[ZODB-Dev] End-of-line translation problem

[Jim Fulton]

(Sorry for the cross posting, but I think that all of the lists have an important
  stake in this issue.)

In the text below, I respond to a report from Tim on the fact that, unlike CVS,
subversion doesn't do any end-of-line translation for text files.  If someone
creates a file on a Unix-like system, it will have Unix line endings. If someone
creates a file on Windows, it will have windows line endings.  Depending on tools
used, we could end up with files having mixed line endings.  This has happened to
us in the past when people using SMB used windows editors to edit files checked
out and checked in using Linux CVS clients (using SMB). When this has happened,
people found it really annoying.

Subversion has a property that can be set on a file to cause end-of-line translation
to take place.  I have argued that the setting of this property should be automated
and that the standard ways to do so with subversion are inadequate.

I started a thread on the subversion users mailing list explaining my
position and describing a work around that Tim suggested to me:

   http://subversion.tigris.org/servlets/BrowseList?list=users&by=thread&from=184061

One of the subversion developers responded suggesting:

- File adds are rare

- Ad hoc mechanisms for setting the subversion eol-style property have worked
   for the subversion project itself. (Text files that I have looked at in the
   subversion repository do, indeed, have this property set.)

It could be argued that we should try subversion's ad hoc approach to
prove that, in fact, it doesn't work for us.

For myself, I'm inclined to do what Tim suggested,
and tell subversion to automatically set the eol-style for
all files to native. This will fail loudly when binary files
are checked in and I think that I can deal with that inconvenience. Note that
this is a big improvement over CVS as generally, binary files will automatically
*not* be treated as text files. (The danger being that the binary detection algorithm
will occasionally fail.)

Thoughts?

If we think that it's worth *trying* the subversion developers ad hoc approach, then I
think we could move forward with the transition sooner, perhaps the week after next.
(I still need to arrange for setting the property during conversion, but I think
I can manage that.)

Jim




Jim Fulton wrote:
> Tim Peters wrote:
> 
>> [Jim Fulton]
>> ...
>>
>>> You will be able to do read-only anonymous checkouts like so:
>>>
>>>   svn co svn://svn.zope.org/repos/main/<project>/trunk
>>>
>>> For example:
>>>
>>>   svn co svn://svn.zope.org/repos/main/ZConfig/trunk
>>
>>
>>
>> FYI, I tried that on Windows (XP), and it worked fine.
>>
>> One glitch, which may be all over the place:  some of the "text files" 
>> got
>> checked out with Windows line ends, but most did not.  For example, 14 of
>> the 19 *.txt files in ZConfig ended up with Windows line ends, but 
>> none of
>> the 37 *.py files did.
>>
>> Ack, no, none of the checked-out .txt files did either.  The .txt 
>> files that
>> had Windows line ends were all created by svn for its own purposes
>> (README.txt files in .svn directories).
>>
>> I'm not sure what to do about this.  Best I can tell from the docs so 
>> far,
>> svn wants a
>>
>>     svn:eol-style
>>
>> property added to every line-oriented file, with value
>>
>>     native
>>
>> in order to get platform-sane line-end conversions.  The doc's 
>> explanation
>> of the effect of that matches my understanding of what CVS does for all
>> non-binary files, which is usually exactly right.
>>
>> I noticed that Fredrik Lundh complained about something similar here:
>>
>>     http://effbot.org/zone/subversion.htm
>>     ...
>>     Properties are nice, but having to use three different commands
>>     to check in a text file from Windows is pretty annoying.
>>
>> Looks like svn *expected* us to do this by setting enable-auto-props 
>> during
>> the intial imports, with a bunch of [auto-props] settings in a config 
>> file;
>> like
>>
>> """
>> [auto-props]
>> *.c = svn:eol-style=native
>> *.cpp = svn:eol-style=native
>> *.h = svn:eol-style=native
>> *.py = svn:eol-style=native
>> *.dsp = svn:eol-style=CRLF
>> *.dsw = svn:eol-style=CRLF
>> *.sh = svn:eol-style=native;svn:executable
>> *.txt = svn:eol-style=native
>> *.png = svn:mime-type=image/png
>> *.jpg = svn:mime-type=image/jpeg
>> """
> 
> 
> I found this to be so unbelievable, that I had to resoearch it myself.
> After looking this up in the book and expressing my amazement on the #svn
> channel (and recieving confirmation from svn developers there), I have to
> admit that you are right. I know better than to doubt you, but this is 
> just so
> unbelievable, I couldn't help it.
> 
> 
>> I think we'll have to develop a standard set of config file settings like
>> that for committers to add to their personal svn configs --
> 
> 
> I don't think that this is practical. I think it will be very hard to 
> communicate
> this to everyone.  Plus, every time someone comes up with a new dang 
> file suffix,
> everyone will have to update their config files.
> 
> I think the "real" answer, the answer that the svn (and arch) developers 
> believe
> in the heart of hearts is that windows users should be using tools that 
> understand,
> well, understand and always produce Unix line endings.
> 
> Is it practical to require windows users to use tools that understand 
> and produce
> Unix line endings?
> 
>  > or can that be
> 
>> done on the server side?
> 
> 
> I suppose it could.  I think that a post-commit script could inspect new 
> files and,
> for any new file that has no mine-type property, or has one with a text 
> type,
> set the svn:eol-style proprty to native. It would have to do this in a 
> separate
> transaction.
> 
> Does anyone want to volunteer to write this script?
> 
> We'll also need to fix cvs2svn to do something similar.
> 
> Jim
> 


-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org