>2GB Data.fs files on FreeBSD
OK, I've seen it bandied about that Linux doesn't support large files but FreeBSD does, and to run Zope on FreeBSD if you want a large database file. So how do you do that? On my BSDI box I started getting "File too large" errors when the file exceeded some magic number around 2GB. I moved the file to a FreeBSD box hoping to be able to trim it there, but I get an error when I try to start Zope: Traceback (innermost last): File "/usr/local/zope/Zope-2.1.2-src/z2.py", line 436, in ? exec "import "+MODULE in {} File "<string>", line 1, in ? File "/usr/local/zope/Zope-2.1.2-src/lib/python/Zope/__init__.py", line 109, in ? DB=ZODB.FileStorage.FileStorage(Globals.BobobaseName) File "/usr/local/zope/Zope-2.1.2-src/lib/python/ZODB/FileStorage.py", line 301, in __init__ self._pos, self._oid, tid = read_index( File "/usr/local/zope/Zope-2.1.2-src/lib/python/ZODB/FileStorage.py", line 1231, in read_index if file_size < start: raise FileStorageFormatError, file.name ZODB.FileStorage.FileStorageFormatError: /usr/local/zope/Zope-2.1.2-src/var/Data.fs So it looks like there is a problem using Zope with a large database no matter what the platform. Has anyone figured out how to fix this? My main goal is to be able to get the last transactions out of this database, after which I'll be able to shrink it well under the 2BG limit...I'm going to try just truncating the last few bytes to see if I can get Zope to open it and let me recover most of the transactions. (Side note: tranalyzer dies before it processes the whole file, probably some sort of memory resource limit on my server.) But given the number of people who have said "use FreeBSD if you want big files", I'm really wondering about this. What if later I have an application where I really need a >2GB database? --RDM
[Cc'ed to python-dev from the zope-dev mailing list; trim your follow-ups appropriately] R. David Murray writes:
So it looks like there is a problem using Zope with a large database no matter what the platform. Has anyone figured out how to fix this? ... But given the number of people who have said "use FreeBSD if you want big files", I'm really wondering about this. What if later I have an application where I really need a >2GB database?
Different system calls are used for large files, because you can no longer use 32-bit ints to store file position. There's a HAVE_LARGEFILE_SUPPORT #define that turns on the use of these alternate system calls; see Python's configure.in for the test used to detect when it should be turned on. You could just hack the generated config.h to turn on large file support and recompile your copy of Python, but if the configure.in test is incorrect, that should be fixed. The test is: AC_MSG_CHECKING(whether to enable large file support) if test "$have_long_long" = yes -a \ "$ac_cv_sizeof_off_t" -gt "$ac_cv_sizeof_long" -a \ "$ac_cv_sizeof_long_long" -ge "$ac_cv_sizeof_off_t"; then AC_DEFINE(HAVE_LARGEFILE_SUPPORT) AC_MSG_RESULT(yes) else AC_MSG_RESULT(no) fi I thought you have to use the loff_t type instead of off_t; maybe this test should check for it instead? Anyone know anything about large file support? -- A.M. Kuchling http://starship.python.net/crew/amk/ When I dream, sometimes I remember how to fly. You just lift one leg, then you lift the other leg, and you're not standing on anything, and you can fly. -- Chloe Russell, in SANDMAN #43: "Brief Lives:3"
On Thu, 13 Apr 2000, Andrew M. Kuchling wrote:
longer use 32-bit ints to store file position. There's a HAVE_LARGEFILE_SUPPORT #define that turns on the use of these alternate system calls; see Python's configure.in for the test used to
I just looked in my python config.h on my FreeBSD system, and I see: #define HAVE_LARGEFILE_SUPPORT 1 So it looks like it is on, and it seems to me the problem could be in either Python or FileStorage.py in Zope. This is a Zope 2.1.2 system (but I diffed filestorage.py against the 2.1.6 version and didn't see any relevant changes) running on a FreeBSD 3.1 system. A make test in Python passed all tests, but I don't know if large file support is tested by the tests. --RDM PS: anyone from the python list replying to this please CC me as I am not on that list.
Hello, Sticking my neck out here as I am not really knowledgeable in this area and have never used BSD and currently don't even have a Linux install. :( Since Zope bundles Python for it's own use, is the Python your looking at the Python Zope is using? If not you might adjust your config.h for the Python Zope is using? Just a thought. Lop my head off if I'm making a fool of myself here. :) Jimmie Houchin "R. David Murray" wrote:
On Thu, 13 Apr 2000, Andrew M. Kuchling wrote:
longer use 32-bit ints to store file position. There's a HAVE_LARGEFILE_SUPPORT #define that turns on the use of these alternate system calls; see Python's configure.in for the test used to
I just looked in my python config.h on my FreeBSD system, and I see:
#define HAVE_LARGEFILE_SUPPORT 1
So it looks like it is on, and it seems to me the problem could be in either Python or FileStorage.py in Zope.
This is a Zope 2.1.2 system (but I diffed filestorage.py against the 2.1.6 version and didn't see any relevant changes) running on a FreeBSD 3.1 system. A make test in Python passed all tests, but I don't know if large file support is tested by the tests.
--RDM
PS: anyone from the python list replying to this please CC me as I am not on that list.
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
On Fri, 14 Apr 2000, Jimmie Houchin wrote:
Since Zope bundles Python for it's own use, is the Python your looking at the Python Zope is using?
If not you might adjust your config.h for the Python Zope is using?
Good question, but I don't a Zope/Python bundle, I install them separately from source. There is only one Python instance on my system. I'm punting on this problem for now, but hopefully will be able to come back to it later. --RDM
OK, some more info. The code in FileStorage.py looks like this: ------------------- def read_index(file, name, index, vindex, tindex, stop='\377'*8, ltid=z64, start=4, maxoid=z64): read=file.read seek=file.seek seek(0,2) file_size=file.tell() print file_size, start if file_size: if file_size < start: raise FileStorageFormatError, file.name [etc] ------------------- I stuck that print statement in there. The results of the print are: -2147248811L 4 So it looks to my uneducated eye like file.tell() is broken. The actual on-disk size of the file, by the way, is indeed 2147718485, so it looks like somebody's not using the right size data structure somewhere. So, can anyone tell me what to look for, or am I stuck for the moment? --RDM PS: anyone on pthon-dev replying please CC me as I am only on the zope list.
R. David Murray writes:
So it looks to my uneducated eye like file.tell() is broken. The actual on-disk size of the file, by the way, is indeed 2147718485, so it looks like somebody's not using the right size data structure somewhere.
Look at Objects/fileobject.c, at the implementation for tell(). (The Python source code is nicely organized; don't be afraid to dive into it.) #if defined(HAVE_FTELLO) && defined(HAVE_LARGEFILE_SUPPORT) offset = ftello(f->f_fp); #elif defined(HAVE_FTELL64) && defined(HAVE_LARGEFILE_SUPPORT) offset = ftell64(f->f_fp); #else offset = ftell(f->f_fp); #endif So, presumably neither HAVE_FTELLO nor HAVE_FTELL64 are defined, so it uses the old ftell(). What's the right system call for FreeBSD? Looking at the man pages for 4.0 at freebsd.org, it's ftello(). So, is HAVE_FTELLO defined? If it isn't, then find out why; if it is, then there's some other bug in Python's tell() code. -- A.M. Kuchling http://starship.python.net/crew/amk/
From gotos to the evolution of life in 10 posts; that's comp.lang.python for you! -- A.M. Kuchling, 4 Apr 1998
On Thu, 13 Apr 2000, Andrew M. Kuchling wrote:
So, presumably neither HAVE_FTELLO nor HAVE_FTELL64 are defined, so it uses the old ftell(). What's the right system call for FreeBSD? Looking at the man pages for 4.0 at freebsd.org, it's ftello(). So, is HAVE_FTELLO defined? If it isn't, then find out why; if it is, then there's some other bug in Python's tell() code.
I may persue your suggestions at some point, but I just tried running Zope on a version of the Data.fs truncated to about 700K under 2G. It still hasn't finished trying to load (I'm about to kill it) after half an hour, and it periodically is dumping the following traceback to the screen (-D mode): ------------- Traceback (innermost last): File "/usr/local/zope/Zope-2.1.2-src/z2.py", line 436, in ? exec "import "+MODULE in {} File "<string>", line 1, in ? File "/usr/local/zope/Zope-2.1.2-src/lib/python/Zope/__init__.py", line 109, in ? DB=ZODB.FileStorage.FileStorage(Globals.BobobaseName) File "/usr/local/zope/Zope-2.1.2-src/lib/python/ZODB/FileStorage.py", line 301, in __init__ self._pos, self._oid, tid = read_index( File "/usr/local/zope/Zope-2.1.2-src/lib/python/ZODB/FileStorage.py", line 1267, in read_index if tl+pos+8 > file_size or status=='c': OverflowError: integer addition ------------- So either Python or Zope has some more serious problems with the 2G limit than just what tell is returning. So unless you tell me I'm doing something stupid that is causing this <grin>, for the moment I'm going to punt and just recover all the data I can out of an even more truncated Data.fs. --RDM
R. David Murray writes:
So it looks to my uneducated eye like file.tell() is broken. The actual on-disk size of the file, by the way, is indeed 2147718485, so it looks like somebody's not using the right size data structure somewhere.
So, can anyone tell me what to look for, or am I stuck for the moment?
Hmm. What is off_t defined to be on your platform? In config.h, is HAVE_FTELLO or HAVE_FTELL64 defined? -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> Corporation for National Research Initiatives
participants (4)
-
Andrew M. Kuchling -
Fred L. Drake, Jr. -
Jimmie Houchin -
R. David Murray