I'm using Zope with python and i'm having problems due to my Data.fs file is over 2GB. According to Python's Library Reference 8.1.1 I have to re-compile python with this sentence: CC="-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" export CC ./configure 1- I'm using Red HAt Linux 6.2 How do I know if it's large-file-capable ? 2- How do I re-compile python as described in 8.1.1, there's any step-by-step guide ? Thanks Alejandro Singer
I'm using Zope with python and i'm having problems due to my Data.fs file is over 2GB. According to Python's Library Reference 8.1.1 I have to re-compile python with this sentence: CC="-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" export CC ./configure 1- I'm using Red HAt Linux 6.2 How do I know if it's large-file-capable ? 2- How do I re-compile python as described in 8.1.1, there's any step-by-step guide ? ------------------------ Is this true of all Zope/Python versions ( maximum data base file size of 2GB)???
At 17:28 23-07-2001 -0400, you wrote:
I'm using Zope with python and i'm having problems due to my Data.fs file is over 2GB. According to Python's Library Reference 8.1.1 I have to re-compile python with this sentence:
CC="-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" export CC ./configure
1- I'm using Red HAt Linux 6.2 How do I know if it's large-file-capable ? 2- How do I re-compile python as described in 8.1.1, there's any step-by-step guide ?
------------------------
Is this true of all Zope/Python versions ( maximum data base file size of 2GB)???
AFAIK it isn't a python/Zope problem but a filesystem problem. I remember to have heard somewhere that Ext2 (the filesystem that Linux uses) only support a file size of 2 GB. Perhaps some of the guru's inhere can correct me if I am wrong. The solution to this problem - I think - may be to switch your filesystem from Ext2 to ReiserFS. Regards, Gitte Wange
AFAIK it isn't a python/Zope problem but a filesystem problem. I remember to have heard somewhere that Ext2 (the filesystem that Linux uses) only support a file size of 2 GB. Perhaps some of the guru's inhere can correct me if I am wrong.
The solution to this problem - I think - may be to switch your filesystem from Ext2 to ReiserFS.
Ext2 is fine with large files. The problem lies in the virtual memory system of Linux 2.2 or lesser kernels (by way of the C libraries, I believe). With a 2.4.x kernel, ext2 can deal with very large files. But I would suggest going to a journaling filesystem anyway. My favorite is SGI's XFS. It can be a pain if you don't have tons of empty disk space, but it's fun to kick over your machine while it's running and get a fsck in milliseconds. --jcc (no more fscking)
On 23 Jul 2001 17:14:30 -0500, J. Cameron Cooper wrote:
AFAIK it isn't a python/Zope problem but a filesystem problem. I remember to have heard somewhere that Ext2 (the filesystem that Linux uses) only support a file size of 2 GB. Perhaps some of the guru's inhere can correct me if I am wrong.
The solution to this problem - I think - may be to switch your filesystem from Ext2 to ReiserFS.
Ext2 is fine with large files. The problem lies in the virtual memory system of Linux 2.2 or lesser kernels (by way of the C libraries, I believe). With a 2.4.x kernel, ext2 can deal with very large files.
But I would suggest going to a journaling filesystem anyway. My favorite is SGI's XFS. It can be a pain if you don't have tons of empty disk space, but it's fun to kick over your machine while it's running and get a fsck in milliseconds.
Not to mention you can format a 1 terabyte filesystem in under one second. Literally. :) In testing, I have found problems with ReiserFS under heavy (and I mean heavy!) load, so I, to, reccommend XFS. Oh, and recovery in the seconds range is wonderful. On that same teraybte. :) Bill
1- I'm using Red HAt Linux 6.2 How do I know if it's large-file-capable
At least Redhat 7.X comes with a Python version that has no largefile support.
2- How do I re-compile python as described in 8.1.1, there's any step-by-step guide ?
http://python.sourceforge.net/maint-docs/lib/posix-large-files.html
Is this true of all Zope/Python versions ( maximum data base file size of 2GB)???
You can use Python and Zope with a Data.fs larger than 2GB but your Python need compiled with largefile support (see above). Andreas
Is this true of all Zope/Python versions ( maximum data base file size of 2GB)???
You can use Python and Zope with a Data.fs larger than 2GB but your Python need compiled with largefile support (see above). ---- OK I will be more specific -- is this true of all distributed Zope versions ( including Windows?) -- if so why? -- it seems that Zope doesn't offer a lot of scalability out of the box if this is the case.
On Mon, 23 Jul 2001 17:28:44 -0400, "Brian Sullivan" <brians@meetingbywire.com> wrote:
According to Python's Library Reference 8.1.1 I have to re-compile python with this sentence:
CC="-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" export CC ./configure
There was something on comp.lang.python last week about this command line containing a typo..... Toby Dickenson tdickenson@geminidataloggers.com
1- I'm using Red HAt Linux 6.2 How do I know if it's large-file-capable ?
I think that large files are only available with Linux 2.4.x kernels, which I think rules out RH6.2 unless you've done something special. The sure-fire test, of course, is to create a > 2Gb file and try to see its size. If you don't get an accurate reading, your system doesn't do +2Gb files.
2- How do I re-compile python as described in 8.1.1, there's any step-by-step guide
The Python docs say that there are complete compiling instructions in the README file in the source distribution. My guess is that it goes something like: ./configure && make && make install --jcc (make install)
1- I'm using Red HAt Linux 6.2 How do I know if it's large-file-capable ?
I think that large files are only available with Linux 2.4.x kernels, which I think rules out RH6.2 unless you've done something special.
RH 6.2 comes out of the box with Ext2 file support, which limits individual file size to 2GB.... in fact RH7.1 install does the same (though the kernel can support other file systems). Check out http://oss.sgi.com/projects/xfs/1.0_release.html for a really good one, and easy to use and install.
The sure-fire test, of course, is to create a > 2Gb file and try to see its size. If you don't get an accurate reading, your system doesn't do +2Gb files.
Actually, not a sure fire test, since some of the tools for listing file sizes (especially: GNOME) still choke. From BASH, you can accurately see the sizes.
On 23 Jul 2001 18:26:14 -0400, marc lindahl wrote:
1- I'm using Red HAt Linux 6.2 How do I know if it's large-file-capable ?
I think that large files are only available with Linux 2.4.x kernels, which I think rules out RH6.2 unless you've done something special.
RH 6.2 comes out of the box with Ext2 file support, which limits individual file size to 2GB.... in fact RH7.1 install does the same (though the kernel can support other file systems). Check out
Not True. Once again, (wink) it is the VFS layer, NOT ext2. And furthermore, it is the ext2 layer on intel 32 bit machines. Any Linux FS on any kernel less than 2.4, without a special, mostly operational patch, will not do >2GB files.
http://oss.sgi.com/projects/xfs/1.0_release.html for a really good one, and easy to use and install.
While I love XFS on Linux, You do not need it to get LF support. it is in the 2.4 kernel's newer VFS.
The sure-fire test, of course, is to create a > 2Gb file and try to see its size. If you don't get an accurate reading, your system doesn't do +2Gb files.
Actually, not a sure fire test, since some of the tools for listing file sizes (especially: GNOME) still choke. From BASH, you can accurately see the sizes.
A recompile with the newer glibc and Kernel fixes most of these issues, though I can not at this time, say it fixes them all. (I can say I have yet to have any issues with 15 or even 500 GB files on RH 7.1 with XFS. Bill
From: Bill Anderson <bill@libc.org>
Not True. Once again, (wink) it is the VFS layer, NOT ext2. And
Hmmm.... I didn't notice this option when installing 7.1? ...doing the normal install routine, you partition (or let it do that for you), but I don't see any options for what filesystem to use...?
While I love XFS on Linux, You do not need it to get LF support. it is in the 2.4 kernel's newer VFS.
No, you don't need it, but it's the only one I found with a easy to use installer... since I'm not a unix expert that part was important to me!
On 24 Jul 2001 11:40:53 -0400, marc lindahl wrote:
From: Bill Anderson <bill@libc.org>
Not True. Once again, (wink) it is the VFS layer, NOT ext2. And
Hmmm.... I didn't notice this option when installing 7.1? ...doing the normal install routine, you partition (or let it do that for you), but I don't see any options for what filesystem to use...?
Sorry, I was not clear. The VFS is the filesystem layer in Linux. The problem is _not_ filesystem specific, it was a problem specifically with intel 32 bit hardware, runniong on kernels previous to 2.4. There was an LFS patch (LFS == Large FIle Support), but it was buggy. If you are running 7.1, your kernel, regardless of filesystem, can deal with large files. No additional effor trequired.
While I love XFS on Linux, You do not need it to get LF support. it is in the 2.4 kernel's newer VFS.
No, you don't need it, but it's the only one I found with a easy to use installer... since I'm not a unix expert that part was important to me!
Actually, the reiserfs installer patch is a helluva lot simpler, since it is a floppy update to the installer. :) I still prefer XFS. Bill
From: Bill Anderson <bill@libc.org> No additional effor trequired.
Well, for a newbie, the effort is figuring out how to format the partitions! :)
Actually, the reiserfs installer patch is a helluva lot simpler, since it is a floppy update to the installer. :)
I heard from someone who worked at an (ahem... unix-based) OS company that reiserfs is flaky... that's why I was looking for something else. There are a couple others out there, but SGI has been working on XFS for years, and it's operating in production systems, so I have confidence that it's rock solid. but anyway, with SGI's installer CD, installing RH7.1 with XFS couldn't be easier -- you boot with the XFS disk and start the installation process with that, then eventually it asks you to insert RH CD#1 and #2... viola! done.
"Alejandro T. Singer" wrote:
I'm using Zope with python and i'm having problems due to my Data.fs file is over 2GB.
Personally, I'd use PartionedFileStorage for this... cheers, Chris PS: Less HTML, we don't need the subject line to be all uppercase to read it ;-)
Hi Alejandro, it is not so hard as it looks like at the start. You should get Python as source and Zope as source. There is always a README and mostly an INSTALL file too which describes step by step how to build and install. In short, you set the CC var as you described, go to the directory you unpacked python, then run: ./configure --with-thread make make test make install (ok, you can omit 'make test') then go to the zope source and python (<--- make sure you get _your_ new configured python here, not an older one on your system. If in doubt use the full path, e.g. /usr/local/bin/python ) python wo_pcgi.py after running it, call python zpasswd.py inituser then ./start Good luck Tino --On Montag, 23. Juli 2001 18:02 -0300 "Alejandro T. Singer" <alejandro.singer@homenajear.com> wrote:
I'm using Zope with python and i'm having problems due to my Data.fs file is over 2GB. According to Python's Library Reference 8.1.1 I have to re-compile python with this sentence:
CC="-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" export CC ./configure
1- I'm using Red HAt Linux 6.2 How do I know if it's large-file-capable ? 2- How do I re-compile python as described in 8.1.1, there's any step-by-step guide ?
Thanks
Alejandro Singer
participants (10)
-
Alejandro T. Singer -
Andreas Jung -
Bill Anderson -
Brian Sullivan -
Chris Withers -
Gitte Wange -
J. Cameron Cooper -
marc lindahl -
Tino Wildenhain -
Toby Dickenson