Re: [Zope] FYI: large files & linux.
tsarna@endicor.com (Ty Sarna) writes:
Even simpler than buying new hardware, switch to a system like NetBSD or FreeBSD that has supported large files for years and years :-)
It supports it to no greater or less extent than Linux does. The standard library defaults to using an int or a long for *seek calls. The difference is semantic only, as they're both 32 bits on both systems anyway. Even if you assume unsigned (fseek is signed), that's still 4gb, max. However, on my Linux/Alpha system, a long is 64 bits. This effectively removes the problem. I submit, therefore, that you are not seeing a Linux problem, but an x86 problem :-)
In article <87n1oazuao.fsf@erwin.complete.org>, John Goerzen <jgoerzen@complete.org> wrote:
It supports it to no greater or less extent than Linux does. The
Untrue.
standard library defaults to using an int or a long for *seek calls. The difference is semantic only, as they're both 32 bits on both systems anyway. Even if you assume unsigned (fseek is signed), that's still 4gb, max.
The kernel internals of Linux use a 32 bit off_t, hence discussion of large file support in the next kernel release.
I submit, therefore, that you are not seeing a Linux problem, but an x86 problem :-)
I'm not seeing an x86 problem at all: lukasiewicz$ uname -sp NetBSD i386 lukasiewicz$ ls -l huge -rw-r--r-- 1 tsarna users 17592186044415 Mar 7 12:41 huge Granted, that's a sparse file (I can't afford that much disk space :^), but it demonstrates that file offsets are not a problem. As mentioned, NetBSD's off_t has been a 64 bit type for about 6 years now, even on i386 (and evey other platform). In fact, off_t was made 64 bit in 1994, and NetBSD/alpha didn't exist until 1995, so it was done before there were any native 64 bit platforms supported. :-) However, it turns out that for Zope use, at least with FileStorage, there is a problem, which turns out to be even an x86 issue but a silly ANSI C spec issue. Unfortunaly Python large file support is broken on (32-bit CPU) NetBSD as a result -- I had to make that sparse file with a little C program. Now, you could have a more than 4G BerkeleyStorage Zope system, since the file handling isn't done at the Python level in that case, but FileStorage won't work until I can work around the Python problem (or until NetBSD 1.5, which will fix it in a different way). But at least it's the kind of thing that could be hacked around in an application by one guy in a day, rather than taking drastic kernel and libc changes :-)
Ty Sarna wrote:
In article <87n1oazuao.fsf@erwin.complete.org>, John Goerzen <jgoerzen@complete.org> wrote:
It supports it to no greater or less extent than Linux does. The
Untrue.
standard library defaults to using an int or a long for *seek calls. The difference is semantic only, as they're both 32 bits on both systems anyway. Even if you assume unsigned (fseek is signed), that's still 4gb, max.
The kernel internals of Linux use a 32 bit off_t, hence discussion of large file support in the next kernel release.
I submit, therefore, that you are not seeing a Linux problem, but an x86 problem :-)
I'm not seeing an x86 problem at all:
However, it turns out that for Zope use, at least with FileStorage, there is a problem, which turns out to be even an x86 issue but a silly ANSI C spec issue. Unfortunaly Python large file support is broken on (32-bit CPU) NetBSD as a result -- I had to make that sparse file with a little C program. Now, you could have a more than 4G BerkeleyStorage Zope system, since the file handling isn't done at the Python level in that case, but FileStorage won't work until I can work around the Python problem (or until NetBSD 1.5, which will fix it in a different way).
But at least it's the kind of thing that could be hacked around in an application by one guy in a day, rather than taking drastic kernel and libc changes :-)
Still it would be sensible to do it by splitting the database file (say at 1M boundaries rather than supporting some obscure non-standard system calls that only few platforms support. Probably also one-day hack by someone who knows where to start digging. At least this is how it was solved for PostgreSQL . ------------- Hannu
On Wed, 8 Mar 2000, Hannu Krosing wrote:
Still it would be sensible to do it by splitting the database file (say at 1M boundaries rather than supporting some obscure non-standard system calls that only few platforms support. Probably also one-day hack by someone who knows where to start digging.
At least this is how it was solved for PostgreSQL .
If you remember, it wasn't done in a day. There was a long period of bugs, esp. bugs in VACUUM. I wouldn't buy your idea of easy splitting Data.fs. Exactly like with the VACUUM, packing Data.fs would be complex task if the data will be splitted across multiple files... Oleg. ---- Oleg Broytmann Foundation for Effective Policies phd@phd.russ.ru Programmers don't die, they just GOSUB without RETURN.
Oleg Broytmann wrote:
On Wed, 8 Mar 2000, Hannu Krosing wrote:
Still it would be sensible to do it by splitting the database file (say at 1M boundaries rather than supporting some obscure non-standard system calls that only few platforms support. Probably also one-day hack by someone who knows where to start digging.
At least this is how it was solved for PostgreSQL .
If you remember, it wasn't done in a day. There was a long period of bugs, esp. bugs in VACUUM.
The claim of "a days hacking" was meant as an ironic counterclaim to it being a day's job on *BSD
I wouldn't buy your idea of easy splitting Data.fs. Exactly like with the VACUUM, packing Data.fs would be complex task if the data will be splitted across multiple files...
I would suggest it by implementing a file-like HugeSplitFile class and using that to hide all the splitting from all other layers. If we could implement it in python then it would be perhaps a 1-hour job, leaving the rest of the day for fixing up the parts of zope interfacing with it ;) It would probably consist mostly of adding long() to some initialisation functions. ------------------- Hannu
On Wed, 8 Mar 2000, Hannu Krosing wrote:
The claim of "a days hacking" was meant as an ironic counterclaim to it being a day's job on *BSD
Oops, didn't get it, sorry :)
I would suggest it by implementing a file-like HugeSplitFile class and using that to hide all the splitting from all other layers.
If we could implement it in python then it would be perhaps a 1-hour job, leaving the rest of the day for fixing up the parts of zope interfacing with it ;)
It would probably consist mostly of adding long() to some initialisation functions.
All sounds good... Oleg. ---- Oleg Broytmann Foundation for Effective Policies phd@phd.russ.ru Programmers don't die, they just GOSUB without RETURN.
In article <38C64871.A58F5A72@tm.ee>, Hannu Krosing <hannu@tm.ee> wrote:
1M boundaries rather than supporting some obscure non-standard system calls that only few platforms support. Probably also one-day hack by someone
What "non-standard" syscalls? I'm talking about calls like lseek().
participants (4)
-
Hannu Krosing -
John Goerzen -
Oleg Broytmann -
tsarna＠endicor.com