[Zope] ZODB and large data sets

[Ty Sarna]

In article <3.0.5.32.19990524102307.00822100@texoma.net>,
Jimmie Houchin  <jhouchin@texoma.net> wrote:
> At 10:48 AM 5/24/99 -0400, Jim Fulton wrote:
> >
> >The file format used by the ZODB 3 FileStorage will also support 
> >very large files on systems with large file support (as described in
> >http://www.python.org/doc/current/lib/posix-large-files.html#l2h-1441).
> 
> At the above URL it mentions these OSes/systems as being capable:
> AIX, HPUX, Irix and Solaris
> 
> Do you know if RedHat Linux running on an AlphaServer such as DS10 would be
> capable?

If you're looking for an open source OS for a server with bigtime
storage requirements and want to avoid silly limits like 2GB files and
127M swap partitions (though maybe that one was fixed in 2.2?), you
might want to look at NetBSD. 

NetBSD has had 64 bit off_t's since 1.0 (1994) on both 32 and 64bit
platforms.  I know of a NetBSD box at NASA Ames with a ~155GB
filesystem.  I think they had one at ~600GB for a while, before
splitting into multiple smaller ones up for testing.  You can store a
single 155GB file on it, if you want (though at the 30MB/s that the
storage it's on gets, it would take you and hour and a half to write
it!).  That doesn't even hit the tripple-indirect-block code (at
~256TB), which has been tested (with sparse files) to work fine.

While I was checking the details on this, someone else chimed in that
he'd played with 18GB files on his machine, and someone made a 21G file
on the spot just for fun :-)

The other BSD's should at least theoretically support this as well, but
I know it's actually been used on NetBSD.