[ZODB-Dev] standalone ZODB for large Numeric arrays

[Mike Mueller]

At 22:09 28.02.03 +0100, you wrote:
[Kaspar]


>What is the main reason you want this?
>Would using netCDF from inside ZODB really be faster than what ZODB does now?

I haven't really measured yet. But netCDF is designed for large arrays and 
fast access.
If ZODB.FileStorage files are big (> 1GB) it takes quite a while to open 
them (30 seconds, 2.2 GHZ Pentium 4, 1 GB RAMBUS). A netCDF file opens just 
like a ordinary file, at least in terms of speed. Also, since ZODB makes 
undo possible, files get large rather quickly. With netCDF I can use large 
file support (> 2GB) or just distribute the data over several files using 
ZODB to keep track in which netCDF file things are stored.


>How do you want to use the netCDF functionality? From inside Python or from
>other software?

Right now I use netCDF from FORTRAN also. Actually, I put arrays in netCDF 
on a RAM disk from FORTAN and read them back in Python and vice versa. This 
is something like a compiler independent data exchange. The overhead is an 
additional copy plus some netCDF operations.



>BTW, I just started using ZODB today, I'm also using it for numerical
>calculations and I really think it kicks ass!


I agree. It's very convenient. For instance, I use  mx.DateTime objects as 
keys to store data at different dates. No need to convert the date into 
seconds since some starting point or something like this.

Mike