[ZODB-Dev] Newbie question - Filestorage size ?

[Tim Peters]

[Luis Machado]
> I would like to know how big can a filestorage grow ?

How big does your operating system allow a file to get?  Same answer, at
least on common platforms (Python supports files as large as the OS allows,
for OS in (at least) {Linux, Solaris, Windows}).

> I'm working on an application where I'm only adding data to a ZODB
> database. I'm not modifying existing data. In this application, there
> is one task writing to the database and many clients reading from it.

It's impossible to avoid modifying *some* pre-existing object when you
write.  Everything in ZODB is reachable via some path from the root object,
so adding new data necessarily requires changing *something* on some path
from the root too -- else the new data would be unreachable from the root.
For example, if you store all your data in a single giant BTree B (which is
fine -- BTrees are good for this), then adding a new key+value pair to B
will modify one or more persistent objects that go into the implementation
of B too.  For example, the internal bucket the new pair lives in will grow
via mutation; the bucket may split too if it's gotten "too big"; and
internal nodes of the BTree may grow copies of the new key, and also split.

> Looks like ZODB is appropriate for this application, as I need a Tree
> database due to the nature of the data.

ZODB holds a rooted object graph (more general than a tree).

> If I'm only adding new data and no modifying nor deleting data from the
> database, do I need to regularly pack the database ?

As above, if you add data, you'll also modify objects.  How often you'll
need to pack can't be guessed from what you've said so far.

> What are the chances that the file gets corrupted ? Is this related to
> the size of the file ?

Please read the following.  It's not punishment <wink>, I promise you'll
find it informative:

    http://zope.org/Wikis/ZODB/FileStorageBackup

> Is there a way to read data from a compressed backuped file ?

Not with any storage implementation Zope Corp supplies, no.  I vaguely
recall hearing about someone writing a ZODB storage that did compress, but
maybe they were just speculating, and maybe I'm hallucinating.