[Zope-dev] CompressedStorage

[Ty Sarna]

In article <Pine.LNX.4.21.0105311941160.32029-100000@bcryachts.atsat.com>,
Erik Enge  <erik@thingamy.net> wrote:
> Hi,
> 
> has anyone given this a good run?  I'm a bit confused as to how to make it
> work.  Do I just subclass it in FileStorage?

No, you wrap it around an underlying storage (such as FileStorage).

When Zope starts up, it looks for the module custom_zodb.py to figure
out what storage to use.  Zope basically uses whatever object
custom_zodb.Storage is as the storage. If there isn't a custom_zodby.py,
Zope acts as if there was one that said [*]:

----
import Globals, ZODB.FileStorage

Storage=ZODB.FileStorage.FileStorage(Globals.BobobaseName)
----

What you want to do is create a FileStorage, and wrap it with a
CompressedStorage and use that. Your custom_zodb.py would look like:

----
import Globals, ZODB.FileStorage, ZODB.CompressedStorage

Storage=ZODB.FileStorage.FileStorage(Globals.BobobaseName)
Storage=ZODB.CompressedStorage.CompressedStorage(Storage, 2048)
----

2048 is the compression threshhold. Anything smaller than that
CompressedStorage won't bother trying to compress. Adjust that number as
you see fit. 1024 might be a better defaul. Note that if it tries to
compress data and it doesn't come out any smaller, it will store it
uncompressed. The threshold decides how big data has to be before it's
worth even trying to make it smaller.

BTW, you should be able to wrap a CompressedStorage around just about
anything, not just a FileStorage. You can have a compressed
BerkeleyStorage or whatever.

[*] this is simplified for ease of explanation... it doesn't really work
this way.

> Will it work with existing
> Data.fs or do I need to start anew, so to speak?

See the comments at the start of CompressedStorage.py. You should be
able to layer it over an existing storage (such as an existing
FileStorage Data.fs). Only newly-written objects will be compressed (if
they're big enough). Of course, once you have compressed objects in the
Data.fs, you have to keep using CompressedStorage. So yes, you can
switch, but you can't switch back :-)

BTW, DC people... you should feel free to include CompressedStoage in
Zope.

Also, I'm sure it would be appreciated if anyone (DC or not) took the
time to write a HowTo on custom_zodb.py (hopefully more detailed and
accurate than above)