[ZODB-Dev] Using zodb and blobs
Tres Seaver
tseaver at palladion.com
Mon Apr 12 15:42:19 EDT 2010
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Nitro wrote:
> After lots of googling and browsing the source I can answer some of my
> questions:
>
>> - What's the difference between storing bigger objects as blobs and as
>> plain large strings?
>
> Plain large strings cannot be streamed for instance. Products like Zope
> chop up their file uploads into 64kb chunks which are then stored as
> individual objects in the zodb.
That was the strategy before blobs. ZODB versions since 3.8 support
storage of BLOb data as files on the filesystem.
>> - Can I stream in parts of a blob/large string without having to read all
>> of it?
>
> I can get a file handle to a blob. Strings are always read as a whole.
>
>> - Where can I find example code on zodb blobs? E.g. how do I save a blob,
>> how do I read it back in?
>
> The ZODB/tests directory features a few blob doctests which provide all
> the necessary code to get started. Having this on zodb.org would be nice
> (especially since the doctests are already ReST-formatted).
>
> Additionally I made some quick performance tests. I committed 1kb sized
> objects and I can do about 40 transaction/s if one object is changed per
> transaction. For 100kb objects it's also around 40 transactions/s. Only
> for object sizes bigger than that the raw I/O throughput seems to start to
> matter.
40 tps sounds low: are you pushing blob content over the wire somehow?
> Still don't know the answers to these:
>
> - Does it make sense to use ZODB in this scenario? My data is not suited
> well for an RDBMS.
YMMV. I still default to using ZODB for anything at all, unless the
problem smells very strongly relational.
> - Are there more complications to blobs other than a slightly different
> backup procedure?
You need to think about how the blob data is shared between ZEO clients
(your appserver) and the ZEO storage server: opinions vary here, but I
would prefer to have the blobs living in a writable shared filesystem,
in order to avoid the necessity of fetching their data over ZEO on the
individual clients which were not the one "pushing" the blob into the
database.
> - Is it ok to use cross-database references? Or is this better avoided at
> all cost?
I would normally avoid them out of habit. They seem to work, though.
> And new questions:
>
> - Does the _p_invalidate hooking as outlined at
> http://www.mail-archive.com/zodb-dev@zope.org/msg00637.html work reliably?
Never tried it, nor felt the need.
> - Are there any performance penalties by using very large invalidation
> queues (i.e. 300,000 objects) to reduce client cache verification time?
At a minimum, RAM occupied by that queue might be better used elsewhere.
I just don't use persistent caches, and tend to reboot appservers in
rotation after the ZEO storage has been down for any significant period
(almost never happens).
> From what I've read it only seems to consume memory.
Note that the ZEO storage server makes copies of that queue to avoid
race conditions.
Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 tseaver at palladion.com
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkvDd5sACgkQ+gerLs4ltQ5h4wCghqTQsNO/5XrqHMZyhi8Hz17d
oRcAn1el86604KoTTWB8Bx5R13ZlvQB/
=momg
-----END PGP SIGNATURE-----
More information about the ZODB-Dev
mailing list