[ZODB-Dev] what's the latest on zodb/zeo+memcached?
Marius Gedminas
marius at gedmin.as
Fri Jan 18 14:02:24 UTC 2013
On Thu, Jan 17, 2013 at 12:31:52PM -0500, Claudiu Saftoiu wrote:
> I wrote the following code to preload the indices:
>
> def preload_index_btree(index_name, index_type, btree):
> print "((Preloading '%s' %s index btree...))" % (index_name,
> index_type)
> start = last_print = time.time()
> for i, item in enumerate(btree.items()):
> item
That's a no-op: you might as well just write 'pass' here.
If you want to load the btree item into cache, you need to do
item._p_activate()
> print "((Preloaded '%s' %s index btree (%d items in %.2fs)))" % (
> index_name, index_type, i, time.time() - start,
> )
If you ever get an empty btree, you'll get an UnboundLocalError: 'i' here.
Drop the enumerate() trick and just use len(btree), it's efficient.
> def preload_catalog(catalog):
> """Given a catalog, touch every persistent object we can find to
> force
> them to go into the cache."""
> start = time.time()
> num_indices = len(catalog.items())
> for i, (index_name, index) in enumerate(catalog.items()):
> print "((Preloading index %2d/%2d '%s'...))" % (i+1,
> num_indices, index_name,)
> preload_index_btree(index_name, 'fwd', index._fwd_index)
> preload_index_btree(index_name, 'rev', index._rev_index)
> print "((Preloaded catalog! Took %.2fs))" % (time.time() - start)
>
> And I run it on server start as follows (modified for the relevant parts; I
> tried to make the example simple but it ended up needing a lot of parts).
> This runs in a thread:
>
> from util import zodb as Z
> from util import zodb_query as ZQ
> for i in xrange(3):
> connwrap = Z.ConnWrapper('index')
> print "((Preload #%d...))" % (i+1)
> with connwrap as index_root:
> ZQ.preload_catalog(index_root.index.catalog)
> connwrap.close()
Every thread has its own in-memory ZODB object cache, but if you have
configured a persistent ZEO client cache, it should help.
Marius Gedminas
--
Never trust a computer you can't repair yourself.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.zope.org/pipermail/zodb-dev/attachments/20130118/9a40bfdc/attachment.sig>
More information about the ZODB-Dev
mailing list