[ZODB-Dev] Finding objects by attribute value?

[Tim Peters]

[Leif K-Brooks]
> The speed of a brute-force search isn't terrible when I'm using
> FileStorage; actually, it seems a bit faster than an equivalent MySQL
> search. The big performance problem comes when using ZEO, since each and
> every object has to be transported across the network.

This suggests to me that you're dealing with relatively small databases,
and/or a relatively small collection of objects.  Unpickling N objects to
find M of them matching some criterion obviously doesn't scale as N grows
ever larger than M, ZEO or not.  Indexing does scale, although it's not even
easy to craft scalable indexing schemes.

If your object collection is small enough, you could boost the ZODB cache
size on your ZEO client, and keep all your objects in the client's RAM.
Then brute force will run as fast as if you weren't using ZEO (although
objects will get refetched across the network when other clients modify
them).

> What would be really neat is if I could somehow ask the ZEO server to
> perform a brute-force search of the OOBTree containing my objects and
> only send me the relevant objects. Is there any way to do that?

AFAIK, there is not.  For the most part, a ZEO server doesn't know about
objects, just object ids, transaction ids, and raw (opaque; uninterpreted)
object pickles.  For example, it's not normally expected that the class code
implementing your persistent objects is even available on a ZEO server.  So
there's no code now in a ZEO server that could be told (even in theory) to,
e.g., "look at the `attr` attribute of the values in such-and-such a BTree"
-- it doesn't have an expectation that it *can* unpickle objects of
user-defined classes, so rarely ever tries to (user-defined conflict
resolution methods are an exception, and the only one I can think of).

If you were willing to increase the load on your ZEO server (I doubt most
users would be), you could write a distinct server program to open a direct
ZODB read-only connection to your FileStorage, with layers of network cruft
to accept and service brute-force search requests.  You'd also need to
ensure that the code implementing your classes was available to this program
(not normally needed on a ZEO server box); worry about concurrent mutations
to objects while the brute-force search was running; and so on.  Lots of
nasty details, and in the end it wouldn't scale either.