[ZODB-Dev] what's the latest on zodb/zeo+memcached?

[Jim Fulton]

On Fri, Jan 18, 2013 at 2:38 PM, Claudiu Saftoiu <csaftoiu at gmail.com> wrote:
>
>> I wonder if disk latency is the problem?. As a test you could put the
>> index.fs file into a tmpfs and see if that improves things, or cat
>> index.fs > /dev/null to try and force it into the fs cache.
>
>
> Hmm, it would seem not... the cat happens instantly:
>
>     (env)tsa at sp2772c:~/sports$ time cat Data_IndexDB.fs > /dev/null
>
>     real    0m0.065s
>     user    0m0.000s
>     sys     0m0.064s
>
> The database isn't even very big:
>
>     rw-r--r-- 1 tsa tsa 233M Jan 18 14:34 Data_IndexDB.fs
>
> Which makes me wonder why it takes so long to load it into memory.... it's
> just a bit frustrating that the server has 7gb of RAM and it's proving to be
> so difficult to get ZODB to keep ~300 megs of it up in there. Or, indeed, if
> linux already has the whole .fs file in a memory cache, where are these
> delays coming from? There's something I don't quite understand about this
> whole situation...

Some high-level comments:

- ZODB doesn't simply load your database into memory.

  It loads objects when you try to access their state.

  If you're using ZEO (or relstorage, or neo), each load requires a
  round-trip to the server.  That's typically a millisecond or two,
  depending on your network setup.  (Your database is small, so disk
  access shouldn't be an issue as it is, presumably in your disk
  cache.

- You say it often takes you a couple of minutes to handle requests.
  This is obviously very long.  It sounds like there's an issue
  with the way you're using the catalog.  It's not that hard get this
  wrong.  I suggest either hiring someone with experience in this
  area to help you or consider using another tool, like solr.

  (You could put more details of your application here, but I doubt
  people will be willing to put in the time to really analyze it and
  tell you how to fix it.  I know I can't.)

- solr is so fast it almost makes me want to cry.  At ZC, we're
  increasingly using solr instead of the catalog.  As the original
  author of the catalog, this makes me sad, but we just don't have the
  time to put in the effort to equal solr/lucene.

- A common mistake when using ZODB is to use it like a relational
  database, puting most data in catalog-like data structures and
  querying to get most of your data.  The strength of a OODB is that
  you don't have to query to get data from a well-designed object
  model.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton
Jerky is better than bacon! http://zo.pe/Kqm