[ZODB-Dev] Re: ZODB + ZEO performances problem

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thierry Florac wrote:

> I'm currently trying to migrate an old Zope-2.6.1 application to
> Zope-2.9.9.
> The new application platform is a Sun Sparc Solaris 10 server (dual
> processors).
> 
> Application setup includes a ZEO server managing two databases (a "main"
> one for common data and a "catalog" to store catalog), and two Zope
> clients (one as main frontend and another one for uncommon
> administration tasks, both with standard threads count of 4).
> Database sizes (when packed) are of 280 and 400 MB ; Zope clients caches
> are of 300 and 500 MB, for 40000 and 80000 objects.
> 
> All data are stored on two mirrored (via RAID 1) Hitachi SAN with quite
> good writing speed.

Because you need fast *seek* speed, too, I would recommend trying your
app with the ZODB on local disk, first.  In fact, I never recommend
putting the main ZODB on a non-local disk, for just this reason.  The
usage pattern of FileStorage (one big file, writing only at the end, but
with lots of random-access reads throughout) is nearly worst-case for
such deployments.

I have one client whose setup is similar (Linux, but using a "big iron"
SAN), where the first phase of incremental repozo backup (the part
checking whether incremental is possible) takes longer by a factor of
two than just doing a full backup.  Putting it on local disk both makes
writes go faster and makes incremental backup feasible.

If you must stay with the SAN, I would investigate whether
DirectoryStorage, which uses lots of litte files, is a better fit.

> I actually have two problems with this configuration :
>  - storing an object into ZODB is VERY slow ; it can take more than 30
> seconds when the object wasn't saved recently, event under light load. I
> actually suspect that it's the catalog "full-text" search index which is
> long to load, update and store. The old application under Zope-2.6.1
> didn't have this problem (but also didn't use ZEO !). Is there a
> specific way to handle catalogs with ZEO, should I avoid this kind of
> configuration ?

You can change the application to use QueueCatalog, which defers writes
to a configurable set of indexes (typically the text index(es)) for
later processing.  I often run that processing via a cron job or other
long-running, non-server process (running as a ZEO client).

>  - sometimes (but most often under quite heavy load) the Zope clients
> completely hang and stop responding, without any access or error log,
> and without any disk or CPU activity. I suppose it's a kind of deadlock,
> how can I detect the source of the problem ?
> 
> Many thanks for any help or advise, as the resolution of these problems
> is quite urgent...

As Alan said, the DeadlockDebugger product is designed to help diagnose
such issues.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIfp7e+gerLs4ltQ4RAgmfAJ9K9TtMfLc3i0hndANvvrY/PaA9qwCgrof8
YAGIUADOgpkIJvb2A7c5JJM=
=76B6
-----END PGP SIGNATURE-----