On Mon, Aug 07, 2006 at 09:09:56PM +0200, Dieter Maurer wrote:
Gerhard Schmidt wrote at 2006-8-7 15:54 +0200:
... 2006-08-07T14:29:19 INFO ZEO.StorageServer (97002/10.152.64.23:52518) Transaction +blocked waiting for storage. Clients waiting: 1. ... 2006-08-07T14:29:50 INFO ZEO.StorageServer (97002/10.152.64.17:54463) Blocked transaction restarted. Clients waiting: 1 2006-08-07T14:29:50 INFO ZEO.StorageServer (97002/10.152.64.23:52518) Blocked transaction restarted.
This one was a very quick one only 30 seconds. I have Blocked Transaktion that ware waiting for more than 2 minutes.
This means that you have very long transactions -- transactions that take very long to commit.
ZEO cannot commit two transactions for the same storage at the same time. Therefore, it sets a storage look when a transaction commit begins for the storage.
If another transaction tries to commit to the same storage, the transaction is blocked until the first transaction commit completes. That's your "Transaction blocked waiting for storage...."
When the commit is completed, then a waiting transaction is restarted. That's your "Blocked transaction restarted".
You should try to understand where the huge transactions come from. Very often, they are caused by poor persistency design (either far too huge objects or an immense number of tine objects or just some stupidity (e.g. writing objects unnecessary).
I have benchmarked my Harddisk (which is at the moment an emergency system because of a hardware failure of the main system) it has 40 MB/sec write speed and it doesn't show high io load when we have such a hangup. I have tried to create an object with 50 MB in the storage the ZEO server had no problem with that. calculating this, there has to be an objekt of CD Image size to cause the write to take more then 30 sec. But this whould mean that the Data.fs whould grow at least a 2-3 Gig a Day (we have 5-6 such hangups a day) but it only grows arround 50-100 MB a per Day (difference befor and after Pack) and real growth is 2-10 MB per Day. To the number of tiny objects. I have the zeo.log on debug level. Entries like these seam to be the objekts that are requested to be written. 2006-08-08T07:44:12 DEBUG ZEO.zrpc.Connection(S) (10.152.64.21:50210) calling storea('\x00\x00\x00\x00\x0057\x85', '\x03gZW\x00\xa0\xa1\xcc', '(... I don't see a lot of them bevor a hangup occur. So there are two areas where the Problem could be located. The filesystem of the host system and in the network between the App-Server and the Zeo. To understand this can you tell me when a transaction is started and when its closed. Does the Zeo server wait until all data is recieved bevor the transaction is started or does the transaktion start when the datatransfer from the appserver starts. Bye Estartu ---------------------------------------------------------------------------- Gerhard Schmidt | Nick : estartu IRC : Estartu | Fischbachweg 3 | | PGP Public Key 86856 Hiltenfingen | EMail: estartu@augusta.de | on request Germany | |