Making Zope stop storing old object versions
Hello, We've got lots of Catalog reads and writes and our Data.fs is growing at an immense rate. Is there a way we can tell Zope to stop storing old object versions to limit the growth and not have to pack that often ? Regards, Danny
Hi, Sinang, Danny wrote:
Hello,
We've got lots of Catalog reads and writes and our Data.fs is growing at an immense rate.
Is there a way we can tell Zope to stop storing old object versions to limit the growth and not have to pack that often ?
Well in fact it (Filestorage) does only "store" new object versions. What it does not is removing the old version - this is done during packing. When you have many Catalog writes or other parts of your object tree experiences many writes, try to mount an extra ZODB (and storage) there. You can also try some of the other storages like relstorage (where it depends on the RDBMS how it handles writes) Cheers Tino
Sinang, Danny wrote: Hello,
We've got lots of Catalog reads and writes and our Data.fs is growing at an immense rate.
Is there a way we can tell Zope to stop storing old object versions to limit the growth and not have to pack that often ?
Well in fact it (Filestorage) does only "store" new object versions. What it does not is removing the old version - this is done during packing.
When you have many Catalog writes or other parts of your object tree experiences many writes, try to mount an extra ZODB (and storage) there.
You can also try some of the other storages like relstorage (where it depends on the RDBMS how it handles writes)
Alternatively, you could just replace your Catalog with an external db such as MySQL (using zmysql database connections). Jonathan
How do I go about replacing a Catalog using MySQL ? This is the first time I've heard this. -----Original Message----- From: Jonathan [mailto:dev101@magma.ca] Sent: Thursday, May 08, 2008 7:16 PM To: Sinang, Danny Cc: zope@zope.org Subject: Re: [Zope] Making Zope stop storing old object versions
Sinang, Danny wrote: Hello,
We've got lots of Catalog reads and writes and our Data.fs is growing
at an immense rate.
Is there a way we can tell Zope to stop storing old object versions to limit the growth and not have to pack that often ?
Well in fact it (Filestorage) does only "store" new object versions. What it does not is removing the old version - this is done during packing.
When you have many Catalog writes or other parts of your object tree experiences many writes, try to mount an extra ZODB (and storage) there.
You can also try some of the other storages like relstorage (where it depends on the RDBMS how it handles writes)
Alternatively, you could just replace your Catalog with an external db such as MySQL (using zmysql database connections). Jonathan
<snip> ----- Original Message ----- From: "Sinang, Danny" <D.Sinang@spi-bpo.com> To: "Jonathan" <dev101@magma.ca> Cc: <zope@zope.org> Sent: Thursday, May 08, 2008 10:03 AM Subject: RE: [Zope] Making Zope stop storing old object versions How do I go about replacing a Catalog using MySQL ? This is the first time I've heard this. </snip> Here's the relevant chapter in the Zope Book: http://plope.com/Books/2_7Edition/RelationalDatabases.stx Jonathan
Thanks Jonathan. But I thought you were referring to a way to make Catalog writes transparently write to MySQL. The article you're pointing to teaches how to use use ZSQLMethods, which we're already doing. But the product we're using to do workflows (OpenFlow) uses the Zcatalog. So we either rewrite it to use MySQL, or we find a way to make Catalogs write to MySQL. Regards, Danny -----Original Message----- From: Jonathan [mailto:dev101@magma.ca] Sent: Thursday, May 08, 2008 10:08 PM To: Sinang, Danny Cc: zope@zope.org Subject: Re: [Zope] Making Zope stop storing old object versions <snip> ----- Original Message ----- From: "Sinang, Danny" <D.Sinang@spi-bpo.com> To: "Jonathan" <dev101@magma.ca> Cc: <zope@zope.org> Sent: Thursday, May 08, 2008 10:03 AM Subject: RE: [Zope] Making Zope stop storing old object versions How do I go about replacing a Catalog using MySQL ? This is the first time I've heard this. </snip> Here's the relevant chapter in the Zope Book: http://plope.com/Books/2_7Edition/RelationalDatabases.stx Jonathan
Hi, Jonathan wrote:
Sinang, Danny wrote: Hello, ... You can also try some of the other storages like relstorage (where it depends on the RDBMS how it handles writes)
Alternatively, you could just replace your Catalog with an external db such as MySQL (using zmysql database connections).
well if MySQL is the right tool and does not introduce other problems. It should be pointed out that along the powerful open source DBs there is for example also postgresql which is extreamly good at handling concurrent loads. Cheers Tino
(Thu, May 08, 2008 at 08:08:28AM +0800) Sinang, Danny wrote/schrieb/egrapse:
We've got lots of Catalog reads and writes and our Data.fs is growing at an immense rate.
Sometimes you have to look at your product code and figure out what it's exactly doing. For example many times not very optimal products will change all the instance when only a small attribute changes, resulting in a zodb write that is much larger then really needed. I documented an example of a hunt for a case like that in "Quasi-Normal in Numbers": http://betabug.ch/blogs/ch-athens/708 Regards, Sascha
participants (4)
-
Jonathan -
Sascha Welter -
Sinang, Danny -
Tino Wildenhain