I have a ZCatalog containing some 400,000+ objects. I recently rebuilt the catalog due to corruption, so it should be in pristine condition. Today I issued a simple query (one condition) of the ZCatalog which should have returned a very large number of objects. My intention was to generate a data export that I could then import into another system. The query ran for 2 hours, at which point I discovered that Zope was hung and completely unresponsive. I had to break out the 'kill -9' to get it to die. Does anyone have any hints on getting large amounts of data out of the ZODB using the ZCatalog (or otherwise)? My efforts thus far have proven futile. ID:[{20040902144153.711.468943402-12.6.18.86}]
----- Original Message ----- > I have a ZCatalog containing some 400,000+ objects. I recently rebuilt the
catalog due to corruption, so it should be in pristine condition. Today I issued a simple query (one condition) of the ZCatalog which should have returned a very large number of objects. My intention was to generate a data export that I could then import into another system. The query ran for 2 hours, at which point I discovered that Zope was hung and completely unresponsive. I had to break out the 'kill -9' to get it to die.
Does anyone have any hints on getting large amounts of data out of the ZODB using the ZCatalog (or otherwise)? My efforts thus far have proven futile.
We have had no problem dumping a ZCatalog with about 1million records to a file using an external method. We also mount a separate .fs file which only contains the ZCatalog; this makes it easier to update the zcatalog on anther machine and then just swap in the new zcatalog without impacting any code. Jonathan
nwingfield@dixon-hughes.com wrote at 2004-9-2 14:41 -0400:
I have a ZCatalog containing some 400,000+ objects. I recently rebuilt the catalog due to corruption, so it should be in pristine condition. Today I issued a simple query (one condition) of the ZCatalog which should have returned a very large number of objects. My intention was to generate a data export that I could then import into another system. The query ran for 2 hours, at which point I discovered that Zope was hung and completely unresponsive. I had to break out the 'kill -9' to get it to die.
Learn about how to analyse a "spinning" Zope (there is a HowTo on "zope.org"). I attach a little "GDB source file" (usually used as GDB init file ".gdbinit"). It defines the command "pfr" ("print frame") which can be used in "eval_frame" frames to output source file name and function this "eval_frame" is used for. This allows you to analyse the Python traceback from C level. -- Dieter def ps x/s ({PyStringObject}$arg0)->ob_sval end def pfr ps f->f_code->co_filename ps f->f_code->co_name p f->f_lineno end define pyo print _PyObject_Dump($arg0) end define pyg print _PyGC_Dump($arg0) end
participants (3)
-
Dieter Maurer -
Jonathan Hobbs -
nwingfield＠dixon-hughes.com