Hello, We have a script that does a'database import' into the ZODB. This script iterates through a series of records (currently in text files) and creates corresponding Zope objects. We can do small (<100) imports relatively well, but on a large import (1000+) the Zope process balloons in size and eventually becomes unresponsive. Sending a SIGHUP to Zope restores normal functionality and performance, and the created objects all seem to work fine. However, we don't want to have to restart Zope every time we do an import. My suspicion, based on the above evidence, is that the ZODB caching mechanism is to blame for the slowdown. Can anybody confirm this, or offer an alternate explanation? And, if caching is the culprit, is there a way to programmatically turn off caching for the duration of a script run? Thanks, VanL
Hi Van, You're on the right track.. nice deduction work! ;-) The cache code in 2.5.1 and below is not adequate for keeping memory usage in check when lots of database stores are happening in sequence. The cache code is much improved in 2.6 (thanks to Toby Dickenson), and it does keep memory usage in check. I suggest using 2.6 (or the Zope trunk) for this reason. One further hint: in your object loader code, do a commit every "n" records (maybe 1000), and after the commit call "self._p_jar.cacheMinimize()" ("self" can actually be any persistent object). This attempts to gc the pickle cache for that database connection). This will keep memory usage steady. HTH, - C On Tue, 2002-08-13 at 12:49, VanL wrote:
Hello,
We have a script that does a'database import' into the ZODB. This script iterates through a series of records (currently in text files) and creates corresponding Zope objects. We can do small (<100) imports relatively well, but on a large import (1000+) the Zope process balloons in size and eventually becomes unresponsive. Sending a SIGHUP to Zope restores normal functionality and performance, and the created objects all seem to work fine. However, we don't want to have to restart Zope every time we do an import.
My suspicion, based on the above evidence, is that the ZODB caching mechanism is to blame for the slowdown. Can anybody confirm this, or offer an alternate explanation?
And, if caching is the culprit, is there a way to programmatically turn off caching for the duration of a script run?
Thanks,
VanL
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Thanks to everyone who replied on my earlier question, I think we have it mostly worked out. However, we have turned up another ZODB problem. After running through several hundred records, the process stops with the following error message: Site Error An error was encountered while publishing this resource. ZODB.POSException.ConflictError Sorry, a site error occurred. Traceback (innermost last): File /usr/zope/lib/python/ZPublisher/Publish.py, line 150, in publish_module File /usr/zope/lib/python/ZPublisher/Publish.py, line 127, in publish File /usr/zope/lib/python/ZPublisher/Publish.py, line 127, in publish File /usr/zope/lib/python/ZPublisher/Publish.py, line 127, in publish File /usr/zope/lib/python/ZPublisher/Publish.py, line 122, in publish File /usr/zope/lib/python/Zope/__init__.py, line 126, in zpublisher_exception_hook File /usr/zope/lib/python/ZPublisher/Publish.py, line 98, in publish File /usr/zope/lib/python/ZPublisher/mapply.py, line 88, in mapply File /usr/zope/lib/python/ZPublisher/Publish.py, line 39, in call_object File /usr/zope/lib/python/Shared/DC/Scripts/Bindings.py, line 252, in __call__ File /usr/zope/lib/python/Shared/DC/Scripts/Bindings.py, line 283, in _bindAndExec File /usr/zope/lib/python/Products/PythonScripts/PythonScript.py, line 302, in _exec (Info: ({'traverse_subpath': [], 'container': <Folder instance at 8b86280>, 'context': <Folder instance at 8b7ea20>, 'script': <PythonScript instance at 8b8c9e0>}, ('/usr/home/pjl/test.cdf', 'blah'), {}, None)) File Script (Python), line 91, in ImportCDF File /usr/local/Zope-2.5.1/Extensions/ReadCDF.py, line 96, in ReadCDF File /usr/zope/lib/python/ZODB/Transaction.py, line 234, in commit File /usr/zope/lib/python/ZODB/Connection.py, line 348, in commit (Info: (('BTrees.IIBTree', 'IIBucket'), '\x00\x00\x00\x00\x00\x00\xa7\xbf', '')) File /usr/zope/lib/python/ZODB/FileStorage.py, line 668, in store (Object: /usr/local/Zope-2.5.1/var/Data.fs) ConflictError: database conflict error (oid 000000000000a7bf, serial was 0346c82026402fe6, now 0346c81d4506b211) I'm not sure how to tackle this error. Any ideas? Thanks, VanL
Are there other connections to the same database while your import script is running? Conflict errors are normal ZODB behavior. See http://www.zope.org/Documentation/Books/ZDG/current/Persistence.stx for more information. On Tue, 2002-08-13 at 17:34, VanL wrote:
Thanks to everyone who replied on my earlier question, I think we have it mostly worked out. However, we have turned up another ZODB problem.
After running through several hundred records, the process stops with the following error message:
Site Error An error was encountered while publishing this resource.
ZODB.POSException.ConflictError
Sorry, a site error occurred.
Traceback (innermost last): File /usr/zope/lib/python/ZPublisher/Publish.py, line 150, in publish_module File /usr/zope/lib/python/ZPublisher/Publish.py, line 127, in publish File /usr/zope/lib/python/ZPublisher/Publish.py, line 127, in publish File /usr/zope/lib/python/ZPublisher/Publish.py, line 127, in publish File /usr/zope/lib/python/ZPublisher/Publish.py, line 122, in publish File /usr/zope/lib/python/Zope/__init__.py, line 126, in zpublisher_exception_hook File /usr/zope/lib/python/ZPublisher/Publish.py, line 98, in publish File /usr/zope/lib/python/ZPublisher/mapply.py, line 88, in mapply File /usr/zope/lib/python/ZPublisher/Publish.py, line 39, in call_object File /usr/zope/lib/python/Shared/DC/Scripts/Bindings.py, line 252, in __call__ File /usr/zope/lib/python/Shared/DC/Scripts/Bindings.py, line 283, in _bindAndExec File /usr/zope/lib/python/Products/PythonScripts/PythonScript.py, line 302, in _exec (Info: ({'traverse_subpath': [], 'container': <Folder instance at 8b86280>, 'context': <Folder instance at 8b7ea20>, 'script': <PythonScript instance at 8b8c9e0>}, ('/usr/home/pjl/test.cdf', 'blah'), {}, None)) File Script (Python), line 91, in ImportCDF File /usr/local/Zope-2.5.1/Extensions/ReadCDF.py, line 96, in ReadCDF File /usr/zope/lib/python/ZODB/Transaction.py, line 234, in commit File /usr/zope/lib/python/ZODB/Connection.py, line 348, in commit (Info: (('BTrees.IIBTree', 'IIBucket'), '\x00\x00\x00\x00\x00\x00\xa7\xbf', '')) File /usr/zope/lib/python/ZODB/FileStorage.py, line 668, in store (Object: /usr/local/Zope-2.5.1/var/Data.fs) ConflictError: database conflict error (oid 000000000000a7bf, serial was 0346c82026402fe6, now 0346c81d4506b211)
I'm not sure how to tackle this error. Any ideas?
Thanks,
VanL
Chris McDonough wrote:
One further hint: in your object loader code, do a commit every "n" records (maybe 1000), and after the commit call "self._p_jar.cacheMinimize()" ("self" can actually be any persistent object). This attempts to gc the pickle cache for that database connection). This will keep memory usage steady.
If it still works like it did when I was playing with text indexing in december, then nah, it won't. Give up or use Zope 2.6. hair-less-ly yours, Chris
I've had a similar problem, and have found that periodically commiting the transaction, (say after every 100 objects), in the zodb helps. get_transaction().commit() Mark VanL wrote:
Hello,
We have a script that does a'database import' into the ZODB. This script iterates through a series of records (currently in text files) and creates corresponding Zope objects. We can do small (<100) imports relatively well, but on a large import (1000+) the Zope process balloons in size and eventually becomes unresponsive. Sending a SIGHUP to Zope restores normal functionality and performance, and the created objects all seem to work fine. However, we don't want to have to restart Zope every time we do an import.
My suspicion, based on the above evidence, is that the ZODB caching mechanism is to blame for the slowdown. Can anybody confirm this, or offer an alternate explanation?
And, if caching is the culprit, is there a way to programmatically turn off caching for the duration of a script run?
Thanks,
VanL
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
participants (4)
-
Chris McDonough -
Chris Withers -
Mark Gibson -
VanL