- Zope-Dev - Zope lists

CorruptedDataError sniff !
by Eric Brun 02 Jun '05

02 Jun '05

Hi all, Houston, we've got big a problem ! Here is the traceback in ZEO_EVENTS.log: " ------ 2005-06-02T11:45:21 INFO(0) zrpc-conn:193.49.250.194:37684 zeoLoad() raised exception: 968647571 Traceback (innermost last): File /data/ZeoCVS/lib/python/ZEO/zrpc/connection.py, line 234, in handle_request File /data/ZeoCVS/lib/python/ZEO/StorageServer.py, line 337, in zeoLoad File /data/ZeoCVS/lib/python/ZODB/FileStorage.py, line 694, in modifiedInVersion (Object: /data/ZeoCVS/var/Data.fs) CorruptedDataError: 968647571 " And I can't pack the zodb, when I pack I've got this traceback : " ------ 2005-06-02T08:48:17 ERROR(200) zrpc:1193 Error raised in delayed method Traceback (innermost last): File /data/ZeoCVS/lib/python/ZEO/StorageServer.py, line 865, in run File /data/ZeoCVS/lib/python/ZEO/StorageServer.py, line 402, in pack_impl File /data/ZeoCVS/lib/python/ZODB/FileStorage.py, line 1503, in pack (Object: /data/ZeoCVS/var/Data.fs) File /data/ZeoCVS/lib/python/ZODB/fspack.py, line 680, in pack File /data/ZeoCVS/lib/python/ZODB/fspack.py, line 464, in findReachable File /data/ZeoCVS/lib/python/ZODB/fspack.py, line 478, in buildPackIndex File /data/ZeoCVS/lib/python/ZODB/fspack.py, line 180, in checkTxn File /data/ZeoCVS/lib/python/ZODB/fspack.py, line 170, in fail CorruptedError: /data/ZeoCVS/var/Data.fs:968633329:invalid transaction status: 'g' " I run Zope 2.6.2 , ZEO 2 and ZODB 3.1 I have tried to do a fsrecover, it removes a transaction. Then when I tried to lunch a pack, I've got a very strange error about unpickling : "Error unpickling, ((UBTrees._IIBTreeqUIIBucketqtqNt.((JÇü KJQKJyWKJ}WKJ06KJKJ®KJÃøKJKKJKKJKKJ KKJ¡KKJ¢KKJ¦KKJ8KJÜS KJê!KJGF*KJ%D.KJ.KJ.KJ.KJ;w/KJ>w/KJ?w/KJ~G0KJÎ¿1KJÁ3KJ'^7KJ;KJ¢;KJwû<KJzû<KJ{û<KJÄ>KJ#IGKJPLGKJ|IKJJKKJ¶ÊMKJ ]KJ ]KJ ]KJ ]KJ ]KJ ]KJ ]KJ ]KJ¡ ]KJrdKJæãdKJèãdKJ¿gKJ¿gKJ]¸yKJÆozKJ¸|KJ¹|KJÁ|KJ |KJ}KJ}KJ}KJ}KJCKJIÏKJJÏKJd¹KJf¹KJ#[KJ÷KJM»KJNÖ´ KJ×´ KJÚ´ KJÛ´ KJC) KJ KJ©Ø¦ KJ¢Ò KJòê KJÀDô KJ×Dô KJû KJ$R KJIr| KJðµ KJÛ$û KJ}KJM»KJî²KJò²KJó²KJô²KJ×ØKJ5´9KJ<´9KJ¯¾ãKJ/hûK" !!!! I try to copyTransaction in a new Storage but it failed too. This is a production site. My last backup which is not corrupted date of Sunday morning ! Please Help me ....

2 1

BTrees and Persistance
by Yair Benita 01 Jun '05

01 Jun '05

I recently started to use ZODB and python as my chosen database solution. I am having a few problems with retaining changes in BTrees. I have read the documentation and am aware of the _p_changed attribute. Still, here is what I observe: ############################ # consider this simplified example # lets ignore the open database method for now. T = OOBTree() # my trees contain integers as keys and sets as values T.update({1:set([1,2,3]), 2:set([5,6,7])}) # I would really really like this to work T[1].add(6) T._p_changed = True get_transaction().commit() # but it doesn't. # changes are not saved when I close the database and reopen it # This works: T[1].add(6) T.update({1:T[1]}) ############################ The thing is my sets tend to be very big and I am not sure but I think that using T.update({1:T[1]}) will slow me down since a dictionary is first created with a copy of the set which is very big and then the OOBTree is updated. Or am I wrong here? Can anyone suggest an efficient way that will not require a copy of the set already stored in T[1]? Thanks for the help, Yair

4 4

Re: [Zope-dev] BTrees and Persistance
by Yair Benita 01 Jun '05

01 Jun '05

That is of course true, I only change the set and would like to save the information. Trying T[1]._p_changed = True gives an error: AttributeError: 'set' object has no attribute '_p_changed' so the only way to do it is to set _p_changed=True for the Tree. Does that mean that the entire Tree object is rewritten to file? Yair On Jun 1, 2005, at 11:13, Lennart Regebro wrote: > On 6/1/05, Yair Benita <y.benita(a)wanadoo.nl> wrote: >> # I would really really like this to work >> T[1].add(6) >> T._p_changed = True >> get_transaction().commit() >> >> # but it doesn't. > > Well, the thing you changed is the set, not the BTree... although I > would have thought is both sets and btrees handled that > automatically... > -- > Lennart Regebro, Nuxeo http://www.nuxeo.com/ > CPS Content Management http://www.cps-project.org/ > >

1 0

Re: [Zope-Checkins] SVN: Zope/trunk/ - Collector #1323: applied patch to fix umask problem in zdctl
by Tim Peters 31 May '05

31 May '05

[Andreas Jung] > I thought Fred hooked zdaemon and ZConfig already as external definition. > But apparently this happened only for ZConfig. That's correct; nothing about zdaemon has changed yet. Back on 13 May: [me] >> The first time ZConfig and zdaemon get tagged, it's important that >> whoever do that first search all clients (ZODB and both Zopes) for >> changes to those checked in from a wrong project, and do merges back >> to ZConfig and zdaemon (I explained that dance last time) -- else >> those changes (if any) will get lost. [you] > That's a job for loosers :-) but I will take it over for ZConfig and > ZDaemon.. maybe not for the next beta but hopefully for the final > for 2.8.1 release. so I thought you would be doing this. Fred fixed a bug in ZConfig since then, and "did the right thing" to get it fixed everywhere. The same hasn't been done for zdaemon yet, and I'm sure Fred won't be doing it. If it *had* been done already, SVN would not have allowed you to check in a change to zdaemon made from Zope, forcing you to make the change in the right place (i.e., in zdaemon) instead.

1 0

Re: [Zope-Checkins] SVN: Zope/trunk/ - Collector #1323: applied patch to fix umask problem in zdctl
by Tim Peters 31 May '05

31 May '05

[Andreas Jung] > Log message for revision 30558: > > - Collector #1323: applied patch to fix umask problem in zdctl > > Changed: > U Zope/trunk/doc/CHANGES.txt > U Zope/trunk/lib/python/zdaemon/zdctl.py Andreas, Zope doesn't "own" zdaemon. zdaemon is (so far) *copied* into Zope from repos/main/zdaemon/trunk/. Checkins made to a zdaemon copy from a client project (like Zope) have no effect on the zdaemon code. So, for example, the zdaemon trunk is still broken, the zdaemon copies in Zope3 and in ZODB are still broken, and the changes you checked in to Zope's copy are likely to get lost the next time someone copies zdaemon into Zope. The right way to fix this in Zope is: 1. Apply the patch to repos/main/zdaemon/trunk/. 2. Make a tag of zdaemon trunk. 3. Delete ("svn rm") Zope's copy of zdaemon and commit. 4. From a Zope checkout, "svn propedit svn:externals lib/python", and add a line naming the zdaemon tag you created in step #2. That would make zdaemon like Zope's other external code (the ZopeX3 stuff, assorted ZODB directories, and ZConfig), and the existence of a zdaemon tag (there isn't one now) would make it easy for other projects (like Zope3 and ZODB) to use the _same_ zdaemon code.

1 0

CMFSetup + ZopeTestCase interaction bug
by Lennart Regebro 31 May '05

31 May '05

When you use the installProduct command of ZopeTestCase on a product that has a CMFSetup profile you may get the following error: File "/home/zopes/27test/Products/CMFSetup/registry.py", line 567, in registerProfile raise KeyError, 'Duplicate profile ID: %s' % profile_id KeyError: 'Duplicate profile ID: <profile id>' This typically happens on the profile "CMFDefault:default". This also *only* happens if you try to run all tests in Products. The reason seem to be that all or some of the products gets installed by Zope. When you then try to install them again with installProduct, you get the error. My suggested solution is to "merge" ZopeTestCases list of installed products with Zopes list of installed products, as these for the moment are not the same, as far as I can see. This error happens both with Zope2.7 and 2.8. Full traceback: File "/opt/Zope-2.7/bin/test.py", line 918, in ? process_args() File "/opt/Zope-2.7/bin/test.py", line 908, in process_args bad = main(module_filter, test_filter, libdir) File "/opt/Zope-2.7/bin/test.py", line 698, in main runner(files, test_filter, debug) File "/opt/Zope-2.7/bin/test.py", line 592, in runner s = get_suite(file, runner.result) File "/opt/Zope-2.7/bin/test.py", line 508, in get_suite mod = package_import(modname) File "/opt/Zope-2.7/bin/test.py", line 482, in package_import mod = __import__(modname) File "/home/zopes/Kalender/Products/CMFonFive/tests/test_typeinfo.py", line 10, in ? installProduct('CMFDefault') File "/home/zopes/Kalender/lib/python/Testing/ZopeTestCase/ZopeLite.py", line 128, in installProduct get_folder_permissions(), raise_exc=1) File "/opt/Zope-2.7/lib/python/OFS/Application.py", line 751, in install_product initmethod(context) File "/home/zopes/27test/Products/CMFDefault/__init__.py", line 155, in initialize BASE) File "/home/zopes/27test/Products/CMFSetup/registry.py", line 567, in registerProfile raise KeyError, 'Duplicate profile ID: %s' % profile_id KeyError: 'Duplicate profile ID: CMFDefault:default' -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/

1 1

mechanism for license boilerplate?
by Chad Whitacre 31 May '05

31 May '05

Dear all, I am wondering what mechanism the Zope team uses to manage the copyright/license boilerplate at the top of every file. Is this handled manually or is it automated? I skimmed through http://www.zope.org/DevHome/ but nothing relevant jumped out at me. Thanks. Chad Whitacre http://www.zetadev.com/

4 4

Re: [Zope-Checkins] CVS: Products/PluginIndexes/common - UnIndex.py:1.20.2.7
by Chris Withers 31 May '05

31 May '05

Tres Seaver wrote: > + if set is None: > + set = IISet(()) > + elif type(set) is IntType: # isinstance(set, int) for 2.8 How come isinstance is only to be used for 2.8? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

2 1

hasattr implementation for Zope?
by Chris Withers 30 May '05

30 May '05

Dieter Maurer wrote: >>>I strongly argue against it. Fix "hasattr" in the Zope context, >>>instead! >> > _marker = [] > def hasattr(obj, attr, marker): > a = getattr(obj, attr, _marker) > return a is not _marker > > import __builtin__ > __builtin__.hasattr = hasattr > > Easy enough, isn't it? > > Way easier than to change "hasattr" anywhere it is now used. Anyone think putting this in Zope core is a good idea? If so, where should it go? Any potential performance implications people can think of? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

7 26

RAMcache and container vs. context
by Paul Winkler 28 May '05

28 May '05

I never noticed this before today, but apparently RAMCacheManager is sensitive to whether cached items are acquired from the context or from the container. (Zope 2.7.3 but afaict it should behave the same in any recent version). Is that intended behavior or is this a bug? Let's say I have a CMF site at http://foo:8080/TheSite. And a template at /TheSite/portal_skins/custom/template1 which looks in part like like: ... Let's say that expensive_script is also in portal_skins/custom and is associated with a RAMCacheManager. Let's say that there are *no* request variables configured for this RAMCacheManager, which would seem to indicate that we consider all calls to expensive_script equivalent regardless of the request. If I visit http://foo:8080/TheSite/page1, which page makes use of this template, and I reload several times, and look at the RAMCacheManager statistics view, I see several hits for /TheSite/expensive_script. Every time I reload, the hit count goes up by one. So far so good. But if I then visit another page, http://foo:8080/TheSite/page2, which uses the same template, I get a miss and the entry count goes up by one. Reloading then gives me a hit, presumably from the new entry. If I browse around the site, the entry count increments for every new page I hit. Remember that I have nothing configured in the "REQUEST Variables" field. So the script should be the same regardless of the request right? This was puzzling me until, on a hunch, I looked at the calling template and changed it from this: ... ...to this: ... And presto, I now get only one entry, and every page on the site gives me a hit from that one entry. Is this really intended behavior? It's not obvious to me from the code why this is happening (or why it might be desirable). The cache manager stores and looks up items by their getPhysicalPath(), so I don't see why context vs. container is relevant. If I'd wanted the URL to be relevant, I would have put that in the REQUEST Variables configuration. This could be a performance hit for the unwary ... aside from all the unnecessary misses, we'd be using RAM on the order of N * M * S where N is number of cached items and M is number of pages in the site, and S is average size of return value. We only should be using N * S. -- Paul Winkler http://www.slinkp.com

5 8