June 2003 - Zope-Dev

Re: deadlock patch ?
by Jeremy Hylton 13 Jun '03

13 Jun '03

> The change was too extensive to call it a patch. It was almost a > rewrite. If you want to dig through the repository, look for all > changes made to ZODB and ZEO between about October 31, 2002 and > January 3, 2003. I imagine it would be less work to upgrade to a > more recent version of Zope. > > Shane It was a fairly involved change. The chief changes were in Transaction.py, but it also affected all the storages. If it isn't possible to upgrade to a modern Zope, which has many benefits, it might be possible to integrate a modern ZODB with an old Zope. Jeremy

2 1

ZODB 3.1.2b2 released
by Jeremy Hylton 13 Jun '03

13 Jun '03

We have released ZODB 3.1.2b2 available from the usual place: http://www.zope.org/Products/ZODB3.1 This release includes several critical bug fixes for ZEO. The bugs are more likely to affect sites that see a lot of read conflicts (resolved or unresolved). The bugs can cause data loss or corruption. I would strongly advise ZEO users to consider upgrading. This release has seen more internal stress testing that normal and should be fairly stable for production use. I hope to get a final release out as soon as possible. The release notes (below) describe the fixes in more detail. Jeremy What's new in ZODB3 3.1.2 beta 2? ================================= Fixed critical race conditions in ZEO's cache consistency code that could cause invalidations to be lost or stale data to be written to the cache. These bugs can lead to data loss or data corruption. These bugs are relatively unlikely to be provoked in sites with few conflicts, but the possibility of failure existed any time an object was loaded and stored concurrently. Fixed a bug in conflict resolution that failed to ghostify an object if it was involved in a conflict. (This code may be redundant, but it has been fixed regardless.) The ZEO server was fixed so that it does not perform any I/O until all of a transactions' invalidations are queued. If it performs I/O in the middle of sending invalidations, it would be possible to overlap a load from a client with the invalidation being sent to it. The ZEO cache now handles invalidations atomically. This is the same sort of bug that is described in the 3.1.2b1 section below, but it affects the ZEO cache. Fixed several serious bugs in fsrecover that caused it to fail catastrophically in certain cases because it thought it had found a checkpoint (status "c") record when it was in the middle of the file.

1 0

deadlock patch ?
by Dirk Datzert 13 Jun '03

13 Jun '03

Hi Zope developers, I read that Zope 2.6.1 has a deadlock patch integrated. - Deadlock prevention code added. It was possible for earlier versions of ZODB to deadlock when using multiple storages. If multiple transactions committed concurrently and both transactions involved two or more shared storages, deadlock was possible. This problem has been fixed by introducing a sortKey() method to the transaction and storage APIs that is used to define an ordering on transaction participants. This solution will prevent deadlocks provided that all transaction participants that use locks define a valid sortKey() method. A warning is raised if a participant does not define sortKey(). For backwards compatibility, BaseStorage provides a sortKey() that uses __name__. Can I find anywhere (maybe CVS) this patch ? I want to look if I can backport this to Zope 2.3.3, since we detected such deadlocks on our server. Regards, Dirk

2 1

R: [Zope-dev] RE: DBtab and BDBStorage
by Fabio Paracchini 12 Jun '03

12 Jun '03

>-----Messaggio originale----- >Da: Chris Withers [mailto:chrisw@nipltd.com] >Inviato: giovedì 12 giugno 2003 15.40 >A: Andrew R. Halko >Cc: 'Fabio Paracchini'; zope-dev(a)zope.org; 'Shane Hathaway' >Oggetto: Re: [Zope-dev] RE: DBtab and BDBStorage >Andrew R. Halko wrote: >> Well, currently I cannot get python 2.2.3 to work with Plone. >What do you need Python 2.2.3 for? >> 2.2.3 will support up to 4TB, which would solve all my problems, but it >> doesn't work. >I have LFS in 2.1.3, have you tried recompiling Python 2.1.3 with LFS? >cheers, >Chris I would add that 2.1.3 on Windows does not support LFS. At least from what I see from here: http://mail.zope.org/pipermail/zope/2002-October/124030.html Anyone has different info ? Another way is if you install Python 2.2 and ZODB3.2a, you can use ZEO and mount via DBTab. In this way you can use Py2.3+ZODB3.2a as ZEO server, and use LFS, and then mount the storage via ZEO Client on a Py2.1+ZODB3.2a+ZOPE. I tried this only on an experimental setup, and it seemed to work, at least I was able to mount a DB and create a Plone site; I didn't do any stress test, i.e. load more than 2Gb. If someone is willing to try more, I'll be happy to assist... Ciao Fabio

3 3

Re: [Zope-dev] RE: DBtab and BDBStorage
by Chris Withers 12 Jun '03

12 Jun '03

(keep it on the list please) Andrew R. Halko wrote: > Are you running Windows or Linux? Windows and python do not support LFS Why are you trying to run a production system on Windows? Chris

2 1

Zope 3 newsletter 8th edition
by Gary Poster 12 Jun '03

12 Jun '03

WELCOME TO THE ZOPE 3 NEWSLETTER: ISSUE 8 (12 JUNE 2003) Welcome to the eighth Zope 3 newsletter. Information about Zope 3 and newsletter contributions and suggestions can be found at the bottom of this newsletter. GLOSSARY FOR THE UNINITIATED Encounter a term in this newsletter you don't know? Try this glossary: http://dev.zope.org/Zope3/NewsletterGlossary NEWS SNIPPETS: - Jim Fulton plans to lead a sprint in Fredericksburg, VA, in the Zope Corporation offices, June 21-23. Spaces are still available. See http://dev.zope.org/Zope3/Fredericksburg2003Sprint2 for more information. - A number of talented Zope 3 developers are sprinting prior to EuroPython in Louvain-la-Neuve, Belgium, June 22-24. See http://dev.zope.org/Zope3/EuroPython2003Sprint for more information. - Jim Fulton plans for a Zope 3 beta for end of June. - Jim Fulton is also scheduled to lead a sprint in Melbourne, Australia, July 9-13. Spaces are still available. See http://www.ozzope.org/OzSprintWiki for more information. - Note that some people's reports below have an "Older News" section: blame it on the newsletter editor. We haven't had a newsletter since March 25. SATHYA RANGASWAMY (ZEOMEGA): i18n conversion ZeOmega is converting all ZPTs and string constants in the Zope 3 application subdirectory (zope.app) to make them i18n aware. GUIDO VAN ROSSUM (PYTHONLABS, ZOPE CORP): Filesystem Synchronization This feature (abbreviated "fssync") lets you create a representation of a subtree of a Zope 3 server's object space on the filesystem, edit the files on the filesystem, and push them back to the server. There is a command line utility which works somewhat like CVS that manages synchronization between filesystem and server. The filesystem representation of common objects is natural and easy to edit: folders map to directories, page templates map to files containing the text of the page template, and so on. Unrecognized objects are represented as a blob of XML. The command line utility is the script src/zope/fssync/main.py; invoke it without arguments to get help. The README.txt file in the same directory has more project information, and links to Wiki pages. It is possible to synchronize part of the site manager subtree (/++etc++site/) as well; the next stage of the project will be to extend the fssync machinery to allow installing and managing site management bundles (the envisioned primary mechanism for distributing Zope 3 software). STEPHAN RICHTER (IUVENO): Internationalization The Pre-PyCon sprint was a great success! We got almost all tasks required for the Zope X3 release done. Page Templates are correctly hooked up to the Translation Services, the user's Locale is available in the request, translation of Python code is possible, the JobBoardI18n demo is updated and translation extraction tools have been written. The only tasks left to do is to implement the I18n support for ZCML, internationalize Zope 3 and translate it to a couple of languages. For a complete summary see http://dev.zope.org/Zope3/PyConSprint. Documentation I am working heavily these days on the Zope 3 Cookbook to provide some extensive and narrative documentation by example. All of the 10 introduction chapters have been outlined and some recipes have been written. I am working currently on a section that builds a Message Board application from scratch. The cookbook can be found at http://dev.zope.org/Zope3/DevelCookbook and the example code is accessable via http://cvs.zope.org/Products3/demo/messageboard/. The book's content is currently placed under the Open Publication License. I recently had another Wiki cleanup session. The Wiki pages look pretty good at this point and most FrontPage linked pages are up-to-date. I am also pretty confident about pages having status badges, though there needs to be more work done in this area. For details see http://mail.zope.org/pipermail/zope3-dev/2003-June/007246.html SQL TALES Expressions This is a little product that adds another TALES expression to page templates. SQL expressions allow you to write SQL statements in your page templates directly without the need for creating new SQL Scripts, connection services and database adapters. The used database can be either specified with a connection object name or by directly specifying the RDB DA and the DSM. Example:: <html tal:define="rdb string:psycopgda; dsn string:dbi://test"> <body> <ul> <li tal:repeat="contact sql: SELECT * FROM contact"> <b tal:content="contact/name" /> </li> </ul> </body> </html> Thanks goes to Alan Runyaga, who tickled the idea out of me. He was looking for allowing Zope to be more similar to PHP and other inline scripting tools, since they are more familiar to the Web scripter and therefore easier to learn. Link: http://cvs.zope.org/Products3/sqlexpr/ ZWiki for Zope 3 I have rently ported the core functionality of ZWiki to Zope 3. Thanks goes to Simon Michael for his help. ZWiki for Zope 3 supports Plain Text, STX and ReST as input types, can mail modification notification, and has parenting support. For more details see http://dev.zope.org/Zope3/ZwikiForZope3 On this page you will also find link to three very positive reviews about the product! ALBERTAS AGEJEVAS, MARIUS GEDMINAS (CODEWORKS): Mailing service Lately, we've been working on the transactionally-safe mailing service. It's not yet there, but expect it in HEAD soon. As an off-shoot of this project, Marius has written a proposal for a nice API for creating RFC822 messages, which would take care of things like MIME attachments and proper encoding of Unicode data. Authenticated user names in logs Albertas has traced and added a minor feature which was missing: authenticated user names in the HTTP access logs. Need to know what the users are doing! <wink> Decorators (see also Steve's entry below) Marius has been pairing with Steve implementing decorators, but that's a separate story; besides, it was more than two weeks ago. Older news from Marius (April 16) One thing worth attention is the new functional testing framework. Functional tests can test large parts of the system as a whole by simulating the publication process in an automated way. One of the immediate merits of the functional test suite is that errors in ZCML files will be immediately noticed. If you're interested, you can find the documentation in doc/FTEST.txt in your Zope 3 source tree, and the FunctionalTestingFramework page of the Zope 3 wiki. Albertas and I have also been working on virtual hosting. It is now basically working in conjunction with Apache rewrite rules. We have already deployed it on some of our production systems. There are still some rough edges, e.g. the Javascript tree in the ZMI is not yet aware of virtual hosting. More details can be found on the VirtualHosting page of the Zope 3 wiki. BTW virtual hosting has full functional tests. ;-) One of the things we use virtual hosting for is a product I've written called z3checkins. It displays a list of the most recent changes made to the Zope 3 CVS repository, and you can add it to your Mozilla sidebar. Check it out at http://www.codeworks.lt/zope3-checkins . If you like it, thank Steve Alexander for the idea. STEVE ALEXANDER (CODEWORKS): Decorators With help from Marius, I implemented a prototype of Decorators. Later, Jim and I simplified Decorators and implemented them. Decorators are specific kinds of Context Wrapper that you use to separate the conetext-specific implementation of a some content from the plain ole' Python implementation. So, a Folder class implements basic folder functionality, and its decorator, ZopeContainerDecorator handles context-wrapping, sending events, calling hooks such as the 'afterAddHook', and that kind of thing. In practical terms, you no longer have to adapt a container to IZopeContainer before using it. The equivalent behaviour with a decorator now happens automatically when the container gets context-wrapped. There's some preliminary documentation in src/zope/app/context.txt. __implements__ to implements With help from Philipp von Weitershausen, I converted almost all (about 350) of the uses of __implements__ in the Zope 3 code the new implements(...) syntax. In a lot of cases, the implements declaration was made shorter, or could be removed altogether. Context Wrappers Jim and I reworked context wrappers a bit, breaking out zope-specific functionality in to a separate subclass in zope.app.context. I also removed the ContextAware base class because it had unpredictable and implicit side-effects, which broke some xml code Martijn Faassen had just checked into CVS under Products3. Sorry Martijn! Shortly after, I wrote a new ContextAware base class that works in a more explicit and predictable way. Easier unit tests for local (placeful) components Jim refactored the PlacefulSetup code for writing unit tests. This made writing location-dependent tests much less painful. Event service bug I fixed a subtle persistence related bug in the event service. Kudos to Marcus J. Ertl for tracking the bug down. Older news from Steve (April 8): I rewrote in C the ContextWrapper code that was in Python. This resulted in much simpler wrapper code in Python. Now the unit tests run somewhere between 10% and 15% faster. CHRISTIAN THEUNE (GOCEPT): Open source certification There has been a meeting with the TUEV-IT which is a German institution that is able to issue common criteria certificates. There is a sponsorship by the Mitcon GmbH who pays the fees for a EAL-1 certification which makes Zope 3 the world's first certified open source product. The certification is in preparation and there will be a workshop in July where the first documents are drafted together with the TUEV and our people, who include Aroldo Souza-Leite, Steve Alexander, some other people and me. If you want to know more about the certification please see http://www.commoncriteria.org Stay zoped! GARY POSTER (ZOPE CORPORATION): Zope 3 plus Zope 2 I have been working on integrating Zope 3 with Zope 2 on a Zope Corporation customer project. The project leverages ZCML to only load the parts of Zope 3 pertinent to the project. Jim Fulton hopes to leverage this experience for a Zope 2 release in the fall of this year that incorporates Zope 3. CONTRIBUTING Please send Zope 3 news, and newsletter suggestions and requests, to gary(a)zope.com, with "Zope 3 newsletter" somewhere in the subject line. As you can see above, casual and quick news items are acceptable and even desirable. MORE INFORMATION ON ZOPE 3 The central place to find Zope 3 information is currently http://dev.zope.org/Zope3 The Zope 3 mailing list is archived and managed at http://lists.zope.org/mailman/listinfo/zope3-dev The Zope 3 development IRC channel, #zope3-dev at irc.openprojects.net is (strictly) for discussion of Zope 3 development issues. Newsletters are archived at http://dev.zope.org/Zope3/Zope3Newsletter

1 0

R: DBtab and BDBStorage
by Fabio Paracchini 12 Jun '03

12 Jun '03

Hi Andrew, here are the answers: >-----Messaggio originale----- >Da: Andrew R. Halko [mailto:ahalko@insivia.com] >Inviato: mercoledì 4 giugno 2003 17.02 >A: fparacchini(a)alteanet.it >Cc: zope-dev(a)zope.org >Oggetto: DBtab and BDBStorage >Hello Fabio, >I read your post at: >http://zope.nipltd.com/public/lists/dev-archive.nsf/AGByKey/35A02E5C64B9 >3519 about DBtab and BDBStorage. I am interested in doing the same >thing. I have a Plone site that I expect to become very large and I >need to find some other solutions than ZODB and its 2GB limit. Do you >think you could help me with figuring this out? I have been reading >documentation, posts and everything else and am a little confused. Here >are a few questions I have from reading your post. >1. What did you need to install to make this work and did you do it >after you already had a Plone/Zope site going? There is a document at http://plone.org/Members/fparacchini/Document.2003-02-14.0631 that explains what you need and what you have to do. It was last updated mid-February, so now there can be later versions of pyBSDDB3 and DBTab. Be careful that the right combination of Python 2.1.x/pyBSDDB3/ZODB has to be carefully checked. I got results with the versions specified in that document, your mileage may vary ;) >2. What is the code that goes in DBtab.conf, I read through it about 10 >times and can't understand how I need to set it up? In DBTab.conf you have to specify a Storage that goes on BDB and a Database that is mounted on a Zope path. For example, to have the whole site on BDB the very basic configuration is: [Database: Main-BDB] mount_paths=/ open_at_startup=1 [Storage: Main-BDB] type=BDBFullStorage This way you'll create a directory below <zope>/Data/var called Main-BDB, that contains a Berkeley DB with logs. >3. Do you have to create a new DB manually every time you want to >separate info, is there no way to setup so that it dynamically adds as >you need space. Well, you can also add a storage later. The procedure is a bit involved but not so much: - I suppose you already have a working DBTab configuration, like the one specified before :) - Create a folder in Zope/Plone where you will put your data, e.g. '/Docs' - Shutdown Zope/Plone - Copy the whole data directory (in the example before, it should be in C:\program files\plone\data\var\Main-BDB, copy it in C:\program files\plone\data\var\Docs-BDB) - Edit DBTab.conf, and add a Storage and a Database like this: [Database: Docs-BDB] mount_paths=/Docs open_at_startup=1 [Storage: Docs-BDB] type=BDBFullStorage - Start Zope/Plone - from management interface, delete folder /Docs - from management interface, create a new 'DBTab mount point' that should be enough: your /Docs folder should be in the new database, where it grows independently from the rest of the system. Check filesizes on both directory, to be sure that everything works. I just have to thank once again Shane Hathaway, for the countless answers on this topic. I also migrated one DB from FileStorage to BDBStorage, the python script I used, launched directly from python interpreter and not from Zope is like this: from ZODB.FileStorage import FileStorage from BDBStorage.BDBFullStorage import BDBFullStorage DBSrc = FileStorage( 'C:\temp\Data.fs', read_only=1) DBDst = BDBFullStorage( 'C:\temp\BDB-Data' ) DBDst.copyTransactionsFrom( DBSrc ) DBSrc.close() DBDst.close() You can also mix different database flavours, thanks to DBTab. In my configuration I keep 'main' database on FileStorage, to make it faster and easier to administer, and I created a couple of different DBs with BDB for folders '/Customers' and '/Members'. >4. I read a little on BerkleyDB Storage, does that give more space than >ZODB? Yes, you no longer have a 2Gb limit, as in ZODB (at least this is true on every platform, to be honest I don't know if on Linux/Unix you can overcome this limit also with ZODB). >Thanks for any help on this that you can give. There should be a >tutorial on this somewhere; it seems like something a lot of people >would be interested in. A 2GB limit is just horrible. Thanks! >Andrew R. Halko I hope that this helps, let me know if you have more suggestions. Ciao Fabio

6 9

oh boy [id,getId() madness]
by Romain Slootmaekers 12 Jun '03

12 Jun '03

Yo, I have an error in this code fragment: ------------- fragment ---------- for childObject in container.objectValues(): id=childObject.getId() childObject2=container[id] # sometimes gives a KeyError ???? #... ------------- /fragment ---------- So it means that there are contained objects with an getId member that returns something different than the key that is used for these objects in the container object. As it turns out, I indeed have such objects in my ZODB, and they cause all kinds of problems (fe, you can't rename them with the management interface) The questions are: - has anyone else experienced this ? - what could be causing this ? TIA, Romain.

4 3

serialization with Ape
by cs＠comlounge.net 11 Jun '03

11 Jun '03

Hi list! :) I am just playing around with Ape (very cool, btw :) and wondering how it can be used from the outside, e.g. not controlled by transactions. Main idea is to manually copy parts of the ZODB to the filesystem (from a product). I first tried the transaction based process, as I found it in some tests: # init (createMapper imported from fsmapper) root_mapper, conns = createMapper(fspath) conn=conns['fs'] resource = StaticResource(root_mapper) storage = ApeStorage(resource, conns) db = ApeDB(storage, resource) get_transaction().begin() # test object to store object = getattr(self,'standard_template.pt') c=db.open() app=c.root()['Application'] id=object.getId() app._setObject(id,object.aq_base,set_owner=0) get_transaction().commit() db.close() which actually stores it but does not seem to me that clean because of all the transaction stuff. I would here then recursively run through all the objects and do that inside one transaction. I then detected the fascades in io.py which I tried to use (dunno if they're thought to be used for such a purpose actually ;-): root_mapper, conns = createMapper(fspath) ei=ExportImport(root_mapper,conns) ei.exportObject(object) Unfortunately this results in an exception Type: MappingError Error Value: Path keychains require a name So can maybe somebody point me to the easiest way to serialize an object using the fsmapper (Shane? ;-) ? I am also interested in reading a serialized object from the fs again and writing it to the fs. Also I wonder if I need to delete objects which have been modified before another export or if there's a possibility to just change it). (and the best would be of course some simple methods for reading/writing a whole directory recursively. I don't know if something like this is already in there somehow or how it works internally.. seems all a bit magic to me ;-) So thanks in advance if anybody can help me :) cheers, Christian -- COM.lounge http://comlounge.net/ communication & design info(a)comlounge.net

2 5

what is manage_workspace supposed to do?
by Oliver Bleutgen 11 Jun '03

11 Jun '03

I've a problem with a product I'm writing and the way manage_workspace works. There's this code in App/Management.py: def manage_workspace(self, REQUEST): """Dispatch to first interface in manage_options """ options=self.filtered_manage_options(REQUEST) try: m=options[0]['action'] if m=='manage_workspace': raise TypeError except: raise Unauthorized, ( 'You are not authorized to view this object.') (*) if m.find('/'): raise 'Redirect', ( "%s/%s" % (REQUEST['URL1'], m)) return getattr(self, m)(self, REQUEST) My question is about the marked block. I'd guess that the intent is to send a redirect if m (== options[0]['action']) contains a '/'. But m.find('/') evaluates to false only if m[0] == '/', otherwise it yields either -1 (which is true), if there's no '/' in m, or something greater 0, if there's a slash after the first char. Is this intended behavior or a bug? cheers, oliver

2 2