April 2004 - Zope-Dev

Subversion repository layout
by Jim Fulton 27 Apr '04

27 Apr '04

The standard subversion repository layout is by project: proj1 /trunk /branches /br1 /br2 ... /tags /tag1 /tag2 ... proj2 /trunk /branches /br1 /br2 ... /tags /tag1 /tag2 ... ... With this layout, when you want to checkout the main development branch (aka head) of ZODB, you do: svn co svn+ssh://svn.zope.org/repos/ZODB/trunk ZODB That is, you need to include "/trunk" at the end and provide "ZODB" as the name of the output directory so as not to get a directory named "trunk". If you forget the "/trunk", you'll get a checkout that includes copies of all of the various tags and branches, which could be huge. I suggest, instead to use the following layout: proj1 proj2 tags /proj1 /tag1 /tag2 ... /proj2 /tag1 /tag2 ... branches /proj1 /br1 /br2 ... /proj2 /br1 /br2 ... With this layout, when you want to checkout the main development branch (aka head) of ZODB, you do: svn co svn+ssh://svn.zope.org/repos/ZODB which seems cleaner and less error prone. The only disadvantage I see in this layout is that we couldn't have projects named "tags" or "branches", but I could live with that. Alternatively, we could have a top-level trunk directory: trunk /proj1 /proj2 tags /proj1 /tag1 /tag2 ... /proj2 /tag1 /tag2 ... branches /proj1 /br1 /br2 ... /proj2 /br1 /br2 ... But that would require inclusion of a "/trunk" dead chicken in checkouts: svn co svn+ssh://svn.zope.org/repos/trunk/ZODB which seems unnecessary to me. Thoughts? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

5 10

is threading in zope really useful ?
by sathya 27 Apr '04

27 Apr '04

greetings, I read somewhere that each zope thread acquires the global python interpreter lock before processing a request and until it releases it the other zope threads are essentially locked out. Does that mean zope threads are eventually serialized and there are no benefits to be gained from concurrent execution ? Any thoughts ? your ideas are much appreciated ty sathya

1 0

Re: The bleak Future of Zope?
by Andreas Jung 27 Apr '04

27 Apr '04

Some remarks from my side as a Zope2 core developer on this issue: The Z2 community and development is currently at a bad point: - very few people are contributing to the Z2 in terms of new code and bug fixes (see the tons of open bugs in the collector) - very few people are willing to contribute to documentation The reasons for this situation from my prospective: - Lots of Z2 people are working now on Plone projects. Plone currently attracts more people because the important and interesting projects are done there. Paul Everits goal to grow Zope by 10 times might happen through Plone, not through Zope itself - The Z2 development is badly managed. The 2.7 release has been delayed for one year or so. - ZC is currently the bottleneck for Z2. Several important people have left and I don't see any new blood there. I see that ZC is focusing on Z3 for the future. That is a legitimate goal from their own prospective but it does not reflect the needs of lots of users. Speaking for the company I am mainly working for have invest a lot time, code and money into Z2 development. E.g. our complete company-critical CMS is currently being rewritten on the base of Zope2. Several other products will be also Zope 2 based. Zope 3 is in the current stage not really an alternative for complex sites, portals and applications in the style we are working on. Z3 might become interesting if we have Plone-like functionality available. To speak with Maiks words: Z3 is attractive as an academic project to try out things and concepts but it does not attract people in the current stage...maybe in two years from now but currently most people are attracted by working and usable solutions like Plone. - The zope.org community site is a mess. Lots of outstanding problems are not fixed, the performance of the site is more than poor (it takes ages to login, it takes ages to load pages), usability (e.g. when you perform a software release) is bad. We need for Zope2 - a better and open management for Z2 releases: If ZC can not provide the resources in terms of time and manpower, the coordination and release management should be given to the community. I am sure that more are willing to contribute more than at the moment. Several companies maintain their own Zope version with lots of extensions and it would be fine to see this stuff in a common repository. - a clear statement from ZC to the future of Zope 2. Zope 2.8 and Zope 2.9 are considered as a migration path for Zope 3 where the Z2 support should be dropped after these releases (as far as I can remember the vision of Rob a while ago). Who will manage Z2 releases after dropping the Z2 support? The last solution would be a Z2 code fork if we can not come to common agreement on the Z2 future but a code fork would be really the last and absolutely worst solution for everyone. - Fix the most outstanding critical bugs on zope.org and speed it up. Zope.org is currently a very bad figurehead for Zope. If ZC can not solve the problems....either give the responsibility to other people or just close zope.org. >From my own prospective as developer I would like to see that Z2 development over the next two or three years continues because there is too much Z2 legacy code in the world and not everyone is interested in following the migration path for Z3. To be honest I doubt that large custom applications can be migrated with a justifiable amount of time and money (just because they are completely bound to Z2 components and its architecture). To clarify my standpoint: I am not an opponent of Zope3 but Zope 3 does not convince me in the current stage and gives me little attraction for the projects I am working on....it just can not compete with Zope 2 if you are building large-scale systems at this time. Andreas ----- ZOPYX - Software Development & Consulting Andreas Jung www.zopyx.com -- Andreas Jung -------------------------------- www.zopyx.com Software-Entwicklung und Consulting Andreas Jung

16 34

performance tuning of ZODB
by Syver Enstad 27 Apr '04

27 Apr '04

I've used the cache_size paramter to the constructor of the DB to good effect. Are there any further gotchas for ensuring that the ZODB stays in memory as much as possible?

4 7

What do we want to bring from CVS to Subversion
by Jim Fulton 27 Apr '04

27 Apr '04

I'm working on the cvs to subversion conversion for the ZODB, Zope 2, and Zope 3 projects. I'm currently doing the conversion of the full history with tags and branches. This is taking a long time and creating a huge repository, which is OK, but, do we really need that much history? I see 3 options: 1. Convert the full history with branches. This will create a rather large and complex repository. 2. Convert the mainline history, but leave off the branches. 3. Start with a clean slate and simply import the current head. Note that, for Zope 2 and ZODB, current maintenance branches will remain in CVS. I think that option 2 provides a nice compromise. The main disadvantage of it is that it will leave current development branches high and dry. I'm not sure how big an issue this is. In theory, these could be committed to the subversion heav via patch files. Thoughts? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

4 8

Unicode in Zope 2 (ZMI, Archetypes, Plone, Formulator)
by Bjorn Stabell 27 Apr '04

27 Apr '04

While we're all waiting for Zope 3 and Plone 3, I'd like to know what the "standard practice" way of using Unicode with Zope 2. In particular, we'd like to store all text as Unicode in the ZODB, and have Zope do the encoding/decoding as automatically and transparently as possible. We've been using Zope 2's ZPublisher to do this encoding/decoding for over 2 years, and it's working fine. We just have to ensure that we set the appropriate encoding in a HTTP Content-type header, and that we add :utext/ustring:ENCODING to HTML form field names. Regardless of what you may have heard, THIS WORKS FINE! We also store Unicode, not UTF-8 (or other encodings), strings in the ZODB. The problem we're running into are with other components, basically making our Unicode-with-Zope experience, shall we say, less than ecstatic (To put it this way, I seem to lose hair much faster when dealing with Unicode problems :) I'm wondering why components/products aren't all relying on the ZPublisher for Unicode encoding/decoding? Is there another standard way? Here is a summary of what we've found: ZMI * gets charset from manage_page_charset encoding * relies on ZPublisher for encoding (but doesn't do decoding, see below) * in PropertyManager you can add ustrings, but since it doesn't add :ENCODING to the field names, you get a Unicode error when trying to save since it tries to decode the text assuming ASCII (big problem) * DTML Methods/Documents: doesn't support Unicode (annoying) * can't use Unicode id's (not a big problem) Archetypes: * gets charset from portal_url.getCharset() or portal_properties.site_properties.default_charset * doesn't rely on ZPublisher, does its own encoding/decoding * returns encoded strings, not Unicode strings, to Zope apps, leading to problems such as: - SearcableText() encodes, and as such can't be used with Unicode-aware ZCatalogs - transform() encodes (and because of that SearchableText() sometimes decodes/encodes 2 times instead of 0 times) - get()ing field values will encode them, so if you want Unicode, you have to decode yourself (adding both unnecessary overhead for data access, and unnecessary dependency on the global variable for the charset) Plone: * no special Unicode support for HTML forms; relies on Archetypes Formulator: * gets charset from manage_page_charset (same as ZMI), but can be overridden * stores field values as encoded text (not Unicode), but lets you specify which encoding to use (confusingly calls this "unicode" mode) * messages are stored as UTF-8 (hardcoded) I suggest this way of dealing with Unicode right now in Zope 2: (1) Let ZPublisher do the encoding/decoding of form input and HTML output: a. Always set a character encoding in a HTTP Content-type request b. Always append :ustring/utext/ulines/utokens:ENCODING to field names of fields that support Unicode (we may need some library code to make this easier) (2) Store Unicode strings directly in the ZODB. The ZODB is perfectly capable of storing strings in Python's internal Unicode format; no need to encode the text to UTF-8 or some other encoding. (3) Encode/decode yourself when reading from/ writing to other external data sources such as files and other databases. Do it just before you write, or just after you read, so that as much code as possible can be encoding-agnostic. Keep the encoding/decoding as close to the "source data" as possible. The best way to do it is (in most cases) to specify the encoding on the IO stream, and let Python do the encoding/decoding for you transparently. If possible, get the encoding from the external data source (e.g., the file) instead of relying on a magical global variable. If you have to rely on a global variable, let it be manage_page_charset. (4) [This is really just advice...] Resist patching your code to work with components that doesn't deal with Unicode. Others are likely having the same problem, so to avoid ending up with lots of ugly patches (that are the source of mysterious Unicode problems), fix the problem at its source: the other component. It's really not that difficult to fix (if we agree on how it should be fixed ;) None of the above components handles Unicode in this way, but it seems to be how the Unicode support in Zope 2 was meant to be used. Let me know if there is another better way, but please do let me know... I think we need to resolve this once and for all or I know some people that'll just go mad (or bald, or both) :) I'll be willing to contribute patches, but since this applies to so many products, it would be good to get some consensus first. At the very least, can we create a "Standard Unicode Practices" page? Bye, -- Bjorn Stabell <mailto:bjorn@exoweb.net>

6 8

Z2 HEAD in rough shape for b/c
by Chris McDonough 26 Apr '04

26 Apr '04

Using the HEAD with old databases and code is a real adventure these days for these reasons: - It doesn't start, due to a tiny config bug which I'll check a fix in for shortly. - Many products create instances of the class Persistence.PersistentMapping.PersistentMapping which no longer exists (CMFCore's SkinsTool and CMFDefault DiscussionTool are two of these). These things either need to be migrated somehow to Persistence.PersistentMapping (*before* an update to the HEAD) or we need to provide some sort of b/c alias. That seems tricky given the current source layout. I've added an alias that seems to work: sys.modules['Persistence.PersistentMapping'] = sys.modules['Persistence.mapping'] But this is really odd because I don't know under what circumstance this alias shadows the effect of the line: from Persistence.mapping import PersistentMapping in Persistence's __init__.py module. If there is a shadowing problem with this alias, I'd suggest we go back to the old layout (where there really was a PersistentMapping.py module in Persistence) on the HEAD for sanity's sake. I'm sure there are reasons against this, but the other option is to require every CMF/Plone user to go through a really delicate migration before upgrading to 2.8, which doesn't seem like a good idea. - The HEAD apparently no longer allows URL access to methods that have inherited function docstrings, where older versions did. Lots of things depend for better or worse on this behavior and this breaks a lot of things. They are all shallow fixes (go add a docstring, typically) but it's difficult to figure out where you need to go do this. - The ZODBMountPoint product relies on a method of Connection objects named _getMountedConnection, which apparently no longer exists. This breaks any mounted databases (which breaks dbtab, which breaks sessions, which breaks lots of other things). Hopefully this is simple to fix, I haven't looked yet. - ZopeVersionControl doesn't work anymore due to various ZODB changes. - Various security declarations that used to allow access now deny. Bug reports don't get any more vague than that, but then again nobody's security machinery is any more vague about reporting its reasons for denying access than Zope's. ;-) I can try to make headway on some of this. Is anyone else willing to help? FWIW, after working through (and around) some of these issues I was able to get Plone running on the current HEAD (well, at least plope.com, which uses a lot of Plone), so I can happily report that Zope 2.8 isn't 100% backwards-incompatible yet. ;-) dig-the-hole-and-fill-it-up-again'ly y'rs, - C

2 2

RE: [Zope3-dev] Re: [Zope-dev] Re: The bleak Future of Zope?
by Tim Peters 26 Apr '04

26 Apr '04

[Max M, tests a zip file of .pyd files in Zope3, on Windows] > I finally got around to testing this, and it works *exactly* as I > hoped. I downloaded Z3 from CVS, and Tim's zip file. > > I unpacked the zip file into the Z3 directory, and it started the > first time. Yippee! That's what I expected, but it's always a pleasant surprise when software works <0.8 wink>. > I don't know about others, but I don't need anything more than this. > > If either Chris or Tim would put that structure up at a saner > location, I will write a How-to about it. > > Basically I just need a web adress that stays current to use in the > How-to, so that I could point to somewhere like: > > http://zope.org/Members/tim_one/z3builds > > > Where they could get the latest zip file. http://zope.org/Members/tim_one/Zope3-pyd.zip is in place now. I'll put up a similarly named one for Zope2 this afternoon (EDT). They'll be replaced from time to time (when Zope 2/3's C code changes), but will retain theses names.

1 0

Need help with http://dev.zope.org/Subversion
by Jim Fulton 26 Apr '04

26 Apr '04

I copied: http://dev.zope.org/CVS to: http://dev.zope.org/Subversion Can help by converting the Subversion wiki to reflect subversion? :) The new writable subversion repository will be at: svn+ssh://svn.zope.org/repos I hope to have something in place for testing on Monday. For example, to do a writable check out of Zope 3, you will use: cvs co svn+ssh://svn.zope.org/repos/Zope3/trunk Zope3 I still have to figure out whether to use the sv server or HTTP for public checkouts, but I'll *probably use HTTP, in which case you'd use: cvs co http://svn.zope.org/repos/Zope3/trunk Zope3 for public checkouts. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

4 6

Decouple Interface and zope.interface (Martijn was right)
by Jim Fulton 26 Apr '04

26 Apr '04

My plan has been to make zope.interface interfaces and Interface interfaces interoperate. Martijn Faassen has been complaining about this for a while, because, to use zope.interface and Interface together required a hard-to-install facade. If we drop the requirement to make the interfaces interoperate (e.g. extend one another, or be used in the same declarations), then, with a small change that Martijn has suggested, we can use zope.interface in Zope 2 without any other changes. The Interface package isn't used very heavily in Zope 2. When it is used, it's used solely for documentation. After further reflection, I don't really see a compelling reason to endure pain to make Interface interfaces usable for Zope 3 interface operations. IOW: Martijn was right and I was wrong. :) I propose to drop the dubious requirement for interoperability between Interface and zope.interface. I propose, instead, to add a requirement that they be able to coexist. This will require changing the name of an attribute used internally by zope.interfaces. It will also allow us to simplify certain aspects of the implementation in zope.interface that exists solely to support Interface interfaces. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

3 2