- Zope-Dev - Zope lists

Re: [Zope-dev] [ZODB-Dev] Splitting a large ZODB file storage - any light-weight approach?
by Jens Vagelpohl 30 Dec '09

30 Dec '09

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Andreas Jung wrote: > Hi there, > > we have some huge ZODB file storage (60-80 GB each) and we want > to split the storage into a number of smaller storage (easier > to handle for backup and packing). The storages themselves only > contain a few (Zope 2) folders - let's say folders A, B, C - > within the root and we want to create seperate storages for A, B, C. > > Is there some easier approach than using the Zope 2 CopyManager API > for moving data around. The ZODB storage iterator API comes to my mind... > is this a suitable approach or is there some other light-weight approach > (compared to using CopyManager API)? Why move any data between ZODBs at all? In this situation I would try the following: - create a new empty ZODB. This one holds the root. - copy the old Data.fs into three new storage server buildouts. Attach to each with a ZEO client and delete the folders you do not need in that particular storage in the ZMI or via zopectl debug. - pack each storage so the deleted data gets trashed and the files shrink. - mount the "folder" storages into the new empty root using mount points on the clients that will use the storages. jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAks7CpEACgkQRAx5nvEhZLKjYgCdGI71f/yNJS3EIpZxkyFVBmSe qhMAoJ8RBK2eq27P67Iki4h4MYMZ1kZB =FC8f -----END PGP SIGNATURE-----

3 2

Zope Tests: 6 OK
by Zope Tests Summarizer 30 Dec '09

30 Dec '09

Summary of messages to the zope-tests list. Period Tue Dec 29 12:00:00 2009 UTC to Wed Dec 30 12:00:00 2009 UTC. There were 6 messages: 6 from Zope Tests. Tests passed OK --------------- Subject: OK : Zope-2.10 Python-2.4.6 : Linux From: Zope Tests Date: Tue Dec 29 20:38:10 EST 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-December/013287.html Subject: OK : Zope-2.11 Python-2.4.6 : Linux From: Zope Tests Date: Tue Dec 29 20:40:10 EST 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-December/013288.html Subject: OK : Zope-2.12 Python-2.6.4 : Linux From: Zope Tests Date: Tue Dec 29 20:42:10 EST 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-December/013289.html Subject: OK : Zope-2.12-alltests Python-2.6.4 : Linux From: Zope Tests Date: Tue Dec 29 20:44:10 EST 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-December/013290.html Subject: OK : Zope-trunk Python-2.6.4 : Linux From: Zope Tests Date: Tue Dec 29 20:46:10 EST 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-December/013291.html Subject: OK : Zope-trunk-alltests Python-2.6.4 : Linux From: Zope Tests Date: Tue Dec 29 20:48:10 EST 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-December/013292.html

1 0

Zope 2.12.2 SyntaxError on installation
by Marius Gedminas 29 Dec '09

29 Dec '09

Following the guide at http://docs.zope.org/zope2/releases/2.12/INSTALL.html#buildout-instances I get the following error: mg@platonas:~/src/akl-website-z2.12-experiment $ python2.5 bootstrap.py Creating directory '/home/mg/src/akl-website-z2.12-experiment/bin'. Creating directory '/home/mg/src/akl-website-z2.12-experiment/parts'. Creating directory '/home/mg/src/akl-website-z2.12-experiment/develop-eggs'. Generated script '/home/mg/src/akl-website-z2.12-experiment/bin/buildout'. mg@platonas:~/src/akl-website-z2.12-experiment $ time bin/buildout Upgraded: zc.buildout version 1.4.3, setuptools version 0.6c11; restarting. Generated script '/home/mg/src/akl-website-z2.12-experiment/bin/buildout'. Installing instance. Getting distribution for 'Zope2'. src/AccessControl/cAccessControl.c:598: warning: ‘intargfunc’ is deprecated src/AccessControl/cAccessControl.c:599: warning: ‘intargfunc’ is deprecated src/AccessControl/cAccessControl.c:600: warning: ‘intintargfunc’ is deprecated src/AccessControl/cAccessControl.c:606: warning: ‘intargfunc’ is deprecated src/Record/_Record.c:340: warning: ‘intargfunc’ is deprecated src/Record/_Record.c:341: warning: ‘intargfunc’ is deprecated src/Record/_Record.c:342: warning: ‘intintargfunc’ is deprecated File "build/bdist.linux-i686/egg/Zope2/utilities/load_site.py", line 248 body = ("<html><head><title><dtml-var title_or_id></title> ^ SyntaxError: EOL while scanning single-quoted string File "/home/mg/tmp/buildout-eggs/tmprWUwxL/Zope2-2.12.2-py2.5-linux-i686.egg/Zope2/utilities/load_site.py", line 248 body = ("<html><head><title><dtml-var title_or_id></title> ^ SyntaxError: EOL while scanning single-quoted string Got Zope2 2.12.2. which seems to be https://bugs.launchpad.net/zope2/+bug/501265 Then buildout proceeds as if nothing is wrong. Getting distribution for 'zope.app.publication==3.7.0'. Got zope.app.publication 3.7.0. Getting distribution for 'zope.app.form==3.8.1'. Got zope.app.form 3.8.1. Getting distribution for 'zope.viewlet==3.5.0'. Got zope.viewlet 3.5.0. Getting distribution for 'zope.contentprovider==3.5.0'. Got zope.contentprovider 3.5.0. Getting distribution for 'zope.component==3.7.1'. Got zope.component 3.7.1. Getting distribution for 'zLOG==2.11.1'. Got zLOG 2.11.1. Getting distribution for 'tempstorage==2.11.2'. Got tempstorage 2.11.2. Getting distribution for 'Persistence==2.11.1'. Got Persistence 2.11.1. Getting distribution for 'ExtensionClass==2.11.3'. Got ExtensionClass 2.11.3. Getting distribution for 'DateTime==2.12.0'. Got DateTime 2.12.0. Getting distribution for 'Acquisition==2.12.4'. Got Acquisition 2.12.4. Getting distribution for 'zope.app.testing==3.6.2'. Got zope.app.testing 3.6.2. Getting distribution for 'zope.app.appsetup==3.11'. Got zope.app.appsetup 3.11. Generated script '/home/mg/src/akl-website-z2.12-experiment/bin/runzope'. Generated script '/home/mg/src/akl-website-z2.12-experiment/bin/zopectl'. Generated interpreter '/home/mg/src/akl-website-z2.12-experiment/bin/py'. After that, it refuses to create a Data.fs and start up: mg@platonas:~/src/akl-website-z2.12-experiment $ bin/runzope Traceback (most recent call last): File "bin/runzope", line 93, in <module> Zope2.Startup.run.run() File "/home/mg/tmp/buildout-eggs/Zope2-2.12.2-py2.5-linux-i686.egg/Zope2/Startup/run.py", line 21, in run starter.prepare() File "/home/mg/tmp/buildout-eggs/Zope2-2.12.2-py2.5-linux-i686.egg/Zope2/Startup/__init__.py", line 87, in prepare self.startZope() File "/home/mg/tmp/buildout-eggs/Zope2-2.12.2-py2.5-linux-i686.egg/Zope2/Startup/__init__.py", line 264, in startZope Zope2.startup() File "/home/mg/tmp/buildout-eggs/Zope2-2.12.2-py2.5-linux-i686.egg/Zope2/__init__.py", line 47, in startup _startup() File "/home/mg/tmp/buildout-eggs/Zope2-2.12.2-py2.5-linux-i686.egg/Zope2/App/startup.py", line 72, in startup DB = dbtab.getDatabase('/', is_root=1) File "/home/mg/tmp/buildout-eggs/Zope2-2.12.2-py2.5-linux-i686.egg/Zope2/Startup/datatypes.py", line 283, in getDatabase name = self.getName(mount_path) File "/home/mg/tmp/buildout-eggs/Zope2-2.12.2-py2.5-linux-i686.egg/Zope2/Startup/datatypes.py", line 300, in getName self._mountPathError(mount_path) File "/home/mg/tmp/buildout-eggs/Zope2-2.12.2-py2.5-linux-i686.egg/Zope2/Startup/datatypes.py", line 273, in _mountPathError "No root database configured") ZConfig.ConfigurationError: No root database configured Huh? Result of that load_site.py error, or a missing manual step that I should have known to do despite it being not mentioned in the installation docs? I was brave enough to specify INSTANCEHOME as '.' in my zope.conf, because I strongly believe hardcoding absolute paths is dumb. mg@platonas:~/src/akl-website-z2.12-experiment $ cat etc/zope.conf %define INSTANCE . python $INSTANCE/bin/py instancehome $INSTANCE Cheers! Marius Gedminas -- http://pov.lt/ -- Zope 3 consulting and development

4 14

moving the zope.app.* packages out of the ZTK: towards a plan
by Martijn Faassen 29 Dec '09

29 Dec '09

Hi there, Let me sketch out some ideas for a plan: * we create a new project, zopeapp, with the same structure as the zopetoolkit (sans documentation) * we pull the zope.app.* packages from the ZTK and move them into zopeapp. * we make sure we have a way to regularly run the zopeapp tests in conjunction with the ZTK. * we then move towards a release of ZTK 1.0 (see Hanno's mail for lots of things we need to work out) * we also release zopeapp 1.0 in conjunction with it. This is just a backwards compatibility KGSish thing. We explicitly don't strive to document it, etc. * we announce that we strive to deprecate what's in zopeapp 1.0 and that we expect users to shift to use ZTK packages as much as possible and report back any difficulties they have with this. * this will help us determine whether there is still useful code left in zopeapp that we wish to salvage into the ZTK or otherwise maintain. * this will also help us determine which packages in zopeapp are more generally useful, and strive to isolate them from the rest of zopeapp. zope.formlib is an example of this. * depending on our experiences in Zope 2, Zope 3 and Grok apps we can decide whether we can put zopeapp in the deep freeze: not maintain it anymore. * we then also look into shunting away the zope.app.* packages from the top-level SVN into an "this is old stuff area" by that time. Note that we also have a long-standing idea of a "wider KGS" which contains the ZTK plus perhaps zopeapp plus more, such as z3c.form. This is a related exercise. Once we have some discussion we can hopefully flesh out this plan, put in some dates and such, and put the plan in the Zope Toolkit documentatino. Regards, Martijn

2 3

Zope Toolkit - packages with zope.app dependencies
by Hanno Schlichting 29 Dec '09

29 Dec '09

Hi. I allowed myself to work on the Zope Toolkit over x-mas and make some decisions to get some progress ;-) The good news is that all our herculean efforts to unclutter the dependency structures have really payed off. Only very few zope.* packages still have dependencies on zope.app packages. I excluded those misbehaving packages from the ZTK to reflect this and noted them in a "unresolved-dependencies" option. The ZTK no longer contains any zope.app packages with one exception. There's a number of problematic packages, where I'd like some other opinions on how to proceed. zope.traversing It has a dependency on zope.app.publication. Given the central role of zope.traversing I allowed it and zope.app.publication to stay in the ZTK, but moved it to the under-review option. We have talked about refactoring the traversal and publication story in the past, without seeing any consensus on how to move forward with it. My feeling tells me, that a pure renaming of zope.app.publication to zope.publication or similar code reshuffling into other packages isn't really helpful in this case. ACTION: Until we see further progress on clarifying the semantics of traversal and publication, I'd leave zope.app.publication in the ZTK. zope.catalog and zope.intid Both of these have a dependency on zope.app.testing. Their tests heavily rely on a functional test setup with a ZODB and persistent data. They don't seem to need a publisher or a web server, though. ACTION: Remove them from the ZTK. Once their test setup no longer relies on zope.app.testing, they should move back into the ZTK. zope.file I think this package is just misnamed and should have been zope.app.file. It's a full blown content type implementation, with dependencies on zope.app.form, providing browser view code, browser menus and quite a bit of zope.app integration. It's test setup needs a full blown ZODB and publisher environment. ACTION: I'd remove this from the ZTK until someone is willing to refactor it to provide a minimal file implementation. zope.formlib and friends zope.formlib depends on zope.app.form which in turn depends on many zope.app packages. zope.html and zope.mimetype both depend on zope.formlib or zope.app.form. Various people myself included looked at the zope.formlib / zope.app.form split to see if it can be refactored in a sensible way. I think the conclusion has so far been: there's just no way. There isn't a sensible set of basic principles or interfaces in formlib, that can form a base for a higher level zope.app.form. All the packages using formlib today actually make use of the widgets in zope.app.form and thus basically on the entire functionality. With z3c.form we have another independent form library from the Zope community, which has seen at least as widespread if not more adoption. Preferring one over the other doesn't make much sense to me. ACTION: At this point I'd suggest to drop zope.formlib, zope.app.form and packages that depend on it from the ZTK. zope.testbrowser This package offers functional browser tests and consequently needs a way to set up a ZODB and publisher with some default application configuration. I don't see how this can be achieved without depending on a specific application environment like zope.app.appsetup or Zope2/App. Zope2 depends on the package and it makes sense to include it on the application server level - but not on a toolkit level for building frameworks. ACTION: I'd suggest to drop this package from the ZTK. Merry x-mas :) Hanno

7 17

Documentation on the Zope 2 release process
by Andreas Jung 29 Dec '09

29 Dec '09

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I started documenting the Zope 2 release process (Zope 2.12 only for now): svn+ssh://svn.zope.org/repos/main/zope2docs/trunk/maintenance/index.rst Hints and feedback appreciated. Andreas - -- ZOPYX Ltd. & Co KG \ zopyx group Charlottenstr. 37/1 \ The full-service network for your D-72070 Tübingen \ Python, Zope and Plone projects www.zopyx.com, info(a)zopyx.com \ www.zopyxgroup.com - ------------------------------------------------------------------------ E-Publishing, Python, Zope & Plone development, Consulting -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAks6PwAACgkQCJIWIbr9KYyqTgCeLfknpgqeGLKpPM9Lj6aU57of kDgAoMUXidpW3eOfpIY8AQI7O5lFK8cM =Tr+l -----END PGP SIGNATURE-----

2 1

Zope Tests: 6 OK
by Zope Tests Summarizer 29 Dec '09

29 Dec '09

Summary of messages to the zope-tests list. Period Mon Dec 28 12:00:00 2009 UTC to Tue Dec 29 12:00:00 2009 UTC. There were 6 messages: 6 from Zope Tests. Tests passed OK --------------- Subject: OK : Zope-2.10 Python-2.4.6 : Linux From: Zope Tests Date: Mon Dec 28 20:55:03 EST 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-December/013281.html Subject: OK : Zope-2.11 Python-2.4.6 : Linux From: Zope Tests Date: Mon Dec 28 20:57:03 EST 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-December/013282.html Subject: OK : Zope-2.12 Python-2.6.4 : Linux From: Zope Tests Date: Mon Dec 28 20:59:03 EST 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-December/013283.html Subject: OK : Zope-2.12-alltests Python-2.6.4 : Linux From: Zope Tests Date: Mon Dec 28 21:01:03 EST 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-December/013284.html Subject: OK : Zope-trunk Python-2.6.4 : Linux From: Zope Tests Date: Mon Dec 28 21:03:03 EST 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-December/013285.html Subject: OK : Zope-trunk-alltests Python-2.6.4 : Linux From: Zope Tests Date: Mon Dec 28 21:05:03 EST 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-December/013286.html

1 0

Avoid deprecation warnings in the testrunner
by Fabio Tranchitella 29 Dec '09

29 Dec '09

Hello, I think we should provide a zope.testing package which does not emit deprecation warnings while running the testrunner on other packages, otherwise there is no way to really understand why the package you are working on is using a deprecated API. I propose the patch attached to this message to zope.testing. It tries to avoid importing zope.testing.doctest unless it is really needed. In this way, developers can see the DeprecationWarning with the path of *their* code, and fix it, instead of receiving the DeprecationWarning because the testrunner imports zope.testing.doctest itself. Before applying the patch, running the tests on another package produced this output: ----------------------------------------------------------------------- .../zope/testing/testrunner/debug.py:23: DeprecationWarning: zope.testing.doctest is deprecated in favour of the Python standard library doctest module from zope.testing import doctest Running zope.testing.testrunner.layer.UnitTests tests: ... ----------------------------------------------------------------------- After applying my patch, I get this: ----------------------------------------------------------------------- .../zope/interface/tests/test_adapter.py:405: DeprecationWarning: zope.testing.doctest is deprecated in favour of the Python standard library doctest module from zope.testing import doctest Running zope.testing.testrunner.layer.UnitTests tests: ... ----------------------------------------------------------------------- Comments? Thanks, Fabio

4 19

PyPI access for zope.contenttype
by Fred Drake 28 Dec '09

28 Dec '09

Can someone set me up with PyPI privs for zope.contenttype? Thanks! -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller

2 2

Zope Tests: 6 OK
by Zope Tests Summarizer 28 Dec '09

28 Dec '09

Summary of messages to the zope-tests list. Period Sun Dec 27 12:00:00 2009 UTC to Mon Dec 28 12:00:00 2009 UTC. There were 6 messages: 6 from Zope Tests. Tests passed OK --------------- Subject: OK : Zope-2.10 Python-2.4.6 : Linux From: Zope Tests Date: Sun Dec 27 20:37:04 EST 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-December/013275.html Subject: OK : Zope-2.11 Python-2.4.6 : Linux From: Zope Tests Date: Sun Dec 27 20:39:05 EST 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-December/013276.html Subject: OK : Zope-2.12 Python-2.6.4 : Linux From: Zope Tests Date: Sun Dec 27 20:41:05 EST 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-December/013277.html Subject: OK : Zope-2.12-alltests Python-2.6.4 : Linux From: Zope Tests Date: Sun Dec 27 20:43:05 EST 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-December/013278.html Subject: OK : Zope-trunk Python-2.6.4 : Linux From: Zope Tests Date: Sun Dec 27 20:45:05 EST 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-December/013279.html Subject: OK : Zope-trunk-alltests Python-2.6.4 : Linux From: Zope Tests Date: Sun Dec 27 20:47:05 EST 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-December/013280.html

1 0