September 2002 - Zope

RE: ZODB/ZSS High Availablity, was: RE: [Zope] Zope Myths?
by sean.upton＠uniontrib.com 13 Sep '02

13 Sep '02

Thought about this using Intermezzo - basically the same idea. I think the problem is that these are on-demand replication mechanisms that get the file when they need it instead of push-to-other-node-on-write sort of deals. I'm not posive about that. The other problem is how these filesystems handle changes to large files (like FileStorage). There, might, in theory, be a way to integrate this with something like DirectoryStorage, but I think you would need to do some heavy scripting to get things to work just right (perhaps to the point that you might as well manually set up scripts to low-tech replicate files?). Sean -----Original Message----- From: Paul Winkler [mailto:pw_lists@slinkp.com] Sent: Thursday, September 12, 2002 4:13 PM To: zope(a)zope.org Subject: Re: ZODB/ZSS High Availablity, was: RE: [Zope] Zope Myths? On a related note... thinking of using coda (http://www.coda.cs.cmu.edu/) to address the single point of failure for our zeo storage. But I've never used coda. Anybody done it with zope? Issues? --PW -- Paul Winkler "Welcome to Muppet Labs, where the future is made - today!" _______________________________________________ Zope maillist - Zope(a)zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )

2 1

Problem with DCOracle2
by Mateusz 13 Sep '02

13 Sep '02

Hi! I have problem with "DCOracle2" compilation under Linux7.1. I use gcc 2.96 and python2.1 and I have Oracle 8.17. gcc -fPIC -g -O2 -Wall -Wstrict-prototypes -I/var/zope/include/python2.1 -I/var/zope/include/python2.1 -DHAVE_CONFIG_H -I/home/oracle/product/8.1.7/rdbms/demo -I/home/oracle/product/8.1.7/network/public -I/home/oracle/product/8.1.7/plsql/public -I/home/oracle/product/8.1.7/rdbms/public -DORACLE8i -c ././dco2.c -o ./dco2.o In file included from /home/oracle/product/8.1.7/rdbms/demo/oci.h:1648, from ././dco2.c:109: /home/oracle/product/8.1.7/rdbms/demo/oci1.h:148: warning: function declaration isn't a prototype In file included from /home/oracle/product/8.1.7/rdbms/demo/oci.h:1673, from ././dco2.c:109: /home/oracle/product/8.1.7/rdbms/demo/ociap.h:6271: warning: function declaration isn't a prototype /home/oracle/product/8.1.7/rdbms/demo/ociap.h:8753: warning: function declaration isn't a prototype /home/oracle/product/8.1.7/rdbms/demo/ociap.h:8759: warning: function declaration isn't a prototype /tmp/ccWy24E5.s: Assembler messages: /tmp/ccWy24E5.s:1038: Warning: Unrecognized .section attribute: want a,w,x /tmp/ccWy24E5.s:1038: Warning: Unrecognized .section attribute: want a,w,x /tmp/ccWy24E5.s:1038: Error: Rest of line ignored. First ignored character is `,'. /tmp/ccWy24E5.s:1071: Warning: Unrecognized .section attribute: want a,w,x /tmp/ccWy24E5.s:1071: Warning: Unrecognized .section attribute: want a,w,x /tmp/ccWy24E5.s:1071: Error: Rest of line ignored. First ignored character is `,'. /tmp/ccWy24E5.s:1104: Warning: Unrecognized .section attribute: want a,w,x /tmp/ccWy24E5.s:1104: Warning: Unrecognized .section attribute: want a,w,x /tmp/ccWy24E5.s:1104: Error: Rest of line ignored. First ignored character is `,'. /tmp/ccWy24E5.s:1150: Warning: Unrecognized .section attribute: want a,w,x /tmp/ccWy24E5.s:1150: Warning: Unrecognized .section attribute: want a,w,x /tmp/ccWy24E5.s:1150: Error: Rest of line ignored. First ignored character is `,'. /tmp/ccWy24E5.s:1183: Warning: Unrecognized .section attribute: want a,w,x /tmp/ccWy24E5.s:1183: Warning: Unrecognized .section attribute: want a,w,x /tmp/ccWy24E5.s:1183: Error: Rest of line ignored. First ignored character is `,'. /tmp/ccWy24E5.s:1229: Warning: Unrecognized .section attribute: want a,w,x /tmp/ccWy24E5.s:1229: Warning: Unrecognized .section attribute: want a,w,x /tmp/ccWy24E5.s:1229: Error: Rest of line ignored. First ignored character is `,'. /tmp/ccWy24E5.s:1262: Warning: Unrecognized .section attribute: want a,w,x /tmp/ccWy24E5.s:1262: Warning: Unrecognized .section attribute: want a,w,x /tmp/ccWy24E5.s:1262: Error: Rest of line ignored. First ignored character is `,'. /tmp/ccWy24E5.s:1324: Warning: Unrecognized .section attribute: want a,w,x /tmp/ccWy24E5.s:1324: Warning: Unrecognized .section attribute: want a,w,x /tmp/ccWy24E5.s:1324: Error: Rest of line ignored. First ignored character is `,'. /tmp/ccWy24E5.s:1357: Warning: Unrecognized .section attribute: want a,w,x /tmp/ccWy24E5.s:1357: Warning: Unrecognized .section attribute: want a,w,x /tmp/ccWy24E5.s:1357: Error: Rest of line ignored. First ignored character is `,'. Could anybody help me, please? Regards Mateusz

4 4

Memory consumtion of a zope instance
by Andreas Lauser 13 Sep '02

13 Sep '02

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi List, I'm doing shared zope hosting, where everbody gets his own instance (using the INSTANCE_HOME environment variable). but when I looked at the memory consumtion I noticed that every instance grabed between 11 and 18 Megabytes of RAM. Since our server is not too blessed with RAM I wondered, whether it was possible to reduce the RAM comsumtion of the instances. I thought at doing a 'late' fork: start a supervisor process at first which loads python and initializes the default products. An Instance would fork this process and initialize the products special to it. In this scheme python's and most of zope's RAM pages are copy on write on most UNIX systems (-> they don't eat your mem). As a site effect, instance startup would be greatly accelerated. Is something like this available, or planned? Maybe I can code this on my own, but at first I've got to gather a lot more experience with zope. Thanks AND - -- "I used to think that the brain was the most wonderful organ in my body. Then I realized who was telling me this." -- Emo Phillips -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE9f7oThlekfiEFJv4RAhMFAKC5ZUTcVFLXnKfHaz8ahYnn9ulvlQCfXu1R WH2H5dNYyrAFRgdc9XF7rbg= =SFpK -----END PGP SIGNATURE-----

6 10

Zope digest, Vol 1 #2342 - 48 msgs
by TGERMAINE＠UPei.CA 13 Sep '02

13 Sep '02

I am away on vacation until Sept 16. If your inquiry is urgent please resend your request to Chris Martin (cmartin(a)upei.ca) and he will look after it. Alternatively, you can call Chris at 566-0604 Tom Germaine Access Services Administrator

1 0

RE: [Zope] problem accessing Photo Folder
by Meilicke, Scott 13 Sep '02

13 Sep '02

Do you have imagemagick installed? I used to get that error when imagemagick wasn't installed correctly or the convert program wasn't in the path. Scott -----Original Message----- From: Roel Van den Bergh [mailto:roel@planetinterior.com] Sent: Thursday, September 12, 2002 9:39 AM To: Zope Org Subject: [Zope] problem accessing Photo Folder When accessing a Photo Folder we get the following error: Site Error An error was encountered while publishing this resource. Error Type: IOError Error Value: [Errno 32] Broken pipe Zope 2.5.1 build from source, Python 2.1.3 build from source rpm, RH 7.3, Dell PowerApp 120 How can we correct this? TIA - - - Roel Van den Bergh _______________________________________________ Zope maillist - Zope(a)zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev ) The information contained in this email message may be privileged and is confidential information intended only for the use of the recipient or any employee or agent responsible for delivering it to the intended recipient. Any unauthorized use, distribution or copying of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify the sender immediately and destroy the original message and all attachments from your electronic files.

2 1

Major problems with slow catalog queries
by Lea Smith 13 Sep '02

13 Sep '02

Hi, I have been developing a site around Zope and the CMF and have got to a stage of loading on the data we have and testing the site. (zope 2.5.1, python 2.1.3, testing on win2000) The loaded data creates around 11,000 objects which are all cataloged in the default portal_catalog. The more objects that are added and cataloged the slower searches become as well as the whole site in general. Have used the product callprofiler and found 90% of the page creation time is more often than not taken up by the script which queries the catalog and returns the catalog results. The script does nothing else. This time generally varies between 8 to 30 seconds, but sometimes as long as 5 minutes! I can't work out what is going on with the catalog, or what is a normal query time for a catalog loaded with this amount of info. And the slow response is going to get worse as visitors add more content. Not good! Have tried recreating the catalog just incase something had gone wrong there. Played around with caching using RAMCache Manager as well as the control_panel cache settings with no reliable improvement. (Need to read up on the cache usage as wasn't quite sure of how best to set it up). Any ideas on normal response times for the catalog and anything I can look at or try would be fantastic. Thanks for any help. __________________________________________________ Do You Yahoo!? Everything you'll ever need on one web page from News and Sport to Email and Music Charts http://uk.my.yahoo.com

4 3

Re: [Zope] Memory consumtion of a zope instance
by David Shaw 13 Sep '02

13 Sep '02

On Thursday, September 12, 2002, at 12:00 PM, zope-request(a)zope.org wrote: > I'm glad this thread was started, because I was wondering about memory > consumption on my win2k machine running one instance. It seems to > start off > using between 25-30 megs. Now it's sitting at 38 megs. When I login > or > access certain folders like the root, it goes up between 70-90 megs. It > takes along time for met to access the root folder. I only have 10-15 > objects in the root. We used Zope at my previous job in a very large-scale way. We had two zope front-ends and a ZEO backend. Our Zope instances (one per host, ZEO on its own box) would usually stabilize with about 30,000 items in the cache and about 255MB of RAM each (our ZCatalog had over 1 million objects in it, and most every page was catalog-driven).

1 0

ZODB/ZSS High Availablity, was: RE: [Zope] Zope Myths?
by sean.upton＠uniontrib.com 12 Sep '02

12 Sep '02

I have been doing a lot of thinking about odb/storage/zss replication lately, but I haven't had a chance to implement these practices yet, so your mileage, insights, and opinions may vary from these thoughts... If the thing that makes replication hard is constant change of lots of interdependant data, a meaningful snapshot system as close to the database software as possible (i.e. DirectoryStorage's snapshots, not LVM's) likely mitigates that risk by providing reasonable assurance of atomicity. If the replication process itself has problems part way through transfer (a low tech soutions like find+cpio over nfs would), it is up to the sysadmin to write scripts to: 1 - Keep multiple areas for replication -> Stage the entire replication in a temp dir before putting it in the place that it is used by ZSS software -> since there is no way to do a transactional file copy of multiple files, how about using symlinks, and moving the symlink on completion of a full, atomic transfer and completed storage consistency check? 2 - Have clustering software resource takeover scripts (i.e. heartbeat resource scripts) evaluate: a. if the storage it is about to use is good, & b. if the last transfer failed, use the last _good_ full replicated set of files. c. The above two checks must be done before starting the ZSS process on the backup server node. Mostly, I can't see how shared storage (DAS/SAN) can provide the same risk-avoidance levels that could be done with the above practices, unless you have some ways of mirroring the last good copy of your odb storage within the same shared storage (replication between two places on the same storage; I assume snapshots and scrips on the secondary node to check consistency of storage/db like 2(a) above could come in handy for this too)? Sean -----Original Message----- From: tomas(a)fabula.de [mailto:tomas@fabula.de] Sent: Thursday, September 12, 2002 1:55 PM To: Bill Anderson Cc: sean.upton(a)uniontrib.com; pw_lists(a)slinkp.com; zope(a)zope.org Subject: Re: [Zope] Zope Myths? On Thu, Sep 12, 2002 at 11:12:27AM -0600, Bill Anderson wrote: > On Thu, 2002-09-12 at 00:21, tomas(a)fabula.de wrote: > > On Wed, Sep 11, 2002 at 03:46:43PM -0700, sean.upton(a)uniontrib.com wrote: [Big hardware vs. replication] > Well, in that case, your network is a single point of failure, too. :^) Assuming just one network, assuming just one connectivity provider. Problem is that replication solutions depend heavyly on the type of application (slowly changing sets of files being the easiest and rapidly changing data sets with complex interdependencies (e.g. high-volume databases) the hardest. > Expensive, well that depend son what you are doing. For under 35000 you > can have just shy of a 1TB of file space, with snapshot capability, > multi-machine fail over, and a whole lot more. That cost includes two > machines running Linux with fail over. It depends on your needs, and > your uptime/availability requirements. ...or you can have ten cheapo off the shelf servers hosted at wildly different places... (OK, it's more like 0.5TB then ;) > For example, if you are running a site like cbsnewyork.com, 25-35 grand > is not that much. If you are running a small site, then you don't need > it. My point was that it (it being ZEO/ZODB/ZOPE) _can_ scale to that. > I've done it. > [...] > > Well, speaking as a former tester of SAN technology, it would appear > things have changed dramatically since your experiences. :) > Yes, but this was mainly my point: if you have access to knowledge and experience with those things -- then you may go for it. If you don't... it's just a point against it. > The configuration/setup is essentially the same as with SCSI, in fact, > Fiber channel uses the SCSI subsystem in the OS. The underlying system > is as robust as the SCSI system, since it is SCSI just over a different > medium. > > [RAID system doing funny things] > > I've never seen this with the Fiber Channel Arrays I dealt with. But > then again, they had two or more controllers. :^) Of course not -- but I take that you *know* what you are doing. This vendor didn't (at some point I realized that), but heck, it wasn't my job, I had enough on the plate myself. [...] > Same thing with fibre channel SAN tech, the range is measured in miles. > I know of several SANs that are spread over multiple states. You can > literally have a fail over datacenter. Yes. It's a tradeoff. I just wanted to point out that experience with those things is one of the points to consider (besides application type, requirements and cost). It'd be interesting to know (I'm not a Zope guy) how well Zope as an application would play in each camp. Thanks -- tomas

3 2

RE: [Zope] Collaborative syncing?
by Adam Getchell 12 Sep '02

12 Sep '02

Jaroslav, I've been rather happy with Zsync. It will tell you when files are different versions between source and server. You could set up Zsync'ing to be bidirectional (e.g., DEVEL --> PROD and PROD --> DEVEL), then you'd go into whatever instance you wish to push the changes from. It is a push mechanism, but it's amazingly useful and robust. After adding the appropriate methods in my Zsyncer, I was actually able to use it to copy an entire site from one server to another. (In this case, it caused a nasty loop because I had a Virtual Site Root and Squid sitting in front of it, but that was my configuration, and I was able to fix it.) In our case, we have DEVEL-->TEST-->PROD, and occasionally I set PROD-->DEVEL. (I turn it off when not needed.) --Adam -----Original Message----- From: Dieter Maurer [mailto:dieter@handshake.de] Sent: Thursday, September 12, 2002 2:42 PM To: lsh(a)wo.cz Cc: zope(a)zope.org Subject: Re: [Zope] Collaborative syncing? Jaroslav Lukesh writes: > I was install and try Zsync (latest on Zope 251/W32), it is nice, but I do > not know to make simple sync in case when source and target was changed > each different file and have some files more or deleted. ZSyncer requires that changes are only made at one place. ZSyncing is an unidirectional process only. Dieter _______________________________________________ Zope maillist - Zope(a)zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )

1 0

Sudden ZClass breakage
by Tim Wilson 12 Sep '02

12 Sep '02

Hi everyone, I use a simple little ZClass to post announcements on my site. Recently, and without any help from me, adding new announcements fails with the following traceback. I didn't change a single thing related to this ZClass, but it just broke. Can anyone offer a suggestion about what is going on? -Tim  -- Tim Wilson | Visit Sibley online: | Check out: Henry Sibley HS | http://www.isd197.org | http://www.zope.com W. St. Paul, MN | | http://slashdot.org wilson(a)visi.com | <dtml-var pithy_quote> | http://linux.com

2 2