- Zope-Dev - Zope lists

RE: [Zope-dev] Zope server speed
by MANOTTI Alessandro 06 Dec '01

06 Dec '01

Thank you for your suggestions (thanks to Matt). I will try to make smaller but more requests (1000 requests of 1000 records). I will even try to "bypass" Zope and write an "external method" in Python (I know Pytho very well). About this solution: can you tell me what I have to do to use ZOracle directly in Python (import ... ? ) ? Thanks to everybody! -----Original Message----- From: Matthew T. Kromer [mailto:matt@zope.com] Sent: Thursday, December 06, 2001 15:27 To: MANOTTI Alessandro; zope-dev(a)zope.org Subject: Re: [Zope-dev] Zope server speed Hi Alessandro, Zope and Python combined aren't too bad if you write some wrappers that help you manipulate data; the problem comes about when you try to do large naive queries -- there is a LOT of overhead in taking Oracle data, encapsulating it in Python objects, and then wrapping that in the Zope machinery. I have a sample database in Oracle of US Zip codes -- it has about 45,000 records in it. I can pull this into DCOracle2 on the C adapter side (dco2.so) in less than a second. However, wrapping each result cell in a python object blows that up into about 9 seconds. If I were to iterate over the whole thing in Zope (which I haven't timed) it would get a LOT longer, because Zope is enforcing policy on every object accessed. So, one of the ways to make that go faster is to use python modules 'outside' of Zope (ie external methods) to perform large calculations which takes a ZOracleDA connection object to get a cursor object on the database, does the query directly, and formats and returns the results. I realize this isn't ideal, but Zope is not optimized for the case of "extract thousands of records from the database and format them in a huge table." Zope likes "extract hundreds of records from the database and format them in a big table" instead. With each release, Zope has more features that enable you to program through the web, but when you're doing performance intensive calculations, it's usually time to write a python module and plug it in to zope (via an external method, or by writing a product). If you have the time to learn, I suggest the product route as being more satisfying, but it can be more frustrating as well. ----- Original Message ----- From: "MANOTTI Alessandro" <alessandro.manotti(a)bfeurope.com> To: <zope-dev(a)zope.org> Sent: Thursday, December 06, 2001 8:52 AM Subject: [Zope-dev] Zope server speed > > > HI all, > > > > I have a problem: > > > > my company is looking for an application server. I know Zope (I presented > > it), and a friend of mine presented Macromedia ColdFusion. > > > > We made some tests about the speed of the systems, arghh!!! Zope is VERY > > slow compared to ColdFusion! > > > > Some examples: > > > > I used Zope with DCOracle2 drivers to connect and extract 257000 (yes! it > > is a test... two hundred fiftyseven thousand records!) and Zope took more > > than 10 minutes, since I stopped it after that time (call SQL and print > > data via <dtml-var ... > TAG). > > > > The same PC, the same server, ColdFusion took about 2 minutes to complete > > the operation!!! > > > > Coldfusion is more than 5 times faster than Zope!!! > > > > PHASE 2: > > > > I said: ok, maybe Zope is slower that Coldfusion in TAG formatting... and > > I tried to get the records and perform a calculation, without no output... > > > > Sigh! the results are the same! > > > > I am depressed, since I talk about Zope very well in my company, but > > now... > > > > Coldfusion and Zope are similar, so we shall select ColdFusion to develop > > web applications. > > > > SIGH !!! > > > > Can I try something to increase Zope power?! > > > > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > Alessandro Manotti > > Presidente dell'Associazione "RIUSA" > > > > Sito web: http://riusa.apritisesamo.net > > email: riusa(a)email.it > > mailing-list: riusa-ml(a)yahoogroups.com > > > > > > _______________________________________________ > Zope-Dev maillist - Zope-Dev(a)zope.org > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) >

2 1

Zope server speed
by MANOTTI Alessandro 06 Dec '01

06 Dec '01

> HI all, > > I have a problem: > > my company is looking for an application server. I know Zope (I presented > it), and a friend of mine presented Macromedia ColdFusion. > > We made some tests about the speed of the systems, arghh!!! Zope is VERY > slow compared to ColdFusion! > > Some examples: > > I used Zope with DCOracle2 drivers to connect and extract 257000 (yes! it > is a test... two hundred fiftyseven thousand records!) and Zope took more > than 10 minutes, since I stopped it after that time (call SQL and print > data via <dtml-var ... > TAG). > > The same PC, the same server, ColdFusion took about 2 minutes to complete > the operation!!! > > Coldfusion is more than 5 times faster than Zope!!! > > PHASE 2: > > I said: ok, maybe Zope is slower that Coldfusion in TAG formatting... and > I tried to get the records and perform a calculation, without no output... > > Sigh! the results are the same! > > I am depressed, since I talk about Zope very well in my company, but > now... > > Coldfusion and Zope are similar, so we shall select ColdFusion to develop > web applications. > > SIGH !!! > > Can I try something to increase Zope power?! > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > Alessandro Manotti > Presidente dell'Associazione "RIUSA" > > Sito web: http://riusa.apritisesamo.net > email: riusa(a)email.it > mailing-list: riusa-ml(a)yahoogroups.com > >

4 3

Re : Open Letters and Zope 3
by Godefroid Chapelle 06 Dec '01

06 Dec '01

Q: What is Zope 3X? A: Zope 3X is Zope rebuilt from the ground up, applying the lessons learned from Zope 2 and CMF. <snip> Q: Will Zope 3X support DTML? A: Probably not. Q: Will Zope 3 support DTML? A: Yes. I would like to get a better idea of what 'Zope rebuilt from the ground up includes'. For me, Zope is also used to produce non Markup language (for ex emails) which you cannot use ZPT for. So would the new Zope been limited to ML ? -- Godefroid Chapelle BubbleNet sprl rue Victor Horta, 18 / 202 1348 Louvain-la-Neuve Belgium Tel + 32 (10) 459901 Mob + 32 (477) 363942 TVA 467 093 008 RC Niv 49849

9 13

Re: [Zope-dev] More signal 11 restarts....
by Gilles Lavaux 06 Dec '01

06 Dec '01

Hello Which config/tool make that your zope restart automatically??? Gilles -----Original Message----- From: Leonardo Rochael Almeida <leo(a)hiper.com.br> To: zope-dev <zope-dev(a)zope.org> Date: Wednesday, December 05, 2001 20:07 Subject: Re: [Zope-dev] More signal 11 restarts.... > > >Hi Harald, > >On Wed, 2001-12-05 at 14:32, Harald Koschinski wrote: >> Daniel Duclos wrote: >> >> > I have a zope that is dumping signal 11 every 40 minutes or so. I have >tried >> >recompile python 2.1.1 with-threads without-pymalloc, recompile Zope with it, >> >recompile ZPAtterns, recompile and instal MYSQL for Python 0.9.1, upgraded to >> >Zope 2.4.3, all this on a Debian Linux box. >> > Nothing changed... still restarting... >> > Anybody, please, has any ideia on this matter? >> > Please, let me know if there's any relevant info that I forgot to mention >> >abot my case! >> > Thanks in advance!! >> > >> >> I have the same problem since we are live :-((( >> >> I have the same versions running like you and I tried the same to fix - >> but no change. >> >> What is your state with this problem. Fixed ? How? > >It's not fixed, but we managed to make it bearable so as not to lose the >client. > >by replacing LoginManager with exUserFolder we managed to bring down the >zope restart time from 5 min (we have a HUGE Data.fs) to 20 secs, and by >installing ZEO we brought down the restart time to between 2 and 8 secs. >By increasing the caching of requests we managed to increase the time >between restart from 15 min to 2 hours. That and a nicely formated >Apache error page for Proxy Errors, for the lucky bastards who happen to >hit the server in the exactly 2 seconds of restart, managed to calm down >the client enough for us to breath. > >I'll try the requestprofiler tip Chris gave, but I don't have much hope, >since we use mysql for authentication and the site is authenticated >mostly everywhere. > >We are seriously considering droping mysql for postgresql. > > Cheers, Leo > >-- >Ideas don't stay in some minds very long because they don't like >solitary confinement. > > > >_______________________________________________ >Zope-Dev maillist - Zope-Dev(a)zope.org >http://lists.zope.org/mailman/listinfo/zope-dev >** No cross posts or HTML encoding! ** >(Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) > > >

2 1

Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB
by Chris Withers 06 Dec '01

06 Dec '01

Casey Duncan wrote: > > I'm not sure I want to store the indexes in the ZODB, just index ZODB data at > a low level. Ah, okay, and yes, in that case, I am in complete agreement ;-) (the level I'm aiming at is just to be able to index python objects, I'll leave plugging that into the ZODB architecture up to someone who understands it better...) > Yup, I think I have a solution, but it'll involve some coding ;^) Ooo...care to explain? :-) > I would rather avoid having to use a relational database unless I have to. > Perhaps the index pluggability could be made to support different backends > (like FileStorage et al does). Yeah, unfortunately, the difficult bit is combining queries: gimme the results where index1=='fish' and index2 is between 2 and 5kg. if index1 is in SQL and index2 is in ZODB, for example, how would you go about efficiently combining results? > > That said, I wasn't aware of Matt's work up until very recently. I'd love > > to see an Indexer that didn't require an RDB (or BerkleyDB :-P) and scaled > > to GigaBytes of Data... > > Yup, me too. Well, I'm just purchasing my copy of Managing Gigabytess now ;-) > OK, I'm available all this week, but I'm not as available the next two weeks. > Lets find a good time. I'm available any time and date, just as long as I get a coupla days notice... cheers, Chris

6 19

corrupted data.fs
by Chris Kratz 05 Dec '01

05 Dec '01

Hello all, My apologies if this is not appropriate on this list and my apologies for this "book". But, we have been having more and more problems with Zope and it is getting quite frustrating. I thought since I had posted on the zope list and no one had any suggestions that I would try this list next. The behavior we are seeing is as follows. We are using versions heavily. There are two of us primary developers, soon to be three working heavily within the same data.fs. We use versions, often running several at the same time. When the code is ready, we save the version incorporating the changes to the live site. Ever since we upgraded to Zope 2.4.1, saving the version rarely actually does that. It will save some objects, but usually only a few. We have to then go to each object and save it again (sometimes making changes) and continue saving the version until we get all the locked objects unlocked and committed. This is very annoying, but not a show stopper. What is starting to scare me is that I wonder if this problem with the version commits is somehow causing the corruption of the database. Now, I say all this because it may or may not be related to the other problems we continue to have. We periodically, get folders that are no longer visible within the ZMI and produce a keyerror in filestorage.py (traceback at end of msg). But, the same folder is still happily being served externally (ie it generates the web pages based within that folder without error to our customers). This tells me that it is still in the ZODB, it is just unreachable via the ZODB. Since we cannot access it from within zope any longer, we have to load the data.fs from backup and restore the offending folder losing any work done since the previous night. It never seems to effect any other parts of the website and it has happened in several different areas. This is becoming more and more of a showstopper especially when we lose code. 1. I have packed the database. 2. I have run fsrecover.py over the data.fs. - Nothing to recover. So, if we aren't corrupted, why can we no longer work with that folder? 3. I have exported and imported the offending folder without effect (XML and regular). 4. We were packing the zope database nightly via a cron job to control ZODB bloat. I have turned that off for now on the theory that somehow mixing overnight packs with versions is causing problems. Help!!! This continues to happen and it is really scary when your database continues to get corrupted underneath you on a live site when you are in heavy development. Can anyone give any ideas on how to track down this problem? Could it be with the versioning code? Is there a better way to work with a production site and still get the benefits of the versioning? Is versioning just dangerous and shouldn't be used? Pardon my tone, but I am a little frustrated. I will say that the last time this happened, thankfully, that particular folder was being worked on in a version and even though the folder was inaccessible outside of the version, when working within the version, it was still there. So, when that version was cut in, we were again able to work with that folder. This time however, the folder in question wasn't begin worked on in a version. Thank you for your time, Sincerely -Chris ------------- **Traceback** Error Type: KeyError Error Value: ?f Traceback Traceback (innermost last): File /usr/local/zope2.4/Zope-2.4.0-linux2-x86/lib/python/ZPublisher/Publish.py, line 171, in publish File /usr/local/zope2.4/Zope-2.4.0-linux2-x86/lib/python/ZPublisher/mapply.py, line 160, in mapply (Object: manage_main) File /usr/local/zope2.4/Zope-2.4.0-linux2-x86/lib/python/ZPublisher/Publish.py, line 112, in call_object (Object: manage_main) File /usr/local/zope2.4/Zope-2.4.0-linux2-x86/lib/python/Shared/DC/Scripts/Bindings.py, line 324, in __call__ (Object: manage_main) File /usr/local/zope2.4/Zope-2.4.0-linux2-x86/lib/python/Shared/DC/Scripts/Bindings.py, line 354, in _bindAndExec (Object: manage_main) File /usr/local/zope/lib/python/App/special_dtml.py, line 244, in _exec (Object: manage_main) File /usr/local/zope/lib/python/DocumentTemplate/DT_In.py, line 711, in renderwob (Object: objectItems) File /usr/local/zope/lib/python/DocumentTemplate/DT_In.py, line 839, in sort_sequence (Object: objectItems) File /usr/local/zope2.4/Zope-2.4.0-linux2-x86/lib/python/ZODB/Connection.py, line 519, in setstate File lib/python/ZODB/FileStorage.py, line 588, in load (Object: /usr/local/zope/var/Data.fs) File lib/python/ZODB/FileStorage.py, line 564, in _load (Object: /usr/local/zope/var/Data.fs) KeyError: ?f ---------------- **Instalation Info** Platform: RH 7.2 Zope Version: (Zope 2.4.1 (binary release, python 2.1, linux2-x86), python 2.1.1, linux2) Python Version: 2.1.1 (#1, Aug 8 2001, 11:57:42) [GCC 2.96 20000731 (Red Hat Linux 7.1 2.96-81)] System Platform: linux2 Products installed: eCrumbler (Installed product CookieCrumbler (CookieCrumbler-0.3)) ExternalMethod (Installed product ExternalMethod (External Method-1-0-0)) MIMETools (Installed product MIMETools) MailHost (Installed product MailHost (MailHost-1-2-0)) OFSP (Installed product OFSP (OFSP-1-0-0)) PlugIns (Installed product PlugIns (PlugIns-0-4-3p1)) PluginIndexes (Installed product PluginIndexes) PythonScripts (Installed product PythonScripts (PythonScripts-2-0-0)) Renderable (Installed product Renderable) SiteAccess (Installed product SiteAccess (SiteAccess-2-0-0)) Squishdot (Installed product Squishdot (Squishdot-1-2-1)) StandardCacheManagers (Installed product StandardCacheManagers (StandardCacheManagers-1-1-0)) ThreadSafeCounter (Installed product ThreadSafeCounter (0.0.3)) TinyTablePlus (Installed product TinyTablePlus (TinyTablePlus-0-9)) UpdateSupport (Installed product UpdateSupport (UpdateSupport-1_0_0)) WarpFramework (Installed product WarpFramework (2001-05-02-1222)) ZCatalog (Installed product ZCatalog (ZCatalog-2-2-0)) ZGDChart (Installed product ZGDChart (ZGDChart 0.5.1)) ZGadflyDA (Installed product ZGadflyDA) ZPatterns (Installed product ZPatterns (ZPatterns-0-3-0)) ZPoPyDA (Installed product ZPoPyDA) ZPsycopgDA (Installed product ZPsycopgDA) ZSQLMethods (Installed product ZSQLMethods) ZSyncer (Installed product ZSyncer) ZWiki (Installed product ZWiki (ZWiki-0-9-6)) ZopeTutorial (Installed product ZopeTutorial (Zope Tutorial 1.0)) iTrack (Installed product iTrack (0.2.0)) -- Systems Analyst/Programmer VistaShare LLC chris.kratz-no-spam-please-vistashare.com (replace -no-spam-please- by @) www.vistashare.com

2 1

RE: [Zope-dev] Searching
by sean.upton＠uniontrib.com 05 Dec '01

05 Dec '01

I think this could be done well at the application level, above the Catalog if done right, but one would have to have good synonym translation table specific to their application. I do this in an application using a TinyTable with translation information for abbreviations (i.e. VW printed in a classified auto ad gets expanded to Volkswagen, so people can search for both)... If there was any sort of thesaurus database of sorts available to your application, I suppose, in theory, one could do some synonym-oriented indexing, profided that you index a method that brute-force weights the real words by outputting them a greater factor of times than the synonym (i.e. index a method that outputs the original text 3 times and synonyms for the meaningful words (non-stop words) in that text perhaps once. This is not difficult from a development perspective, but involves having synonym data and deciding what words should be translated to their synonyms, and what ones should not. I think the real issue here is not having a database of synonyms - and, at that, you would have to have one for whatever language(s) you needed to support. Sean -----Original Message----- From: Dieter Maurer [mailto:dieter@handshake.de] Sent: Wednesday, December 05, 2001 1:24 PM To: Dirk Datzert Cc: zope-dev(a)zope.org Subject: Re: [Zope-dev] Searching Dirk Datzert writes: > I'm looking for a search tool which can do semantic/synonym search. > > Can this be done via ZCatalog ? Not out of the box. You could place something before ZCatalog (which parses subquerie for search term and expands them into a set of synonyms). But you would need to work quite hard to get true semantic search... Dieter _______________________________________________ Zope-Dev maillist - Zope-Dev(a)zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )

1 0

Catalog wierdness ...
by Joachim Werner 05 Dec '01

05 Dec '01

Hi! Any ZCatalog experts out there? I am suddenly getting this with a ZCatalog (it is actually a subclassed version of the ZCatalog, almost a one-to-one copy of the CMF Catalog tool): ------ Zope Error Zope has encountered an error while publishing this resource. Error Type: KeyError Error Value: -" ---------------------------------------------------------------------------- ---- Troubleshooting Suggestions This resource may be trying to reference a nonexistent object or variable - ". The URL may be incorrect. The parameters passed to this resource may be incorrect. A resource that this resource relies on may be encountering an error. For more detailed information about the error, please refer to the HTML source for this page. If the error persists please contact the site maintainer. Thank you for your patience. Traceback (innermost last): File /home/ftp/root/kue-projekt/dev_8080/lib/python/ZPublisher/Publish.py, line 224, in publish_module File /home/ftp/root/kue-projekt/dev_8080/lib/python/ZPublisher/Publish.py, line 187, in publish File /home/ftp/root/kue-projekt/dev_8080/lib/python/Zope/__init__.py, line 226, in zpublisher_exception_hook (Object: TransparencySupport) File /home/ftp/root/kue-projekt/dev_8080/lib/python/ZPublisher/Publish.py, line 171, in publish File /home/ftp/root/kue-projekt/dev_8080/lib/python/ZPublisher/mapply.py, line 160, in mapply (Object: manage_addSmartTextBlock) File /home/ftp/root/kue-projekt/dev_8080/lib/python/ZPublisher/Publish.py, line 112, in call_object (Object: manage_addSmartTextBlock) File /home/ftp/root/kue-projekt/dev_8080/lib/python/Products/SmartSections/SmartT extBlock.py, line 63, in manage_addSmartTextBlock (Object: TransparencySupport) File /home/ftp/root/kue-projekt/dev_8080/lib/python/Products/SmartSections/SmartS ectionElement.py, line 30, in manage_addSectionElement (Object: TransparencySupport) File /home/ftp/root/kue-projekt/dev_8080/lib/python/Products/OrderedFolder/Ordere dObjectManager.py, line 118, in _setObject (Object: TransparencySupport) File /home/ftp/root/kue-projekt/dev_8080/lib/python/OFS/ObjectManager.py, line 330, in _setObject (Object: TransparencySupport) File /home/ftp/root/kue-projekt/dev_8080/lib/python/Products/SmartSections/SmartS ectionElement.py, line 177, in manage_afterAdd (Object: ElementWithAttributes) File /home/ftp/root/kue-projekt/dev_8080/lib/python/Products/SmartSections/SmartS ectionElement.py, line 172, in manage_routineTasks (Object: ElementWithAttributes) File /home/ftp/root/kue-projekt/dev_8080/lib/python/Products/SmartSections/SmartS ection.py, line 89, in contentIndexObject (Object: TransparencySupport) File /home/ftp/root/kue-projekt/dev_8080/lib/python/Products/Kontentor/Katalog.py , line 130, in indexObject (Object: LockableItem) File /home/ftp/root/kue-projekt/dev_8080/lib/python/Products/ZCatalog/ZCatalog.py , line 498, in catalog_object (Object: LockableItem) File /home/ftp/root/kue-projekt/dev_8080/lib/python/Products/ZCatalog/Catalog.py, line 438, in catalogObject File /home/ftp/root/kue-projekt/dev_8080/lib/python/Products/PluginIndexes/TextIn dex/TextIndex.py, line 381, in index_object (Object: PluggableIndex) File /home/ftp/root/kue-projekt/dev_8080/lib/python/Products/PluginIndexes/TextIn dex/GlobbingLexicon.py, line 166, in getWordId File /home/ftp/root/kue-projekt/dev_8080/lib/python/Products/PluginIndexes/TextIn dex/GlobbingLexicon.py, line 206, in assignWordId File /home/ftp/root/kue-projekt/dev_8080/lib/python/ZODB/Connection.py, line 519, in setstate File /home/ftp/root/kue-projekt/dev_8080/lib/python/ZODB/FileStorage.py, line 588, in load (Object: /home/ftp/root/kue-projekt/dev_8080/var/Data.fs) File /home/ftp/root/kue-projekt/dev_8080/lib/python/ZODB/FileStorage.py, line 584, in _load (Object: /home/ftp/root/kue-projekt/dev_8080/var/Data.fs) File /home/ftp/root/kue-projekt/dev_8080/lib/python/ZODB/FileStorage.py, line 1905, in _loadBack KeyError: (see above) ----- Very strange, as it seems to be a rather low-level error. Any hints? Joachim

2 2

absolute_url FTP upload and CatalogAware
by Niclas Kuehne 05 Dec '01

05 Dec '01

2 1

absolute_url FTP upload and CatalogAware (PLAIN TEXT!)
by Niclas Kuehne 05 Dec '01

05 Dec '01

When I upload a file of a certain class (HTMLDocument) that is CatalogAware the entry in the catalog does not contain the real absolute_url. Instead it shows "http://myserver:21/Zope/Dir1/Dir2/object.html" where "myserver" is the hostname of the linux server, 21 is the FTP port used by Zope and Dir1 & Dir2 are directories. When I clear the catalog and add the documents by clicking "Find Objects" the absolute_url is correct. What is happening here? Any help? Niclas Kuehne Reply to: niclas(a)eho2000.de

1 0