From: Toby Dickenson [mailto:tdickenson@geminidataloggers.com] Yes. The default of 4 threads is arguably too high. All my applications work better with 2 threads, and double the cache size (but still using the same total amount of memory)
I'm experimenting with that now. A catalog query that used to take 40-100 seconds with the default target number of objects in the cache now takes 1-2 seconds (or less) with the target number increased to 15,000. This 100x speed increase comes at the cost of using 0.5 GB of resident memory with 4 threads.
They need to be seperate to give isolation between transactions. Some classes that allocate large amounts of memory for immutabe objects have custom optimisations to share those between threads.
This is very unfortunate, especially given that most of these objects won't be changed often. Something like the the way virtual memory works (copy-on-change) would have saved a lot of memory, or allowed you to keep many more objects in the cache, increasing speed drastically. Is this in any way related to the inherent global interpreter lock limitation in Python? Bye, -- Bjorn
On Thursday 23 October 2003 10:12, Bjorn Stabell wrote:
From: Toby Dickenson [mailto:tdickenson@geminidataloggers.com] Yes. The default of 4 threads is arguably too high. All my applications work better with 2 threads, and double the cache size (but still using the same total amount of memory)
I'm experimenting with that now. A catalog query that used to take 40-100 seconds with the default target number of objects in the cache now takes 1-2 seconds (or less) with the target number increased to 15,000. This 100x speed increase comes at the cost of using 0.5 GB of resident memory with 4 threads.
15k objects sounds like alot for a catalog query. 0.5G/15k = average 33k per object...... that sounds more like the size of a document object rather than a btree node. Both of these facts suggest you might be able to improve performance by optimising your application. There are pages in control panel that will tell you what these 15k objects are.
They need to be seperate to give isolation between transactions. Some classes that allocate large amounts of memory for immutabe objects have custom optimisations to share those between threads.
This is very unfortunate, especially given that most of these objects won't be changed often. Something like the the way virtual memory works (copy-on-change) would have saved a lot of memory, or allowed you to keep many more objects in the cache, increasing speed drastically.
yes, this is certainly true.
Is this in any way related to the inherent global interpreter lock limitation in Python?
no. -- Toby Dickenson
Dear Zopers Sorry for taking your time, but I am looking for advice by someone with practical Zope/Python experience(*). At the moment I am starting to learn Python and Zope. I wanted to do this for some time, but recently I got in touch with a musician who is interested in a music database application(**). Thus, I thought I'd take my chance and do this in Zope. After reading quite some material about Zope I am still hesitating how to approach the problem. Should I use ZODB or MySQL? DTML or ZPT? I want to make sure that the database is safe and can be backed up. Can I access the ZODB using Python programs? How does this work? Should I first learn Python or Zope? Can I convert an Excel sheet to read into Zope (using Python)? What would you suggest for a good solution and how do I learn the required skills in the shortest amount of time? Are there good comprehensive examples to look at? It seems that I learn best from looking at other people's code. Thanks a lot and kind regards André Meyer Delft, The Netherlands (*) I have 20 years of experience in computing and worked mainly in Java for the last 8 years after using more exciting languages and have a lot of experience in OO, database modelling, knowledge engineering, etc. (*) This is about pieces of old music that are played in various instrumental settings. One query is to search for combinations of pieces with a given set of instruments in order to prepare for a concert. The interface needs to be easy to handle and avoid re-entering the same information more than once (e.g. a composer or publisher). Therefore, I foresee that comboboxes are used a lot for selecting previously entered values/objects. The application first used on a personal workstation, but may be made available for multi-user access (eventually contributions) on the Web later on.
On Thu, 2003-10-23 at 02:40, Andre Meyer wrote:
I am looking for advice by someone with practical Zope/Python experience(*).
Ah... well, I'll try to help anyway. :-)
At the moment I am starting to learn Python and Zope.
Excellent.
After reading quite some material about Zope I am still hesitating how to approach the problem.
Reading about Zope is good, but *doing* something in Zope is the only real way it makes sense. Initial confusion is normal.
Should I use ZODB or MySQL?
Depends. If you have heavy experience with LAMP-style web apps or RDBMS administration, you might find using MySQL more intuitive. If not, ZODB is a probably perfectly reasonable choice.
DTML or ZPT?
Also depends. They're both well-supported, though ZPT is more fervently advocated. If you have any experience with ASP, you'll probably find DTML more intuitive. But ZPT is the better tool. The main thing is you should only use templating languages for templating... plan on using Python scripts for any significant logic.
I want to make sure that the database is safe and can be backed up.
Good idea. Both ZODB and an RDBMS can meet this criteria if managed well.
Can I access the ZODB using Python programs?
Well... Zope is a Python program. But if you mean, can the data be used by a non-zope program, the answer is sort of. Yes, you can... but if sharing data with another system is a key priority, I'd let that tip me toward using an RDBMS.
How does this work?
First, set aside two or three weeks. :-)
Should I first learn Python or Zope?
People with several years of programming experience can learn 80% of Python in a day or so. I'd start here: http://diveintopython.org/ It's going to be tough to do much in Zope without understanding how to construct basic Python expressions. Fortunately, Python expressions are really quite intuitive... for the most part.
Can I convert an Excel sheet to read into Zope (using Python)?
Yes. Install Python's csv module, export the spreadsheet to csv, import csv into an external method and follow the docs for how to parse a csv and how to communicate with an external method.
What would you suggest for a good solution and how do I learn the required skills in the shortest amount of time?
Start working on a real-world problem. And have the Zope Book at your side: http://www.zope.org/Documentation/Books/ZopeBook/current
Are there good comprehensive examples to look at? It seems that I learn best from looking at other people's code.
Zope comes with examples... I'd start by looking at those. HTH, Dylan
On Thu, Oct 23, 2003 at 08:45:53AM -0700, Dylan Reinhardt wrote:
What would you suggest for a good solution and how do I learn the required skills in the shortest amount of time?
Start working on a real-world problem. And have the Zope Book at your side: http://www.zope.org/Documentation/Books/ZopeBook/current
Also, since zope is a quite different paradigm than what you're used to, I'd follow the "Plan to throw one away" rule. I'm still maintaining a site that was built by "porting" a Weblogic Portal site to Zope. Let's just say it has some serious code smell. I'm also still maintaining another site built by the same group. It was their second effort and it's *much* nicer.
Are there good comprehensive examples to look at? It seems that I learn best from looking at other people's code.
Zope comes with examples... I'd start by looking at those.
There's also a lot of good examples in the online Zope Book. #zope on irc is a good resource too (when it's active... sometimes it's pretty dead). And this mailing list is invaluable. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's THUNDEROUS STINKY MONITOR DOG! (random hero from isometric.spaceninja.com)
Andre Meyer wrote at 2003-10-23 11:40 +0200:
... Should I use ZODB or MySQL?
If you have highly structure mass data, use a relational database. If you have unstructured data (text, images, binary data, ...) use the ZODB.
DTML or ZPT?
Use ZPT (with a few exceptions: in Z SQL Methods, for Javascript/CSS/email generation).
I want to make sure that the database is safe and can be backed up. Can I access the ZODB using Python programs?
Yes. But a ZODB storage can only be accessed by a single process at a time. You will use ZEO (Zope Enterprise Objects) when several processes need to access the ZODB at the same time. ZEO is a storage server that centralized access to the storage.
How does this work? Should I first learn Python or Zope?
Like with all OO systems, the basics are quite simple but the complete story is very complex. Thus, the best approach (in my view): Look at the Python tutorial (--> python.org). Read the Zope book. Look at the Zope tutorial (--> in Zope integrated) These give you the basics. Learn more as needed later...
Can I convert an Excel sheet to read into Zope (using Python)?
You can do this via "csv" as an intermediary. You will need Zope's "External Method"s. -- Dieter
Thanks a lot for your help. I have a better picture now and started with reading Python in Nutshell, the Python tutorial, the Zope book and tutorial, as well as the Zope Bible. Together with your tips, this should get me further soon. The basics are quick to learn, but the complete picture takes some time to grasp. I think I will use a ZPT/ZODB solution as this is closest to OO and separation of logic and presentation. thanks again and best wishes André Meyer Delft, The Netherlands
Toby Dickenson wrote at 2003-10-23 10:33 +0100:
On Thursday 23 October 2003 10:12, Bjorn Stabell wrote:
From: Toby Dickenson [mailto:tdickenson@geminidataloggers.com] Yes. The default of 4 threads is arguably too high. All my applications work better with 2 threads, and double the cache size (but still using the same total amount of memory)
I'm experimenting with that now. A catalog query that used to take 40-100 seconds with the default target number of objects in the cache now takes 1-2 seconds (or less) with the target number increased to 15,000. This 100x speed increase comes at the cost of using 0.5 GB of resident memory with 4 threads.
15k objects sounds like alot for a catalog query.
We observed that the "BTrees.IOBTree.IOBTree" records used for ZCatalog Metadata can be very large (our Metadata contains a "Description" field which often has several kb). -- Dieter
participants (6)
-
Andre Meyer -
Bjorn Stabell -
Dieter Maurer -
Dylan Reinhardt -
Paul Winkler -
Toby Dickenson