RE: [Zope] Thousands of Objects - how do they manifest? (newbie)
I have put 100s of thousands of items in to a folder before, and I would offer the following advice, for whatever it is worth: - Use BTreeFolder2. If you are okay with sized batches of objects using the default sort, this is going to help, if you code your application to use this functionality. If you need sorted batches of objects, use the catalog to get these listings instead. - ObjectManager methods like _setObject and _getObject using Zope security machinery may slow things down a bit in a folder with lots of items (at least it did in the old BTreeFolder, this may no longer be the case in BTreeFolder2; the workaround for the old BTreeFolder product, which had to load the entire list of objects as a dictionary, was to use _setOb() and _getOb(), bypassing the security machinery). - Backup and scalability concerns might be helped if you choose to use DirectoryStorage and the right filesystem: http://dirstorage.sf.net. Sean -----Original Message----- From: Paul Winkler [mailto:pw_lists@slinkp.com] Sent: Monday, January 06, 2003 9:57 AM To: zope@zope.org Subject: Re: [Zope] Thousands of Objects - how do they manifest? (newbie) On Mon, Jan 06, 2003 at 03:04:39PM -0000, Crosbie Fitch wrote:
1) Are all objects in Zope persistent? ZClass instances? Python objects?
No, usually, and not necessarily. Objects are persistent if their classes inherit (directly or indirectly) from Persistent, and if the object is added to another object which is an ObjectManager, e.g. a Folder instance. Otherwise they are not persistent. In order to meet these requirements, the object should be constructed either as a ZClass or as a python Product. YOu can define and instantiate classes in an External Method, but making them persistent would be Wrong. The reason I said ZClass instances are "usually" persistent is that you might add one to e.g. a TemporaryFolder whose sub-objects are never written to disk. But in most cases, you add instances to a Folder and they are persistent.
2) Are all object instances visible as entries in a Zope UI folder?
Yes.
Is it possible to have persistent objects without them being visible?
not with a standard Folder.
Or does it not matter that the UI will attempt to list them all (millions or more)?
It would be very hard to use. :) Have a look at Shane's BTreeFolder2, which is designed to solve this kind of problem. http://hathaway.freezope.org/Software/BTreeFolder2
3) If I am expecting a very large number of objects would I always be better off using an external database? or are there circumstances where ZODB is quite able to look after them?
don't know, sorry. i've never pushed a zope folder beyond 100 or so items. My biggest ZODB at the moment contains 150727 items for a total of 2.4 GB, according to the control panel.
4) If I need to be assured of being able to backup/restore the database and/or rebuild the site in the event of catastrophe does an external database provide any greater facility/reliability for achieving this?
depends on the database I guess. You can back up zope just by copying the ZODB, but that gets interesting when it's very large. I've heard on this list that copying a live ZODB might be problematic if zope writes to it while you're backing it up. I don't know if it leads to simply missing some updates in the copy, or worse corruption. So I'm stuck with some downtime while making the backup copy, which is not insignificant with 2.4 GB of data. One strategy I'm looking at to mitigate this would be: 1) restart zope in read-only mode. 2) make a copy of the zodb. 3) restart zope in normal read-write mode. So I'd have only two very brief outages instead of one long one.
Or is ZEO the thing to use?
ZEO does not address backup issues at all. It just allows you to run multiple Zope servers from one ZODB. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's PERVERT EPSILON! (courtesy of isometric.spaceninja.com) _______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Thanks for the surprisingly prompt and helpful advice everyone. :-) The primary question I had in my mind was whether persistent objects were necessarily manifest in a folder. If at least hundreds of users might create umpteen new objects/records each day, I was wondering if there might be two kinds of persistent object, i.e. visible and invisible (and whether there would be performance issues if not). I think this is pretty well answered now. BTreeFolder2 sounds like a reasonable solution to the folder containment issue. However, given that creation of new objects/records (bids) is likely to be quite frequent and simultaneous by a wide variety of users, I'm beginning to think that perhaps I'd be better off using an external database. (NB we're talking around 100K bids per item) An external database is also likely to help separate out reliability and scalability issues of the eCommerce data separately from taht of the web site. Now down to the external database. I've gathered so far that probably the best database to start off with would be MySQL for Python (with ZMySQLDA). I expect this can be swapped out with another SQL database if MySQL starts creaking under the load.
Am Montag, 6. Januar 2003 19:51 erhalten:
Now down to the external database. I've gathered so far that probably the best database to start off with would be MySQL for Python (with ZMySQLDA). I expect this can be swapped out with another SQL database if MySQL starts creaking under the load.
i use a self built object-interface from zope (python) to mySQL about performance you have to test, if it can fullfill your needs ... (time consuming is not so much searching big data, but showing big results of searches = constructing many objects) you can find it at sourceforge.net (O2DB - interface for mapping objects to relational DB) regards fritz (-:fs)
From: fritz i use a self built object-interface from zope (python) to mySQL ... you can find it at sourceforge.net (O2DB - interface for mapping objects to relational DB)
Thanks. Sounds like it could save me a bit of work. I'll have a look! :-)
On Mon, 6 Jan 2003 18:51:47 -0000 "Crosbie Fitch" <crosbie@cyberspaceengineers.org> wrote:
Now down to the external database. I've gathered so far that probably the best database to start off with would be MySQL for Python (with ZMySQLDA). I expect this can be swapped out with another SQL database if MySQL starts creaking under the load.
i use a self built object-interface from zope (python) to mySQL about performance you have to test if it fullfills your needs ... (time consuming is not so much searchin big data, but showing big results of searches = constructing many objects) you can find it at sourceforge.net (O2DB - interface for mapping objects to relational DB) regards fritz (-:fs)
Crosbie Fitch wrote:
Now down to the external database. I've gathered so far that probably the best database to start off with would be MySQL for Python (with ZMySQLDA).
I'd reconsider. I won't contribute to any database flamewars (but this is not slashdot anyway ;)), because I lack the direct knowledge of mysql, but a short search on google for - postgres mysql concurrent - brings up a lot of links which indicate that postgres would be the way to go, see for instance the often cited http://www.phpbuilder.com/columns/tim20001112.php3?print_mode=1 Performance under concurrent usage, mature implementation for transactions since ages and an outstanding locking mechanism (MVCC) seem to be the things which your application could profit by from the beginning. But use what you're familiar with. cheers, oliver
participants (5)
-
Crosbie Fitch -
fritz -
Oliver Bleutgen -
sean.upton＠uniontrib.com -
zope