We have been looking at caching in Zope as a way of tweaking performance. Heres an example of what I think happens: - Supposing I have a 1,000 object catalog. If one person changes an catalog aware object, that instance of the catalog will be pulled out of the ZODB and changed. It will then be written to the ZODB. That last copy will stay in the cache for as long as the "Cache Parameters" are set to allow. - If somebody changes another catalog aware object, that will repeat the above process. - However simply accessing the catalog (no changes) will pull the object from the cache. - What would be really nice is if the object only got written to the cache when it is no longer used, that way every time the catalog changed it didnt write to ZODB instead it changed the cached version. Of course that does bring up the recovering from disaster problem. So do I understand it correctly? -- Andy McKay, Developer. ActiveState.
Statement of possible BS: I am speaking from how I understand the catalog to work, and possibly not how it actually *does* work. :-) Andy McKay wrote:
We have been looking at caching in Zope as a way of tweaking performance. Heres an example of what I think happens:
- Supposing I have a 1,000 object catalog. If one person changes an catalog aware object, that instance of the catalog will be pulled out of the ZODB and changed. It will then be written to the ZODB.
Actually, a subobject of the catalog which is stored in a BTree will change, and the Catalog instance will change, but it will not write the entire contents of the catalog back out to the ZODB. The current BTree implementation has some problems, so other unrelated objects in it may be changed on the write. This will be solved by updates to the catalog which use a new "broadtree" BTree implementation.
That last copy will stay in the cache for as long as the "Cache Parameters" are set to allow.
The changed subobjects will, yes. Unless they're manually ghosted for some other reason (rarely happens, if ever).
- If somebody changes another catalog aware object, that will repeat the above process.
Yes.
- However simply accessing the catalog (no changes) will pull the object from the cache.
Yes.
- What would be really nice is if the object only got written to the cache when it is no longer used, that way every time the catalog changed it didnt write to ZODB instead it changed the cached version. Of course that does bring up the recovering from disaster problem.
This would break transactional integrity. It's important that the catalog is in a self-consistent state as well as be consistent with other objects in the ZODB, and transactions help to maintain this. Is it essential that all your cataloged objects be catalog-aware? -- Chris McDonough Digital Creations, Publishers of Zope http://www.zope.org
Chris McDonough wrote:
changed on the write. This will be solved by updates to the catalog which use a new "broadtree" BTree implementation.
Any idea when this will land? We'ev had to give up on the Catalog for the mailing list archives and go with MySQL's full text indexing :-( cheers, Chris
No, sorry... hopefully within the next month. ----- Original Message ----- From: "Chris Withers" <chrisw@nipltd.com> To: "Chris McDonough" <chrism@digicool.com> Cc: "Andy McKay" <andym@ActiveState.com>; <zope-dev@zope.org> Sent: Monday, September 25, 2000 6:08 AM Subject: Broadtree Catalog?
Chris McDonough wrote:
changed on the write. This will be solved by updates to the catalog which use a new "broadtree" BTree implementation.
Any idea when this will land?
We'ev had to give up on the Catalog for the mailing list archives and go with MySQL's full text indexing :-(
cheers,
Chris
Andy McKay wrote:
We have been looking at caching in Zope as a way of tweaking performance. Heres an example of what I think happens:
- Supposing I have a 1,000 object catalog. If one person changes an catalog aware object, that instance of the catalog will be pulled out of the ZODB and changed. It will then be written to the ZODB. That last copy will stay in the cache for as long as the "Cache Parameters" are set to allow.
- If somebody changes another catalog aware object, that will repeat the above process.
- However simply accessing the catalog (no changes) will pull the object from the cache.
- What would be really nice is if the object only got written to the cache when it is no longer used, that way every time the catalog changed it didnt write to ZODB instead it changed the cached version. Of course that does bring up the recovering from disaster problem.
So do I understand it correctly?
Not exactly. Have you seen the ZODB paper? http://www.zope.org/Members/jim/Info/IPC8/ZODB3 You can think of the normal ZODB cache as being a tool that manages the objects in memory. Whenever an object has been accessed, it is in the cache. The cache, together with the database are responsible for moving objects, and their state in and out of memory. An object can be in serveral in memory states. Two that are of interest, from a caching perspective, are the "ghost" state and the up-to-date state. The up-to-date state is a state in which the object and it's state are loaded. In the ghost state, the object is in memory, but it has now state. The cache manager tries to release an object's state and remove the object from memory when it is not used. An object's state is released when it hasn't been accessed in a long time. It can be removed when no other objects (not counting the cache itself) refer to it. Note that the cache parameters are only guides for an algorithm that is somewhat adaptive and a bit more complicated than the parameters suggest. For example, the target cache size is a target, not a limit. The cache sizes are usually mich higher than the target. Another thing to be aware of is that large objects are generally designed to spread their state over many subobjects so that they can make effective use of the cache. A catalog may have hundreds of sub-objects, only a small fraction of which are kept in memory at a time. Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
participants (4)
-
Andy McKay -
Chris McDonough -
Chris Withers -
Jim Fulton