[Michel Pelletier, on Fri, 10 Mar 2000] :: What is needed is a Folder object based on a more efficient storage :: mechanism, like BTree. Which brings us full circle to previous discussions about the using the reiserfs -- not only BTree, but journalled. :>)

[Pavlos Christoforou, on Fri, 10 Mar 2000] :: Hi Patrick - :: :: What happens if one manually deletes a file that is part of a transaction :: involving many objects/files? How would transaction boundaries be enforced :: on a filesystem? I am interested in hearing your thoughts on the issue :: since it seems you have already done some work on it. I have done nothing on this, unless you count daydreaming. ;>) At one point, I had idly asked if anyone thought this might work, expecting that some knowledgeable soul would immediately shoot the idea down. Surprisingly, both Jim Fulton and Hans Reiser seemed to see that it was in the realm of possibility, but not easy. Of course, you have identified the hard part. Back in December, Jim Fulton had the following reaction (below). Afterward, someone else at DC told me that when Jim says it "getting transactional semantics right might be a tad tricky" he means your head will explode. :) Still ... since Linus Thorvalds has refused to add the complexity of journalling to the Linux buffer cache, the people hacking on reiserfs and other journalling filesystems, have been forced to implement their own mechanisms for keeping track of dirty data. So one wonders if these might be adaptable in some fashion, much as they are used to support high availability network filesystems like Intermezzo, which exploits an existing filesystem as a cache and wraps around it. So, in answer to your question, I have no idea. ;) =========== [Jim Fulton] Would you imagine using the reiserfs as a ZODB storage? If one was going to use a file system as an object store, then the reiserfs would be attractive due to it's effective support for small files. Is this what you had in mind? The ZODB has a well-defined "storage interface" that should make implementation of a reiserfs- (or just an fs-) based storage reasonably straightforward, although getting transactional sematics right might be a tad tricky. Jim :: :: :: _______________________________________________ :: Zope maillist - Zope@zope.org :: http://lists.zope.org/mailman/listinfo/zope :: ** No cross posts or HTML encoding! ** :: (Related lists - :: http://lists.zope.org/mailman/listinfo/zope-announce :: http://lists.zope.org/mailman/listinfo/zope-dev )

From: "Michel Pelletier" <michel@digicool.com>

alan runyan wrote:

...

is it me or is the Members list take FOREVER to navigate through? and

why

...

would it take so long (more than minute) to go from first page of members to the next page? it wasn't like this a few months ago. is it the load on the server?(I tried last night @4AM to the same results) is it the way its fetching and ordering members from the catalog (this definetly shouldnt be the problem - its only 6000 people, w/ listed boolean and alphabetizing results).

It doesn't use the Catalog. It's a plain Zope folder. This means that the folder object, including it's dictionary of childer, is loaded into memory every time it is accessed. This is the inefficient part. Folders were never meant to contain this many objects. This is not neccesarrily a lack, Linux directories are not that efficient either. Try putting 6000 files in one ext2 directory and then type 'ls'. Pack a lunch.

Actually, the Members roster is a bit more efficient. It uses caching to store the results. This was added only recently, before the roster was offline for a while because it was taking too much resources. What happens is this: If the cache has expired, or is empty, we do go through all 6000 Member folders, and check wether or not they are listed. From listed Folders (just over a 1000 now), we store the id and title in a list. This list is sorted on id. Next we store the total number of Members found. Then, we loop over the 6000 Member objects, and check wether or not they have been active in the past two weeks. The number of active accounts is also stored. Together with a timestamp, this info is then stored as a volatile dictionary attribute, and returned. A next call to Members will return this cached information. Timeout of the cache currently is 60 minutes. So, the batched pages and the searches use this cached information. There are a few reasons why it still may be slow: 1/ The cache has expired or is empty: we have to fill it again. 2/ The cache is per-thread. It isn't persistent, and therefor not shared between threads. Your request could hit on thread first, your next request a different thread, and then have to wait for the cache to fill twice. 3/ The volatile attributes don't hang around as long as I expect them to. Maybe the object cache is a bit too aggressive, I haven't figured this out yet. I haven't been able to get the cache to stay around for more than 20 minutes, but it is hard to verify and debug with 7 threads. I am not sure about this one, because in the course of some extra testing I got to see cache older than 20 minutes again. 4/ The dictionary containing the cache may be coming from swapped memory. Currently, the Zope.org process grows quite large. This may have to do with the up to 42000 Folder objects loaded into memory at any time. The Members Folder holds references to 6000 Member Folders, and there can be up to 7 of them (7 threads). This is an area where a BTree based folder would _really_ help. 5/ Plain network lag. Check other pages on Zope.org. 6/ CPU is hogged. The rendering of the ZQR is very expensive. If it runs, the whole of Zope.org suffers. I am seriously considering disabeling it. People really onlly need the pre-rendered version. I was testing just now, had to wait way too long, and noticed someone was looking at both the normal ZQR (rendering from XML), and the print preview version. The Members roster came from cache. I CCed David Kankiewicz on this, so he knows =) Martijn Pieters | Software Engineer mailto:mj@digicool.com | Digital Creations http://www.digicool.com/ | Creators of Zope http://www.zope.org/ | The Open Source Web Application Server ---------------------------------------------

From: "Michel Pelletier" <michel@digicool.com>

Martijn Pieters wrote:

...

Actually, the Members roster is a bit more efficient. It uses caching to store the results. This was added only recently, before the roster was offline for a while because it was taking too much resources.

folders, and check wether or not they are listed. From listed Folders (just over a 1000 now), we store the id and title in a list. This list is sorted

1000 is still too many to be efficient in a regular folder. You're right that a BTree is probably the way to go here. I didn't know about the caching efforts, it seems like you went through some hoops to come up with a better solution.

Note that the 1000 items is a sorted list of (id, title) tuples, _not_ Folder object references. Once the cache is filled, we only have to display the precooked numbers, take one _.len() of the listed members list for display, and have dtml-in do it's magic on the afore-mentioned list. The problem lies mostly in the fact that we have to access each and every Member Folder to check if it is listed. Even if we use a BTree, we'd have to load every Folder into memory. The same for the User objects in acl_user. Martijn Pieters | Software Engineer mailto:mj@digicool.com | Digital Creations http://www.digicool.com/ | Creators of Zope http://www.zope.org/ | The Open Source Web Application Server ---------------------------------------------

On Sat, 11 Mar 2000, Martijn Pieters wrote:

4/ The dictionary containing the cache may be coming from swapped memory. Currently, the Zope.org process grows quite large. This may have to do with the up to 42000 Folder objects loaded into memory at any time. The Members

For my own information, could you tell me how big big is? I'm running a database of 60K objects, and when there is a lot of activity on the database, the process grows to about 150M. At around 180M, it locked up and I had to restart Zope. I think my server ran out of available virtual memory. (I'm moving the database to a dedicated server Real Soon Now). --RDM