[Zope] zope.org Members list

[Martijn Pieters]

From: "Michel Pelletier" <michel@digicool.com>
> alan runyan wrote:
> >
> > is it me or is the Members list take FOREVER to navigate through?  and
why
> > would it take so long (more than minute) to go from first page of
members to
> > the next page?  it wasn't like this a few months ago.  is it the load on
the
> > server?(I tried last night @4AM to the same results)  is it the way its
> > fetching and ordering members from the catalog (this definetly shouldnt
be
> > the problem - its only 6000 people, w/ listed boolean and alphabetizing
> > results).
>
> It doesn't use the Catalog.  It's a plain Zope folder.  This means that
> the folder object, including it's dictionary of childer, is loaded into
> memory every time it is accessed.  This is the inefficient part.
> Folders were never meant to contain this many objects.  This is not
> neccesarrily a lack, Linux directories are not that efficient either.
> Try putting 6000 files in one ext2 directory and then type 'ls'.  Pack a
> lunch.

Actually, the Members roster is a bit more efficient. It uses caching to
store the results. This was added only recently, before the roster was
offline for a while because it was taking too much resources.

What happens is this:

If the cache has expired, or is empty, we do go through all 6000 Member
folders, and check wether or not they are listed. From listed Folders (just
over a 1000 now), we store the id and title in a list. This list is sorted
on id. Next we store the total number of Members found.

Then, we loop over the 6000 Member objects, and check wether or not they
have been active in the past two weeks. The number of active accounts is
also stored.

Together with a timestamp, this info is then stored as a volatile dictionary
attribute, and returned.

A next call to Members will return this cached information. Timeout of the
cache currently is 60 minutes. So, the batched pages and the searches use
this cached information.

There are a few reasons why it still may be slow:

1/ The cache has expired or is empty: we have to fill it again.

2/ The cache is per-thread. It isn't persistent, and therefor not shared
between threads. Your request could hit on thread first, your next request a
different thread, and then have to wait for the cache to fill twice.

3/ The volatile attributes don't hang around as long as I expect them to.
Maybe the object cache is a bit too aggressive, I haven't figured this out
yet. I haven't been able to get the cache to stay around for more than 20
minutes, but it is hard to verify and debug with 7 threads. I am not sure
about this one, because in the course of some extra testing I got to see
cache older than 20 minutes again.

4/ The dictionary containing the cache may be coming from swapped memory.
Currently, the Zope.org process grows quite large. This may have to do with
the up to 42000 Folder objects loaded into memory at any time. The Members
Folder holds references to 6000 Member Folders, and there can be up to 7 of
them (7 threads). This is an area where a BTree based folder would _really_
help.

5/ Plain network lag. Check other pages on Zope.org.

6/ CPU is hogged. The rendering of the ZQR is very expensive. If it runs,
the whole of Zope.org suffers. I am seriously considering disabeling it.
People really onlly need the pre-rendered version. I was testing just now,
had to wait way too long, and noticed someone was looking at both the normal
ZQR (rendering from XML), and the print preview version. The Members roster
came from cache.

I CCed David Kankiewicz on this, so he knows =)

Martijn Pieters
| Software Engineer    mailto:mj@digicool.com
| Digital Creations  http://www.digicool.com/
| Creators of Zope       http://www.zope.org/
|   The Open Source Web Application Server
---------------------------------------------