[Zope-dev] RE: Memory issue 2.1.1 wrap up (Long)

[Jason Spisak]

Thanks to everyone who gave me advise on what was going on.

I think the problem was that I need some tuning in a variety of areas. 
(One of them being my brain) I am going to write up a little ditty on
what I discovered so that people can check it before they claim leakage
is occuring.

I don't know enough about OS's to really put the *technical*
explanations of some of the things to look for so, Matt, if you'd take a
look at it when I post the url, I'd appreciate it.

Just a quick rehash--

1. I was running on an SMP machine, andthe setcheckinterval in Python
wasn't helping me realize the machine's potential.  This contributed to
the slow down. 
Points:(2 physical, 1 psychological)

2. The 'Cache' tab in Zope was set to give objects a long, leisurely
time in memory.  This was causing Zope to rapidly claim and hold onto a
ton of memory for objects it wan't even using. I adjusted it to 3
seconds. I worked the number until I found a happy medium. (BTW, I
didn't really know *exactly* what this meant, and I hope there's a help
system file on it in 2.2) 
Points:(1 physical, 1 psychological)

3. The 'Cache tab in Zope was set to allow a large amount of objects
(20,000) to be a target size.  This made those large amounts of objects
that were lying around just happy to be there, I guess, I'm stil not
sure what it does exaclty.
Points:(1 physical, 1 psychological)

4. I store ton of information in my ZCatalog.  The way the ZCatalog does
an update (from what Shane told me) is a memory intensive task, loading
lots of objects (I'd love help here), and when you combine that with 2 &
3, you get maxed out pretty quick. Catalog updates were taking a long
time.
Points:(2 physical, 2 psychological)

5. I use the standard Folder object to store 12,000 objects.  Whenever a
new object is added there is a huge hit on performance.  Also, that
object just so happens to get ZCataloged!  That's turns Zope into a dog
for a bit.  [A good solution here is ZPatterns' RackMountable, but this
is a production environment, and I want to let ZPatterns stabilize a
bit.  I am very excited about it though.;)]
Points:(2 physical, 2 psychological)

6. The 'free' command on GNU/Linux shows just how much Linux buffers.  A
ton.  Linux basically took my box from 1 gig of memory down to 1 MB,
everytime.  And that's is where it likes to stay.  That's scary, but not
fatal.  Then the swap increased to about 40MB.  When staying right on
top of those numbers, it looks pretty bad while it's happening, so I
believed things were headed down the drain.  Zope's performance was poor
(due to 1,2,3,4,5), so I thought the end was near. "If I don't restart
Zope, things will only get worse." This is not true, the swap has
hovered at 12MB average, and the free mem is averaging 4MB bobbing like
a rubber duck.
Points:(1 physical, 4 psychological)

7. There is in fact a memory leak in the SQLSession product. ;)
Points:(0 physical, 2 psychological)

All of these together with the fact that I learn more everyday about
Zope and Linux, and you have the prime situation for Memorious Falsis. 
A syndrome where the hacker sees memory leaks that aren't there.

Thanks wholeheartedly,
-- 
Jason Spisak
444@hiretechs.com