[Andy Yates]
... For example if I have ab make X requests, X new sessions are created and memory use goes up by 100MB. I also see about X objects in the transient object container. Now I wait for the sessions to expire based on the "Data object timeout value". Then I make a few calls to tickle the Transient system's garbage collection. The objects in the toc goes down but memory use does not. Some have posted that python does not release memory it just reuses it. Well, when I make X more requests the memory is not reused. Now it is using 200MB.
I know as much about the gory details of Python's memory-management as anyone on Earth, so you can believe this: if anyone gives you a catchy one-line summary, don't believe them. Python has many distinct memory-management subsystems, and details are crucial. In addition, Python builds on top of the platform C library's malloc/free/realloc system, and that builds in turn on the platform operating system policies. If you pursue it, you'll discover that it's a long discussion just trying to figure out what "release memory" could possibly mean. That said, it's not generally expected that the virtual memory highwater mark (which I'm guessing you're really talking about when you say "now it is using 200MB") will grow rapidly and consistently.
... Could it be possible that the BTrees code which is written in C is leaking?
Yes -- and if you use an old enough version of ZODB, it's guaranteed to leak <wink>.
Memory leaked here would not show up as refcounts in Zope right?
For the most part, it should: BTrees are built out of Python objects, each with its own refcount (as is true of all Python objects). To Zope, BTree building blocks look like all other kinds of persistent Python objects. I haven't done this for a while, but when leaks in the BTree code were still suspected (they truly aren't any more), I used to run the ZODB test suite under a debug-build ZODB and a debug-build Python, for days at a time in a loop. There are special gimmicks you can use in a debug-build Python to detect leaks. This did find leaks, too. I stopped doing this after the last leak it pointed at took days to track down, and turned out to be in a BTree endcase soooooo rare that it may never have occurred in real life (but did occur once in one of the test cases for "pathological" BTrees). That doesn't mean there can't be more leaks, but it would be pretty amazing if there were still a leak on any common path thru the BTree code.
It is FULL of malloc and free calls. Isn't this a fairly new addition to Zope?
No. The BTree code is really part of ZODB, and has been there (as far as I know) forever; Jeremy Hylton and I did a lot concentrated work to squash memory leaks in ZODB (including its BTree code) a couple years ago. The BTree code has been very stable (== has been changed very little) since then.