May 2000 - Zope-Dev - Zope lists

Re: [Zope-Dev] Zope 2.1.6 Memory issue
by Jason Spisak 22 May '00

22 May '00

Shane Hathaway writes: > Jason Spisak wrote: > > > > Shane Hathaway writes: > > > > > Jason Spisak wrote: > > > > > You must have a large site. > > > > > > > > 300,000 + objects > > > > > > Whoa... ;-) > > > > > > Tell me, what are you doing with 300K objects? > > > > > > Shane > > > > Technical recruiting. > > I am the ultimate test case for ZDOB scalability. ;) > > > > I just recently started having problems thoug, so I am going back and > > disabling the small fixes I have made recently in my code. > > > > I just restarted Zope again. > > > > Most of the objects just sit around. > > > > Wow. There it goes again. It's got to be DateTime.DateTime.DateTime. It > > just shot to 118003. > > That's actually a natural thing to happen. DateTime objects get > attached to many kinds of objects. The real problem is that so many > objects are being loaded into memory. > > I just did some research and it looks as though garbage collection is > not fully implemented in Python. I was not aware of this; do you know > whether it has been corrected? No, I had no idea. > > After minimizing your in-memory cache, look at the manage_debug screen > again. Are the refcounts still high? I bet they are... > No. The ref counts drop to hundreds. Also, is see reference to 161 Products.GadflyDA.gadfly.kjparser.ParseRule, and I'm not even useing Gadfly! What the heck is that about? Anyone? Thing's seem to be a bit nicer now that I took out an External Method that returns a DateTime object the Catalog for Indexing. However, there's only about 14,000 of objects with that property in existsnce in my DB, and they don't change often. Here's the code: def calendar_day(self) return int(self.calendar_date)/1000000 Used to return the unique integer day. All my best, Jason Spisak CIO HireTechs.com 6151 West Century Boulevard Suite 900 Los Angeles, CA 90045 P. 310.665.3444 F. 310.665.3544 Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

4 5

Re: Memory 2.1.6
by Jason Spisak 22 May '00

22 May '00

Shane, Well I lied. The memory is now fluctuating. It went as high as 12MB available (out of a gig) and down to 2MB. This seems fine, it's just that I'm scraping the bottom of the barrel. Also, I really am curious why flushing the cache doesn't reclaim memory, too. All my best, Jason Spisak CIO HireTechs.com 6151 West Century Boulevard Suite 900 Los Angeles, CA 90045 P. 310.665.3444 F. 310.665.3544 Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

1 0

Re: [Zope-dev] Please HELP: Nested ZClass, Nested instance
by Karl Anderson 22 May '00

22 May '00

"Park, Karl" <Karl.Park(a)korea.bowneglobal.com> writes: > I created a ZClass in the Product folder. And anther ZClass under the > ZClass(thus nested class) > To do it, I created a DTML Method which accepts user's input and then, if > needed, it have to create an object. > First part is to create the parent object, which works just fine. > But the sencond part to create the child object doesn't seem to work. > If I take the easy way, that is, creating othere ZClass on the top of the > products folder of Control Panel, I can do it. > But I don't think it is a good desing from the perspective of OOP.... I had some hassles with this recently. There's a HOWTO at: http://www.zope.org/Members/gtk/containers Can you create the child zclass from the manage screen inside the parent zclass? Does the parent zclass inherit from ObjectManager before any other superclasses? -- Karl Anderson karl(a)digicool.com

1 0

Re: Zope 2.1.6 Memory issue
by Jason Spisak 22 May '00

22 May '00

David Kankiewicz writes: > Jason Spisak wrote: > > > > After being up for about a half-hour my objects reading are: > > > > Total number of objects in the database 300280 > > Total number of objects in all of the caches combined 150995 > > > > Is this typical? Half? I doubt it. My target size is about 20,000. > > > > All my best, > > > Not saying I know anything, but how many threads are you running? is > it 8? > Unfortunately, it is 10. We almost had a winner. Thanks for playing the Zope memory Lotto. ;) Jason Spisak CIO HireTechs.com 6151 West Century Boulevard Suite 900 Los Angeles, CA 90045 P. 310.665.3444 F. 310.665.3544 Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

1 0

XMLImporter integration issue
by Chimezie Thomas-Ogbuji 22 May '00

22 May '00

posted a new issue regarding XMLImporter integration http://www.zope.org/Wikis/zope-xml/XMLImporter%20mapping%20dictionary -- Chimezie Thomas-Ogbuji Junior Consultant Fourthought Inc. (303) 583 9900 ext 104 cogbuji(a)fourthought.com

1 0

Re: [Zope-dev] Please HELP: Nested ZClass, Nested instance
by Rik Hoekstra 22 May '00

22 May '00

<snip> >First part is to create the parent object, which works just fine. >But the sencond part to create the child object doesn't seem to work. >If I take the easy way, that is, creating othere ZClass on the top of the >products folder of Control Panel, I can do it. >But I don't think it is a good desing from the perspective of OOP.... > Nope, but it won't work right now (there were some threads on this on the main Zope list recently). If you want my advice: do not nest ZClasses, or you'll find yourself redesigning your product sooner or later :-( >Could any one of you have solution to this, so that I can keep the Class >Hierarchy as above and create the object nested? No, make a flattened design. My 2 cents (eurocents that is) Rik

1 0

Re: [Zope-Dev] Zope 2.1.6 Memory issue
by Jason Spisak 22 May '00

22 May '00

Shane Hathaway writes: > Jason Spisak wrote: > > > You must have a large site. > > > > 300,000 + objects > > Whoa... ;-) > > Tell me, what are you doing with 300K objects? > > Shane After being up for about a half-hour my objects reading are: Total number of objects in the database 300280 Total number of objects in all of the caches combined 150995 Is this typical? Half? I doubt it. My target size is about 20,000. All my best, Jason Spisak CIO HireTechs.com 6151 West Century Boulevard Suite 900 Los Angeles, CA 90045 P. 310.665.3444 F. 310.665.3544 Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

1 0

Re: [Zope-Dev] Zope 2.1.6 Memory issue
by Jason Spisak 22 May '00

22 May '00

Shane Hathaway writes: > Go to Control Panel -> Database Management -> Flush Cache and click > "minimize". Then go to Control Panel -> Database Management -> Cache > Parameters. After a minimize operation, the "Total number of objects > in all of the caches combined" should be relatively low (< 1000), as > well as your system memory usage. Is that the case? > i did it again, and now I'm down to 160 objects total. That's more liek it. But system memory is still dropping. I was at 173632 available (This is only a half our after the Zope restart BTW. It ate 100MB in under that time.) before the second pass and now I'm at 173616. Jason Spisak CIO HireTechs.com 6151 West Century Boulevard Suite 900 Los Angeles, CA 90045 P. 310.665.3444 F. 310.665.3544 Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

2 2

Re: [Zope-Dev] Zope 2.1.6 Memory issue
by Jason Spisak 22 May '00

22 May '00

Shane Hathaway writes: > Jason Spisak wrote: > > > > Shane Hathaway writes: > > > > I had to restart Zope. I was about to start swapping, and now everyone's a > > lunch. > > > > It's so weird. Even though there is really no one on it that memory is > > just flying out the window. > > > > > Jason Spisak wrote: > > > > > > > > Shane Hathaway writes: > > > > > > > > Okay, here's the output. (BTW is was trying manage_debug without the > > > > Control_Panel. That was not intelligent;) > > > > > > > > Class May 22, 2000 4:21 am May 22, 2000 4:22 am Delta > > > > OIBTree.Bucket 7132 8135 +1003 > > > > IOBTree.BTree 2570 2827 +257 > > > > BTree.BTree 1028 1105 +77 > > > > IIBTree.Bucket 4590 4619 +29 > > > > > > Since you say you are not using any DA's, the culprit must be > > > ZCatalog. What happens when you minimize the memory cache (setting the > > > limit to 3 seconds)? Do the OIBTree.Bucket objects get unloaded? > > > > > > Shane > > > > How would I tell if they get unloaded? I am new to the manage_debug BTW. > > I set the Cache Parameters target time between accesses to 3. What am I > > looking for number-wise? > > I studied your debug screens a little more and realized that the > handling of BTree, etc. is probably in order. The refcount is indeed > decreasing--I think you just happened to hit a spike. > > You must have a large site. 300,000 + objects >I'm guessing that you have a lot of > objects that get cataloged. Yep. That's why I have the machine I do, though. >In order for them to be cataloged, they > have to be loaded into memory then unloaded. > I see. > Go to Control Panel -> Database Management -> Flush Cache and click > "minimize". Then go to Control Panel -> Database Management -> Cache > Parameters. After a minimize operation, the "Total number of objects > in all of the caches combined" should be relatively low (< 1000), as > well as your system memory usage. Is that the case? No. After the 'minimize' I now have 15501 objects in cache combined. And the free command shows less memory than before the minimize command. As Wyle E. Coyote would say...Yipe. :o Jason Spisak CIO HireTechs.com 6151 West Century Boulevard Suite 900 Los Angeles, CA 90045 P. 310.665.3444 F. 310.665.3544 Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

1 0

Zope 2.1.6 Memory issue
by Jason Spisak 22 May '00

22 May '00

Zopists, I am spoiled. I have a large server and until now I have never really monitored memory usage. But I noticed things slowing down a great deal and was curious. Swapping. Machine: RH 6.1 2.2.12-smp Dual Xenon 1 gig memory Zope install: Zope 2.1 Binary ZODB pure. No DA's Home grown ZClasses. 'ps' tells me Zope is the culprit. It's using 64% when things get swappy. There are no prosesses running on this machine BTW. It's just a Zope server. My Data.fs file is ~300 packed and gets to be about 1GB at the end of every day. (A lot of transactions :) Zope is eating memory at the rate of 10MB every 2 minutes. Is that appropriate? I never noticed a slowdown until I switched to 2.1, really. I've considered writing a small python progam to compare the output of 'free' to the Zope log, but I'd love to see if there is another way to nail what's eating mem. If someone can tell me the best way to narrow it down from experience, I'll do it this afternoon. Hopefully, -- Jason Spisak 444(a)hiretechs.com

2 5