[ZODB-Dev] Memory Errors reading large ZODB
Jim Carroll
jimpy at oublic.org
Wed Aug 1 13:57:18 EDT 2007
Hi, I'm trying to iterate over all the comments in my discussion .fs
and I'm getting different errors each time. The latest is Memory Error (direct
fs access), sometimes it's trouble starting a thread (access through ZEO.)
I'm running ZODB 3.3 on a FreeBSD system.
I've tried doing periodic commit() and cacheGC(), but the memory use just keeps
growing until things blow up.
My question is: How can I get my zodb client to forget objects very shortly
after I read them? (I'm just counting objects, and want to get them out of
memory right away.)
I don't think I can del object because that would also remove it from the
database... isn't that correct?
my main loop is below...
Thanks,
-Jim
for groupid, group in db.group_db.root.items():
group_short_name = group.get_user_id()
# touch each discussion once
for item in group.get_blog().get_items():
for comment in item.get_all_comments():
author_id = comment.author.get_user_id()
counts[author_id] = counts.get(author_id, 0) + 1
transaction_commit()
db.connection.cacheGC()
print "counted in ", groupid
#
More information about the ZODB-Dev
mailing list