Hello,
We've had requests from several of our users for the ability to have a drop in page counter within zope. However creating a page counter python script which increments some value in zope will bloat the ZODB.
Solutions exist where values are stored on the file system or in a database. Unfortunately our users don't have file system access and it is unacceptable to expect them to request a database account and setup database connections and methods just to create a page counter.
I would like to create a Page Counter product that doesn't bloat. If a product is created that doesn't subclass History or UndoSupport does it still bloat?
Zope is transactional, but products like ZLDAPConnection have the ability to be "non-transactional" what does this mean? Could I use this in my counter?
-Brian
I would like to create a Page Counter product that doesn't bloat. If a product is created that doesn't subclass History or UndoSupport does it still bloat?
those have nothing to do with the fact that every time that hit counter fires some object will get updated and thus saved again.
Zope is transactional, but products like ZLDAPConnection have the ability to be "non-transactional" what does this mean? Could I use this in my counter?
the meaning of transactional as employed by the ZLDAPConnection has nothing to do with your concern about ZODB bloat. the ZLDAPConnection tried (it never got successfully finished) to hook into the ZODB transaction machinery to make sure writes to LDAP only happen if the whole transaction was indeed successful.
i personally don't see a way to completely avoid ZODB bloat with a counter product stored in the ZODB itself. IMHO it is inherently bad to use ZODB-based hit counters in general for that very reason.
jens
On Tue, 2002-11-05 at 23:02, Jens Vagelpohl wrote:
i personally don't see a way to completely avoid ZODB bloat with a counter product stored in the ZODB itself. IMHO it is inherently bad to use ZODB-based hit counters in general for that very reason.
Just an idea from the 'could-be-done-if-needed-department'. Generally there is quite often the need to have non-undoable properties in different objects of a site. There is one way to store it in the temporary storage, which is ram based, and not shared between ZEO instances. But if a second non-undoable storage is mounted it could be stored there and could be shared between ZEOC.
To make this more transparent, one could perhaps implement something like a special propertysheetobject, which stores its properties in this mounted storage.
Is there anything obviously wrong with this idea?
__Janko
On Wednesday 06 November 2002 8:33 am, Janko Hauser wrote:
Just an idea from the 'could-be-done-if-needed-department'. Generally there is quite often the need to have non-undoable properties in different objects of a site. There is one way to store it in the temporary storage, which is ram based, and not shared between ZEO instances. But if a second non-undoable storage is mounted it could be stored there and could be shared between ZEOC.
To make this more transparent, one could perhaps implement something like a special propertysheetobject, which stores its properties in this mounted storage.
Is there anything obviously wrong with this idea?
the problem is that cross-storage references are tricky.
Mixing undoable and non-undoable *objects* in the same storage is on the to-do list for DirectoryStorage, http://dirstorage.sourceforge.net
A solution that I've employed in the past is to keep the counters in RAM (perhaps in a dict stored as a module global keyed by object path) and write the values that have changed persistently every 5-10 minutes.
The drawback to this is that you can loose some counts if the server goes down, plus the count won't be accurate across ZEO app servers sharing the same storage (but they'll be close). I generally deal with this by reading the commited values rather than the volatile ones.
-Casey
On Tuesday 05 November 2002 03:55 pm, Brian R Brinegar wrote:
Hello,
We've had requests from several of our users for the ability to have a drop in page counter within zope. However creating a page counter python script which increments some value in zope will bloat the ZODB.
Solutions exist where values are stored on the file system or in a database. Unfortunately our users don't have file system access and it is unacceptable to expect them to request a database account and setup database connections and methods just to create a page counter.
I would like to create a Page Counter product that doesn't bloat. If a product is created that doesn't subclass History or UndoSupport does it still bloat?
Zope is transactional, but products like ZLDAPConnection have the ability to be "non-transactional" what does this mean? Could I use this in my counter?
-Brian
Brian R Brinegar writes:
We've had requests from several of our users for the ability to have a drop in page counter within zope. However creating a page counter python script which increments some value in zope will bloat the ZODB.
Solutions exist where values are stored on the file system or in a database. Unfortunately our users don't have file system access and it is unacceptable to expect them to request a database account and setup database connections and methods just to create a page counter.
I would like to create a Page Counter product that doesn't bloat. If a product is created that doesn't subclass History or UndoSupport does it still bloat?
It does.
The bloat is a consequence of how "FileStorage" works (I appends changes at the end of the file and does never override something).
You need a different storage (a non-undoable one) to prevent this behaviour, e.g. BerkeleyStorage or OracleStorage.
When your load is light and your counter consists justs of an integer, then your bloat is only several dozens of byte and you may be able to live with it....
Dieter
On Friday 08 November 2002 01:55 pm, Dieter Maurer wrote:
Brian R Brinegar writes:
We've had requests from several of our users for the ability to have a drop in page counter within zope. However creating a page counter python script which increments some value in zope will bloat the ZODB.
Solutions exist where values are stored on the file system or in a database. Unfortunately our users don't have file system access and it is unacceptable to expect them to request a database account and setup database connections and methods just to create a page counter.
I would like to create a Page Counter product that doesn't bloat. If a product is created that doesn't subclass History or UndoSupport does it still bloat?
It does.
The bloat is a consequence of how "FileStorage" works (I appends changes at the end of the file and does never override something).
You need a different storage (a non-undoable one) to prevent this behaviour, e.g. BerkeleyStorage or OracleStorage.
is there a useable undoable storage?
When your load is light and your counter consists justs of an integer, then your bloat is only several dozens of byte and you may be able to live with it....
if you can live with that, then you should try to use a subclass of lib/python/BTrees/Length.py
more efficient pickling state and app conflict resolution.
stepping up beyond that while remaining in the zodb means using something like the strategy casey suggested of updating in memory variables and committing them periodically (and you definitely want conflict resolution for something like that on your persistent ob).
-k
"kt" == kapil thangavelu kvthan@wm.edu writes:
>> e.g. BerkeleyStorage or OracleStorage.
kt> is there a useable undoable storage?
I think you meant "useable non-undoable storage", right?
If so, then I believe that ZODB 3.2 will have a such a useable storage. We've had some recent discussions with Sleepycat about some of the worst problems with the BerkeleyDB storages, and I think we now have some good strategies for greatly improving their utility.
In the ZODB cvs tree there are now two completely rewritten implementations of the Full and Minimal storages. Full storage, as its name implies supports undo and versions. It reference counts objects and does mark-and-sweep garbage collection, but you need to pack in order to get rid of old object revisions or unreferenced objects.
An interesting byproduct of the rewrite is that Full now supports something called "autopacking" -- essentially a background thread that can be configured to pack the database, say as of 4 hours ago, once per hour (these parameters are of course configurable). Autopacking usually only eliminates old object revisions (with reference counting), but it can be configured to occasionally do a "classic pack", which would also do a mark-and-sweep of the entire object space to get rid of unreferenced cycles. For example, you could tell autopack to do one of those once per day.
Minimal storage does not support undo or versions because it keeps only the most current object revision. The current implementation of Minimal isn't quite complete because while it performs reference counting garbage collection, it doesn't yet do mark-and-sweep to collect unreferenced object cycles. I'll be adding this probably next week, with pack and autopack interfaces.
One downside to the rewrite is that the table schemas aren't compatible with the older versions of these storages, so if you have data in an existing Berkeley storage, you'll need to use something like copyTransactionsFrom() to move them to the new storages.
I plan on making a beta release of the new storages sometime early next week. They will be included in the ZODB 3.2 release, which will like get alpha'd soon, with hopefully a final release by the end of the year. If you want to take an early look, check out the ZODB3 cvs tree. What's there passes all the tests.
Cheers, -Barry
-
barry＠zope.com -
Brian R Brinegar -
Casey Duncan -
Dieter Maurer -
Janko Hauser -
Jens Vagelpohl -
kapil thangavelu -
Toby Dickenson