Counter and concurrent access
Hello, I try to create a counter of which I am certain that it sends back in every Call a different result. I should so manage concurent acces on this one. The following method is not valid I have to look for a product which could make works: ---- ID = context.nextID context.manage_changeProperties( { 'nextID' : ID+1 } ) ---- But products found (FSCounter and to threadSafeCounter) uses the File System to manage the concurrent access on the counter value. I want to use the method described in the article " Advanced ZODB for python Programmers " ( Http://www.zope.org/Documentation/Articles/ZODB2) by creating a product based on the following code: ------------ class Counter(Persistent): self.count = 0 def hit(self): self.count = self.count + 1 return self.count def _p_resolveConflict(self, oldState, savedState, newState): # Figure out how each state is different: savedDiff= savedState['count'] - oldState['count'] newDiff= newState['count']- oldState['count'] # Apply both sets of changes to old state: return oldState['count'] + savedDiff + newDiff ---------- Is the class "Persistent" assure the value " self.count " to be not identical in two simultaneous calls of the "hit" method (not identical return value)? -- Yann Lugrin yann.lugrin@gmproject.net
Beware of implementing counters within ZODB! Anytime you change a property of a persistent object like a Folder or DTML Document, Zope stores a new version of that object in ZODB. This leads to Data.fs severe bloat!!! Use a relational database or some other Python-only persistence mechanism to store your counter. Best regards, Luciano Yann Lugrin wrote:
Hello,
I try to create a counter of which I am certain that it sends back in every Call a different result. I should so manage concurent acces on this one.
The following method is not valid I have to look for a product which could make works: ---- ID = context.nextID context.manage_changeProperties( { 'nextID' : ID+1 } ) ----
But products found (FSCounter and to threadSafeCounter) uses the File System to manage the concurrent access on the counter value.
I want to use the method described in the article " Advanced ZODB for python Programmers " ( Http://www.zope.org/Documentation/Articles/ZODB2) by creating a product based on the following code:
------------ class Counter(Persistent): self.count = 0
def hit(self): self.count = self.count + 1 return self.count
def _p_resolveConflict(self, oldState, savedState, newState): # Figure out how each state is different: savedDiff= savedState['count'] - oldState['count'] newDiff= newState['count']- oldState['count']
# Apply both sets of changes to old state: return oldState['count'] + savedDiff + newDiff ----------
Is the class "Persistent" assure the value " self.count " to be not identical in two simultaneous calls of the "hit" method (not identical return value)?
-- Yann Lugrin yann.lugrin@gmproject.net
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
From: Luciano Ramalho <luciano@hiper.com.br>
Beware of implementing counters within ZODB! Anytime you change a property of a persistent object like a Folder or DTML Document, Zope stores a new version of that object in ZODB. This leads to Data.fs severe bloat!!! Use a relational database or some other Python-only persistence mechanism to store your counter.
Actually, for example, take a look at ThreadSafeCounter (part of ZopeGUD) - it's an example of how to control persistence to avoid updating a counter object for every tick. Other than that, FSCounter works quite nicely...
Luciano is right, but only if you do it by hanging the counter directly off the piece of content as a property. If instead you have a persistent subobject, it simply becomes a matter of math. For instance, let's say it costs 50 bytes to store counter every time it increments. Let's then say you do 10k page views a day, which would be a site getting more than a million page views a month. You're talking about ZODB growth around 500Kb per day, or 15Mb per month. With *any* amount of packing, this shouldn't be too much of a problem. Even better are the approaches previously mentioned where counting goes in a central datastructure that queues up increments for an interval, then writes them out to some kind of storage. I don't think you _have_ to leave the ZODB to do counters. Just don't do them as a property or attribute directly attached to the thing you're counting. --Paul Luciano Ramalho wrote:
Beware of implementing counters within ZODB! Anytime you change a property of a persistent object like a Folder or DTML Document, Zope stores a new version of that object in ZODB. This leads to Data.fs severe bloat!!! Use a relational database or some other Python-only persistence mechanism to store your counter.
Best regards,
Luciano
Yann Lugrin wrote:
Hello,
I try to create a counter of which I am certain that it sends back in every Call a different result. I should so manage concurent acces on this one.
The following method is not valid I have to look for a product which could make works: ---- ID = context.nextID context.manage_changeProperties( { 'nextID' : ID+1 } ) ----
But products found (FSCounter and to threadSafeCounter) uses the File System to manage the concurrent access on the counter value.
I want to use the method described in the article " Advanced ZODB for python Programmers " ( Http://www.zope.org/Documentation/Articles/ZODB2) by creating a product based on the following code:
------------ class Counter(Persistent): self.count = 0
def hit(self): self.count = self.count + 1 return self.count
def _p_resolveConflict(self, oldState, savedState, newState): # Figure out how each state is different: savedDiff= savedState['count'] - oldState['count'] newDiff= newState['count']- oldState['count']
# Apply both sets of changes to old state: return oldState['count'] + savedDiff + newDiff ----------
Is the class "Persistent" assure the value " self.count " to be not identical in two simultaneous calls of the "hit" method (not identical return value)?
-- Yann Lugrin yann.lugrin@gmproject.net
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
From: Paul Everitt <paul@zope.com>
I don't think you _have_ to leave the ZODB to do counters. Just don't do them as a property or attribute directly attached to the thing you're counting.
Would this be a use for 'precondition' in File/Image?
You certainly _could_ use the precondition as a way to call a counter before each request. More likely you'd just adapt the existing File/Image. As the one behind "precondition" several years ago, I cringe to see it brought up in public. :^) An example of a reasonable idea that, years later, you just hate to be associate with. --Paul marc lindahl wrote:
From: Paul Everitt <paul@zope.com>
I don't think you _have_ to leave the ZODB to do counters. Just don't do them as a property or attribute directly attached to the thing you're counting.
Would this be a use for 'precondition' in File/Image?
From: Paul Everitt <paul@zope.com>
As the one behind "precondition" several years ago, I cringe to see it brought up in public. :^) An example of a reasonable idea that, years later, you just hate to be associate with.
So you're the one! :) Seriously, though... I uncovered this (for myself, anyway) when working on my Audio product, and though the it seemed a good idea... but I'm still wondering for what :) Is it a nifty solution in search of a problem, or has someone found a truly unique use for it? Or should it be, er, demoted?
We wanted to enforce some rules that had to be enforced before someone could get to a File. For instance, you had to register, or you were on a list of beta candidates, or you had to come from a certain page, or you wanted to record a log of who got to it, etc. When this was done there were no ZClasses, so it was a pretty heavyweight activitiy to make a slight customization to enforce these policies. --Paul marc lindahl wrote:
From: Paul Everitt <paul@zope.com>
As the one behind "precondition" several years ago, I cringe to see it brought up in public. :^) An example of a reasonable idea that, years later, you just hate to be associate with.
So you're the one! :)
Seriously, though... I uncovered this (for myself, anyway) when working on my Audio product, and though the it seemed a good idea... but I'm still wondering for what :) Is it a nifty solution in search of a problem, or has someone found a truly unique use for it? Or should it be, er, demoted?
Makes sense... I'm interested in this thread because I think it would be handy to keep track of 'number of downloads' of various file types - for example, to build a list of 'most popular downloads', etc. It seems like perhaps, using the Catalog for that makes sense?
From: Paul Everitt <paul@zope.com> Date: Tue, 25 Sep 2001 11:19:30 -0400 To: marc lindahl <marc@bowery.com> Cc: zope@zope.org Subject: Re: [Zope] Counter and concurrent access
We wanted to enforce some rules that had to be enforced before someone could get to a File. For instance, you had to register, or you were on a list of beta candidates, or you had to come from a certain page, or you wanted to record a log of who got to it, etc.
When this was done there were no ZClasses, so it was a pretty heavyweight activitiy to make a slight customization to enforce these policies.
Paul Everitt wrote:
You're talking about ZODB growth around 500Kb per day, or 15Mb per month. With *any* amount of packing, this shouldn't be too much of a problem.
ZODB bloat was one problem, the other problem used to be write conflicts on high volume sites. But, with the application conflict resolution, this isn't a problem either... In fact, I think Mike did a counter just like this in one of his recent ZODB programming articles on zope.org... cheers, Chris
On Tue, 25 Sep 2001 11:45:47 +0100, Chris Withers <chrisw@nipltd.com> wrote:
Paul Everitt wrote:
You're talking about ZODB growth around 500Kb per day, or 15Mb per month. With *any* amount of packing, this shouldn't be too much of a problem.
ZODB bloat was one problem, the other problem used to be write conflicts on high volume sites.
But, with the application conflict resolution, this isn't a problem either...
A problem of the future might be extra intra-ZEO-server traffic, when replicated ZEO is ready. or maybe not...... Toby Dickenson tdickenson@geminidataloggers.com
participants (6)
-
Chris Withers -
Luciano Ramalho -
marc lindahl -
Paul Everitt -
Toby Dickenson -
Yann Lugrin