hi all! i make heavy use of indexes in my extension classes. these all inherit from catalogpathaware, so i have to call object.reindex_object() on each changed instance. calling it from attribute getters/setters f.ex. is not a good idea, because changing 3 attributes will reindex the object 3 times. what i'd like to have is that such objects are reindexed automatically before comitting a transaction. is it possible? where should i start looking in the source, is there possibly a before_transaction_commit hook? regards, juergen herrmann _______________________________________________________________________
XLhost.de - eXperts in Linux hosting <<
Juergen Herrmann Weiherweg 10, 93051 Regensburg, Germany Fon: +49 (0)700 XLHOSTDE [0700 95467833] Fax: +49 (0)721 151 463027 ICQ: 27139974 - IRC: #XLhost@quakenet WEB: http://www.XLhost.de
----- Original Message ----- From: "Jürgen Herrmann" <Juergen.Herrmann@XLhost.de>
i make heavy use of indexes in my extension classes. these all inherit from catalogpathaware, so i have to call object.reindex_object() on each changed instance. calling it from attribute getters/setters f.ex. is not a good idea, because changing 3 attributes will reindex the object 3 times.
what i'd like to have is that such objects are reindexed automatically before comitting a transaction.
You can't index before committing. I think you need a different strategy: how about creating all of the new objects and then call the catalog/indexing mechanism. We do this for one of our large volume catalogs (1m+ records), by having an object class/catalog that are not 'aware' and then manually (well, thru a python routine) cataloging/indexing the objects. HTH Jonathan
[ Jonathan wrote:]
----- Original Message ----- From: "Jürgen Herrmann" <Juergen.Herrmann@XLhost.de>
i make heavy use of indexes in my extension classes. these all inherit from catalogpathaware, so i have to call object.reindex_object() on each changed instance. calling it from attribute getters/setters f.ex. is not a good idea, because changing 3 attributes will reindex the object 3 times.
what i'd like to have is that such objects are reindexed automatically before comitting a transaction.
You can't index before committing. I think you need a different strategy: how about creating all of the new objects and then call the catalog/indexing mechanism. We do this for one of our large volume catalogs (1m+ records), by having an object class/catalog that are not 'aware' and then manually (well, thru a python routine) cataloging/indexing the objects.
hmm, it's not about newly created objects. i want to have automatic REindexing, in cleartext - i want to make it easy for the programmer: he/she should just call an attribute setter or maybe change a relation and just before the transaction would be committed to the storage, i'd like to cycle through all dirty objects and reindex them, if they're catalogpathaware (or implement reindex_object()...).
HTH
Jonathan
juergen _______________________________________________________________________
XLhost.de - eXperts in Linux hosting <<
Juergen Herrmann Weiherweg 10, 93051 Regensburg, Germany Fon: +49 (0)700 XLHOSTDE [0700 95467833] Fax: +49 (0)721 151 463027 ICQ: 27139974 - IRC: #XLhost@quakenet WEB: http://www.XLhost.de
Personally I think one should stay the hell away from transactions. They're not for you to fiddle with. I understand your question and even though I think it's a bit crazy I can see a benefit (simplicity for the programmer). How about a trickery solution like this: class CatalogPathAware: def index_object(self, *a, **k): print self.__class__.__name__, "in for indexing!" class _MyProduct(CatalogPathAware): def __init__(self, id, title=''): self.id = id self.title = title def setTitle(self, title): # could be much more complicated self.title = title class MyProduct(_MyProduct): def setTitle(self, title): _MyProduct.setTitle(self, title) self.index_object() inst = MyProduct("instanceA") inst.setTitle('Peter') print inst.title You would basically rename your existing class from "Whatever" to "_Whatever" and continue as before. This does mean however that you have to explicitly write a function for each setter. On 6/17/05, Jürgen Herrmann <Juergen.Herrmann@xlhost.de> wrote:
[ Jonathan wrote:]
----- Original Message ----- From: "Jürgen Herrmann" <Juergen.Herrmann@XLhost.de>
i make heavy use of indexes in my extension classes. these all inherit from catalogpathaware, so i have to call object.reindex_object() on each changed instance. calling it from attribute getters/setters f.ex. is not a good idea, because changing 3 attributes will reindex the object 3 times.
what i'd like to have is that such objects are reindexed automatically before comitting a transaction.
You can't index before committing. I think you need a different strategy: how about creating all of the new objects and then call the catalog/indexing mechanism. We do this for one of our large volume catalogs (1m+ records), by having an object class/catalog that are not 'aware' and then manually (well, thru a python routine) cataloging/indexing the objects.
hmm, it's not about newly created objects. i want to have automatic REindexing, in cleartext - i want to make it easy for the programmer: he/she should just call an attribute setter or maybe change a relation and just before the transaction would be committed to the storage, i'd like to cycle through all dirty objects and reindex them, if they're catalogpathaware (or implement reindex_object()...).
HTH
Jonathan
juergen _______________________________________________________________________
XLhost.de - eXperts in Linux hosting <<
Juergen Herrmann Weiherweg 10, 93051 Regensburg, Germany Fon: +49 (0)700 XLHOSTDE [0700 95467833] Fax: +49 (0)721 151 463027
ICQ: 27139974 - IRC: #XLhost@quakenet WEB: http://www.XLhost.de _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
-- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com
[ Peter Bengtsson wrote:]
Personally I think one should stay the hell away from transactions. They're not for you to fiddle with. I understand your question and even though I think it's a bit crazy I can see a benefit (simplicity for the programmer).
How about a trickery solution like this:
class CatalogPathAware: def index_object(self, *a, **k): print self.__class__.__name__, "in for indexing!"
class _MyProduct(CatalogPathAware): def __init__(self, id, title=''): self.id = id self.title = title def setTitle(self, title): # could be much more complicated self.title = title
class MyProduct(_MyProduct): def setTitle(self, title): _MyProduct.setTitle(self, title) self.index_object()
inst = MyProduct("instanceA") inst.setTitle('Peter')
print inst.title
You would basically rename your existing class from "Whatever" to "_Whatever" and continue as before. This does mean however that you have to explicitly write a function for each setter.
hmm, i can't see how this would help. if i call index_object in every attribute getter method, the indexing is done possibly several times, unnecessary and (cpu)time consuming. juergen _______________________________________________________________________
XLhost.de - eXperts in Linux hosting <<
Juergen Herrmann Weiherweg 10, 93051 Regensburg, Germany Fon: +49 (0)700 XLHOSTDE [0700 95467833] Fax: +49 (0)721 151 463027 ICQ: 27139974 - IRC: #XLhost@quakenet WEB: http://www.XLhost.de
On 6/17/05, Jürgen Herrmann <Juergen.Herrmann@xlhost.de> wrote:
[ Peter Bengtsson wrote:]
Personally I think one should stay the hell away from transactions. They're not for you to fiddle with. I understand your question and even though I think it's a bit crazy I can see a benefit (simplicity for the programmer).
How about a trickery solution like this:
class CatalogPathAware: def index_object(self, *a, **k): print self.__class__.__name__, "in for indexing!"
class _MyProduct(CatalogPathAware): def __init__(self, id, title=''): self.id = id self.title = title def setTitle(self, title): # could be much more complicated self.title = title
class MyProduct(_MyProduct): def setTitle(self, title): _MyProduct.setTitle(self, title) self.index_object()
inst = MyProduct("instanceA") inst.setTitle('Peter')
print inst.title
You would basically rename your existing class from "Whatever" to "_Whatever" and continue as before. This does mean however that you have to explicitly write a function for each setter.
hmm, i can't see how this would help. if i call index_object in every attribute getter method, the indexing is done possibly several times, unnecessary and (cpu)time consuming.
Writes happen a lot less often but they also cost a lot more. It'd be nice to be able to hotwire the parentless class so that it magically "wraps" all functions that match on:: re.compile('set[A-Z]\w+') It would be hard to find out what the functions are, eg:: class MyProduct(_MyProduct): def __init__(self, *a, **k): _MyProduct.__init__(self, *a, **k) for fun in dir(_MyProduct): if camel_setters_regex.findall(fun): print "create a wrapping method called %s" % fun How to magically create a method upon itself with an arbitary name is something I don't know how to do but it's not impossible because I know that Archetypes (cmf, plone) does it.
juergen _______________________________________________________________________
XLhost.de - eXperts in Linux hosting <<
Juergen Herrmann Weiherweg 10, 93051 Regensburg, Germany Fon: +49 (0)700 XLHOSTDE [0700 95467833] Fax: +49 (0)721 151 463027
ICQ: 27139974 - IRC: #XLhost@quakenet WEB: http://www.XLhost.de
-- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com
On Fri, Jun 17, 2005 at 02:19:26PM +0200, J?rgen Herrmann wrote:
hi all!
i make heavy use of indexes in my extension classes. these all inherit from catalogpathaware, so i have to call object.reindex_object() on each changed instance. calling it from attribute getters/setters f.ex. is not a good idea, because changing 3 attributes will reindex the object 3 times.
what i'd like to have is that such objects are reindexed automatically before comitting a transaction.
is it possible? where should i start looking in the source, is there possibly a before_transaction_commit hook?
I vaguely recall a similar discussion on zodb-dev; a search of the archives of that list may be a good idea. You might also find QueueCatalog useful. AFAICT, if you are using QueueCatalog and an object asks to be reindexed three times before the next time the queue is processed, the QueueCatalog will only actually reindex the object once. The tradeoff is that the catalog results can be stale. It's only available via CVS, and doesn't come with docs, but the embedded docstrings look pretty good. Note, I haven't used it yet myself. http://cvs.zope.org/Products/QueueCatalog/ Or to get a tarball, http://cvs.zope.org/Products/QueueCatalog/QueueCatalog.tar.gz?tarball=1 It looks like you need some external process to actually trigger the processing of the queue. A cron job calling a wget script would do the job, or (better imho because it runs as a separate process) if you are using zeo you could run a pretty trivial python script via bin/zopectl ... something like (untested): import time while 1: time.sleep(30) events_count = app.path.to.my.queuecatalog.process(max=1000) print "processed %d events" % events_count # or log it somewhere. Hope that helps, -PW -- Paul Winkler http://www.slinkp.com
Jürgen Herrmann wrote at 2005-6-17 14:19 +0200:
i make heavy use of indexes in my extension classes. these all inherit from catalogpathaware, so i have to call object.reindex_object() on each changed instance. calling it from attribute getters/setters f.ex. is not a good idea, because changing 3 attributes will reindex the object 3 times.
what i'd like to have is that such objects are reindexed automatically before comitting a transaction.
is it possible? where should i start looking in the source, is there possibly a before_transaction_commit hook?
It is impossible with "ZODB 3.2" (unless you patch "ZODB.Transaction.Transaction"). "ZODB 3.4" (which is used for Zope 2.8/3.1) has hooks that makes it possible. -- Dieter
[ Dieter Maurer wrote:]
Jürgen Herrmann wrote at 2005-6-17 14:19 +0200:
i make heavy use of indexes in my extension classes. these all inherit from catalogpathaware, so i have to call object.reindex_object() on each changed instance. calling it from attribute getters/setters f.ex. is not a good idea, because changing 3 attributes will reindex the object 3 times.
what i'd like to have is that such objects are reindexed automatically before comitting a transaction.
is it possible? where should i start looking in the source, is there possibly a before_transaction_commit hook?
It is impossible with "ZODB 3.2" (unless you patch "ZODB.Transaction.Transaction").
"ZODB 3.4" (which is used for Zope 2.8/3.1) has hooks that makes it possible.
-- Dieter
that's what i wanted to hear, thanks! i'll have a look at zope 2.8 immediately :) regards, juergen herrmann _______________________________________________________________________
XLhost.de - eXperts in Linux hosting <<
Juergen Herrmann Weiherweg 10, 93051 Regensburg, Germany Fon: +49 (0)700 XLHOSTDE [0700 95467833] Fax: +49 (0)721 151 463027 ICQ: 27139974 - IRC: #XLhost@quakenet WEB: http://www.XLhost.de
Dieter Maurer <dieter@handshake.de> wrote:
Jürgen Herrmann wrote at 2005-6-17 14:19 +0200:
i make heavy use of indexes in my extension classes. these all inherit from catalogpathaware, so i have to call object.reindex_object() on each changed instance. calling it from attribute getters/setters f.ex. is not a good idea, because changing 3 attributes will reindex the object 3 times.
what i'd like to have is that such objects are reindexed automatically before comitting a transaction.
is it possible? where should i start looking in the source, is there possibly a before_transaction_commit hook?
It is impossible with "ZODB 3.2" (unless you patch "ZODB.Transaction.Transaction").
The CPSCompat module of CPS has monkey-patches that backport this from ZODB 3.4, among others, to be used in Zope 2.7. http://svn.nuxeo.org/trac/pub/file/CPSCompat/trunk/PatchZODBTransaction.py Florent
"ZODB 3.4" (which is used for Zope 2.8/3.1) has hooks that makes it possible.
-- Florent Guillaume, Nuxeo (Paris, France) CTO, Director of R&D +33 1 40 33 71 59 http://nuxeo.com fg@nuxeo.com
hi all! as i had time to look at all the stuff, i realized, that i'm getting closer, but this is not exactly what i wanted... as i can see from the path, one has to call: beforeCommitHook(method, **args, **kwargs) on each transaction, correct? this is contrary to my idea of everything doing it's work "automagically". what i want is a kind of callback to each dirty object, something like this: class Transaction(...): def commit(self, ...): # insert this: for object in self._objects: try: object._before_transaction_commit() except AttributeError: pass # rest of original commit() follows... could this impose any unforseen behaviour? btw: what version of the two following is better (i.e. faster and more elegeant, i do python coding since abt. 6months only...) try: object._before_transaction_commit() except AttributeError: pass or hook = object.get('_before_transaction_commit', None) if hook: hook() another question: is it hook() or hook(object) in the previous line? i never know if the self parameter has to be passed in if not called as self.method() - are there any (simple) rules? regards, juergen herrmann [ Florent Guillaume wrote:]
Dieter Maurer <dieter@handshake.de> wrote:
Jürgen Herrmann wrote at 2005-6-17 14:19 +0200:
i make heavy use of indexes in my extension classes. these all inherit from catalogpathaware, so i have to call object.reindex_object() on each changed instance. calling it from attribute getters/setters f.ex. is not a good idea, because changing 3 attributes will reindex the object 3 times.
what i'd like to have is that such objects are reindexed automatically before comitting a transaction.
is it possible? where should i start looking in the source, is there possibly a before_transaction_commit hook?
It is impossible with "ZODB 3.2" (unless you patch "ZODB.Transaction.Transaction").
The CPSCompat module of CPS has monkey-patches that backport this from ZODB 3.4, among others, to be used in Zope 2.7.
http://svn.nuxeo.org/trac/pub/file/CPSCompat/trunk/PatchZODBTransaction.py
Florent
"ZODB 3.4" (which is used for Zope 2.8/3.1) has hooks that makes it possible.
-- Florent Guillaume, Nuxeo (Paris, France) CTO, Director of R&D +33 1 40 33 71 59 http://nuxeo.com fg@nuxeo.com _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
_______________________________________________________________________
XLhost.de - eXperts in Linux hosting <<
Juergen Herrmann Weiherweg 10, 93051 Regensburg, Germany Fon: +49 (0)700 XLHOSTDE [0700 95467833] Fax: +49 (0)721 151 463027 ICQ: 27139974 - IRC: #XLhost@quakenet WEB: http://www.XLhost.de
[ Jürgen Herrmann wrote:] ...
hook = object.get('_before_transaction_commit', None) if hook: hook()
... of course it should be: getattr(object, '_before_transaction_commit', None) regards, juergen herrmann _______________________________________________________________________
XLhost.de - eXperts in Linux hosting <<
Juergen Herrmann Weiherweg 10, 93051 Regensburg, Germany Fon: +49 (0)700 XLHOSTDE [0700 95467833] Fax: +49 (0)721 151 463027 ICQ: 27139974 - IRC: #XLhost@quakenet WEB: http://www.XLhost.de
Have a look at how CPS uses this hook to delay indexing to the end of the transaction. Code is at http://svn.nuxeo.org/trac/pub/file/CPSCore/trunk/ IndexationManager.py Then in all objects for which we want to delay indexing, we replace the reindexObject() method with: def reindexObject(self, idxs=[]): """Schedule object for reindexation """ get_indexation_manager().push(self, idxs=idxs) And the IndexationManager calls _reindexObject at the end of the transaction. This is done for instance in http://svn.nuxeo.org/trac/pub/file/ CPSCore/trunk/ProxyBase.py Florent On 22 Jun 2005, at 14:58, Jürgen Herrmann wrote:
hi all!
as i had time to look at all the stuff, i realized, that i'm getting closer, but this is not exactly what i wanted...
as i can see from the path, one has to call: beforeCommitHook(method, **args, **kwargs) on each transaction, correct? this is contrary to my idea of everything doing it's work "automagically".
what i want is a kind of callback to each dirty object, something like this:
class Transaction(...): def commit(self, ...): # insert this: for object in self._objects: try: object._before_transaction_commit() except AttributeError: pass # rest of original commit() follows...
could this impose any unforseen behaviour? btw: what version of the two following is better (i.e. faster and more elegeant, i do python coding since abt. 6months only...)
try: object._before_transaction_commit() except AttributeError: pass
or hook = object.get('_before_transaction_commit', None) if hook: hook()
another question: is it hook() or hook(object) in the previous line? i never know if the self parameter has to be passed in if not called as self.method() - are there any (simple) rules?
regards, juergen herrmann
[ Florent Guillaume wrote:]
Dieter Maurer <dieter@handshake.de> wrote:
Jürgen Herrmann wrote at 2005-6-17 14:19 +0200:
i make heavy use of indexes in my extension classes. these all inherit from catalogpathaware, so i have to call object.reindex_object() on each changed instance. calling it from attribute getters/setters f.ex. is not a good idea, because changing 3 attributes will reindex the
object
3 times.
what i'd like to have is that such objects are reindexed automatically before comitting a transaction.
is it possible? where should i start looking in the source, is there possibly a before_transaction_commit hook?
It is impossible with "ZODB 3.2" (unless you patch "ZODB.Transaction.Transaction").
The CPSCompat module of CPS has monkey-patches that backport this from ZODB 3.4, among others, to be used in Zope 2.7.
http://svn.nuxeo.org/trac/pub/file/CPSCompat/trunk/ PatchZODBTransaction.py
Florent
"ZODB 3.4" (which is used for Zope 2.8/3.1) has hooks that makes it possible.
-- Florent Guillaume, Nuxeo (Paris, France) CTO, Director of R&D +33 1 40 33 71 59 http://nuxeo.com fg@nuxeo.com _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
______________________________________________________________________ _
XLhost.de - eXperts in Linux hosting <<
Juergen Herrmann Weiherweg 10, 93051 Regensburg, Germany Fon: +49 (0)700 XLHOSTDE [0700 95467833] Fax: +49 (0)721 151 463027
ICQ: 27139974 - IRC: #XLhost@quakenet WEB: http://www.XLhost.de
-- Florent Guillaume, Nuxeo (Paris, France) CTO, Director of R&D +33 1 40 33 71 59 http://nuxeo.com fg@nuxeo.com
participants (6)
-
Dieter Maurer -
Florent Guillaume -
Jonathan -
Jürgen Herrmann -
Paul Winkler -
Peter Bengtsson