The view has access to the original request that ended in the error. The view is looked-up and rendered in a new transaction.

A new transaction means that *arbitrary* changes may have occurred to the application state in between: 1. the transaction that raise the error, and 2. the transaction that reports it In principal it is posssible to delete the database state and import a whole new application between these two points. The object containing the originally published method might no longer exist. I guess that might not matter if your error page just says "sorry an internal error has occurred" - if you want your users to be shielded from the details of any problem. I have some applications at the other extreme, where the users are all capable of debugging any problem and I want the error page to be a "core dump" of application state.

Can you give an example of the kind of situation where you'd need access to objects in a doomed transaction in their doomed state, in order to make an error report?

Access to the doomed state at the end of the transaction is what zope 2 gives us today. Access to the objects restored to their state as they were at the beginning of the failing transaction would be ok too (maybe even better) but zodb cant do that today. Either of these two are necessary for easily providing an error report that displays application state explaining why the error occurred. Under the Zope 3 model I think I would need to take a non-persistent copy of the application state in an exception handler towards the end of the first transaction. The second, error-reporting transaction would dump the copy, rather than dump what it sees to be the current world state. It is important that the objects passed from one transaction to the next are non-persistent, otherwise bad zodb-level things can happen. Does Zope 3 guard against this? -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson

On Friday 04 April 2003 8:55 pm, Dieter Maurer wrote:

Toby Dickenson wrote at 2003-4-4 15:54 +0100:

...

.... Access to the doomed state at the end of the transaction is what zope 2 gives us today.

This is not the case.

Zope 2 as it is now performs error handling in a separate transaction.

Yes. I was aware of this of course, but hadnt fully appreciated the implications for this debate. Thanks for the reminder.

...

It is important that the objects passed from one transaction to the next are non-persistent, otherwise bad zodb-level things can happen. Does Zope 3 guard against this?

I do not see this.

Caches routinely pass persistent objects from one transaction to the next.

Are there any examples in stock Zope 2? I would consider this a bug.

And think of a non-Zope ZODB application. This application is likely to have references to persistent objects in its variables and on its stack and will probably use it across transactions. At transaction boundaries the state of some of these objects is invalidated and reloaded transparently in the following transaction, if necessary.

I have some applications that work this way too. It is ok for applications that dont have a need need to be highly robust, but I dont think this approach belongs in the zope core.

Sure, when the object was deleted, a POSKeyError will result. This is what should happen.

There are scenarios where this POSKeyError is permanent, not transient. If a dangling reference has been committed into the storage then there may be no way to recover without manual repair of the storage. Im not sure that it cant do other damage too. it seems to me that zodb wasnt designed to operate in this manner.

I have two use cases:

* sending email about (some) severe errors.

We can use a non-transactional mail sender for this purpose although it would be nice to send all mail through a single (transactional) mailhost.

* gathering statistics about/logging errors in a relational database

IMO none of the proposed solutions are ideal for this. If the log-to-relational bit of the error reporting fails, you probably still want the error log email to be sent anyway and you probably dont want to fall back to zopes default (ugly) error page. Also, you dont necessarily need these actions to be run before sending the error page back to the server. It seems to me that these use cases call for a transaction that runs soon *after* the error event. It should be *seperate* to the method which generates the error report seen by the user. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson

On Monday 07 April 2003 5:19 pm, Florent Guillaume wrote:

...

...

Caches routinely pass persistent objects from one transaction to the next.

Are there any examples in stock Zope 2? I would consider this a bug.

Thanks for the pointers. All are in code that I am not familiar with, so I am not entirely confident in these analysis:

ZCTextIndex caches a persistent lexicon in _v_lexicon.

This class does have a clear() method, but I cant see where it is called from. I am assuming it is never called. At an application level this cache is not transparent. I think the following sequence is possible: 1. One transaction calls getLexicon. The appropriate object is found by name through acquisition, and placed in the cache. 2. Another transaction replaces the lexicon object with a different one. 3. A third transaction calls getLexicon again and gets the cached object, not the new object. This can last indefinitely, until Zope is shut down. At a zodb level: 4. Another transaction calls getLexicon and the return value is assigned as an attribute of a persistent object. The transaction commits. Things might continue to work until that lexicon object is deactivated (that is, possibly not until Zope is shut down). After that, there is no guarantee that the lexicon object can be accessed. Attempted accesses may give a POSKeyError. If this happens you will need to restore a backup. This should be 100% reproducable on bsddb.Minimal with no further steps. It would have purged the pickled state of the lexicon at the end of step 2 when the original lexicon object became unreachable from the zodb root. FileStorage will see this problem only if it is packed between step 3 and 4. I think the fix here is to empty the cache at the end of each transaction that fills it. This fixes the application-level and zodb-level problems. I would appreciate a comment from someone more familiar with this code. In this scenario DirectoryStorage would raise an exeception in step, preventing the commit that corrupts the storage.

DC.ZRDB.DA caches Bucket() which I think are persistent in _v_cache.

These objects are "persistent" in the sense that they derive from the persistent base class, but not in the sense that they are registered with a DB and written to disk. Only this second class of persistent objects cause problems when passed between transactions, so I believe this is safe at ZODB level. At application level, I understand the purpose of this cache is to be non-transparent. There is code in place to ensure that the cache is regularly refreshed.

CMFCore.MemberDataTool does a cache of persistent MemberData objects in _v_temps.

I dont have a CMF handy - I may check this tomorrow.

Is all this wrong ?

? -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson

On Monday 07 April 2003 10:09 pm, Florent Guillaume wrote:

Ok, from your exlanations, MemberDataTool should be now safe -- I patched it a few weeks ago to clear the cache at the end of the transaction using REQUEST._hold.

BTW, I use the attatched class to delete an attribute from an object at the end of a transaction. This is a little safer than REQUEST._hold in cases where request boundares are not aligned with transactions. Should this go in ZODB somewhere?

...

...

Is all this wrong ?

?

I meant, is all the above code using ad-hoc caches buggy in one way or another?

I meant, is it buggy enough to need fixing? -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson

Toby Dickenson wrote:

...

Ok, from your exlanations, MemberDataTool should be now safe -- I patched it a few weeks ago to clear the cache at the end of the transaction using REQUEST._hold.

BTW, I use the attatched class to delete an attribute from an object at the end of a transaction. This is a little safer than REQUEST._hold in cases where request boundares are not aligned with transactions. Should this go in ZODB somewhere?

It would be nice, yes. It would provide a better abstraction that REQUEST._hold and if you say it's safer, then that's a win too. Regarding Steve's suggestion:

I'd like to see attributes that get cleared at transaction boundaries made a first-class feature of the ZODB. This might be implemented for attributes starting with _x_ where x is The letter of your choice, following a similar pattern to _v_ and _p_ attributes.

It would be also very useful. _v_ is volatile, but not volatile enough... :) Florent

------------------------------------------------------------------------

""" attribute_cleaner

A class to remove an attribute from an object at the end of a transaction. This is useful for attributes that hold transaction-specific caches.

Normal operation is::

if self._v_cache is None: self._v_cache = create_data_to_cache() attribute_cleaner(self,'_v_cache') """

class attribute_cleaner: def __init__(self,client,attr): self.client = client self.attr = attr get_transaction().register(self)

def sortKey(self): return repr(self)

def ClearCache(self,*args): try: delattr(self.client, self.attr) except AttributeError: pass except KeyError: pass

tpc_finish = tpc_abort = abort = abort_sub = ClearCache

def tpc_begin(self,transaction,subtransaction=None): pass def commit(self,object,transaction): pass def tpc_vote(self,transaction): pass def commit_sub(self,transaction): pass

-- Florent Guillaume, Nuxeo (Paris, France) +33 1 40 33 79 87 http://nuxeo.com mailto:fg@nuxeo.com

Toby Dickenson wrote at 2003-4-6 20:50 +0100:

...

...

Caches routinely pass persistent objects from one transaction to the next.

Are there any examples in stock Zope 2? I would consider this a bug.

The ZODB caches (implemented by yourself) do.

.... It seems to me that these use cases call for a transaction that runs soon *after* the error event. It should be *seperate* to the method which generates the error report seen by the user.

I am starting to agree that your proposal is the better one: * The standard case will be that error handling does not change the ZODB state. For this standard case, it is not relevant whether it is executed in the old transaction or a new one. * Your use case requires that error handling is done in the original transaction. When the framework aborts it, there is no way to implement it. * My use cases can explicitely abort the old transaction and start a new one -- in case the framework delays abortion. * Having several transactions in a single request makes handling of "ConflictError"s in later transactions mode difficult. Dieter