Hooking persistent.Persistent.__setstate__ was Re: [ZODB-Dev] Analyzing a ZODB.

[Jim Fulton]

On Apr 5, 2008, at 6:30 PM, Alan Runyan wrote:
> Here is something more ZODB and less Zope related (kinda).
>
> I was talking with Benji a few weeks ago about a problem that should
> be easy to debug but was not.  Here is the scenerio:
>
>  - Customer has software on a remote machine.  They are seeing
>  unnecessary transaction commits.  Just like the guy 'Analyzing a  
> ZODB'.
>
>  - Customer is completely incapable of doing anything other than  
> putting
>  a script on the filesystem.
>
>  - Benji and I thought about it and he proposed 'the simplest thing  
> that
>  c/should work, monkey patch persistent.__setstate__' so that I  
> could see
>  what objects were being mutated.

That doesn't show you which objects are being mutated. It only shows  
you which objects are being loaded.

If you want to see what objects are actually being mutated, it would  
be better, IMO, to set up some sort of event channel on object  
invalidations.

If you want to see what code is mutating objects, it would be better  
to set breakpoints in Connection.register.


>
>   - Unfortunately ZODB 3.x does not have a Python fallback of
>  persistent.Persistent -- its in C.  The customer did not have a C  
> compiler
>  on their box.
>
> IIRC how I solved it was increase ZEO event log to see the oid's.   
> Then I
> walked him through loading the oid up to see what object was being
> mutated.  It was more painful than it should have been.

Yup, but automating something like this might have been informative  
without being painful.

> Question:  Is it possible for ZODB 3.9 to have a pure python
> implementation of persistent.Persistent?

Probably.

>  Maybe this would be a good
> ZODB GSOC project?

The current C-level persistence implementation is rather nasty because  
of the close coupling with the persistent cache implementation.

My goal is to come up with a new persistence implementation that is  
much simpler and more flexible with both Python and C  
implementations.  I haven't had time to work on this though. :(  If  
someone wanted to work on this, I'd be happy to go into more detail.

...

>  - increasing zeo server log level and watching oid's being changed
> is sort-of the equivalent of turning on RDBMS logging to see SQL
> stmt's being executed.  Unfortunately I believe without having a
> hook in persistent.Persistent we can never really get that level
> of granularity (i.e. __getattribute__ is only accessible in client)
> with only ZEO server logs.

Regardless of whether it's implementation is in C or Python, the  
current implementation is complex and highly coupled.  A more flexible  
system that I envision, would make it easier to experiment with  
alternate cache implementations and would also make it easier to  
substitute implementations that did more logging for debugging purposes.

> This is a transparent plea for a "new feature".  But I believe it  
> would
> significantly help people writing ZODB applications.
>
> Maybe people should always have a base class that you override
> those methods which delegate to persistent.Persistent.  i.e.
> class MyMixin(persistent.Persistent) and mixin MyMixin
> instead of mixin persistent.Persistent directly?  Then
> you can instrument the MyMixin with the logging?

That would introduce lots of overhead and complexity that I don't  
think is warranted.

Jim

--
Jim Fulton
Zope Corporation