[Zope] [Further investigations] Re: A question about __setstate__ in Shared/DC/ZRDB/Connection.py

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Marco Bizzarri wrote:
> On Fri, Sep 19, 2008 at 9:23 AM, Marco Bizzarri
> <marco.bizzarri at gmail.com> wrote:
>> Hi all.
>>
>> I'm working on an application which uses Zope (2.8, at the moment) and
>> ZPsycopgDA (toghter with a number of other products).
>>
>> While writing an acceptance test, I encountered a strange problem: the
>> test locks up.
>>
>> A further investigation shown that there were two connections at the
>> database; one of them was not committed, the other one was blocked
>> waiting for the other to commit.
>>
>> I therefore used the pdb in order to stop the execution of the test
>> inside the connect method of the ZPsycopgDA.DA. Once I had that
>> breakpoint, I was able to get the logs of the two transactions on the
>> database, and I had the confirmation that indeed there were two
>> different transactions.
>>
>> So, I wondered what could possibily happen, I mean why during a test
>> there could be a second connect to the database.
>>
>> I issued a "bt" to see the stack of calls leading to the connect, and
>> what I could see was that the coonect was called inside the
>> __setstate__ method of Shared/DC/ZRDB/Connection.py.
>>
>> I assume therefore that the ZPsycopgDA object has been "ghostified",
>> during the transaction. But this "assumption" is not supported by any
>> evidence. In particular, it is not supported by my knowledge of the
>> internal behaviour of ZODB on objects during a single transaction.
>>
>> Can anyone provide suggestion on this topic?
>>
>> Regards
>> Marco
>> --
>> Marco Bizzarri
>> http://notenotturne.blogspot.com/
>> http://iliveinpisa.blogspot.com/
>>
> 
> I did further investigation on the topic, and I think I've pinned the
> problem. I don't know the solution, but I can reproduce the problem
> with a small sample. Here is the sample:
> 
> 
> import os
> import sys
> import unittest
> 
> if __name__ == '__main__':
>     execfile(os.path.join(sys.path[0], '../framework.py'))
> 
> from Testing import ZopeTestCase
> 
> from OFS import Image
> 
> from Products.ZPsycopgDA.DA import manage_addZPsycopgConnection
> from Products.ZSQLMethods import SQL
> 
> 
> class DoubleTransactionTest(ZopeTestCase.ZopeTestCase):
> 
>     def _add_big_image(self, value, data):
>         Image.manage_addFile(self.app, "f%06s" % value, data , "a title")
> 
>     def test_showdouble(self):
>         manage_addZPsycopgConnection(self.app, "db_connection", "",
> "host=localhost user=postgres dbname=template1")
>         self.app._setObject('sql', SQL.SQL("sql", "", "db_connection",
> "", "select * from pg_tables"))
>         self.app.sql()
>         data =  "*" * (1 << 20)
>         for x in range(1000):
>             self._add_big_image(x, data)
>             print "Added %s " % x
>         self.app.sql()
> 
> if __name__ == '__main__':
>     unittest.main()
> 
> 
> I'm doing three things here:
> 
> - creating a db connection
> - making a query to the db (this causes a transaction to begin)
> - creating a lot of "big" files (expecially, larger than 2 * 2 ^ 16 *)
> - making another query to the db;
> 
> Once I create a big file I fall into the following branch inside the
> OFS.Image._read_data
> 
> 
>         if size <= 2*n:
>             seek(0)
>             if size < n: return read(size), size
>             return Pdata(read(size)), size
> 
>         # Make sure we have an _p_jar, even if we are a new object, by
>         # doing a sub-transaction commit.
>         transaction.savepoint(optimistic=True)
> 
> This causes, at the end, to call the ZODB.Connection.savepoint which,
> just before returning, calls a cacheGC to be called, which, I'm
> afraid, causes the db_connection to be "sent" out of the cache itself,
> thus leaving it without the _v_ attributes.
> 
> Hope this can help in giving suggestions.

Thanks for digging further into it;  I couldn't imagine how that was
occurring.  In this case, the large number of created Pdata objects (one
per 64k chunk of each of your images) are causing your connection object
to be evicted from the cache at one of the savepoints, and thus
ghostified (which is where it loses its volatiles).

There is a special 'STICKY' state which prevents ghostifying, but it
can't be set from Python code.  You could, however, set '_p_changed' on
the connection at the beginning of the method, and then delete it at the
end:  changed objects can't be ghostified.  E.g.:

   def my_method(self):
       self.connection._p_changed = 1
       try:
           self.sql()
           # now do the stuff which used to ghostify the connection
       finally:
           del self.connection._p_changed

It is a nasty workaround, but should help prevent the lockup.

Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFI06oM+gerLs4ltQ4RAgq8AJ9yTjrFTtIt+IEPtghZIX/627IBjACeLG1f
wm9dSVcCcB/wT5N4DXMumSw=
=MI9F
-----END PGP SIGNATURE-----