[Zope] [Further investigations] Re: A question about __setstate__ in Shared/DC/ZRDB/Connection.py

[Marco Bizzarri]

On Fri, Sep 19, 2008 at 9:23 AM, Marco Bizzarri
<marco.bizzarri at gmail.com> wrote:
> Hi all.
>
> I'm working on an application which uses Zope (2.8, at the moment) and
> ZPsycopgDA (toghter with a number of other products).
>
> While writing an acceptance test, I encountered a strange problem: the
> test locks up.
>
> A further investigation shown that there were two connections at the
> database; one of them was not committed, the other one was blocked
> waiting for the other to commit.
>
> I therefore used the pdb in order to stop the execution of the test
> inside the connect method of the ZPsycopgDA.DA. Once I had that
> breakpoint, I was able to get the logs of the two transactions on the
> database, and I had the confirmation that indeed there were two
> different transactions.
>
> So, I wondered what could possibily happen, I mean why during a test
> there could be a second connect to the database.
>
> I issued a "bt" to see the stack of calls leading to the connect, and
> what I could see was that the coonect was called inside the
> __setstate__ method of Shared/DC/ZRDB/Connection.py.
>
> I assume therefore that the ZPsycopgDA object has been "ghostified",
> during the transaction. But this "assumption" is not supported by any
> evidence. In particular, it is not supported by my knowledge of the
> internal behaviour of ZODB on objects during a single transaction.
>
> Can anyone provide suggestion on this topic?
>
> Regards
> Marco
> --
> Marco Bizzarri
> http://notenotturne.blogspot.com/
> http://iliveinpisa.blogspot.com/
>

I did further investigation on the topic, and I think I've pinned the
problem. I don't know the solution, but I can reproduce the problem
with a small sample. Here is the sample:


import os
import sys
import unittest

if __name__ == '__main__':
    execfile(os.path.join(sys.path[0], '../framework.py'))

from Testing import ZopeTestCase

from OFS import Image

from Products.ZPsycopgDA.DA import manage_addZPsycopgConnection
from Products.ZSQLMethods import SQL


class DoubleTransactionTest(ZopeTestCase.ZopeTestCase):

    def _add_big_image(self, value, data):
        Image.manage_addFile(self.app, "f%06s" % value, data , "a title")

    def test_showdouble(self):
        manage_addZPsycopgConnection(self.app, "db_connection", "",
"host=localhost user=postgres dbname=template1")
        self.app._setObject('sql', SQL.SQL("sql", "", "db_connection",
"", "select * from pg_tables"))
        self.app.sql()
        data =  "*" * (1 << 20)
        for x in range(1000):
            self._add_big_image(x, data)
            print "Added %s " % x
        self.app.sql()

if __name__ == '__main__':
    unittest.main()


I'm doing three things here:

- creating a db connection
- making a query to the db (this causes a transaction to begin)
- creating a lot of "big" files (expecially, larger than 2 * 2 ^ 16 *)
- making another query to the db;

Once I create a big file I fall into the following branch inside the
OFS.Image._read_data


        if size <= 2*n:
            seek(0)
            if size < n: return read(size), size
            return Pdata(read(size)), size

        # Make sure we have an _p_jar, even if we are a new object, by
        # doing a sub-transaction commit.
        transaction.savepoint(optimistic=True)

This causes, at the end, to call the ZODB.Connection.savepoint which,
just before returning, calls a cacheGC to be called, which, I'm
afraid, causes the db_connection to be "sent" out of the cache itself,
thus leaving it without the _v_ attributes.

Hope this can help in giving suggestions.

Regards
Marco

-- 
Marco Bizzarri
http://notenotturne.blogspot.com/
http://iliveinpisa.blogspot.com/