[ZODB-Dev] Repair corrupted RelStorage database

Rudá Porto Filgueiras rudazz at gmail.com
Sat Jan 30 11:51:34 EST 2010 

On Sat, Jan 30, 2010 at 6:56 AM, Shane Hathaway <shane at hathawaymix.org>wrote:

> Rudá Porto Filgueiras wrote:
>
>> How can I repair or verify a RelStorage database?
>> I´m using MySQL Adapter, there is any script like fsrecover.py
>>
>> Follow the failure when RelStorage try to load some objects:
>>
>> 2010-01-30T04:25:14 ERROR ZODB.Connection Couldn't load state for
>> 0x04ef0f59
>> Traceback (most recent call last):
>>  File
>> "/usr/local/zope-agecom/plone/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
>> line 847, in setstate
>>    self._setstate(obj)
>>  File
>> "/usr/local/zope-agecom/plone/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
>> line 888, in _setstate
>>    p, serial = self._storage.load(obj._p_oid, self._version)
>>  File
>> "/usr/local/zope-agecom/plone/eggs/RelStorage-1.4.0b1-py2.4.egg/relstorage/storage.py",
>> line 432, in load
>>    raise POSKeyError(oid)
>> POSKeyError: 0x04ef0f59
>>
>
> To diagnose this, I would:
>
> - Examine the logs in detail.  There should be a log entry giving details
> about that specific POSKeyError.
>


Follow error from november 2009 (there are some repeated (five) errors like
that on the date/time).

550308-2009-11-28T05:31:51 ERROR ZODB.Connection Couldn't load state for
0x03ebd4
550309-Traceback (most recent call last):
550310-  File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
line 797, in setstate
550311-    self._setstate(obj)
550312-  File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
line 855, in _setstate
550313-    self._reader.setGhostState(obj, p)
550314-  File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/serialize.py",
line 604, in setGhostState
550315-    state = self.getState(pickle)
550316-  File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/serialize.py",
line 597, in getState
550317-    return unpickler.load()
550318-  File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/serialize.py",
line 471, in _persistent_load
550319-    return self.load_oid(reference)
550320-  File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/serialize.py",
line 537, in load_oid
550321-    return self._conn.get(oid)
550322-  File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
line 225, in get
550323-    p, serial = self._storage.load(oid, self._version)
550324-  File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
line 1196, in load
550325-    return self._storage.load(oid, self._base_version)
550326-  File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZEO/ClientStorage.py",
line 727, in load
550327-    return self.loadEx(oid, version)[:2]
550328-  File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZEO/ClientStorage.py",
line 750, in loadEx
550329-    data, tid, ver = self._server.loadEx(oid, version)
550330-  File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZEO/ServerStub.py",
line 196, in loadEx
550331-    return self.rpc.call("loadEx", oid, version)
550332-  File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZEO/zrpc/connection.py",
line 650, in call
550333-    raise inst # error raised by server
550334:POSKeyError: 0x06294e

This errros is today errors, there are repeated (10) errors like that:

835744:2010-01-30T09:26:47 WARNING relstorage POSKeyError on oid 1793877: no
tid found; Current transaction is 253262639700997905; Recent object tids: []
835745-------
835746-2010-01-30T09:26:47 ERROR ZODB.Connection Couldn't load state for
0x1b5f55
835747-Traceback (most recent call last):
835748-  File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
line 847, in setstate
835749-    self._setstate(obj)
835750-  File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
line 888, in _setstate
835751-    p, serial = self._storage.load(obj._p_oid, self._version)
835752-  File
"/usr/local/zope/agecom-virtual/eggs/RelStorage-1.4.0b1-py2.4.egg/relstorage/storage.py",
line 432, in load
835753:    raise POSKeyError(oid)
835754:POSKeyError: 0x1b5f55
835756-2010-01-30T09:26:48 INFO Archetypes
ESC[00mESC[01;32m/usr/local/zope/agecom-virtual/parts/plone/Archetypes/UIDCatalog.py[106]:getObject
835757-ESC[00mUIDCatalogBrains getObject raised an error:
835758- Traceback (most recent call last):
835759-
835760-  File
"/usr/local/zope/agecom-virtual/parts/plone/Archetypes/UIDCatalog.py", line
87, in getObject
835761-    path = self.getPath()
835762-
835763-  File
"/usr/local/zope/agecom-virtual/parts/zope2/lib/python/Products/ZCatalog/CatalogBrains.py",
line 33, in getPath
835764-    return self.aq_parent.getpath(self.data_record_id_)
835765-
835766-  File
"/usr/local/zope/agecom-virtual/parts/zope2/lib/python/Products/ZCatalog/ZCatalog.py",
line 608, in getpath
835767-    return self._catalog.paths[rid]
835768-
835769-  File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
line 847, in setstate
835770-    self._setstate(obj)
835771-
835772-  File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
line 888, in _setstate
835773-    p, serial = self._storage.load(obj._p_oid, self._version)
835774-
835775-  File
"/usr/local/zope/agecom-virtual/eggs/RelStorage-1.4.0b1-py2.4.egg/relstorage/storage.py",
line 432, in load
835776:    raise POSKeyError(oid)
835777-
835778:POSKeyError: 0x1b5f55


> - Shut down MySQL and back up the database as-is.
>
> I discover the problem after restore database on another MySQL/Zope server
and try to do Plone migration 2.5 -> 3.2.


> - Run the MySQL database verification and repair tools.  It's possible that
> the object you need is there, but MySQL can't find it due to corrupted
> indexes or similar.
>
>
I will do that, but it must afert 02:00 am. :-)
Did you think PostgreSQL is more resistent to such failures than MySQL?


> - Run a pack in dry-run mode, which will fill the object_refs table so I
> can find out exactly what object(s) refer to that OID (note that 0x04ef0f59
> is 82775897 in decimal).
>
>
Yes, I know that, there is a function to do this conversion on ZODB package.
Is
>>> from ZODB.utils import U64
>>> U64('0x1b5f55')
3492595808898266421L



> - Use both SQL and "zopectl debug" to examine the breakage.
>

On SQL I need to inspect current_object, object_ref and object_state to find
these "ghost" objects and who is trying to load then?


>
> Running a pack in dry-run mode performs verification without actually
> deleting anything.  There is no fsrecover-like script, but I could be
> convinced to create one if the need arises.
>
>
I don't know how dificult is, but I'm sure this will be a high value for now
and the future. :-)
For me it can be something like zodbconvert (moving from the same or another
Storage type) but doing these corrections when it find transactions or lod
object failure.

I try to use ZODB convert to moving from MySQL to FileStorage, but it
generates a failure, but I do that to discover the exactly transaction and
date time that generates other failure.

After that I correct the problem by creating  a "stub dict" pickle
object_state to be able to delete the "father" content becouse before that
if I try to list the folder contents it generates POSKeyError exception.

I expect not need to do this again, but know that happen again, could be a
server problem, and I have backup.
But how I can restore old database and apply all good objects created after
the corrupted transaction?

Anyway, thanks for your support.


> Shane
>


-- 
Rudá Porto Filgueiras
http://python-blog.blogspot.com
http://twitter.com/rudaporto
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.zope.org/pipermail/zodb-dev/attachments/20100130/1fb8558f/attachment-0001.html

More information about the ZODB-Dev mailing list