[Zope] Data.fs falls into 33GB hole

[sean.upton@uniontrib.com]

Hmm... I heard somewhere that packing a filestorage used memory proportional
to the storage size.  I would consider posting to the ZODB-dev list, perhaps
for some more pointers on how to get out of this?

Sean

-----Original Message-----
From: Tino Wildenhain [mailto:tino@wildenhain.de]
Sent: Thursday, November 14, 2002 5:15 PM
To: Howard Hansen; zope@zope.org
Subject: Re: [Zope] Data.fs falls into 33GB hole


Hi Howard,

why fsrecover? Your Data.fs is big but should not be
corrupt. I would even suspect if you simply pack
it, it should go to around 360-400MB again.
Zope packs by renaming it and then copys the most
recent objects over. So you only need the space
for actual Data.fs + Data.fs after packing.
Your last backup was at 360MB, so the packet Version
schould be at this level.
Since you have a backup on your Windows box, you dont
have to worry, I'd say.

HTH
Tino Wildenhain

--On Donnerstag, 14. November 2002 13:36 -0800 Howard Hansen 
<howardahansen@yahoo.com> wrote:

> I have several, fairly-large sites running on my
> machine and at the last backup, my Data.fs file stood
> at 340MB.  Zope itself only took up 50MB of RAM, so I
> didn't worry about the size of the file.  Disk space
> is essentially infinite nowadays, right?
>
> Yes, but imagine my shock when I checked out why my
> backup routine took a long time and then failed.  I
> found that Data.fs took up 33GB of space on the disk!
> Yikes!
>
> I must step back for a moment and salute everyone who
> made this possible.  Everyone at Zope, Linus, Guido,
> Maxtor, AMD, Intel, and the manufacturers of all of
> the little components who have conspired to make a
> sticky mess like this possible.
>
> My Data.fs file grew to 33GB and I DIDN'T NOTICE.
>
> Looking over the Undo lists, I found hundreds of
> transactions that said simply "by Zope".  Some
> experimentation led me to discover that these were
> inserted by JRedirector.  It summarizes redirections,
> and my assumption that it did it under the Undo radar
> proved incorrect.  I've since disabled JRedirector.
>
> Running df on my disk, I find that I've filled 74% of
> the 55GB data partition, so there's no chance of
> making a copy locally and figuring it out from there.
> Serendipidously, I did have a 60GB drive literally
> sitting on the shelf.  I popped it into my Windows PC
> (I'd been able to get fsrecover.py to run on it, but
> not on the Linux box) started copying over the network
>  This took a couple of hours.  I tried to run
> fsrecover on the big file and it failed immediately:
>
>> python.exe fsrecover.py -v 2 -P 184000 j:\Data.fs
> j:\Data_recovered_all.fs
> Recovering j:\Data.fs into j:\Data_recovered_all.fs
> Traceback (most recent call last):
>   File "fsrecover.py", line 328, in ?
>     if __name__=='__main__': recover()
>   File "fsrecover.py", line 221, in recover
>     seek(0,2)
> IOError: [Errno 22] Invalid argument
>
> I guess there are indeed some limits.  I wrote a
> quick program to make a copy of the first
> 1,000,000,000 bytes of the file.  I ran fsrecover on
> that without problems.  This tells me that the file
> itself is OK.  It just needs to be packed or filtered.
>
> I've got thousands of good transactions interspersed
> with tens of thousands of noise transactions.
>
> Questions:
>
> 1) Can I safely pack a database this huge on my live
> machine?
> 2) If that doesn't work, can I easily figure out
> transaction boundaries?  If so, I could either chop
> the big file up into smaller chunks and serially
> attach new tails to the file, then fsrecover the whole
> thing.
> 3) If I can figure out transaction boundaries, can I
> filter out transactions by type?  I've got
> approximately 32GB of JRedirector transactions that
> I'd be happy to remove.
>
> As always, thanks for listening and I eagerly await
> your wisdom.
>
>
> Howard Hansen
> http://howardsmusings.com
>
>
> __________________________________________________
> Do you Yahoo!?
> Yahoo! Web Hosting - Let the expert host your site
> http://webhosting.yahoo.com
>
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )



_______________________________________________
Zope maillist  -  Zope@zope.org
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )