[Zope] Advice on Blob Storage?

Michael McFadden mcfaddenm at rfa.org
Mon Sep 21 18:24:45 CEST 2015


This may be more of a zodb / relstorage question - I hope it's ok to ask 
on the Zope list.

I'm seeing behavior using relstorage and blobs that I didn't expect:
    If I upload a large file, say 2 gigs, I am noticing that our SQL 
database also grows by 2 Gigs, along with the blob storage.
   After a pack, the space is reclaimed on the SQL side, and everyone is 
happy.
   FWIW - it's videos that are doing this.

I am pretty sure it's the undo log that's growing, based on the fact 
that a pack reclaims the space.

Can this behavior be turned off for a specific field or content type?  
So undo logs are preserved for everything BUT this monster of a content 
type?

Seems strange to do this tho.

Are there other alternatives, like calling .pack() directly on the 
field's storage after it's set?

Our problem is that our sql database grows to a huge size between our 
weekly packs, and backups of the sql dumps are becoming unmanageable.  
Our blob backups are ready to deal with this kind of size, but not the 
sql backups.

----------
Going deeper down the rabbit hole, although I don't think it's relevant, 
is the fact that I hacked and replaced the storage class for the field.
Instead of using AnnotationStorage - which I found used as default for 
ImageField - I intercept the data during storage.set(), ship it out to a 
separate storage facility, and replace the data with a happy message 
"This is not where your data is" which is then written to the blobs.   
It works just great - keeping our blob storage growth from going 
crazy.    If you try to 'download' the file from Plone, you'll get the 
text file with the happy message.

Now that I've been shown that the Blob Storage is functioning just fine, 
but the SQL storage size is going off the charts, I hope I'm not back at 
square one.

The goal is to allow users to think they are uploading 4Gb videos into 
Plone, when under the covers, we're actually shipping the video files 
off to some fancy off-site storage. (Akamai)  So we don't have to store 
them and back them up on-site, and our blob directories remain 
manageable in size.

The storage hack can be seen here: 
https://github.com/RadioFreeAsia/rfa.kaltura/blob/master/rfa/kaltura/storage/storage.py 


I'm not proud of it, but it works.


-- 
Mike McFadden
Radio Free Asia
Technical Operations Division
2025 M Street NW
Washington DC 20036 USA

This e-mail message is intended only for the use of the addressee and may contain information that is privileged and confidential.  Any unauthorized dissemination, distribution or copying is strictly prohibited.  If you receive this transmission in error, please contact network at rfa.org.




More information about the Zope mailing list