[Zope] Advice on Blob Storage?

[Michael McFadden]

On 09/24/2015 08:47 AM, Jean Jordaan wrote:
>
> If relstorage is growing for blob uploads, I would think something is
> wrongly configured.
I'm really thinking the same thing myself, but I wouldn't know the first 
place to look to configure this.

This installation was a Plone 2 archetypes-based build that was upgraded 
to Plone 4, and that's when the Relstorage changeover was done.  I don't 
know if that gives any hints.

We have so much archetypes content with specialized code that we've 
stuck with archetypes.  I have a feeling that archetypes is tightly 
coupled with filestorage somehow.

>> Can this behavior be turned off for a specific field or content type?  So
>> undo logs are preserved for everything BUT this monster of a content type?
>>
>> Seems strange to do this tho.
> Yes, that seems like a plaster on top of a broken bone.
I hope we don't go down that path.

>
>> Going deeper down the rabbit hole, although I don't think it's relevant, is
>> the fact that I hacked and replaced the storage class for the field.
>> Instead of using AnnotationStorage
> This sounds dangerous to me ..
It's actually working perfectly, and was the original intent.   When I 
did some quick maths to show how much blob storage would grow based on 
how much video content we create, it became cost-preventative to store 
the videos in blob storage.   The subclassed AnnotationStorage works.   
However, I'll be looking into collective.xsendfile to see if I can make 
things a bit better.

In a nutshell, Blob Storage is happy - the data are stored elsewhere 
happily.  The upshot is that you cannot fetch the file from Plone, and 
that's just fine for now.  If you do fetch it through plone's download, 
you get about 80 bytes that say "your file is not here"

The fact that relstorage grows with the upload (and shrinks back with 
the pack) is what's troubling.

My spelunking (which I enjoy) has gone deep enough to confirm that the 
relstorage growth happens with the transaction's tcp_finish() call, and 
I haven't gone deeper yet.  What's strange is that the data of the File 
field has been replaced by then (in the dangerous manner mentioned 
above) and I'm not sure where it's finding it.

All the tom foolery can be seen on github where I replace the field 
storage for the file field.

Thanks for the reply.

-- 
Mike McFadden
Radio Free Asia
Technical Operations Division
2025 M Street NW
Washington DC 20036 USA

This e-mail message is intended only for the use of the addressee and may contain information that is privileged and confidential.  Any unauthorized dissemination, distribution or copying is strictly prohibited.  If you receive this transmission in error, please contact network at rfa.org.