[Ape] APE issue

[Shane Hathaway]

On Tuesday 12 October 2004 02:21 am, Andrew Veitch wrote:
> I've been trying to migrate a 5GB ZODB onto APE/filesystem. It's worked
> fine on the whole but I've had a few issues:
>
> First, a dozen or so of files in the ZODB are quite large, 100MB-200MB. I
> know that neither ZODB nor APE are really designed for files of this size
> but it works fine under ZODB although it is a little slow sometimes.
> However under APE we frequently get 'out of memory' errors - both when
> creating and viewing the files. The server has 4GB of RAM plus swap space
> and I was the only user at the time so I am surprised that a 100MB file
> causes this error. Unfortunately there is no traceback in the log - I
> guess that running out of memory stops the logging.

I imagine there was some limit imposed on the process that caused it to fail 
long before reaching 4 GB.  Perhaps the kernel can not create a single 
process of such size.  Regardless, Ape needs to handle large files better.  
It actually has the potential to do an excellent job with large files, I 
think, but it currently punts.

Right now, Ape loads and saves all files as a big chunk.  We could make File 
objects read and write the filesystem directly when Ape is running.  The one 
complication is preserving transactions, but I hope that's not too 
complicated.

The first part of the code that needs to change somehow is the FilePData class 
in apelib.zope2.ofsserial.  It currently converts the "data" attribute 
between a chain of PData objects (see OFS.Image) and a simple string.  
FilePData could sneakily replace the "data" attribute with something of its 
own making--something that reads the data directly from a file.  Then it 
could pass the filename instead of the file data to and from the gateway.  Do 
you see what I mean?

> Secondly, I've occasionally had 'maximum recursion depth exceeded'
> messages when pasting content. This is when pasting 4 or 5 files.
> Traceback appended. This looks to me like a problem with Archetypes and
> the Pickler rather than APE per se. But again, it has never happened with
> the ZODB. Increasing Python's maximum recursion did resolve the issue.

This is probably the correct resolution.  Ape uses deeper pickles than 
standard ZODB.

Shane