[Ape] APE issue

Shane Hathaway shane at hathawaymix.org
Thu Oct 21 04:42:29 EDT 2004


On Tuesday 12 October 2004 02:21 am, Andrew Veitch wrote:
> I've been trying to migrate a 5GB ZODB onto APE/filesystem. It's worked
> fine on the whole but I've had a few issues:
>
> First, a dozen or so of files in the ZODB are quite large, 100MB-200MB. I
> know that neither ZODB nor APE are really designed for files of this size
> but it works fine under ZODB although it is a little slow sometimes.
> However under APE we frequently get 'out of memory' errors - both when
> creating and viewing the files. The server has 4GB of RAM plus swap space
> and I was the only user at the time so I am surprised that a 100MB file
> causes this error. Unfortunately there is no traceback in the log - I
> guess that running out of memory stops the logging.

I imagine there was some limit imposed on the process that caused it to fail 
long before reaching 4 GB.  Perhaps the kernel can not create a single 
process of such size.  Regardless, Ape needs to handle large files better.  
It actually has the potential to do an excellent job with large files, I 
think, but it currently punts.

Right now, Ape loads and saves all files as a big chunk.  We could make File 
objects read and write the filesystem directly when Ape is running.  The one 
complication is preserving transactions, but I hope that's not too 
complicated.

The first part of the code that needs to change somehow is the FilePData class 
in apelib.zope2.ofsserial.  It currently converts the "data" attribute 
between a chain of PData objects (see OFS.Image) and a simple string.  
FilePData could sneakily replace the "data" attribute with something of its 
own making--something that reads the data directly from a file.  Then it 
could pass the filename instead of the file data to and from the gateway.  Do 
you see what I mean?

> Secondly, I've occasionally had 'maximum recursion depth exceeded'
> messages when pasting content. This is when pasting 4 or 5 files.
> Traceback appended. This looks to me like a problem with Archetypes and
> the Pickler rather than APE per se. But again, it has never happened with
> the ZODB. Increasing Python's maximum recursion did resolve the issue.

This is probably the correct resolution.  Ape uses deeper pickles than 
standard ZODB.

Shane


More information about the Ape mailing list