[Zope] Large BTreeFolder2 batching/pagination

[Ken Ara]

I take your suggestion to find a Catalog-based solution. I have set the subtransactions very low (100).

But I have got the following error:

Traceback (innermost last):
  Module ZPublisher.Publish, line 119, in publish
  Module ZPublisher.mapply, line 88, in mapply
  Module ZPublisher.Publish, line 42, in call_object
  Module Products.ZCatalog.ZCatalog, line 348, in manage_catalogFoundItems
  Module Products.ZCatalog.ZCatalog, line 800, in ZopeFindAndApply
  Module Products.ZCatalog.ZCatalog, line 559, in catalog_object
  Module transaction._manager, line 110, in savepoint
  Module transaction._transaction, line 312, in savepoint
  Module transaction._transaction, line 309, in savepoint
  Module transaction._transaction, line 737, in __init__
  Module ZODB.Connection, line 1034, in savepoint
  Module ZODB.Connection, line 543, in _commit
  Module ZODB.Connection, line 571, in _store_objects
  Module ZODB.Connection, line 1168, in store
IOError: [Errno 28] No space left on device

I am guessing that this process is using the /tmp directory which unfortunately was incorrectly sized at only 1.9Gb. Can someone confirm this? 

Then either I must find a way to change the /tmp size, or is there a solution through Zope?

Thank you.

Ken


--- On Mon, 11/2/09, Andreas Jung <lists at zopyx.com> wrote:
>
> We have not been successful to create a Catalog of this many objects.
> The process seemed to time out after many hours, probably
> hardware-bound. Also, the size of the resulting ZODB is of concern,
> but we may try again with the Catalog on a mounted database.
>
we have ZCatalogs with millions of objects..likely you are indexing
everything within one big transaction
without using savepoints or subtransactions..


>
-aj