[Zope] Large TextIndexNG problem

Andreas Jung Andreas Jung <lists@andreas-jung.com>
Mon, 09 Dec 2002 06:35:12 +0100


The largest installation with TextIndexNG running I have seen was about
70,000 documents large. It ran on a fast Athlon machine with 256 MB.
I assume you are running into some kind of memory problem. I suggest
to run "top" when you are trying to index the documents. Watch
the memory usage of the running Zope process and checkout your swap
process. Also you can try to play around with the number of subtransactions
uses. I have seen a memory explosion on a solaris system some
time ago but never seen such a memory problem under Linux i386.

If you have further problems please contact me again.

Andreas


--On Sonntag, 8. Dezember 2002 17:26 -0600 David Mackay 
<mackay_d@bellsouth.net> wrote:

> I am having a problem creating an index for about 276,000 documents.  I
> successfully created a TextIndexNG index for the documents by indexing
> 50,000 at a time.  If I tried to do everything at one time, it would work
> for about six or seven hours, and then return with zer0 items indexed.
> Before I added the indexed items, my database size was about 360MB.
> Well, I decided that I needed to make a few changes, so I cleared the
> index.  Then I packed the database.  However, the database size is now
> 2023.7M.  When I try to repack the database, I get an error message
> telling me that there have been no changes since the last pack.   Also, I
> can't add any indices.  I get a message that the connection was refused
> when attempting to contact the server.  There are no messages in Z2.log
> indicating that there is a problem, and no tracebacks from python.  The
> environment is: Zope 2.5.1
> Python 2.1.3
> Linux (Red Hat 8.0) kernel 2.4.18-3
> The machine has 768MB of memory and 11GB of free hard drive space.
>
> Can anyone give me some pointers?
>
> Dave
>
>
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )




    ---------------------------------------------------------------------
   -    Andreas Jung                     http://www.andreas-jung.com   -
  -   EMail: andreas at andreas-jung.com                              -
   -            "Life is too short to (re)write parsers"               -
    ---------------------------------------------------------------------