Seb,

I've replaced the following date indexes that werer FieldIndexes with DateIndexes

Date effective expires modified

i'll let you know what kind of speed improvement I get.

Jason Spisak wrote:

...

Zopistas,

My ZCatalog is fast as admin, but dog slow as anonymous and other users. Anyone had this same experience? Details:

This is certainly to do with queries which run against FieldIndexes only for anonymous users. This is something the CMF does (e.g. expires and effective dates only applied for anonymous).

FieldIndexes are really bad for DateTimes. Delete your date-related FieldIndexes and replace them with DateIndexes.

seb

Jason Spisak

Tres,

You are right. I've also got that permission turned off to take advantage of that part of the CMF workflow, which compounds the problem since it's site wide. I'm building the individual DateIndexs now and when that's done, I'll check the performance (for my own edification if nothing else 8-). Then I'll add that DateRange index and patch the CMF and check again.

Do you guys want metrics? If so, preferred tool/output method? (although my guess is, you already know how spiffy this makes things 8-)

Tres Seaver wrote:

Jason Spisak wrote:

...

Zopistas,

My ZCatalog is fast as admin, but dog slow as anonymous and other users. Anyone had this same experience? Details:

marjors:

Zope 2.6.2 CMF 1.3 Plone 1.0.5

about 50,000 cataloged objects (dual xenon server, plenty of ram, RAID)

User folder has 15k users in it, but admin is one of those users.

i've factored it way down and a simple search with just 1 Event loaded, like:

return context.portal_catalog(Type='Event')

is instantaneous for the 'admin' user, but takes about 10 seconds to return 1 SINGLE object. (only 1 Event loaded) imagine the time it takes to search and display 1300 events (which is what's normally in there). ouch. i've rebuilt the indexes, and still no dice.

Any help would be greatly appreciated.

This symptom probably has to do with the login in the CMF catalog which filters results based on the effective - expiration dates, for anybody without the "View inactive content" permission. I would guess that your portal_catalog is *not* using a DateRangeIndex to filter such content, but is still using the individual 'effective' and 'expires' indexes.

To fix this:

• Add a DateRangeIndex, 'effectiveRange', to your portal_catalog; set its start attribute to 'effective' and its stop attribute to

'expires'.

• Patch CMFCore/CatalogTool.py using the attached patch file (made against the released 1.3 version).

Tres.

---

Index: CMFCore/CatalogTool.py

RCS file: /cvs-repository/CMF/CMFCore/CatalogTool.py,v retrieving revision 1.30.4.6 diff -c -r1.30.4.6 CatalogTool.py *** CMFCore/CatalogTool.py 1 Aug 2002 19:07:55 -0000 1.30.4.6 --- CMFCore/CatalogTool.py 30 Jan 2004 18:16:37 -0000

---

*** 12,18 **** ############################################################################## """ Basic portal catalog.

! $Id$ """

import os --- 12,18 ---- ############################################################################## """ Basic portal catalog.

! $Id: CatalogTool.py,v 1.30.4.6 2002/08/01 19:07:55 tseaver Exp $ """

import os

---

*** 202,215 **** if not _checkPermission( CMFCorePermissions.AccessInactivePortalContent, self ): base = aq_base( self ) ! now = DateTime() ! if hasattr( base, 'addIndex' ): # Zope 2.4 and above ! kw[ 'effective' ] = { 'query' : now, 'range' : 'max' } ! kw[ 'expires' ] = { 'query' : now, 'range' : 'min' } ! else: # Zope 2.3 ! kw[ 'effective' ] = kw[ 'expires' ] = now ! kw[ 'effective_usage'] = 'range:max' ! kw[ 'expires_usage' ] = 'range:min'

      return apply(ZCatalog.searchResults, (self, REQUEST), kw)

--- 202,208 ---- if not _checkPermission( CMFCorePermissions.AccessInactivePortalContent, self ): base = aq_base( self ) ! kw[ 'effectiveRange' ] = DateTime()

      return apply(ZCatalog.searchResults, (self, REQUEST), kw)

---

---

Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )

Consider me awaiting my socks to be blown off. 8-)

Tres Seaver wrote:

Jason Spisak wrote:

...

You are right. I've also got that permission turned off to take advantage of that part of the CMF workflow, which compounds the problem since it's site wide. I'm building the individual DateIndexs now and when that's done, I'll check the performance (for my own edification if nothing else 8-). Then I'll add that DateRange index and patch the CMF and check again.

Do you guys want metrics? If so, preferred tool/output method? (although my guess is, you already know how spiffy this makes things 8-)

While *I* have a gut feel for what the result should be ("using DateIndexes will ease most of your pain; using a DateRange index will blow your socks off"), it would be good to confirm that I'm right (or not :) in public.

Tres.

Jason Spisak wrote:

Tres and Seb,

Got rid of all the date FieldIndexes and am reindexing the new DateIndexes one at a time from the indexes tab, but I've hit a snag.

when I go to reindex 'expires' I get this error immediately:

Site error

This site encountered an error trying to fulfill your request. The errors were: Error Details

Error Type OverflowError

Error Value integer multiplication

Update lib/python/Products/PluginIndexesn/DateIndex/DateIndex.py to the current head of the 2.6 branch, e.g. from:

As a quick workaround, you could patch CMFDefault/DublinCore.py::

---------------- 8< ----------------------------------- *** CMFDefault/DublinCore.py 14 Nov 2002 06:48:20 -0000 1.19.4.3 --- CMFDefault/DublinCore.py 30 Jan 2004 22:22:53 -0000 *************** *** 253,259 **** date = getattr( self, 'creation_date', None ) return date is None and self.__FLOOR_DATE or date

! __CEILING_DATE = DateTime( 9999, 0 ) # never expires

security.declarePublic( 'expires' ) def expires( self ): --- 253,259 ---- date = getattr( self, 'creation_date', None ) return date is None and self.__FLOOR_DATE or date

! __CEILING_DATE = DateTime( 2037, 0 ) # never expires

security.declarePublic( 'expires' ) def expires( self ): ---------------- 8< -----------------------------------

The better fix would involve not returning the silly _CEILING_DATE at all, and then telling the index whether to treat 'None' as low or high.

Tres.

Tres,

I updated the DateIndex.py file to the one from the current 2.6 head. But I'm still getting the 'integer multiplication' error. Is there another possibility? Also, it only does this for the 'expires' index. The 'effective' index reindexed just fine.

Thanks in advance,

Jason Spisak

Tres Seaver wrote:

Jason Spisak wrote:

...

Tres and Seb,

Got rid of all the date FieldIndexes and am reindexing the new DateIndexes one at a time from the indexes tab, but I've hit a snag.

when I go to reindex 'expires' I get this error immediately:

Site error

This site encountered an error trying to fulfill your request. The errors were: Error Details

Error Type OverflowError

Error Value integer multiplication

Update lib/python/Products/PluginIndexesn/DateIndex/DateIndex.py to the current head of the 2.6 branch, e.g. from:

As a quick workaround, you could patch CMFDefault/DublinCore.py::

---------------- 8< ----------------------------------- *** CMFDefault/DublinCore.py 14 Nov 2002 06:48:20 -0000 1.19.4.3 --- CMFDefault/DublinCore.py 30 Jan 2004 22:22:53 -0000

---

*** 253,259 **** date = getattr( self, 'creation_date', None ) return date is None and self.__FLOOR_DATE or date

! __CEILING_DATE = DateTime( 9999, 0 ) # never expires

  security.declarePublic( 'expires' )
  def expires( self ):

--- 253,259 ---- date = getattr( self, 'creation_date', None ) return date is None and self.__FLOOR_DATE or date

! __CEILING_DATE = DateTime( 2037, 0 ) # never expires

  security.declarePublic( 'expires' )
  def expires( self ):

---------------- 8< -----------------------------------

The better fix would involve not returning the silly _CEILING_DATE at all, and then telling the index whether to treat 'None' as low or high.

Tres.

<I didn't see this come through from the list, so I'm doing a quick resend.>

Tres and Seb,

Got rid of all the date FieldIndexes and am reindexing the new DateIndexes one at a time from the indexes tab, but I've hit a snag.

when I go to reindex 'expires' I get this error immediately:

Site error

This site encountered an error trying to fulfill your request. The errors were: Error Details

Error Type OverflowError

Error Value integer multiplication

any thoughts on how to get past this?

Jason Spisak wrote:

...

You are right. I've also got that permission turned off to take advantage of that part of the CMF workflow, which compounds the problem since it's site wide. I'm building the individual DateIndexs now and when that's done, I'll check the performance (for my own edification if nothing else 8-). Then I'll add that DateRange index and patch the CMF and check again.

Do you guys want metrics? If so, preferred tool/output method? (although my guess is, you already know how spiffy this makes things 8-)

While *I* have a gut feel for what the result should be ("using DateIndexes will ease most of your pain; using a DateRange index will blow your socks off"), it would be good to confirm that I'm right (or not :) in public.

Tres.

Toby Gustafson wrote:

Hi,

Within a python file I do a call to thread.start_new_thread(...). Before this call, I am the admin user (verified by calling AccessControl.getSecurityManager().getuser().getUserName()). After the call, however, in the new thread, the user is now "Anonymous User".

Is there any way to somehow "pass" the user between threads because the code executed in the new thread needs the same permissions as the code executed in the old thread? I can obviously pass the user name to the new thread, but I'm not sure if that does me any good, because I doubt an Anonymous User would have permission to make itself another user.

Any hints or suggestions would be appreciated.

Given that you trust yourself ;), you can add a security context from within the second thread; you could pass the user ID to the thread via one of several forms of "currying", e.g. via instance variables::

class TrustedSecurityTask:

def __init__(self, user_id): self._user_id = user_id

def __call__(self, *args, **kw): sm = getSecurityManager() sm.newSecurityContext(None, User(self._user_id)) # your code here .....

thread = Thread(TrustedSecurityTask(user_id)) thread.start()

Tres.

Toby Gustafson wrote:

Your code below did not work for me because I could not find the newSecurityContext method. However, I was able to use the SecurityManagement.newSecurityManager instead, and that worked great. Thanks for pointing me in the right direction.

Great! Sorry for leaving off the <untested>...</untested> brackets, and for typing code from memory after an all-day "planes, trains, and automobiles" trip.

Tres.