Timing the opening of network ports
Some time ago the following patch was proposed to control the timing for opening the socket ports upon startup: <http://mail.zope.org/pipermail/zope-dev/2005-November/025709.html> I propose to adapt the patch for Zope 2.11 since it is pretty useful especially when you deal with loadbalancers and have Zope instances that take a lot time for starting up (e.g. Plone :-)). Any objections? Andreas
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 28 Aug 2006, at 19:19, Andreas Jung wrote:
Some time ago the following patch was proposed to control the timing for opening the socket ports upon startup:
<http://mail.zope.org/pipermail/zope-dev/2005-November/025709.html>
I propose to adapt the patch for Zope 2.11 since it is pretty useful especially when you deal with loadbalancers and have Zope instances that take a lot time for starting up (e.g. Plone :-)). Any objections?
+1 If you have load balancers that only look for open ports (or a single instance with no load balancers even) this will improve the user experience for people who hit the site upon startup. IMHO it's more helpful to get a "connection refused" reply in a browser than a browser that sits and sits and sites... jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (Darwin) iD8DBQFE8zJrRAx5nvEhZLIRAqUTAJ4hPrqildFLXLg+vdYIA0F26J/o1gCgpBZT bvhVObipCKUessLOC1ANc/Q= =D5Fx -----END PGP SIGNATURE-----
Jens Vagelpohl wrote at 2006-8-28 20:14 +0200:
On 28 Aug 2006, at 19:19, Andreas Jung wrote:
Some time ago the following patch was proposed to control the timing for opening the socket ports upon startup:
<http://mail.zope.org/pipermail/zope-dev/2005-November/025709.html>
I propose to adapt the patch for Zope 2.11 since it is pretty useful especially when you deal with loadbalancers and have Zope instances that take a lot time for starting up (e.g. Plone :-)). Any objections?
+1
If you have load balancers that only look for open ports (or a single instance with no load balancers even) this will improve the user experience for people who hit the site upon startup. IMHO it's more helpful to get a "connection refused" reply in a browser than a browser that sits and sits and sites...
Can we make this configurable? Earlier Zope versions had bound the ports rather late. When later Zope versions bound the ports early, I was quite happy. Unlike you, I prefer when the browser waits until Zope has come up over me having to reload manually until it finally is ready... -- Dieter
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dieter Maurer wrote:
Jens Vagelpohl wrote at 2006-8-28 20:14 +0200:
On 28 Aug 2006, at 19:19, Andreas Jung wrote:
Some time ago the following patch was proposed to control the timing for opening the socket ports upon startup:
<http://mail.zope.org/pipermail/zope-dev/2005-November/025709.html>
I propose to adapt the patch for Zope 2.11 since it is pretty useful especially when you deal with loadbalancers and have Zope instances that take a lot time for starting up (e.g. Plone :-)). Any objections? +1
If you have load balancers that only look for open ports (or a single instance with no load balancers even) this will improve the user experience for people who hit the site upon startup. IMHO it's more helpful to get a "connection refused" reply in a browser than a browser that sits and sits and sites...
Can we make this configurable?
Earlier Zope versions had bound the ports rather late. When later Zope versions bound the ports early, I was quite happy.
Unlike you, I prefer when the browser waits until Zope has come up over me having to reload manually until it finally is ready...
The branch Andreas just merged leaves the "fast-bind" option on by default. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFE9HYS+gerLs4ltQ4RAtfnAJwMe8G1irFvF/hktI735zLfmgq6EwCeIMHb 0YIFT47ydWAHIQLOKxIFdw0= =z0U9 -----END PGP SIGNATURE-----
Tres Seaver wrote:
Unlike you, I prefer when the browser waits until Zope has come up over me having to reload manually until it finally is ready...
The branch Andreas just merged leaves the "fast-bind" option on by default.
Cool, although I can't really see the use ;-) All this does is let app server instances get creamed at restart if the load is high... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote:
Tres Seaver wrote:
Unlike you, I prefer when the browser waits until Zope has come up over me having to reload manually until it finally is ready...
The branch Andreas just merged leaves the "fast-bind" option on by default.
Cool, although I can't really see the use ;-)
All this does is let app server instances get creamed at restart if the load is high...
a) it's the current behaviour b) it's more convenient for developers c) it's a good thing if you have 'smart' load balancers -- gocept gmbh & co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - ct@gocept.com - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development
Christian Theune wrote:
b) it's more convenient for developers
Why?
c) it's a good thing if you have 'smart' load balancers
How so? Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote:
Christian Theune wrote:
b) it's more convenient for developers
Why?
Early open port means: zopectl restart and reload in your browser immediately without getting "Connection refused". Dieter already mentioned this use case
c) it's a good thing if you have 'smart' load balancers
How so?
Tres mentioned that to me: 'smart' load balancers can probe the server and decide that the port is there, but the response time is too high and won't include it in the pool again yet, but they can know that the port is already open again. Christian -- gocept gmbh & co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - ct@gocept.com - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 30 Aug 2006, at 14:32, Christian Theune wrote:
Chris Withers wrote:
Christian Theune wrote:
b) it's more convenient for developers
Why?
Early open port means: zopectl restart and reload in your browser immediately without getting "Connection refused". Dieter already mentioned this use case
That's a matter of taste IMHO. Everyone handles that differently doing development. I always do "zopectl fg" and never hit "reload" until the console tells me Zope is ready. Let's not discuss this any further, taste knows no right or wrong ;)
c) it's a good thing if you have 'smart' load balancers How so?
Tres mentioned that to me: 'smart' load balancers can probe the server and decide that the port is there, but the response time is too high and won't include it in the pool again yet, but they can know that the port is already open again.
I don't quite get this. I'd have said "smart load balancers can work around the fact that Zope opens ports way before it can start serving". So in essence, the "smartness" papers over Zope's early port opening issue. I'd never say "Zope opening ports too early is a good thing for smart load balancers"... Besides, there are tons of stupid load balancers out there who only look at open ports. But just like the argument above, this seems to boil down to personal preference. The setting is configurable, let's stop beating that old mare ;) jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (Darwin) iD8DBQFE9YebRAx5nvEhZLIRAh5PAJ4s9ij4ssii3PbQLBDQFGEkwHR89wCdHsDJ C0aoOvlUC7vbcEuk1spKx2Q= =msv1 -----END PGP SIGNATURE-----
Christian Theune wrote:
b) it's more convenient for developers
Why?
Early open port means: zopectl restart and reload in your browser immediately without getting "Connection refused". Dieter already mentioned this use case
I don't really buy that, but since it's configurable, it doesn't matter...
c) it's a good thing if you have 'smart' load balancers
How so?
Tres mentioned that to me: 'smart' load balancers can probe the server and decide that the port is there, but the response time is too high and won't include it in the pool again yet, but they can know that the port is already open again.
Again, I can only see this as being a bad thing: LB: Hmmm, this one's alive but absolutely creamed, I better not send it any requests for a long time. D'oh Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
--On 29. August 2006 18:37:45 +0100 Chris Withers <chris@simplistix.co.uk> wrote:
Tres Seaver wrote:
Unlike you, I prefer when the browser waits until Zope has come up over me having to reload manually until it finally is ready...
The branch Andreas just merged leaves the "fast-bind" option on by default.
Cool, although I can't really see the use ;-)
The usecase is pretty simple: you have a loadbalancer and remove one backend Zope. The LB detects the removal and stops forwarding request. When the client comes back (means Zope opens the ports early) the LB will start forwarding to the client although it might take a while until the Zope instance is really up and ready to serve request. So by opening the ports late we are able to reduce the latency and avoid pending requests. -aj
Andreas Jung wrote:
The usecase is pretty simple: you have a loadbalancer and remove one backend Zope. The LB detects the removal and stops forwarding request. When the client comes back (means Zope opens the ports early) the LB will start forwarding to the client although it might take a while until the Zope instance is really up and ready to serve request. So by opening the ports late we are able to reduce the latency and avoid pending requests.
Er yes, I know why this is a very good thing. I was commenting that I can't see why anyone would want anything else ;-) Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote at 2006-8-31 08:15 +0100:
... Er yes, I know why this is a very good thing. I was commenting that I can't see why anyone would want anything else ;-)
You did not read the thread carefully: I explained why I prefer early port binding... -- Dieter
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Andreas Jung wrote:
--On 29. August 2006 18:37:45 +0100 Chris Withers <chris@simplistix.co.uk> wrote:
Tres Seaver wrote:
Unlike you, I prefer when the browser waits until Zope has come up over me having to reload manually until it finally is ready...
The branch Andreas just merged leaves the "fast-bind" option on by default.
Cool, although I can't really see the use ;-)
The usecase is pretty simple: you have a loadbalancer and remove one backend Zope. The LB detects the removal and stops forwarding request. When the client comes back (means Zope opens the ports early) the LB will start forwarding to the client although it might take a while until the Zope instance is really up and ready to serve request. So by opening the ports late we are able to reduce the latency and avoid pending requests.
The other reason for wanting "early binding" to the ports is if the ports are in the "reserved for root" range (< 1024); in that case, the ports *must* be bound early, before dropping privileges to those of the "effective user". Tres - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFE9ucV+gerLs4ltQ4RAt+9AJkBLjRKmZaRQWPq2iiKIRAZy+W2hwCfTeW/ FaLZGLsRx8J4JJHNPtnJxy4= =mir0 -----END PGP SIGNATURE-----
On Aug 31, 2006, at 9:41 AM, Tres Seaver wrote:
The other reason for wanting "early binding" to the ports is if the ports are in the "reserved for root" range (< 1024); in that case, the ports *must* be bound early, before dropping privileges to those of the "effective user".
Not necessarily. The basic sequence for opening a server socket connection is: listener = socket(AF_INET,SOCK_STREAM,0) bind(listener, address_info) listen(listener, LISTEN_QUEUE_LENGTH) For unix and unix-like systems that reserve some range of ports for privileged users, it is the bind() call that has the user check. It is at the listen() call where the port is seen as open and available for connections. It might be that Zope's HTTPServer class bunches all of them in a single method for convenience, but it is possible to bind() early, drop privs, and listen() late.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Andrew Langmead wrote:
On Aug 31, 2006, at 9:41 AM, Tres Seaver wrote:
The other reason for wanting "early binding" to the ports is if the ports are in the "reserved for root" range (< 1024); in that case, the ports *must* be bound early, before dropping privileges to those of the "effective user".
Not necessarily.
The basic sequence for opening a server socket connection is:
listener = socket(AF_INET,SOCK_STREAM,0) bind(listener, address_info) listen(listener, LISTEN_QUEUE_LENGTH)
For unix and unix-like systems that reserve some range of ports for privileged users, it is the bind() call that has the user check. It is at the listen() call where the port is seen as open and available for connections.
It might be that Zope's HTTPServer class bunches all of them in a single method for convenience, but it is possible to bind() early, drop privs, and listen() late.
Thanks, I'd forgotten that distinction. ZServer does indeed hardwire the two of them together, and it looks like it would be hard to split it apart. Andreas' hack elegant as it is, seems adequate for the purpose. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFE925G+gerLs4ltQ4RAmezAJ49ikaCgveaSPcLhug3NHVS0MemUgCfdzlm aswciRwTad8heeTWnaThItk= =KTDx -----END PGP SIGNATURE-----
I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog. Here are the clues: 1. When I re-index the catalog, everything works fine. Unfortunately, I cannot re-index the catalog very frequently because it takes a long time 2. The objects that are indexed are "Citation" objects, a class I defined. 3. The "Citation" class extends CatalogAware (as well as Item,Persistent,Implicit, PropertyManager) 4. Every time I change a citation, I call "reindex_object()" (on the citation) 5. When I add a citation, I call reindex_object() on the citation immediately. 6. I'm using "Zope 2.8.4-final", built from source. 7. I'm using python 2.3.5 Thank you in advance. I can provide more information upon request. Best, Brian Rosenthal -------------------- Brian Rosenthal General Partner, RoboCommerce, LLC brian@robocommerce.com http://www.robocommerce.com 800-644-7626
--On 1. September 2006 02:54:26 -0400 zope-dev@rosenthaltechnologies.com wrote:
I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog.
Read or Write conflict errors? Are the objects modified at the same time by other threads? -aj
Read or Write conflict errors?
I'm getting the "Key Error" traceback on Reads, but my guess is that the actual "error" happens sometime before the "Key Error", when the ZCatalog itself is corrupted. I am not sure what that is, because it does not generate an error. However, I see the error when I call ZCatalogInstance.Search(...), and only when the Search includes objects that are corrupted in the ZCatalog.
Are the objects modified at the same time by other threads?
I'm not sure about this. I'm running ZEO, and in general, I think I can cause the error when I am the only user on the system, only issuing one request. However, I'm getting another ConflictError when I try to set an attribute value of a ZODB object during the execution of a script, and I'm told that may be because there is a problem accessing the same object from multiple threads. Is there a good article that you know of that I could find out more about thread-safe programming in ZEO? Best, Brian On Fri, 01 Sep 2006 09:27:41 +0200, "Andreas Jung" <lists@zopyx.com> said:
--On 1. September 2006 02:54:26 -0400 zope-dev@rosenthaltechnologies.com wrote:
I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog.
Read or Write conflict errors? Are the objects modified at the same time by other threads?
-aj
-------------------- Brian Rosenthal General Partner, RoboCommerce, LLC brian@robocommerce.com http://www.robocommerce.com 800-644-7626
zope-dev@rosenthaltechnologies.com wrote at 2006-9-1 02:54 -0400:
I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog.
Here are the clues:
You forget to provide us with an essential glue: The complete error information (error type, error value and traceback). You find it in your "error_log" object (in Zope's "Root Folder"). -- Dieter
Traceback for my Key Error (Dieter, thank you for alerting me on where to find this): Traceback (innermost last): Module ZPublisher.Publish, line 113, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 40, in call_object Module Shared.DC.Scripts.Bindings, line 311, in __call__ Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec Module Products.PythonScripts.PythonScript, line 323, in _exec Module None, line 11, in to_javascript - <PythonScript at /nephron_links/to_javascript used for /nephron_links/news_dtml> - Line 11 Module OFS.DTMLDocument, line 128, in __call__ - <DTMLDocument at /nephron_links/news_dtml> - URL: http://nephron.org/news_dtml/manage_main - Physical Path: /nephron_links/news_dtml Module DocumentTemplate.DT_String, line 476, in __call__ Module DocumentTemplate.DT_In, line 465, in renderwb Module DocumentTemplate.DT_In, line 747, in sort_sequence Module Products.ZCatalog.Lazy, line 158, in __getitem__ Module Products.ZCatalog.Catalog, line 520, in getScoredResult KeyError: -464571725 On Fri, 1 Sep 2006 19:11:12 +0200, "Dieter Maurer" <dieter@handshake.de> said:
zope-dev@rosenthaltechnologies.com wrote at 2006-9-1 02:54 -0400:
I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog.
Here are the clues:
You forget to provide us with an essential glue:
The complete error information (error type, error value and traceback). You find it in your "error_log" object (in Zope's "Root Folder").
-- Dieter
Brian Rosenthal General Partner, RoboCommerce, LLC brian@robocommerce.com http://www.robocommerce.com 800-644-7626
zope-dev@rosenthaltechnologies.com wrote at 2006-9-1 13:29 -0400:
Traceback for my Key Error (Dieter, thank you for alerting me on where to find this):
Traceback (innermost last): ... - Physical Path: /nephron_links/news_dtml Module DocumentTemplate.DT_String, line 476, in __call__ Module DocumentTemplate.DT_In, line 465, in renderwb Module DocumentTemplate.DT_In, line 747, in sort_sequence Module Products.ZCatalog.Lazy, line 158, in __getitem__ Module Products.ZCatalog.Catalog, line 520, in getScoredResult KeyError: -464571725
This does not look like a catalog corruption. Instead, it looks as if the catalog changed since the search was done. Are you caching your search results? -- Dieter
Dieter Maurer wrote at 2006-9-3 19:22 +0200:
zope-dev@rosenthaltechnologies.com wrote at 2006-9-1 13:29 -0400:
Traceback for my Key Error (Dieter, thank you for alerting me on where to find this):
Traceback (innermost last): ... - Physical Path: /nephron_links/news_dtml Module DocumentTemplate.DT_String, line 476, in __call__ Module DocumentTemplate.DT_In, line 465, in renderwb Module DocumentTemplate.DT_In, line 747, in sort_sequence Module Products.ZCatalog.Lazy, line 158, in __getitem__ Module Products.ZCatalog.Catalog, line 520, in getScoredResult KeyError: -464571725
This does not look like a catalog corruption.
Instead, it looks as if the catalog changed since the search was done. Are you caching your search results?
Sorry! I was wrong! It has nothing to do with a catalog modification after the search. Looks indeed as if an index reported a document which is no longer indexed. Looks like an index bug. Which (types of) indexes are involved in your failing queries? -- Dieter
Thank you so much for following-up with me, here. I think you're right about this having to do with the indexes on the ZCatalog. The catalog has: 1. Seven "FieldIndex" indexes. 2. A "DateIndex" index 3. Three "ZCTextIndex" indexes I actually create these from python in a function that accepts the "catalog" as a parameter during the Product Initialization process. I have a function "catalog_add" that creates and initializes the a ZCatalog when this product is created and initialized, so let me actually post the sourcecode here for that function and the functions it calls. It's fairly short. There are two functions of note "catalog_add" and "addIndexes". I'd actually be grateful if you would take a look and let me know if you see anything. In particular, am I using an index that you know to have problems, or that I'm using improperly. class Newsdesk(Folder,Persistent,Implicit): meta_type = "Newsdesk" # ... def catalog_add(self): "setup the catalog" self.manage_addProduct["ZCatalog"].manage_addZCatalog("Catalog", "Default Catalog") catalog = self._getOb('Catalog') self.addLexicon(catalog) self.addIndexes(catalog) self.addMetas(catalog) catalog = self._getOb('Catalog') return catalog def addIndexes(self, cat): cat.addIndex('date_found', 'FieldIndex', 'date_found') cat.addIndex('deleted_p', 'FieldIndex', 'deleted_p') cat.addIndex('uri', 'FieldIndex', 'uri') cat.addIndex('categories_field', 'FieldIndex', 'categories') cat.addIndex('categorized_p', 'FieldIndex', 'categorized_p') cat.addIndex('approved_p', 'FieldIndex', 'approved_p') cat.addIndex('search_id', 'FieldIndex', 'search_id') extras = Empty() extras.doc_attr = 'title' extras.index_type = 'Okapi BM25 Rank' extras.lexicon_id = 'Lexicon' cat.addIndex('title', 'ZCTextIndex', extras) extras2 = Empty() extras2.doc_attr = 'source_name' extras2.index_type = 'Okapi BM25 Rank' extras2.lexicon_id = 'Lexicon' cat.addIndex('source_name', 'ZCTextIndex', extras2) extras3 = Empty() extras3.doc_attr = 'categories' extras3.index_type = 'Okapi BM25 Rank' extras3.lexicon_id = 'Lexicon' cat.addIndex('categories', 'ZCTextIndex', extras3) cat.addIndex('bobobase_modification_time', 'DateIndex') def addMetas(self, cat): "create the metadata columns" cat.manage_addColumn("categorized_p") cat.manage_addColumn("deleted_p") cat.manage_addColumn("search_id") cat.manage_addColumn("categories") cat.manage_addColumn("date_found") cat.manage_addColumn("source_name") cat.manage_addColumn("title") cat.manage_addColumn("uri") def addLexicon(self, cat): elem = [] wordSplitter = Empty() wordSplitter.group = 'Word Splitter' wordSplitter.name = 'HTML aware splitter' caseNormalizer = Empty() caseNormalizer.group = 'Case Normalizer' caseNormalizer.name = 'Case Normalizer' stopWords = Empty() stopWords.group = 'Stop Words' stopWords.name = 'Remove listed and single char words' elem.append(wordSplitter) elem.append(caseNormalizer) elem.append(stopWords) cat.manage_addProduct['ZCTextIndex'].manage_addLexicon('Lexicon', 'Default Lexicon', elem) On Sun, 3 Sep 2006 20:11:33 +0200, "Dieter Maurer" <dieter@handshake.de> said:
Dieter Maurer wrote at 2006-9-3 19:22 +0200:
zope-dev@rosenthaltechnologies.com wrote at 2006-9-1 13:29 -0400:
Traceback for my Key Error (Dieter, thank you for alerting me on where to find this):
Traceback (innermost last): ... - Physical Path: /nephron_links/news_dtml Module DocumentTemplate.DT_String, line 476, in __call__ Module DocumentTemplate.DT_In, line 465, in renderwb Module DocumentTemplate.DT_In, line 747, in sort_sequence Module Products.ZCatalog.Lazy, line 158, in __getitem__ Module Products.ZCatalog.Catalog, line 520, in getScoredResult KeyError: -464571725
This does not look like a catalog corruption.
Instead, it looks as if the catalog changed since the search was done. Are you caching your search results?
Sorry! I was wrong!
It has nothing to do with a catalog modification after the search.
Looks indeed as if an index reported a document which is no longer indexed. Looks like an index bug. Which (types of) indexes are involved in your failing queries?
-- Dieter
Brian Rosenthal General Partner, RoboCommerce, LLC brian@robocommerce.com http://www.robocommerce.com 800-644-7626
zope-dev@rosenthaltechnologies.com wrote at 2006-9-3 18:06 -0400:
Thank you so much for following-up with me, here. I think you're right about this having to do with the indexes on the ZCatalog.
The catalog has: 1. Seven "FieldIndex" indexes. 2. A "DateIndex" index 3. Three "ZCTextIndex" indexes
I have not seen reports that these indexes cause inconsistencies of the type you have described. A long time ago, there have been problems in the "BTrees" conflict resolution which caused catalog inconsistencies (of a different kind). They affected document lists of length one in the index. When the last remaining document was removed and a new one concurrently inserted, the insertion could be lost. But this was fixed a long time ago. Apart from bugs, inconsistencies are often caused by "try: ... except: ..." clauses. If the "try" clause modifies the persistent state and then is aborted by an exception, it is vital that the exception reaches the ZPublisher such that it aborts the transaction. If the "except" prevents this, then the ZPublisher will commit and you are likely to get inconsistencies. In older Zope versions (before Zope 2.8.1), such a dangerous "try: ... except: ..." has been in "OFS.ObjectManager.ObjectManager.manage_beforeDelete". In more modern versions, the code has been slightly cleaned up but it still is error prone. Skim your log files whether you see "manage_beforeDelete () threw" ERROR log entries. You can get more hints towards the problem by examining the query that results in your exception: If this query just consists of a subquery to a single index, then you know the problematic index. If it combines several indexes (by an "and"), then all of these indexes would need to be inconsistent (which is possible but far less likely than a single index). In this case, one would expect that not the index but the catalog primary data got corrupted. In the latter case, the problem would need to happen during initial indexing or final unindexing. Studying the code may exclude some possibilities. -- Dieter
Where would I find API documentation on get_transaction().commit(...)? I am calling get_transaction().commit(1) at various points in my program and it seems like it's not actually committing those transactions. I'd be grateful on a quick explanation for how this works. I'd like to commit sub-transactions as I go, but I don't want to stop the execution of the script (without the "1", it seems to arrest execution). I do not care about the ability to roll back the transaction. Best, Brian -------------------- Brian Rosenthal General Partner, RoboCommerce, LLC brian@robocommerce.com http://www.robocommerce.com 800-644-7626
zope-dev@rosenthaltechnologies.com wrote at 2006-9-5 19:10 -0400:
... I am calling get_transaction().commit(1) at various points in my program and it seems like it's not actually committing those transactions.
"commit(1)" commits a subtransaction. As the name suggests, you have to commit the main transaction as well (at some time). You do this by "commit()" (or "commit(0)"). -- Dieter
Dieter, thanks. On Thu, 7 Sep 2006 22:09:27 +0200, "Dieter Maurer" <dieter@handshake.de> said:
zope-dev@rosenthaltechnologies.com wrote at 2006-9-5 19:10 -0400:
... I am calling get_transaction().commit(1) at various points in my program and it seems like it's not actually committing those transactions.
"commit(1)" commits a subtransaction.
As the name suggests, you have to commit the main transaction as well (at some time). You do this by "commit()" (or "commit(0)").
-- Dieter
Brian Rosenthal General Partner, RoboCommerce, LLC brian@robocommerce.com http://www.robocommerce.com 800-644-7626
Tres Seaver wrote:
The other reason for wanting "early binding" to the ports is if the ports are in the "reserved for root" range (< 1024); in that case, the ports *must* be bound early, before dropping privileges to those of the "effective user".
Ah, that's true enough, but then again, anyone binding zope to a low-numbered port requires the gun politely pointing away from their foot anyway ;-) Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
--On 29. August 2006 18:50:43 +0200 Dieter Maurer <dieter@handshake.de> wrote:
Jens Vagelpohl wrote at 2006-8-28 20:14 +0200:
On 28 Aug 2006, at 19:19, Andreas Jung wrote:
Some time ago the following patch was proposed to control the timing for opening the socket ports upon startup:
<http://mail.zope.org/pipermail/zope-dev/2005-November/025709.html>
I propose to adapt the patch for Zope 2.11 since it is pretty useful especially when you deal with loadbalancers and have Zope instances that take a lot time for starting up (e.g. Plone :-)). Any objections?
+1
If you have load balancers that only look for open ports (or a single instance with no load balancers even) this will improve the user experience for people who hit the site upon startup. IMHO it's more helpful to get a "connection refused" reply in a browser than a browser that sits and sits and sites...
Can we make this configurable?
The patch fast-listen patch is already available on the trunk. However I don't have plans for any further work in that area. -aj
Andreas Jung wrote:
--On 29. August 2006 18:50:43 +0200 Dieter Maurer <dieter@handshake.de> wrote:
Jens Vagelpohl wrote at 2006-8-28 20:14 +0200:
On 28 Aug 2006, at 19:19, Andreas Jung wrote:
Some time ago the following patch was proposed to control the timing for opening the socket ports upon startup:
<http://mail.zope.org/pipermail/zope-dev/2005-November/025709.html>
I propose to adapt the patch for Zope 2.11 since it is pretty useful especially when you deal with loadbalancers and have Zope instances that take a lot time for starting up (e.g. Plone :-)). Any objections?
+1
If you have load balancers that only look for open ports (or a single instance with no load balancers even) this will improve the user experience for people who hit the site upon startup. IMHO it's more helpful to get a "connection refused" reply in a browser than a browser that sits and sits and sites...
Can we make this configurable?
The patch fast-listen patch is already available on the trunk. However I don't have plans for any further work in that area.
This sounds as if it wasn't configurable. It actually *is* configurable, so everyone should be happy. Those who want to bind late can do so now. Otherwise Zope will bind early as usual. That should end this topic :)
participants (10)
-
Andreas Jung -
Andrew Langmead -
Brian Rosenthal -
Chris Withers -
Christian Theune -
Dieter Maurer -
Jens Vagelpohl -
Philipp von Weitershausen -
Tres Seaver -
zope-dev＠rosenthaltechnologies.com