May 2003 - Zope-Dev - Zope lists

[CRITICAL] Conflict Errors, Transactions, Retries, Oh My....
by Jeffrey P Shell 31 May '03

31 May '03

I need to know more about Conflict Errors. We're running into a lot of them lately, it seems, on production Zope 2.6.1 sites (running on FreeBSD). The primary culprit seems to be Temporary Storage/Sessions. Something that has happened, and is causing a small amount of alarm, is that a large method that interfaces to external non-transactional systems seems to (on occasion) send their information off to that external system twice, but there's only one matching set of Zope data. As the two writes to the non-transactional system are very close to each other and contain nearly identical data (except for one bit that gets regenerated in the method), and there are conflict INFO reports in the Event Log from around the same time, I'm assuming that a conflict error is happening somewhere in this method and causing the transaction to be retried (if I'm understanding how Conflict Errors work). Zope and the relational databases seem to do things fine with rolling back the data, but the non-transactional systems now have duplicate data that they **absolutely should not have**. This doesn't happen often, but (as stated), this is a critical operation that needs to be better protected. All other exceptions and bits and pieces in the block of code in question has been tested thoroughly and we have not had any other problems that cause erroneous writes. Is there a way I can protect against Conflict Error retries as well? Is there some sort of Try/Except or Try/Finally I can wrap around the code that won't interfere with the ZODB? Is there any other sort of best-practice here that could help me (and others) who might unknowingly trigger this problem? I know there are some fixes likely to be in Zope 2.6.2 that may help with the situation, but I'd like to put extra protections around this code regardless of what may be coming in the future. Thanks in advance, Jeffrey P Shell jeffrey(a)cuemedia.com

6 14

WebDAV File Descriptor Leak
by Brian Brinegar 29 May '03

29 May '03

Hello, We've run into a problem where we get messages like this in our Zope logs: 2003-05-23T18:56:08 PROBLEM(100) ZServer Server accept() threw an exception we occasionally get a lot of these and then restart Zope on the ZEO Client and everything is okay. After a little more investigation we found that these occur when we run out of File Descriptors. So, why are we running out of file descriptors??? Tracked it down to this: each time a WebDAV client connects to zope cluster a temporary file is created. lsof shows the following: python 19005 httpd 1016u REG 0,9 227 15857725 /var/tmp/(a)19004.1930 (deleted) python 19005 httpd 1017u REG 0,9 227 15857727 /var/tmp/(a)19004.1931 (deleted) python 19005 httpd 1018u REG 0,9 227 15857732 /var/tmp/(a)19003.1933 (deleted) python 19005 httpd 1019u REG 0,9 227 15857738 /var/tmp/(a)19003.1934 (deleted) python 19005 httpd 1020u REG 0,9 227 15857740 /var/tmp/(a)19003.1935 (deleted) python 19005 httpd 1021u REG 0,9 227 15857743 /var/tmp/(a)19003.1936 (deleted) python 19005 httpd 1022u REG 0,9 227 15857745 /var/tmp/(a)19003.1937 (deleted) python 19005 httpd 1023u REG 0,9 227 15857886 /var/tmp/(a)19003.1956 (deleted) they are all roughly the same size. There seems to be a file descriptor leak. Is this a known problem? We are running Zope 2.5.1 right now. Any ideas? Thanks, -Brian

6 12

manage_addZClass* permission question
by Jamie Heilman 29 May '03

29 May '03

I can't fathom the ZClass code. Can somebody tell me if manage_addZClass, manage_addZClassForm, and manage_subclassableClassNames are supposed to be protected by the 'Add Zope Class' permission, or if the code in ZClasses/__init__.py is pure fluf? That permission never shows up in any folder's security settings that I can see. VerboseSecurity has this to say about manage_addZClassForm: Unauthorized: Your user account does not have the required permission. Access to 'manage_addZClassForm' of (Product instance at 89189e0) denied. Your user account, meh, exists at /acl_users. Access requires one of the following roles: ['Manager']. Your roles in this context are ['Authenticated']. So it doesn't look like there is a named permission associated with those methods. I have to wonder if thats intentional. -- Jamie Heilman http://audible.transient.net/~jamie/ "Most people wouldn't know music if it came up and bit them on the ass." -Frank Zappa

2 3

The Return of the Son of Session Errors, redux
by Jean Jordaan 29 May '03

29 May '03

Hi all We're being chowed heavily by the KeyError discussed from the 13th to the 18th March in this thread: http://mail.zope.org/pipermail/zope-dev/2003-March/019081.html Has there been any definite resolution of that issue? We're running Zope 2.6.1, has the issue gone away on Zope 2.6.2? In that thread a patch to TemporaryFolder.py was suggested to deal with ReadConflictErrors, but we don't get those. The suggestions regarding the KeyErrors was: 1) to move to a different storage, such as BerkeleyStorage or DirectoryStorage, for the session data, or 2) to "touch the objects that are likely to change early in the transaction." (Toby Dickenson). Our errors are triggered in the bowels of XMLWidgets, so I don't think 1) is a simple solution, and we don't want to do 2) if we can avoid it. Have specific alternative session storages been found to be free of this issue? The initial report has been logged as a 'critical' error as http://collector.zope.org/Zope/848 but it's still Pending. -- Jean Jordaan http://www.upfrontsystems.co.za

3 6

ZEO install/runtime issues
by Jeremy Hylton 29 May '03

29 May '03

[Please followup to zodb-dev.] Richard, You made some changes to the mkzeoinst.py script in April. I was busy then, and I've just had a chance to look at the changes now. I'd like to discuss some of the changes, and I'm including a wider discussion list to make sure we include anyone else who is interested. A number of the changes are Zope specific. (For example, you can't even run mkzeoinst.py without having a directory named "Zope" hanging off of sys.path.) ZEO and ZODB are intended for use separately from the rest of Zope, so we need to find a way to factor this out into a generic configuration and a Zope-specific configuration. One other requirement for ZEO is that it work with Zope 2.6. I expect there will be a ZODB 3.2 release long before there is a Zope 2.7 release. So we can't depend on any Zope 2.7 features in mkzeoinst.py. The other question I have is about the organization of software into a Zope home and an instance home. I'm not sure what the history of this arrangement is, but I recommend that people do not configure their ZEO servers to share software with their Zope app servers. It can cause fairly severe problems! The problem with sharing software is that the ZEO server can load arbitrary modules when it attempts to perform conflict resolution. If there is a conflict for an instance of class A.B.C, then ZEO will load A.B.C and see if it has an _p_resolveConflict() method. If the modules A or B have any side-effects at import time, then those side-effects will occur in the ZEO server. I've seen this method lookup cause all of CMF to get imported and try to initialize itself. This ended up brining down the ZEO server. A safer way to run a ZEO server is to have an isolated copy of the software that only contains software for objects that need to perform conflict resolution. In practice, many sites only need conflict resolution for BTrees. So none of the other Zope or product code needs to be accessible to the ZEO server. Do you have any ideas about how to support the features you need given the requirements I've suggested here? Do those requirements make sense? Jeremy

3 2

App.Permission security hole
by Jamie Heilman 28 May '03

28 May '03

Tooling through restructuring of my site I discovered a stupid permissions problem. While App.Permission declares the 'Define permission' perm it never gets initialized and thus manage_addPermission{,Form} basically had weakened security. The permission 'Access contents information' was still protecting the method, but thats not adequate--that permission was never intented (afaik) to represent "write access" to the zodb. By default, as the 'Access contents information' permission is granted to the Anonymous user, anybody could fire off a request to http://victimhost/Control_Panel/Products/x/manage_addPermission?id=foo&titl… where 'x' is some installed product (hey, why not use the HelpSys vuln to find one that fits your fancy!) to add a permission object to Product x. Anonymous users being allowed to bloat the zodb at the least, possibly other issues at the worst (I don't know, I don't use the ZClass machinery). A quick refactor of App/Permission.py may be found at http://audible.transient.net/zope/Permission.py which protects the methods in question, however I have a hunch there may be more broken here than this permission alone. Testimonies on #zope indicate this affects 2.6.x as well as CVS, I don't know how far back the bug goes (only due to sheer lazyness). Collector report will be filed shortly, expect it to be hidden. Workarounds: use the above Permission.py, or remove 'Access contents information' from the Control_Panel/Products folder (and possibly others) -- Jamie Heilman http://audible.transient.net/~jamie/ "It's almost impossible to overestimate the unimportance of most things." -John Logue

2 1

a better propertyLabel as standard?
by Lennart Regebro 27 May '03

27 May '03

What about replacing the default propertyLabel of PropertyManager with the one from CMF: def propertyLabel(self, id): """Return a label for the given property id """ for p in self._properties: if p['id'] == id: return p.get('label', id) return id The current one: def propertyLabel(self, id): """Return a label for the given property id """ return id Is kinda silly and this little change would make things neater, and remove the use of having it a separate function in CMF, where it really doesn't belong. Thoughts?

4 5

Adding user.
by Ignacio Dosil Lago 26 May '03

26 May '03

Hi all, I wanted to add a new user to Zope, so I copied it's login name from a document and pasted it into the user add form. That login name included strange characters which I couldn't see. Now I can't remove that user!! I'm using zope-2.6.0. Does version 2.6.1 solve this problem? Any ideas on how to remove that user?? Thank you.

3 3

Re: [Zope-dev] TextIndexNG and ExternalFile
by Craeg Strong 23 May '03

23 May '03

That is a good question. Does it work when using the Zope "File" object instead of ExternalFile? Or how about trying the standard Zope TextIndex instead of TextIndexNG? Either situation will provide the clue needed to find the problem. Regards, --Craeg > Hello, > I am using TextIndexNG and ExternalFile to index documents. I am able > to index all documents that are in the ZODB and also pdf files in a > LocalFS directory. However, the indexing fails for pdf documents > setup as External Files. > Am I missing anything in the setup? Incidentally, it is able to index > ExternalFile text documents, only the pdf files have a problem. Any > help would be appreciated. > Setup is on Linux. Converter is xpdf. > Regards/Biddu > > > > > _______________________________________________ > Zope-Dev maillist - Zope-Dev(a)zope.org > http://mail.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope ) >

1 0

Zpydoc RFC
by alan milligan 23 May '03

23 May '03

Hi, This is my first posting on this list - so I hope I'm at the right place. I've just integrated pydoc into Zope. This in itself is of limited use, so I've extended it a bit: * applied Zope permissions to control and limit documentable content * full content customisation with ZPT * addition of zope-specific stuff - like documenting zope permissions on functions It's currently in release candidate status, because I think that with one or two hours effort from someone who knows more about Zope internals than myself, it would be a massively better product. I am hoping that someone on this list is interested in this work. I have a page at http://www.last-bastion.net/Zpydoc where current efforts are being coordinated. All help (and comments) about progressing this to general release is greatfully received. Cheers, Alan _________________________________________________________________ Check out the Xtra gaming servers at http://xtramsn.co.nz/gaming !

1 0