Question about possible Data.fs corruption
hello all, I recently took over managing a zope site for a small nonprofit. I'm new to Zope and python, and the people who built the site don't have time to maintain it--it arrived in my hands with problems. If a certain kind of object (a "reader's guide") is added to the database, it breaks the site as soon as a person tries to view the new object. We have to restart from a previous version of Data.fs (running Zope 2.5.1, python 2.1, on Debian). I have a copy of the site on an OS 10.3 box with python 2.3 and Zope 2.6.4rc2--on this, the Zope server doesn't come down entirely, but does throw this error when I try to nav to the index page after an update: File /.../Traversable.py, line 106, in unrestrictedTraverse (Object: Zope) (Info: (['105586758764', 'Neela Vaswani', 'Authors'], '')) IndexError: string index out of range The file referred to is the new reader's guide object--I'm not sure why it's being called when I'm simply going to the site index page from an unrelated page, but this is no doubt integral to the problem. The folks who built the site and myself have peeled back this transaction numerous times. Things are packed before that. Nothing else causes problems. Thinking this could be a Data.fs corruption issue, I've run fstest, fsrefs, checkbtrees(none of which complain), and tranalyzer (which breaks almost immediately with "Not enough data left" back in transactions 3 years old). Planning to begin taking back enough transactions that things cease breaking, I've also used a verbose fsrecover to find recent transaction locations, and truncated a copy of the Data.fs file to these--this has created a POSKeyError at 0000000000000002 (site won't start, can't even set app = Zope.app() in python because it throws me an error there, too, so I can't view the object). And...that's where I stand. Any suggestions on how to handle this? I would appreciate any advice on either how to proceed, or rethink the problem. thanks in advance, ira
Ira Sher wrote:
If a certain kind of object (a "reader's guide") is added to the database, it breaks the site as soon as a person tries to view the new object. We have to restart from a previous version of Data.fs (running Zope 2.5.1, python 2.1, on Debian). I have a copy of the site on an OS 10.3 box with python 2.3 and Zope 2.6.4rc2--on this, the Zope server doesn't come down entirely, but does throw this error when I try to nav to the index page after an update: File /.../Traversable.py, line 106, in unrestrictedTraverse (Object: Zope) (Info: (['105586758764', 'Neela Vaswani', 'Authors'], '')) IndexError: string index out of range
A view on an object should not be able to destry the ZODB. The problem is most certainly in the "readers guide". I suspect it does something badly. Has this product ever worked. What kind of product is it? a zClass or a Python product? Does the cpu go to a full 100%? It could be caused by a recursive function somewhere in the product. You could also have a race condition somewhere, where a lock is being used even though it shouldn't be. The strangest things can happen with homegrown Products :-) regards Max M
Max, Thanks for writing back about this...in answer to your questions, yes, the product has worked without incident over a period of years on maybe twenty occasions. It's a zClass product. CPU doesn't do anything remarkable when I view the page on the site (though along with the index page becoming inaccessible, the page with the particular object doesn't update to various parts of the site, and instead must be navigated to as a URL (I see an error in the log saying KeyError: '105586687914' for the object that is the author just edited). This happens regardless of text being used or the particular author object being updated with this product. Previous objects of the same type can be edited without any problems. Any other thoughts much appreciated ira On Mar 3, 2004, at 6:10 PM, Max M wrote:
A view on an object should not be able to destry the ZODB. The problem is most certainly in the "readers guide". I suspect it does something badly. Has this product ever worked. What kind of product is it? a zClass or a Python product?
Does the cpu go to a full 100%? It could be caused by a recursive function somewhere in the product.
You could also have a race condition somewhere, where a lock is being used even though it shouldn't be. The strangest things can happen with homegrown Products :-)
regards Max M
Ira Sher wrote:
instead must be navigated to as a URL (I see an error in the log saying KeyError: '105586687914' for the object that is the author just edited).
Do you use Zope's sessioning machinery at all? It's be good to have as much of the log entry that containing this error anyway... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris, I know that I am running Zope with CoreSessionsTracking 0.9, but I don't know if the site uses sessions...I've taken it over recently (and without knowing Zope...it's a long story) and am trying to find my way around it while fixing some significant problems. The error log for the KeyError says: 2004-03-04T12:47:25 ERROR(200) SiteError http://localhost:8080/sarabande//index_html Traceback (innermost last): File /Volumes/isher/Sites/sarabande/Zope-2.6.4rc2-src/lib/python/ZPublisher/ Publish.py, line 98, in publish File /Volumes/isher/Sites/sarabande/Zope-2.6.4rc2-src/lib/python/ZPublisher/ mapply.py, line 88, in mapply (Object: index_html) File /Volumes/isher/Sites/sarabande/Zope-2.6.4rc2-src/lib/python/ZPublisher/ Publish.py, line 39, in call_object (Object: index_html) File /Volumes/isher/Sites/sarabande/Zope-2.6.4rc2-src/lib/python/OFS/ DTMLDocument.py, line 126, in __call__ (Object: index_html) File /Volumes/isher/Sites/sarabande/Zope-2.6.4rc2-src/lib/python/ DocumentTemplate/DT_String.py, line 474, in __call__ (Object: index_html) File /Volumes/isher/Sites/sarabande/Zope-2.6.4rc2-src/lib/python/ DocumentTemplate/DT_In.py, line 602, in renderwb (Object: Catalog.searchResults( meta_type='Book', sort_on='entry_date', sort_order='reverse')) File /Volumes/isher/Sites/sarabande/Zope-2.6.4rc2-src/lib/python/ DocumentTemplate/DT_Let.py, line 75, in render (Object: book="getobject(data_record_id_, REQUEST)") File /Volumes/isher/Sites/sarabande/Zope-2.6.4rc2-src/lib/python/ DocumentTemplate/DT_Util.py, line 201, in eval (Object: getobject(data_record_id_, REQUEST)) (Info: data_record_id_) File <string>, line 1, in <expression> (Object: getobject) File /Volumes/isher/Sites/sarabande/Zope-2.6.4rc2-src/lib/python/Products/ ZCatalog/ZCatalog.py, line 562, in getobject (Object: Catalog) File /Volumes/isher/Sites/sarabande/Zope-2.6.4rc2-src/lib/python/OFS/ Traversable.py, line 155, in unrestrictedTraverse (Object: Zope) (Info: ([], '105586687914')) File /Volumes/isher/Sites/sarabande/Zope-2.6.4rc2-src/lib/python/OFS/ ObjectManager.py, line 657, in __getitem__ (Object: Arthur Vogelsang) KeyError: '105586687914' I'm trying to do as much troubleshooting as possible on my local machine rather than the live site because the errors that are making the homepage inaccessible were actually knocking down the whole server on the older version of Zope the live site runs. Any ideas about this? thanks ira On Mar 5, 2004, at 5:51 AM, Chris Withers wrote:
Ira Sher wrote:
instead must be navigated to as a URL (I see an error in the log saying KeyError: '105586687914' for the object that is the author just edited).
Do you use Zope's sessioning machinery at all?
It's be good to have as much of the log entry that containing this error anyway...
cheers,
Chris
-- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Ira Sher wrote at 2004-3-5 13:25 -0500:
... The error log for the KeyError says: ... (Object: book="getobject(data_record_id_, REQUEST)") File /Volumes/isher/Sites/sarabande/Zope-2.6.4rc2-src/lib/python/ DocumentTemplate/DT_Util.py, line 201, in eval (Object: getobject(data_record_id_, REQUEST)) (Info: data_record_id_) ... /Volumes/isher/Sites/sarabande/Zope-2.6.4rc2-src/lib/python/OFS/ ObjectManager.py, line 657, in __getitem__ (Object: Arthur Vogelsang) KeyError: '105586687914'
Your catalog references an object that is no longer there. The missing object is "Arthur Vogelsang/105586687914". -- Dieter
Chris, FYI, the problem with the site seems to affect all of the tools that affect a particular object--Book--and all attempts to add new book objects or other objects related to book objects. Old objects can be modified, but new ones either fail, or while appearing to succeed, cause subsequent errors. An example of trying to add a new Book object causes this error: Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module OFS.DTMLMethod, line 127, in __call__ Module DocumentTemplate.DT_String, line 474, in __call__ Module DocumentTemplate.DT_With, line 61, in render Module DocumentTemplate.DT_Util, line 195, in eval - __traceback_info__: Book Module AccessControl.Owned, line 123, in getWrappedOwner ValueError: unpack list of wrong size Apparently, the updating tools on the site simply broke at some point four months ago (the previous site administrator doesn't have time to deal with the site anymore, and can't think of anything that he might have changed at that time). I assume these various different errors have their source in a common problem or corruption. Any ideas of how such a thing might work? thanks again, ira On Mar 5, 2004, at 5:51 AM, Chris Withers wrote:
Ira Sher wrote:
instead must be navigated to as a URL (I see an error in the log saying KeyError: '105586687914' for the object that is the author just edited).
Do you use Zope's sessioning machinery at all?
It's be good to have as much of the log entry that containing this error anyway...
cheers,
Chris
-- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Ira Sher wrote at 2004-3-5 13:59 -0500:
FYI, the problem with the site seems to affect all of the tools that affect a particular object--Book--and all attempts to add new book objects or other objects related to book objects. Old objects can be modified, but new ones either fail, or while appearing to succeed, cause subsequent errors. An example of trying to add a new Book object causes this error: Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module OFS.DTMLMethod, line 127, in __call__ Module DocumentTemplate.DT_String, line 474, in __call__ Module DocumentTemplate.DT_With, line 61, in render Module DocumentTemplate.DT_Util, line 195, in eval - __traceback_info__: Book Module AccessControl.Owned, line 123, in getWrappedOwner ValueError: unpack list of wrong size
That was a bug in one Zope version triggered for "unowned" objects. It may be fixed in current Zope versions. In principle, "unowned" objects should only live below "Control_Panel". You should not find unowned objects outside this subhierarchie. -- Dieter
Ira Sher wrote at 2004-3-3 17:58 -0500:
... If a certain kind of object (a "reader's guide") is added to the database, it breaks the site as soon as a person tries to view the new object. We have to restart from a previous version of Data.fs (running Zope 2.5.1, python 2.1, on Debian). I have a copy of the site on an OS 10.3 box with python 2.3 and Zope 2.6.4rc2--on this, the Zope server doesn't come down entirely, but does throw this error when I try to nav to the index page after an update: File /.../Traversable.py, line 106, in unrestrictedTraverse (Object: Zope) (Info: (['105586758764', 'Neela Vaswani', 'Authors'], '')) IndexError: string index out of range
You should look at the traceback, post it if necessary together with a few lines of source code refered to in the last traceback entry.
Thinking this could be a Data.fs corruption issue,
This is very unlikely. -- Dieter
Dieter, This is the traceback I always get from adding this particular object and then going to view the site homepage: Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module OFS.DTMLDocument, line 126, in __call__ Module DocumentTemplate.DT_String, line 474, in __call__ Module DocumentTemplate.DT_In, line 602, in renderwb Module DocumentTemplate.DT_Let, line 75, in render Module DocumentTemplate.DT_Util, line 201, in eval - __traceback_info__: data_record_id_ Module <string>, line 1, in <expression> Module Products.ZCatalog.ZCatalog, line 562, in getobject Module OFS.Traversable, line 106, in unrestrictedTraverse - __traceback_info__: (['105586758764', 'Neela Vaswani', 'Authors'], '') IndexError: string index out of range Based on feedback I've been getting here, I'm definitely not concentrating on data corruption anymore...any ideas about where best to place my attentions? thanks for your thoughts, ira On Mar 5, 2004, at 4:39 PM, Dieter Maurer wrote:
Ira Sher wrote at 2004-3-3 17:58 -0500:
... If a certain kind of object (a "reader's guide") is added to the database, it breaks the site as soon as a person tries to view the new object. We have to restart from a previous version of Data.fs (running Zope 2.5.1, python 2.1, on Debian). I have a copy of the site on an OS 10.3 box with python 2.3 and Zope 2.6.4rc2--on this, the Zope server doesn't come down entirely, but does throw this error when I try to nav to the index page after an update: File /.../Traversable.py, line 106, in unrestrictedTraverse (Object: Zope) (Info: (['105586758764', 'Neela Vaswani', 'Authors'], '')) IndexError: string index out of range
You should look at the traceback, post it if necessary together with a few lines of source code refered to in the last traceback entry.
Thinking this could be a Data.fs corruption issue,
This is very unlikely.
-- Dieter
Your ZCatalog tries to retrieve the object reference from the object path which is stored as a list. This path in the ZCatalog is now incorrect. I.e. the object does not exist on that path. Try to update your ZCatalog. Either manually from the ZMI or manually index each object. Ira Sher wrote:
Dieter, This is the traceback I always get from adding this particular object and then going to view the site homepage: Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module OFS.DTMLDocument, line 126, in __call__ Module DocumentTemplate.DT_String, line 474, in __call__ Module DocumentTemplate.DT_In, line 602, in renderwb Module DocumentTemplate.DT_Let, line 75, in render Module DocumentTemplate.DT_Util, line 201, in eval - __traceback_info__: data_record_id_ Module <string>, line 1, in <expression> Module Products.ZCatalog.ZCatalog, line 562, in getobject Module OFS.Traversable, line 106, in unrestrictedTraverse - __traceback_info__: (['105586758764', 'Neela Vaswani', 'Authors'], '') IndexError: string index out of range
Based on feedback I've been getting here, I'm definitely not concentrating on data corruption anymore...any ideas about where best to place my attentions? thanks for your thoughts, ira
On Mar 5, 2004, at 4:39 PM, Dieter Maurer wrote:
Ira Sher wrote at 2004-3-3 17:58 -0500:
... If a certain kind of object (a "reader's guide") is added to the database, it breaks the site as soon as a person tries to view the new object. We have to restart from a previous version of Data.fs (running Zope 2.5.1, python 2.1, on Debian). I have a copy of the site on an OS 10.3 box with python 2.3 and Zope 2.6.4rc2--on this, the Zope server doesn't come down entirely, but does throw this error when I try to nav to the index page after an update: File /.../Traversable.py, line 106, in unrestrictedTraverse (Object: Zope) (Info: (['105586758764', 'Neela Vaswani', 'Authors'], '')) IndexError: string index out of range
You should look at the traceback, post it if necessary together with a few lines of source code refered to in the last traceback entry.
Thinking this could be a Data.fs corruption issue,
This is very unlikely.
-- Dieter
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
.
-- Peter Bengtsson, http://www.peterbe.com
Ira Sher wrote at 2004-3-5 18:02 -0500:
This is the traceback I always get from adding this particular object and then going to view the site homepage: ... - __traceback_info__: data_record_id_ Module <string>, line 1, in <expression> Module Products.ZCatalog.ZCatalog, line 562, in getobject Module OFS.Traversable, line 106, in unrestrictedTraverse - __traceback_info__: (['105586758764', 'Neela Vaswani', 'Authors'], '') IndexError: string index out of range
Based on feedback I've been getting here, I'm definitely not concentrating on data corruption anymore...any ideas about where best to place my attentions?
One of your objects (the one your application currently tries to access) is catalogued with a bad "uid" (unique id). Usually, the "uid" is a path derived from "getPhysicalPath" (the path from the root to the object). In the case of this object, the uid is a path "...//Authors/Neela Vaswani/105586758764" but the path contains an empty segment. "unrestrictedTraverse" (called by "getObject") cannot handle this case. We can fix this manually when you have enough information to identify the object. For this, you must determine the path with the empty segment. For this, you would replace the "getObject" by a "getPath" and output the path somehow. Then, in a Python Script (or interactively, when you know how you do this) you call "...catalog.uncatalog_object(yourBadPath)". This removes the broken object from the catalog. After that, you recatalog the object again "...catalog.catalog_object(yourObject)". Of course, we then still do not know how the bad path happened to enter the catalog. Other objects might suffer from the same problem.... -- Dieter
Dieter, Thank you for your responses. I've determined from the catalog that the forty-odd records with path problems generally have a double-backslash at the beginning, and otherwise contain the correct path. I'm going to try to update them. The objects (like the one I originally enquired about) that were causing such problems with the site have been removed, but when re-added, while I think I've solved the condition that was causing the path to appear incorrectly, I have noticed that these objects don't otherwise behave as expected (and cause problems on the live site) because where scripts look for the following data: <dtml-in "Catalog.searchResults(meta_type='Book',sie='on')"> and the dtml script writing the data is writing: <dtml-call "REQUEST.set('id','readers_guide')"> <dtml-call "REQUEST.set('sie','on')"> <dtml-with "Readers_Guide.createInObjectManager(REQUEST['id'], REQUEST)"> <dtml-call "propertysheets.Basic.manage_editProperties(REQUEST)"> <dtml-call "aq_parent.propertysheets.Basic.manage_changeProperties(REQUEST)"> <dtml-call "reindex_object()"> <dtml-call "aq_parent.reindex_object()"> </dtml-with> which would seem to set sie as "on", the newer records return sie as True rather than "on", and so fail. The version of python was updated on the live server. Could this have caused the problem? How can I be more explicit in the call to ensure that "on" is preserved rather than converted to True? thanks again ira On Mar 6, 2004, at 3:27 PM, Dieter Maurer wrote:
Ira Sher wrote at 2004-3-5 18:02 -0500:
This is the traceback I always get from adding this particular object and then going to view the site homepage: ... - __traceback_info__: data_record_id_ Module <string>, line 1, in <expression> Module Products.ZCatalog.ZCatalog, line 562, in getobject Module OFS.Traversable, line 106, in unrestrictedTraverse - __traceback_info__: (['105586758764', 'Neela Vaswani', 'Authors'], '') IndexError: string index out of range
Based on feedback I've been getting here, I'm definitely not concentrating on data corruption anymore...any ideas about where best to place my attentions?
One of your objects (the one your application currently tries to access) is catalogued with a bad "uid" (unique id). Usually, the "uid" is a path derived from "getPhysicalPath" (the path from the root to the object). In the case of this object, the uid is a path "...//Authors/Neela Vaswani/105586758764" but the path contains an empty segment. "unrestrictedTraverse" (called by "getObject") cannot handle this case.
We can fix this manually when you have enough information to identify the object. For this, you must determine the path with the empty segment. For this, you would replace the "getObject" by a "getPath" and output the path somehow. Then, in a Python Script (or interactively, when you know how you do this) you call "...catalog.uncatalog_object(yourBadPath)". This removes the broken object from the catalog. After that, you recatalog the object again "...catalog.catalog_object(yourObject)".
Of course, we then still do not know how the bad path happened to enter the catalog. Other objects might suffer from the same problem....
-- Dieter
Hi Ira, Ira Sher wrote: This is truly horrible code, and a good example of why DTML sux so bad:
<dtml-in "Catalog.searchResults(meta_type='Book',sie='on')"> and the dtml script writing the data is writing: <dtml-call "REQUEST.set('id','readers_guide')"> <dtml-call "REQUEST.set('sie','on')"> <dtml-with "Readers_Guide.createInObjectManager(REQUEST['id'], REQUEST)"> <dtml-call "propertysheets.Basic.manage_editProperties(REQUEST)"> <dtml-call "aq_parent.propertysheets.Basic.manage_changeProperties(REQUEST)"> <dtml-call "reindex_object()"> <dtml-call "aq_parent.reindex_object()"> </dtml-with>
Lets try the above again in a Script (Python): REQUEST = context.REQUEST # NB: no REQUEST polution from the following... id = 'readers_guide' sie = 'on' for brain in Catalog.searchResults(meta_type='Book',sie='on'): # dunno what else you have in REQUEST, so leaving as is... rg = brain.Readers_Guide.createInObjectManager(id, REQUEST) # you're using ZClasses and editProperties - ouch! # I hope this method is # getting called from a web form which shows ALL the properties... rg.propertysheets.Basic.manage_editProperties(REQUEST) rg_container = rg.aq_parent # The following line makes no sense, what's it trying to change? rg_container.propertysheets.Basic.manage_changeProperties(REQUEST) rg.reindex_object() rg_container.reindex_object() ...which is still pretty croaky, but at least debuggable now. I'd really want to get ride of those REQUEST's as arguments to createInObjectManager and the property changers...
which would seem to set sie as "on", the newer records return sie as True rather than "on", and so fail.
Using the REQUEST in the way the above do is pretty dodgy...
The version of python was updated on the live server. Could this have caused the problem?
Yes, if the properties are of type Boolean and you upgraded Zope as well..
How can I be more explicit in the call to ensure that "on" is preserved rather than converted to True?
Don't use ZClasses? ;-) Well, look at the property sheets, see what version fo Zope was and is in use, etc... cheers, Chris - nothing beats a badly written Zope app (or perl) for pain and suffering. -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris, Ack. Thanks for the feedback. The site was written long ago in Zope 1.x, and has been nosed along to it's current state, when the last of the developers didn't have time to take care of it. It's too large (and the client too nonprofit) to overhaul the whole thing, but it looks like I'll be pulling apart bits of it as it breaks to rewrite. Didn't know there was such a strong feeling about using dtml request (having worked with zope/python for all of a month). regards, ira On Mar 10, 2004, at 9:10 AM, Chris Withers wrote:
Hi Ira,
Ira Sher wrote:
This is truly horrible code, and a good example of why DTML sux so bad:
<dtml-in "Catalog.searchResults(meta_type='Book',sie='on')"> and the dtml script writing the data is writing: <dtml-call "REQUEST.set('id','readers_guide')"> <dtml-call "REQUEST.set('sie','on')"> <dtml-with "Readers_Guide.createInObjectManager(REQUEST['id'], REQUEST)"> <dtml-call "propertysheets.Basic.manage_editProperties(REQUEST)"> <dtml-call "aq_parent.propertysheets.Basic.manage_changeProperties(REQUEST)"> <dtml-call "reindex_object()"> <dtml-call "aq_parent.reindex_object()"> </dtml-with>
Lets try the above again in a Script (Python):
REQUEST = context.REQUEST # NB: no REQUEST polution from the following... id = 'readers_guide' sie = 'on' for brain in Catalog.searchResults(meta_type='Book',sie='on'): # dunno what else you have in REQUEST, so leaving as is... rg = brain.Readers_Guide.createInObjectManager(id, REQUEST) # you're using ZClasses and editProperties - ouch! # I hope this method is # getting called from a web form which shows ALL the properties... rg.propertysheets.Basic.manage_editProperties(REQUEST) rg_container = rg.aq_parent # The following line makes no sense, what's it trying to change? rg_container.propertysheets.Basic.manage_changeProperties(REQUEST) rg.reindex_object() rg_container.reindex_object()
...which is still pretty croaky, but at least debuggable now. I'd really want to get ride of those REQUEST's as arguments to createInObjectManager and the property changers...
which would seem to set sie as "on", the newer records return sie as True rather than "on", and so fail.
Using the REQUEST in the way the above do is pretty dodgy...
The version of python was updated on the live server. Could this have caused the problem?
Yes, if the properties are of type Boolean and you upgraded Zope as well..
How can I be more explicit in the call to ensure that "on" is preserved rather than converted to True?
Don't use ZClasses? ;-) Well, look at the property sheets, see what version fo Zope was and is in use, etc...
cheers,
Chris - nothing beats a badly written Zope app (or perl) for pain and suffering.
-- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
You weren't around to hear everyone rejoicing at the arrival of the Python script. It was a beautiful day. Troy Ira Sher wrote:
...Didn't know there was such a strong feeling about using dtml request (having worked with zope/python for all of a month). regards, ira
participants (6)
-
Chris Withers -
Dieter Maurer -
Ira Sher -
Max M -
Peter Bengtsson -
Troy Farrell