RE: [Zope] Integrating Zope with Excel.
Joe, Hmmm, well yes, Zope is multi-threaded ... But this only means that one thread does one piece of work at a time ... In a production environment, there's nothing preventing 2 threads from processing 2 similar requests for an Excel conversion, in which ase you'd run into the issues I mentionned. Apart from that I can't help much more ... Your best bet would be with ActiveState support or the win32all mailing lists, if any ... J.F. -----Original Message----- From: zope-bounces+jean-francois.doyon=ccrs.nrcan.gc.ca@zope.org [mailto:zope-bounces+jean-francois.doyon=ccrs.nrcan.gc.ca@zope.org]On Behalf Of Joe Goldthwaite Sent: August 25, 2004 6:25 PM To: zope@zope.org Subject: RE: [Zope] Integrating Zope with Excel. Thanks for the reply J.F. I can use all the input I cat get. Although I might run into problems with re-entrancy when this goes into production, I don't think that's the case here. I'm the only one using the machine and I'm not submitting concurrent requests. I also thought Zope was multi-threaded so it has to finish a request before it can start another one. I could be wrong on that though. Joe Goldthwaite -----Original Message----- From: Jean-Francois.Doyon@CCRS.NRCan.gc.ca [mailto:Jean-Francois.Doyon@CCRS.NRCan.gc.ca] Sent: Wednesday, August 25, 2004 3:06 PM To: joe@goldthwaites.com; zope@zope.org Subject: RE: [Zope] Integrating Zope with Excel. I don't know much of anything about coding on windows, but I had looked at doing something smiliar once ... To convert PPT's on demand to HTML in my case, but I looked at using Python's COM features to do it (Regular Python + win32all, not ActiveState's). One thing that stopped me dead in my tracks was many comments form Microsoft that all the Office related COM objects are NOT re-entrant. Therefore Excel.Application can only exist once and so on, and you have to take care with such issues ... This is why nonbody's written an MS Office based Web Service :) You're not really using it as a web service, but the same concecpt applies I think? Basically, I'd look at object re-entrancy and concurency issues on the MS Office side. I ended up never needing mine, but I think I'd had it down to the fact that the best was to have the Application instance running and use Document instances to do the work ... Thing is I really didn't want to have PowerPoint instanciated permanently on the server. Anyways, just some thoughts, though I'm not sure they're of any help :) J.F. -----Original Message----- From: zope-bounces@zope.org [mailto:zope-bounces@zope.org]On Behalf Of Joe Goldthwaite Sent: August 25, 2004 5:40 PM To: zope@zope.org Subject: [Zope] Integrating Zope with Excel. Hello Again, I've been working on a Zope application and one of the tasks is to integrate the web reports with Excel. I've got some financial reports that the user can browse through using a web interface. What I'm trying to do is have a link on the page that will download the report they're viewing into a local Excel worksheet. I decided to use com because I want the worksheet to have colors, print settings, formatting, etc. I already have working code in VB. I've looked into XML and Pyxlwriter but it looks like a new learning curve compared to just translating my VB com commands into Python. I'm using Visual Studio .Net 2003 with Active Python as my development environment. When I submit the URL that creates the XL worksheet in development, it works like a top. In testing, I've generated 50 different worksheets without any problems. When I try to run the same code in Zope I have problems. Under Zope, I take the submitted URL and create a worksheet that I save to a file. Then I return a meta command to redirect the browser to the newly created file which is downloaded to the users computer. The good news is that it works perfectly - sometimes. Other times I get this error message; Module ExcelReport, line 48, in __init__ Module win32com.client, line 95, in Dispatch Module win32com.client.dynamic, line 91, in _GetGoodDispatchAndUserName Module win32com.client.dynamic, line 79, in _GetGoodDispatch com_error: (-2147221008, 'CoInitialize has not been called.', None, None) The error message is coming back from the line; XLApp = Dispatch("Excel.Application"). The really strange thing is that the error seems to be random. Sometimes it works, sometimes it doesn't. I haven't been able to determine any rhyme or reason to it. When I restart Zope, it will work one to as many as 10 times. Then I'll get the error message. If I keep trying the same URL, it might eventually work or I may have to restart Zope to get it working again. Does anyone have any idea as to what's going on here? Thanks, Joe Goldthwaite _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
J.F., I've never understood the threading issues - too complex for my feeble brain. I know that if one user performs a large job through Zope, it stops responding to other users until the big job is finished. I've also seen posts discussing putting Zope behind Apache so that all the requests can run independently. It's not an issue in my app because I've only got about 20 users per server and seldom have more than five or six accessing it at the same time. The suggestion of posting on the win32 mailing lists is a good one. In fact, I had posted there first. Mark Hammond himself actually posted a response. He said to try using the DispatchEx() function instead of Dispatch(). I gather that Dispatch() just tries to get a reference to the object. If Excel is already running, it just returns a reference to the existing session. DispatchEx() always creates a new Excel session. That sounds safer in my application so I changed it to that. That didn't fix the problem though. He also suggested that I manually call pythoncom.CoInitialize(). That fixed the problem. He also said that he didn't think either suggestion was a "great idea". I'm not sure what he meant by that. It's a little worrysome but it seems to be working fine now. Thanks again for your help. -----Original Message----- From: Jean-Francois.Doyon@CCRS.NRCan.gc.ca [mailto:Jean-Francois.Doyon@CCRS.NRCan.gc.ca] Sent: Thursday, August 26, 2004 10:08 AM To: joe@goldthwaites.com; zope@zope.org Subject: RE: [Zope] Integrating Zope with Excel. Joe, Hmmm, well yes, Zope is multi-threaded ... But this only means that one thread does one piece of work at a time ... In a production environment, there's nothing preventing 2 threads from processing 2 similar requests for an Excel conversion, in which ase you'd run into the issues I mentionned. Apart from that I can't help much more ... Your best bet would be with ActiveState support or the win32all mailing lists, if any ... J.F. -----Original Message----- From: zope-bounces+jean-francois.doyon=ccrs.nrcan.gc.ca@zope.org [mailto:zope-bounces+jean-francois.doyon=ccrs.nrcan.gc.ca@zope.org]On Behalf Of Joe Goldthwaite Sent: August 25, 2004 6:25 PM To: zope@zope.org Subject: RE: [Zope] Integrating Zope with Excel. Thanks for the reply J.F. I can use all the input I cat get. Although I might run into problems with re-entrancy when this goes into production, I don't think that's the case here. I'm the only one using the machine and I'm not submitting concurrent requests. I also thought Zope was multi-threaded so it has to finish a request before it can start another one. I could be wrong on that though. Joe Goldthwaite -----Original Message----- From: Jean-Francois.Doyon@CCRS.NRCan.gc.ca [mailto:Jean-Francois.Doyon@CCRS.NRCan.gc.ca] Sent: Wednesday, August 25, 2004 3:06 PM To: joe@goldthwaites.com; zope@zope.org Subject: RE: [Zope] Integrating Zope with Excel. I don't know much of anything about coding on windows, but I had looked at doing something smiliar once ... To convert PPT's on demand to HTML in my case, but I looked at using Python's COM features to do it (Regular Python + win32all, not ActiveState's). One thing that stopped me dead in my tracks was many comments form Microsoft that all the Office related COM objects are NOT re-entrant. Therefore Excel.Application can only exist once and so on, and you have to take care with such issues ... This is why nonbody's written an MS Office based Web Service :) You're not really using it as a web service, but the same concecpt applies I think? Basically, I'd look at object re-entrancy and concurency issues on the MS Office side. I ended up never needing mine, but I think I'd had it down to the fact that the best was to have the Application instance running and use Document instances to do the work ... Thing is I really didn't want to have PowerPoint instanciated permanently on the server. Anyways, just some thoughts, though I'm not sure they're of any help :) J.F. -----Original Message----- From: zope-bounces@zope.org [mailto:zope-bounces@zope.org]On Behalf Of Joe Goldthwaite Sent: August 25, 2004 5:40 PM To: zope@zope.org Subject: [Zope] Integrating Zope with Excel. Hello Again, I've been working on a Zope application and one of the tasks is to integrate the web reports with Excel. I've got some financial reports that the user can browse through using a web interface. What I'm trying to do is have a link on the page that will download the report they're viewing into a local Excel worksheet. I decided to use com because I want the worksheet to have colors, print settings, formatting, etc. I already have working code in VB. I've looked into XML and Pyxlwriter but it looks like a new learning curve compared to just translating my VB com commands into Python. I'm using Visual Studio .Net 2003 with Active Python as my development environment. When I submit the URL that creates the XL worksheet in development, it works like a top. In testing, I've generated 50 different worksheets without any problems. When I try to run the same code in Zope I have problems. Under Zope, I take the submitted URL and create a worksheet that I save to a file. Then I return a meta command to redirect the browser to the newly created file which is downloaded to the users computer. The good news is that it works perfectly - sometimes. Other times I get this error message; Module ExcelReport, line 48, in __init__ Module win32com.client, line 95, in Dispatch Module win32com.client.dynamic, line 91, in _GetGoodDispatchAndUserName Module win32com.client.dynamic, line 79, in _GetGoodDispatch com_error: (-2147221008, 'CoInitialize has not been called.', None, None) The error message is coming back from the line; XLApp = Dispatch("Excel.Application"). The really strange thing is that the error seems to be random. Sometimes it works, sometimes it doesn't. I haven't been able to determine any rhyme or reason to it. When I restart Zope, it will work one to as many as 10 times. Then I'll get the error message. If I keep trying the same URL, it might eventually work or I may have to restart Zope to get it working again. Does anyone have any idea as to what's going on here? Thanks, Joe Goldthwaite _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Joe Goldthwaite wrote:
I've never understood the threading issues - too complex for my feeble brain. I know that if one user performs a large job through Zope, it stops responding to other users until the big job is finished.
That's odd, you should look into that, it's not the norm...
I've also seen posts discussing putting Zope behind Apache so that all the requests can run independently.
That's NOT why you put Zope behind Apache ;-)
the problem. He also said that he didn't think either suggestion was a "great idea". I'm not sure what he meant by that. It's a little worrysome but it seems to be working fine now.
Ask him :-) cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On Tue, Aug 31, 2004 at 11:47:26AM +0100, Chris Withers wrote:
Joe Goldthwaite wrote:
I've never understood the threading issues - too complex for my feeble brain. I know that if one user performs a large job through Zope, it stops responding to other users until the big job is finished.
That's odd, you should look into that, it's not the norm...
I've seen the same behavior in a few situations: 1) an External Method was making an http request using httplib to a host that had become (wrongly) firewalled. I don't know how this could completely block zope, but that was the behavior I observed. A bit of using gcc as described in the "Debug a Spinning Zope" article helped me identify the ext. method in question and rewrite it so it can't possibly block like that. 2) When using ZEO with an inadequate client cache, it is possible to end up with zope apparently doing nothing but waiting for ZEO while the cache thrashes. This happened to us when somebody was downloading a very large file (more than twice the size of the client cache). A look at the event logs showed a long stream of ZEO client cache flip messages during the time in question. We drastically increased the size of the client cache and haven't had that problem since. I think that in this case, zope was still servicing requests, it was just doing so extremely slowly. When the download finished, everything instantly cleared right up. -- Paul Winkler http://www.slinkp.com
Paul Winkler wrote:
1) an External Method was making an http request using httplib to a host that had become (wrongly) firewalled. I don't know how this could completely block zope, but that was the behavior I observed. A bit of using gcc as described in the "Debug a Spinning Zope" article helped me identify the ext. method in question and rewrite it so it can't possibly block like that.
This sounds like something that should be discussed on zope-dev, but if it comes down to something C-ish blocking python's thread stuff, then ti may even end up on python-dev...
2) When using ZEO with an inadequate client cache, it is possible to end up with zope apparently doing nothing but waiting for ZEO while the cache thrashes. This happened to us when somebody was downloading a very large file (more than twice the size of the client cache). A look at the event logs showed a long stream of ZEO client cache flip messages during the time in question. We drastically increased the size of the client cache and haven't had that problem since. I think that in this case, zope was still servicing requests, it was just doing so extremely slowly. When the download finished, everything instantly cleared right up.
Yeah, ZEO's storage server is currently single threaded, so it can cause problems if abused... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On Wed, Sep 01, 2004 at 10:17:34AM +0100, Chris Withers wrote:
Paul Winkler wrote:
1) an External Method was making an http request using httplib to a host that had become (wrongly) firewalled. I don't know how this could completely block zope, but that was the behavior I observed. A bit of using gcc as described in the "Debug a Spinning Zope" article helped me identify the ext. method in question and rewrite it so it can't possibly block like that.
This sounds like something that should be discussed on zope-dev, but if it comes down to something C-ish blocking python's thread stuff, then ti may even end up on python-dev...
I don't think it's really a problem anymore: I was using either urllib or urllib2 (don't remember). At that time (python 2.1), socket.setdefaulttimeout() did not exist so any request to a socket could potentially spin forever. My theory is that, each time a request came for my external method, the current worker thread was then waiting forever for a reply from the firewalled socket. So, inevitably, sooner or later all threads were doing this and the server was effectively hung. But, I haven't been able to reproduce this behavior today with Zope 2.7.2, even with socket.setdefaulttimeout(None). I've currently got 16 pending requests for the never-finishing external method, and I can still browse zope just fine. -- Paul Winkler http://www.slinkp.com
Paul Winkler wrote:
I was using either urllib or urllib2 (don't remember). At that time (python 2.1), socket.setdefaulttimeout() did not exist so any request to a socket could potentially spin forever.
Cool, I didn't know python had finally grown one of those. Yay!
My theory is that, each time a request came for my external method, the current worker thread was then waiting forever for a reply from the firewalled socket. So, inevitably, sooner or later all threads were doing this and the server was effectively hung.
That makes sense...
But, I haven't been able to reproduce this behavior today with Zope 2.7.2, even with socket.setdefaulttimeout(None). I've currently got 16 pending requests for the never-finishing external method, and I can still browse zope just fine.
...but this doesn't. Have you upped the number to work threads to > 16? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On Thursday 02 September 2004 09:32, Chris Withers wrote:
...but this doesn't. Have you upped the number to work threads to > 16?
...Or do you have something like squid in front of zope that is combining 16 requests into 1? do you get the same results with 16 different URLs that all use this blocking method? -- Toby Dickenson
On Thu, Sep 02, 2004 at 09:46:58AM +0100, Toby Dickenson wrote:
On Thursday 02 September 2004 09:32, Chris Withers wrote:
...but this doesn't. Have you upped the number to work threads to > 16?
...Or do you have something like squid in front of zope that is combining 16 requests into 1?
Nope, test was just hitting zope on http://localhost:8080
do you get the same results with 16 different URLs that all use this blocking method?
Don't see how that would matter in the absence of squid, but i'll try... yep, no difference :-) btw, here's the external method: def pagesuck(site, page): socket.setdefaulttimeout(None) remote = urllib.urlopen('%s/%s' % (site, page)) data = remote.read() return data -- Paul Winkler http://www.slinkp.com
On Thu, Sep 02, 2004 at 09:32:54AM +0100, Chris Withers wrote:
Paul Winkler wrote:
I was using either urllib or urllib2 (don't remember). At that time (python 2.1), socket.setdefaulttimeout() did not exist so any request to a socket could potentially spin forever.
Cool, I didn't know python had finally grown one of those. Yay!
My theory is that, each time a request came for my external method, the current worker thread was then waiting forever for a reply from the firewalled socket. So, inevitably, sooner or later all threads were doing this and the server was effectively hung.
That makes sense...
But, I haven't been able to reproduce this behavior today with Zope 2.7.2, even with socket.setdefaulttimeout(None). I've currently got 16 pending requests for the never-finishing external method, and I can still browse zope just fine.
...but this doesn't. Have you upped the number to work threads to > 16?
Nope. This was a 2.7.2 server with the default 4 threads. Btw, I notice that even with socket.setdefaulttimeout(None), the requests eventually timed out after some minutes :-) Don't know how many, I didn't keep track. -- Paul Winkler http://www.slinkp.com
Paul Winkler wrote:
But, I haven't been able to reproduce this behavior today with Zope 2.7.2, even with socket.setdefaulttimeout(None). I've currently got 16 pending requests for the never-finishing external method, and I can still browse zope just fine.
...but this doesn't. Have you upped the number to work threads to > 16?
Nope. This was a 2.7.2 server with the default 4 threads.
Hmmm, then I'd hypothesise that only 4 of those threads were connected to Zope, with the other other 12 waiting to connect to Zope. Do you have any evidence which could back this up, or prove it wrong? Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On Fri, Sep 03, 2004 at 09:49:46AM +0100, Chris Withers wrote:
Paul Winkler wrote:
But, I haven't been able to reproduce this behavior today with Zope 2.7.2, even with socket.setdefaulttimeout(None). I've currently got 16 pending requests for the never-finishing external method, and I can still browse zope just fine.
...but this doesn't. Have you upped the number to work threads to > 16?
Nope. This was a 2.7.2 server with the default 4 threads.
Hmmm, then I'd hypothesise that only 4 of those threads were connected to Zope, with the other other 12 waiting to connect to Zope. Do you have any evidence which could back this up, or prove it wrong?
Doesn't sound right, because why was another browser able to browse the site just fine with no noticeable performance issues? I'm quite sure it wasn't caching, there's no squid or anything here and I was visiting pages that hadn't been visited before. Unfortunately I don't really have time to investigate this further. If I did, I'd probably set up a 2.6.2/py-2.1.3 test server and see what behavior I get there (IIRC that was the platform on which the hanging behavior was observed). -- Paul Winkler http://www.slinkp.com
On Friday 27 August 2004 18:39, Joe Goldthwaite wrote:
He also suggested that I manually call pythoncom.CoInitialize().
You need to call it once in every thread, to initialize com *for* *that* *thread*. calling it before every request is a quick way to ensure that it is called at least once. the redundant calls probably wont do any short term harm, but its bad style. Traditionally CoInitialize is called by the code that creates the thread - it doesnt belong in application code. pythoncom.CoInitialize() will initialize your threads using com's apartment threading mode, which is not 100% identical to zope's threading rules. To be safe you really should be using CoInitializeEx and some other magic parameters. This has come up many times before..... check out http://www.zopelabs.com/cookbook/993002968, or google for "dickenson zope coinitializeex" -- Toby Dickenson
participants (5)
-
Chris Withers -
Jean-Francois.Doyon＠CCRS.NRCan.gc.ca -
Joe Goldthwaite -
Paul Winkler -
Toby Dickenson