Running methods on Zope startup.
Is there a Zope product out there which implements a "Startup" like folder for Zope? That is, I want to be able to have a folder like object called "Startup" in the root directory, and when Zope is first started I want Zope to automatically make a call against the "Startup" object which could then in turn run methods of any user defined objects which have been added into the "Startup" folder to perform special processing only when Zope is started. Imagine a situation similar to database connections. With database connections when you create it you can indicate that you want the connection established. Now when Zope is restarted, those database connections will be automatically restarted when Zope restarts. Although, I am not sure whether this is when Zope actually starts or when the next request using that database happens. Anyway, I ultimately want to be able to have a Zope product which has a persistent interprocess messaging connection to another process. Although when first added into Zope the connection can be initiated, I know of no way to have Zope restart that connection when it restarts. I can't rely on the connection only starting when someone tries to access the resource, I want it to happen straight away upon Zope restart. When I thought about it, I figured the most basic thing to do would be to have a Zope "Startup" folder as described in a known place. I could then stick in that folder, methods to be triggered at startup which could startup the interprocess communications link for me. This seemed better than having a startup mechanism specific to what I was doing as it could be used for other things as well. Can anyone point me to such a Zope product. I have started looking at how to do it myself and have some of the bits in place. The first issue is that when the "initialize()" method in the product "__init__.py" file is called not all Zope startup has occured, so it is too early to trigger things from that point. Best I could come up with there was to register a callback to asyncore to be called when asyncore mainloop is actually run, which is after all Zope startup has been done. The next problem is getting an appropriate context to be able to start making calls into Zope objects. The "initialize()" method gets a product context, but doesn't have public access to the application context. Is there some way of accessing the Zope application context and thus root folder where you don't otherwise have a handle to it? Can Main.bobo_application be used? Even if the application context is obtained, is it safe to be making calls into Zope objects from a callback from asyncore? Is there a better way? I am very new to Zope and this is my first adventure into making a Zope product and thus could be totally off track. If anyone understands what I am trying to do and can suggest how to do it or point at something that does what I want, then even better. Personally I would be surprised if this hadn't already been done or even that there is some inbuilt way of doing it now, as it opens up all sorts of interesting possibilities. Appreciated if responses are cc'd to me as well as going to the list as I am not a member of the list yet and am using list archives. Thanks in advance. -- Graham Dumpleton (grahamd@dscpl.com.au)
Something like this is slated for Zope 2.6. If possible (and that's a big if at the moment, with the work we've got lined up), it is my intention to create a service which does the following: - Searches for an appropriately-named on-disk directory (var/startup probably). - Iterates over the contents of the directory, finding all .py files and turns them into python modules. - Searches each module for a well-known top-level function (named "onstart" or something similar) and calls the function with a single argument, which will be the Zope "application" (root) object. - The code does whatever arbitrary thing that it needs to do, using the application object as a gateway with which it can manipulate persistently-stored objects in the ZODB if necessary. - Optionally, the code deletes itself (if this is a run-once sort of thing) or places a marker in the database (or on the filesystem) that says it should not run again at startup. - The code finishes. The two things that I think are different about this than what you specified is that in my conception, the startup folder is not a persistent Zope object, and you would need to place files in a directory on disk to hook the startup process. It also solves the problem of needing to have a handle to I think your idea of registering a callback to ensure that startup is complete before performing any actions is very sound. I think a callback can be registered almost anywhere, but a plausible place to create the registration method might be in lib/python/OFS/Application.py. Then the callback should probably be called by lib/python/Zope/__init__.py (somewhere after "OFS.Application.initialize(c)", after which point all Products should be initialized). This is a little brittle, though. A better way to do this would be to create a generic Zope scheduling module. This is where hooking the asyncore mainloop would be very appropriate. Unfortunately, the implementation is a bit more complex because you'd probably need to make sure the scheduler didn't hold up the mainloop, and that means kicking off one or more threads to do the work instead of using the main thread just at startup. However, it would be much more generally useful and forward-thinking to implement it this way, even if it was a stub that didn't actually do anything but startup stuff at first. If I were to do this, I would: - Implement a stub scheduler module that was really dumb and only knew how to do "run once, now" jobs (no run-many jobs, no run-then jobs). Do this so as to not go insane trying to reinvent cron and at at the moment. ;-) - Implement a non-Product "Startup" module that did what I explain above, but registered a callback with the scheduler instead of with the method called by Zope/__init__. - Change medusa's asyncore so that the "poll" (and poll2, and poll3) function kept track of when the scheduler had last been run. - In asyncore, when either the select call times out or the last time the schduler was run was over "X" seconds in the past, call in to the scheduler, telling it to wake up, modifying the "last run" time. - The (dumb) scheduler would wake up and notice that it had a job and would kick off a thread which would call back in to the startup code. Then it would remove the job from its queue and go on with its life, which would be absolutely desolate until the next time Zope started up. This architecture would facilitate innovations (like a *real* scheduling facility ;-) in the future. HTH, - C ----- Original Message ----- From: <grahamd@dscpl.com.au> To: <zope-dev@zope.org> Sent: Wednesday, April 24, 2002 11:56 PM Subject: [Zope-dev] Running methods on Zope startup.
Is there a Zope product out there which implements a "Startup" like folder for Zope? That is, I want to be able to have a folder like object called "Startup" in the root directory, and when Zope is first started I want Zope to automatically make a call against the "Startup" object which could then in turn run methods of any user defined objects which have been added into the "Startup" folder to perform special processing only when Zope is started.
Imagine a situation similar to database connections. With database connections when you create it you can indicate that you want the connection established. Now when Zope is restarted, those database connections will be automatically restarted when Zope restarts. Although, I am not sure whether this is when Zope actually starts or when the next request using that database happens.
Anyway, I ultimately want to be able to have a Zope product which has a persistent interprocess messaging connection to another process. Although when first added into Zope the connection can be initiated, I know of no way to have Zope restart that connection when it restarts. I can't rely on the connection only starting when someone tries to access the resource, I want it to happen straight away upon Zope restart.
When I thought about it, I figured the most basic thing to do would be to have a Zope "Startup" folder as described in a known place. I could then stick in that folder, methods to be triggered at startup which could startup the interprocess communications link for me. This seemed better than having a startup mechanism specific to what I was doing as it could be used for other things as well.
Can anyone point me to such a Zope product. I have started looking at how to do it myself and have some of the bits in place. The first issue is that when the "initialize()" method in the product "__init__.py" file is called not all Zope startup has occured, so it is too early to trigger things from that point. Best I could come up with there was to register a callback to asyncore to be called when asyncore mainloop is actually run, which is after all Zope startup has been done.
The next problem is getting an appropriate context to be able to start making calls into Zope objects. The "initialize()" method gets a product context, but doesn't have public access to the application context. Is there some way of accessing the Zope application context and thus root folder where you don't otherwise have a handle to it? Can Main.bobo_application be used?
Even if the application context is obtained, is it safe to be making calls into Zope objects from a callback from asyncore? Is there a better way?
I am very new to Zope and this is my first adventure into making a Zope product and thus could be totally off track. If anyone understands what I am trying to do and can suggest how to do it or point at something that does what I want, then even better.
Personally I would be surprised if this hadn't already been done or even that there is some inbuilt way of doing it now, as it opens up all sorts of interesting possibilities.
Appreciated if responses are cc'd to me as well as going to the list as I am not a member of the list yet and am using list archives.
Thanks in advance.
-- Graham Dumpleton (grahamd@dscpl.com.au)
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Chris McDonough wrote:
[snip]
There is product ZExternalNews which is working in background from Zope startup. You can take a look there. There are several Startup issues in the code. Regards, Myroslav -- Myroslav Opyr zope.net.ua <http://zope.net.ua/> ° Ukrainian Zope Hosting e-mail: myroslav@zope.net.ua <mailto:myroslav@zope.net.ua> cell: +380 50.3174578
On Apr 25 22:19, Myroslav Opyr <myroslav@zope.net.ua> wrote:
Subject: Re: [Zope-dev] Running methods on Zope startup.
Chris McDonough wrote:
[snip]
There is product ZExternalNews which is working in background from Zope startup. You can take a look there. There are several Startup issues in the code.
Yes, this product is doing a similar thing to what I want to do. I note though that it does: app=context._ProductContext__app Ie., it breaks abstraction and accesses the private application context which is available in the product context. Yes, it is possible, but it is (is it) cheating. :-) Is there any reason why the application context in the product context just can't be public in the first place? It would make things easier and doesn't seem that big a deal to change it. At least Zope could provide a public method which gives access to it. I also note that this product does what it needs direct from the product initialize routine. As long as this product doesn't rely in anyway on other products this would be okay, but if it did, it could be a problem as threads created by the product might try to access another product or even part of Zope which hadn't been initialised yet. Anyway, I might just break the encapsulation in the same way for now, will save a lot of work. Thanks for the pointer to this package. -- Graham Dumpleton (grahamd@dscpl.com.au)
On Apr 25 00:57, "Chris McDonough" <chrism@zope.com> wrote:
Subject: Re: [Zope-dev] Running methods on Zope startup.
Something like this is slated for Zope 2.6. If possible (and that's a big if at the moment, with the work we've got lined up), it is my intention to create a service which does the following:
...
The two things that I think are different about this than what you specified is that in my conception, the startup folder is not a persistent Zope object, and you would need to place files in a directory on disk to hook the startup process. It also solves the problem of needing to have a handle to
The important thing as far as I can see is that the hook is at least there in such a way that the application context is available. As long as one has that, it would then be very easy to add in a method which went looking for the "Startup" folder in the database and went from there.
I think your idea of registering a callback to ensure that startup is complete before performing any actions is very sound. I think a callback can be registered almost anywhere, but a plausible place to create the registration method might be in lib/python/OFS/Application.py. Then the callback should probably be called by lib/python/Zope/__init__.py (somewhere after "OFS.Application.initialize(c)", after which point all Products should be initialized).
Yes, worked out where it could be done, for now I was trying to work out how it might be done without modifying the Zope code.
This is a little brittle, though. A better way to do this would be to create a generic Zope scheduling module. This is where hooking the asyncore mainloop would be very appropriate. Unfortunately, the implementation is a bit more complex because you'd probably need to make sure the scheduler didn't hold up the mainloop, and that means kicking off one or more threads to do the work instead of using the main thread just at startup. However, it would be much more generally useful and forward-thinking to implement it this way, even if it was a stub that didn't actually do anything but startup stuff at first.
As long as one has the startup hook, a scheduler can then be a Zope product. Steps would then be for someone to install "Startup" product and if need be manually register callback in internal Zope startup directory to execute "Startup" folder in known location. Someone installing the "Scheduler" product would then register instance of it with "Startup", or it could even be automatic in some way by using registration ability of ZCatalog. In respect of a separate thread, I agree. I actually already have a full scheduler/dispatcher written I can hook in. It actually has its own select loop and thus must be run in a separate thread. It even has mechanism to be woken up from select if new job was scheduled to be running but dispatcher was otherwise blocking. The only issue with this separate dispatcher is that the core is actually written in C++ and isn't pure Python but relying on Python wrappers that already exist.
If I were to do this, I would:
- Implement a stub scheduler module that was really dumb and only knew how to do "run once, now" jobs (no run-many jobs, no run-then jobs). Do this so as to not go insane trying to reinvent cron and at at the moment. ;-)
I already have the cron code in the dispatcher system I have. :-)
- Implement a non-Product "Startup" module that did what I explain above, but registered a callback with the scheduler instead of with the method called by Zope/__init__.
- Change medusa's asyncore so that the "poll" (and poll2, and poll3) function kept track of when the scheduler had last been run.
How is that important if it is run in a separate thread?
- In asyncore, when either the select call times out or the last time the schduler was run was over "X" seconds in the past, call in to the scheduler, telling it to wake up, modifying the "last run" time.
Hmmm, still not sure why, but then the dispatcher code I have probably has a much more developed job model than asyncore does. Not only does it support i/o activity, timeouts, alarms, cron alarms, it also has three different types of jobs, ie., priority, standard and idle. Add to that that it has a separate thread, not sure why they would need to communicate.
- The (dumb) scheduler would wake up and notice that it had a job and would kick off a thread which would call back in to the startup code. Then it would remove the job from its queue and go on with its life, which would be absolutely desolate until the next time Zope started up.
This architecture would facilitate innovations (like a *real* scheduling facility ;-) in the future.
Which is what I am interested in now. Right now though, I am more interested in what is using the dispatcher in parallel to Zope as opposed to scheduling persistent methods in Zope. Rather get something working first before I go into it further. ;-) Anyway, you have given me some ideas. Knowing that this is sort of planned at least might at least get me motivated to hack in the Zope code a bit to get something working and try out a few ideas. -- Graham Dumpleton (grahamd@dscpl.com.au)
On Wed, 24 Apr 2002 21:56:52 -0600 (MDT), grahamd@dscpl.com.au wrote:
Imagine a situation similar to database connections. With database connections when you create it you can indicate that you want the connection established. Now when Zope is restarted, those database connections will be automatically restarted when Zope restarts. Although, I am not sure whether this is when Zope actually starts or when the next request using that database happens.
The connection is made when the next request comes in, and the connection is broken when ZODB decides it is time to deactivate the DA object; which is hopefully only happens if the connection has not been used for a while.
I can't rely on the connection only starting when someone tries to access the resource, I want it to happen straight away upon Zope restart.
I think you will have to fight ZODB if you need: * The connection to always stay up * one connection per instance of a class stored in ZODB (like database adapters)
Can anyone point me to such a Zope product. I have started looking at how to do it myself and have some of the bits in place. The first issue is that when the "initialize()" method in the product "__init__.py" file is called not all Zope startup has occured, so it is too early to trigger things from that point.
Initialize looks like the right place to me. What do you think this is too early for? One thing that might be nice to add is *dependencies* between product initialization. Today you have to rely on Zope sorting product names alphabetically before initialization, which slightly sucks.
The next problem is getting an appropriate context to be able to start making calls into Zope objects. The "initialize()" method gets a product context, but doesn't have public access to the application context. Is there some way of accessing the Zope application context and thus root folder where you don't otherwise have a handle to it?
Public access? no. currently you have to hack through the jar. I agree there should be an API for this. Toby Dickenson tdickenson@geminidataloggers.com
On Apr 25 10:02, Toby Dickenson <tdickenson@devmail.geminidataloggers.co.uk> wrote:
Subject: Re: [Zope-dev] Running methods on Zope startup.
On Wed, 24 Apr 2002 21:56:52 -0600 (MDT), grahamd@dscpl.com.au wrote:
Imagine a situation similar to database connections. With database connections when you create it you can indicate that you want the connection established. Now when Zope is restarted, those database connections will be automatically restarted when Zope restarts. Although, I am not sure whether this is when Zope actually starts or when the next request using that database happens.
The connection is made when the next request comes in, and the connection is broken when ZODB decides it is time to deactivate the DA object; which is hopefully only happens if the connection has not been used for a while.
Ok, wasn't quite sure which way it did it.
I can't rely on the connection only starting when someone tries to access the resource, I want it to happen straight away upon Zope restart.
I think you will have to fight ZODB if you need: * The connection to always stay up * one connection per instance of a class stored in ZODB (like database adapters)
On the latter point of searching automatically for all instances of a particular type and automatically starting up something for it, that was my first thought and in the end I figured that could get real inefficient. That is when I came to the lowest common denominator of a "Startup" folder and figured that a manual association could be made. At least an association with a product instance which could then do the hard work. Anyway, I know I will potentially have to fight ZODB though, as I can see lots of instances where I will need to recognise existance of objects in ZODB and create parallel transient objects which do something in relation to those instances. Given I am new to Zope this might turn out to be messier than I think it will.
Can anyone point me to such a Zope product. I have started looking at how to do it myself and have some of the bits in place. The first issue is that when the "initialize()" method in the product "__init__.py" file is called not all Zope startup has occured, so it is too early to trigger things from that point.
Initialize looks like the right place to me. What do you think this is too early for?
Not all products might have been initialized at that point, plus possibly other stuff as well in Zope itself. The asyncore loop also isn't running at that point. For reasons unknown I have also already found that if I create some separate code of my own from a thread started in the initialize routine, that even though the main thread of control returned back into Zope to finishing doing everything and start up asyncore, Zope wouldn't then service any requests. Was most strange, would only work if I created the separate thread from callback in asyncore once it had been started.
One thing that might be nice to add is *dependencies* between product initialization. Today you have to rely on Zope sorting product names alphabetically before initialization, which slightly sucks.
This is an issue with the idea of a "Startup" folder as well as what order to you execute register methods. First thought was to have ability to say that method relied on other method being run first, but then figure that for simplicity, might be easier to let user indicate specific order they were run through management interface.
The next problem is getting an appropriate context to be able to start making calls into Zope objects. The "initialize()" method gets a product context, but doesn't have public access to the application context. Is there some way of accessing the Zope application context and thus root folder where you don't otherwise have a handle to it?
Public access? no. currently you have to hack through the jar.
I agree there should be an API for this.
But then, as suggested by someone else, if a startup mechanism is inbuilt the hook would be placed in the Application.py file which has access to the application context anyway. Feedback appreciated. -- Graham Dumpleton (grahamd@dscpl.com.au)
participants (4)
-
Chris McDonough -
grahamd＠dscpl.com.au -
Myroslav Opyr -
Toby Dickenson