RE: [Zope] SMP and Zope.
Yes, it is ... You can do it on Solaris, and Linux 2.6 kernels I believe. Like I said, I just decided to buy single CPU machines for my Zope's and void the issue altogether :) On Linux, there's a way to do it using /proc I think ... Look for "CPU affinity" on google or something like that, you should find what you need for your OS. J.F. -----Original Message----- From: zope-bounces@zope.org [mailto:zope-bounces@zope.org] On Behalf Of Tom Trelvik Sent: March 11, 2005 3:59 PM To: zope@zope.org Subject: Re: [Zope] SMP and Zope. Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote:
So if you start an instance, you should bind it to a specific CPU.
But how is this part done? I'm assuming it's OS specific? Tom _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote: | Yes, it is ... | | You can do it on Solaris, and Linux 2.6 kernels I believe. | | Like I said, I just decided to buy single CPU machines for my Zope's | and void the issue altogether :) | | On Linux, there's a way to do it using /proc I think ... | | Look for "CPU affinity" on google or something like that, you should | find what you need for your OS. It is our experience in hosting Zope on SMP Linux boxen (2.4 series kernles) that the CPU scheduler will tend to migrate the Python processes to separate CPUs, without need for explicit binding. Tres. - -- =============================================================== Tres Seaver tseaver@zope.com Zope Corporation "Zope Dealers" http://www.zope.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCMhGVGqWXf00rNCgRAovNAJ9/UTnAHlNYLPYW877aFq2gzeDyuwCfQ3y2 Zco4LNMQLhxJXqkzwKqQZLo= =0aNU -----END PGP SIGNATURE-----
Tres Seaver wrote:
It is our experience in hosting Zope on SMP Linux boxen (2.4 series kernles) that the CPU scheduler will tend to migrate the Python processes to separate CPUs, without need for explicit binding.
I don't suppose anyone has similar (or contrary) anecdotes from FreeBSD (5.3) servers? Thanks, Tom (who admittedly hasn't even googled for it yet, but since we're sharing, I figured I'd ask)
Tres Seaver wrote:
It is our experience in hosting Zope on SMP Linux boxen (2.4 series kernles) that the CPU scheduler will tend to migrate the Python processes to separate CPUs, without need for explicit binding.
Hi Tres, So, Do I create 4 instances of Zope to use with a dual Xeon HT? :-) Thanks. Fernando Lujan
On Fri, 2005-03-11 at 17:05, Fernando Lujan wrote:
Tres Seaver wrote:
It is our experience in hosting Zope on SMP Linux boxen (2.4 series kernles) that the CPU scheduler will tend to migrate the Python processes to separate CPUs, without need for explicit binding.
Hi Tres,
So, Do I create 4 instances of Zope to use with a dual Xeon HT? :-)
I don't know that there's a ready-made answer for this. You can create as many instances as you want on a single machine, pretty much, but for better or worse, it will be up to you to figure out how many is the "right number on any given hardware. - C
FWIW, at a former company we were running something like 20 Zope instances on a quad Xeon machine with impressive results. No "hanging" or anything like it to report. Note that an important tuning point is the system's checkinterval. Once we had this set to a high enough value CPU binding proved completely unnecessary and even counterproductive, YMMV. Stefan On 11. Mär 2005, at 23:05, Fernando Lujan wrote:
Tres Seaver wrote:
It is our experience in hosting Zope on SMP Linux boxen (2.4 series kernles) that the CPU scheduler will tend to migrate the Python processes to separate CPUs, without need for explicit binding.
Hi Tres,
So, Do I create 4 instances of Zope to use with a dual Xeon HT? :-)
-- Software Engineering is Programming when you can't. --E. W. Dykstra
Stefan H. Holek wrote:
FWIW, at a former company we were running something like 20 Zope instances on a quad Xeon machine with impressive results. No "hanging" or anything like it to report. Note that an important tuning point is the system's checkinterval. Once we had this set to a high enough value CPU binding proved completely unnecessary and even counterproductive, YMMV.
Stefan
This is interesting - how high was high enough, and how did you test that this was a proper setting? Thanks, /dario -- -- ------------------------------------------------------------------- Dario Lopez-Kästen, IT Systems & Services Chalmers University of Tech. "...and click? damn, I need to kill -9 Word again..." - b using macosx
Rule of thumb is: checkinterval = pystones / 50 Stefan On 13. Mär 2005, at 14:18, Dario Lopez-Kästen wrote:
Stefan H. Holek wrote:
FWIW, at a former company we were running something like 20 Zope instances on a quad Xeon machine with impressive results. No "hanging" or anything like it to report. Note that an important tuning point is the system's checkinterval. Once we had this set to a high enough value CPU binding proved completely unnecessary and even counterproductive, YMMV. Stefan
This is interesting - how high was high enough, and how did you test that this was a proper setting?
Thanks,
/dario
-- Software Engineering is Programming when you can't. --E. W. Dykstra
On Mar 15, 2005, at 11:31 AM, Stefan H. Holek wrote:
Rule of thumb is: checkinterval = pystones / 50
Before everyone goes adjusting their zope.conf's python-check-interval, I think I should point out that the configuration option is ignored in versions of Zope 2.7 previous to the Zope-2.7.5 beta release. In Zope 2.7.0 through 2.7.4, you are getting Python 2.3's default checkinterval of 100 no matter what your zope.conf file says.
Right, this does not however keep you from putting import sys sys.setcheckinterval(1000) someplace where it *does* get picked up... Stefan On 15. Mär 2005, at 17:48, Andrew Langmead wrote:
On Mar 15, 2005, at 11:31 AM, Stefan H. Holek wrote:
Rule of thumb is: checkinterval = pystones / 50
Before everyone goes adjusting their zope.conf's python-check-interval, I think I should point out that the configuration option is ignored in versions of Zope 2.7 previous to the Zope-2.7.5 beta release. In Zope 2.7.0 through 2.7.4, you are getting Python 2.3's default checkinterval of 100 no matter what your zope.conf file says.
-- Software Engineering is Programming when you can't. --E. W. Dykstra
Andrew Langmead wrote:
On Mar 15, 2005, at 11:31 AM, Stefan H. Holek wrote:
Rule of thumb is: checkinterval = pystones / 50
Before everyone goes adjusting their zope.conf's python-check-interval, I think I should point out that the configuration option is ignored in versions of Zope 2.7 previous to the Zope-2.7.5 beta release. In Zope 2.7.0 through 2.7.4, you are getting Python 2.3's default checkinterval of 100 no matter what your zope.conf file says.
Just a last question: Actually, what does checkinterval do? Thanks. Fernando Lujan
On 18 Mar 2005 at 20:24, Fernando Lujan wrote:
Actually, what does checkinterval do?
Hi. I am creating a product on the filesystem. Outside of the skins that contain generic scripts and templates, I am organizing a series of folders and their contents in the same hierarchy that I to want it appear in my CMF site. Some of these folders are two or three levels deep. Can someone suggest a method to recursively read the folders and files from the product and move them to the site during install (or alternatively point me to an existing product install script that demonstrates this). Many thanks. David
David Pratt wrote:
Hi. I am creating a product on the filesystem. Outside of the skins that contain generic scripts and templates, I am organizing a series of folders and their contents in the same hierarchy that I to want it appear in my CMF site. Some of these folders are two or three levels deep. Can someone suggest a method to recursively read the folders and files from the product and move them to the site during install (or alternatively point me to an existing product install script that demonstrates this).
Many thanks. David _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
your product can ask zope where it is installed like: product_path = globals()['__builtins__']['INSTANCE_HOME'] + '/Products' robert
robert wrote at 2005-3-19 14:39 +0100: ....
your product can ask zope where it is installed like: product_path = globals()['__builtins__']['INSTANCE_HOME'] + '/Products'
You can shorten this to product_path = INSTANCE_HOME + '/Products' This works for any builtin (not just "INSTANCE_HOME"). -- Dieter
robert wrote:
your product can ask zope where it is installed like: product_path = globals()['__builtins__']['INSTANCE_HOME'] + '/Products'
As Dieter pointed out, this can easily be shorted to just INSTANCE_HOME+'/Products', but that doesn't mean it'll be correct ;-) Your best bet is DirectoryViews, but if you do ever want ot know the file system path of a Product, then put something like the following in the Product's __init__.py: import os.path my_path = os.path.abspath(os.path.split(__file__)[0]) ...and then you can get it from elsewhere like: from Products.MyProduct import my_path cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Am Montag, den 21.03.2005, 08:12 +0000 schrieb Chris Withers:
robert wrote:
your product can ask zope where it is installed like: product_path = globals()['__builtins__']['INSTANCE_HOME'] + '/Products'
As Dieter pointed out, this can easily be shorted to just INSTANCE_HOME+'/Products', but that doesn't mean it'll be correct ;-)
Your best bet is DirectoryViews, but if you do ever want ot know the file system path of a Product, then put something like the following in the Product's __init__.py:
import os.path my_path = os.path.abspath(os.path.split(__file__)[0])
os.path.dirname(os.path.abspath(__file__)) please ;)
-- Tino Wildenhain <tino@wildenhain.de>
Tino Wildenhain wrote:
import os.path my_path = os.path.abspath(os.path.split(__file__)[0])
os.path.dirname(os.path.abspath(__file__))
please ;)
why? Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On 18 Mar 2005 at 22:43, David Pratt wrote:
Hi. I am creating a product on the filesystem. Outside of the skins that contain generic scripts and templates, I am organizing a series of folders and their contents in the same hierarchy that I to want it appear in my CMF site. Some of these folders are two or three levels deep. Can someone suggest a method to recursively read the folders and files from the product and move them to the site during install (or alternatively point me to an existing product install script that demonstrates this).
First off, why develop your files on the file system? Why not develop them in Zope and then publish them by exporting a .zexp file? If you must move files back and forth between the file system and Zope, then you could use FTP. Most decent FTP clients will allow a recursive up/download. Lastly, if you must use the file system and you need an installer instead of a manual process, then look into Python's ftplib module. It won't do the work for you, but it's a pretty easy module to use. Gre7g
First off, why develop your files on the file system? Why not develop them in Zope and then publish them by exporting a .zexp file?
Huh? There's a lot of very good reasons why some content/logic is *way* better off on the file system, especially content/logic that rarely changes. Think "real text editors" or "CVS/SVN code management" or "grep and other great utilities" or "really simple deployment". jens
David Pratt wrote at 2005-3-18 22:43 -0400:
I am creating a product on the filesystem. Outside of the skins that contain generic scripts and templates, I am organizing a series of folders and their contents in the same hierarchy that I to want it appear in my CMF site.
You use "os.listdir" and "os.path.isdir" on the filesystem side and "manage_addFolder" for directories and appropriate other constructors for files on the ZODB side. Or you use a "DirectoryView" (in case, your file hiearchy does not contain content but templates and scripts). -- Dieter
Thank you Dieter and others for responding. I was looking at doing something with a class and os.path.walk to do something like this - if I could not find something already in use but put a hold on this to look at the DirectoryView idea more closely. I have only ever seen a DirectoryView used in skins such as; from Products.CMFCore import DirectoryView DirectoryView.registerDirectory('skins/my_scripts_and_templates', globals()) I like very much the idea of what you have suggested and hadn't really considered a DirectoryView in the site root before. I guess as long as you register the directory you can create a directory view pretty much anywhere - which is interesting and at the same time similar to products out there like LocalFS. In my situation, the folder hierarchy and templates are responsible for giving the site its path structure. I also have generic templates in a skins folder with most scripts to deal with the normal things the site will have to do. So it was important to keep the heirarchy of folders and templates in tact. I did not want to simply create a folder in skins that would get any of its templates through acquisition. DirectoryView is recursive and gets everything under it just fine. On top of it is the real benefit that to make any future changes to this base material will only ever need do be done in the product. The only issue real issue for me is that there are some properties that I need to maintain on the folders as well for certain types of functionality. So this is where I may be hooped. DirectoryView doesn't appear to allow any properties on the folder other than its path. I guess the question is there a way to add to the properties of a DirectoryView folder?? from the filesystem. I have found a reference to a FSPropertyObject but have yet to find any examples of using it for something like this. I have string, boolean, int, lines and text properties that are folder specific. Regards, David On Saturday, March 19, 2005, at 03:03 PM, Dieter Maurer wrote:
David Pratt wrote at 2005-3-18 22:43 -0400:
I am creating a product on the filesystem. Outside of the skins that contain generic scripts and templates, I am organizing a series of folders and their contents in the same hierarchy that I to want it appear in my CMF site.
You use "os.listdir" and "os.path.isdir" on the filesystem side and "manage_addFolder" for directories and appropriate other constructors for files on the ZODB side.
Or you use a "DirectoryView" (in case, your file hiearchy does not contain content but templates and scripts).
-- Dieter
David Pratt wrote:
functionality. So this is where I may be hooped. DirectoryView doesn't appear to allow any properties on the folder other than its path.
Sure it does. Google for .metadata files in the context of DirectoryView ;-) cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On Mar 21, 2005, at 2:37, David Pratt wrote:
The only issue real issue for me is that there are some properties that I need to maintain on the folders as well for certain types of functionality. So this is where I may be hooped. DirectoryView doesn't appear to allow any properties on the folder other than its path.
Not true for CMF CVS-HEAD. You can add .metadata files for folders in that version. jens
Hi Jens. This sounds promising. Is this the HEAD of 1.4 or 1.5? If I check this out, I guess I am going to need to upgrade my Zope. I am on 2.7.3 at the moment. On Monday, March 21, 2005, at 04:52 AM, Jens Vagelpohl wrote:
On Mar 21, 2005, at 2:37, David Pratt wrote:
The only issue real issue for me is that there are some properties that I need to maintain on the folders as well for certain types of functionality. So this is where I may be hooped. DirectoryView doesn't appear to allow any properties on the folder other than its path.
Not true for CMF CVS-HEAD. You can add .metadata files for folders in that version.
jens
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
On Mar 21, 2005, at 11:32, David Pratt wrote:
Hi Jens. This sounds promising. Is this the HEAD of 1.4 or 1.5? If I check this out, I guess I am going to need to upgrade my Zope. I am on 2.7.3 at the moment.
It's HEAD directly, meaning the trunk, or what's eventually going to turn into CMF 1.6. Get it this way: cvs -d :pserver:anonymous@cvs.zope.org:/cvs-repository co CMF Speaking of 2.7.3, now that 2.7.5 is out you should upgrade, anyway. It fixed some access control problems that led to weird security bugs. The upgrade is painless. The way I upgrade is very simple: - download the new Zope and store it separately from the old Zope - go though the build process without creating a new INSTANCE_HOME - stop the existing site - edit all files in the existing INSTANCE_HOME/etc and INSTANCE_HOME/bin and do a simple search-and-replace to replace references to the old Zope path with the new path. - restart the site jens
Hi jens. This is totally helpful. I am just in the process of upgrading to 2.7.5 now. Many thanks, David On Monday, March 21, 2005, at 07:01 AM, Jens Vagelpohl wrote:
On Mar 21, 2005, at 11:32, David Pratt wrote:
Hi Jens. This sounds promising. Is this the HEAD of 1.4 or 1.5? If I check this out, I guess I am going to need to upgrade my Zope. I am on 2.7.3 at the moment.
It's HEAD directly, meaning the trunk, or what's eventually going to turn into CMF 1.6. Get it this way:
cvs -d :pserver:anonymous@cvs.zope.org:/cvs-repository co CMF
Speaking of 2.7.3, now that 2.7.5 is out you should upgrade, anyway. It fixed some access control problems that led to weird security bugs. The upgrade is painless. The way I upgrade is very simple:
- download the new Zope and store it separately from the old Zope - go though the build process without creating a new INSTANCE_HOME - stop the existing site - edit all files in the existing INSTANCE_HOME/etc and INSTANCE_HOME/bin and do a simple search-and-replace to replace references to the old Zope path with the new path. - restart the site
jens
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
The side effect of upgrading to HEAD at this point is a few broken products that can't find CMFCorePermissons from CMFCore Where did this go? Regards, David On Monday, March 21, 2005, at 07:01 AM, Jens Vagelpohl wrote:
On Mar 21, 2005, at 11:32, David Pratt wrote:
Hi Jens. This sounds promising. Is this the HEAD of 1.4 or 1.5? If I check this out, I guess I am going to need to upgrade my Zope. I am on 2.7.3 at the moment.
It's HEAD directly, meaning the trunk, or what's eventually going to turn into CMF 1.6. Get it this way:
cvs -d :pserver:anonymous@cvs.zope.org:/cvs-repository co CMF
Speaking of 2.7.3, now that 2.7.5 is out you should upgrade, anyway. It fixed some access control problems that led to weird security bugs. The upgrade is painless. The way I upgrade is very simple:
- download the new Zope and store it separately from the old Zope - go though the build process without creating a new INSTANCE_HOME - stop the existing site - edit all files in the existing INSTANCE_HOME/etc and INSTANCE_HOME/bin and do a simple search-and-replace to replace references to the old Zope path with the new path. - restart the site
jens
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
On Mar 21, 2005, at 14:22, David Pratt wrote:
The side effect of upgrading to HEAD at this point is a few broken products that can't find CMFCorePermissons from CMFCore
Where did this go?
CMFCore.CMFCorePermissions is now CMFCore.permissions I suppose whatever products you run could stand an upgrade, too. And you better make sure you test everything thoroughly before migrating your live site. jens
Ok. Thanks for this fine piece of knowledge and good advice. I am pretty current on most all. I'll just have to change a couple of statements here and there is what it looks like. Regards, David On Monday, March 21, 2005, at 09:26 AM, Jens Vagelpohl wrote:
On Mar 21, 2005, at 14:22, David Pratt wrote:
The side effect of upgrading to HEAD at this point is a few broken products that can't find CMFCorePermissons from CMFCore
Where did this go?
CMFCore.CMFCorePermissions is now CMFCore.permissions
I suppose whatever products you run could stand an upgrade, too. And you better make sure you test everything thoroughly before migrating your live site.
jens
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
participants (15)
-
Andrew Langmead -
Chris McDonough -
Chris Withers -
Dario Lopez-Kästen -
David Pratt -
Dieter Maurer -
Fernando Lujan -
Gre7g Luterman -
Jean-Francois.Doyon＠CCRS.NRCan.gc.ca -
Jens Vagelpohl -
robert -
Stefan H. Holek -
Tino Wildenhain -
Tom Trelvik -
Tres Seaver