Hi Andrew, here are the answers:
-----Messaggio originale----- Da: Andrew R. Halko [mailto:ahalko@insivia.com] Inviato: mercoledì 4 giugno 2003 17.02 A: fparacchini@alteanet.it Cc: zope-dev@zope.org Oggetto: DBtab and BDBStorage
Hello Fabio,
I read your post at: http://zope.nipltd.com/public/lists/dev-archive.nsf/AGByKey/35A02E5C64B9 3519 about DBtab and BDBStorage. I am interested in doing the same thing. I have a Plone site that I expect to become very large and I need to find some other solutions than ZODB and its 2GB limit. Do you think you could help me with figuring this out? I have been reading documentation, posts and everything else and am a little confused. Here are a few questions I have from reading your post.
1. What did you need to install to make this work and did you do it after you already had a Plone/Zope site going?
There is a document at http://plone.org/Members/fparacchini/Document.2003-02-14.0631 that explains what you need and what you have to do. It was last updated mid-February, so now there can be later versions of pyBSDDB3 and DBTab. Be careful that the right combination of Python 2.1.x/pyBSDDB3/ZODB has to be carefully checked. I got results with the versions specified in that document, your mileage may vary ;)
2. What is the code that goes in DBtab.conf, I read through it about 10 times and can't understand how I need to set it up?
In DBTab.conf you have to specify a Storage that goes on BDB and a Database that is mounted on a Zope path. For example, to have the whole site on BDB the very basic configuration is: [Database: Main-BDB] mount_paths=/ open_at_startup=1 [Storage: Main-BDB] type=BDBFullStorage This way you'll create a directory below <zope>/Data/var called Main-BDB, that contains a Berkeley DB with logs.
3. Do you have to create a new DB manually every time you want to separate info, is there no way to setup so that it dynamically adds as you need space.
Well, you can also add a storage later. The procedure is a bit involved but not so much: - I suppose you already have a working DBTab configuration, like the one specified before :) - Create a folder in Zope/Plone where you will put your data, e.g. '/Docs' - Shutdown Zope/Plone - Copy the whole data directory (in the example before, it should be in C:\program files\plone\data\var\Main-BDB, copy it in C:\program files\plone\data\var\Docs-BDB) - Edit DBTab.conf, and add a Storage and a Database like this: [Database: Docs-BDB] mount_paths=/Docs open_at_startup=1 [Storage: Docs-BDB] type=BDBFullStorage - Start Zope/Plone - from management interface, delete folder /Docs - from management interface, create a new 'DBTab mount point' that should be enough: your /Docs folder should be in the new database, where it grows independently from the rest of the system. Check filesizes on both directory, to be sure that everything works. I just have to thank once again Shane Hathaway, for the countless answers on this topic. I also migrated one DB from FileStorage to BDBStorage, the python script I used, launched directly from python interpreter and not from Zope is like this: from ZODB.FileStorage import FileStorage from BDBStorage.BDBFullStorage import BDBFullStorage DBSrc = FileStorage( 'C:\temp\Data.fs', read_only=1) DBDst = BDBFullStorage( 'C:\temp\BDB-Data' ) DBDst.copyTransactionsFrom( DBSrc ) DBSrc.close() DBDst.close() You can also mix different database flavours, thanks to DBTab. In my configuration I keep 'main' database on FileStorage, to make it faster and easier to administer, and I created a couple of different DBs with BDB for folders '/Customers' and '/Members'.
4. I read a little on BerkleyDB Storage, does that give more space than ZODB?
Yes, you no longer have a 2Gb limit, as in ZODB (at least this is true on every platform, to be honest I don't know if on Linux/Unix you can overcome this limit also with ZODB).
Thanks for any help on this that you can give. There should be a tutorial on this somewhere; it seems like something a lot of people would be interested in. A 2GB limit is just horrible. Thanks!
Andrew R. Halko
I hope that this helps, let me know if you have more suggestions. Ciao Fabio
Thanks for the help!, a few more questions (lets hope). I haven't done anything yet by the way, I wanted to research my options and do some testing before I went ahead and did anything major. I did a little testing, but haven't gotten very far. The setup for Plone that I am thinking will support everything I need and work is: Main Plone site - ZODB 4gb once I install python 2.2 and enable large_file_support Plone/images/ - folder with all my images stores on Local File System Plone/files/ - folder with all my main files (word, pdf, etc.) stores on Local File System Plone/Members/ - BerkleyDB to allow a large number of users a large DB to create whatever they need in their areas Now, I am trying to setup my dbtab.conf to first test this before I try and integrate it into my current site, which has limited info now, but needs to be ready for major population in a week or two at most. Here is what I am guessing for config, maybe you can tell me if I am close and what more I need to do. [Storage: Main] type=FileStorage file_name=%(CLIENT_HOME)s/Data.fs [Database: Main] mount_paths=/ [Storage: Images] type=FileStorage basepath=%(CLIENT_HOME)s/images [Database: Images] mount_paths=/Plone/images/ container_class=OFS.Folder.Folder cache_size=4000 [Storage: Files] type=FileStorage basepath=%(CLIENT_HOME)s/files [Database: Files] mount_paths=/Plone/files/ container_class=OFS.Folder.Folder cache_size=4000 [Database: Members] mount_paths=/Plone/Members/ [Storage: Members] type=BDBFullStorage Thanks for all your help, it is very much appreciated. Andrew R. Halko
Andrew R. Halko wrote:
Now, I am trying to setup my dbtab.conf to first test this before I try and integrate it into my current site, which has limited info now, but needs to be ready for major population in a week or two at most. Here is what I am guessing for config, maybe you can tell me if I am close and what more I need to do.
Some suggestions: - Use ZEO. It makes starting/stopping Zope much faster and isolates the application from the database. - Put /Plone/portal_catalog in its own database, allowing you to tune its cache size and pack it more frequently than the rest of the data. - Watch the activity graphs in the Control_Panel to determine proper cache sizes. Note that DBTab 1.2 fixes a bug related to this. Shane
Thanks Shane. Is all of your suggestions along with the setup I was considering? I have to research ZEO, cause I know nothing about it. Do you mind explaining the last two things just a slight more as I am unfamiliar. Such as how you figure out cache and what is packing? Andrew R. Halko -----Original Message----- From: Shane Hathaway [mailto:shane@zope.com] Sent: Thursday, June 05, 2003 4:17 PM To: Andrew R. Halko Cc: 'Fabio Paracchini'; zope-dev@zope.org Subject: Re: DBtab and BDBStorage Andrew R. Halko wrote:
Now, I am trying to setup my dbtab.conf to first test this before I try and integrate it into my current site, which has limited info now, but needs to be ready for major population in a week or two at most. Here is what I am guessing for config, maybe you can tell me if I am close and what more I need to do.
Some suggestions: - Use ZEO. It makes starting/stopping Zope much faster and isolates the application from the database. - Put /Plone/portal_catalog in its own database, allowing you to tune its cache size and pack it more frequently than the rest of the data. - Watch the activity graphs in the Control_Panel to determine proper cache sizes. Note that DBTab 1.2 fixes a bug related to this. Shane
Andrew R. Halko wrote:
Thanks Shane. Is all of your suggestions along with the setup I was considering? I have to research ZEO, cause I know nothing about it. Do you mind explaining the last two things just a slight more as I am unfamiliar. Such as how you figure out cache and what is packing?
FileStorages only append until you pack. Visit the database control panel to remove old versions of objects. No wonder your database is big--since you've never packed, your database contains all revisions ever committed. Packing will help a lot. To figure out the right cache size, start with 4000 (as you have done.) Let your site run and look at the activity graph for each database after an hour of peak usage. If it's loading a lot of objects on every request, increase the cache size to use more RAM (but don't make Zope swap!) If it's hardly loading anything, decrease the cache size to give RAM back to the system. Mounting portal_catalog is a new trick I've been asking people to try. Just put it in a mounted database. Once you've done that, your main database will probably grow much more slowly than before. Shane
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday, June 6, 2003, at 06:17 AM, Shane Hathaway wrote:
- Use ZEO. It makes starting/stopping Zope much faster and isolates the application from the database.
I've noticed that this pattern doesn't hold with Zope 2.7 as it currently stands. Even on a powerful machine the ZEO clients can take quite some time to start up. Does anyone know why this might be? I'm talking here about a brand-new ZEO server with practically empty ZODB. No additional Products. Richard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (Darwin) iD8DBQE+38vorGisBEHG6TARAiG5AJ0XBcUZkMnOX/4ECk/JChEu1NUhZgCeLs+a kvn9GuVvOswbWOplJr04WIc= =xlTW -----END PGP SIGNATURE-----
Richard Jones wrote at 2003-6-6 09:01 +1000:
- Use ZEO. It makes starting/stopping Zope much faster and isolates the application from the database.
I've noticed that this pattern doesn't hold with Zope 2.7 as it currently stands. Even on a powerful machine the ZEO clients can take quite some time to start up. Does anyone know why this might be? I'm talking here about a brand-new ZEO server with practically empty ZODB. No additional Products.
Starting Zope loads several hundreds of Python modules. This may take some time. Dieter
Andrew R. Halko wrote:
Plone/Members/ - BerkleyDB to allow a large number of users a large DB to create whatever they need in their areas
I don't see why you're using BDB here. If you have a 4GB Data.fs in another part of your setup, what's the problem with just using a FileStorage for this? cheers, Chris
Well, currently I cannot get python 2.2.3 to work with Plone. I got a first error and was able to fix the problem, but then got another and have not been able to fix it for the life of me. I tried the Plone mailing list for the bug I got after installing python and got nothing. 2.2.3 will support up to 4TB, which would solve all my problems, but it doesn't work. Right now, my best solution is to scratch BDB and APE's Local Files System storage and go with 5 separate ZODBs at 2GB and then when the new version of Plone comes out, hopefully it will support Python 2.2.3 and all my DBs will have up to 4TB (pretty much unlimited) of available space. Andrew R. Halko -----Original Message----- From: Chris Withers [mailto:chrisw@nipltd.com] Sent: Tuesday, June 10, 2003 8:27 AM To: Andrew R. Halko Cc: 'Fabio Paracchini'; zope-dev@zope.org; 'Shane Hathaway' Subject: Re: [Zope-dev] RE: DBtab and BDBStorage Andrew R. Halko wrote:
Plone/Members/ - BerkleyDB to allow a large number of users a large DB to create whatever they need in their areas
I don't see why you're using BDB here. If you have a 4GB Data.fs in another part of your setup, what's the problem with just using a FileStorage for this? cheers, Chris
Andrew R. Halko wrote:
Well, currently I cannot get python 2.2.3 to work with Plone.
What do you need Python 2.2.3 for?
2.2.3 will support up to 4TB, which would solve all my problems, but it doesn't work.
I have LFS in 2.1.3, have you tried recompiling Python 2.1.3 with LFS? cheers, Chris
participants (6)
-
Andrew R. Halko -
Chris Withers -
Dieter Maurer -
Fabio Paracchini -
Richard Jones -
Shane Hathaway