BTreeFolder for "huge" ZODB (was : populating a "huge" ZODB)
Thank you very much Igor, however I am still a bit afraid because I see a scrolling window for the products contrained by the BTreeFolder on the product homepage (http://www.zope.org/Members/hathawsh/BTreeFolder). In the exmple given (4096 items), it could still be OK, but I think having 1,000,000 + objects might be a problem. There is an interface problem here : the contents of such a folder can only be reasonably displayed in successive batches. (in fact it is also another aspect of the problem of having thousands of objects in an ordinary folder : if you click on the folder to open it, the Management interface will hang) Pierre
On Fre, 2002-06-21 at 11:57, Pierre Godefroy wrote: [ huge folder ] Have a look at BTreeFolder, the whole structure is stored in a btree, so a lookup for an id should be pretty fast... greetings, Igor
--
Hi everybody,
How would you populate a "huge" ZODB with hundreds of thousands of objects which have no specific reason to be hierarchically distributed in a complex tree of folders and sub-folders?
They could in fact all be in the same folder. It seems to be OK for hundreds of objects, but afterwards there is an incremental cost : each time you add a new object, there is a check on the existence of an identical ID before adding/creating it. When you reach the level of already existing thousands/tens of thousands/hundreds of thousands objects in the same folder it seems you are in serious trouble...
I know the question is a bit naive, and that usually you have a data structure which dictates a naturally "hierarchy" of folders which means that no folder would contain maybe more than a few hundred objects or sub-folders.
Maybe I should "artificially" design an ID structure which would correspond to a hierarchy : for instance ojects would be identified by numbers with 7 digits, each digit corresponding to a lower level in the hierarchy? It means that an object with ID 2343789 would be placed in the "/2/3/4/3/7/8/9" sub-folder.
What do you think? Any other idea?
Pierre Godefroy 52 rue des Archives 75004 PARIS FRANCE Tél. : +33 (0)1 42 74 46 05
If I understand you right then you want to use ZODB as thé DB for an application? \Oliver
-----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Pierre Godefroy Sent: 21. juni 2002 12:37 To: zope@zope.org Subject: [Zope] BTreeFolder for "huge" ZODB (was : populating a "huge" ZODB)
Thank you very much Igor, however I am still a bit afraid because I see a scrolling window for the products contrained by the BTreeFolder on the product homepage (http://www.zope.org/Members/hathawsh/BTreeFolder). In the exmple given (4096 items), it could still be OK, but I think having 1,000,000 + objects might be a problem. There is an interface problem here : the contents of such a folder can only be reasonably displayed in successive batches. (in fact it is also another aspect of the problem of having thousands of objects in an ordinary folder : if you click on the folder to open it, the Management interface will hang)
Pierre
On Fre, 2002-06-21 at 11:57, Pierre Godefroy wrote: [ huge folder ] Have a look at BTreeFolder, the whole structure is stored in a btree, so a lookup for an id should be pretty fast... greetings, Igor
--
Hi everybody,
How would you populate a "huge" ZODB with hundreds of thousands of objects which have no specific reason to be hierarchically distributed in a complex tree of folders and sub-folders?
They could in fact all be in the same folder. It seems to be OK for hundreds of objects, but afterwards there is an incremental cost : each time you add a new object, there is a check on the existence of an identical ID before adding/creating it. When you reach the level of already existing thousands/tens of thousands/hundreds of thousands objects in the same folder it seems you are in serious trouble...
I know the question is a bit naive, and that usually you have a data structure which dictates a naturally "hierarchy" of folders which means that no folder would contain maybe more than a few hundred objects or sub-folders.
Maybe I should "artificially" design an ID structure which would correspond to a hierarchy : for instance ojects would be identified by numbers with 7 digits, each digit corresponding to a lower level in the hierarchy? It means that an object with ID 2343789 would be placed in the "/2/3/4/3/7/8/9" sub-folder.
What do you think? Any other idea?
Pierre Godefroy 52 rue des Archives 75004 PARIS FRANCE
Tél. : +33 (0)1 42 74 46 05
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev ) --- Incoming mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.371 / Virus Database: 206 - Release Date: 13-06-2002
Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.371 / Virus Database: 206 - Release Date: 13-06-2002
Oliver, Yes, in fact it would be for the management of bibliographic records. Pierre At 12:43 21/06/2002 +0200, Oliver Marx wrote:
If I understand you right then you want to use ZODB as thé DB for an application?
\Oliver
-----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Pierre Godefroy Sent: 21. juni 2002 12:37 To: zope@zope.org Subject: [Zope] BTreeFolder for "huge" ZODB (was : populating a "huge" ZODB)
Thank you very much Igor, however I am still a bit afraid because I see a scrolling window for the products contrained by the BTreeFolder on the product homepage (http://www.zope.org/Members/hathawsh/BTreeFolder). In the exmple given (4096 items), it could still be OK, but I think having 1,000,000 + objects might be a problem. There is an interface problem here : the contents of such a folder can only be reasonably displayed in successive batches. (in fact it is also another aspect of the problem of having thousands of objects in an ordinary folder : if you click on the folder to open it, the Management interface will hang)
Pierre
On Fre, 2002-06-21 at 11:57, Pierre Godefroy wrote: [ huge folder ] Have a look at BTreeFolder, the whole structure is stored in a btree, so a lookup for an id should be pretty fast... greetings, Igor
--
Hi everybody,
How would you populate a "huge" ZODB with hundreds of thousands of objects which have no specific reason to be hierarchically distributed in a complex tree of folders and sub-folders?
They could in fact all be in the same folder. It seems to be OK for hundreds of objects, but afterwards there is an incremental cost : each time you add a new object, there is a check on the existence of an identical ID before adding/creating it. When you reach the level of already existing thousands/tens of thousands/hundreds of thousands objects in the same folder it seems you are in serious trouble...
I know the question is a bit naive, and that usually you have a data structure which dictates a naturally "hierarchy" of folders which means that no folder would contain maybe more than a few hundred objects or sub-folders.
Maybe I should "artificially" design an ID structure which would correspond to a hierarchy : for instance ojects would be identified by numbers with 7 digits, each digit corresponding to a lower level in the hierarchy? It means that an object with ID 2343789 would be placed in the "/2/3/4/3/7/8/9" sub-folder.
What do you think? Any other idea?
Pierre Godefroy 52 rue des Archives 75004 PARIS FRANCE
Tél. : +33 (0)1 42 74 46 05
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev ) --- Incoming mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.371 / Virus Database: 206 - Release Date: 13-06-2002
Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.371 / Virus Database: 206 - Release Date: 13-06-2002
Pierre Godefroy 52 rue des Archives 75004 PARIS FRANCE Tél. : +33 (0)1 42 74 46 05
Hi Pierre, I would suggest to store your bibliographic records on a external DB like MySQL, PostgreSQL or Oracle. Or if you have a lot of unstructured data use a XML-Database like Xindice. You would than create an application on zope to work with these data-records. Regards, Dirk ----- Original Message ----- From: "Pierre Godefroy" <pierre.godefroy@noos.fr> To: "Oliver Marx" <oliver@tekk.dk>; <zope@zope.org> Sent: Friday, June 21, 2002 12:55 PM Subject: RE: [Zope] BTreeFolder for "huge" ZODB (was : populating a "huge" ZODB) Oliver, Yes, in fact it would be for the management of bibliographic records. Pierre At 12:43 21/06/2002 +0200, Oliver Marx wrote:
If I understand you right then you want to use ZODB as thé DB for an application?
\Oliver
-----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Pierre Godefroy Sent: 21. juni 2002 12:37 To: zope@zope.org Subject: [Zope] BTreeFolder for "huge" ZODB (was : populating a "huge" ZODB)
Thank you very much Igor, however I am still a bit afraid because I see a scrolling window for the products contrained by the BTreeFolder on the product homepage (http://www.zope.org/Members/hathawsh/BTreeFolder). In the exmple given (4096 items), it could still be OK, but I think having 1,000,000 + objects might be a problem. There is an interface problem here : the contents of such a folder can only be reasonably displayed in successive batches. (in fact it is also another aspect of the problem of having thousands of objects in an ordinary folder : if you click on the folder to open it, the Management interface will hang)
Pierre
On Fre, 2002-06-21 at 11:57, Pierre Godefroy wrote: [ huge folder ] Have a look at BTreeFolder, the whole structure is stored in a btree, so a lookup for an id should be pretty fast... greetings, Igor
--
Hi everybody,
How would you populate a "huge" ZODB with hundreds of thousands of objects which have no specific reason to be hierarchically distributed in a complex tree of folders and sub-folders?
They could in fact all be in the same folder. It seems to be OK for hundreds of objects, but afterwards there is an incremental cost : each time you add a new object, there is a check on the existence of an identical ID before adding/creating it. When you reach the level of already existing thousands/tens of thousands/hundreds of thousands objects in the same folder it seems you are in serious trouble...
I know the question is a bit naive, and that usually you have a data structure which dictates a naturally "hierarchy" of folders which means that no folder would contain maybe more than a few hundred objects or sub-folders.
Maybe I should "artificially" design an ID structure which would correspond to a hierarchy : for instance ojects would be identified by numbers with 7 digits, each digit corresponding to a lower level in the hierarchy? It means that an object with ID 2343789 would be placed in the "/2/3/4/3/7/8/9" sub-folder.
What do you think? Any other idea?
Pierre Godefroy 52 rue des Archives 75004 PARIS FRANCE
Tél. : +33 (0)1 42 74 46 05
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev ) --- Incoming mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.371 / Virus Database: 206 - Release Date: 13-06-2002
Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.371 / Virus Database: 206 - Release Date: 13-06-2002
Pierre Godefroy 52 rue des Archives 75004 PARIS FRANCE Tél. : +33 (0)1 42 74 46 05 _______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
On Friday 21 Jun 2002 11:37 am, Pierre Godefroy wrote:
Thank you very much Igor, however I am still a bit afraid because I see a scrolling window for the products contrained by the BTreeFolder on the product homepage (http://www.zope.org/Members/hathawsh/BTreeFolder). In the exmple given (4096 items), it could still be OK, but I think having 1,000,000 + objects might be a problem. There is an interface problem here
: the contents of such a folder can only be reasonably displayed in
successive batches. (in fact it is also another aspect of the problem of having thousands of objects in an ordinary folder : if you click on the folder to open it, the Management interface will hang)
I can suggest several different approaches: 1. Like Igor suggested, but subclass BTreeFolder and implement your own management interface. 2. Put the objects in a heirarchical structure anyway, and access them through a ZCatalog. Choose a structure that corresponds to the adminstrative patterns, not access patterns. 3. Implement your own Zope Product that behaves exactly how you want. You might think of this as building a 'Better BTreeFolder', and BTrees might be a key part of the implementation. I hope this helps
participants (4)
-
Dirk Datzert -
Oliver Marx -
Pierre Godefroy -
Toby Dickenson