At what point does it make sense to store data outside of the ZODB? For example, I have a site which wants to have access to a news article archive. If the articles were stored in Zope, the ZCatalog searching would make this task almost trivial. I'm concerned about the size of the database, however. The archives may go back several years and represent information from a weekly news publication. I have a sample archive of one "issue" which constitutes about 130k of raw text (no graphics at this point). I expect that a typical "issue" may be around 130k - 200k. If I store that information in Zope, probably as instances of a ZClass with properties to hold some information and perhaps DTMLMethods to hold the many article content, how many weeks of archive information could I reasonably store in Zope before the thing would become too big? This archive solution is likely to be replaced in the future so I'm interested in finding a solution which is simple and quick to implement but will provide adequate performance for my users. Any info on this subject would be appreciated. Thanks. James W. Howe mailto:jwh@allencreek.com Allen Creek Software, Inc. pgpkey: http://ic.net/~jwh/pgpkey.html Ann Arbor, MI 48103
James, Take a look at http://www.zope.org/Members/anthony/sql_vs_ZODB for some views on this subject. Chris "James W. Howe" wrote:
At what point does it make sense to store data outside of the ZODB? For example, I have a site which wants to have access to a news article archive. If the articles were stored in Zope, the ZCatalog searching would make this task almost trivial. I'm concerned about the size of the database, however.
The archives may go back several years and represent information from a weekly news publication. I have a sample archive of one "issue" which constitutes about 130k of raw text (no graphics at this point). I expect that a typical "issue" may be around 130k - 200k. If I store that information in Zope, probably as instances of a ZClass with properties to hold some information and perhaps DTMLMethods to hold the many article content, how many weeks of archive information could I reasonably store in Zope before the thing would become too big?
This archive solution is likely to be replaced in the future so I'm interested in finding a solution which is simple and quick to implement but will provide adequate performance for my users.
Any info on this subject would be appreciated.
Thanks.
James W. Howe mailto:jwh@allencreek.com Allen Creek Software, Inc. pgpkey: http://ic.net/~jwh/pgpkey.html Ann Arbor, MI 48103
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
That helps, but I'm more interested on knowing if there is a size issue. Does performance degrade once the database gets to a particular size, for example. I think, performance not withstanding, that the data I want to store would fit better in a ZODB than an external database. Thanks. At 04:26 PM 2/3/00 +0000, Chris Withers wrote:
James,
Take a look at http://www.zope.org/Members/anthony/sql_vs_ZODB for some views on this subject.
Chris
"James W. Howe" wrote:
At what point does it make sense to store data outside of the ZODB? For example, I have a site which wants to have access to a news article archive. If the articles were stored in Zope, the ZCatalog searching would make this task almost trivial. I'm concerned about the size of the database, however.
The archives may go back several years and represent information from a weekly news publication. I have a sample archive of one "issue" which constitutes about 130k of raw text (no graphics at this point). I expect that a typical "issue" may be around 130k - 200k. If I store that information in Zope, probably as instances of a ZClass with properties to hold some information and perhaps DTMLMethods to hold the many article content, how many weeks of archive information could I reasonably store in Zope before the thing would become too big?
This archive solution is likely to be replaced in the future so I'm interested in finding a solution which is simple and quick to implement but will provide adequate performance for my users.
Any info on this subject would be appreciated.
Thanks.
James W. Howe mailto:jwh@allencreek.com Allen Creek Software, Inc. pgpkey:
http://ic.net/~jwh/pgpkey.html
Ann Arbor, MI 48103
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
James W. Howe mailto:jwh@allencreek.com Allen Creek Software, Inc. pgpkey: http://ic.net/~jwh/pgpkey.html Ann Arbor, MI 48103
James,
That helps, but I'm more interested on knowing if there is a size issue. Does performance degrade once the database gets to a particular size, for example.
I'm afraid that I don't know, but I'd very much like to ;-) I'm working on the basis that I'll deal with it when it becomes a problem, but I understand that may not be an acceptable proposition in your case. The following may interest you, although I don't know the size of the information stored in Zope.org: http://www.zope.org/AboutZopeOrg If you learn any more, please let me know too! :-) cheers, Chris
Let's see if I can put your mind somewhat at rest. I have a Zope database which is roughly 500Mb on an NT box. The rudimentary tests I've done (using the Apache ab command), indicate no difference between the 500mb ZODB and one of 10mb, this of course is where the same documents/methods are being called on both. I, so far, have absolutely no worries about the size of the ZODB, apart from running out of disk of course. HTH Phil phil.harris@zope.co.uk | -----Original Message----- | From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Chris | Withers | Sent: Thursday, February 03, 2000 4:50 PM | To: James W. Howe | Cc: Zope Mailing List | Subject: Re: [Zope] Zope Database Size | | | James, | | > That helps, but I'm more interested on knowing if there is a size | > issue. Does performance degrade once the database gets to a particular | > size, for example. | | I'm afraid that I don't know, but I'd very much like to ;-) | I'm working on the basis that I'll deal with it when it becomes | a problem, but I | understand that may not be an acceptable proposition in your case. | | The following may interest you, although I don't know the size of the | information stored in Zope.org: | http://www.zope.org/AboutZopeOrg | | If you learn any more, please let me know too! :-) | | cheers, | | Chris | | _______________________________________________ | Zope maillist - Zope@zope.org | http://lists.zope.org/mailman/listinfo/zope | ** No cross posts or HTML encoding! ** | (Related lists - | http://lists.zope.org/mailman/listinfo/zope-announce | http://lists.zope.org/mailman/listinfo/zope-dev ) |
----- Original Message ----- From: "Phil Harris" <phil.harris@zope.co.uk> To: "Chris Withers" <chrisw@nipltd.com>; "James W. Howe" <jwh@allencreek.com> Cc: "Zope Mailing List" <zope@zope.org> Sent: Thursday, February 03, 2000 12:04 PM Subject: RE: [Zope] Zope Database Size [snippage]
I have a Zope database which is roughly 500Mb on an NT box.
I, so far, have absolutely no worries about the size of the ZODB, apart from running out of disk of course.
How frequently do you pack the DB, and how long does that take? That's something I've been curious about as my DB has been growing (but we're nowhere near 500MB right now...) Kevin
Kevin Dangoor wrote:
----- Original Message ----- From: "Phil Harris" <phil.harris@zope.co.uk> To: "Chris Withers" <chrisw@nipltd.com>; "James W. Howe" <jwh@allencreek.com> Cc: "Zope Mailing List" <zope@zope.org> Sent: Thursday, February 03, 2000 12:04 PM Subject: RE: [Zope] Zope Database Size
[snippage]
I have a Zope database which is roughly 500Mb on an NT box.
I, so far, have absolutely no worries about the size of the ZODB, apart from running out of disk of course.
How frequently do you pack the DB, and how long does that take? That's something I've been curious about as my DB has been growing (but we're nowhere near 500MB right now...)
Note that in Zope 2, packing is done in the background and doesn't block reads or writes except during a small part of the overall packing operation. Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
On Thu, 3 Feb 2000, James W. Howe wrote:
At what point does it make sense to store data outside of the ZODB? For example, I have a site which wants to have access to a news article archive. If the articles were stored in Zope, the ZCatalog searching would make this task almost trivial. I'm concerned about the size of the database, however.
You will probably want to consult the How-To on this topic at http://www.zope.org/Members/anthony/sql_vs_ZODB Good luck. -Tim -- Tim Wilson | Visit Sibley online: | Check out: Henry Sibley H.S. | http://www.isd197.k12.mn.us/ | http://www.zope.org/ W. St. Paul, MN | | http://slashdot.org/ wilson@visi.com | <dtml-var pithy_quote> | http://linux.com/
"James W. Howe" wrote:
At what point does it make sense to store data outside of the ZODB? For example, I have a site which wants to have access to a news article archive. If the articles were stored in Zope, the ZCatalog searching would make this task almost trivial. I'm concerned about the size of the database, however.
The archives may go back several years and represent information from a weekly news publication. I have a sample archive of one "issue" which constitutes about 130k of raw text (no graphics at this point). I expect that a typical "issue" may be around 130k - 200k. If I store that information in Zope, probably as instances of a ZClass with properties to hold some information and perhaps DTMLMethods to hold the many article content, how many weeks of archive information could I reasonably store in Zope before the thing would become too big?
This archive solution is likely to be replaced in the future so I'm interested in finding a solution which is simple and quick to implement but will provide adequate performance for my users.
Any info on this subject would be appreciated.
There isn't really any size limit in ZODB, however, the underlying storage may impose some limitation. ZODB uses an open storage interface, and we provide a default storage implementation, the FileStorage. It's unclear how scalable FileStorage is. The answer probobably depends on your application. Here are some things to consider with FileStorage: - The database size is limited by the maximum size of a file in whatever underlying file system you use. Some systems limit this to 2GB, but other's don't. - There is a small fixed per-object memory cost when using FileStorage. This is due to the in-memory index that FileStorage uses to keep track of object record locations. Essentially, each object in the database consumes a Python dictionary entry with an 8-byte string and a short or long integer. This is probably on the order of 100 bytes per object. (Someone could invent a more optimized data structure that could cut this down to about 20 bytes per object.) Note that Rob is quite right in saying that the ZODB cache keeps only a fraction of your data in memory. In your application, you have large objects, but you don't have that many objects. So the second bullet doesn't really apply to you. Of course, file size limits are likely to limit many alternative approaches too. Ty Sarna's CompressedStorage, http://www.zope.org/Members/tsarna/CompressedStorage, can be used with FileStorage to reduce database size. This would probably be a big win for you. The per-object overhead in FileStorage could be addressed with more efficient indexing data structures, or with some mechanism, like dbm or Berkeley DB files to store the index. On systems that support very large files a derivative of FileStorage that used a dbm index should be able to support extremely large databases. Of course, you can implement (or you can pay someone to implement) alternative storages that used whetever underying storage mechanism you want and then plug it into ZODB. For example, it should be straightforward to implement a ZODB storage that uses some underlying RDBMS as it's storage manager. Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
On Thu, 03 Feb 2000 12:31:48 -0500, Jim Fulton <jim@digicool.com> wrote:
Here are some things to consider with FileStorage:
- There is a small fixed per-object memory cost when using FileStorage. This is due to the in-memory index that FileStorage uses to keep track of object record locations.
The per-object overhead in FileStorage could be addressed with more efficient indexing data structures, or with some mechanism, like dbm or Berkeley DB files to store the index.
This problem has just come onto my radar for my current project (pending some performance tests later this week). Is anyone else looking at this and would care to share some effort? Toby Dickenson tdickenson@geminidataloggers.com
participants (7)
-
Chris Withers -
James W. Howe -
Jim Fulton -
Kevin Dangoor -
Phil Harris -
Timothy Wilson -
Toby Dickenson