ANNOUNCE: OracleStorage 1.0.0 beta 1
The initial beta release of an Oracle-based storage for ZODB is available at: http://www.zope.org/Products/OracleStorage The Oracle storage provides full ZODB storage capabilities, including undo and versions, by storing serialized objects and meta data in Oracle tables. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
Jim Fulton wrote:
The initial beta release of an Oracle-based storage for ZODB is available at:
http://www.zope.org/Products/OracleStorage
The Oracle storage provides full ZODB storage capabilities, including undo and versions, by storing serialized objects and meta data in Oracle tables.
Chris McDonough pointed out that I had left some specific table names in some SQL statements rather than using "prefix" specific names. This would lead to pack failures (unless you chose the prefix "jim_" foy your Oracle objects. :) Thanks Chris. I've fixed this and made beta 2 available at the location above. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
Hello sports fans. Last week we released a beta of OracleStorage. There was a pretty strong drumbeat in Zopeland for us to go ahead and get this out. We spent some extra time on the installation process and instructions to make sure it was straightforward to use. We at DC would like to get OracleStorage hardened as fast as possible. Thus, I have a serious of requests: a. If you are interested in it, please download it this week and give it a shot. b. If you downloaded it, please send me a note and tell me so. c. If you find any bugs with it, please file something in the Collector. We'll make sure there's a keyword for OracleStorage. d. If you have a general question, request, or discussion, start a thread here in zope-dev. My indebtedness goes out to anybody that can help us on this. Thanks! --Paul
Well, two betas of OracleStorage in one day, then a month and a half of silence. What's the status? What about the other Storage projects? BerkeleyStorage has been dead for an year, and I heard pretty nasty words about InterbaseStorage. What about someone who wanted to try to port OracleStorage to Postgres or some other RDBMS? Please help stamp out Data.fs! :-) []s, |alo +---- -- Hack and Roll ( http://www.hackandroll.org ) News for, uh, whatever it is that we are. http://zope.gf.com.br/lalo mailto:lalo@hackandroll.org pgp key: http://zope.gf.com.br/lalo/pessoal/pgp Brazil of Darkness (RPG) --- http://zope.gf.com.br/BroDar
Lalo Martins wrote:
Well, two betas of OracleStorage in one day, then a month and a half of silence. What's the status?
What about the other Storage projects? BerkeleyStorage has been dead for an year, and I heard pretty nasty words about InterbaseStorage. What about someone who wanted to try to port OracleStorage to Postgres or some other RDBMS?
Please help stamp out Data.fs! :-)
[]s,
i've been interested in doing a postgres storage, but i haven't been able to convince myself that it would be useful in practice at all. any suggestions? Kapil
On Tue, Nov 28, 2000 at 06:01:15PM -0800, Ender wrote:
Lalo Martins wrote:
Well, two betas of OracleStorage in one day, then a month and a half of silence. What's the status?
What about the other Storage projects? BerkeleyStorage has been dead for an year, and I heard pretty nasty words about InterbaseStorage. What about someone who wanted to try to port OracleStorage to Postgres or some other RDBMS?
Please help stamp out Data.fs! :-)
i've been interested in doing a postgres storage, but i haven't been able to convince myself that it would be useful in practice at all. any suggestions?
Of course it would, for the same reasons as OracleStorage (eg FileStorage/Data.fs is inefficient), plus PostgreSQL is free and Oracle costs thousands of dollars. What's the doubt? :-) []s, |alo +---- -- Hack and Roll ( http://www.hackandroll.org ) The biggest site for whatever-it-is-that-we-are. http://zope.gf.com.br/lalo mailto:lalo@hackandroll.org pgp key: http://zope.gf.com.br/lalo/pessoal/pgp Brazil of Darkness (RPG) --- http://zope.gf.com.br/BroDar
Of course it would, for the same reasons as OracleStorage (eg FileStorage/Data.fs is inefficient)
Actually, it's the other way around. OracleStorage is 30-to-50 times slower than FileStorage on writes. Reads are slow too but the slowness is somewhat negated by caching.
On Wed, Nov 29, 2000 at 07:02:50AM -0500, Chris McDonough wrote:
Of course it would, for the same reasons as OracleStorage (eg FileStorage/Data.fs is inefficient)
Actually, it's the other way around. OracleStorage is 30-to-50 times slower than FileStorage on writes. Reads are slow too but the slowness is somewhat negated by caching.
Chris, that's only true for small databases. At about 100M of Data.fs, OracleStorage starts being faster. It depends on hardware too. We made some benchmarks on a major Brazilian portal, and well, it's currently running OracleStorage. Anyway, I said "inefficient", not "slow". []s, |alo +---- -- Hack and Roll ( http://www.hackandroll.org ) The biggest site for whatever-it-is-that-we-are. http://zope.gf.com.br/lalo mailto:lalo@hackandroll.org pgp key: http://zope.gf.com.br/lalo/pessoal/pgp Brazil of Darkness (RPG) --- http://zope.gf.com.br/BroDar
It sounds like you've done more comprehensive speed testing than we have. Can you share some numbers? ----- Original Message ----- From: "Lalo Martins" <lalo@hackandroll.org> To: <zope-dev@zope.org> Sent: Wednesday, November 29, 2000 8:42 AM Subject: Re: [Zope-dev] OracleStorage, and possibly others
On Wed, Nov 29, 2000 at 07:02:50AM -0500, Chris McDonough wrote:
Of course it would, for the same reasons as OracleStorage (eg FileStorage/Data.fs is inefficient)
Actually, it's the other way around. OracleStorage is 30-to-50 times slower than FileStorage on writes. Reads are slow too but the slowness is somewhat negated by caching.
Chris, that's only true for small databases. At about 100M of Data.fs, OracleStorage starts being faster. It depends on hardware too. We made some benchmarks on a major Brazilian portal, and well, it's currently running OracleStorage.
Anyway, I said "inefficient", not "slow".
[]s, |alo +---- -- Hack and Roll ( http://www.hackandroll.org ) The biggest site for whatever-it-is-that-we-are.
http://zope.gf.com.br/lalo mailto:lalo@hackandroll.org pgp key: http://zope.gf.com.br/lalo/pessoal/pgp
Brazil of Darkness (RPG) --- http://zope.gf.com.br/BroDar
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
On Wed, Nov 29, 2000 at 10:13:15AM -0500, Chris McDonough wrote:
It sounds like you've done more comprehensive speed testing than we have. Can you share some numbers?
Unfortunately no, I can't. We made measurements during a consulting job I did for another company, and it was made by them, not me, *plus* I wasn't smart enough to save a copy of the numbers :-/ Anyway, the biggest problem they had was stability, their ZODB grows and shrinks insickening speeds, and Data.fs would break for some reasons I don't remember anymore, and OracleFS solved that. I'll try to convince them to write a report to the list.
----- Original Message ----- From: "Lalo Martins" <lalo@hackandroll.org> To: <zope-dev@zope.org> Sent: Wednesday, November 29, 2000 8:42 AM Subject: Re: [Zope-dev] OracleStorage, and possibly others
On Wed, Nov 29, 2000 at 07:02:50AM -0500, Chris McDonough wrote:
Of course it would, for the same reasons as OracleStorage (eg FileStorage/Data.fs is inefficient)
Actually, it's the other way around. OracleStorage is 30-to-50 times slower than FileStorage on writes. Reads are slow too but the slowness is somewhat negated by caching.
Chris, that's only true for small databases. At about 100M of Data.fs, OracleStorage starts being faster. It depends on hardware too. We made some benchmarks on a major Brazilian portal, and well, it's currently running OracleStorage.
Anyway, I said "inefficient", not "slow".
[]s, |alo +---- -- Hack and Roll ( http://www.hackandroll.org ) The biggest site for whatever-it-is-that-we-are. http://zope.gf.com.br/lalo mailto:lalo@hackandroll.org pgp key: http://zope.gf.com.br/lalo/pessoal/pgp Brazil of Darkness (RPG) --- http://zope.gf.com.br/BroDar
I actually need to get a BerkeleyStorage against BSDDB3 going for a customer fairly soon. Jim has done a lot of work on it, and it's looking like I'll probably end up finishing it. Robin Dunn has a Python extension module against the bsddb3 libraries that we're using. It may actually be released as several storages (undoing and nonundoing).
In article <01ba01c05a42$edabbb10$1f48a4d8@kurtz>, Chris McDonough <chrism@digicool.com> wrote:
I actually need to get a BerkeleyStorage against BSDDB3 going for a customer fairly soon. Jim has done a lot of work on it, and it's looking like I'll probably end up finishing it. Robin Dunn has a Python extension module
Ah, cool!
against the bsddb3 libraries that we're using. It may actually be released as several storages (undoing and nonundoing).
Once upon a time, I had planned to do (and had some code towards): no-versions, no-undo versions, undo versions, undo only within versions with a "table" structure such that you could migrate from one to another. The last option is possibly useful for sites with objects that change a lot (so you don't want undo), but where you want to use versions for code development (which requires undo... note the lack of a "versions, no-undo" combo. That one won't work usefully, because you can trivially get stuck. You get something locked in the version, and there's no way to recover short of committing the entire version. "no-versions, undo" is possible, but doesn't seem to really buy anything)
Lalo Martins wrote:
On Wed, Nov 29, 2000 at 07:02:50AM -0500, Chris McDonough wrote:
Of course it would, for the same reasons as OracleStorage (eg FileStorage/Data.fs is inefficient)
Actually, it's the other way around. OracleStorage is 30-to-50 times slower than FileStorage on writes. Reads are slow too but the slowness is somewhat negated by caching.
Chris, that's only true for small databases. At about 100M of Data.fs, OracleStorage starts being faster. It depends on hardware too. We made some benchmarks on a major Brazilian portal, and well, it's currently running OracleStorage.
Anyway, I said "inefficient", not "slow".
I think you really mean "scalable". I suspect that FileStorage will be faster and use less disk that OracleStorage, or any other storage, at least when supporting undo. To get even comparible speed from Oracle requires substantial optimization on the Oracle side. Our customer was able to substantially improve DCOracleStorage performance by doing things like putting tables and indexes on different disk *controllers*. :) OTOH, FileStorage has a huge memory hit for lage databases, because it keeps an in memory index from oid to file position. It currently uses a Python dictionary for this. We could probably reduce the memory hit by about an order of magnitude by switching to a different data structure, but DCOracleStorage or a Berkeley DB-based storage doesn't need this index. Initial tests with my latest Berkeley DB-based storage show it to be about three times slower than FileStorage. I suspect that if the logs and data files were put on separate disks, as Sleepycat Software recommends, that the speed difference could be reduced quite a bit. Note that, until recently, FileStorage couldn't be used with databases over 2GB in size, due to a bug that was fixed at around the same time that we released the DCOracleStorage. We have a customer who has a 10GB FileStorage. You need large-file support in Python for your OS to be able to go over 2GB. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org
On Thu, Nov 30, 2000 at 08:27:00AM -0500, Jim Fulton wrote:
Lalo Martins wrote:
Anyway, I said "inefficient", not "slow".
I think you really mean "scalable".
No, I meant "inefficient", a purposely vague term ;-) yes, I meant to imply it isn't scalable, but not only that, and as I didn't have the numbers, I prefered to be vague. What you say below for memory usage also falls under "inefficient". Eating into swap can really make a big machine crawl.
OTOH, FileStorage has a huge memory hit for lage databases, because it keeps an in memory index from oid to file position. It currently uses a Python dictionary for this. We could probably reduce the memory hit by about an order of magnitude by switching to a different data structure, but DCOracleStorage or a Berkeley DB-based storage doesn't need this index.
[]s, |alo +---- -- Hack and Roll ( http://www.hackandroll.org ) The biggest site for whatever-it-is-that-we-are. http://zope.gf.com.br/lalo mailto:lalo@hackandroll.org pgp key: http://zope.gf.com.br/lalo/pessoal/pgp Brazil of Darkness (RPG) --- http://zope.gf.com.br/BroDar
If you're at all interested, Kapil, it'd really be great if someone were to take over InterbaseStorage. It needs to be changed to use Bob Tierney's newer multithreaded GVInterbaseDA Python DB-API adapter (Kinterbasdb doesn't seem to get along with newer versions of Zope, and it's singlethreaded). Additionally, the pack code is broken. It would be a big win for folks who want a versioning, undoing storage backed by a relational opensource database. ----- Original Message ----- From: "Ender" <kthangavelu@earthlink.net> To: "Lalo Martins" <lalo@hackandroll.org> Cc: <zope-dev@zope.org> Sent: Tuesday, November 28, 2000 9:01 PM Subject: Re: [Zope-dev] OracleStorage, and possibly others
Lalo Martins wrote:
Well, two betas of OracleStorage in one day, then a month and a half of silence. What's the status?
What about the other Storage projects? BerkeleyStorage has been dead for an year, and I heard pretty nasty words about InterbaseStorage. What about someone who wanted to try to port OracleStorage to Postgres or some other RDBMS?
Please help stamp out Data.fs! :-)
[]s,
i've been interested in doing a postgres storage, but i haven't been able to convince myself that it would be useful in practice at all. any suggestions?
Kapil
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Lalo Martins wrote:
What about the other Storage projects? BerkeleyStorage has been dead for an year.
It's not dead, it's just pining for the fjords! Seriously, AFAIK it still works, and it's mainly just stalled waiting for some interest. It could stand to be updated to BerkeleyDB 3, but that was blocked on having an updated Python BerkeleyDB module for a while. Now there is an updated version, so it should be easy to update the storage. The main thing BerkeleyStorage is lacking is interest -- I don't need it currently for the purpose for which it was developed, and nobody else seems to need it much either. It looked for a while like it might get a boost from the Sessions project, but that's stalled for the moment as well.
On Wed, Nov 29, 2000 at 04:28:26PM +0000, Ty Sarna wrote:
Lalo Martins wrote:
What about the other Storage projects? BerkeleyStorage has been dead for an year.
It's not dead, it's just pining for the fjords!
Seriously, AFAIK it still works, and it's mainly just stalled waiting for some interest. It could stand to be updated to BerkeleyDB 3, but that was blocked on having an updated Python BerkeleyDB module for a while. Now there is an updated version, so it should be easy to update the storage.
The main thing BerkeleyStorage is lacking is interest -- I don't need it currently for the purpose for which it was developed, and nobody else seems to need it much either. It looked for a while like it might get a boost from the Sessions project, but that's stalled for the moment as well.
I think a BerkeleyDB3-based Storage would be a very cool option if versioning was optional instead of missing. (Then again, this is almost getting to my dreamed XDeltaStorage...) []s, |alo +---- -- Hack and Roll ( http://www.hackandroll.org ) The biggest site for whatever-it-is-that-we-are. http://zope.gf.com.br/lalo mailto:lalo@hackandroll.org pgp key: http://zope.gf.com.br/lalo/pessoal/pgp Brazil of Darkness (RPG) --- http://zope.gf.com.br/BroDar
Lalo Martins wrote:
Well, two betas of OracleStorage in one day, then a month and a half of silence. What's the status?
From our perspective, it's what it was then. We built it for a customer who later decided they didn't want it. I'm glad to hear you found it useful.
What about the other Storage projects? BerkeleyStorage has been dead for an year, and I heard pretty nasty words about InterbaseStorage. What about someone who wanted to try to port OracleStorage to Postgres or some other RDBMS?
We're working on a new suite of Berkeley storages that we are writing to the the latest Berkeley database APIs. As Chris pointed out, these are based on a new Berkeley DB extension that Andrew Kuchling started and that Robin Dunn is finishing. There are a number of interesting new features that will eventually result from these storages: - A new pack/garbage collection approach. The current storages perform a mark-and-sweep garbage collection when packing. It turns out that this really isn't very scalable. I'm going two switch to using the same garbage-collection strategy that Python uses, which is reference counting supplimented by an incremental cyclic garbage detection algorithm. I have a simple storage (no undo or versions) that needs no packing of your data structures are free of cycles. I'm hopeful that we can make packing automatic and incremental. - Undo-log generation will be much faster, at least for common cases. Generation of undo logs (for selecting transactions to undo) is becoming a significant performance problem with FileStorage. - ZEO verification, performed when a client with a persistent cache starts up, will be much faster. - Policies to control whether multiple revisions are stored or whether revisions are removed by packing on a object-by-object or transaction-by-transaction basis. For example, you could have a poll/votting product that didn't allow undo of and didn't require storing multiple records for votes. This would be cleaner than mounting a separate database with a simple storage. You could keep significant historical revisions for important objects, such as Wiki pages, templates, scripts, etc., with a policy to decide which revisions are significant. - The ability to set policies to implement quotas. I expect that we'll work out a lot of these ideas for Berkeley storages and then implement them in OrcaleStorage. Others should then be able to map the implementation onto other storages. I'll note, in passing, that it's much nicer working these ideas out with the BerkelyDB API than dealing with PLSQL or Python SQL. We are also working on ZEO storage replication. This may have a big impact on the storage API, or lead to specialized APIs for replicated storages.
Please help stamp out Data.fs! :-)
I don't think Data.fs will go away. I do expect it to be relagated to initial evaluation and development projects. Use of Berkely DB in transactional mode requires a significant andminstration commitment. Log files need to be purged. Backup and recovery processes need to be in place. A similar cost is associated with using Oracle and many other databases, I expect. People aren't going to want to deal with these issues when initially trying and learning Zope (or ZODB). Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org
At 08:10 AM 11/30/00 -0500, Jim Fulton wrote:
I don't think Data.fs will go away. I do expect it to be relagated to initial evaluation and development projects. Use of Berkely DB in
transactional mode requires a significant andminstration commitment. Log files need to be purged. Backup and recovery processes need to be in place.
FWIW, Ty and I talked about using a daemonic thread in BerkeleyStorage to run periodic checkpoints and purge logfiles. AFAIK, backups of BerkeleyDB require only that you make a copy of all the associated files. Recovery's a bit trickier, of course. (We never got around to implementing it because our tests showed that BerkeleyDB didn't solve the performance issue we were trying to solve. So we quit working on it at that point.)
On Thu, Nov 30, 2000 at 08:10:15AM -0500, Jim Fulton wrote:
Lalo Martins wrote:
Please help stamp out Data.fs! :-)
I don't think Data.fs will go away. I do expect it to be relagated to initial evaluation and development projects. Use of Berkely DB in transactional mode requires a significant andminstration commitment. Log files need to be purged. Backup and recovery processes need to be in place. A similar cost is associated with using Oracle and many other databases, I expect. People aren't going to want to deal with these issues when initially trying and learning Zope (or ZODB).
Sometime it will, I hope. Or at least, be relegated to non-Zope ZODBs where the data volume is very small. But FileStorage will not be replaced by any of the current alternatives, I agree. []s, |alo +---- -- Hack and Roll ( http://www.hackandroll.org ) The biggest site for whatever-it-is-that-we-are. http://zope.gf.com.br/lalo mailto:lalo@hackandroll.org pgp key: http://zope.gf.com.br/lalo/pessoal/pgp Brazil of Darkness (RPG) --- http://zope.gf.com.br/BroDar
Jim Fulton wrote:
- Policies to control whether multiple revisions are stored or whether revisions are removed by packing on a object-by-object or transaction-by-transaction basis.
You could keep significant historical revisions for important objects, such as Wiki pages, templates, scripts, etc., with a policy to decide which revisions are significant.
- The ability to set policies to implement quotas.
We are also working on ZEO storage replication. This may have a big impact on the storage API, or lead to specialized APIs for replicated storages.
very cool :-) Any idea when these will be around to play with?
Please help stamp out Data.fs! :-)
Nah... we like it, especially once Shane's PartionedFile storage makes it into production... cheers, Chris
Chris Withers wrote:
Jim Fulton wrote:
- Policies to control whether multiple revisions are stored or whether revisions are removed by packing on a object-by-object or transaction-by-transaction basis.
You could keep significant historical revisions for important objects, such as Wiki pages, templates, scripts, etc., with a policy to decide which revisions are significant.
- The ability to set policies to implement quotas.
We are also working on ZEO storage replication. This may have a big impact on the storage API, or lead to specialized APIs for replicated storages.
very cool :-)
Any idea when these will be around to play with?
Uh, no. We are actually using a very limited version in production, but the full-featured version is waiting for someone with bandwidth to finish it. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org
participants (8)
-
Chris McDonough -
Chris Withers -
Ender -
Jim Fulton -
Lalo Martins -
Paul Everitt -
Phillip J. Eby -
tsarna＠endicor.com