[Zope-ZEO] Re: [Zope-dev] OracleStorage, and possibly others

[Jim Fulton]

Lalo Martins wrote:
> 
> On Wed, Nov 29, 2000 at 07:02:50AM -0500, Chris McDonough wrote:
> > > Of course it would, for the same reasons as OracleStorage (eg
> > > FileStorage/Data.fs is inefficient)
> >
> > Actually, it's the other way around.  OracleStorage is 30-to-50 times slower
> > than FileStorage on writes.  Reads are slow too but the slowness is somewhat
> > negated by caching.
> 
> Chris, that's only true for small databases. At about 100M of
> Data.fs, OracleStorage starts being faster. It depends on
> hardware too. We made some benchmarks on a major Brazilian
> portal, and well, it's currently running OracleStorage.
> 
> Anyway, I said "inefficient", not "slow".

I think you really mean "scalable".  I suspect that
FileStorage will be faster and use less disk that OracleStorage, 
or any other storage, at least when supporting undo.

To get even comparible speed from Oracle requires substantial
optimization on the Oracle side. Our customer was able to substantially
improve DCOracleStorage performance by doing things like putting
tables and indexes on different disk *controllers*. :)

OTOH, FileStorage has a huge memory hit for lage databases, because
it keeps an in memory index from oid to file position.  It currently
uses a Python dictionary for this. We could probably reduce the memory 
hit by about an order of magnitude by switching to a different data
structure, but DCOracleStorage or a Berkeley DB-based storage doesn't
need this index.

Initial tests with my latest Berkeley DB-based storage show it to be
about three times slower than FileStorage.  I suspect that if the logs
and data files were put on separate disks, as Sleepycat Software recommends, 
that the speed difference could be reduced quite a bit.

Note that, until recently, FileStorage couldn't be used with databases
over 2GB in size, due to a bug that was fixed at around the same time that 
we released the DCOracleStorage.  We have a customer who has a 10GB 
FileStorage. You need large-file support in Python for your OS to 
be able to go over 2GB.

Jim

--
Jim Fulton           mailto:jim@digicool.com   Python Powered!        
Technical Director   (888) 344-4332            http://www.python.org  
Digital Creations    http://www.digicool.com   http://www.zope.org