[Zope] storing in ZODB vs filesystem

[Toby Dickenson]

On Monday 26 Aug 2002 5:45 pm, Dieter Maurer wrote:
> Phil Glaser writes:
>  > ...
>  > My understanding is that your process space needs
>  > to have enough RAM to contain the ZODB index, which is typtically ab=
out
>  > 20% the size of Data.fs.
>
> I do not believe this number (20 %).

Thats higher than I typically see, but not much. All my FileStorages have=
 a=20
RAM/disk ratio between 2% and 12%. I usually work with 10% when planning =
a=20
new server

That is for normal use. During packing FileStorage needs more than twice =
as=20
much, which may explain Phils 20%=20

>    The index size depends only on the number of objects in the ZODB and
>    not on the objects size. I expect (at most) a few hundred bytes
>    per object.

If you are limited by RAM for your filesystem index, then Content objects=
 are=20
probably not your problem. If a typical content object is 10k, then they =
have=20
a RAM/disk ratio of 1%. Moving content out into the filesystem wont=20
significantly reduce RAM requirements.

Your problem is probably the metadata. For example BTrees create many obj=
ects=20
that are less than 100 bytes.=20


>> Is all this plausible, or am I barking up the wrong tree? Please corre=
ct me
>> if I'm wrong because this is an issue for an app I'm in the process of
>> spec'ing out.

There tools which tell you the size and number of objects in a FileStorag=
e: I=20
think translyzer can do this, and something else released more recently w=
hose=20
name I dont remember. You could use this to get a better estimate for you=
r=20
data set.

If it looks like a problem you could consider a more scalable storage;=20
possibly the various flavours of bsddbStorage (in a perpetual beta releas=
e),=20
or DirectoryStorage (still at alpha release, and under active development=
)

(Hmmm. If you use DirectoryStorage then you can see the mix of object siz=
es=20
using 'ls'. Thats handy)