[Dirstorage-users] Re: [ZODB-Dev] POSKeyError plea for help

Tue, 14 Jan 2003 10:59:33 -0800

For whatever its worth, my initial experience is that there are scalability
problems with ext3 and DirectoryStorage.  Once the ext3 filesystem exceeds
millions of files, filesystem metadata operations slow to a crawl on a very
fast, otherwise unused/unloaded machine (actually, I think "crawl" is a very
forgiving term; results of typing ls in the nearly-empty root dir of the
filesystem take 10-20 seconds).  This of course, is a purely informal
observation; we are in the process of moving this over to reiserfs (and I
don't have a decent comparative assessment yet).  

Sean

-----Original Message-----
From: Toby Dickenson [mailto:tdickenson@geminidataloggers.com]
Sent: Tuesday, January 14, 2003 5:02 AM
To: Chris Withers
Cc: zodb-dev@zope.org; dirstorage-users@lists.sourceforge.net
Subject: [Dirstorage-users] Re: [ZODB-Dev] POSKeyError plea for help

(Reply-To set to dirstorage-users list)

On Tuesday 14 January 2003 12:26 pm, Chris Withers wrote:
> Toby Dickenson wrote:
> >>PS: What file systems would you recommend DirectoryStorage on?
> >
> > reiserfs on linux.
>
> How about ext2?

On ext2 there are certain operations that can not be performed safely,
without 
the risk of a crash or power loss dumping files in /lost+found. It is 
certainly possible to manually rebuild the directory structure after that, 
but I dont recomend it.

(Hmmm. 99% possible. I could easily fix that last 1% if you need it)

ext3 should be robust, but you will lose out on speed and space.

> Sorry, I meant how much difference does the type of filesystem make?

It is believed to be critical.  Nearly half of all files are 8 bytes long,
and 
I dont know of any filesystem other than reiser or NTFS than can handle this

load efficiently.

> > You will get a healthy NotImplementedError. It should be possible to
> > create an excellent NTFS version, but I know I will never have time for
> > this.
>
> What eneds to be done?

1. study the semantics of the win32 equivalent to 'fsync'. Getting this
   right on posix was harder than the man page suggested.
2. find out how to perform an atomic file replace.
3. implement a scalable os.listdir (extension module?)
4. probably more.

Have a look at:
PosixFilesystem.py - an example of how an NTFSFilesystem would work
BaseFileSystem.py - which defines its interface.
LocalFilesystem.py - which probably has other requirements that
  really should be documented in BaseFileSystem.py

-- 
Toby Dickenson
http://www.geminidataloggers.com/people/tdickenson

-------------------------------------------------------
This SF.NET email is sponsored by: FREE  SSL Guide from Thawte
are you planning your Web Server Security? Click here to get a FREE
Thawte SSL guide and find the answers to all your  SSL security issues.
http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0026en
_______________________________________________
dirstorage-users mailing list
dirstorage-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dirstorage-users