On Friday 31 January 2003 7:17 am, Wankyu Choi wrote:
While we're at the subject... I'd be pulling, say, 10,000 objects from directory storage in one go if a NeoBoard forum gets heavily populated ( 10,000 articles or 5,000 articles with each article having one attachment object ).
Either I misunderstand this statement, or your application design may be flawed. What is 'one go'? How many 'goes per second' do you expect to sustain?
I meant ( basically the same, but the actual code is slightly different than the following ): --- code excerpt from NeoBoard --- def getArticles(...): ... prepare.... articles = [article for article in board.ZopeFind( obj_metatypes='NeoBoard Article' , search_sub=1)] # I use ZopeFind instead of objectValues since articles get nested ...sort articles... ...return articles... --------------------------- Note that my python/zope experience is limited. I'm still learning :-) Suppose the 'board' folderish object ( let's assume it's a BTreeFolder; if it's present NeoBoard uses BTreeFolder2, otherwise Zope Folder ) holds about 10,000 articles. Whenever a list of articles gets displayed, this statement should be executed. Most RDBMS-based boards do the same with a select/limit combo like the PHP NeoBoard does: "SELECT article_field_list from board limit start_num, end_num" Am I missing something? Is there a way in Zope to limit the returned results as a limit clause would do in an SQL statement? Even the Zope core source code seems to get every subobject from folderish objects no matter how many there might be. That gave me the impression that Zope doesn't give you that option. Correct me if I'm wrong. With RDBMS, I can limit the numver of returned rows with a limit clause that is constructed with a request variable, say, 'b_start'. In Zope, I have to pull all the rows (objects) first and get a specified batch from them by wrapping them into batches. I didn't mean 'goes per second'. Just wanted to know how much slower this one go of pulling too many objects from a folder would be in comparison to RDBMS's select/limit combo.
How slow would it be compared to an SQL "select * from heavily_populated_board limit 50" (supposing a pageful of article list would spew out 50 articles )? ( I just assume it'd be slower than using MySQL. Correct me if I'm wrong. )
The DirectoryStorage overhead for this is tiny: nothing more than reading 100 files from some nested directories and some simple validation on each file
header. Not much different to FileStorage, which would have 50 seeks and 50 reads in one file. Both storages would transfer the same number of data bytes.
The key performance issue is how fast your filesystem can traverse those directories. Or for FileStorage, the cost of keeping that directory/dictionary in memory. If you have tried FileStorage and it lacked raw speed, then I doubt DirectoryStorage will be faster.
I applologize for not being specific with the query. I didn't mean to compare DS with FS. I already know DS could be slower than FS and guess that's acceptable given its benefits. I was just curious how slower NeoBoard might get if I rely solely on ZODB ( irregardless of its storage model ) than on RDBMS. In fact, PHP/MySQL NeoBoard is pretty fast. Given the skins and stuff, Zope NeoBoard should be slower. But I found (and am a bit worried) that pulling data from ZODB is too much slower than selecting rows from MySQL. I'd have to re-design NeoBoard to use MySQL as storage backend like I did in good ol' days. bummer... Bear with my ignorance^^ You have every right not to answer this stupid question ;-) Thanks for your time anyway. Best Regards, Wankyu Choi --------------------------------------------------------------- To the dedicated staff at NeoQuest, language is not a problem to be dealt with, but an art waiting to be performed. --------------------------------------------------------------- Wankyou Choi CEO/President NeoQuest Communications, Inc. 3rd Floor, HMC Bldg., 730-14, Yoksam-dong, Kangnam-gu Seoul, Korea Tel: 82-2 - 501 - 7124 Fax: 82-2-501-7058 Corporate Home: http://www.neoqst.com Personal Home: http://www.zoper.net, http://www.neoboard.net e-mail: wankyu@neoqst.com ---------------------------------------------------------------