Storing Data inside Zope
Just how bad an idea is it to use Zope object Properties to store numeric/text application data? My first Linux (or otherwise) Zope app will be a gratis case tracking system for a friend's small (himself plus 3 staff) law office. Other than setting up new clients and cases (maybe a few a day) the most frequent data changes will be adding new activity detail records which are small. Only a couple of reports are run each day. Overall transactions will be few and small. My initial idea was to use MySQL or PostgreSQL for data storage. While developing the framework of the app I want to be able to store/retrieve a few records for testing. Since every bit of Zope (HTML, DTML, Python, etc.) is new to me, every time I start to do something I have to learn something else first. I'm not complaining - that's how I expected it to be - but if I could sort of skip over learning ZSQL and defining/using MySQL tables right now and stay focused on my input forms and reports, then come back to data storage later, that would really nice. My plan is to use scripts for all I/O which would allow me to change I/O methods later without changing whatever code is calling the scripts. I considered using text files or XML data files within Zope for testing but in each case I'd have to write the parsing and update-data code myself which would take time and be thrown out later by the final ZSQL code. Then I thought about defining some object that takes Properties, adding Property names/types for (example) the client master record and saving that as a data template (not ZPT). Then each time there was a new entry I'd make a copy of the template and update the Properties using Zope's built-in APIs without ever giving the data objects any content. I think I could also use Zope catalog features for data searches. Without knowing more about Zope the only downside I see is the data being encapsulated in the app tree, therefore requiring any data to be purged before copying/exporting the app somewhere. It will also make the Zope object grow but adding data is going to make any database grow no matter which is used. I can also imagine there might be a philosophical point about keeping the data outside the Zope object. But is it a bad idea for some practical reason? That's a question with two branches. The first is, is it completely bad and I shouldn't do it even temporarily for testing? The second is, is it bad for production but okay for development/testing? If there isn't a good reason for not doing it then it would certainly be nice to be able to stay in Zope for everything so that only Zope would have to be installed to move the package to new machines and so I can stay on track without having to branch off into data storage issues. I was thinking about using the File object as my data object since it supports Properties. Any other suggestions? Thanks, Tom P. Allred, MCSE, RHCE, CCNA Allred Data Consulting, Inc. 704-633-0311 x826 www.AllredData.com Tom@AllredData.com
Tom, I've been spending a fair amount of time attempting to answer that very set of questions. It shouldn't be necessary to set up an RDBMS just to cook up a simple app that needs to store a small number of records. I may have something to release late next week, but then again this project been within a week of release for several months now. :-) Drop me a line privately if I don't announce something by Friday and I'll give you a copy of what I've got so far. Dylan On Sat, 2003-05-31 at 08:22, Tom P. Allred wrote:
Just how bad an idea is it to use Zope object Properties to store numeric/text application data?
My first Linux (or otherwise) Zope app will be a gratis case tracking system for a friend's small (himself plus 3 staff) law office. Other than setting up new clients and cases (maybe a few a day) the most frequent data changes will be adding new activity detail records which are small. Only a couple of reports are run each day. Overall transactions will be few and small.
My initial idea was to use MySQL or PostgreSQL for data storage. While developing the framework of the app I want to be able to store/retrieve a few records for testing. Since every bit of Zope (HTML, DTML, Python, etc.) is new to me, every time I start to do something I have to learn something else first. I'm not complaining - that's how I expected it to be - but if I could sort of skip over learning ZSQL and defining/using MySQL tables right now and stay focused on my input forms and reports, then come back to data storage later, that would really nice. My plan is to use scripts for all I/O which would allow me to change I/O methods later without changing whatever code is calling the scripts. I considered using text files or XML data files within Zope for testing but in each case I'd have to write the parsing and update-data code myself which would take time and be thrown out later by the final ZSQL code.
Then I thought about defining some object that takes Properties, adding Property names/types for (example) the client master record and saving that as a data template (not ZPT). Then each time there was a new entry I'd make a copy of the template and update the Properties using Zope's built-in APIs without ever giving the data objects any content. I think I could also use Zope catalog features for data searches.
Without knowing more about Zope the only downside I see is the data being encapsulated in the app tree, therefore requiring any data to be purged before copying/exporting the app somewhere. It will also make the Zope object grow but adding data is going to make any database grow no matter which is used. I can also imagine there might be a philosophical point about keeping the data outside the Zope object. But is it a bad idea for some practical reason? That's a question with two branches. The first is, is it completely bad and I shouldn't do it even temporarily for testing? The second is, is it bad for production but okay for development/testing? If there isn't a good reason for not doing it then it would certainly be nice to be able to stay in Zope for everything so that only Zope would have to be installed to move the package to new machines and so I can stay on track without having to branch off into data storage issues. I was thinking about using the File object as my data object since it supports Properties. Any other suggestions?
Thanks, Tom P. Allred, MCSE, RHCE, CCNA Allred Data Consulting, Inc. 704-633-0311 x826 www.AllredData.com Tom@AllredData.com
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Tom P. Allred wrote:
Just how bad an idea is it to use Zope object Properties to store numeric/text application data?
It is not bad at all. Usually, I use this way to rapidly prototype applications. Once initial thoughts are ironed out, I make these into Zope Products. You might want to look for Abracadabra Object in www.zope.org - this packages exactly what you were describing (template folder; copying it to make new objects) in a nice way. - Babu http://vsbabu.org/
Just how bad an idea is it to use Zope object Properties to store numeric/text application data?
Depends. See below for a less glib response.
My first Linux (or otherwise) Zope app will be a gratis case tracking system for a friend's small (himself plus 3 staff) law office. Other than setting up new clients and cases (maybe a few a day) the most frequent data changes will be adding new activity detail records which are small. Only a couple of reports are run each day. Overall transactions will be few and small.
My initial idea was to use MySQL or PostgreSQL for data storage. While developing the framework of the app I want to be able to store/retrieve a few records for testing. Since every bit of Zope (HTML, DTML, Python, etc.) is new to me, every time I start to do something I have to learn something else first. ...
Unless you already have a relational database or your data is terrbily relational, I recommend not using an external database. Things become simpler that way. Zope objects are a good place to store data, but you should be thoughtful about how it is done. Which is to say, you should design object-orientedly and not expect to use Zope objects like tables. If you make one object to store a large set of data, you'll run into problems. Not the least of which is a rapidly growing storage due to the undo mechanisms. So factor your data space down to sets of properties or natural objects, and make those, either as a product or (as previously suggested) a PropertiesObject/Abracadabra system. You could also use ZClasses, but I wouldn't: they are, shall we say, problem prone. In your case you'll probably have a Client, which will have Cases, which will contain ActivityDetailRecords. A Client might have name, address, and whatnot properties, a Case might have description, judge, dates, etc properties, and ActivityDetailRecords might have date and contents properties. (I am of course just guessing about these.) This way you get a clean containment, plus most of the objects are immutable or rarely changed, so you only store the data you actually need to store. --jcc
Just how bad an idea is it to use Zope object Properties to store numeric/text application data?
Depends. See below for a less glib response.
Zope objects are a good place to store data, but you should be thoughtful about how it is done. Which is to say, you should design object-orientedly and not expect to use Zope objects like tables.
--jcc
I'm using the Client/Cases/Detail/etc granularity and making each record a single object (using File objects for testing) in an appropriate data folder (e.g. Case/Data, Client/Data or maybe Data/Client, Data/Case). I create properties with field names and assign them the field values. That way I can use the ZMI to create/clone/edit/delete 'data records' easily in addition to my programs. I'm using Formulator so I can tap the field definitions programmatically to get the info needed to define the File data object properties appropriately for each field. I thought about making a template File data object and cloning it as needed but I haven't figured out how to do that yet. Besides the dynamic Formulator approach lets me change form AND 'table' definitions in one spot which is nice for development. Being shiny new to Zope I'm still grasping the building blocks but thought ZClasses sounded like what might be useful for this. Thanks for letting me know that might not be a good idea. I'll continue to fumble through alternate approaches. It's painfully slow but at least it's interesting. Thanks to all who have contributed help, Tom P. Allred, MCSE, RHCE, CCNA Allred Data Consulting, Inc. 704-633-0311 x826 www.AllredData.com Tom@AllredData.com
I'm using the Client/Cases/Detail/etc granularity and making each record a single object (using File objects for testing) in an appropriate data folder (e.g. Case/Data, Client/Data or maybe Data/Client, Data/Case). I create properties with field names and assign them the field values. That way I can use the ZMI to create/clone/edit/delete 'data records' easily in addition to my programs. I'm using Formulator so I can tap the field definitions programmatically to get the info needed to define the File data object properties appropriately for each field. I thought about making a template File data object and cloning it as needed but I haven't figured out how to do that yet. Besides the dynamic Formulator approach lets me change form AND 'table' definitions in one spot which is nice for development.
Sounds like you might like http://www.zope.org/Members/mjablonski/PropertyObject and friends.
Being shiny new to Zope I'm still grasping the building blocks but thought ZClasses sounded like what might be useful for this. Thanks for letting me know that might not be a good idea. I'll continue to fumble through alternate approaches. It's painfully slow but at least it's interesting.
ZClasses does seem like a good idea. But regular products aren't that hard (esp. with mxmEasyProduct) and they're more portable, more powerful, and usually more understandable than the current ZClasses idea and implementation. I think that PropertyObject/Abracadabra is a good player in the TTW-product area. Also, Plone's Archetypes (which also works in vanilla CMF) is very cool: based on a simple schema it'll make an object/type with the right forms and properties, even with actions and input validation. Dunno if it's quite far enough along yet to be newbie usable, but it's almost to a release and should be included in future Plones. CMF/Plone might be a little much to swallow at the same time, but if you're interested, http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/archetypes/Archety... http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/archetypes/Archetypes/docs/qu... http://sourceforge.net/projects/archetypes http://www.plone.org/ http://cmf.zope.org/ --jcc
Sorry if this has already been suggested in this thread.. It seems to me that you are really in need of a hybrid system which uses both Zope and a relational data base. Zope and some specialized products to manage the front-end and query interfaces, and a relational database to handle complex queries. MySQL, at least the recent versions, supports full text indexing as well as the usual SQL relational queries. I suspect that would be invaluable for your application. You might consider (if your budget allows) incorporating a google-style search engine.
I'm using the Client/Cases/Detail/etc granularity and making each record a single object (using File objects for testing) in an appropriate data folder (e.g. Case/Data, Client/Data or maybe Data/Client, Data/Case). I create properties with field names and assign them the field values. That way I can use the ZMI to create/clone/edit/delete 'data records' easily in addition to my programs. I'm using Formulator so I can tap the field definitions programmatically to get the info needed to define the File data object properties appropriately for each field. I thought about making a template File data object and cloning it as needed but I haven't figured out how to do that yet. Besides the dynamic Formulator approach lets me change form AND 'table' definitions in one spot which is nice for development.
Being shiny new to Zope I'm still grasping the building blocks but thought ZClasses sounded like what might be useful for this. Thanks for letting me know that might not be a good idea. I'll continue to fumble through alternate approaches. It's painfully slow but at least it's interesting.
Thanks to all who have contributed help, Tom P. Allred, MCSE, RHCE, CCNA Allred Data Consulting, Inc. 704-633-0311 x826 www.AllredData.com Tom@AllredData.com
Dennis Allison wrote:
MySQL, at least the recent versions, supports full text indexing as well as the usual SQL relational queries.
Yeah, but both MySQL and PostGres full text indexing is pretty weak compared to ZCTextIndex... cheers, Chris
Chris, Could you comment on the trade-offs in features and performance. I'd assumed (perhaps wrongly) that this application might have a very large amount of text (multiple GB) and assumed that, in this case, the scale of the problem would be a mismatch to ZCTextIndex but not to the RLDB approach. On Tue, 3 Jun 2003, Chris Withers wrote:
Dennis Allison wrote:
MySQL, at least the recent versions, supports full text indexing as well as the usual SQL relational queries.
Yeah, but both MySQL and PostGres full text indexing is pretty weak compared to ZCTextIndex...
Dennis Allison wrote:
Chris, Could you comment on the trade-offs in features and performance. I'd assumed (perhaps wrongly) that this application might have a very large amount of text (multiple GB) and assumed that, in this case, the scale of the problem would be a mismatch to ZCTextIndex but not to the RLDB approach.
ZCTextIndex provides thinks like phrase matching, globbing, decent boolean combination and a search syntax that mirrors Google not too badly. The amount of text indexed is obviously related to the size of the index, but decent indexing should mean that the idnex is a lot smaller than the original text. Besides, this common misconception that ZODB isn't suitable for large amounts of data is still just that; a misconception... cheers, Chris
--On Dienstag, 3. Juni 2003 7:45 Uhr -0700 Dennis Allison <allison@sumeru.stanford.EDU> wrote:
Chris, Could you comment on the trade-offs in features and performance. I'd assumed (perhaps wrongly) that this application might have a very large amount of text (multiple GB) and assumed that, in this case, the scale of the problem would be a mismatch to ZCTextIndex but not to the RLDB approach.
I have not followed the complete thread but I would *not* store several GB ot texts to be be indexed inside the ZODB. You can store the text inside the ZODB but I would not recommend to use the ZCatalog infrastructure for a larger fulltext database. -aj
but I would not recommend to use the ZCatalog infrastructure for a larger fulltext database.
Why not? __________________________________ Do you Yahoo!? Yahoo! Calendar - Free online calendar with sync to Outlook(TM). http://calendar.yahoo.com
there is some point in terms of performance, size, reliability and usability where I would choose a commercial fulltext system over Zope. It is hard to say where is point is. The largest fulltext database I have ever build had about 10 million documents and about 60-70 GB of texts..I can not imagine having stored all these docs inside the ZODB + indexes. -aj --On Mittwoch, 4. Juni 2003 11:13 Uhr -0700 Stan Baptista <sbaptista@yahoo.com> wrote:
but I would not recommend to use the ZCatalog infrastructure for a larger fulltext database.
Why not?
__________________________________ Do you Yahoo!? Yahoo! Calendar - Free online calendar with sync to Outlook(TM). http://calendar.yahoo.com
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Andreas Jung wrote:
there is some point in terms of performance, size, reliability and usability where I would choose a commercial fulltext system over Zope. It is hard to say where is point is. The largest fulltext database I have ever build had about 10 million documents and about 60-70 GB of texts..I can not imagine having stored all these docs inside the ZODB + indexes.
It would certainly be interesting to try ;-) What solution did you use for the above in the end? cheers, Chris PS: What number and size of documents are involved in the project being discussed in this thread?
Chris, Six or seven tables/datasets and very small amounts of data from a handful of users. My original question was whether it was bad from a philosophical or practical perspective to use ZODB to store user data. The answer appears to be 'no'. I'm currently using a File object for each record and storing the data as Properties. My record creation script uses the input form data in my Formulator forms to define the File Properties so any changes I make in the input form are automatically reflected in new data records. I thought about using ZClasses to define data objects but someone suggested ZClasses had problems I didn't want to deal with. Thanks, Tom P. Allred, MCSE, RHCE, CCNA Allred Data Consulting, Inc. 704-633-0311 x826 www.AllredData.com Tom@AllredData.com -----Original Message----- From: Chris Withers [mailto:chrisw@nipltd.com] Sent: Thursday, June 05, 2003 5:26 AM To: Andreas Jung Cc: Stan Baptista; Dennis Allison; Tom P. Allred; 'J Cameron Cooper'; zope@zope.org Subject: Re: [Zope] Storing Data inside Zope Andreas Jung wrote:
there is some point in terms of performance, size, reliability and usability where I would choose a commercial fulltext system over Zope. It is hard to say where is point is. The largest fulltext database I have ever build had about 10 million documents and about 60-70 GB of texts..I can not imagine having stored all these docs inside the ZODB + indexes.
It would certainly be interesting to try ;-) What solution did you use for the above in the end? cheers, Chris PS: What number and size of documents are involved in the project being discussed in this thread?
Tom P. Allred wrote:
I thought about using ZClasses to define data objects but someone suggested ZClasses had problems I didn't want to deal with.
I think it's easier to use python classes instead. ZClasses are overly complicated. They should be rewritten into something that is easier to use, but has much less functionality, IMO. You could also use CMF and define up portal_types, but I think that may be a less neat solution.
participants (9)
-
Andreas Jung -
Chris Withers -
Dennis Allison -
Dylan Reinhardt -
J Cameron Cooper -
Lennart Regebro -
Satheesh Babu -
Stan Baptista -
Tom P. Allred