December 2011 - Zope

Re: [Zope] Help in deciding approach to Web App
by Sareesh Sudhakaran 06 Dec '11

06 Dec '11

Thank you all for your replies. I have an idea on how to begin: I'll go with mySQL and see how it works out. I will use both PHP and Python for the prototype. My issue no longer fits within the scope of this mailing list. Without your support it would have been impossible for me to get started. Thanks!sareesh

1 0

Help in deciding approach to Web App
by Sareesh Sudhakaran 06 Dec '11

06 Dec '11

I have a personal project - a web application I wanted to develop - but I'm confused on which route to take. I am not under any time constraint. About the App:The best I can describe it is as a kind of expert system (but not AI) that needs to find the best workflow for a process, given a set of initial and final parameters. E.g. a 'capsule' of data must pass through many 'tools' or 'environments' to reach a desired output - something like a very complicated car wash.Let's say there are many tools that can be used at various stages in the process. I have estimated there are at least 500 tools as of now, and it is bound to grow in the future as newer tools are introduced. Existing tools will also have version updates.Each tool, on average, has at least 100 properties that define the tool. Some of them have as high as 1000 unique properties. Some of these tools are linked to each other - e.g if one tool is selected, there are only n tools that can correspond to it for the next step in the process. I also have the problem of 'matching' the tools for analysis. E.g. Tool A might have only three fixed rpms - 100, 200 and 500, but Tool B might have rpms from 20 to 2000. I'm not sure how I can construct a database without spelling out each number, as in the example above.The total number of tools needed for the process can be defined at the beginning, however, it will change as the application becomes more complex in the future. I plan to address every contingency in the process. The idea is - if the user inputs the initial parameters and the desired outcome (another set of parameters), the app must find the 'best' path - sort of like a decision tree. The best path can be the fastest, cheapest, etc. I would like the user to choose what is best for him/her.Unfortunately, parameters might change, relationships might change (but not regularly) - the 'rules' I will be using might be revised for better accuracy in prediction.I also need to track each user's path and solutions' for future reference (but no personal details except username and email address for logging in). Maybe when the app is up and running, I'd like to make it more democratic, with users contributing to refining the logic/rules involved.If possible, I would also like the app to output a graphical flowchart at the end showing the workflow with all tools grouped in an easy to understand layout. My questions:Will the app be better served with a relational DB like mySQL or an Object database? After a lot of research I've guessed that my particular case might be better served with Python and Zope/ZODB. But I might be wrong? Maybe PHP+mySQL or Django is a better fit?Can anyone provide general advice on how to go about beginning such a project in ZOPE. Which is the best place to start learning for a newbie? Can anyone recommend a good shared hosting provider that supports Zope fully but is not expensive? Is there a module or app that is open source that I can use to output a graphical flowchart based on the results, or will I be better served programming it from scratch with Python?I would appreciate any help in getting started. Thank you in advance. I have tried most online forums but have not good any productive answers. Most of the answers I got were pro-PHP+mySQL. Adam

6 7

Re: [Zope] Zope Digest, Vol 91, Issue 2
by Sareesh Sudhakaran 06 Dec '11

06 Dec '11

Hi John and FernandoIntuitively I feel that my project fits in with an Object DB, which is why I have spent a lot of time trying to understand its methodology. But now I'm more confused than ever. To answer your questions - Only an expert can perform data entry for the tools, obviously. Since I have no clue how to enter data into mySQL or ZODB I have no idea what's in store for me. The ultimate goal is for the engine to become an expert system that is capable of taking inputs, learning from mistakes, scraping data from reports and from the internet, and become better and better at its predictions. The logic engine will get better and more complex as more tools are added - but once all the 'current' tools are added, the system must be able to go 'solo', with an accuracy that reaches six sigma levels. As Fernando has suggested, I will have to go with C/C++, or even my own language in the future, but that day is far off. I plan to start with 50 higher-level tools and try to build a prototype. The front-end is really very simple. The choices available for the start parameters and end parameters are pretty basic, and most of the logic happens under the hood - probably at the back end if possible. Once I've built this prototype, I will know whether mqSQL can handle scaling up (in my particular case). If not, I can confidently invest in ZODB. If I can make my position any clearer in order to get a conclusive answer either way - I'll be glad to. Thanks again for your help. I really am indebted to you guys.sareesh > From: zope-request(a)zope.org > Subject: Zope Digest, Vol 91, Issue 2 > To: zope(a)zope.org > Date: Mon, 5 Dec 2011 12:00:02 +0000 > > Send Zope mailing list submissions to > zope(a)zope.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.zope.org/mailman/listinfo/zope > or, via email, send a message with subject or body 'help' to > zope-request(a)zope.org > > You can reach the person managing the list at > zope-owner(a)zope.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Zope digest..." > > > Today's Topics: > > 1. Re: Help in deciding approach to Web App (Fernando Martins) > 2. Re: Help in deciding approach to Web App (Sareesh Sudhakaran) > 3. Re: Help in deciding approach to Web App (Fernando Martins) > 4. Re: Help in deciding approach to Web App (John Schinnerer) > 5. Re: Help in deciding approach to Web App (Sareesh Sudhakaran) > 6. Re: Help in deciding approach to Web App (Fernando Martins) > 7. Re: Help in deciding approach to Web App (John Schinnerer) > 8. Re: Help in deciding approach to Web App (Niels Dettenbach) > 9. Re: Help in deciding approach to Web App (Fernando) > 10. Re: Help in deciding approach to Web App (Sareesh Sudhakaran) > 11. Re: Help in deciding approach to Web App (Niels Dettenbach) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 04 Dec 2011 13:28:36 +0100 > From: Fernando Martins <fernando(a)cmartins.nl> > To: Sareesh Sudhakaran <aysand(a)hotmail.com> > Cc: zope(a)zope.org > Subject: Re: [Zope] Help in deciding approach to Web App > Message-ID: <4EDB6774.1050603(a)cmartins.nl> > Content-Type: text/plain; charset="iso-8859-1"; Format="flowed" > > On 12/04/2011 09:52 AM, Sareesh Sudhakaran wrote: > > Thanks Niels. Just to clarify: > > Does my particular instance fall under an OODBMS model or a RDBMS > > model (with ORM if necessary)? > > > > > Data modelling is a bit of an art and probably you could tackle your > problem with any approach. I think the important is for you to figure > out which model suits more your personality. No kidding. > > I would personally start with the RDBMS approach considering only the > information you provide. Also, you can easily use zope in combination > with a RDBMS. When you read the book, beware that zope has been changing > from a "through the web" approach, to a typical file system based > approach, which is a loss, but it seems to be what suits the needs of > the zope developers. > > The approach I use is: > > zpt page -> (one) Python Script -> (some) ZSQL Methods -> MySQL > > The zpt defines the layout of the pages, the ZSQL Methods retrieve the > data needed and the Python script massages the data to make it suitable > for the ZPT page. Or the other way around, from user input in a form to > storage in the database. > > The advantage of the relational approach is that it is a very well > understood model and although different people will still come to > different models there are sufficient objective guidelines out there > (e.g., the normalization rules, and then criteria for when to > denormalise) to help you along. Furthermore, there are lots of people in > db-related forums that can help you. > > Also, RDBMS provides you with a "standard" query language, SQL, which > plenty of systems and tools can use. In general, RDBMS gives you the > safest approach to keep your data and not loose it when you need to > migrate either the front-end or the back-end. This language is very > powerful and can avoid you a lot of low level programming. > > However, plenty of people can not deal well with SQL because it follows > a paradigm so different from the classic imperative programming. With > SQL, you specify the "what", with the other languages you specify the > "how". The advantage of the "what" is that you can do a lot of data > processing in a few lines of code. The problem with the "what" is that > because you don't know the "how" of it, you feel you don't have control > and you are led to say the language is "obscure" or "unreadable". > > However, even if you are not comfortable with the "what" (you have to > try to know), you can still rely on an library like SQLAlchemy to keep > you a bit in the comfort zone of the "how". So instead of learning SQL, > you need to learn the API of a specific library. Your choice. I > recommend the first by far. > > The real main issue with Relational is that it is a highly structured > data model. It allows you to keep high quality data but if you don't get > it right soon enough in the development cycle, some later changes can > have a huge impact in the application, requiring rewrites. Furthermore, > it works the best when you have many objects with the same properties. > If you have many entities all different from each other (the tools in > your case, maybe), then maybe a OODBMS might be better. But here, there > is no standard language, or standard whatever. Perosnally, I would avoid > as much as possible to put data in something like ZODB (I use it merely > to store and manage the application). > > The problem with your specific case is that it does not seem to be a > typical case of books and authors, which might be a risk for someone > without experience. The issue "Tool A might have only three fixed rpms - > 100, 200 and 500, but Tool B might have rpms from 20 to 2000", is indeed > tricky. I suspect in general the needs of your system will be too > specific to be able to rely only on SQL queries. You would need to put a > lot of intelligence in the data (really highly structured) and it might > become unmanageable or not scalable. > > I guess you will need to put a lot of intelligence in the Python Script. > So, the ZSQL retrieves the relevant tool data and then makes the tool > choice. The knowledge of the meaning of the attributes is maintained in > programming. > > I should say I am not totally sure the Python Script is the best > approach here, in terms of management facilities. But Python is surely a > very good language due to its readability. However, you might need to > use External methods or a more typical file-system based Python approach. > > Or maybe you actually need to create a Domain Specific Language to > encode the intelligence needed for your tool selection process. If your > python code becomes repetitive, with patterns showing up, then a DSL > might be a good approach, but this might be too much engineering for you > at this stage. It looks like you are in a typical CIM scenario and I > remember handling a similar problem 20 years ago. I vaguely remember at > that time to use custom graph structures in C and the the intelligence > was coded in Lisp/Scheme. So, there is a big range of solutions to try > out :) > > If you have time, then the simple approach > > zpt page -> (one) Python Script -> (some) ZSQL Methods -> MySQL database > > might be a good starting point. You should not need to spend much time > to implement a prototype using this approach. In the worse case scenario > it helps you understand better your problem and what could be a better > approach with little investment. Essentially you try the classical > approach and if it does not work well, either you are doing something > wrong, or you have a better understanding of your needs and where to go. > > Good luck, > Fernando > >

3 4

Fwd: Re: Re: Help in deciding approach to Web App
by Fernando 05 Dec '11

05 Dec '11

On Dec 5, 2011 14:38 "Niels Dettenbach" <nd(a)syndicat.com> wrote: > Am Montag, 5. Dezember 2011, 14:23:57 schrieben Sie: > > How would you put 500+ objects (for the tools) each with hundreds or > > thousands of attributes in in ZODB? > > This depends from the object structure and transforms to do with it, > but - as > i wrote - it could make sense to use different database solutions in > parallel > for the same application and/or different parts of the same data > objects. > > I was asking about tools and technology, not about object design. Would you develop a Zope Product (or set of Products) or is there a different approach. The Zope book tells us nothing about creating custom objects in ZODB. Assuming a simple case of a web form to input a set of fixed attributes to define a tool object in ZODB, what is the simplest approach to create/update an object in ZODB? > Putting 500+ data objects with very different and changing structures > into a RDBMS could be a much larger mess... > > I agree, that's why I mentioned in my first post that this is not a typical Books and Authors data model. In a case like this, without more requirements information, I would say a table to define tools only with common attributes, another table to define relationships between tools, another table to list all possible attribute names and another one to define the specific attribute values for the tools. But this discussion is off-topic and even irrelevant with so little specification. Regards, Fernando

1 0

Re: [Zope] Help in deciding approach to Web App
by Sareesh Sudhakaran 05 Dec '11

05 Dec '11

Thanks Niels. Just to clarify:Does my particular instance fall under an OODBMS model or a RDBMS model (with ORM if necessary)? I will begin by reading the Zope Book. Thanks for your assistance. Appreciate it. -ss > Subject: Re: [Zope] Help in deciding approach to Web App > From: nd(a)syndicat.com > Date: Sun, 4 Dec 2011 09:24:10 +0100 > To: aysand(a)hotmail.com; zope(a)zope.org > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > > > > > Sareesh Sudhakaran <aysand(a)hotmail.com> schrieb: > > >My questions:Will the app be better served with a relational DB like > >mySQL or an Object database? After a lot of research I've guessed that > >my particular case might be better served with Python and Zope/ZODB. > >But I might be wrong? Maybe PHP+mySQL or Django is a better fit?Can > >anyone provide general advice on how to go about beginning such a > >project in ZOPE. > This hardly depends from your data structure, but with Zope you have the option to use ZODB and SQL DBs like MySQL in parallel. Typical relational data should go into a SQLDB while complex / oo Data structures should go into ZODB. > > We developed several complex to very complex web based applications - incl. expert systems - on Zope ZODB plus MySQL. I can't believe that someone would be able to solve such issues with PHP/SQL within the same time / ressources. > > > Which is the best place to start learning for a > >newbie? > You should start by trying the short zope practice / excercise as described in the Zope book. > > The Zope Book should be the best source for getting into Zope step by step from "nothing". ß) > > > Can anyone recommend a good shared hosting provider that > >supports Zope fully but is not expensive? > Looking for "zope hosting" or similiar in google should advice you. Our company - as one of many - offers Zope hosting to. > > > Is there a module or app that > >is open source that I can use to output a graphical flowchart based on > >the results, or will I be better served programming it from scratch > >with Python? > There are different modules as i.e. Python Imaging (PIL) and higher level modules. > > > I would appreciate any help in getting started. Thank you > >in advance. I have tried most online forums but have not good any > >productive answers. Most of the answers I got were pro-PHP+mySQL. > > Yes, because the peoples did not know anything other solution usually... > > > cheers, > > > Niels. > > - -- > Niels Dettenbach > Syndicat IT&Internet > http://www.syndicat.com > -----BEGIN PGP SIGNATURE----- > Version: APG v1.0.8 > > iIEEAREIAEEFAk7bLio6HE5pZWxzIERldHRlbmJhY2ggKFN5bmRpY2F0IElUJklu > dGVybmV0KSA8bmRAc3luZGljYXQuY29tPgAKCRBU3ERlZRyiDUhlAJ4/XPl3Oet6 > XN4UlkQ611FNoWqZCwCfZ1IPVjaLMD32wOlFE9cDnrm6bJQ= > =ukyi > -----END PGP SIGNATURE----- >

4 8