How can I get a listing of all objects in the Zope 2.62 ZODB?
In Oracle it is easy to get a listing of everything that is in the Oracle database. One simply queries the system view dba_objects. Is there anything similar for the Zope 2.62 ZODB? I have been using Squishdot at goodbyejim.com for years. I still don't understand it. I would love to be able to see everything that it dumps into the ZODB. I understand that one can look at the Squishdot source to find out. But it ought to be possible to query the ZODB directly and observe all of the objects that are in it. Jonathan Mark
--On 12. September 2006 11:26:13 -0400 "Mark, Jonathan (Integic)" <jonathan.mark@integic-hc.com> wrote:
In Oracle it is easy to get a listing of everything that is in the Oracle database. One simply queries the system view dba_objects.
Is there anything similar for the Zope 2.62 ZODB? I have been using Squishdot at goodbyejim.com for years. I still don't understand it. I would love to be able to see everything that it dumps into the ZODB.
Mark, Jonathan (Integic) wrote at 2006-9-12 11:26 -0400:
In Oracle it is easy to get a listing of everything that is in the Oracle database. One simply queries the system view dba_objects.
Is there anything similar for the Zope 2.62 ZODB? I have been using Squishdot at goodbyejim.com for years. I still don't understand it. I would love to be able to see everything that it dumps into the ZODB.
In modern ZODB versions (I think ZODB 3.6 and above), there is an iterator which allows you to iterate over all objects in the ZODB. It main use case is to perform migrations between major updates. But, if you know how to "list" your objects intelligently (this is much more difficult than with Oracle where all objects have a strict schema), you definitely can do this as well (but I doubt, you will be happy).
I understand that one can look at the Squishdot source to find out. But it ought to be possible to query the ZODB directly and observe all of the objects that are in it.
The ZODB level is *very* *very* low! You do *NOT* want to work on this level when you have an alternative... -- Dieter
On 9/12/06, Mark, Jonathan (Integic) <jonathan.mark@integic-hc.com> wrote:
In Oracle it is easy to get a listing of everything that is in the Oracle database. One simply queries the system view dba_objects.
Is there anything similar for the Zope 2.62 ZODB?
ZODB is an hierarchical object database. So, yes, there is and no there isn't. You can't just make a query and get all the data, the data is stored as objects. On the other hand, getting tha data is dead easy, although you don't make a query. You just write python. For example, to get all the subobjects for a particular object, you write subobjects = the_object.objectValues() Tada! Do that recursively, and you pretty much get all the objects you would be interested in. Then you need to understand each object type, but that's another thing. ;-)
Mark, Jonathan (Integic) wrote:
Is there anything similar for the Zope 2.62 ZODB? I have been using Squishdot at goodbyejim.com for years. I still don't understand it. I would love to be able to see everything that it dumps into the ZODB.
OK. Can I ask why you're interested? (the answer to that question will help give you more sensible answers to your first question.)
to find out. But it ought to be possible to query the ZODB directly and observe all of the objects that are in it.
It is, but, as with your Oracle example, that's rarely very helpful... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers, the developer of Squishdot, asks:
OK. Can I ask why you're interested?
(the answer to that question will help give you more sensible answers to your first question.)
My experience using Squishdot to run goodbyejim.com for several years has been a positive one. I am grateful to Chris for writing it and am glad that he has moved on to other cutting edge Zope projects. However, there is one problem. As the number of posts gets past 2000 the application slows to a crawl. You can observe the problem by comparing http://jonathanmark.com/Blog/1128110364/index_html, which is in a Squishdot instance with 3000 posts and is slow, and http://goodbyejim.com, which has 800 posts. The front page of jonathanmark.com/Blog is cached, hence one must visit an interior page to observe the problem. A second Squishdot instance at http://jonathanmark.com/2005Blog/1147676261/index_html also seemed to be slowing down after 2000 posts. All of these Squishdot instances run in the same Zope instance. So I decided that I would modify Squishdot so that it uses a relational database to store posts. I am certain that through proper use of RDBMS indexes Squishdot would be scalable into the tens of thousands of messages. For a small blog such as mine with several thousand posts a year even Gadfly might be good enough. To convert Squishdot to use an RDBMS I would have to modify the Posting and other product scripts in Squishdot. If I could look at the Squishdot-related objects in the ZODB I would have a better understanding of what Squishdot is doing when it posts and retrieves a message. I could then duplicate that behavior with Gadfly. If I may add my two cents on a related matter, what I like best about Zope 2 is the VHM, undo, and content management. I work with RDBMSs every day and understand them far better than I do the ZODB. Thus storing textual transactional data in the ZODB makes the system far more complex for DTML scripters like me. If one were to ask why Ruby on Rails and Django get the attention that Zope does not, I feel that a reliance on using the ZODB for textual transactional data is one reason. It sounds more difficult to set up an RDBMS app in Zope than in Django or Ruby on Rails. I have never done any of those things, but that is the hype. Since users prefer RDBMSs (that is what we know) new developers are going to gravitate towards Django or RoR. Wouldn't it be great if there could be a Zope app that would set up RDBMS tables the way Django does, or generate code to access tables the way RoR does? But for lack of a better idea in the meantime it is best for me to migrate my version of Squishdot to use an RDBMS.
Mark, Jonathan (Integic) wrote:
has been a positive one. I am grateful to Chris for writing it and am glad that he has moved on to other cutting edge Zope projects.
What gives you the idea I've "moved on" anywhere? I still use Squishdot heavily and deployed a new instance for a customer a couple of months ago...
However, there is one problem. As the number of posts gets past 2000 the application slows to a crawl.
That's not been my experience at all.. But, it does highlight one thing: you must be a developer - coming with a solution rather than a problem and asking a question that has little to do with either ;-) The question I believe you meant to ask is "My squishdot site is slow, how can I speed it up?" rather than this stuff about zodb internals. You could have even asked on the Squishdot mailing list ;-)
You can observe the problem by comparing http://jonathanmark.com/Blog/1128110364/index_html, which is in a Squishdot instance with 3000 posts and is slow, and http://goodbyejim.com, which has 800 posts. The front page of jonathanmark.com/Blog is cached, hence one must visit an interior page to observe the problem.
Caching can be a legitimate solution to these problems, though I've never needed it myself...
All of these Squishdot instances run in the same Zope instance.
Okay, in this order: - what version of Squishdot are you using? - what version of Zope? - what customisations have you done to the various templates? (I'd put money on something in here inadvertantly trying to drag all the posts into memory...) - what size is your Data.fs? - How much memory does the box have? - What configuration changes have you made to Zope to ensure the object cache is of a useful size?
So I decided that I would modify Squishdot so that it uses a relational database to store posts.
"hey, this hammer's broken, gimme that wrench and I'll see if I can bash this nail in" This is an awful lot of work for very little gain, why don't you actually try and solve the problem you're having rather than rewriting the entire app/framework/whatever?
I am certain that through proper use of RDBMS indexes Squishdot would be scalable into the tens of thousands of messages.
I'm certain that proper use of ZCatalog indexes, which modern Squishdots already use, will scale just as well ;-)
For a small blog such as mine with several thousand posts a year even Gadfly might be good enough.
Hahahaha... ok, you made me laugh. Seriously though, don't even go anywhere near gadflu...
To convert Squishdot to use an RDBMS I would have to modify the Posting and other product scripts in Squishdot.
...which will take about 100 times more effort than figuring out what's slowing down your sites ;-)
If I could look at the Squishdot-related objects in the ZODB I would have a better understanding of what Squishdot is doing when it posts and retrieves a message.
No, you really wouldn't.
If I may add my two cents on a related matter, what I like best about Zope 2 is the VHM, undo, and content management.
You loose undo with your plan, as well as incurring an awful lot more work...
I work with RDBMSs every day and understand them far better than I do the ZODB.
Really? It doesn't show ;-) In all fairness, yes, if you understand RDB more, then your temptation will naturally be to run to it, but that is a mistake here...
more complex for DTML scripters like me. If one were to ask why Ruby on Rails and Django get the attention that Zope does not, I feel that a reliance on using the ZODB for textual transactional data is one reason.
I rather strongly disagree with this statement, but this isn't the right time or place to debate it...
It sounds more difficult to set up an RDBMS app in Zope than in Django or Ruby on Rails.
Indeed, whereas it is trivial to set up a ZODB-based app with indexing done using ZCatalog.
the hype. Since users prefer RDBMSs (that is what we know) new developers are going to gravitate towards Django or RoR.
They're welcome to it them...
Wouldn't it be great if there could be a Zope app that would set up RDBMS tables the way Django does, or generate code to access tables the way RoR does?
No. For me, it would suck slimy rocks through straws. Now, a really good ORM that worked transparently with Zope's transaction management and allowed interaction in a similar style to zodb _would_ be of interest, but Zope 3 is your best bet for that...
But for lack of a better idea in the meantime it is best for me to migrate my version of Squishdot to use an RDBMS.
I'm afraid I won't waste my time or yours helping you with that. If you want help with getting your sites faster, quickly, just answer the questions above... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Mark, Jonathan (Integic) wrote at 2006-9-13 11:06 -0400:
... To convert Squishdot to use an RDBMS I would have to modify the Posting and other product scripts in Squishdot. If I could look at the Squishdot-related objects in the ZODB I would have a better understanding of what Squishdot is doing when it posts and retrieves a message. I could then duplicate that behavior with Gadfly.
The "Gadfly" implementation included in Zope is bad -- do not use it in earnest. *If* you really want to go with a relational database, use a real one (Postgres, MySQL (with transactional tables), Oracle, ...). The alternative is to learn what make Squishdot slow and fix that. A profile (e.g. with "ZopeProfiler") may help you to understand what happens. -- Dieter
participants (5)
-
Andreas Jung -
Chris Withers -
Dieter Maurer -
Lennart Regebro -
Mark, Jonathan (Integic)