I've been trying to roll my own RSS parsing in Zope by using Mark Hammond's feedparser python module. I've been able to get it to install it successfully, and have been able to get Zope to let Scripts (Python) import it natively, by creating a new directory called "Other" in the Products directory, and putting an __init__.py file in it with the following 2 lines: from Products.PythonScripts.Utility import allow_module allow_module('feedparser') Basically, the parser is supposed to return a dictionary containing the rss feed information. If I create and run the following python script: import feedparser uri = 'http://slashdot.org/slashdot.rdf' return feedparser.parse(uri) and click the "test" view in the ZMI, I see a single, REALLY long line of text that is the string representation of the output dictionary (the real output, with live data from slashdot). And this seems to be a real dictionary, not just a string, because len(feedparser.parse(uri)) returns 9. But I can't seem to access any of the dictionary functionality, because when I try to, the ZMI keeps asking me to log in again (even though I'm logged in as the site manager), won't accept any of my login credentials (either site manager or emergency user), and then, when I ultimately click "cancel", tells my I don't have the authorization to access this element in this context. For example, if I create and run the following python script import feedparser data = feedparser.parse('http://slashdot.org/slashdot.rdf') return data['channel'] I receive the error message "You are not authorized to access 'channel' in this context.". What's going on? Placing the feedparser.parse() functionality in an external method doesn't help either, I get the same results (viewing the top dictionary returns shows a string representation of the dictionary, viewing any subelement of the dictionary causes an "Unauthorized error). Feedparser doesn't have any outside dependencies (at least, I certainly didn't install any, and it works perfectly in the python shell). Does Zope restrict the use of dictionaries somehow? Are there some other access granting commands I should put in __init__.py ? I'm using Zope 2.7 with python 2.3 on linux. Any help is greatly appreciated! Greg Steffensen
Greg Steffensen wrote at 2004-5-3 12:17 -0400:
... And this seems to be a real dictionary, not just a string, because len(feedparser.parse(uri)) returns 9. But I can't seem to access any of the dictionary functionality, because when I try to, the ZMI keeps asking me to log in again (even though I'm logged in as the site manager)
Apparently, it is not a true dictionary but instead an object that implements the dictionary interface. You must provide security declarations for such objects. You find details in the Zope Developer Guide. -- Dieter
basic i know - but could someone point me towards any instructions on how to backup and restore a zope installation? i can't find anything in the admin guide and deja is not bringing anything up either. thanks, kev bailey
Kevin Bailey <kbailey@freewayprojects.com> on 5/23/04 wrote:
basic i know - but could someone point me towards any instructions on how to backup and restore a zope installation?
i can't find anything in the admin guide and deja is not bringing anything up either.
checkout http://www.plope.com/Members/chrism/repozo <--> george donnelly ~ http://www.zettai.net/ ~ "Quality Zope Hosting" Shared and Dedicated Zope Hosting ~ Zope Servers ~ Zope Websites Yahoo, AIM: zettainet ~ ICQ: 51907738 ~ Sales (USA): 1-866-967-3669
Kevin Bailey wrote at 2004-5-23 17:47 +0100:
basic i know - but could someone point me towards any instructions on how to backup and restore a zope installation?
Backup: You copy "var/Data.fs" to a safe place Restore: You restore "var/Data.fs" from this safe place. -- Dieter
On Mon, 2004-05-24 at 18:57, Dieter Maurer wrote:
Kevin Bailey wrote at 2004-5-23 17:47 +0100:
basic i know - but could someone point me towards any instructions on how to backup and restore a zope installation?
Backup: You copy "var/Data.fs" to a safe place Restore: You restore "var/Data.fs" from this safe place.
Don't you have to stop zope first?
Nope. I use rsync every night to back up the Data.fs. http://zopezone.com/faq/1085419040 The secret sauce is rsync which only moves the part of the file that has changed. Jake -- http://www.ZopeZone.com Jerome R. Westrick said:
On Mon, 2004-05-24 at 18:57, Dieter Maurer wrote:
Kevin Bailey wrote at 2004-5-23 17:47 +0100:
basic i know - but could someone point me towards any instructions on how to backup and restore a zope installation?
Backup: You copy "var/Data.fs" to a safe place Restore: You restore "var/Data.fs" from this safe place.
Don't you have to stop zope first?
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
RSYNC is unsuitable for backing up files that might change as rsync examines/backs it up. Do not use it. jens On May 24, 2004, at 1:18 PM, Jake wrote:
Nope.
I use rsync every night to back up the Data.fs.
http://zopezone.com/faq/1085419040
The secret sauce is rsync which only moves the part of the file that has changed.
Jake -- http://www.ZopeZone.com
Jerome R. Westrick said:
On Mon, 2004-05-24 at 18:57, Dieter Maurer wrote:
Kevin Bailey wrote at 2004-5-23 17:47 +0100:
basic i know - but could someone point me towards any instructions on how to backup and restore a zope installation?
Backup: You copy "var/Data.fs" to a safe place Restore: You restore "var/Data.fs" from this safe place.
Don't you have to stop zope first?
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Been using it for years, no problems. Jake -- http://www.ZopeZone.com Jens Vagelpohl said:
RSYNC is unsuitable for backing up files that might change as rsync examines/backs it up. Do not use it.
jens
On May 24, 2004, at 1:18 PM, Jake wrote:
Nope.
I use rsync every night to back up the Data.fs.
http://zopezone.com/faq/1085419040
The secret sauce is rsync which only moves the part of the file that has changed.
Jake -- http://www.ZopeZone.com
Jerome R. Westrick said:
On Mon, 2004-05-24 at 18:57, Dieter Maurer wrote:
Kevin Bailey wrote at 2004-5-23 17:47 +0100:
basic i know - but could someone point me towards any instructions on how to backup and restore a zope installation?
Backup: You copy "var/Data.fs" to a safe place Restore: You restore "var/Data.fs" from this safe place.
Don't you have to stop zope first?
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
You'd see problems if your databases grow into the gigabyte range. We have and we threw out rsync really fast. jens On May 24, 2004, at 2:50 PM, Jake wrote:
Been using it for years, no problems.
Jake -- http://www.ZopeZone.com
Jens Vagelpohl said:
RSYNC is unsuitable for backing up files that might change as rsync examines/backs it up. Do not use it.
jens
On May 24, 2004, at 1:18 PM, Jake wrote:
Nope.
I use rsync every night to back up the Data.fs.
http://zopezone.com/faq/1085419040
The secret sauce is rsync which only moves the part of the file that has changed.
Jake -- http://www.ZopeZone.com
Jerome R. Westrick said:
On Mon, 2004-05-24 at 18:57, Dieter Maurer wrote:
Kevin Bailey wrote at 2004-5-23 17:47 +0100:
basic i know - but could someone point me towards any instructions on how to backup and restore a zope installation?
Backup: You copy "var/Data.fs" to a safe place Restore: You restore "var/Data.fs" from this safe place.
Don't you have to stop zope first?
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
If the Data.fs changes during the rsync session, rsync will laugh at you and copy crap. The recommended way is to create a copy of the Data.fs and rsync the copy of the Data.fs. -aj --On Montag, 24. Mai 2004 14:50 Uhr -0400 Jake <jake@zopezone.com> wrote:
Been using it for years, no problems.
Jake
Or use repozo: http://www.plope.com/Members/chrism/repozo On Mon, 2004-05-24 at 14:54, Andreas Jung wrote:
If the Data.fs changes during the rsync session, rsync will laugh at you and copy crap. The recommended way is to create a copy of the Data.fs and rsync the copy of the Data.fs.
-aj
--On Montag, 24. Mai 2004 14:50 Uhr -0400 Jake <jake@zopezone.com> wrote:
Been using it for years, no problems.
Jake
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Hmm... I have used it on a Data.fs from 300 -> 800 MB. What is to say the copy won't go bad if it gets written to at the time of the copy? Jake -- http://www.ZopeZone.com Andreas Jung said:
If the Data.fs changes during the rsync session, rsync will laugh at you and copy crap. The recommended way is to create a copy of the Data.fs and rsync the copy of the Data.fs.
-aj
--On Montag, 24. Mai 2004 14:50 Uhr -0400 Jake <jake@zopezone.com> wrote:
Been using it for years, no problems.
Jake
On Mon, May 24, 2004 at 02:59:06PM -0400, Jake wrote:
Hmm...
I have used it on a Data.fs from 300 -> 800 MB.
What is to say the copy won't go bad if it gets written to at the time of the copy?
This is my understanding: At least on unix, cp simply starts reading at the front of the file and reads bytes until EOF. Zope only appends to Data.fs. (It has been mentioned that it is *possible* to twiddle stuff in the middle of a filestorage, but the core Zope code is not supposed to do this, and I've never heard an example of any add-on product that does so either.) So, if Zope writes some data while you're copying, it doesn't matter - one of two things happen: 1) cp hits EOF before the new data is flushed to disk. Your new data doesn't make it into the backup, but the backup is perfectly good otherwise... yay transactions :-) 2) cp isn't up to the end yet. So the new data will be copied before cp gets to EOF. Either way, you're fine. rsync is different because it actually analyzes the file for differences and this can get confused if the file changes while it's working. Apparently this is more likely to be a problem as the file gets larger. -- Paul Winkler http://www.slinkp.com
[mailto:zope-bounces+zope=benlast.com@zope.org]On Behalf Of Paul Winkler Zope only appends to Data.fs. (It has been mentioned that it is *possible* to twiddle stuff in the middle of a filestorage, but the core Zope code is not supposed to do this, and I've never heard an example of any add-on product that does so either.) Out of curiosity, what then prevents Data.fs growing without bound? Does it get truncated from the beginning? Even if it does only get appended to, I assume there's a race condition where cp may encounter Data.fs with half or a record written to it and thus copy an inconsistent file.
ben
On Tue, May 25, 2004 at 06:40:01AM +0100, Ben Last (Zope) wrote:
[mailto:zope-bounces+zope=benlast.com@zope.org]On Behalf Of Paul Winkler Zope only appends to Data.fs. (It has been mentioned that it is *possible* to twiddle stuff in the middle of a filestorage, but the core Zope code is not supposed to do this, and I've never heard an example of any add-on product that does so either.)
Out of curiosity, what then prevents Data.fs growing without bound? Does it get truncated from the beginning?
No, never. Instead, we "pack" it - a process which deletes transactions older than a given date. I've never really looked into how packing is implemented for the default storage, but I know that it makes a backup copy (Data.fs.old), tells Zope to write new transactions since the pack started to Data.fs.tmp, and finally when done with its work appends Data.fs.tmp to the new Data.fs. If you never pack, your Data.fs will indeed only ever get larger... OTOH, be sure you do your homework and know what version of Zope you have and whether it has known pack bugs. I know that 2.6.1-2.6.2 had pack bugs and could lose data during a pack. To the best of my knowledge, 2.6.4 and 2.7.0 do not have this bug.
Even if it does only get appended to, I assume there's a race condition where cp may encounter Data.fs with half or a record written to it and thus copy an inconsistent file.
Not a problem. Due to the ZODB's transactional nature, all this means is that the last transaction in the backup is incomplete and invalid and Zope will ignore it. If we number transactions with consecutive integers, then a Data.fs with complete transactions 0..N and incomplete transaction N+1 is functionally equivalent to a Data.fs that ends with complete transaction N. -- Paul Winkler http://www.slinkp.com
--On Montag, 24. Mai 2004 14:59 Uhr -0400 Jake <jake@zopezone.com> wrote:
Hmm...
I have used it on a Data.fs from 300 -> 800 MB.
What is to say the copy won't go bad if it gets written to at the time of the copy?
The different is that copying a live Data.fs is like tracking a moving target, a static copy of the Data.fs is static and rsync knows how to handle static files but it does not handle files that grow or change during the copy phase. An inconsistent copy of the Data.fs is no problem since Zope will through away the last inconsistent transcation during the next startup. -aj
Andreas Jung wrote:
If the Data.fs changes during the rsync session, rsync will laugh at you and copy crap. The recommended way is to create a copy of the Data.fs and rsync the copy of the Data.fs.
Or, pack the ZODB, then rsync Data.fs.old :) -Matt -- Matt Hamilton matth@netsight.co.uk Netsight Internet Solutions, Ltd. Business Vision on the Internet http://www.netsight.co.uk +44 (0)117 9090901 Web Design | Zope/Plone Development & Consulting | Co-location | Hosting
participants (12)
-
Andreas Jung -
Ben Last (Zope) -
Chris McDonough -
Dieter Maurer -
george donnelly -
Greg Steffensen -
Jake -
Jens Vagelpohl -
Jerome R. Westrick -
Kevin Bailey -
Matt Hamilton -
Paul Winkler