October 2000 - Zope-Dev

PythonPiece, perlpiece. You can use the Big PP or the little pp!
by James Johnson 25 Oct '00

25 Oct '00

Is it not just another piece of the puzzle to figure out? :-) Get your Free E-mail at http://tacoma.zzn.com ____________________________________________________________ Get your own Web-Based E-mail Service at http://www.zzn.com

6 6

RE: [Zope-dev] Large file support
by Toby Dickenson 25 Oct '00

25 Oct '00

> I should also note that if you create a producer, you will have to > override the __len__ method to return the entire length of the data. > > This is because RESPONSE.write doesn't allow you to set the > length of a > write and there code during output that checks the size of the written > object. > > Wahay! thats been on my todo list for ages. Ill take a look > when I get > > some time. Damn. Its back on my to-do list then ;-)

1 0

Large file support
by seant＠superchannel.org 25 Oct '00

25 Oct '00

I have been building an "ExternalFile" class which stores the body of the file in an external file, mirroring the Zope path/hierarchy. This will allow easy integration with servers that can mount the external representation of the content and serve it with a consistent namespace. To make life zimple, I tried to move all file manipulation to Zope, including upload/download/copy/cut/paste/delete and permissions. These external files are transaction aware, blah blah.. Working with files > 20MB I notices some serious performance/scalability issues and investigated. Here are the results. A diff with my changes against version 2.2.2 is available at <http://www.superchannel.org/Playground/large_file_zope2.2.2_200010241.diff> Concerns: Zope objects like File require data as a seekable file or as a coherent block, rather than as a stream. Initializing/updating these objects *may* require loading the entire file into memory. In memory buffering of request or response data could cause excessive swapping of the working set. Multi-service architecture (ZServer->ZPublisher) could limit the reuse of stream handles. Creating temporary files as FIFOs buffers between the services causes signficant swapping. Modifications: Using pipes I found that FTPServer.ContentCollector was using a StringIO to buffer the uploads from FTP clients. I changed this into a TemporaryFile for a while which revealed the leaked file descriptor bug (see below). This intermediary temp file caused 1 extra file copy for each request. The goal is to not have any intermediary files at all, and pipeline the content directly into the Zope objects. To remove this FTP upload file buffer, I converted the FTP collector again from a TemporaryFile into a pipe with a reader and writer file objects. The FTPRequest receives the reader from which it can process the input on the publish thread in processInputs. Since we are dealing with blocking pipes it is OK to have a reader on the publish thread and a writer on the ZServer thread. The major considerations were regarding the proper way to read from a pipe through the chain of control, especially in cgi.FieldStorage. Stdin is treated as the reader of the pipe throughout the code. All seek()s and tell()s on sys.stdin type objects (a tty not a seekable file) should be considered illegal and removed. Usage of FieldStorage from FTP (Unknown content-length) To gain access to the body of a request, one typically calls REQUEST['BODY'] or REQUEST['BODYFILE']. This returns the file object the FieldStorage copied from stdin. To prevent FieldStorage from copying the file from stdin to a temporary file, we can set the CONTENT_LENGTH header to '0' in the FTP _get_env for a STOR. In this case, FieldStorage creates a temporary file but doesn't read any data from stdin so we can return stdin directly when BODYFILE is requested and 'content-length' is '0'. However, BODYFILE could be a pipe which doesn't support 'seek' or 'tell'. The code used to suck the data off the BODYFILE needs to be modified to adapt to the possibly of being passed a pipe. Updating Image.File to play with pipes The _read_data method of Image.File pulls the data out of the BODYFILE and sticks it in the instance as a string, pdata object, or a linked list of pdata objects. The existing code reads and builds the list in one clean sweep back-to-front. I belive this keeps the pdata.data chunks out of memory, quickly (sub)committing then deactivating (_p_changed = None) them. Since we can no longer safely assume 'seek' is valid for BODYFILE, I tried to read and build the list front-to-back. This kept the data in memory, even though I tried to deactivate the objects quickly. As a tradeoff, I read the data front-to-back then built the list back-to-front taking another pass to reverse the list so it is in the correct order. Memory usage appears to be steady, meaning the whole file is not loaded into the working set. This also prevents unecessary reading into a temporary FieldStorage file during an FTP upload. Web based uploads... ...suck. I do not recommend doing a web based upload for files > 1mb. First, a content-length is known, so we don't get the advantage of pipelining the data directly from the socket, a temporary file must be created, written and read. Second, I believe the content is encoded so the transferred bitcount is much higher than using FTP. Plus, most browsers today do not support a progress bar for posts, so there is no indication of status, causing most people to click 'Upload' multiple times. I haven't done any optimization for this case, but have tested that is still works properly. Cleaning up (leaked file descriptor bug) I noticed that when uploading 20+ MB a couple of times, I ran out of hard drive space. This didn't make sense and I looked into what files were open by Zope. Doing an 'lsof' I found that the temporary files which are immediatly unlinked after creation, were still open until the end of the Zope process. These files (created by tempfile.TemporaryFile) needed to be closed after the end of the REQUEST and RESPONSE, rather than at the end of the Zope process. After publishing, the close method of the REQUEST gets called. Here I added closing of stdin and the FieldStorage created TemporaryFile '_file'. Output producers The ZServer.HTTPResponse object makes a good attempt of keeping large results out of memory but does so by creating a temporary file and copying any written data to it then pushing a file_part_producer onto the channel output queue. If the Zope object knows how to produce the data themselves, they could push producer(s) directly to the channel. I added a single check in ZServer.HTTPResponse(256) where a temporary file is only created if the data is larger than the in-memory buffer *and* doesn't already look like a producer with 'more' as a method. If the temporary file doesn't exist the rest of the code simply writes the data to the channel and the channel produces the output directly from the producer created by the Zope object. Using a file producer from my Zope object cuts out a file copy, and those get expensive when one is dealing with 20+ MB files. The response time is also dramatically reduced because the file copy step before streaming to the client was removed. I would like to apply the same concept to Image.File.index_html where rather than creating a temporary file in the RESPONSE to queue the contents, create a producer to pull the data directly out of the backend when it is ready to write. I am experiencing a 10 second latency (233Mhz laptop) between requesting a 10MB file and receiving the first byte with the current code. If an output producer is used, this latency would drop < 1 sec. I made an attempt to create a pdata_producer but failed because of ZODB errors reloading the object. I get a traceback like: 2000-10-24T09:19:08 ERROR(200) ZODB Couldn't load state for '\000\000\000\000\000\000&\370' Traceback (innermost last): File /usr/local/zope/lib/python/ZODB/Connection.py, line 442, in setstate AttributeError: 'None' object has no attribute 'load' My hunch is that the Image, pdata_producer or pdata object gets deactivated and can't find its DB to load itself. I tried setting a _p_jar on the pdata_producer, but I don't really know what happens when the object context leaves publish_module. Since the object activation happens in the ZServer thread, some voodoo may be needed to get the proper state in the pdata_producer.... any takers? Testing... I have only tested these changes with FTPServer and HTTPServer, not PCGIServer or FCGIServer. I have tested round-trip coherency because of the change in Image.File._read_data. I haven't completely tested boundary conditions, where Image.File._read_data makes descisions. The extent has been large files 10+ MB and small files < 64K. I haven't tested HTTPRequest.retry which will probably fail because HTTPRequest.stdin now may be a pipe. 3rd party products which treat BODYFILE as a seekable file object may fail during FTP uploads. Summary: Most of these efforts are geared towards FTP, as HTTP form uploads don't seem to be worth the effort. I haven't taken a look at HTTP PUT, for webdav clients etc... Similar pipelining could be used, however I doubt they would be possible without modifing cgi.FieldStorage. Zope seems to be doing a lot with TempStorage and other ZODB magic I didn't care about checking out. Some performance improvements could be included here. FTP I/O with my changes including my ExternalFile custom output producer dramatically increases Zopes performance and scalability. -Sean

3 5

WriteLocking RiskFactors updated
by Jeffrey P Shell 25 Oct '00

25 Oct '00

The WriteLocking RiskFactors artifact has had some new work done on it and I'm interested in some editorial comments and review before work begins on Architecture and UseCases. <http://dev.zope.org/Wikis/DevSite/Projects/WriteLocking/RiskFactors> Make any comments you have at: <http://dev.zope.org/Wikis/DevSite/Projects/WriteLocking/ProjectDiscussion> -- Jeffrey P Shell, jeffrey(a)Digicool.com http://www.zope.org/ | http://www.digicool.com/

2 1

extending objectValues() and objectIds() or ?
by Dennis Nichols 25 Oct '00

25 Oct '00

I have developed a flock of Python products that have attributes 'first_date', 'last_date', and 'ok_to_publish' with the intention that these attributes will be set by the content managers. Each product also has an 'isPublishable' method that returns true if 'ok_to_publish' is true _and_ today >= 'first_date' _and_ today <= 'last_date'. This all works fine but I find myself writing DTML in many places to retrieve a list of objects with objectValues() and then looping over them making yet another list of only the objects that are publishable. What I would really like is an objectValues() and an objectIds() that return only publishable objects/ids. It looks like this could be easily accomplished by {tremble} adding an additional parameter to those two routines, say chkPub=None so that I could do a call such as objectValues('Poll',1) when I wanted just the publishable Poll items returned. However... 1. modifying ObjectManager.py doesn't seem like the best move I've ever thought of 2. I'm confused about the ObjectManager internals - can I call my isPublishable methods on objects that ObjectManager keeps in its lists? How? I guess a better question, rather than asking about this one possible solution, would be: How can I extend Zope to give me a list of objects qualified by more than just the meta_type? Guidance gleefully gathered. Thanks! -- Dennis Nichols nichols(a)tradingconnections.com

2 2

RE: Security requires Acquisition?!
by Toby Dickenson 25 Oct '00

25 Oct '00

> Okay, this did the trick, but I'm not very happy with the result :-( > > I don't want the DisplayClass to be acquiring and I don't really see > (from a moral standpoint ;-) why I should need to mix in an Acquisiton > class to make security work :-S I suspect that was a rhetorical question, but ill answer anyway. Zope security is context based: Users can be defined in a subfolder and only have access under that folder, they can also be given local roles for a given folder. The role:permission mapping is set per-folder. Any security aware object needs to know its context. > That said, I think Shane said that Zope security is > predicated a lot on > Acquisition. Now, can I get the solution I'm looking for by mixing in > Aquisition.Explicit, still have the security stuff work and > not have the > DisplayClass acquiring attributes I don't want it do? Yes, you will need to set Acquisition.Acquired for the necessary attributes. Alternatively there may be other ways of avoiding an object using some specific acquired attributes: I often define an index_html=None to avoid a class using its container's default view. Wanting to make an object non-acquiring may be a danger-sign of some other problems. If the correctness of your program depends on the absence of certain attributes (acquired or otherwise) then you need to take extra care over PropertyManager-like features, which might allow a user to add the critical attribute. hth,

2 1

The Great Python (and Perl) Method Renaming Poll
by Michel Pelletier 25 Oct '00

25 Oct '00

We will be conducting a community poll to decide what to call Python (Perl, insert your language here) Methods. Many candidates have been discussed, but I'm afraid I don't have enough time to cull all the candidates from the discussions. So, before the poll, we are calling for nominiations from the community. Please send me an email containing one or more candidate names. These names will be added to the list. No pre-screening will be done, so please exercise some discretion if your favorite name is more tounge-in-cheek than practical (you never know what the masses will decide though!). Later this week, I will create a web poll where you can vote for your favorite. thanks, -Michel

2 1

Re: Unit testing, ZUnit
by Butch Landingin 25 Oct '00

25 Oct '00

--- In zope(a)egroups.com, Lalo Martins <lalo@h...> wrote: <snip> > We first conceived it as a kind of Zope-based version of > unittestgui.py - you create a "TestRunner" object giving it the > package, module and name of a TestSuite object generator (see > the PyUnit documentation) and inside it you may click on some > widget to run the tests and produce a "TestResult" object which > you can inspect later. Does this sound like a good design? > > Then I figured in the long run a Product isn't the best > sollution; instead, fiddling with App/Product* sounds more like > it, to allow developers to register tests just like they > register classes, ZClass superclasses, _misc and help. Of > course, just like the current registerHelp and others, it > wouldn't be mandatory and not using it wouldn't break anything. > > Then, in the Product's page in the Control Panel, there would > be a tab "Test", where you'd be allowed to run the unit tests. > > What do people think of this? What does DC think of this? Can I > go ahead and develop it in this direction? Kewl. ;^) please do! I've been test-infected ever since I started using JUnit about 8 months ago so I know the value of building a unittest framework for Zope. I even started playing around with PyUnit and integrating it with Zope... I got to the point where I had a ZPublisher-based framework that allowed me to have something like a function that had a relative url as its argument and it would return the rendered HTML... I then stored the string into a file and it would allow me easily build tests that compared the string output to the one stored in the file... I was planning to do more (e.g. get to the point where I have a function with a relative url as its argument which would return me the actual object -- this would then allow me to run its methods and check its properties, check assertions, etc., etc... but then, my "real" work interfered and I had to put my plans in the backburner for a while...(I might get back to this in a month or so, so I'm very interested in your project...keep us posted) I had envisioned it as something that ran on the commandline (not as a TTW thing) and the context in which I was using it for was that I would be developing a lot of python code and that running my test suite on the commandline would return the familiar "...." output that the commandline versions of Junit and PyUnit provide... it would also allow me to run the GUI version without changing any of my unittests... My intention was that most of the testing would be the testing of the application, so I didnt intend to test the ZServer part of Zope (I assumed that ZServer would be passing the right URL to the ZPublisher) -- instead, everything would be run locally and single-threaded...(of course, this framework is limited and can't do the testing of ZServer-related stuff -- e.g. virtual hosting stuff like ZServer -- but since I was more focused on the 'application', this wasn't such a big deal compared to the benefit of being able to run my tests locally and with the flexibility of running them either thru the commandline or thru a GUI... So I do have some questions about how you intend the ZUnit to be used... e.g. -- lets say I'm developing a Zope Application... I begin by writing the Python code interface, then I start writing the PyUnit test... in the the Setup section, I put in the code that hooks me into the Zope ZODB, installs the product, sets the properties, etc.; in the test section, I can get to the object, test my assertions, etc... and in the TearDown -- I back out my changes, etc... Then I add this test to my Test suite...etc., etc...Then to run this test, I...do...what? Also one coding caveat: when I started building my ZPublisher-based framework, I encountered a bug I couldn't easily solve: I could run my PyUnit tests fine using the commandline but when I tried to run them via the PyUnit GUI they failed : as far as I could tell, the GUI version of PyUnit did something esoteric with the "import" function (which would allow automatic reloading whenever you started it -- allowing you to leave the GUI up all the time and you could modify your python code and it would reload correctly (this is equivalent to JUnit's LoadingTestRunner) -- unfortunately, it reacted badly with Zope's 'esoteric' use of the "import" function as well ;^)... HTH. Butch Landingin butchland(a)yahoo.com http://squishdot.org PS. Sorry for the rambling, its been a while since I last posted to the list and I wanted to get a lot of things off my chest ;^) __________________________________________________ Do You Yahoo!? Yahoo! Messenger - Talk while you surf! It's FREE. http://im.yahoo.com/

2 1

Re: Using LoginManager with users stored in several Specialists
by Itai Tavor 25 Oct '00

25 Oct '00

Ty Sarna <tsarna(a)endicor.com> wrote: >Itai Tavor <itai(a)optusnet.com.au> wrote: >> - I'm not sure how to customize a UserSource to access the >> propertysheets on the Customer and Reseller classes. The easiest way >> seems to be to define userExists, userRoles and userAuthenticate >> methods in a GenericUserSource, but I don't think it's a good idea - >> I still wouldn't know how to make changes to the propertysheet from >> the LoginManager (for example, changePassword should be a method of >> acl_users, not of the user classes, because it does the same thing >> for all user types), and there is no caching. Should I write my own >> UserSource? Or can I do it with Data Plug-ins? I'd like to keep the >> solution simple, but I do want it to be efficient (fast and cached). > >Generic User Source is basically for people who are familar with GUF and >want some degree of backward compatability. There was also a time where >it was the only choice :-) > >Now there is "User Source", which is to LoginManager what Rack is to >Specialist. It's completely general, defaulting to storing things >persistently, but overrideable with SkinScript and the "load from >existence of attribute blah" to do anything you want. It's much more >Generic than Generic User Source, actually :) > >Have a look at the SkinScript reference for the "Object Remapping" >example (last one under WITH ... COMPUTE ...) of how to make a Rack >that retrieves proxyies for objects from a different Specialist. You >can direcly apply this same technique to have a User Source provide >users based on objects in other Specialists. I guess I still don't understand enough about object remapping... I got a CustomerUserSource with this SkinScript: WITH Customers.getCustomerByName(self.id) COMPUTE login_name, password, roles, original_object=RESULT NOTE: I use getCustomerByName rather than getItem because I use auto-generated codes for the customer objects id's, this lets me allow users to change their login name - but it means that finding a user is a more expensive operation than getItem, which is another reason I worry about speed and caching. Now I need to set the 'loaded by accessing attribute' bit, because the UserSource has no persistent items in it, right? What do I set it to? I tried to set it to 'id' but it does nothing - CustomerUserSource.getItem(some_user) always returns None. Also, if a Rack is using object remapping, can I still use getPersistentItemIDs() on it? -- Itai Tavor "Je sautille, donc je suis." C3Works itai(a)c3works.com - Kermit the Frog "If you haven't got your health, you haven't got anything"

1 0

Getting all objects matching a given meta_type and more
by morten＠esol.no 24 Oct '00

24 Oct '00

I've managed using the ZCatalog now. However, I have a small problem: When I instansiate a ZCatalog at the root of a tree, and search for all objects matching a given meta-type, all the objects in the entire tree are returned. Is there a way to instruct the ZCatalog instance that it should only return objects from within a certin part of the tree, without instansiating a new ZCatalog instance in the given subtree? I.e.: Zope Catalog (meta-type ZCatalog) Folder1 Folder2 Object1 (meta-type test) Folder3 Object2 (meta-type test) Folder4 Object3 (meta-type test) FolderA FolderB ObjectA (meta-type test) FolderC ObjectB (meta-type test) FolderD ObjectC (meta-type test) Now, is there a way I can tell the ZCatalog instance at the root folder to only return the objects mathing meta-type test in thee FolderA folder? Thanks in advance. -Morten

2 1