- Zope-Dev - Zope lists

Re: [Zope] CatalogAware and multiple meta_types
by ed colmar 01 May '00

01 May '00

I have started down the path of trying to convert my squishdot based product to use the Catalog instead of the "old" Squishdot indexing system. I have set up the _searchable_result_columns, and _searchable_arguments methods for 4 of the variables I am interested in. I have placed the self.reindex_object() call in the manage_edit method. I commented out the old search interface. Now, all seememd good until I tried to add variables that belonged to a different meta_type. Which results in an AttributeError value: 0 Within my base class I have four different meta_types that can be added. People Promoters Events Hosts Each of these objects contain different variables. People objects contain "name", and promoter objects contain "promotername". How do I set it up so that all of these values can be cataloged and searched? Do I have to standardize naming conventions so that all objects share common variable names? Or, can I define searchable paramaters within these objects? Thanks for any ideas or suggestions! -ed-

1 0

ANN: JPicture 0.0.2 released
by Jens Vagelpohl 01 May '00

01 May '00

Hi everybody, Having read a lot of peoples' questions about how to create types of objects that act as a container and into which you can upload stuff like pictures, and because I had a personal need for this type of object, I sat down and created "JPicture". I know, very imaginative name... JPicture is an "old-style" Zope product (meaning it is not ZClass-based and all components are on the file system) that is based on Zope's own Image class with the following additions: * a field of type "text" called "description" that allows you to store more than just a title with the picture * a second image can be uploaded into a JPicture as a "thumbnail" attribute. This can be done either when a JPicture is first created or at any point afterwards. Otherwise JPicture quacks and waddles just like a normal Image in Zope. Apart from my own need I hope it can serve as a model for those who have had problems in the past tackling the "uploading a file into an object"-problem. And if you're a big ZClass fan it is very easy to make JPicture into a Base class that can be inherited by a new ZClass by either replacing the code in __init__.py that starts with "context.registerClass..." with a line that reads "context.registerBaseClass(JPicture.JPicture, 'JPicture') or simply adding this after the "context.registerClass..." bit. Find the whole shebang including legalese and other documentation at: http://www.zope.org/Members/jens/JPicture ! jens ---- Jens Vagelpohl jens(a)digicool.com Software Engineer www.digicool.com Digital Creations (888) 344-4332 Got Zope? ----

1 0

Adding Pike ability to Zope
by David Brown 01 May '00

01 May '00

I have just submitted a product to be reviewed on the main Zope site. It's called ZFDocument, and it subclasses DTML Document to allow it to be editted by Pike (as well as in a browser -- this was always possible, but I made it easier to expose). Now all I really had to do was add four methods to my subclass, as well as one instance var. That, and the PikeCompatibility framework I built for ZopeFish are all we need to be able to talk to Pike. I propose that I modify (or submit patches for) DTML Method. By adding Pike compatibility there, we get it for almost everything. XML Document will take a patch, but that's easy enough to do, if I make the additions a mixin. Who do I talk to/What do I have to do to make this a reality? dave

2 1

Want to help on a Zope Project?
by David Brown 30 Apr '00

30 Apr '00

I'm working on a Content Management product called ZopeFish, and I have run into some areas where I'd be grateful for some expertise. I've documented where I'm having problems here: http://zopefish.weblogs.com/2000/04/30 You can download the code there as well. Thanks in advance! dave

1 0

Re: [Zope-dev] Data.fs corruption when creating lots of objects.
by R. David Murray 29 Apr '00

29 Apr '00

Thanks for the explanation! On Fri, 28 Apr 2000, Phillip J. Eby wrote: > NOT designed for high-volume updates like this. Your actual best bet would > be to load all 60,000 objects first, then ZCatalog them all in one single > long transaction, with subtransactions to prevent memory overflow. > ZCatalog+FileStorage has terrific speed and great space performance in that > configuration. But if you have an application that requires frequent > indexing, you may be better off with an RDBMS for certain aspects. I came to this conclusion myself about two hours ago and am in the process of doing the load that way. The object load without the Catalog was relatively fast and not disk intensive. I do find it interesting to note that raw data occupying 8MB on disk takes up 28MB on disk loaded into ZClass instance objects, *without* cataloging. This is not a good storage ratio, so I'm not sure what you mean by "great space performance"; but then again, ZODB's purpose is definately not storage optimization! I'll be curious to see how much overhead the Catalog adds; I'm guessing the space performance you are refering to is in that overhead. Do you have an example of doing subtransactions? I tried just calling get_transaction().commit(1), but it didn't seem to write anything to disk and the memory usage just kept growing. There remains the fact that there is a data corruption bug in ZODB somewhere, which seems to only be triggered by the massive update activity associated with something like the load-with-ZCatalog I tried to do. I just hope it doesn't bite me again when I do the Cataloging run. I have not been successful at narrowing down an easily reproduced example of the error. When it strikes seems to depend on the update load, so debugging it is going to be an annoyingly long process, I think. --RDM

1 0

Re: [Zope-dev] Data.fs corruption when creating lots of objects.
by Phillip J. Eby 29 Apr '00

29 Apr '00

What's happening is pretty normal for the type of operation you describe. Your 52 objects are BTree pages/buckets and IntSet objects being rewritten for each transaction. The expected number of objects rewritten by a transaction is based on the depth of each BTree. Each ZCatalog index has a forward and backward BTree that gets updated. The ZCatalog itself has from one to three additional BTrees. The index forward BTrees end in IntSet objects. And of course there are the objects you're adding. The objects get larger each time because the pages and IntSets are filling up with entries, and of course they are written out again at the end of every transaction. To ease the storage burden, you should probably load your objects in larger tranasactions. ZODB only writes changed objects at the end of a transaction, so if you do, say, 100 objects per transaction, you will have nearly 100 times fewer objects written and disk space used. ZCatalog is NOT designed for high-volume updates like this. Your actual best bet would be to load all 60,000 objects first, then ZCatalog them all in one single long transaction, with subtransactions to prevent memory overflow. ZCatalog+FileStorage has terrific speed and great space performance in that configuration. But if you have an application that requires frequent indexing, you may be better off with an RDBMS for certain aspects. At 07:08 PM 4/28/00 -0400, R. David Murray wrote: > >What's going on here? The amount of data and objects being thrown >around here seems *totally* crazy. I'm especially puzzled by those >larger and larger object sizes during a load, given that the size >of the input records is consistent. The objects that remain after >packing are of a very reasonable size (around 350-400 bytes each). >And why does the addition of a single new ZClass instance result >in *52* objects in the transaction, when I would assume that only >the Catalog and the containing folder are getting modified in the >transaction? > >(For review, I'm adding a simple catalog aware ZClass object that >has about a dozen properties, but I'm trying to add a little less >than 60,000 of them). > >Can anyone with some knowledge of how the ZODB in general and FileStorage >in particular works give me a clue what is going on here? Is this >really normal behavior, to need this huge amount of disk space to >handle what seem to be simple updates?

1 0

ZClass arguments dilemma
by Kevin Howe 29 Apr '00

29 Apr '00

Hi All, I have a ZClass and would like to add a method to it which would allow overriding arguments to be passed to it via dtml: Example: <dtml-with myZClassInstance> <dtml-var "myMethod( link='http://domain' )"> </dtml-with> The values for the given variables (link) would be used instead of the object's Property values. The problem I have run into is that when an object method is called via DTML as above, the entire REQUEST of the object is replaced, denying accessing to the object's Property values and Methods. Example: myMethod() would no longer have access to <dtml-var id> because "id" wasn't supplied in the DTML call. All ZClass Methods or Properties are no longer available. In a Python Product you can still access the values because "self" is passed: Example: def myMethod (self, title): # use "command line" argument if given if title: pass # otherwise use the Property value else: title = self.title But with the ZClass, I can find no clear way to pass the self values for that object. Example: <dtml-with myZClassInstance> <dtml-var "myMethod( self, link='http://domain' )"> </dtml-with> What is the correct way to go about this? Thanks :) kh

3 2

Data.fs corruption when creating lots of objects.
by R. David Murray 28 Apr '00

28 Apr '00

More info on the database corruption. I took out my reindex_object call (to reduce the ZCatalog activity), and did a get_transaction().commit() after every object addition. The load failed when it filled the disk at 6GB of data. So I went back and started loading it in 2500 record chunks and packing after each one. The first chunk resulted in a 177 MB Data.fs file. A pack on this took it down to 8MB. Loading the second chunk ballooned it to 295MB. Packing that resulted in the error I saw before when doing the load by sections: Error Type: CorruptedTransactionError Error Value: /usr/local/zope/directory/var/Data.fs data record exceeds transaction record at 185738594 Tranalyzer says: Traceback (innermost last): File "tranalyzer.py", line 333, in ? main(sys.argv) File "tranalyzer.py", line 325, in main t = Transaction(f, off, refs) File "tranalyzer.py", line 170, in __init__ self.obs.append(Record(f, refs)) File "tranalyzer.py", line 211, in __init__ self.vstring = f.c(vl - 16) File "tranalyzer.py", line 105, in c self.checkleft(n) File "tranalyzer.py", line 87, in checkleft raise 'Not enough data left, %d < %d' % (self.left, n) Not enough data left, 6816 < 23567 so it finds the same problem. Looking at the Tranalyzer output, it looks like I have three objects per record added after a pack, but about 52 objects in each transaction before packing. As the load progresses, some of these objects get rather large (>16K). What's going on here? The amount of data and objects being thrown around here seems *totally* crazy. I'm especially puzzled by those larger and larger object sizes during a load, given that the size of the input records is consistent. The objects that remain after packing are of a very reasonable size (around 350-400 bytes each). And why does the addition of a single new ZClass instance result in *52* objects in the transaction, when I would assume that only the Catalog and the containing folder are getting modified in the transaction? (For review, I'm adding a simple catalog aware ZClass object that has about a dozen properties, but I'm trying to add a little less than 60,000 of them). Can anyone with some knowledge of how the ZODB in general and FileStorage in particular works give me a clue what is going on here? Is this really normal behavior, to need this huge amount of disk space to handle what seem to be simple updates? I'm trying to narrow down the corruption to a quickly repeatable operation so I can start to insturment what's going on, but I'm really puzzled by what the Zope transaction/object machinery is doing here. --RDM

1 0

RE: [Zope-dev] Problem with reading zope objects from external me thod.
by Chris McDonough 28 Apr '00

28 Apr '00

The Image object's data attribute is what you want. AFAIK, you need to use consruct a StringIO object with the data in from the data attribute (it makes a chunk of data look like a file object). PIL doesnt seem to support any other method of reading a chunk of data but from an object that looks like a file. > -----Original Message----- > From: Anders Lide'n [mailto:al@netg.se] > Sent: Friday, April 28, 2000 2:54 AM > To: zope-dev(a)zope.org > Subject: [Zope-dev] Problem with reading zope objects from external > method. > > > > I want to draw a circle on a picture that I have stored > inside zope. Since > the image manipulation module (PIL) is a module not usable from inside > zope I have to use an external method. > > The external method now needs to read in the file (the > picture) and this > file is located inside zope as a zope object (image file). > Hence : I need > to get the external method to read a zope object. But I > continually fail > to read the zope object, how do I do? > > Anders Lidén NetGuide / TerraTel AB > al(a)netg.se Tankegången 4 > Url:http://www.netg.se/ 417 56 GÖTEBORG > Mobile : 0736 - 73 55 32 +46 - (0)31 - 50 79 69 > > > _______________________________________________ > Zope-Dev maillist - Zope-Dev(a)zope.org > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) >

1 0

Transparent folders?
by Martijn Faassen 28 Apr '00

28 Apr '00

Hi there, Recently I've had an idea. Problem description: I'm sure lots of people will recognize this. The root folder fills up with lots and lots and lots of methods; both factored HTML layout components and methods that should apply to all subfolders. Ideally you'd want to put this stuff in subfolders. But you run into the following problem: acquisition works no more. That is, acquisition works, but to override any specific layout component that's in a subfolder, you need to override the entire subfolder and copy over *all* layout elements. That's not what we want. I haven't seen an easy solution to this problem yet, unless I'm missing something. Analysis: So, why do you want to use folders at all? To compartementalize the system so we can easily find stuff. To avoid long lists of methods in the root folder. But proper folders hurt acquisition, so.. Solution: We need a new type of folder, the 'transparent' or 'virtual' folder. If we have a folder 'folder', and a subfolder 'sub', and sub is a transparent folder, anything we place in 'sub' is actually in 'folder' as well. It'll be acquired. It can be overridden (either in a subfolder of the same name, or in the folder that's doing that acquiring directly). This way, we get to compartementalize things while keeping the benefits of acquisition. Actually, this idea could have deep consequences, as it's getting very close to (multiple) inheritance. That is, if you put a transparent folder in your folder, it's like saying the current folder inherits from that transparent folder. Perhaps transparent folders could then also be generalized for a new type of ZClass inheritance that may or may not be easier to manage than the current system. Imagine being able to drop a transparent folder in your ZClass to make it inherit the methods in that transparent folder. ZClass inheritance needs an extra bit, though; the ability to make a 'reference' transparent folder that does not have actual methods inside itself, but just refers to some other ZClass (or Python base class, or whatever). Let's just stick to my original transparent folder idea for now, keeping all this inheritance zen in mind for later. My questions are: * is this a good idea at all? Any disastrous complications or consequences? * is this technically feasible? Regards, Martijn

9 13