Re: [Zope-dev] objectValues performance
Ok, I'll bite. Why doesn't the standard folder scale? Seems like a design flaw to me - why doesn't the default folder use catalogs or BTrees? -Brett
"Casey" == Casey Duncan <cduncan@kaivo.com> writes:
Casey> Brett Carter wrote: >>> I have a folder with greater than 5000 ZClass instances in it. It >>> takes > 5mins to do an objectValues for every object in the folder - >>> is there a higher perfomance call I could make? >>> -Brett Casey> Standard folder performance degrades pretty quickly once you get Casey> a lot of objects in it. There are two solutions to this: Casey> Subdivide your objects into multiple folders. Casey> Use a BTreeFolder which should be much faster. Casey> You can download the BTreeFolder product Casey> here: http://www.zope.org/Members/hathawsh/BTreeFolder/ Casey> hth, Casey> Casey Duncan
Brett Carter wrote:
Ok, I'll bite. Why doesn't the standard folder scale? Seems like a design flaw to me - why doesn't the default folder use catalogs or BTrees? -Brett
Because massive scale is not a requirment of folders, they are meant to organize content for humans, not to be large-collection containers. A folder with 5000 elements is not very useful to a human. On a similar note, create 5000 files in a linux directory on a ext2 (standard) filesystem and then type 'ls'. You'll notice they don't scale very well either, which is why there are filesystems like ReiserFS. -Michel
"Casey" == Casey Duncan <cduncan@kaivo.com> writes:
Casey> Brett Carter wrote: >>> I have a folder with greater than 5000 ZClass instances in it. It >>> takes > 5mins to do an objectValues for every object in the folder - >>> is there a higher perfomance call I could make? >>> -Brett
Casey> Standard folder performance degrades pretty quickly once you get Casey> a lot of objects in it. There are two solutions to this:
Casey> Subdivide your objects into multiple folders. Casey> Use a BTreeFolder which should be much faster.
Casey> You can download the BTreeFolder product Casey> here: http://www.zope.org/Members/hathawsh/BTreeFolder/
Casey> hth, Casey> Casey Duncan
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Brett Carter wrote:
Ok, I'll bite. Why doesn't the standard folder scale? Seems like a design flaw to me - why doesn't the default folder use catalogs or BTrees? -Brett
AFAIK a standard folder uses a linear search when you request an object from it (ala Python dictionaries, someone please correct me if I'm wrong). This works great except that the search time grows linearly (by n) as you add objects. The BTreeFolder as the name implies creates a binary tree of the objects where the search time grows by only log n. For small folders the search time difference is minimal to non-existant, but as n increases the BTreeFolder search time increases minimally. B-trees are fairly complex entities to manage and for the vast majority of folders are total overkill. That is why standard folders work the way they do, the implementation is simple and efficient for 99.9% of applications. Your case is fairly atypical of most Zope folders. Perhaps a future implementation of Zope folders could automatically use a b-tree after a certain threshold is reached, for now you must explicitly select them. Andy's idea of using objectIds instead of objectValues is also a good one which will save significant amounts of memory. You can always access each object individually via id if you need to. Using a ZCatalog could also help in this because you can query the objects without loading them into memory and the returned result does not load the objects themselves, only the meta-data and only once a result item is explicitly accessed (By using so-called lazy sequences). However the catalog will not speed up your actual object access time unless you divide them up amongst several folders or use a BTreeFolder. The latter being a simpler solution from a design standpoint. Good luck! Casey Duncan
Casey Duncan wrote:
Brett Carter wrote:
Ok, I'll bite. Why doesn't the standard folder scale? Seems like a design flaw to me - why doesn't the default folder use catalogs or BTrees? -Brett
i believe folders store objects as attrs which implies using a hash in __dict__ for lookup. some quick conclusions. the btree folder is slower on average for creation by anywhere from 15-25% andt btree's __getattr is on average almost twice as slow as the folder's getattr. btrees folders are great, but they currently have a performance penalty at the benefit of larger folders that behave better during concurrent changes, mainly because they implement a __getattr__ hook in python instead of c. i did some tests a few months ago(posted to zope-dev) and i believe the performance penalty varied depending on the operation, approx 15-25% on ob. creation and 40-50% slower on access. unless you know you'll have a large number of objects or lots of concurrent changes you'd be better off with a standard folder. as for what constitues a large number, i'm not sure, it depends on object size and number of objects, during my tests i didn't see any changes with scaling the numbers up to 5000, but granted my objects were lightweight and poor examples on anything resembling realworld. regarding objectids in this context its the difference between dict.keys() and dict.items(). my catalog suggestion was under the assumption that in getting a filtered set from the catalog would allow you to do dict[id] on individual items for the set, using the metadata to perform any runtime application filtering computations. cheers kapil
AFAIK a standard folder uses a linear search when you request an object from it (ala Python dictionaries, someone please correct me if I'm wrong). This works great except that the search time grows linearly (by n) as you add objects. The BTreeFolder as the name implies creates a binary tree of the objects where the search time grows by only log n. For small folders the search time difference is minimal to non-existant, but as n increases the BTreeFolder search time increases minimally. B-trees are fairly complex entities to manage and for the vast majority of folders are total overkill. That is why standard folders work the way they do, the implementation is simple and efficient for 99.9% of applications. Your case is fairly atypical of most Zope folders.
Perhaps a future implementation of Zope folders could automatically use a b-tree after a certain threshold is reached, for now you must explicitly select them.
Andy's idea of using objectIds instead of objectValues is also a good one which will save significant amounts of memory. You can always access each object individually via id if you need to. Using a ZCatalog could also help in this because you can query the objects without loading them into memory and the returned result does not load the objects themselves, only the meta-data and only once a result item is explicitly accessed (By using so-called lazy sequences). However the catalog will not speed up your actual object access time unless you divide them up amongst several folders or use a BTreeFolder. The latter being a simpler solution from a design standpoint.
Good luck!
Casey Duncan
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
participants (4)
-
Brett Carter -
Casey Duncan -
Ender -
Michel Pelletier