For all who are interested in breaking database size barriers without moving to Oracle or Berkeley Storage, here's something to try out. http://www.zope.org/Members/hathawsh/PartitionedFileStorage I *highly* recommend you back up Data.fs before installing. What is known to work: - Reads and writes of logical files greater than 3 GB on versions of Linux that support only 2 GB files. - Undo and packing. - Fast random reads and writes. - Using simple binary concatenation to move back to a standard single file. - Using PartitionedFile completely independent of Zope or ZODB. What has not been tested, but will very likely work: - Tranalyzer. - Putting the partitions on different physical devices. (Poor man's RAID :-) ) What couldn't be tested by my limited resources: - Platforms other than Linux. - Running this on a busy Zope site. (No, www.zope.org is not running PartitionedFileStorage. :-) ) BTW to all in the U.S., Happy Thanksgiving! Shane
Thank Shane! -- Andy McKay, Developer. ActiveState. ----- Original Message ----- From: "Shane Hathaway" <shane@digicool.com> To: <zope-dev@zope.org> Sent: Wednesday, November 22, 2000 1:48 PM Subject: [Zope-dev] ANNOUNCE: PartitionedFileStorage
For all who are interested in breaking database size barriers without moving to Oracle or Berkeley Storage, here's something to try out.
http://www.zope.org/Members/hathawsh/PartitionedFileStorage
I *highly* recommend you back up Data.fs before installing.
What is known to work: - Reads and writes of logical files greater than 3 GB on versions of Linux that support only 2 GB files. - Undo and packing. - Fast random reads and writes. - Using simple binary concatenation to move back to a standard single file. - Using PartitionedFile completely independent of Zope or ZODB.
What has not been tested, but will very likely work: - Tranalyzer. - Putting the partitions on different physical devices. (Poor man's RAID :-) )
What couldn't be tested by my limited resources: - Platforms other than Linux. - Running this on a busy Zope site. (No, www.zope.org is not running PartitionedFileStorage. :-) )
BTW to all in the U.S., Happy Thanksgiving!
Shane
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Shane Hathaway wrote:
What couldn't be tested by my limited resources: - Platforms other than Linux.
I'm running this on my NT4 Workstation, not high load but I'll let you know if anything weird happens (that isn't caused by me ;-) I wonder if this will help with the conflict errors I've been getting? cheers, Chris
Chris, Any hints/tips on how you did this (just to save me from having to do any work of course ;)) Phil ----- Original Message ----- From: "Chris Withers" <chrisw@nipltd.com> To: "Shane Hathaway" <shane@digicool.com> Cc: <zope-dev@zope.org> Sent: Thursday, November 23, 2000 10:31 AM Subject: Re: [Zope-dev] ANNOUNCE: PartitionedFileStorage
Shane Hathaway wrote:
What couldn't be tested by my limited resources: - Platforms other than Linux.
I'm running this on my NT4 Workstation, not high load but I'll let you know if anything weird happens (that isn't caused by me ;-)
I wonder if this will help with the conflict errors I've been getting?
cheers,
Chris
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Phil Harris wrote:
Chris,
Any hints/tips on how you did this (just to save me from having to do any work of course ;))
I just used WinZip to extract the tarball in the /ZODB directory. Sadly, it didn't work, and won't do until the next version fo Zope is released :-( cheers, Chris
Chris Withers wrote:
Phil Harris wrote:
Chris,
Any hints/tips on how you did this (just to save me from having to do any work of course ;))
I just used WinZip to extract the tarball in the /ZODB directory. Sadly, it didn't work, and won't do until the next version fo Zope is released :-(
I just realized I could use the "patch" command to backport. So I made a new tarball. Have fun! http://www.zope.org/Members/hathawsh/PartitionedFileStorage/PartitionedFileS... Shane
Shane Hathaway wrote:
I just realized I could use the "patch" command to backport. So I made a new tarball. Have fun!
http://www.zope.org/Members/hathawsh/PartitionedFileStorage/PartitionedFileS...
great :-) It's installed and running now. No problems yet, btu then I haven't seen any partitioning either... how long doI have to wait? ;-) cheers, Chris
Chris Withers wrote:
Shane Hathaway wrote:
I just realized I could use the "patch" command to backport. So I made a new tarball. Have fun!
http://www.zope.org/Members/hathawsh/PartitionedFileStorage/PartitionedFileS...
great :-)
It's installed and running now. No problems yet, btu then I haven't seen any partitioning either... how long doI have to wait? ;-)
Until 512 MB. However, you can change the default partition size in FileStorage.py. Note that it behaves correctly with partitions of varying size, so you don't have to worry that it'll get confused if you set it to a partition size of 10 MB when your first partition is already 50 MB. BTW, Evan helped me find and correct a bug. You'll only run into it if you set your partition size to less than the size of the largest ZODB objects. --- PartitionedFile.py~ Wed Nov 22 16:14:27 2000 +++ PartitionedFile.py Mon Dec 11 12:01:18 2000 @@ -343,7 +343,7 @@ file = self.__seekToNextPartition() if file is not None: state = self.__state - toAppend = file.read(size) + toAppend = file.read(size - len(data)) data = data + toAppend else: break Shane
Shane Hathaway wrote:
BTW, Evan helped me find and correct a bug. You'll only run into it if you set your partition size to less than the size of the largest ZODB objects.
That's fixed in 0.0.2, right? Well, anyway, I changed the line in FileStorage.py as follows: # Use 0.5 GB partitions because that's what fits on a CD. defaultOps = PartitionedFileOperations( DefaultPartitionNamingStrategy(partlen=40000)) # DefaultPartitionNamingStrategy(partlen=2L**29)) ...so I could try and provoke a split, but, now, when I try to add objects, I get: Error Type: CorruptedDataError Error Value: 41904222L Traceback (innermost last): File E:\Zope\22C38D~1.5B1\lib\python\ZPublisher\Publish.py, line 222, in publish_module File E:\Zope\22C38D~1.5B1\lib\python\ZPublisher\Publish.py, line 187, in publish File E:\Zope\22C38D~1.5B1\lib\python\Zope\__init__.py, line 221, in zpublisher_exception_hook (Object: ApplicationDefaultPermissions) File E:\Zope\22C38D~1.5B1\lib\python\ZPublisher\Publish.py, line 171, in publish File E:\Zope\22C38D~1.5B1\lib\python\ZPublisher\mapply.py, line 160, in mapply (Object: manage_main) File E:\Zope\22C38D~1.5B1\lib\python\ZPublisher\Publish.py, line 112, in call_object (Object: manage_main) File E:\Zope\22C38D~1.5B1\lib\python\App\special_dtml.py, line 120, in __call__ (Object: manage_main) (Info: E:\Zope\2.2.5b1\lib\python\OFS\main.dtml) File E:\Zope\22C38D~1.5B1\lib\python\DocumentTemplate\DT_String.py, line 528, in __call__ (Object: manage_main) File E:\Zope\22C38D~1.5B1\lib\python\App\special_dtml.py, line 120, in __call__ (Object: manage_tabs) (Info: E:\Zope\2.2.5b1\lib\python\App\manage_tabs.dtml) File E:\Zope\22C38D~1.5B1\lib\python\DocumentTemplate\DT_String.py, line 528, in __call__ (Object: manage_tabs) File E:\Zope\22C38D~1.5B1\lib\python\DocumentTemplate\DT_With.py, line 146, in render (Object: _(manage_options=filtered_manage_options())) File E:\Zope\22C38D~1.5B1\lib\python\App\PersistentExtra.py, line 114, in locked_in_version (Object: ApplicationDefaultPermissions) File E:\Zope\22C38D~1.5B1\lib\python\ZODB\Connection.py, line 432, in modifiedInVersion File E:\Zope\22C38D~1.5B1\lib\python\ZODB\DB.py, line 364, in modifiedInVersion File E:\Zope\2.2.5b1\lib\python\ZODB\FileStorage.py, line 679, in modifiedInVersion (Object: E:\Zope\2.2.5b1/var/Data.fs) CorruptedDataError: (see above) Any ideas? cheers, Chris
Chris Withers wrote:
Shane Hathaway wrote:
BTW, Evan helped me find and correct a bug. You'll only run into it if you set your partition size to less than the size of the largest ZODB objects.
That's fixed in 0.0.2, right?
Should be, although I just realized I don't know whether 0.0.2 was intended for Zope 2.2 or 2.3.
Well, anyway, I changed the line in FileStorage.py as follows: # Use 0.5 GB partitions because that's what fits on a CD. defaultOps = PartitionedFileOperations( DefaultPartitionNamingStrategy(partlen=40000)) # DefaultPartitionNamingStrategy(partlen=2L**29))
...so I could try and provoke a split, but, now, when I try to add objects, I get:
Error Type: CorruptedDataError Error Value: 41904222L
Ugh. I have no idea what might have happened. That's why I'm wary: if something goes wrong, the symptoms will show up a long time after the problems occur. Now BerkeleyStorage is on its way and once that's ready, PartitionedFileStorage will have such a small audience that it won't be worth the trouble. Oh well, it was a fun experiment. Shane
Shane Hathaway <shane@digicool.com> wrote:
Now BerkeleyStorage is on its way and once that's ready, PartitionedFileStorage will have such a small audience that it won't be worth the trouble. Oh well, it was a fun experiment.
Can you pleae expand on this.. what about BerkeleyStorage is on its way and when ? Been using it for soem tests and like it very much, sincerely hoping it might get rolled into Zope and Python before long. thanks - Jason ___________________________________________________________ Jason CUNLIFFE = NOMADICS['Interactive Art and Technology']
Jason Cunliffe wrote:
Shane Hathaway <shane@digicool.com> wrote:
Now BerkeleyStorage is on its way and once that's ready, PartitionedFileStorage will have such a small audience that it won't be worth the trouble. Oh well, it was a fun experiment.
Can you pleae expand on this.. what about BerkeleyStorage is on its way and when ?
Here is the info on the BerkeleyStorage work. Jim sent me the URL. http://lists.zope.org/pipermail/zope-zeo/2000-November/000196.html It will soon be a fishbowl project. Shane
Shane Hathaway wrote:
Should be, although I just realized I don't know whether 0.0.2 was intended for Zope 2.2 or 2.3.
There is no difference between the PartitionedFile.py files in the 2.2 and 2.3 versions, so I guess 0.0.2 will do as well for either. -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net
Steve Alexander wrote:
Shane Hathaway wrote:
Should be, although I just realized I don't know whether 0.0.2 was intended for Zope 2.2 or 2.3.
There is no difference between the PartitionedFile.py files in the 2.2 and 2.3 versions, so I guess 0.0.2 will do as well for either.
Silly me, of course you're right. :-) BTW I'm happy to hear someone's putting it to good use. Shane
Shane Hathaway wrote:
BTW, Evan helped me find and correct a bug. You'll only run into it if you set your partition size to less than the size of the largest ZODB objects.
The new version of FileStorage.py in PartitionedFileStorage for 2.3 overwrites some of the changes made the FileStorage locking protocol in CVS. These are the ones using _commit_lock_acquire _commit_lock_release. -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net
Steve Alexander wrote:
Shane Hathaway wrote:
BTW, Evan helped me find and correct a bug. You'll only run into it if you set your partition size to less than the size of the largest ZODB objects.
The new version of FileStorage.py in PartitionedFileStorage for 2.3 overwrites some of the changes made the FileStorage locking protocol in CVS.
These are the ones using _commit_lock_acquire _commit_lock_release.
Well, it's pretty easy to use diff and patch to correct it. But like I told ChrisW, I think BerkeleyStorage will fill the need that PartitionedFileStorage was only partitially addressing. OTOH I've been running it for a while now on my box and have had no signs of problems. Shane
Shane Hathaway wrote:
But like I told ChrisW, I think BerkeleyStorage will fill the need that PartitionedFileStorage was only partitially addressing.
Is BerkleyStorage going to be as easy to set up and use? I seem to remember Jim F making some comments to the contrary a while back... What about on WinNT? ;-) cheers, Chris
On Tue, 02 Jan 2001 15:33:03 -0500, Shane Hathaway <shane@digicool.com> wrote:
But like I told ChrisW, I think BerkeleyStorage will fill the need that PartitionedFileStorage was only partitially addressing.
FileStorage has fewer license compatability issues. Im not yet sure about the extent of any problems this may cause. Toby Dickenson tdickenson@geminidataloggers.com
Chris Withers wrote:
I'm running this on my NT4 Workstation, not high load but I'll let you know if anything weird happens (that isn't caused by me ;-)
Okay, weirdness... when I try to copy/delete/etc stuff, I get: Error Type: AttributeError Error Value: _tstatus Traceback (innermost last): File E:\Zope\2.2.2\lib\python\ZPublisher\Publish.py, line 222, in publish_module File E:\Zope\2.2.2\lib\python\ZPublisher\Publish.py, line 187, in publish File E:\Zope\2.2.2\lib\python\Zope\__init__.py, line 221, in zpublisher_exception_hook (Object: Traversable) File E:\Zope\2.2.2\lib\python\ZPublisher\Publish.py, line 171, in publish File E:\Zope\2.2.2\lib\python\ZPublisher\mapply.py, line 160, in mapply (Object: manage_pasteObjects) File E:\Zope\2.2.2\lib\python\ZPublisher\Publish.py, line 112, in call_object (Object: manage_pasteObjects) File E:\Zope\2.2.2\lib\python\OFS\CopySupport.py, line 216, in manage_pasteObjects (Object: Traversable) File E:\Zope\2.2.2\lib\python\OFS\CopySupport.py, line 431, in _getCopy (Object: Traversable) File E:\Zope\2.2.2\lib\python\ZODB\ExportImport.py, line 219, in importFile File E:\Zope\2.2.2\lib\python\ZODB\BaseStorage.py, line 207, in tpc_finish (Object: E:\Zope\2.2.2/var/Data.fs) File E:\Zope\2.2.2\lib\python\ZODB\FileStorage.py, line 821, in _finish (Object: E:\Zope\2.2.2/var/Data.fs) AttributeError: (see above) ...or some other wierdness involving _tstatus. Extra bonus wierdness: despite the above error, deleting a property actually works while the paste part of copying and pasting a folder didn't work :-S Any ideas? Chris
Chris Withers wrote:
Error Type: AttributeError Error Value: _tstatus
Problem found... Shane, you did your stuff on version 1.48 of FileStorage.py, while Zope 2.2.x uses version 1.37.12.8 of SileStorage.py Sadly, it appears the two are incompatible :-( Is it just me, or should plugging in a new type of storage not feel quite as hacky as this? Looks cool anyway, cheers, Chris
On Thu, 23 Nov 2000, Chris Withers wrote:
Chris Withers wrote:
Error Type: AttributeError Error Value: _tstatus
Problem found...
Shane, you did your stuff on version 1.48 of FileStorage.py, while Zope 2.2.x uses version 1.37.12.8 of SileStorage.py
Sadly, it appears the two are incompatible :-(
It would appear so. I didn't realize until now that there are significant new features for 2.3.x FileStorage. So a new qualification: you have to be using Zope CVS in order to try out this version.
Is it just me, or should plugging in a new type of storage not feel quite as hacky as this?
Some things are just hacky by nature. :-) BTW this *won't* reduce any conflict errors. Shane
participants (7)
-
Andy McKay -
Chris Withers -
Jason Cunliffe -
Phil Harris -
Shane Hathaway -
Steve Alexander -
Toby Dickenson