Hmmm ive looked at fsrecover.py and its not happy either. Anyone else solved this problem? Im on Win2k BTW. -- Andy McKay, Developer. ActiveState. ----- Original Message ----- From: "Andy McKay" <andym@ActiveState.com> To: <zope-dev@zope.org> Sent: Friday, November 03, 2000 1:04 PM Subject: 2gb file size
My database has suddenly ballooned to 2 gig's stopping my Zope from running. Are there any utils to delete the last transaction or so?
This is kinda urgent... :(
Thanks.
-- Andy McKay, Developer. ActiveState.
Ive tried truncating the file and then running fsrecover.py, but get: File "ZODB\FileStorage.py", line 1530, in read_index if read(4) != packed_version: raise FileStorageFormatError, name FileStorage.FileStorageFormatError: q:\var\Data.fs As a desperation I commented out line 1530 and reran. It runs through the entire file and reduces the data.fs to 1k. Am I totally hooped? I thought ZODB could handle being truncated, ive never had an problems reading from a data.fs even when my box has blown up. Anyway looks like Im talking to myself here... -- Andy McKay, Developer. ActiveState. ----- Original Message ----- From: "Andy McKay" <andym@ActiveState.com> To: <zope-dev@zope.org> Sent: Friday, November 03, 2000 1:28 PM Subject: [Zope-dev] Fw: 2gb file size
Hmmm ive looked at fsrecover.py and its not happy either. Anyone else solved this problem? Im on Win2k BTW.
-- Andy McKay, Developer. ActiveState. ----- Original Message ----- From: "Andy McKay" <andym@ActiveState.com> To: <zope-dev@zope.org> Sent: Friday, November 03, 2000 1:04 PM Subject: 2gb file size
My database has suddenly ballooned to 2 gig's stopping my Zope from running. Are there any utils to delete the last transaction or so?
This is kinda urgent... :(
Thanks.
-- Andy McKay, Developer. ActiveState.
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
I didn't think this was a problem on Windows? I wonder if Python needs to be compiled in a special way to support large files on Windows. Have you tried Tranalyzer to see if will tell you where the transaction breaks are in the file? There's a howto by Itamar on Zope.org about how to do this. ----- Original Message ----- From: "Andy McKay" <andym@activestate.com> To: "Andy McKay" <andym@activestate.com>; <zope-dev@zope.org> Sent: Friday, November 03, 2000 6:39 PM Subject: Re: [Zope-dev] Fw: 2gb file size
Ive tried truncating the file and then running fsrecover.py, but get:
File "ZODB\FileStorage.py", line 1530, in read_index if read(4) != packed_version: raise FileStorageFormatError, name FileStorage.FileStorageFormatError: q:\var\Data.fs
As a desperation I commented out line 1530 and reran. It runs through the entire file and reduces the data.fs to 1k. Am I totally hooped? I thought ZODB could handle being truncated, ive never had an problems reading from a data.fs even when my box has blown up. Anyway looks like Im talking to myself here...
-- Andy McKay, Developer. ActiveState. ----- Original Message ----- From: "Andy McKay" <andym@ActiveState.com> To: <zope-dev@zope.org> Sent: Friday, November 03, 2000 1:28 PM Subject: [Zope-dev] Fw: 2gb file size
Hmmm ive looked at fsrecover.py and its not happy either. Anyone else solved this problem? Im on Win2k BTW.
-- Andy McKay, Developer. ActiveState. ----- Original Message ----- From: "Andy McKay" <andym@ActiveState.com> To: <zope-dev@zope.org> Sent: Friday, November 03, 2000 1:04 PM Subject: 2gb file size
My database has suddenly ballooned to 2 gig's stopping my Zope from running. Are there any utils to delete the last transaction or so?
This is kinda urgent... :(
Thanks.
-- Andy McKay, Developer. ActiveState.
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Andy, Are you running Zope 2.2.1 or later? There were pointer arithmetic errors that occurred on earlier versions, even with operating systems that support >2 GB files. Shane Chris McDonough wrote:
I didn't think this was a problem on Windows? I wonder if Python needs to be compiled in a special way to support large files on Windows.
Have you tried Tranalyzer to see if will tell you where the transaction breaks are in the file? There's a howto by Itamar on Zope.org about how to do this.
----- Original Message ----- From: "Andy McKay" <andym@activestate.com> To: "Andy McKay" <andym@activestate.com>; <zope-dev@zope.org> Sent: Friday, November 03, 2000 6:39 PM Subject: Re: [Zope-dev] Fw: 2gb file size
Ive tried truncating the file and then running fsrecover.py, but get:
File "ZODB\FileStorage.py", line 1530, in read_index if read(4) != packed_version: raise FileStorageFormatError, name FileStorage.FileStorageFormatError: q:\var\Data.fs
As a desperation I commented out line 1530 and reran. It runs through the entire file and reduces the data.fs to 1k. Am I totally hooped? I thought ZODB could handle being truncated, ive never had an problems reading from a data.fs even when my box has blown up. Anyway looks like Im talking to myself here...
-- Andy McKay, Developer. ActiveState. ----- Original Message ----- From: "Andy McKay" <andym@ActiveState.com> To: <zope-dev@zope.org> Sent: Friday, November 03, 2000 1:28 PM Subject: [Zope-dev] Fw: 2gb file size
Hmmm ive looked at fsrecover.py and its not happy either. Anyone else solved this problem? Im on Win2k BTW.
-- Andy McKay, Developer. ActiveState. ----- Original Message ----- From: "Andy McKay" <andym@ActiveState.com> To: <zope-dev@zope.org> Sent: Friday, November 03, 2000 1:04 PM Subject: 2gb file size
My database has suddenly ballooned to 2 gig's stopping my Zope from running. Are there any utils to delete the last transaction or so?
This is kinda urgent... :(
Thanks.
-- Andy McKay, Developer. ActiveState.
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Shane Hathaway wrote:
Are you running Zope 2.2.1 or later? There were pointer arithmetic errors that occurred on earlier versions, even with operating systems that support >2 GB files.
So on W2K, Zope can have >2GB databases? I'd really like to hear if you get it to work Andy (or if anyone else did.) -- Itamar S.T. itamar@maxnm.com Fingerprint = D365 7BE8 B81E 2B18 6534 025E D0E7 92DB E441 411C
Thank you everyone for your help. I truncated the file and restarted Zope successfully. I had a brief period of problems when I forgot use binary mode for truncating the file - duh. Anyway, I am running 2.2.1 and it doesnt work with a 2gb file. I didnt analyse where the problem lies, however I do believe its a python issue. Thanks. -- Andy McKay, Developer. ActiveState. ----- Original Message ----- From: "Itamar Shtull-Trauring" <itamar@maxnm.com> To: <zope-dev@zope.org> Sent: Sunday, November 05, 2000 2:37 AM Subject: Re: [Zope-dev] Fw: 2gb file size
Shane Hathaway wrote:
Are you running Zope 2.2.1 or later? There were pointer arithmetic errors that occurred on earlier versions, even with operating systems that support >2 GB files.
So on W2K, Zope can have >2GB databases? I'd really like to hear if you get it to work Andy (or if anyone else did.)
-- Itamar S.T. itamar@maxnm.com Fingerprint = D365 7BE8 B81E 2B18 6534 025E D0E7 92DB E441 411C
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
I did do a How-to as well http://www.zope.org/Members/andym/2gig -- Andy McKay, Developer. ActiveState. ----- Original Message ----- From: "Andy McKay" <andym@ActiveState.com> To: <zope-dev@zope.org> Sent: Monday, November 06, 2000 9:26 AM Subject: Re: [Zope-dev] Fw: 2gb file size
Thank you everyone for your help. I truncated the file and restarted Zope successfully. I had a brief period of problems when I forgot use binary mode for truncating the file - duh. Anyway, I am running 2.2.1 and it doesnt work with a 2gb file. I didnt analyse where the problem lies, however I do believe its a python issue.
Thanks. -- Andy McKay, Developer. ActiveState. ----- Original Message ----- From: "Itamar Shtull-Trauring" <itamar@maxnm.com> To: <zope-dev@zope.org> Sent: Sunday, November 05, 2000 2:37 AM Subject: Re: [Zope-dev] Fw: 2gb file size
Shane Hathaway wrote:
Are you running Zope 2.2.1 or later? There were pointer arithmetic errors that occurred on earlier versions, even with operating systems that support >2 GB files.
So on W2K, Zope can have >2GB databases? I'd really like to hear if you get it to work Andy (or if anyone else did.)
-- Itamar S.T. itamar@maxnm.com Fingerprint = D365 7BE8 B81E 2B18 6534 025E D0E7 92DB E441 411C
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Andy McKay wrote:
I did do a How-to as well http://www.zope.org/Members/andym/2gig
Not long ago I created a wrapper around File objects which automatically splits the file before it reached a predefined limit. It could handle any number of partitioned file segements. I tried it with FileStorage and even integrated it in. That's how we were able to find and fix some arithmetic errors that occurred in FileStorage when reaching the 2GB barrier. If there's interest in the community I can release the code (in fact, there is no dependency on Zope and it's pretty fast.) OTOH, I can verify there are production sites that have succesfully reached around 10 GB in Data.fs size. Shane
-- Andy McKay, Developer. ActiveState. ----- Original Message ----- From: "Andy McKay" <andym@ActiveState.com> To: <zope-dev@zope.org> Sent: Monday, November 06, 2000 9:26 AM Subject: Re: [Zope-dev] Fw: 2gb file size
Thank you everyone for your help. I truncated the file and restarted Zope successfully. I had a brief period of problems when I forgot use binary mode for truncating the file - duh. Anyway, I am running 2.2.1 and it doesnt work with a 2gb file. I didnt analyse where the problem lies, however I do believe its a python issue.
Thanks. -- Andy McKay, Developer. ActiveState. ----- Original Message ----- From: "Itamar Shtull-Trauring" <itamar@maxnm.com> To: <zope-dev@zope.org> Sent: Sunday, November 05, 2000 2:37 AM Subject: Re: [Zope-dev] Fw: 2gb file size
Shane Hathaway wrote:
Are you running Zope 2.2.1 or later? There were pointer arithmetic errors that occurred on earlier versions, even with operating systems that support >2 GB files.
So on W2K, Zope can have >2GB databases? I'd really like to hear if you get it to work Andy (or if anyone else did.)
-- Itamar S.T. itamar@maxnm.com Fingerprint = D365 7BE8 B81E 2B18 6534 025E D0E7 92DB E441 411C
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Shane Hathaway wrote:
Not long ago I created a wrapper around File objects which automatically splits the file before it reached a predefined limit. It could handle any number of partitioned file segements.
Yes please! Perhaps it could be integrated into the ZODB codebase while you're at it?
Itamar Shtull-Trauring wrote:
Shane Hathaway wrote:
Not long ago I created a wrapper around File objects which automatically splits the file before it reached a predefined limit. It could handle any number of partitioned file segements.
Yes please! Perhaps it could be integrated into the ZODB codebase while you're at it?
I'm working on cleaning it up so that it integrates cleanly with FileStorage. Then I'll release it. BTW your roadmap document is fantastic. I hope you're getting lots of good feedback. It really brings everything together. Shane
Shane This would be very helpfull to share. One of the reasons I tried the Berkeley Database was because I kept blowing this 2Gb limit. Prehaps if this was released I could finally finish the zope mail archives. :-) Cheers -Andy
-----Original Message----- From: shane@shane.digicool.com [mailto:shane@shane.digicool.com]On Behalf Of Shane Hathaway Sent: 06 November 2000 21:51 To: Andy McKay; zope-dev@zope.org Subject: Re: [Zope-dev] Fw: 2gb file size
Andy McKay wrote:
I did do a How-to as well http://www.zope.org/Members/andym/2gig
Not long ago I created a wrapper around File objects which automatically splits the file before it reached a predefined limit. It could handle any number of partitioned file segements. I tried it with FileStorage and even integrated it in. That's how we were able to find and fix some arithmetic errors that occurred in FileStorage when reaching the 2GB barrier. If there's interest in the community I can release the code (in fact, there is no dependency on Zope and it's pretty fast.)
OTOH, I can verify there are production sites that have succesfully reached around 10 GB in Data.fs size.
Shane
-- Andy McKay, Developer. ActiveState. ----- Original Message ----- From: "Andy McKay" <andym@ActiveState.com> To: <zope-dev@zope.org> Sent: Monday, November 06, 2000 9:26 AM Subject: Re: [Zope-dev] Fw: 2gb file size
Thank you everyone for your help. I truncated the file and restarted Zope successfully. I had a brief period of problems when I forgot use binary mode for truncating the file - duh. Anyway, I am running 2.2.1 and it doesnt work with a 2gb file. I didnt analyse where the problem lies, however I do believe its a python issue.
Thanks. -- Andy McKay, Developer. ActiveState. ----- Original Message ----- From: "Itamar Shtull-Trauring" <itamar@maxnm.com> To: <zope-dev@zope.org> Sent: Sunday, November 05, 2000 2:37 AM Subject: Re: [Zope-dev] Fw: 2gb file size
Shane Hathaway wrote:
Are you running Zope 2.2.1 or later? There were pointer arithmetic errors that occurred on earlier versions, even with operating systems that support >2 GB files.
So on W2K, Zope can have >2GB databases? I'd really like to hear if you get it to work Andy (or if anyone else did.)
-- Itamar S.T. itamar@maxnm.com Fingerprint = D365 7BE8 B81E 2B18 6534 025E D0E7 92DB E441 411C
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
participants (5)
-
Andy Dawkins -
Andy McKay -
Chris McDonough -
Itamar Shtull-Trauring -
Shane Hathaway