Re: ZEO and 2gb max limit for linux .. not for solaris etc.
Sven Hohage wrote:
Hallo Jorgen, I write you cause I've got a question about Zope. You've posted on the mailing list on my question with the filesize of the data.fs or in ZEO ZSS. My problem is that I'd like to know which Unix - versions don't have this filesize limitations and if there is a linux version without that limit. Or do you know another way to solve this problem?? Many thanks Sven
Hmm.. well .. I can tell you what I know as of today. Linux has this limitation as of today. At least all the variants I have come over (mostly Red Hat and Suse). The limitation is in the filesystem implementation itself; meaning that the number of "references" that can be created cannot exceed the address limit of 2gb. Solaris does not have this limitation and neither does Hp-Ux or SGI Irix. I know previous versions of SunOS (v4.x) had the same limitation as linux is experiencing today. Though.. there are ways "around" the problem. Though; all different over the variuos unix versions as of today. When running Zope you really don't have much choice on linux. When you hit the 2gb barrier .. you have problems. I have not tried it .. but as I can recall there was a product for Zope that made it possible to "mount" several databases at once .. though it was beta. .. this might solve a problem .. and would from my point of view be a nice way to go. This way; the administrator of the system also have some choices as to where he want to put his data. Then you can move certain object databases to fast storage .. and so forth. .. so; if you have those problems; get Solaris v7/v8 for Intel (or if you can afford it-> sparc!) .. and run Zope of that. I do so .. but then again; I am not near the 2gb barrier that many people have come over. ;) .. on the other hand; I am not aware of the various "beta" filesystem implementations and kernel "to-do"'s for linux, but I suspect that it will not be long before that problem is solved. .. oh! When I think of it.. SGI has ported their XFS to linux. Not sure how far that implementation has gone yet.. but the link to check out is; http://oss.sgi.com/projects/xfs/ Looks promising.. and a "production" like filesystem for linux is a "must" as far as I can see. ;) Just read the brief here; Sub-second filesystem recovery after crashes or power failures (never wait for long fscks again) 64-bit scalability: millions of terabytes, millions of files, and a million files per directory (no more 2 GB limits) High reliability and performance from journaling and other advanced algorithms Hope that helped a bit?? .. just shoot if there's more I can help with! ;) -- Med vennlig hilsen, Jørgen Skogstad /\ System Specialist \\ \ \ \\ / Sun Microsystems AS / \/ / / Gjerdrums vei 12 / / \//\ N-0486 Oslo, NORWAY \//\ / / / / /\ / Phone: + 47 22 02 39 00 / \\ \ Cellular: + 47 92 41 30 23 \ \\ Fax: + 47 22 02 39 99 \/ EMail: jorgen.skogstad@Norway.Sun.Com Web: www.sun.no BTW; Opinions formed by myself in informal mailings are mine and not of my employer.
If you don't wan't to run Sun or HP-UX you could always look at one of the open *BSD clones. As I can recall both OpenBSD and FreeBSD have filsystems that manage >2GB files. /Johan
Sven Hohage wrote:
Hallo Jorgen, I write you cause I've got a question about Zope. You've posted on the mailing list on my question with the filesize of the data.fs or in ZEO ZSS. My problem is that I'd like to know which Unix - versions don't have this filesize limitations and if there is a linux version without that limit. Or do you know another way to solve this problem?? Many thanks Sven
Hmm.. well .. I can tell you what I know as of today. Linux has this limitation as of today. At least all the variants I have come over (mostly Red Hat and Suse). The limitation is in the filesystem implementation itself; meaning that the number of "references" that can be created cannot exceed the address limit of 2gb.
Solaris does not have this limitation and neither does Hp-Ux or SGI Irix. I know previous versions of SunOS (v4.x) had the same limitation as linux is experiencing today.
Though.. there are ways "around" the problem. Though; all different over the variuos unix versions as of today. When running Zope you really don't have much choice on linux. When you hit the 2gb barrier .. you have problems. I have not tried it .. but as I can recall there was a product for Zope that made it possible to "mount" several databases at once .. though it was beta.
.. this might solve a problem .. and would from my point of view be a nice way to go. This way; the administrator of the system also have some choices as to where he want to put his data. Then you can move certain object databases to fast storage .. and so forth.
.. so; if you have those problems; get Solaris v7/v8 for Intel (or if you can afford it-> sparc!) .. and run Zope of that.
I do so .. but then again; I am not near the 2gb barrier that many people have come over. ;)
.. on the other hand; I am not aware of the various "beta" filesystem implementations and kernel "to-do"'s for linux, but I suspect that it will not be long before that problem is solved.
.. oh! When I think of it.. SGI has ported their XFS to linux. Not sure how far that implementation has gone yet.. but the link to check out is;
http://oss.sgi.com/projects/xfs/
Looks promising.. and a "production" like filesystem for linux is a "must" as far as I can see. ;) Just read the brief here;
Sub-second filesystem recovery after crashes or power failures (never wait for long fscks again)
64-bit scalability: millions of terabytes, millions of files, and a million files per directory (no more 2 GB limits)
High reliability and performance from journaling and other advanced algorithms
Hope that helped a bit??
.. just shoot if there's more I can help with! ;)
--
Med vennlig hilsen, Jørgen Skogstad
/\ System Specialist \\ \ \ \\ / Sun Microsystems AS / \/ / / Gjerdrums vei 12 / / \//\ N-0486 Oslo, NORWAY \//\ / / / / /\ / Phone: + 47 22 02 39 00 / \\ \ Cellular: + 47 92 41 30 23 \ \\ Fax: + 47 22 02 39 99 \/ EMail: jorgen.skogstad@Norway.Sun.Com Web: www.sun.no
BTW; Opinions formed by myself in informal mailings are mine and not of my employer.
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Jørgen Skogstad wrote:
Sven Hohage wrote:
Hallo Jorgen, I write you cause I've got a question about Zope. You've posted on the mailing list on my question with the filesize of the data.fs or in ZEO ZSS. My problem is that I'd like to know which Unix - versions don't have this filesize limitations and if there is a linux version without that limit. Or do you know another way to solve this problem?? Many thanks Sven
Hmm.. well .. I can tell you what I know as of today. Linux has this limitation as of today. At least all the variants I have come over (mostly Red Hat and Suse). The limitation is in the filesystem implementation itself; meaning that the number of "references" that can be created cannot exceed the address limit of 2gb.
Not quite true. The problem only exixts in Linux on 32 bit x86 hardware. Wherever Linux is 64 bit, the problem is non-extant. BTW, both RedHat and Suse (and Debian, IIRC) have distribions for the Alpha platform. Also, all this is detailed several times in the archives. Bill Anderson -- Do not meddle in the affairs of sysadmins, for they are easy to annoy, and have the root password.
Hmm.. well .. I can tell you what I know as of today. Linux has this limitation as of today. At least all the variants I have come over (mostly Red Hat and Suse). The limitation is in the filesystem implementation itself; meaning that the number of "references" that can be created cannot exceed the address limit of 2gb. Not quite true. The problem only exixts in Linux on 32 bit x86 hardware. Wherever Linux is 64 bit, the problem is non-extant. BTW, both RedHat and Suse (and Debian, IIRC) have distribions for the Alpha platform.
Yep.. didn't think of that as I have never reasoned for running linux on any of those platforms. ;) I've used to running Solaris on sparc.. and when needed linux on x86 hardware (when solaris for intel was not the correct choice). Though; I look forward for SGI's implementation og XFS when it arrives in beta this summer. Though; are there any possibilities that Zope may implement "Oracle style" distribution og the databases in due time? Like then you have a bit more "control" over where you put your data .. and what kind of devices you put them on? .. possibly this can be done through ZEO and the beta shared object database (mount more than one instance of the Data.fs files). Have a real nice day! ;) Kindest, Jørgen Skogstad
Jørgen Skogstad wrote:
Hmm.. well .. I can tell you what I know as of today. Linux has this limitation as of today. At least all the variants I have come over (mostly Red Hat and Suse). The limitation is in the filesystem implementation itself; meaning that the number of "references" that can be created cannot exceed the address limit of 2gb. Not quite true. The problem only exixts in Linux on 32 bit x86 hardware. Wherever Linux is 64 bit, the problem is non-extant. BTW, both RedHat and Suse (and Debian, IIRC) have distribions for the Alpha platform.
Yep.. didn't think of that as I have never reasoned for running linux on any of those platforms. ;) I've used to running Solaris on sparc.. and when needed linux on x86 hardware (when solaris for intel was not the correct choice).
Though; I look forward for SGI's implementation og XFS when it arrives in beta this summer.
I believe, though I may be wrong, that the 2.4 kernel also removes the problem. Which also is due out RSN. :) I am given to understand though, that some have had no success in getting python to create (and use?) a >2GB file, even though the uiderlying mechanisms could (this was on Linux ). So I am hopefull, yet cautios. ANd yes, I am eagerly awaiting for XFS to be done. I already have played with a snapshot of it. Not bad, needs work, but hey, it isn't released yet so, what could I expect, eh? ;)
Though; are there any possibilities that Zope may implement "Oracle style" distribution og the databases in due time? Like then you have a bit more "control" over where you put your data .. and what kind of devices you put them on?
.. possibly this can be done through ZEO and the beta shared object database (mount more than one instance of the Data.fs files).
not quite! You can _either_ use ZEO OR the MountedFileStorage product, but not both. I was working on a MountedZEOStorage, but haven't had as much time for it as I thought I would. Should be relatively easy though. -- Do not meddle in the affairs of sysadmins, for they are easy to annoy, and have the root password.
Please note that we recently found a bug in ZODB FileStorage that causes it to fail miserably if a file gets over 2GB. This bug is fixed in Zope 2.2.1, which is being released in beta today. Note that even with this fix, your OS/File-System/Python-config must support files over 2GB to allow large FileStorage. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
participants (5)
-
Bill Anderson -
Jim Fulton -
Jørgen Skogstad -
Jørgen Skogstad -
Zope