MemoryError exception when importing a zexp file
Hi, I'm trying to import a .zexp file that's 1.6GB but it continually fails due to a MemoryError exception. I guess that adding more RAM would probably fix this problem but does anyone know if there's any way of getting around this sort of problem without increasing the RAM? This is the traceback: -- Site Error An error was encountered while publishing this resource. exceptions.MemoryError Sorry, a site error occurred. Traceback (innermost last): a.. Module ZPublisher.Publish, line 163, in publish_module_standard b.. Module Products.PlacelessTranslationService.PatchStringIO, line 51, in new_publish c.. Module ZPublisher.Publish, line 108, in publish d.. Module Zope.App.startup, line 226, in abort e.. Module ZODB.Transaction, line 149, in abort f.. Module ZODB.Connection, line 474, in abort_sub MemoryError -- Thanks Ben -- Ben Lobo Application Developer Adaptive Technologies Ltd. +44 (0)1273 728128 http://www.adaptivetechnologies.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 29 May 2007, at 08:36, Ben Lobo wrote:
Hi,
I'm trying to import a .zexp file that's 1.6GB but it continually fails due to a MemoryError exception. I guess that adding more RAM would probably fix this problem but does anyone know if there's any way of getting around this sort of problem without increasing the RAM?
Short of increasing memory on that specific host you could also create a sandbox on a different machine with more RAM and then copy the database when it's done. jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iD8DBQFGW9GTRAx5nvEhZLIRAoyRAJwM2QyR764vdHecw/xLNkFUcHWdyACfRK+o YZIVsaudyBaMUjEgl/X0xkU= =uFQ8 -----END PGP SIGNATURE-----
Hi, Maybe you could try giving the machine more swap space (partition in Linux, file in Windows) You'd probably have performance issues, but if the import is a one-timer, it might be acceptable. Jonathan -----Message d'origine----- De : zope-bounces@zope.org [mailto:zope-bounces@zope.org] De la part de Jens Vagelpohl Envoyé : mardi 29 mai 2007 09:09 À : zope list user Objet : Re: [Zope] MemoryError exception when importing a zexp file -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 29 May 2007, at 08:36, Ben Lobo wrote:
Hi,
I'm trying to import a .zexp file that's 1.6GB but it continually fails due to a MemoryError exception. I guess that adding more RAM would probably fix this problem but does anyone know if there's any way of getting around this sort of problem without increasing the RAM?
Short of increasing memory on that specific host you could also create a sandbox on a different machine with more RAM and then copy the database when it's done. jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iD8DBQFGW9GTRAx5nvEhZLIRAoyRAJwM2QyR764vdHecw/xLNkFUcHWdyACfRK+o YZIVsaudyBaMUjEgl/X0xkU= =uFQ8 -----END PGP SIGNATURE----- _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
Thanks for the tip. I tried increasing the swap file to 10GB but still got the same memory error. I'm now considering exporting the Plone site (cos that's what it is) one piece at a time and reimporting it into a clean Plone site. Any thoughts on whether or not this is a good idea? Cheers Ben "Winterflood, Jonathan" wrote:
Hi,
Maybe you could try giving the machine more swap space (partition in Linux, file in Windows) You'd probably have performance issues, but if the import is a one-timer, it might be acceptable.
Jonathan
-----Message d'origine----- De : zope-bounces@zope.org [mailto:zope-bounces@zope.org] De la part de Jens Vagelpohl Envoyé : mardi 29 mai 2007 09:09 À : zope list user Objet : Re: [Zope] MemoryError exception when importing a zexp file
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 29 May 2007, at 08:36, Ben Lobo wrote:
Hi,
I'm trying to import a .zexp file that's 1.6GB but it continually fails due to a MemoryError exception. I guess that adding more RAM would probably fix this problem but does anyone know if there's any way of getting around this sort of problem without increasing the RAM?
Short of increasing memory on that specific host you could also create a sandbox on a different machine with more RAM and then copy the database when it's done.
jens
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 30 May 2007, at 11:31, Ben Lobo wrote:
Thanks for the tip. I tried increasing the swap file to 10GB but still got the same memory error.
I'm now considering exporting the Plone site (cos that's what it is) one piece at a time and reimporting it into a clean Plone site. Any thoughts on whether or not this is a good idea?
I guess the first question should be why you are doing that export/ import at all? Is there any good reason for it? jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iD8DBQFGXUuhRAx5nvEhZLIRAg8LAJ9EvvNAMjCw8M6KhmnXv6gO23A2iwCfQmvi 00br+QjNySuhxGDF4QQ9dXg= =qYau -----END PGP SIGNATURE-----
Hi, Failing with 10 GB seems weird... Did you monitor real memory usage to see where it failed? I'm thinking maybe the python interpreter / zope / something is limiting its memory usage (IIRC, Java VMs are limited to 128M by default) Could someone confirm/infirm this? I don't have any experience w/ Plone, so I don't know whether it would work, but I imagine you might have to re-import the pieces in the right order, so that there wouldn't be any broken dependencies (just speculating here) I don't know what get's exported in the .zexp file, but maybe packing the database to remove old versions of the Zope objects will reduce the size (not sure whether the history is included in the export... but maybe) Cheers Jonathan -----Message d'origine----- De : zope-bounces@zope.org [mailto:zope-bounces@zope.org] De la part de Ben Lobo Envoyé : mercredi 30 mai 2007 11:32 À : zope@zope.org Objet : [Zope] Re: MemoryError exception when importing a zexp file Thanks for the tip. I tried increasing the swap file to 10GB but still got the same memory error. I'm now considering exporting the Plone site (cos that's what it is) one piece at a time and reimporting it into a clean Plone site. Any thoughts on whether or not this is a good idea? Cheers Ben "Winterflood, Jonathan" wrote:
Hi,
Maybe you could try giving the machine more swap space (partition in Linux, file in Windows) You'd probably have performance issues, but if the import is a one-timer, it might be acceptable.
Jonathan
-----Message d'origine----- De : zope-bounces@zope.org [mailto:zope-bounces@zope.org] De la part de Jens Vagelpohl Envoyé : mardi 29 mai 2007 09:09 À : zope list user Objet : Re: [Zope] MemoryError exception when importing a zexp file
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 29 May 2007, at 08:36, Ben Lobo wrote:
Hi,
I'm trying to import a .zexp file that's 1.6GB but it continually fails due to a MemoryError exception. I guess that adding more RAM would probably fix this problem but does anyone know if there's any way of getting around this sort of problem without increasing the RAM?
Short of increasing memory on that specific host you could also create a sandbox on a different machine with more RAM and then copy the database when it's done.
jens
This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 30 May 2007, at 12:12, Winterflood, Jonathan wrote:
Hi,
Failing with 10 GB seems weird... Did you monitor real memory usage to see where it failed? I'm thinking maybe the python interpreter / zope / something is limiting its memory usage (IIRC, Java VMs are limited to 128M by default) Could someone confirm/infirm this?
There is no such limiting mechanism anywhere in the stack. However, there may always be OS-level limitations imposed by the systems administrator. jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iD8DBQFGXVxkRAx5nvEhZLIRAuZbAJ90Y6NLwZVo/AErmNbGGhIfUzdLmACfZbd4 GAgyEd3UI+0h5vj6820oIA0= =Igdw -----END PGP SIGNATURE-----
Winterflood, Jonathan wrote at 2007-5-30 12:12 +0200:
... Failing with 10 GB seems weird... Did you monitor real memory usage to see where it failed?
A 32 bit Linux will limit the amount of virtual memory per process to somewhat between 2.5 and 3 GB -- for reasons of address bit restrictions: With 32 bit addresses you can only address about 4 GB and parts of the address space is reserved for other purposes (than heap). You need a 64 bit architecture to have a significantly higher virtual address space. -- Dieter
(Wed, May 30, 2007 at 10:31:31AM +0100) Ben Lobo wrote/schrieb/egrapse:
Thanks for the tip. I tried increasing the swap file to 10GB but still got the same memory error.
Some operating systems limit the amount of RAM a single process can consume. Sometimes that amount can be adjusted. Also 32bit systems can only use a limited amount of RAM per process due to the limit of what you can address with 32bits -- over that amount you can give the system 100GB or 1000GB of swap, it won't change a thing. Even if those do not apply, I guess it's very much possible for a process to eat up all the RAM, no matter how much that is, all it would need is a little bit of recursion.
I'm now considering exporting the Plone site (cos that's what it is) one piece at a time and reimporting it into a clean Plone site. Any thoughts on whether or not this is a good idea?
Not mentioning Plone in the first place is problematic. In fact *plone* is problematic. Did you verify with plohn people (e.g. on a Plone mailing list) that current plone versions can be imported/exported? (Because the last time I made the mistake to do a project with plone you couldn't import/export a plone site -- or rename it, move it to another instance or anything similar for that matter -- without major pain. It's been a while though, so things could have changed.) Summary: Go and ask the plone people, if you can't import anyway, the whole exercise is in vain. Good luck! Regards, Sascha
I'm now considering exporting the Plone site (cos that's what it is) one piece at a time and reimporting it into a clean Plone site. Any thoughts on whether or not this is a good idea?
Not mentioning Plone in the first place is problematic. In fact *plone* is problematic. Did you verify with plohn people (e.g. on a Plone mailing list) that current plone versions can be imported/exported? (Because the last time I made the mistake to do a project with plone you couldn't import/export a plone site -- or rename it, move it to another instance or anything similar for that matter -- without major pain. It's been a while though, so things could have changed.) Summary: Go and ask the plone people, if you can't import anyway, the whole exercise is in vain.
I've happily moved, renamed, exported and imported Plone sites, both within a single zope instance and between separate instances on many occasions so I don't think Plone is the problem. The only difference in this case is the size of the Plone site and the amount of RAM it seems to want to use. I'm not much of an expert on diagnosing and managing memory problems so I'm not sure how to check things like where the import failed other than by looking at Windows task manager (it's Windows 2003 server). If there are any OS limitations on memory usage, it's only what would normally be set up on a standard Windows installation. Someone in this thread asked why I'm doing this export in the first place: I want to import it back into zope but into a separate storage file so that I can back it up separately from the main storage. Someone suggested that I might be able to do it by copying the entire Data.fs file to the new strage location, restarting zope and then deleting the bits that I don't want instead of exporting then importing. I tried this but it didn't work unfortunately. The new mount point just appeared in the zmi root with a broken icon and gave me an 'unsubscriptable object type error when I tried to access it. Any further tips would be gratefully welcomed. Cheers Ben
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 31 May 2007, at 13:56, Ben Lobo wrote:
Someone in this thread asked why I'm doing this export in the first place: I want to import it back into zope but into a separate storage file so that I can back it up separately from the main storage. Someone suggested that I might be able to do it by copying the entire Data.fs file to the new strage location, restarting zope and then deleting the bits that I don't want instead of exporting then importing.
I tried this but it didn't work unfortunately. The new mount point just appeared in the zmi root with a broken icon and gave me an 'unsubscriptable object type error when I tried to access it.
Good that I asked. That's not a valid reason to do import/export. As the other person mentioned, you should copy the database and do it that way. If you run into problems that way you need to post useful error messages (meaning, something better than "didn't work") and paste the full error traceback and all symptoms. jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iD8DBQFGXs4JRAx5nvEhZLIRAloCAJ92O4GoSGMDeMek0qKk/vEkdwkwawCdFle+ n+jnt5XauqI/JxsyG52zxdg= =gZuZ -----END PGP SIGNATURE-----
On Thu, May 31, 2007 at 09:55:26AM +0200, Sascha Welter wrote:
... Did you verify with plohn people (e.g. on a Plone mailing list) that current plone versions can be imported/exported? (Because the last time I made the mistake to do a project with plone you couldn't import/export a plone site -- or rename it, move it to another instance or anything similar for that matter -- without major pain. It's been a while though, so things could have changed.)
I spent several hours recently to import a .zexp of a Plone 2.0 site. Got nowhere. One suggestion I got (which I wasn't able to follow) was the following. If you have access to the original server, try this: 1) Make a full backup of the entire ZODB, eg. using repozo.py. 2) Set up a new Zope instance with all the same products installed, using your new backup copy of the ZODB. 3) Go into the ZMI for this new instance, and delete everythign EXCEPT this plone site. 4) Pack the database to get rid of all the other stuff. Or make a new clean backup. There. You now have a Data.fs that contains only your Plone site. Instead of importing the .zexp, just mount the database at the appropriate path. Disclaimer: I haven't tried this yet.
Summary: Go and ask the plone people, if you can't import anyway, the whole exercise is in vain.
+1 -- Paul Winkler http://www.slinkp.com
"Paul Winkler" <pw_lists@slinkp.com> wrote in message news:20070531150739.GA5772@slinkp.com...
I spent several hours recently to import a .zexp of a Plone 2.0 site. Got nowhere.
One suggestion I got (which I wasn't able to follow) was the following. If you have access to the original server, try this:
1) Make a full backup of the entire ZODB, eg. using repozo.py. 2) Set up a new Zope instance with all the same products installed, using your new backup copy of the ZODB. 3) Go into the ZMI for this new instance, and delete everythign EXCEPT
this
plone site. 4) Pack the database to get rid of all the other stuff. Or make a new clean backup.
There. You now have a Data.fs that contains only your Plone site. Instead of importing the .zexp, just mount the database at the appropriate path.
Disclaimer: I haven't tried this yet.
Thanks Paul. Someone else on another list also suggested this method and I tried it and it seems to work well. To make it work, there are a couple of additional bits that need to be followed: ... 5) Go back to the orignal Zope instance and delete the Plone site that you want to set up the new file storage for (make a backup first) 6) Make sure that the new mount point has the same name as the original Plone site in zope.conf Then when you create the new ZODB mount point, it will appear in the root of the ZMI as a Plone site just as before so there's not even any need to to re-create the catalogue. Cheers Ben
participants (6)
-
Ben Lobo -
Dieter Maurer -
Jens Vagelpohl -
Paul Winkler -
Sascha Welter -
Winterflood, Jonathan