Error in Copy/Import during Migration
I'm fairly well versed on the inner workings of Zope, but yesterday I've run into a problem that I can grasp. First, some background on what I'm doing. I've got a 200GB Data.fs file which I'm splitting up into multiple storages. The way I'm doing this is to create a zope instance which mounts a new storage and the old storage and then a python script connects to the application and uses a modified version of manage_clone (which maintains ownership) to copy content into the new storage. This works correctly for most things, but yesterday I tried to move a folder with about 3.1GB of stuff in it. On a side note, I ran into an issue with running out of /tmp space during the clone. The _getCopy method of CopySourse exports an object to a tempfile and then imports it to create the copy. When copying something larger than available tmp space zope crashes. I worked around this by explicitly setting tempfile.tempdir to somewhere with plenty of free space. There may be an environment variable to do this as well. Just an FYI. Anyway, here's what I see. The copy starts, a tempfile is created (on the client) and grows to the appropriate size (3.1GB). Then, a Data.fs.temp appears on the new server and grows to the appropriate size. However, when my script tries to commit the transaction I get an EOF error: Traceback (most recent call last): File "./migrate_site.py", line 45, in ? transaction.get().commit() File "/var/local/zope/Zope-2.11.4-1/lib/python/transaction/_transaction.py", line 325, in commit self._commitResources() File "/var/local/zope/Zope-2.11.4-1/lib/python/transaction/_transaction.py", line 427, in _commitResources rm.tpc_vote(self) File "/var/local/zope/Zope-2.11.4-1/lib/python/ZODB/Connection.py", line 744, in tpc_vote s = vote(transaction) File "/var/local/zope/Zope-2.11.4-1/lib/python/ZEO/ClientStorage.py", line 1004, in tpc_vote self._server.vote(id(txn)) File "/var/local/zope/Zope-2.11.4-1/lib/python/ZEO/ServerStub.py", line 262, in vote return self.rpc.call('vote', trans_id) File "/var/local/zope/Zope-2.11.4-1/lib/python/ZEO/zrpc/connection.py", line 699, in call raise inst # error raised by server EOFError I don't seem to have any space issues on any devices, and the process works for smaller site. Any input would be greatly appreciated. Thanks, -- Brian Brinegar Web Services Coordinator Engineering Computer Network
+-------[ Brian Brinegar ]---------------------- | I'm fairly well versed on the inner workings of Zope, but yesterday I've | run into a problem that I can grasp. [snip] | Anyway, here's what I see. The copy starts, a tempfile is created (on | the client) and grows to the appropriate size (3.1GB). Then, a | Data.fs.temp appears on the new server and grows to the appropriate | size. However, when my script tries to commit the transaction I get an | EOF error: File size limits on the new server restricting to say 2GB? -- Andrew Milton akm@theinternet.com.au
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 1/7/10 2:30 PM, Andrew Milton wrote:
+-------[ Brian Brinegar ]---------------------- | I'm fairly well versed on the inner workings of Zope, but yesterday I've | run into a problem that I can grasp.
[snip]
| Anyway, here's what I see. The copy starts, a tempfile is created (on | the client) and grows to the appropriate size (3.1GB). Then, a | Data.fs.temp appears on the new server and grows to the appropriate | size. However, when my script tries to commit the transaction I get an | EOF error:
File size limits on the new server restricting to say 2GB?
No. It's an x86-64 RedHat 5 install. We've had 250GB files on this system. - -Chris -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.12 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktGNxoACgkQ+zGkEDwn8Uam5QCcDOe62Znr9CrfuwpvCPXXCW0T oKEAoMjde1r8SKzdv38UTbf6ozaAeNWE =O65C -----END PGP SIGNATURE-----
+-------[ Christopher N. Deckard ]---------------------- | -----BEGIN PGP SIGNED MESSAGE----- | Hash: SHA1 | | On 1/7/10 2:30 PM, Andrew Milton wrote: | > +-------[ Brian Brinegar ]---------------------- | > | I'm fairly well versed on the inner workings of Zope, but yesterday I've | > | run into a problem that I can grasp. | > | > [snip] | > | > | Anyway, here's what I see. The copy starts, a tempfile is created (on | > | the client) and grows to the appropriate size (3.1GB). Then, a | > | Data.fs.temp appears on the new server and grows to the appropriate | > | size. However, when my script tries to commit the transaction I get an | > | EOF error: | > | > File size limits on the new server restricting to say 2GB? | | No. It's an x86-64 RedHat 5 install. We've had 250GB files on this system. Got any errors in your ZEO log ? -- Andrew Milton akm@theinternet.com.au
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 1/7/10 2:39 PM, Andrew Milton wrote:
+-------[ Christopher N. Deckard ]---------------------- | -----BEGIN PGP SIGNED MESSAGE----- | Hash: SHA1 | | On 1/7/10 2:30 PM, Andrew Milton wrote: | > +-------[ Brian Brinegar ]---------------------- | > | I'm fairly well versed on the inner workings of Zope, but yesterday I've | > | run into a problem that I can grasp. | > | > [snip] | > | > | Anyway, here's what I see. The copy starts, a tempfile is created (on | > | the client) and grows to the appropriate size (3.1GB). Then, a | > | Data.fs.temp appears on the new server and grows to the appropriate | > | size. However, when my script tries to commit the transaction I get an | > | EOF error: | > | > File size limits on the new server restricting to say 2GB? | | No. It's an x86-64 RedHat 5 install. We've had 250GB files on this system.
Got any errors in your ZEO log ?
Aha! Our zeo log shows that we are running out of space in /tmp. What is the best way to redefine the location for temp files for the ZEO server? - -Chris -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.12 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktGPuYACgkQ+zGkEDwn8UbojQCglylx+tFbmuALBmEK7dFXACRK 6BcAoMooVxYsbfSCwRjUbV3VDAyqfYMn =XHFb -----END PGP SIGNATURE-----
+-------[ Christopher N. Deckard ]---------------------- | -----BEGIN PGP SIGNED MESSAGE----- | Hash: SHA1 | | On 1/7/10 2:39 PM, Andrew Milton wrote: | > +-------[ Christopher N. Deckard ]---------------------- | > | -----BEGIN PGP SIGNED MESSAGE----- | > | Hash: SHA1 | > | | > | On 1/7/10 2:30 PM, Andrew Milton wrote: | > | > +-------[ Brian Brinegar ]---------------------- | > | > | I'm fairly well versed on the inner workings of Zope, but yesterday I've | > | > | run into a problem that I can grasp. | > | > | > | > [snip] | > | > | > | > | Anyway, here's what I see. The copy starts, a tempfile is created (on | > | > | the client) and grows to the appropriate size (3.1GB). Then, a | > | > | Data.fs.temp appears on the new server and grows to the appropriate | > | > | size. However, when my script tries to commit the transaction I get an | > | > | EOF error: | > | > | > | > File size limits on the new server restricting to say 2GB? | > | | > | No. It's an x86-64 RedHat 5 install. We've had 250GB files on this system. | > | > Got any errors in your ZEO log ? | | Aha! Our zeo log shows that we are running out of space in /tmp. What | is the best way to redefine the location for temp files for the ZEO server? Set the TMPDIR env var to point to where you want tmp files created in the shell script that starts your ZEO server -- Andrew Milton akm@theinternet.com.au
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 1/7/10 3:10 PM, Andrew Milton wrote:
Set the TMPDIR env var to point to where you want tmp files created in the shell script that starts your ZEO server
We got our tmpdir issues worked out, but then got this error: 2010-01-07 15:40:34 CRITICAL txn.47988082355376 A storage error occurred during the second phase of the two-phase commit. Resources may be in an inconsistent state. Traceback (most recent call last): File "./migrate_site.py", line 45, in ? transaction.get().commit() File "/var/local/zope/Zope-2.11.4-1/lib/python/transaction/_transaction.py", line 325, in commit self._commitResources() File "/var/local/zope/Zope-2.11.4-1/lib/python/transaction/_transaction.py", line 432, in _commitResources rm.tpc_finish(self) File "/var/local/zope/Zope-2.11.4-1/lib/python/ZODB/Connection.py", line 758, in tpc_finish self._storage.tpc_finish(transaction, callback) File "/var/local/zope/Zope-2.11.4-1/lib/python/ZEO/ClientStorage.py", line 1079, in tpc_finish raise ClientDisconnected( ZEO.Exceptions.ClientDisconnected: Calling tpc_finish() on a disconnected transaction - -Chris -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.12 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktGSBwACgkQ+zGkEDwn8UY/TwCggnFf2KLjId+J/wI4LIESuuM+ b1wAoKz8p/JpbA/WOQHZQ3fgMyglmpaE =zKI2 -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Christopher N. Deckard wrote:
On 1/7/10 3:10 PM, Andrew Milton wrote:
Set the TMPDIR env var to point to where you want tmp files created in the shell script that starts your ZEO server
We got our tmpdir issues worked out, but then got this error:
2010-01-07 15:40:34 CRITICAL txn.47988082355376 A storage error occurred during the second phase of the two-phase commit. Resources may be in an inconsistent state. Traceback (most recent call last): File "./migrate_site.py", line 45, in ? transaction.get().commit() File "/var/local/zope/Zope-2.11.4-1/lib/python/transaction/_transaction.py", line 325, in commit self._commitResources() File "/var/local/zope/Zope-2.11.4-1/lib/python/transaction/_transaction.py", line 432, in _commitResources rm.tpc_finish(self) File "/var/local/zope/Zope-2.11.4-1/lib/python/ZODB/Connection.py", line 758, in tpc_finish self._storage.tpc_finish(transaction, callback) File "/var/local/zope/Zope-2.11.4-1/lib/python/ZEO/ClientStorage.py", line 1079, in tpc_finish raise ClientDisconnected( ZEO.Exceptions.ClientDisconnected: Calling tpc_finish() on a disconnected transaction
Try restarting the appservers after the restart of the ZEO server. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktGS3IACgkQ+gerLs4ltQ5vyQCdHUGdM13eHGh0YFibqovvxuPu rIsAmgL02kC/XczD59oGhyY8pDhicByN =+owh -----END PGP SIGNATURE-----
+-------[ Christopher N. Deckard ]---------------------- | -----BEGIN PGP SIGNED MESSAGE----- | Hash: SHA1 | | On 1/7/10 2:30 PM, Andrew Milton wrote: | > +-------[ Brian Brinegar ]---------------------- | > | I'm fairly well versed on the inner workings of Zope, but yesterday I've | > | run into a problem that I can grasp. | > | > [snip] | > | > | Anyway, here's what I see. The copy starts, a tempfile is created (on | > | the client) and grows to the appropriate size (3.1GB). Then, a | > | Data.fs.temp appears on the new server and grows to the appropriate | > | size. However, when my script tries to commit the transaction I get an | > | EOF error: | > | > File size limits on the new server restricting to say 2GB? | | No. It's an x86-64 RedHat 5 install. We've had 250GB files on this system. Also have you done a cmp(1) of the chunk and the data.fs.tmp file to make sure you're not just seeing a sparse file of the right size that's internally truncated? -- Andrew Milton akm@theinternet.com.au
participants (4)
-
Andrew Milton -
Brian Brinegar -
Christopher N. Deckard -
Tres Seaver