I was looking, and trying to understand why 'tempstorage' is not included in the ZODB. Since some people might want to share sessions between ZEO clients, and that 'tempstorage' is what is mounted as '/temp_folder/session_data', why not make that easy to mount via ZEO? Does anyone see a problem with this? What I'm proposing is to move 'tempstorage' to 'ZODB', since it has no dependency on anything other than 'zLOG'. -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
Fine by me. - C On May 31, 2006, at 1:22 AM, Sidnei da Silva wrote:
I was looking, and trying to understand why 'tempstorage' is not included in the ZODB.
Since some people might want to share sessions between ZEO clients, and that 'tempstorage' is what is mounted as '/temp_folder/session_data', why not make that easy to mount via ZEO? Does anyone see a problem with this?
What I'm proposing is to move 'tempstorage' to 'ZODB', since it has no dependency on anything other than 'zLOG'.
-- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Sidnei da Silva schrieb:
I was looking, and trying to understand why 'tempstorage' is not included in the ZODB.
Since some people might want to share sessions between ZEO clients, and that 'tempstorage' is what is mounted as '/temp_folder/session_data', why not make that easy to mount via ZEO? Does anyone see a problem with this?
actually it is easy mountable via ZEO. I'm doing this already... Do you mean including the schema for zeo.conf per default? Regards Tino
On Wed, May 31, 2006 at 09:53:14AM +0200, Tino Wildenhain wrote: | Sidnei da Silva schrieb: | >I was looking, and trying to understand why 'tempstorage' is not | >included in the ZODB. | > | >Since some people might want to share sessions between ZEO clients, | >and that 'tempstorage' is what is mounted as | >'/temp_folder/session_data', why not make that easy to mount via ZEO? | >Does anyone see a problem with this? | | actually it is easy mountable via ZEO. I'm doing this already... | Do you mean including the schema for zeo.conf per default? Yeah, that too. I was actually thinking about the standalone ZODB package. Or does that not exist anymore? I was also wondering if this is the right thing to do (mounting tempstorage via ZEO) since I don't recall seeing any tutorial that recommended it. -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
On May 31, 2006, at 9:08 AM, Sidnei da Silva wrote:
On Wed, May 31, 2006 at 09:53:14AM +0200, Tino Wildenhain wrote: | Sidnei da Silva schrieb: | >I was looking, and trying to understand why 'tempstorage' is not | >included in the ZODB. | > | >Since some people might want to share sessions between ZEO clients, | >and that 'tempstorage' is what is mounted as | >'/temp_folder/session_data', why not make that easy to mount via ZEO? | >Does anyone see a problem with this? | | actually it is easy mountable via ZEO. I'm doing this already... | Do you mean including the schema for zeo.conf per default?
Yeah, that too. I was actually thinking about the standalone ZODB package. Or does that not exist anymore?
I was also wondering if this is the right thing to do (mounting tempstorage via ZEO) since I don't recall seeing any tutorial that recommended it.
Should be OK, although if you're doing it because you want a "fast" storage (or one that "produces fewer conflicts"), ZEO takes away much hope of that intrinsically. - C
On Wed, May 31, 2006 at 09:40:01AM -0400, Chris McDonough wrote: | Should be OK, although if you're doing it because you want a "fast" | storage (or one that "produces fewer conflicts"), ZEO takes away much | hope of that intrinsically. My goal was actually sharing sessions between zeo clients, without resorting to zsqlsessions or similar. -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
I've done this (at least with FileStorage) and it's sllllllooooooooowwww. Might be OK for low-traffic sites, but better to implement a custom session data container that stores stuff in SQL. I have the beginnings of one of these if you want to see it. - C On May 31, 2006, at 9:44 AM, Sidnei da Silva wrote:
On Wed, May 31, 2006 at 09:40:01AM -0400, Chris McDonough wrote: | Should be OK, although if you're doing it because you want a "fast" | storage (or one that "produces fewer conflicts"), ZEO takes away much | hope of that intrinsically.
My goal was actually sharing sessions between zeo clients, without resorting to zsqlsessions or similar.
-- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
On Wed, May 31, 2006 at 09:49:49AM -0400, Chris McDonough wrote: | I've done this (at least with FileStorage) and it's | sllllllooooooooowwww. Might be OK for low-traffic sites, but better | to implement a custom session data container that stores stuff in | SQL. I have the beginnings of one of these if you want to see it. Slow for using sessions? Why didn't you use tempstorage then? Or you are just asserting that FileStorage over ZEO is noticeably slower than directly? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
On May 31, 2006, at 9:55 AM, Sidnei da Silva wrote:
On Wed, May 31, 2006 at 09:49:49AM -0400, Chris McDonough wrote: | I've done this (at least with FileStorage) and it's | sllllllooooooooowwww. Might be OK for low-traffic sites, but better | to implement a custom session data container that stores stuff in | SQL. I have the beginnings of one of these if you want to see it.
Slow for using sessions? Why didn't you use tempstorage then?
I think I thought it wouldn't have mattered. The difference between using sessions against a local filestorage and one on a ZEO server was something like 20X.
Or you are just asserting that FileStorage over ZEO is noticeably slower than directly?
Yes, at least for writes. As I assume tempstorage would be too. And since sessions are all about writes, it's very noticeable. - C
On Wed, May 31, 2006 at 09:59:36AM -0400, Chris McDonough wrote: | | On May 31, 2006, at 9:55 AM, Sidnei da Silva wrote: | | >On Wed, May 31, 2006 at 09:49:49AM -0400, Chris McDonough wrote: | >| I've done this (at least with FileStorage) and it's | >| sllllllooooooooowwww. Might be OK for low-traffic sites, but better | >| to implement a custom session data container that stores stuff in | >| SQL. I have the beginnings of one of these if you want to see it. | > | >Slow for using sessions? Why didn't you use tempstorage then? | | I think I thought it wouldn't have mattered. The difference between | using sessions against a local filestorage and one on a ZEO server | was something like 20X. So, since I couldn't believe the 20X figure, I wrote a very dumb test [1] that shows [2] tempstorage to be only slightly slower than filestorage (roughly 10%) and that adding zeo to the mix makes both of them about 4X slower, which would be pretty acceptable by my standards. I'm looking forward for testing tres' memcached stuff tomorrow. In the meantime, it would be great if someone can run the script on different boxes and platforms (I've ran it on Windows on a Intel Dual Core 3.0) to see if there's any difference. [1] http://awkly.org/files/zeo-bench/bench.py [2] http://awkly.org/files/zeo-bench/bench_results.txt -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
Sorry, I've not run your tests but I'll note that they don't take into account any concurrency or time lost due to conflict error retries. There are many more conflicts when writing to ZEO-backed database in general when there is lots of concurrency because write transactions usually take longer. And even if MVCC bails you out, it still takes time to do the conflict resolution. There is a set of tests explicitly created to torture test session conflict rates here: http://cvs.zope.org/Packages/SessionRig/ See also: http://www.plope.com/Members/dunny/conflicts/view - C On May 31, 2006, at 10:58 PM, Sidnei da Silva wrote:
On Wed, May 31, 2006 at 09:59:36AM -0400, Chris McDonough wrote: | | On May 31, 2006, at 9:55 AM, Sidnei da Silva wrote: | | >On Wed, May 31, 2006 at 09:49:49AM -0400, Chris McDonough wrote: | >| I've done this (at least with FileStorage) and it's | >| sllllllooooooooowwww. Might be OK for low-traffic sites, but better | >| to implement a custom session data container that stores stuff in | >| SQL. I have the beginnings of one of these if you want to see it. | > | >Slow for using sessions? Why didn't you use tempstorage then? | | I think I thought it wouldn't have mattered. The difference between | using sessions against a local filestorage and one on a ZEO server | was something like 20X.
So, since I couldn't believe the 20X figure, I wrote a very dumb test [1] that shows [2] tempstorage to be only slightly slower than filestorage (roughly 10%) and that adding zeo to the mix makes both of them about 4X slower, which would be pretty acceptable by my standards.
I'm looking forward for testing tres' memcached stuff tomorrow.
In the meantime, it would be great if someone can run the script on different boxes and platforms (I've ran it on Windows on a Intel Dual Core 3.0) to see if there's any difference.
[1] http://awkly.org/files/zeo-bench/bench.py [2] http://awkly.org/files/zeo-bench/bench_results.txt
-- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 31 May 2006, at 14:55, Sidnei da Silva wrote:
On Wed, May 31, 2006 at 09:49:49AM -0400, Chris McDonough wrote: | I've done this (at least with FileStorage) and it's | sllllllooooooooowwww. Might be OK for low-traffic sites, but better | to implement a custom session data container that stores stuff in | SQL. I have the beginnings of one of these if you want to see it.
Slow for using sessions? Why didn't you use tempstorage then? Or you are just asserting that FileStorage over ZEO is noticeably slower than directly?
Tempstorage over ZEO becomes a conflict problem when things get busy. We removed it in favor of local temp storages. jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (Darwin) iD8DBQFEfa1uRAx5nvEhZLIRAje/AKCINcg6oR6P8rGQVSKP/Xw98hUMAwCeK9g3 aUcDnH/w9mvtCEBquKyOElU= =ONm3 -----END PGP SIGNATURE-----
Hi, I have bad experence with tempstorage on ZEO (FileStorage with no undo). With site with 18000 users (300 per minutes) , the site hang because of conflict on ZEO. I am very interested by a faster tempstorage like SQLDB Cheers Eric Brun Pentila www.pentila.com Chris McDonough a écrit :
I've done this (at least with FileStorage) and it's sllllllooooooooowwww. Might be OK for low-traffic sites, but better to implement a custom session data container that stores stuff in SQL. I have the beginnings of one of these if you want to see it.
- C
On May 31, 2006, at 9:44 AM, Sidnei da Silva wrote:
On Wed, May 31, 2006 at 09:40:01AM -0400, Chris McDonough wrote: | Should be OK, although if you're doing it because you want a "fast" | storage (or one that "produces fewer conflicts"), ZEO takes away much | hope of that intrinsically.
My goal was actually sharing sessions between zeo clients, without resorting to zsqlsessions or similar.
--Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
_______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Eric Brun wrote:
Hi,
I have bad experence with tempstorage on ZEO (FileStorage with no undo). With site with 18000 users (300 per minutes) , the site hang because of conflict on ZEO. I am very interested by a faster tempstorage like SQLDB
I am working on a memcached-based shared session data container, which should be both faster than ZEO + tempstorage and conflict-free. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFEfdyg+gerLs4ltQ4RAtXIAKCTtlsM3Vz75eGsBmt2HB/e4hdsOACgxQX+ +md4NlLnybXuZFzzWyyI+3I= =xLIQ -----END PGP SIGNATURE-----
On Wed, May 31, 2006 at 02:12:48PM -0400, Tres Seaver wrote: | -----BEGIN PGP SIGNED MESSAGE----- | Hash: SHA1 | | Eric Brun wrote: | > Hi, | > | > I have bad experence with tempstorage on ZEO (FileStorage with no undo). | > With site with 18000 users (300 per minutes) , the site hang because of | > conflict on ZEO. | > I am very interested by a faster tempstorage like SQLDB | | I am working on a memcached-based shared session data container, which | should be both faster than ZEO + tempstorage and conflict-free. Having written a MemcacheCacheManager, I'm very interested in helping with this. How can I help you? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
Sidnei da Silva wrote:
On Wed, May 31, 2006 at 02:12:48PM -0400, Tres Seaver wrote: | -----BEGIN PGP SIGNED MESSAGE----- | Hash: SHA1 | | Eric Brun wrote: | > Hi, | > | > I have bad experence with tempstorage on ZEO (FileStorage with no undo). | > With site with 18000 users (300 per minutes) , the site hang because of | > conflict on ZEO. | > I am very interested by a faster tempstorage like SQLDB | | I am working on a memcached-based shared session data container, which | should be both faster than ZEO + tempstorage and conflict-free.
Having written a MemcacheCacheManager, I'm very interested in helping with this. How can I help you?
Not sure if you know it, but I guess Tres is referring to http://www.danga.com/memcached/ I was also ofthen thinking if something like that would make a good client cache - w/o blowing up pythons heap so much. Regards Tino
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Sidnei da Silva wrote:
On Wed, May 31, 2006 at 02:12:48PM -0400, Tres Seaver wrote: | -----BEGIN PGP SIGNED MESSAGE----- | Hash: SHA1 | | Eric Brun wrote: | > Hi, | > | > I have bad experence with tempstorage on ZEO (FileStorage with no undo). | > With site with 18000 users (300 per minutes) , the site hang because of | > conflict on ZEO. | > I am very interested by a faster tempstorage like SQLDB | | I am working on a memcached-based shared session data container, which | should be both faster than ZEO + tempstorage and conflict-free.
Having written a MemcacheCacheManager, I'm very interested in helping with this. How can I help you?
I haven't tested it under heavy load yet, but here is my first pass: it registers two ZMI-addable objects: - a 'MemCache Proxy', which knows the set of servers making up the distributed cache (these may be either straight memcached servers or, alternatively, disk-based tugela servers). - a 'MemCache Session Data Container', which uses the proxy to implement the same interface as the TransientObjectContainer. I plan to bang on it a little more, and release it sometime on or before the 12th (I'm giving a talk on it at the ZPUG meeting then). Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFEfgeJ+gerLs4ltQ4RAi2HAKCLvrdVm/Teqg/89VgP3lW+Tab/XwCdHgPV zrgHfoFO8WaNnVuYswl5EjY= =y9Rm -----END PGP SIGNATURE-----
participants (6)
-
Chris McDonough -
Eric Brun -
Jens Vagelpohl -
Sidnei da Silva -
Tino Wildenhain -
Tres Seaver