Try to partition your collision space. Do you mount a shared temp_folder? Don't! Do as Chris said and keep a temp_folder per ZEO client. Let the load balancer send people to the same ZEO client every time (according to the cookie). Note that there is no solution for bad coding practice other than to fix it ;-). Keep session access to an absolute minimum (read *and* write). You may also want to look at SQLSessions, which seem to be able to take a higher load (haven't tried for a while). But only up to a point until you get concurrency issues in the RDBMS. Psycopg in particular defers conflicts to Zope by again raising ConflictErrors... In the end, the key to scaling is to keep state out of the app server. http://www.loudthinking.com/arc/000479.html http://www.danga.com/words/2004_oscon/oscon2004.pdf Stefan On Tue, 15 Nov 2005, Dennis Allison wrote:

Peter,

That's pretty much what we've done, but it is not really enough in our case. Our sites are interactive with lots of per user state. We make fairly heavy use of session variables to track the state. Our number of simultaneous users is also fairly high. In addition, during development, we structured things for ease of development rather than for mimimum liklihood of conflict errors. At this point we are seeing many conflicts and interactions with both the sessioning mechanism and the persistence mechanism--or so it appears.

Even when we resolve the current database interaction problem, a refactoring seems in order to ensure the number of conflicts is kept to a minimum.

Chris, I am aware that using ZEO to back session database is likely to increase the opportunity for conflicts, but using a single session database seems to be reaquired if you want, as we do, to distribute out interactive application acrosss a cluster of processors cleanly. Under our current approach, we have multiple Zopes running on multiple machines, all referencing a shared database of session variables. This allows a HTTP request to be blindly distributed to any of the machines. Given the conflict problems, we may need to rethink this simplistic architecture. On Tue, 15 Nov 2005, Chris McDonough wrote:

On Nov 15, 2005, at 5:54 PM, Dennis Allison wrote:

...

Has anyone prepared a set of best practice guidelines on the techniques to use to minimize conflicts?

It is becoming clear that we need to do to refactor some of our code to get us out of our current conflict pickle.

A quick google produced lots of commentary but nothing I could hand to our programmers as guidelines.

Nothing in general except the (probably too general) "attempt not to allow one process/thread to write to the same object at the same time as another process/thread".

One tip, since you've told us you use sessions heavily: using ZEO to back your session database provides more opportunity for conflicts than if you used a "normal" Zope storage. This is because:

- the opportunity for conflict naturally grows as transaction time increases and using a ZEO storage rather than a local storage adds latency to the total transaction time that would not otherwise exist.

- you have "n" processes attempting to write to essentially the same objects at the same time when you use a shared ZEO sessioning database. If these were instead local databases, each process would only compete "with itself" (between all of its threads).

I tend to try to use local sessioning databases backed by tempstorage plus a frontend load balancer that allows for cookie-based session affinity (Pound).

- C

--

Changing the architecture will likely get you the most bang but note also that there are a few knobs that you can turn on the transient object container UI that may limit the number of conflicts (in the temp_folder/session_data_container): - "Data object timeout value" -- the timeout value in minutes.. make this higher. - "Timeout resolution" -- the timeout resolution in seconds.. make this higher. You can also turn off "inband housekeeping" of session data by calling the method "disableInbandHousekeeping" on a transient object container. At that point, sessions will continue to work properly but "old" session data won't be garbage collected and you will need to do this every so often by calling the "housekeep" method on TOC. See this written by dunny about conflicts and sessions (although this was written before MVCC): http://www.plope.com/Members/dunny/conflicts/view HTH, - C On Nov 16, 2005, at 6:18 PM, Dennis Allison wrote:

Chris,

I am aware that using ZEO to back session database is likely to increase the opportunity for conflicts, but using a single session database seems to be reaquired if you want, as we do, to distribute out interactive application acrosss a cluster of processors cleanly.

Under our current approach, we have multiple Zopes running on multiple machines, all referencing a shared database of session variables. This allows a HTTP request to be blindly distributed to any of the machines. Given the conflict problems, we may need to rethink this simplistic architecture.

On Tue, 15 Nov 2005, Chris McDonough wrote:

...

On Nov 15, 2005, at 5:54 PM, Dennis Allison wrote:

...

Has anyone prepared a set of best practice guidelines on the techniques to use to minimize conflicts?

It is becoming clear that we need to do to refactor some of our code to get us out of our current conflict pickle.

A quick google produced lots of commentary but nothing I could hand to our programmers as guidelines.

Nothing in general except the (probably too general) "attempt not to allow one process/thread to write to the same object at the same time as another process/thread".

One tip, since you've told us you use sessions heavily: using ZEO to back your session database provides more opportunity for conflicts than if you used a "normal" Zope storage. This is because:

- the opportunity for conflict naturally grows as transaction time increases and using a ZEO storage rather than a local storage adds latency to the total transaction time that would not otherwise exist.

- you have "n" processes attempting to write to essentially the same objects at the same time when you use a shared ZEO sessioning database. If these were instead local databases, each process would only compete "with itself" (between all of its threads).

I tend to try to use local sessioning databases backed by tempstorage plus a frontend load balancer that allows for cookie-based session affinity (Pound).

- C

--

On Nov 17, 2005, at 4:46 AM, Dennis Allison wrote:

Thank Chris, good pointsB.

The session variable timeout is currently several hours as requird by our application. The timeout resolution remains at its default value.

Making the resolution higher has been observed to reduce conflicts (see dunny's chart).

I have not dorked with disabling the inband housekeeping. I don't see how it would impact on our problem. Am I missing something?

Turning off inband housekeeping has the potential to reduce the numnber of conflicts because less work is done during normal sessioning operations. Instead of doing the work to figure out whether it needs to clean up after itself during "normal" operations it relies on an external process.

I suppose I could do a custom p_resolveConflict since the vast majority of conflicts are are non-conflicting and actual conflicts are fairly easy to identify.

I couldn't know one way or another, but if you believe so...

Still, on the whole, I think we have to refactor our programs to minimize the opportunities for conflicts as well as tuning the mechanism. As a temporary workaround, I'm backing out of the shared temporary storage in favor of a local temp storage. That should make things a bit better.

Good idea... - C

On Wed, 16 Nov 2005, Chris McDonough wrote:

...

Changing the architecture will likely get you the most bang but note also that there are a few knobs that you can turn on the transient object container UI that may limit the number of conflicts (in the temp_folder/session_data_container):

- "Data object timeout value" -- the timeout value in minutes.. make this higher.

- "Timeout resolution" -- the timeout resolution in seconds.. make this higher.

You can also turn off "inband housekeeping" of session data by calling the method "disableInbandHousekeeping" on a transient object container. At that point, sessions will continue to work properly but "old" session data won't be garbage collected and you will need to do this every so often by calling the "housekeep" method on TOC.

See this written by dunny about conflicts and sessions (although this was written before MVCC):

http://www.plope.com/Members/dunny/conflicts/view

HTH,

- C