[ZODB-Dev] Wrong blob file being returned (similar to https://mail.zope.org/pipermail/zodb-dev/2011-February/014067.html )

[Jim Fulton]

On Tue, Jul 12, 2011 at 6:33 AM, steve <steve at lonetwin.net> wrote:
> Hi,
>
> I have a setup where 4 ZEO clients running on separate machines connect to a
> single DB server which runs on a different system by itself. The ZEO clients and
> the DB server all are at version ZODB3-3.10.2. Now, since the last few weeks
> some of our users have been reporting that they occasionally see incorrect
> images being returned.
>
> On googling I came across the thread below and was wondering whether I am seeing
> the same thing as this:
>
> https://mail.zope.org/pipermail/zodb-dev/2011-February/014067.htm
>
> ...although the setup and version is different (ie: ZODB-3.8 and
RelStorage).

Yeah, quite a bit different.

>
> Unfortunately, sorry but I do not know enough about ZODB internals to be able to
> say for sure. Is there a way I can test whether the problem is indeed with the
> wrong blob file being returned from the blobcache ? FWIW, we haven't figured out
> a way to consistently reproduce this error ourselves.

Dang.

> Other things that
> may/may-not be relevant:
>
> a. our site is image heavy (36293 blob files) and the servers are behind a load
> balancer so in a single request to the web-app (a repoze.bfg site) we might even
> load collectively 20+ blobs from any of the 4 servers.
>
> b. zeo connection string on the clients
> zodb_uri =
> zeo://xxx.xxxx.xxx.xxx:8886/?blob_dir=%(here)s/../var/blobs&shared_blob_dir=false&connection_pool_size=50&cache_size=1024MB&drop_cache_rather_verify=true
>
> c. $ cat var/blobs/.layout
> zeocache
>
> Any comments/suggestion on how to isolate and fix this problem would be appreciated.

We have a number of large apps with multiple terabytes of blobs and a
vaguely similar configuration. We haven't seen this sort of problem.
One difference is that we set the blob cache size.  I don't suppose
you're running of disk space?

The only suggestion I have is to keep an eye on it and try to
reporoduce the problem. I would think that if a request returns an
incorrect Blob, it would continue to. If someone reports a bad blob,
get the URL and see if you can reproduce by making the same request to
each of the app servers, bypassing the load balencer.  If one server
is being bad, you can remove it from the LB pool to debug it.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton