[Zope] Re: Questions about demostorage

[Duncan Booth]

Chris Withers <chris at simplistix.co.uk> wrote:
>> (So in other words, the 
>> connection from the demostorage front-end to the zeo-client isn't
>> frozen at the point when the storage is opened which might have been
>> a plausible alternative).
> 
> Urm, that would involve taking a complete copy of the db at open
> time... 

Not knowing the details of the implementation I don't know that it would. 
I'd have thought it would involve copying the index rather than the data, 
and since I don't know the implementation of the index it might only need 
to do a copy-on-write as and when parts of the index change.

> But really, what you're doing is not what DemoStorage was 
> designed for and will likely end in tears or a lot of hair pulling...

I know, that's why I'm trying to establish the boundaries of what I can 
expect from it.

>> What I'm hoping to achieve is a way of testing upgrades on our live
>> system in an environment as close as possible to the real live
>> environment, but without actually affecting the live system in any
>> way. Demostorage seems to offer that as an option.
> 
> It really doesn't ;-) Do this kinda testing the same way that everyone
> does: build yourself a uat environment that mirrors your production 
> environment, maybe with a few less zeo clients, run it off a copy of 
> your production zodb and then test there...

Got that, but the problem is that the test database isn't exactly identical 
to the live one (e.g. the ldap config points to the test ldap server) but 
I'm sure we could script the tweaks needed. More significantly is the pain 
involved in copying 10Gb from the live system onto dev on more than an 
occasional basis.

The problem I'm trying to address is that sometimes we make mistakes when 
updating the live server because of minor differences between the test and 
live environments. So on the live servers we have a separate zope instance 
for final testing. It has its own copy of the Products folder (checked out 
from the relevant svn tag) and when we are happy that the checkout works as 
expected I just rsync that Products folder onto the live Products.

I'm not suggesting running the system on demostorage for long periods of 
time. Simply long enough to check that whatever weird buildout changes we 
make are doing are going to work. That means that the only guarantee I 
really need is that the live system won't be affected by any changes to the 
test system; if the test system becomes internally inconsistent and falls 
over I'll just restart it and try again.

For example, we are currently running with an ancient svn checkout of 
CacheFu. I know, from playing on the dev system, that migrating from that 
to [a slightly patched copy of] the current version is possible, but it 
took some time and caused site errors until it was complete. I have a 
script which I believe should do the migration quickly but I want to be 
absolutely positive it is going to work without any hitches. If I can do a 
dry run in the live environment then I should be able to get that 
confidence.