[Grok-dev] Re: horizontal scaling

[Martijn Faassen]

Hey,

Christopher Mulcahy wrote:
[snip discussion of horizontal scaling]
> Zope3 is very large and I haven't been able to get the 30,000 foot view 
> yet.
> Most simply I suppose:
> Is it possible for 1 zope3 application server ( or a family of zope3 
> application servers sharing a ZEO or relstorage-like back-end ) to 
> provide multiple Grok application instances for many ( maybe not 200, 
> but 30 or 40 ) different virtual hosts ( fronted by squid, apache et.al ).

Good question. I don't have experience with this amount of scaling. I 
think the best chance to get people who do have such experience (if they 
exist) is to ask this question on zope-dev as well. But there are some 
of those people hanging out on grok-dev, so let's see whether someone 
else replies. Christian Theune (who will be at the grokkerdam sprint 
tomorrow) may be able to give interesting feedback, for instance.

> Can an application that has not been accessed in hours ( or days ) be 
> "passivated" and its runtime resource costs be reduced to disk-space and 
> a tiny entry in an in-memory hashtable?

ZODB (and ZEO) do keep a cache around of objects that have been 
accessed, so I don't that this will happen on a Zope level. That said, 
the operating system may swap out the application if it isn't used a 
lot, but that's probably not what you ask for. I don't know what the 
memory characteristics of RelStorage are - zope-dev or zodb-dev should 
get you better answers.

There are ways emerging to run Zope without the ZODB at all (or just 
with a very tiny ZODB), and store the data in a relational database, for 
instance using Storm or SQLAlchemy. This may have better memory 
characteristics. It's likely a typical relational object mapper still 
has an object cache around, but it may be smaller, or at least it may be 
easier to make this smaller. I imagine a typical PHP application would 
still have a smaller amounts of objects around, as no relational object 
mapper is used at all, and PHP also tends to have a "run the page and 
then forget about whatever happened" nature.

> I know that zope3  ( and thereby Grok ) is a lot heavier than my own 
> framework, but it brings a lot more to the table.
> I just can't afford to run one for every site ( large or small ) that 
> needs dynamic content.
> I have a vision of  one redundant zope3-cluster ( backed by relstorage 
> on a slony-replicated postgresql cluster ) that can provide all of the 
> web-application infrastructure for all of my development work for 
> various disparate customers going forward.
> If this is feasible?

It's possible to run multiple applications in a single Zope instance. 
There are drawbacks to this - if you upgrade Grok to a newer version (or 
just a library), you would need to update all your underlying code, not 
just for one application. Also a restart will be interesting - you could 
restart each front-end separately until all are using the newer version 
of your code.

I think it would also be possible to run different ZEO or RelStorage 
frontends that share the same database backend. This way you could do 
incremental updates while sharing the same database backend. I think 
this would need some special approach though, as even though you'd use 
one big database in the back, an individual frontend server would not be 
able to access all the objects that are in that ZODB, only the ones that 
make sense in the context of its application. I don't think there's 
anything fundamentally stopping you from doing so, but I also suspect 
there are few tools to support you there.

> If this were possible it would be a slam-dunk for the adoption of 
> grok/zope3 over rails, turbogears, django, et.al.

Unfortunately I doubt that there's an easy "yes" or "no" to this answer. 
   It also would need to be measured whether such a setup would have 
reliability and memory advantages in the real world.

The question would be most easily answered if someone already has done 
things like this. I'm not really the right person to answer, but I still 
wanted to provide you some feedback. :) zope-dev again would be a good 
place to ask.

Thanks for your questions - it's interesting to see this kind of 
question coming in; gives us new perspectives and news ideas, even if, 
sadly, we may not end up being able to do what you're asking for.

Regards,

Martijn