SQLSession vs FSSession for ultra-high scalability and speed.
Hi Folks, Sorry if this has been asked before, but can anybody advise on FSSession vs SQLSession for: a) Speed. b) Scalability. Planned architecture employs multiple Zope/Apache frontline servers: +-----+ | Z/A |---+ +-----+ | | +-----+ | | Z/A |---+ +----- DB1 +-----+ | | |------| +-----+ | | | Z/A |---+ +----- DB2 (optional) +-----+ | | | Etc, etc. Z/A = 'front line' webservers running Zope/Apache. Contains application/business logic. We are NOT using the ZODB for data storage. DB1 = Data storage. eg. Oracle/Postgres DB2 = potentially just for managing the sessions, if we use SQLSession. Would MySQL be best ? Just how scalable can it really get ? We need it to at least be able to scale to 12 million sessions per day (and substant- -ially more writes to DB1 per day), even if we will be starting out much more modestly than that. Realistically, should I be looking at something like ATG Dynamo instead ? Thank you very much, chas
Hi Chas, Let me give it a try (or just dump all of what I know). First of all it looks like you are going to want ZEO. If you don't know ZEO, get to know ZEO, it fits perfectly with what you want for the frontline servers. As for the SQLSessions. Since there is a lot of writing in SQLSessions I would suggest that you stick to the RDBMS model, Zope can handle it, but SQL can handle it better and if you are already planning on putting a DB behind the Zope installation it will really not take that much more (if anything) to add session information. You might, however, depending on time, want to invest a little in playing around with your own SQL code to handle session data. There are factors like Zope's ability to cache that you will want to take full advantage of. As for the database, MySQL is hands down the fastest, but certainly not the most robust in terms of features. Oracle is great if you have $10,000 and a full time DBA but PostgreSQL 7.0 is really your best bet. You might also want to take the money you were going to spend on the DB2 and put it right back into DB1 and house everything there. Just get a beefy machine, RAID it up for redundancy and call it a day. Take a look at some of the benchmarks I ran for an idea of what you can do with a plain ole RH 6.1 box and PostgreSQL. http://www.zope.org/Members/BwanaZulia/zope_benchmarks/benchmarks.html The other idea that I have been playing around with in terms of redundancy is to build something into the Zope code for the ZSQL Methods that can go into a "cached" mode if your database dies. I have only begun to play with this, but in theory since, Zope can cache all your common data that comes from SQL you could, in effect, bring down the SQL database and have Zope continue to work with the cached data. I think all that it would involve (and I am sure that I am simplifying the hell out of this) is to have Zope know that the SQL database is no longer there (look for an error) and just turn the "Maximum time (seconds) to cache results" to unlimited until the database was restored. Has anyone else thought about this or played with this idea? Ok.. that is enough... Cheers, J
Sorry if this has been asked before, but can anybody advise on FSSession vs SQLSession for:
a) Speed. b) Scalability.
Planned architecture employs multiple Zope/Apache frontline servers:
+-----+ | Z/A |---+ +-----+ | | +-----+ | | Z/A |---+ +----- DB1 +-----+ | | |------| +-----+ | | | Z/A |---+ +----- DB2 (optional) +-----+ | | | Etc, etc.
Z/A = 'front line' webservers running Zope/Apache. Contains application/business logic. We are NOT using the ZODB for data storage.
DB1 = Data storage. eg. Oracle/Postgres
DB2 = potentially just for managing the sessions, if we use SQLSession. Would MySQL be best ?
Just how scalable can it really get ? We need it to at least be able to scale to 12 million sessions per day (and substant- -ially more writes to DB1 per day), even if we will be starting out much more modestly than that. Realistically, should I be looking at something like ATG Dynamo instead ?
Thank you very much,
chas
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
In article <p04310103b54cde3dbe12@[209.27.56.93]>, J. Atwood <jatwood@bwanazulia.com> writes
As for the database, MySQL is hands down the fastest, but certainly not the most robust in terms of features. Oracle is great if you have $10,000 and a full time DBA but PostgreSQL 7.0 is really your best bet. You might also want to take the money you were going to spend on the DB2 and put it right back into DB1 and house everything there. Just get a beefy machine, RAID it up for redundancy and call it a day.
It may be the way I designed my site, but if the database serving SQLSessions had to be taken down for maintenance, my site became inaccessible. Based upon this experience, I would therefore now prefer to use different database engines for the main database and for the session tables. And since for sessions, all you really want is speed without any fancy SQL transactions, then MySQL for the session data seems an appropriate use. Session tables wouldn't have to be too big as you could run ZScheduler to remove expired sessions. -- Regards, Graham Chiu gchiu<at>compkarori.co.nz http://www.compkarori.co.nz/index.php Powered by Interbase and Zope.
Graham Chiu wrote:
In article <p04310103b54cde3dbe12@[209.27.56.93]>, J. Atwood <jatwood@bwanazulia.com> writes
As for the database, MySQL is hands down the fastest, but certainly not the most robust in terms of features. Oracle is great if you have $10,000 and a full time DBA but PostgreSQL 7.0 is really your best bet. You might also want to take the money you were going to spend on the DB2 and put it right back into DB1 and house everything there. Just get a beefy machine, RAID it up for redundancy and call it a day.
<deletia>
And since for sessions, all you really want is speed without any fancy SQL transactions, then MySQL for the session data seems an appropriate use.
I seem to recall from the MySQL vs Postgres thread a while ago that MySQL was fastest in mostly-reading situations, whereas Postgres was far better when there is quite a lot of writing as well as reading. If this is true, the Postgres should be a better choice for a session-data DBMS. Of course, the *right* way to find out is to run a benchmark :-) -- Steve Alexander Software Engineer Cat-Box limited
In article <39279A2F.BA5AED6D@cat-box.net>, Steve Alexander <steve@cat- box.net> writes
And since for sessions, all you really want is speed without any fancy SQL transactions, then MySQL for the session data seems an appropriate use.
I seem to recall from the MySQL vs Postgres thread a while ago that MySQL was fastest in mostly-reading situations, whereas Postgres was far better when there is quite a lot of writing as well as reading.
If this is true, the Postgres should be a better choice for a session-data DBMS.
For SQLSessions, you only write the session data once, and read multiple times. So, that would suit MySQL ? -- Regards, Graham Chiu gchiu<at>compkarori.co.nz http://www.compkarori.co.nz/index.php Powered by Interbase and Zope
Thank you very much for the comprehensive and speedy reply, J.
Let me give it a try (or just dump all of what I know).
Much more than I expected - and greatly appreciated. Thank you.
First of all it looks like you are going to want ZEO. If you don't know ZEO, get to know ZEO, it fits perfectly with what you want for the frontline servers.
Wow, has it been released for public consumption yet ? (I've been out of the Zope-loop for a while unfortunately) But yes, I had ZEO in mind for keeping the application logic on the frontline servers in sync.
As for the database, MySQL is hands down the fastest, but certainly not the most robust in terms of features. Oracle is great if you have $10,000 and a full time DBA but PostgreSQL 7.0 is really your best bet. You might also want to take the money you were going to spend on the DB2 and put it right back into DB1 and house everything there. Just get a beefy machine, RAID it up for redundancy and call it a day.
Will definitely be getting RAID for the application data.
The other idea that I have been playing around with in terms of redundancy is to build something into the Zope code for the ZSQL Methods that can go into a "cached" mode if your database dies. I have only begun to play with this, but in theory since, Zope can cache all your common data that comes from SQL you could, in effect, bring down the SQL database and have Zope continue to work with the cached data. I think all that it would involve (and I am sure that I am simplifying the hell out of this) is to have Zope know that the SQL database is no longer there (look for an error) and just turn the "Maximum time (seconds) to cache results" to unlimited until the database was restored.
Has anyone else thought about this or played with this idea?
I think in most cases, if the DB dies, it's game over until it goes up again.
Graham Chiu wrote: It may be the way I designed my site, but if the database serving SQLSessions had to be taken down for maintenance, my site became inaccessible.
Same would happen to my site. Rather than try to build around that, I think I'll try to keep the DB up instead - since I won't have much functionality without sessions.
Based upon this experience, I would therefore now prefer to use different database engines for the main database and for the session tables. And since for sessions, all you really want is speed without any fancy SQL transactions, then MySQL for the session data seems an appropriate use.
Exactly what I was hoping - thank you for the confirmation :)
Graham Chiu wrote:
Steve Alexander wrote: I seem to recall from the MySQL vs Postgres thread a while ago that MySQL was fastest in mostly-reading situations, whereas Postgres was far better when there is quite a lot of writing as well as reading.
Interesting. I would have thought that MySQL won in writes b/c it doesn't have the transaction overhead. Then again, Postgres 7 seems to have updated the ante a lot.
If this is true, the Postgres should be a better choice for a session-data DBMS.
For SQLSessions, you only write the session data once, and read multiple times. So, that would suit MySQL ?
I guess it depends what you're doing with your session object. We'll be changing the properties of the session object as the user traverses the site, so the session object will have to be updated (ie. updated on the RDB). Thank you all for your help. chas
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 In article <3.0.5.32.20000522041134.00af6c50@mail.skinnyhippo.com>, chas <panda@skinnyhippo.com> writes
Graham Chiu wrote: It may be the way I designed my site, but if the database serving SQLSessions had to be taken down for maintenance, my site became inaccessible.
Same would happen to my site. Rather than try to build around that, I think I'll try to keep the DB up instead - since I won't have much functionality without sessions.
What I meant was that it should be easy to plug in another database engine for the session serving since there should be nothing database specific about the interface to SQLSession. ( As it happens, there is :-( ) So, taking down the database for Session management should not take down the site. - -- Regards, Graham Chiu gchiu<at>compkarori.co.nz http://www.compkarori.co.nz/index.php Powered by Interbase and Zope -----BEGIN PGP SIGNATURE----- Version: PGPsdk version 1.7.1 iQA/AwUBOSew+7TRdIWzaLpMEQI03gCfU+VlPOn/x3xXkcJaN8Pc0fVtlH0An1YK 2pdxpkp1E9Fy5aDPUdP2qp20 =8kZ/ -----END PGP SIGNATURE-----
On Sun, 21 May 2000, chas wrote:
Hi Folks,
Sorry if this has been asked before, but can anybody advise on FSSession vs SQLSession for:
a) Speed. b) Scalability.
FSSession does not use ZODB to store data, but stores session pickles directly on the harddisk. It will also update session info only if there are modifications to the session object. Given OS file caching I would not be supprised if FSSession is faster than any RDBMs based solution. Also note that if you have to support many writes on the Session objects, a filesystem provides a nice map from users --> files, which then utilizes the 'high concurrency' provided by the FS. OTO an RDBMS maybe a more reliable datastore than the filesystem with better consistency, recovery tools etc etc. Also check whether SQLSession caches info. It used to hit the RDBMs for every variable access which was very prohibitive for us, but Anthony mentioned that he was changing that. Since performance is going to be a major issue in your design I suggest you run a small bechmark to test relative performance. Pavlos
participants (5)
-
chas -
Graham Chiu -
J. Atwood -
Pavlos Christoforou -
Steve Alexander