Hi there Zopists My apoligies for the rather long story which follows, but I think it very interesting for anybody developing serious websites: thats you guys ! The analysis is about which database to use for websites. The author seems to be familiar with about all the commercial en open source databases on the market and finally found KDB. The story is taken from the KDB-mailing list. Very short: KDB is a very small, very efficient, very fast database (because optimal use of ram) written in the language K and has (among other options) a http-based XML interface for easy connecting. Gijs Reulen Date sent: Fri, 5 May 2000 04:59:14 -0700 (PDT) From: Andrei Moutchkine <muchandr@CSUA.Berkeley.EDU> To: kdb@listbox.com Copies to: Andrei Moutchkine <muchandr@CSUA.Berkeley.EDU> Subject: Re: Dot-coms reignite database software industry Send reply to: kdb@listbox.com
The funny thing is how the 'enterprise' databases are completely ill-suited for the purposes of backing a Web site. The Web just rubs them entirely the wrong way. All of a sudden people don't worry about performance of ad hoc queries anymore because they don't have any but worry a lot about connection times. You typical http connection doesn't live long enough for those monsters to scratch their noses. The Web guys would to have their two and a half queries pre-optimized once and for all times and get the fancy dynamic query optimizer the DBMS vendors spent so much effort developing out of the way. I would guess that most DB-backed Web sites out there are actually backed by MySQL, which is outright barebones feature-wise. Last time I tried it there was no support for transactions, no subselects (has been added meanwhile I think) and no query optimization to speak of (you got your performance by generous application of explicit left joins). The only technical merit of MySQL appears to be the fact it goes very very fast on trivial queries. I am not sure there is an understanding with major DBMS vendors that most of all the new users want the trivial bugs fixed. Take Oracle for example. The world's second largest independant software company is busy producing piles of middleware nobody needs and mountains of manuals nobody reads. (Looking into my Oracle box. Application server? WebDB? Developer server? E-commerce server? Yepee, I've been spared their interactive TV server and network computer server) At the same time, the core engine appears outright neglected. (I suspect lurking code written by long gone interns that nobody in the company understands anymore. It would be very hard to lose track of code ownership in 200K, so the smallness of K code is a big advantage here) Still, I have no explanation why Oracle didn't do jack all those years about text support for example:
You still have to set aside a fixed length for varchars. There is also some arbitrary (4K or something) absolute upper limit on the legth of varchars, after which you have to resort to SQL-unfriendly ways of dealing with text. This is a serious problem for people storing whole documents, like Web pages. How do others do? (I know Informix does allow properly arbitrary length text) How hard was it to add a regexp library, ideally for use inside the 'LIKE' SQL clause? (I am not sure how close KDB 'LIKE' comes to fully supporting regexp matching, but a casual look reveals something more powerful than pathetic pattern matching of standard SQL, which is all you are going to get from Oracle) There are probably a few Web sites that are slow because its designers chose to move too much data to webserver for the sole purpose of applying Perl's (exemplary) regexps to selection (I am responsible for one such site :( )
The non-speciality of KDB varbinary datatype could be very interesting to Java folks storing serialized objects. (Typically, if you want to do more with a BLOB than just storing/retrieving you are either out of luck or you have to write your own cartridge/datablade server extension)
I was so anxious to check out Oracle's new object extensions and I am very disappointed. (In fact, this is how I got into KDB) Specifically, the timeseries cartridge is useless. The best timestamp resolution is one second and there is no way to compensate, because you can't add your own functions (plain vanilla ones for a single row, not even mentioning custom aggregates) I think they just released something for the time being to stop Informix from mocking them. (It seems to have worked. The infamous 'dinosaur' billboard is off 101, isn't it?) Even Postgres, the granddaddy of object-relational databases, comes out ahead of Oracle on object features in my book, despite being maintained by some guy from Siberia in his spare time these days.
I think the Web guys should be pretty receptive to the idea of a memory database. Couple of startups even got funded to develop one some time ago. If you want a working example, there is Yahoo. Every server of theirs has the entire static directory mapped into RAM and a hacked version of Apache reads it via shared memory.
Overall, KDB appears very well positioned to take over as THE database to put behind Web sites. (dunno if I am ready to throw out my trusty Apache webserver just yet though :) Maybe this is because I've never seen a nicer database overall. Pretty good for something that appears to be just a parser from ueber-SQL to K :) Does it do anything else? (say the clever transaction reordering scheme from SIGMOD paper posted on kx' site...) Or is it just a fiendish plot to trick unsuspecting souls like myself into learning some K? :) Great idea. I am glad I am learning K now, but I would've never downloaded the standalone K product when I was looking around for a DB with good timeseries support. Its just too unlike anything people came to associate with 'database'. If I have to mention just one thing from what I've learned so far, it would be that it appears to be that K gives you the main advantage of object databases (application-specific arbitrarily shaped datatypes) without the main disadvantages. The official reason nobody uses object databases is supposedly because you have to write very low-level methods to access your object storage and than they end up being very SQL-unfriendly and than all your code breaks if you change your data representation. Appears to be no big deal in K.
To get back on topic, the definitive book on database-backed Web sites can be found online at http://photo.net/wtr/thebook/ Highly recommended, even though I don't always agree with Phil. Its the only database book I am aware of that's actually funny. So far I failed to convince the author to give KDB a spin, unfortunately. He says similar performance, etc. claims have been made for other products in the past, but have always been traced to compromises made in either transaction support or support for concurrency. You go try. (the online book accepts freeform input) Phil's hype machine can make or break a product, easily.
I apologise for my un-APL-like verbosity.
-Andrei
On Thu, 4 May 2000, arthur whitney wrote:
yes
--- The KDB list is managed by majordomo@listbox.com Send a message there, containing the word 'help', for assistance, e.g. unsubscribing.