September 1999 - Zope

Re: [Zope] Zope in Portals
by Andy Dustman 28 Sep '99

28 Sep '99

How about this: Obviously there are some applications which need transactions (ACID properties) and so the DA they use must support transactions. Likewise, there are other applications which don't need a transactional database. So: Is it feasible to add a method to the DA class which succeeds if the database supports transactions fails/raises an exception if it does not? I think the ZSQL Method would then need an extra property: transactional (boolean). If a ZSQL Method is transactional, then it won't accept a database connection which is not transactional. Alternatively, simply distribute the DA as-is, make it very clear that the underlying database does not support transactions, and if you use it anyway and something needs to be rolled back, you're screwed, and you were warned that this sort of thing could happen... -- andy dustman | programmer/analyst | comstar.net, inc. telephone: 770.485.6025 / 706.549.7689 | icq: 32922760 | pgp: 0xc72f3f1d

1 0

RE: [Zope] Zope in Portals
by Paul Everitt 28 Sep '99

28 Sep '99

Andy wrote: > Actually, the patch I have for ZMySQLDA to make it use MySQLdb/_mysql > instead of MySQLmodule-1.4 makes it fully thread-safe. I've flogged > MySQLdb/_mysql with a test script and typically use 3-5 threads and it > works fine, no leaks, no blocking. This really is wonderful news! We've been hoping someone with a lot of MySQL experience would do some magic for Zope's DA support. It's been a shame that people have to run Zope2 with one application thread. Thus, thanks! --Paul

1 0

Re: [Zope] FreeBSD : anyone else noticed kernel "table full" errors whenZope is installed ? Zope is installed ?
by chas 28 Sep '99

28 Sep '99

At 03:36 PM 9/27/99 +0200, Olivier Deckmyn wrote: >I've had exactly the same pb... >(I use FreeBSD 3.1) >The only solution I found was to increase MAXUSERS... Thanks Olivier. At least I'm not imagining things. Does this hint at scalability problems if I'm suffering this problem with zero load yet didn't suffer from this on a CGI-based web server with very heavy load ? Or is this (as I hope) just an initial cost of running PCGI/Zope that won't be extrapolated with load ? At 01:42 PM 9/28/99 -0400, Thaxter, Jason M. wrote: > >I have had the same setup without having this problem. Python is something >of a cpu hog, and I have managed (during load testing) to overload it once >or twice (kernel message: Python exited on signal 11) causing a need to >restart Zope. I've had a couple of "Python exit on signal 11" problems too - again on a development machine (ie. next to zero load) but assumed that's because I was doing something spazzy with a 500,000-record DB. >Once I even bounced the box. But once I started running Zope >with more threads (8), I was no longer able to kill anything, even when it >got swamped. Ah, ok - maybe I'll try that though I'm almost certain that that will cause more of the first problems (above). >Could you have some issue with disk space or inodes on one of your relevant >filesystems? I don't think so. It's a very well endowed machine <g> and was working completely fine on a diet of pure 100% CGI. chas >----- Message d'origine ----- >De : chas <panda(a)skinnyhippo.com> >À : <zope(a)zope.org> >Envoyé : mardi 28 septembre 1999 11:17 >Objet : [Zope] FreeBSD : anyone else noticed kernel "table full" errors >whenZope is installed ? Zope is installed ? > > >> I have 4 almost identical servers : >> - FreeBSD 3.2 >> - MySQL >> - Apache 1.3.6 >> >> All were running various websites using CGI scripts. >> All were stable and have more than sufficient resources. >> >> I've since installed Zope on 3 of them and all 3 in question >> have subsequently started throwing up the following >> kernel log messages as soon as Zope is installed (running >> on the backend of Apache) : >> >> > file: table is full >> > file: table is full >> > file: table is full >> > file: table is full >> > file: table is full >> >> This is easily resolved by increasing the MAXUSERS in the >> kernel but what worries me is that I should even need to >> do this. One of the boxes was handling 2 million hits >> per day last month without creating these kernel errors. >> >> I am tempted to assume that this is all due to Zope because >> the correlation is so strong - run Zope and these errors occur, >> don't run zope and don't get any. Even for one of the servers >> that is still offline with zero traffic. >> >> Has anyone else experienced this on FreeBSD ? >> Any reason why this might be Zope-induced ? >> >> cheers, >> >> chas

1 0

RE: [Zope] Uppercasing a string in a ZSQL method?
by Jeffrey Shell 28 Sep '99

28 Sep '99

> <dtml-let sNewAcctNo="string.upper(sNewAcctNo)"> (doesn't work; > exceptions.NameError) Tried _.string.upper() ?

1 0

Re: [Zope] Zope in Portals
by Andy Dustman 28 Sep '99

28 Sep '99

On Tue, 28 Sep 1999, Christopher Petrilli wrote: > > Incidentally, the "+100 users comment" raises the question: Will Zope > > allow the same database connection to be used simultaneously in more than > > one thread? > > Each thread can have a connection to each database that is accessed, but > connections are not shared between threads in the way you are hypothesizing. > We are talking of Level 3 DAs, those that are fully thread-safe, which the > MySQL DA is not. It must be run in a single thread. With Oracle for > example, if you have 5 application threads, you could theoretically have 5 > Oracle connections. Actually, the patch I have for ZMySQLDA to make it use MySQLdb/_mysql instead of MySQLmodule-1.4 makes it fully thread-safe. I've flogged MySQLdb/_mysql with a test script and typically use 3-5 threads and it works fine, no leaks, no blocking. -- andy dustman | programmer/analyst | comstar.net, inc. telephone: 770.485.6025 / 706.549.7689 | icq: 32922760 | pgp: 0xc72f3f1d

1 0

Re: [Zope] Zope in Portals
by Christopher Petrilli 28 Sep '99

28 Sep '99

>> This is where "SELECT * FROM UPDATE" comes from. > > I don't think MySQL supports SELECT ... FROM UPDATE anyway. It certainly > doesn't support sub-selects (but will in the near future in 3.23). SELECT FOR UPDATE is an idiom for telling the database that you plan to use the data for an UPDATE at some point, so that it becomes locked (row/table/column, whatever). This is tightly wound with the concept of transactions, and I wouldn't expect MySQL to have it. >> This is just the tip of the concurrency iceberg. It's all about keeping >> your data sane under load. I can write all kinds of hacks that will fix it >> in low usage situations, but will they hold up with 100+ concurrent users? > > The answer is: It depends on your application. I don't think I'd use MySQL > for a large e-commerce application, but there are certain classes of > problems where it works fine. As an example, I have a mail application > which presently authenticates out of Solid, and I developed a MySQL > interface because we needed more speed out of, and transactions are not > critical, since the database is read-mostly. (Turns out, we may bypass > MySQL for LDAP.) As you'll note, I specifically talked about heavy update situations. Anything can be fast and safe if it's read-only ;-) > Which leads to scenario #3: Make the developer of the SQL methods write > the necessary SQL statements to implement the desired level of isolation > (if possible), and be prepared to handle an exception and clean up if > rollback occurs. It's not unreasonable to expect someone who's going to > use ZMySQLDA to be familiar with MySQL and know what it's limitations are. I believe it's unrealistic to expect someone to have to clean up after inadequacies in the database model. You COULD say "you've got to lock the table before you do anything with it", but then you're loosing all the benefits of hiding database ideas under OO concepts. One of the great things about using ZSQLMethods as they are inteded is you can reconnect to multiple different databases without rewriting anything if you stick to SQL standard syntax. If you have to insert all kinds of things, then well, it's not logical. Regardless, even if you did LOCK the table, that doesn't let you rollback changes that were done in the event of a later failure. This is very common in database applications of any complexity. > Incidentally, the "+100 users comment" raises the question: Will Zope > allow the same database connection to be used simultaneously in more than > one thread? Each thread can have a connection to each database that is accessed, but connections are not shared between threads in the way you are hypothesizing. We are talking of Level 3 DAs, those that are fully thread-safe, which the MySQL DA is not. It must be run in a single thread. With Oracle for example, if you have 5 application threads, you could theoretically have 5 Oracle connections. > Another question: How long (temporally) can a transaction exist? Is it > limited to a single HTTP request, or can it be maintained over several > requests? I'm suspecting the former. Generally the former, but it might cross many many methods in the process, and start many many transactions in different sources that are then commited as a whole (using a core two-phase protocol). Chris -- | Christopher Petrilli Python Powered Digital Creations, Inc. | petrilli(a)digicool.com http://www.digicool.com

1 0

Re: [Zope] Uppercasing a string in a ZSQL method?
by Scott Anderson 28 Sep '99

28 Sep '99

Actually, I'm trying to solve the problem for inserts, not selects, but thanks anyways. Regards, -scott tpatrick(a)IndigoNetworks.com wrote: > > There are two ways to solve the problem: > > 1. MODIFY THE METHOD AND OUTPUT TO THROUGH AND <DTML-IN> > Modify the SQL METHOD: > > select name, upper(vid_code) <<-----THIS WILL RETURN THE vid_code in > UPPERCASE > from vid > where ACCOUNT_TYPE='secondary' and BROADCAST_STATUS='ACTIVE' > and ascii(substr(lower(name),1,1)) between 48 and 57 > ORDER BY ascii(substr(lower(name),1,1)) > > When you view this via DTML in and <dtml-in> TAGS > > specify the variable as <dtml-var upper(vid_code)> > > We figured this out because the test output of the SQLMETHOD prints > upper(vid_code). If I am incorrect on the syntax above look at the TEST > output from the SQLMETHOD! It should work! > > 2. MODIFY THE OUTPUT VIA DTML with a FORMAT ATTRIBUTE: > > HELP ME OUT HERE!!! > > <dtml-in kdjfajdfljalfjafj> > > <dtml-var thevariable fmt="uppercase??"> <<< THIS IS ONE THAT I WOULD LIKE > TO KNOW!! > > Hope this helps! > > Theodore E. Patrick > > http://www.ishophere.com - Where do you shop? > > -----Original Message----- > From: Scott Anderson [mailto:sanderson@ttm.com] > Sent: Tuesday, September 28, 1999 1:11 PM > To: zope(a)zope.org > Subject: [Zope] Uppercasing a string in a ZSQL method? > > I've tried the following to change a string to upper case in a ZSQL > method: > > <dtml-sqlvar sNewAcctNo type=string upper> (doesn't work; sqlvar doesn't > define 'upper') > > <dtml-let sNewAcctNo="_.upper(sNewAcctNo)"> (doesn't work; > exceptions.AttributeError) > > <dtml-let sNewAcctNo="string.upper(sNewAcctNo)"> (doesn't work; > exceptions.NameError) > > <dtml-let sNewAcctNo="upper(sNewAcctNo)"> (doesn't work; > exceptions.NameError) > > What am I missing? > > Thanks in advance, > -scott > > _______________________________________________ > Zope maillist - Zope(a)zope.org > http://www.zope.org/mailman/listinfo/zope > > (To receive general Zope announcements, see: > http://www.zope.org/mailman/listinfo/zope-announce > > For developer-specific issues, zope-dev(a)zope.org - > http://www.zope.org/mailman/listinfo/zope-dev )

1 0

RE: [Zope] Allaire Spectra
by Alexander Staubo 28 Sep '99

28 Sep '99

Taken from the Spectra FAQ: "What is the difference between Allaire Spectra and ColdFusion? ColdFusion is Allaire's industry-leading Web application development platform with a community of hundreds of thousands of developers building sites today. Allaire Spectra is a new product family and Allaire's solution for organizations wishing to build large-scale systems that require advanced content management, e-commerce, and personalization capabilities. Allaire Spectra is built entirely with ColdFusion and requires the ColdFusion Enterprise Application Server. Interactive developers use Allaire Spectra's visual tools, along with ColdFusion Studio to build custom site implementations." It looks like Spectra is a set of services interfacing with ColdFusion, for creating applications in the three areas they advertise: content management, e-commerce and personalization. We might infer from this that they're focusing on e-commerce-enabled portals. It's a closed, proprietary, commercial system, as seen in this FAQ: "How much will Allaire Spectra cost? Allaire Spectra is priced at $7,495 per server and requires the ColdFusion Enterprise Application Server." ColdFusion EAS is in turn pretty expensive. (Although not expensive in enterprise terms. Just so expensive that somebody can't "install and play around with it".) In other words, everything Zope isn't and will never be. Good news, I think. They're not learning. -- Alexander Staubo http://www.mop.no/~alex/ "`This must be Thursday,' said Arthur to himself, sinking low over his beer, `I never could get the hang of Thursdays.'" --Douglas Adams, _The Hitchhiker's Guide to the Galaxy_ > -----Original Message----- > From: Jay, Dylan [mailto:djay@lucent.com] > Sent: 28. september 1999 03:26 > To: 'zope(a)zope.org' > Subject: [Zope] Allaire Spectra > > > Has anyone had a good look at Allaire Spectra? > It looks like Zope on steriods. Or at least marketed that way. Perhaps > someone can do a feature for feature comparrison. > > > http://www1.allaire.com/products/Spectra > > _______________________________________________ > Zope maillist - Zope(a)zope.org > http://www.zope.org/mailman/listinfo/zope > > (To receive general Zope announcements, see: > http://www.zope.org/mailman/listinfo/zope-announce > > For developer-specific issues, zope-dev(a)zope.org - > http://www.zope.org/mailman/listinfo/zope-dev ) >

8 8

Re: [Zope] Zope in Portals
by Andy Dustman 28 Sep '99

28 Sep '99

On Tue, 28 Sep 1999, Christopher Petrilli wrote: > > There are a couple possibilities for implementing transactions in > > ZMySQLDA: > > > > 1) Automatic use of LOCK TABLES. > > > > Excuse me while I lose my lunch. :-) This is so amazingly hackish as to > almost be a joke. This *might* work partially in some half-brain-dead > fashion in a single-threaded, non concurrent environment, but when you have > other situations where you might have two people touching tables at the same > time, in some cases dependent on each other... In a single-threaded, non-concurrent environment, you wouldn't need LOCK TABLES at all. > This is where "SELECT * FROM UPDATE" comes from. I don't think MySQL supports SELECT ... FROM UPDATE anyway. It certainly doesn't support sub-selects (but will in the near future in 3.23). > This is just the tip of the concurrency iceberg. It's all about keeping > your data sane under load. I can write all kinds of hacks that will fix it > in low usage situations, but will they hold up with 100+ concurrent users? The answer is: It depends on your application. I don't think I'd use MySQL for a large e-commerce application, but there are certain classes of problems where it works fine. As an example, I have a mail application which presently authenticates out of Solid, and I developed a MySQL interface because we needed more speed out of, and transactions are not critical, since the database is read-mostly. (Turns out, we may bypass MySQL for LDAP.) > Not to slam MySQL, but... it has it's place, but I'm not sure it's under > heavy update loads. There were two principles we came up with when working on the Python DB API 2.0 spec that are relevant. The first, and oldest: The query language is database-dependent. The second, and relatively new: Don't support rollback() on databases without transactions. (The old MySQL module implemented rollback() as pass, which is bad news.) The most the DA should reasonably be expected to do is raise an exception in this case, which the caller can (maybe) recover from. Which leads to scenario #3: Make the developer of the SQL methods write the necessary SQL statements to implement the desired level of isolation (if possible), and be prepared to handle an exception and clean up if rollback occurs. It's not unreasonable to expect someone who's going to use ZMySQLDA to be familiar with MySQL and know what it's limitations are. Incidentally, the "+100 users comment" raises the question: Will Zope allow the same database connection to be used simultaneously in more than one thread? I sure hope not, and it seems unlikely: I expect very few, if any, databases can handle multiple users sharing the same connection, particularly with the normal concept of transactions (i.e. one transaction open at one time on a connection). Therefore, it seems there must be some locking mechanism within Zope to prevent this. Sharing a single connection this way would tend to produce a serious bottleneck with 100+ users, but of course, you can have multiple connections. How the application makes use of these connections will affect the MySQL locking scheme to employ somewhat. Another question: How long (temporally) can a transaction exist? Is it limited to a single HTTP request, or can it be maintained over several requests? I'm suspecting the former. -- andy dustman | programmer/analyst | comstar.net, inc. telephone: 770.485.6025 / 706.549.7689 | icq: 32922760 | pgp: 0xc72f3f1d

1 0

Re: [Zope] Uppercasing a string in a ZSQL method?
by Jim Sanford 28 Sep '99

28 Sep '99

This should work. (It works for me.) <dtml-sqlvar "_.upper(sNewAcctNo)" type=string> Jim Sanford Accelerated Technology jsanford(a)atinucleus.com -----Original Message----- From: Scott Anderson <sanderson(a)ttm.com> To: zope(a)zope.org <zope(a)zope.org> Date: Tuesday, September 28, 1999 1:21 PM Subject: [Zope] Uppercasing a string in a ZSQL method? I've tried the following to change a string to upper case in a ZSQL method: <dtml-sqlvar sNewAcctNo type=string upper> (doesn't work; sqlvar doesn't define 'upper') <dtml-let sNewAcctNo="_.upper(sNewAcctNo)"> (doesn't work; exceptions.AttributeError) <dtml-let sNewAcctNo="string.upper(sNewAcctNo)"> (doesn't work; exceptions.NameError) <dtml-let sNewAcctNo="upper(sNewAcctNo)"> (doesn't work; exceptions.NameError) What am I missing? Thanks in advance, -scott _______________________________________________ Zope maillist - Zope(a)zope.org http://www.zope.org/mailman/listinfo/zope (To receive general Zope announcements, see: http://www.zope.org/mailman/listinfo/zope-announce For developer-specific issues, zope-dev(a)zope.org - http://www.zope.org/mailman/listinfo/zope-dev )

1 0