[ZWeb] DNS still fishy?

[Justizin]

On 10/12/06, Lennart Regebro <regebro at gmail.com> wrote:
> Just a couple of notes here.
>
> Although zoneedit has been running fine for me for years without a
> single problem, obviously it would be nice with some backup.
> Preferably something with another ISP and located on like another
> continent or something. Two of these backups would be even better.
>
> But honestly, compare the likelyhood that all three of these would
> fail at one time, together with the increasing likelyhood than one
> server of them is misconfigured and starts disturbing the usage for a
> minor part of the users, then we will quickly realize that the more
> backups and failsafes we have the larger the likelyhood that something
> of this will go wrong.

the worst that happens is that some changes fail to propogate.
changes to DNS should always be approached with the assumption that
this will happen.  What's worse is for there to be no copy of a zone
available.

It should never be necessary for an A record to change immediately,
because this cannot be relied upon.  The best defense to this is,
however, to set TTLs at 300s, or 5 minutes, about a week in advance.

> 8 servers seems to be to be a complete overkill, and it will only
> cause problems. I will change my mind on this the time all zone-edit
> servers stop working at the same time as two of the backups fail.

It could cause problems, and that's why we aren't really using eight
servers right now, but it should not cause problems.  It is a
challenge, also, that our DNS is not hosted in the same location as
the website.  So, it's possible that DNS will be unreachable when an
outage occurs, i.e. a fibre being cut in the middle of the ocean, and
this outage may not actually affect our site.

I bet ten bucks if we rely entirely on zoneedit's nameservers that
this will happen once for at least twelve hours for some significant
region of the world within the next year.

> Don't overcomplicate things. It just makes them fail.

This assumption really has nothing to do with what happened this week.

What happened this week was either:

  (a) a typo

  (b) an erroneously truncated string

If there were only two nameservers, they would have pointed at the
wrong IP, and the site would have been perceptually unavailable for a
few hours to two days for various people.  If there were eight, the
same would happen, for about the same time frame.

So, if you want to only use two nameservers, that's okay with me.
Remember to wake me up when the zone is unreachable for someone and we
want to run more. :)

I always assume, if anything, that some machines, network connections,
disk drives, etc.. will invariably fail, and that you can never have
too many if they are available.  I like the idea of a group of zope
community members collectively providing DNS service.  Maybe we should
even talk about running multiple copies of the flat content in
different places.  If my site goes down, esp if one of my machines
fail, I much prefer to feel comfortable that I can reach zope.org than
rely on the possibility that i might have copies of recent releases in
another location.  if i'm going to keep copies of the releases around
for myself, might as well mirror them, eh?

While having a set of servers configured by various people sounds as
if it would be overcomplicated, with proper planning and coordination,
we should be able to keep it simple.

When making changes to DNS, always assume that for 48 hours there will
be between a 90-10 and 10-90 split between people who have your new
records and people who have old records.  When changing nameservers,
double or triple this, because some people will have cached records
from the old nameserver *and* more recently cached NS records, so they
may continue querying the old nameserver until the cached NS record
itself expires.

When something critical like svn/cvs or the main website need to be
changed, again, it is necessary to drop the TTL, on the entire zone,
even, to something really short like 300s about a week in advance.
This ensures that everyone in the world has a copy of the zone which
says: "no copy of this zone and no records in this zone are good for
longer than five minutes.".  Just before a switch is made, you can
proxy the old front-end apache server to the new host explicitly, and
then update records.  for five or ten minutes some people's requests
will be slow because they are possibly doubling-back across the
internet, but at least they can't really tell what's going on, just
that for a few minutes it is a 'little bit slow'.

-- 
Justizin, Independent Interactivity Architect
ACM SIGGRAPH SysMgr, Reporter
http://www.siggraph.org/