Writing from a little room called "The Matrix" at Diamond Mountain, this is an email I just sent to an employee of Dream Host, the company that hosts most of my web sites, many of which (including this one) had some downtime this weekend:

On Mon,  8 Mar 2010 07:11:19 -0800 (PST), DreamHost Customer Support Team
<support@dreamhost.com> wrote:

> I'm one of the mysql admins with DreamHost and will look at this issue.
> First, I ask that you keep profanity out of support emails.  It does not
> get your issue fixed any faster.

Hi Justin,
My apologies. I will try to refrain in the future but consider it an indication of my frustration.
As I said, this has happened again and again. not this particular problem
but various problems that all involve a lack of prior warning from DH
for a variety of changes, updates, equipment swaps, etc.  it's just lame, IMHO. (can I use the word lame? no,
some would say that's bad, so i'll say... how about "inconsiderate and bad business practice"?)

> I searched through our background
> database mover, and can see no moves scheduled that would have effected
> your databases.

That may be the case but if you'll look at the history of this support ticket you'll see that the support tech that was
helping me claimed that database moves were exactly what caused the sudden lack of proper permissions
that all my mysql user accounts needed to access all of my databases.

to wit, George said:
>>I have just been informed that our MySQL admin moved about 6000 databases
>>over the weekend.

>>So all of the MySQL servers needed to be reconfigured.

>>Unfortunately some were not done automatically

so, i'm not sure what was going on. if they were moved friday evening then
i'm confused as to why you need to move them again, and if they
were not, then i'm confused why your support guy said they were moved.

> So this issue you were having is not us moving your
> databases or touching them in a way that would break them by our doing.
> That said, slinky is a old mysql server and belonged to your previous
> cluster when you weren't on the baghdad server.  I'm moving your
> databases to a proper mysql server in your current cluster.  The downtime
> will just be a few minutes, but it should help since it will be in the
> same physical location and will be on newer hardware.

thanks for explaining all that, though as I said above, i'm still confused and don't know why the problem happened and would like to know.
(but to respond with a question, is there some way that users can themselves help to check whether  their databases are
up to date and on the right machines and matched up with the proper clusters and whatnot?
how might these kinds of mismatches be prevented in the future? I know in the past in the control panel there have been, at times,
notices that indicate the ability to move all my DBs over to one machine, and I have in the past
taken that advice and opted to do that whenever i've seen those.. apparently that wasn't enough or isn't what you're talking about?)

>
> As for notification of system changes, there is an email sent in cases
> where webservers are moved, and databases are not moved often, some
> living on the same servers for years like your databases in this case
> being on slinky mysql server for a longer time.

thanks again.  I know it's probably difficult to account for every single change and problem
left over from legacy systems and such.

in closing, allow me to just relate one anecdote that taught me something I have tried to
apply to every job I have done and that I think applies here.
Years ago I used to be a live sound engineer.  several nights a week I would be in a club
setting up microphones and running the mixing board for bands. all sorts of bands.
some i would like, others i didn't. some were famous, some were tiny little local bands just starting out.
some had huge crowds, others just a handful of friends watching.
the old sound man that trained me said one night toward the beginning of my tenure
at this club:  "to you, tonight is just another night, just a job.
but to these musicians, and especially the audience, this night is a
special night. they don't do this every day. this really matters to them,
even if to you it seems like just another night of the same old same old."

[to explain further, blog readers, i included this mainly because a previous support person told me that i was not warned about the change DH made because it was "only a small percentage" of all their databases. in other words, i didn't matter because i was just a tiny sliver of their business.]

I remembered that and kept it in mind, or tried to, every time I was mixing, and then
later and still now, for anything i'm doing that involves other people. I'm not trying to
condescend or sound like some wise old man, but am just relating a possibly useful parable that will hopefully
inspire others.  I write this from a freezing cold wet drizzly Buddhist retreat center in the mountains
of southern arizona, where i've been for 4 days now and where i kept getting texts and calls friday and saturday
and yesterday about my users' sites being down (i was sort of hoping my phone would not get reception
here, but alas)... so perhaps I'm not only in a slightly annoyed mood but also in a
somewhat reflective state of mind...

best wishes,

steev

>
> The move will be done in a couple minutes, if you notice any further
> issues specifically related to the databases and these issues you are
> having, please let me know at justin@dreamhost.com.
>
> Thanks!
> Justin K