Today's Problems

post reply Forum Index » Website Feedback

1 2 ► Go

Go to page...

12/07/2015 00:49

Subject: Today's Problems

quote

jsteele
Site Admin
Joined: 11/12/2007 23:38
Messages: 50212
Online

One thing that I've found from running this site is a lack of understanding of what is involved in keeping things going and a divergence of opinion about how it should be done. Having just had an unplanned "really f'd up day", I'm inclined to wax poetic on the topic.

Because Sundays are normally fairly slow, I decided to do some upgrades on the servers. We have a cluster of two web servers and another cluster of two database servers. There are also a few random servers that do this and that, but they aren't important -- regardless of their opinions on the matter -- for this discussion. I try to have every service duplicated so that we can always fail over to a backup. But, in reality, that is very difficult. Today, after some firmware upgrades to the primary web server, the server's network interface cards (ethernet connections) went out. That server could not connect to the network and for all intents and purposes became a very large and not particularly efficient paper weight. That's when my blood pressure nearly blew my eyeballs out of their sockets.

I had already failed over everything (so I thought) to the second member of the web cluster. However, the authentication server had not been clustered and was unavailable. As a result nobody could login. Not even me. Then, because whatever higher beings determine our fate decided that I needed a good kick in the guts, the application server on the front end died. That meant that we had no home page. Look at the bright side: no need to worry about the inability to login when you can 't get to the home page which is the only page that allows you to login.

But, I'm a hardcore computer geek who has been through a bunch of tough times. I have backups. Do what you want to me. I'll recover from a backup. I guess I didn't mention that while I was installing those firmware upgrades that caused my primary server to henceforth identify as a paperweight, I decided to run an OS upgrade on the server that does my daily backups. Yes, there was that. Little did I know, that upgrade changed the database that was being used. So, I have a backup program that without fail conducts backups. That program is more faithful than my dog (and I don't even have to feed it). But, it relies on the database that was just replaced with an incompatible one. So, I have no ability at this time to recover the backups. And, did I say that I can't even login to my own website?

This is the point where I really feel like I can share some advice. Despite my best efforts, everything has gone wrong. How do you confront this situation? I focused on what was important. For me, that was being able to manage the website. I needed to be able to login. Therefore, I created an entirely new version of the forums that used a different authentication facility. This version was only available to me. That allowed me to login and manage things as necessary. Then, I focused on the backup server. I had to play a little fast and a little loose, but as long as this is just between friends, I managed to trick the backup program into running. This sort of thing doesn't show up on certification exams, which is probably why I'm not certified for anything (at least that I'll admit). Once I could recover my backups, I was home free.

The lesson is: focus on what is important and have good backups.

I recovered all that I needed from my backups and got things running in respectable, if not perfect, form. The other problems will be solved later. But, if anyone wants a giant paperweight, I have one available at an attractive price.

profile p.m.