Steve's blog

A blog about what's going on in Analysis UK...

When Backup systems bite back.

I’m a firm believer in backups. Data backups, spare part backups, uninterruptible power supplies, raid disks and stuff that should sit around in the background and help you when things go wrong. However the last thing I expect of these systems is to introduce major failures.

I got struct three times with this just recently with backup systems that decided they needed more attention.

All the web sites I run are basically data driven, all the exchange rates for Dollars2Pounds are in a database, all the page impressions are logged into a database, all the used books for sale on BookSwap.ws are all in a database so the backups for these sites is a backup of the databases (which is stored on a separate machine or two), the actual application code doesn’t need regular backups.

The other week after the server had a bit of a hiccup I figured I’d give the automated backups for the web sites a try, after all, if I needed to move to a new server this may save me some time. I only set up 3 sites as it’s a slow process, I’m so glad I only did 3 as two backup attempts latter I had to stop these. I set the backups to occur weekly – as I said, nothing really changes on the actual front end so very little need for backups anyway.

One of the things the web site control panel’s automated backup does is to take the site off line – now I can kind of understand this if data is being changed but really, most websites are static files (html, php, dll’s) so you’d think these could be just copied.

Have you spotted the snag so far, backup are really important, keeping your web site on line is also important – I don’t think being offline for 30mins every day to do backups is all that acceptable for global eCommerce sites.

So what happened, well the backup for BookSwap.ws started, the site was taken off line, then, well, the backup appears to have failed and the site never got put back on line. No details of this in the control panel for the site and I even had to go as far as editing the setup to kick the site back online. It was a very evil situation that I was VERY unhappy with. When I realised what had happened I canned the automated backups immediately.

Then 2 weeks latter I found out the secure server part of BookSwap.ws was off line, this meant transactions, "my account" and member login were all unavailable – not a funny situation. Guess who had setup automated backups and forgotten to disable them on this account!

Having finally got rid of the web site automated backups (remember the database is backed up separately!) the websites have been up and running without any problems since.

The next thing I know I’ve wandered into my home office and it’s all very quite – not normal as I usually have 3 PCs running, this particular day no PC’s were running, a quick look at the UPS – looked OK but the only reason all 3 should be off is a power fail. About a month before we had a major outage that drained the UPS.
Well, I kept an eye on the UPS and it all looked OK, no battery fail light or anything (it’s supposed to be Smart). Then two weeks latter all the PC’s go off again. This time the UPS goes off and complains big time about being switched back on. This time indicating a battery failure.

I got my self a new battery for my UPS from UPSBattery, a great deal and delivered the next day - fantastic. Installed it without any problems and the UPS is now up and running.

I decided to have a go at installing the software to manage the UPS again, last time I tried it was a nightmare (about 3 years ago) and I gave up. Well the software doesn’t appear to have changed much, but fortunately it has the option to remember User/Password details as I always forget that stuff (one feature of FireFox and IE I love).

Naturally I installed the business edition of the UPS software – not because I’m running in a business environment and need the software on my server and to be able to connect remotely to it. Not because I plan to get more UPS, but because the personal edition wouldn’t install without a UPS on the system and it didn’t detect the one that was connected on the USB port. Windows had it, the business edition (which you have to install 3 apps for!) had it, but the personal edition, nope useless.

Once up and running I ran the self test a number of times, all appeared OK, It wouldn’t let me calibrate the battery without it being 100% full. When I did finally get there the UPS died and took down all the PC’s connected to it – when I got them rebooted the UPS control software didn’t have the intelligence to realise it killed the UPS and reported no events – nice!. Mental note – when running UPS cal/self test don’t rely on power from the UPS to drive the machine you are using to monitor the test.

With it being a new battery I think the previous settings in the UPS memory were reporting an incorrect charge level, hence the battery wasn’t all that well charged when I tried the calibration – hence it didn’t survive the calibration!.

I also run a daily backup on my development PC to backup the source code every day, the application for this is nice and simple, but when it kicks in that’s it, the PC’s unusable for an hour – the backing up takes about 5 mins, but 55mins is spent pre-processing – talk about a pain, I’ve got a decent PC with a few spindles and lots of memory but still the backup app needs to hog 100% of the CPU.

So I’ve the backup set for 2am so I’m forced to go to bed then, but also forced to leave the PC on over night – hopefully the new version of the app has the ability to run in the background as I work – maybe I will write to them and ask, after all they keep emailing me to get me to upgrade but don’t bother to tell me any special reason that the new versions is better than the old!.

I’ve not posted any links or named the products that bit me, I don’t want to give them any free advertising or inbound links for products I’m not happy with!

Well, it’s 1:55 am here in the UK and I need to get some bits done before the backup takes over and the PC becomes unusable.

Any backup tips would be much appreciated…

Steve.
Comments are closed