DR - fixing problems found

This is the third blog about DR. Whilst contracting at a major international investment bank, I was involved in their DR testing. Every 6 months, on a Saturday, all major systems would be failed over to the DR site to run a DR simulation. Process documentation had to be followed and any issues/problems or errors reported to the DR co-ordinator. Some systems came up easily, others required some workarounds (that needed to be added to the documentation), whilst others just failed. A week after the DR was done, the co-ordinator would send out a report on what failed and what needed to be updated. I was tasked with looking at a number of failed SQL Server applications. What I discovered was that the same systems had been continually failing the DR testing for years and nobody had ever bothered to actually try and fix them. I spent a few weeks getting failed systems to work and preparing scripts and documenting what needed to happen. It was quite frustrating as some of the fixes were quite easy, but nobody had spent the time and effort.

No use going through the whole process of documenting the problems if nobody is going to do anything about it.