Early in my career at Oracle, I worked as what you might call a “paramedic” for Oracle databases.  Our team was originally called Field Support, but was later renamed Advanced Customer Services (ACS).  Most of us worked on full-time contracts, but we also worked short duration emergency jobs that we called “Fly & Fix” engagements.

During a Fly & Fix engagement, you could encounter just about any problem imaginable with Oracle databases.  We worked on database performance issues, crashing databases, corrupted databases, and corrupted databases with failing recovery.  By far, the most challenging engagement was the failing recovery situation. 

Consider the situation where a database is corrupted and the backup is either corrupted, doesn’t exist, or the backup isn’t usable for some reason.  Most importantly, think about the business impact in these situations as well as the human impact to the people involved.  These are very stressful situations.  People’s lives are disrupted and careers are impacted. Backup/recovery is a critical part of any IT operation, and it normally works unless mistakes were made. Like I said, it’s a very stressful situation.  Making such serious mistakes can be career limiting. 

120 Stories

In this blog, I will tell a sampling of stories from my years doing Fly & Fix work, with a focus on the events that taught important lessons to everyone involved. I will also tell how the failure started and some of the human and business impact it caused.  I went on a Fly & Fix engagement about once per month for 11 years, so I have about 120 stories to draw from.

Some of these stories are known to the industry and business press, but most of them went unreported. All of these cases are highly confidential, so I won’t divulge names of companies or people involved. I hope this blog will serve as a warning to IT professionals so you can avoid these sorts of issues in your own operations.  All of these stories will be told from my perspective and I will tell you the lessons I learned through the process. Others involved might have taken different lessons.  I hope you enjoy the re-telling.