Rising Above Without Sinking Below

Rising Above

The data professionals I've had the honor of working with are some of the most dedicated people I know.

There have been several occasions in my life when a hard deadline has been mandated to deliver a seemingly impossible data solution. And in every single instance, my team and I rose above to meet the challenge. Sometimes you make it, and sometimes you don't. But one thing's for certain–the heart and mind of a data professional under pressure to deliver is an undeniable force to be recognized. This quality, however, puts them at a high risk of being exploited. Here's a story that changed my life forever.

Delivering HiFIRE

It was the Fall of 2004 and I was consulting with a data warehouse team at Hitachi Data Systems. There were various others who came in and out of our team, but our core team was solid–six brilliant and talented database experts, a sharp and experienced business analyst, and an organized and driven manager. We were working with a pretty common stack: Oracle on the back end, Business Objects on the front end, and Informatica to handle ETL. We each had our specialties, but we all knew all the tools inside out. We were a tight-knit group. In fact, I keep in contact with many of them after all these years. We often reminisce about good ol' days of HiFIRE.

HiFIRE stands for Hitachi Feature Interoperability Reporting Engine. After a round of making improvements to our data warehouse, our manager approached us with this important project. Hitachi's data systems must be configured with other components (server, operating system, channel adapter, etc.) made by different vendors. Hitachi had an interoperability lab where they would test various combinations as needed to make sure everything worked properly. When you included varieties and versions of each of these components, the number of permutations and combinations of how a data system could be configured are enormous. So, they needed a data-driven web application to keep track of their testing schedule and results.

Everything seemed reasonable and even exciting, until we heard the deadline. They needed a base functional version stood up and working in three months–three months! It was August, and there was a big show in November where our end user wanted to demo the product (that didn't exist). So, we were charged with building a functioning database, web front end (using the Business Objects API), and ETL transformations from whole cloth, in three months. And like all good data professionals do, we rose to meet the challenge.

Unsustainable Pace

After a quick huddle (mainly to calm each other down), our business analyst promptly produced a Business Requirements Document. This gave us the foundation for negotiating with our end user on what a reasonable first release would look like. Although we had the opportunity to break the functionality down into feature sets, to get a release that was usable, we had to decompose the solution functionally. So, there was no way around getting the fundamental components of the system working–database, web front end, Business Objects integration, and Informatica transformations.

As a team, we committed to making this happen, but we almost killed ourselves in the process. Standing up the database was easy, but the Web development part was new to most of us. We were data professionals, not Web developers. It's no surprise that we latched onto Business Objects' Web Intelligence API. We knew Business Objects, but most of us had to quickly build our Java skills to code the front end. Add to that all the undocumented features of Business Objects in a Web environment, and we had quite a challenge in front of us. Fortunately, one of us was an expert in both Java and the API, but even our expert had to wrestle with the code on more than one occasion.

Some days were very, very long. There were times when I started at 7 am, worked all day and night on HiFIRE with my team until 2 am, and then returned at 7 am to start all over again. The kinds of challenges we faced were unbelievable. I remember going crazy trying to find a memory leak that would suffocate the system within a matter of days. As it turned out it was one of those undocumented features I previously mentioned and, after several phone calls with Business Objects support, the only way we could stabilize the system was with a huge workaround that would monitor memory usage and seamlessly restart the system when it reached a dangerous threshold. With big challenges like this to overcome, we had our doubts about making our deadline–but we pushed on at full steam no matter what.

November or Bust

As we turned the corner on November, things weren't looking good. The conference was in a couple of weeks and our functional testing had more red lights than a Christmas tree. We couldn't move the conference date and we didn't want to disappoint our end user. We pulled together and we worked around the clock. We had breakfast, lunch, dinner, and midnight snacks at the office. Even our manager stayed late with us for moral support. We slid into User Acceptance Testing with only minutes to spare. Crossing our fingers and toes we anxiously watched our end user test through the system. And we passed with flying colors.

With a collective sigh of relief, we almost cried. We were so tired. But we were elated and proud as well. We rose above to meet the challenge in front of us, and we delivered. Our end user was very excited. This product would be a big hit at the conference, and he had nothing but gratitude for us delivering on time. We all went home early that day and got a good night's sleep. We deserved it.

The next morning, we met with our end user to talk about the next release. What we heard him say was astonishing. He felt that, given we could deliver what we did over the last three months, it was fair to assume we could deliver an equivalent set of new features over the next three months. What?! No way. We were traveling at a wildly unsustainable pace and there was no collective gas left in the tank–period.

In fact, this event prompted us to move in an Agile direction. Attracted by the notion of a 40-hour week, Extreme Programming looked like the right transition for us–so that's what we did. The 40-hour week rule in Extreme Programming–which evolved into the Sustainable Pace rule–was put in place to prevent developer burnout. It also prevents the unethical black market hours that we've all spent trying to hit a deadline.

Lessons Learned

Delivering HiFIRE is an experience I will never forget. It fortified great friendships and it reaffirmed my belief that the impossible is only impossible until it's not. Additionally, I learned a great lesson about setting expectations. You teach people what to expect from you based on your performance. So, the Sustainable Pace rule of Extreme Programming was born primarily to set the right expectations with end users. Preventing burnout is a bonus.

Leaders and managers have an ethical responsibility when asking their data professionals to rise above. Data professionals will always do it because it's part of their core values. That's no reason to take advantage of them. Understand that this is burst performance, not business as usual. Professionals should be acknowledged and rewarded for their over-the-top contributions, and expectations should be kept at a reasonable level.

I wouldn't be surprised if my HiFIRE story hits close to home for many. All data professionals are called upon to rise above at some point in their career. I wouldn't trade those memories for all the money in the world, but even more valuable are these lessons that have served me well for decades. Take some time today to reflect on the times you've risen above and the lessons you can personalize from that experience. I'm not sure if HiFIRE still exists, but its effects on me will last for decades to come.