(From Eighth Note, the employee magazine of the Federal Reserve Bank of St. Louis, March/April 2005, Volume 19, Number 2.)
With a tight deadline and facing difficult odds, Eighth District employees proved that they could build a new business continuity facility and state-of-the-art data center through creative thinking, hard work and constant collaboration.
By Alice C. Dames
The Eighth’s Districts decision to sell its Little Rock Branch building and move employees to smaller quarters meant that the Bank not only needed to find a new location for these employees, but also a new home for the District’s business continuity site.
In August, the District found that new site in Washington, Mo., about one hour west of the downtown St. Louis area. But several difficult challenges loomed: The building already occupying that site, an old print shop, had to be converted into a state-of-the-art data center that would meet the District’s business recovery needs.
That meant gutting and rehabbing the building’s interior while also constructing a new driveway and parking lot and adding special security features — all within the space of three months to meet the District’s self-imposed deadline of Nov. 15. The goal was to have the site ready for testing before the end of 2004.
Even the employees involved had doubts that the Bank could make such an ambitious target.
“We all looked at each other and said, ‘We can’t do this; there’s just no way!’” says John Schuermann, a senior technician in ITS.
Facilities Officer Ray McIntyre agrees. “It was a real nail biter. Normally I like to keep a few weeks of extra time in my back packet, just in case. But the day we bought the building we were already into some of the lead times some of our vendors required.”
Beat the Clock
Previously, the District had houses its regional business recovery facility in Little Rock, while another site in Overland serve as a local recovery facility for St. Louis Operations.
In looking for a new site, Business Continuity Manager Derek Schneidt says, “The task was not to replicate the Little Rock and Overland sites, but rather, to create a bigger and better space where the Bank’s critical departments could recover their operations more fully and for three months or more.”
The first steps was to survey critical Bank departments to see what they would need to recover their operations. A Bank-wide workgroup was formed in late 2003 to help choose a new site. The team examined several factors, including security, weather, natural disasters, soil conditions, infrastructure, amenities, utilities and travel time to St. Louis.
As a result of its analysis, the workgroup decided the Bank didn’t have to look too far to get what it needed.
“We found that the city of Washington, in Franklin County, was the best location for us,” says Business Continuity Officer Rao Kaza.
The Bank signed a real estate contract in mid-August, and the challenge of meeting the Nov. 15 deadline began.
To help manage the project’s smaller but still critical details, Chris Whiteman from the Center for Project Management examined each department’s requirements and determined how many workstations, servers and phones the site would require. Then she broke the project into phases, identified milestones and created a project plan. Whiteman also led daily department meetings in ITS, updating the plan after each meeting.
“Those meetings were a fabulous learning experience for all of us,” says Schuermann. “Whenever a problem arose, we’d take a team approach to resolve the issue.”
Most of the problems the team experienced were out of anyone’s control. For instance, storms during late autumn delayed construction of the driveway and parking lot.
ITS Senior Manager Jim Albenesius says, “We also had to wait until Ameren UE installed permanent power in the building before we could install the phone and data network. And later, when SBC was laying fiber cable, they hit bedrock; so, they had to do some blasting before the cable would come through.”
Coordinating the move also posed several challenges. For example, Little Rock and Overland had to remain functional until the Washington site was certified, and TRASS’ equipment had to be moved in three distinct stages.
“We Scored a Touchdown!”
To meet those challenges, Kaza says many employees made personal sacrifices, working extra hours to make sure that the project stayed on track. “Our team didn’t wait for things to happen or make excuses for why things couldn’t get done,” says Kaza. “Instead, they took the initiative.”
Schneidt adds, “We knew we had a limited budget and aggressive schedule, but we didn’t let that limit our thinking. Rao had a very practical idea. He said, ‘Let’s start here and see if we can do more.’”
As a result, the Eighth District now has the only dedicated recovery system within the Fed system, one which has the same computing infrastructure that a full branch would have. This means that if there’s a computing network failure in the St. Louis data center, the District can recover its network in Washington, and employees don’t have to relocate.
“Adds McIntyre: “We had a very good team. We had a good relationship with the city of Washington; we hired a good contractor; and we had the right Bank employees out there working with them. At the end of the day, I think we scored a touchdown when it comes to meeting our customer’s expectations.”