- UASI, Bureaucracy and Terror

Homeland InSecurity

This week I was, in turn, amused, maddened and fasincated by disaster. I had the opportunity to do both leadership and followership as officials from the “Seattle Urban Area” considered projects to submit for grant funding from the fedgov Department of Homeland Insecurity.

This whole process is a look into the little-known culture of “homeland insecurity” which has blossomed over the last 7 years.

Let me say, right at the beginning, that September 11th, 2001, was a horrific event and a real wake-up call for the United States. We’ve been taught the same lesson we learned on December 7th, 1941 – two huge oceans do not make for a nation secure from the enemies of our way of life. My somewhat humorous tone in this blog is not meant to make light of the serious threats we face.

But what is our major response? Pretty typically, we create a giant bureaucracy, slosh a lot of political rhetoric and some money around, and get prepared to fight … the last war.

The major bureaucracy is, of course, the Department of Homeland Security, or Insecurity, as I’ve named it, because it heads an industry and set of programs based upon the fear and feelings of insecurity of the American People. DHS is really a myriad of individual bureaucracies and programs, sometimes working at cross-purposes and often duplicating effort.

But I stray from the subject – the Urban Area Security Initiative (UASI), usually pronounced You-Ah-Zee. In 2009 about $1.8 billion will be granted to urban areas through a series of programs as shown here. The UASI grants are made to “urban areas”, a carefully defined (by the fedgov) set of metropolitan areas which share vulnerabilities and threats. The “Seattle Urban Area” is the City of Seattle, its surrounding King County, and the counties to the north and south. The Seattle urban area is eligible for $11 million, more or less, of that $1.8 billion pie.

Now $11 million may sound like a lot of money, and it is. But think about the Seattle Urban Area – a population of 3 million people, two major ports (Seattle and Tacoma) into which ships with nuclear bombs or bioterrorism agents or chemical weapons could sail, three major military bases (Naval Station Everett, McChord Air Force Base, Fort Lewis), and vulnerability to a potential magnitude 8.0 earthquake. We probably have 12,000 cops and firefighters (Seattle alone has 2,200), each of which needs radios and hazmat gear and training. All of a sudden, $11 million doesn’t sound like a lot, and it isn’t.

Its also a little amazing that places like Vermont and Wyoming – not exactly your typical terrorist or disaster targets – get $6 million each. But that’s the reality of politics in our 2009 world with two senators from every state and a lot of dollars to be allocated.

Each year, a group of officials from the emergency management, law enforcement, firefighting, public health and technology disciplines – the officials responsible to prepare the region for disasters and terrorist events – get together to consider projects for UASI funding. We did that this week in the Seattle Urban Area. We have a strategy to guide us but it is still a give-and-take to determine which projects are most importnat to the region.

I lead the subgroup of technical and operational staff responsible to consider and prioritize interoperable communications projects. These are the networks and systems which allow first, second and third responders to communicate with each other and with their dispatch centers to prepare for and respond to disasters large and small. They include handheld and vehicle-mounted radios, computer and telephone systems, and fiber optic networks.

Small disasters are the ones which occur every day – such as when I fell off my bicycle, shattered my arm and called 911 for help. The large ones are the fires, landslides, earthquakes, and (God forbid) terrorist events which may afflict our region.

Finding projects which prepare us for those disasters and which fit into $11 million is daunting. In the past we’ve procured fireboats and helicopters and personal protective equipment. We’ve found money for training our responders, buying radios and for public outreach and disaster preparedness campaigns. It is hard work to winnow 100 proposed projects and $40 million in needs down to a dozen or so projects with $11 million or so, but we mostly did it this week. In future blog entries I’ll talk about some of those projects – the ones I can reveal without making us more vulnerable to terrorists (and that is most of them).

In the meantime, I’m just proud of the emergency managers, fire and police officials, and technical staff I was able to spend time with this week, prioritizing projects to keep the Seattle urban area safe.

And I’m tired!

- The 108 Degree Data Center

The 108 Degree Data Center

November in Seattle is always cool and rainy and sometimes stormy – windstorms, that is. Seattle’s all time high temperature – for any day of the year – is 100 degrees. That all time high is, of course, outside. But it reached 108 degrees here on Sunday November 16th. Inside a data center. The City of Seattle’s data center.

To make a short blog entry even shorter, I’ll skip to the root cause: a failed power breaker on a pump for the domestic water supply to the building housing the data center. The water supply flows to CRAC (“crack” or computer room air conditioning) units which, in turn, cool the data center. For HomeCity Security reasons, I won’t reveal the actual location of the data center, but let’s just say it is in a downtown 60 story skyscraper which also houses about 3500 office workers during the week. The problem started about noon and was fixed at about 8:00 PM.

The data center holds about 500 servers, storage systems and other equipment. We shut down a lot of servers and many services starting almost immediately. Nevertheless the temperature in the data center rose to that toasty 108 degrees, setting a new record high (sort of) for Seattle.

So why is this notable? For two reasons: the problem and the response.

In terms of the “problem”, let me assure you (especially if you live in Seattle) that cooling problems like this will be rare to non-existent in the future. Years ago we installed a one megawatt generator for backup power. This year we’ve been working a project to install “dry coolers”. These aren’t really “dry”, but the water cooling the data center will flow in a “closed loop” between the new coolers and the center, so we’ll no longer be dependent on external water or power supplies. Unfortunately, the dry coolers don’t come online until January, which is why we went to 108 degrees last Sunday.

But there’s a more general issue here – every city and county government has data centers and servers and vital information. Every area of the country is subject to some sort of a disaster and every government needs to have a backup and recovery plan.

But for what disaster should we prepare?

Here in Seattle, everyone is concerned about the “big one” – a magnitude 8.0 earthquake. While we need to be ready for an major earthquake, we have about one of those “big ones” every 300 years. Much more likely are disasters like last Sunday – a failure of water and cooling, a “meltdown” if you will (non-radioactive, however!). Or perhaps the disaster will be the opposite – too much water from a broken pipe, and a flood drowning those servers. Or – and this also happens in computer centers – a fire followed by (drum roll), a flood as the fire suppression system kicks in. Should we have a plan for “the big one”, that earthquake? Sure. But most of our disaster preparation effort should plan for the much more probable disaster of fire and water.

Finally, any disaster response plan has one element which is vastly more important than any other: people. And, on November 14th, the “people” (employees) of the City of Seattle and its Department of Information Technology performed splendidly. A dozen IT professionals showed up on site within two hours (despite interference from the traffic around a nearby Seahawks football game). The computer center manager – a 44 year employee and true hero Ken Skraban – was on site and immediately in charge. Two employees set up an IT operations center with an incident commander and support staff. Several responded to the data center and shut down servers in an orderly, pre-planned, color coded (red-green-orange-yellow) fashion, with the most critical servers (for example “Blackberry” support) staying up continuously. Server administrators from every major department in City government responded on site.

And when the crisis was past and cool water was again flowing to the “crack” units, those same folks brought all services up in an orderly fashion. And there was not a single call to the help desk on Monday morning as a result of our unanticipated “summer” high.

Disasters happen. Careful planning and skilled, trained staff will always mitigate their effects.

- Tech Nightmares Frighten a CIO

Tech Horror

Sometimes my job as a City CIO keeps me up at night. There are some pretty horrible things which can happen to the technology which keeps City and County governments running. Halloween seems like a perfect time to confront a few of our most frightful fears, and here are a few of mine.

Water. And Fire. Or Fire followed by Water. In my data center. The City of Seattle has multiple data centers, but our main one, constructed in 2001, has well over $15 million of stuff in it. Things like the e-mail servers used by the entire City government, or the disk array holding all our financial data. And about 500 mid-range servers. Our data center is in one of the most modern, earthquake-resistant buildings in town. But my real fear – and much more likely than the predicted 8.0 earthquake – is a fire or a gushing water leak. I guess it’s time to test that disaster recovery plan again!

E-mail Horror

E-mail. Gosh, e-mail is the most important application we have – more important than utility billing systems or computer-aided dispatch or financial management systems. We all get an avalanche of e-mail every day, and the City of Seattle’s great Postini spam filter from Google cleans out most of the viruses and junk mail. But it is really the content of the e-mail which scares me. Like that occasional email which says “hey, we’ve decided to cut your budget for xxx (fill in the blank) by $500,000 but you still get to do the project, on time, with reduced budget” or “oh, hey, Mr. CTO, your Wi-Fi network in the University District is down. Again. And the Mayor has a public meeting there at 3:00 PM”. The only thing more frightening than some of the e-mail messages is arriving in the morning to find that the e-mail system is … ah … “down”. And down HARD!

Tablet Computer

Tablet computers. Ah the great promise of laptops and tablets! You sit at your desk, and it is a desktop computer. You unhook it. You take it to every one of your meetings so you can view documents electronically, and don’t have to print paper to take along. You take notes using Microsoft One-Note on the tablet, rather than writing stuff on paper (and, like me, promptly losing the paper in one of the giant piles in my office). You demonstrate that you are “friendly to the environment” by personally reducing your paper use by storing everything electronically. Then you forget to back the tablet up, you trip on the stairs with the laptop in your hands, and it crashes. Into the wall. Literally.

BlackBerry

BlackBerries. An extraordinary combination of the two most nefarious technologies known to humankind, the cellular telephone and electronic mail. Now you get to be available to your customers 24 hours a day, 7 days a week, 365 days a year. The real killer is discovering, one September day, that one of my last havens where most cell phones and BlackBerries didn’t work – beautiful little Republic, Washington – had been jerked into the modern era (see blog entry Fossils and Technology). The Blackberry worked there! Arggh. The only thing worse than a fully functioning BlackBerry is one which doesn’t work, so you are out of touch! Arrgh!

Mayor’s briefing. You show up to brief the Mayor and his senior staff on your latest new hotshot tech project, hoping to convince them to make a relatively small (less than a million bucks) investment. But, as you walk into the Mayor’s Office, the Dow drops 500 points, Lehman Brothers fails, AIG needs a $85 billion bailout, sales tax revenues drop precipitously right along with consumer confidence, Boeing goes on strike and the room’s technology systems go on the fritz.

The Help Desk. So you call the Help Desk (206-386-1212 for the City) about any one of the problems above, and they fix the problem over the phone. Quickly. Efficiently. Surprisingly well. And you – the Chief Technology Guy – are really frightened, because the problem was “user error” and the user is you!