- Tech Lessons from the Seattle Snowstorm

Seattle EOC Activated

The Seattle area and I just went through a four day snow/ice storm event.  The City of Seattle’s emergency operations center (EOC) was activated and coordinated the City government’s response.  That response received high marks from the public and media for a variety of reasons (see Seattle Times editorial here), including the leadership of Mayor Michael McGinn.

I was able to personally observe that response and lead the technology support of it.   Information technology materially contributed to the improved response, nevertheless I see a number of further potential enhancements using technology .  And that’s the purpose of this blog entry.

GIS GIS GIS (Maps)

Every city, county and state is all about geography and maps.  Maps are the way we deploy resources (think “snowplows”).  Maps are the way we understand what’s happening in our jurisdiction.  Everyone who has lived and traveled inside a city can look on a map and instantly visualize locations – what the “West Seattle bridge” or any other street, infrastructure or geographical feature (think “hill”) looks like.

SDOT Snowplow Map

For this storm, we have some great mapping tools in place, especially a map which showed which streets had been recently plowed and de-iced.   This map used GPS technology attached to the snowplow trucks.  That same map had links to over 162 real-time traffic cameras so people could see the street conditions and traffic.  (Other cities, like Chicago, have similar maps.)

Electrical System Outage Map

Another useful map is the electrical utility’s system status map, which shows the exact locations of electrical system outages, the number of outages, the number of customers affected and the estimated restoration times.  This is really useful if you are a customer who is affected – at least you know we’ve received your problem and a crew will be on the way.

What could we do better?  We could put GPS on every City government vehicle and with every City crew and display all that information on a map.   That way we’d immediately know the location of all our resources.  If there was a significant problem – let’s say a downed tree blocking a road or trapping people – we could immediately dispatch the closest resources.  In that case we’d typically dispatch a transportation department tree-clearing crew.  But that crew might have to travel across the City when a parks department crew with the proper equipment might be a block away. 

This same sort of map could show a variety of other information – the location of police and fire units, which streets are closed due to steep hills and ice, where flooding is occurring, blocked storm drains, as well as water system and electrical outages.   This “common operating picture”, across departments, would be enormously useful – as just one example, the fire department needs water to fight fires, and it needs good routes to get its apparatus to the fire and perhaps it would need a snowplow to clear a street as well.

Obviously we wouldn’t want to show all of this information to the public – criminals would have a field day if they knew the location of police units!  But a filtered view certainly could be presented to show the City government in action.

Perceptions and Citizen Contact

A lot of media descended on Seattle this week.  Partly that was due to the uniqueness of the storm – it doesn’t snow much in this City.  And perhaps it was a slow news week in the world.  A lot of news crews filmed inside the EOC.  The Mayor and other key department spokespeople were readily available with information.  This is quite important – the television, radio and print/blog media are really important in advising the public on actions they should take (“public transit to commute today, don’t drive”) and actions they should avoid (“don’t use a charcoal grill to cook when you are without power”).  Our joint information center (JIC) was a great success.  

Mayor McGinn’s family even contributed to this – his 11 year old son filmed him in a public service announcement about how to clear a storm drain of snow and ice which is now posted on the Seattle Channel.  

What could we do better?   We need better video conferencing technology, so the Mayor and senior leaders can be reached quickly by news media without sending a crew to the EOC.  This video conferencing would also be quite useful in coordinating action plans between departments with leaders in different locations.   In a larger, regional, disaster, such capability would allow the governor, mayors and county executives to rapidly and easily talk to each other to coordinate their work.  It is much easier for anyone to communicate if they can see the visual cues of others on the call.  

Also, Seattle, like many cities, is a place of many languages and nationalities.   We need to have translators available to get communications out in the languages our residents speak.  This might include a volunteer-staffed translation team but at least could include recording and rapidly distributing written, video and audio/radio public service announcements in multiple languages.

Commuting; Telecommuting

In these emergencies, many people elect to use public transit – buses and trains for commuting.  (I actually took my “boat” – the water taxi - to work twice this week.)  Yet snowstorms are also the times when buses jackknife or get stuck in snowdrifts and going up hills.  

In this emergency, the coordination between the transit agency (“Metro”) and the City was quite improved, because we had people – liaisons – from each agency embedded with the other.   This allowed snowplows to help keep bus routes clear and help clear streets near trapped buses.  

And, with recent technology advances and sorta-broadband networks, many workers can now telecommute.  Seattle had few outages of Internet service this week, although in suburban areas trees and snow brought down not just power lines, but telephone and cable lines as well causing more widespread Internet issues.

What could we do better?  The easiest and most useful advance, I think, would be GPS on every bus and train and water taxi boat.   That, combined with real-time mapping, would allow people to see the location of their rides right on their smartphones.  If we deployed it right, such technology might also show how full the bus is and the locations of stuck buses.  This sort of technology would be useful every day for public transit users – but is especially important during snow emergencies.

Another huge necessity – which I’ve advocated often and loudly – is very high speed fiber broadband networks.   With fiber broadband – and Gigabit (a billion bits per second), two way, telecommuting and tele-education becomes really possible.  Kids could continue their school day with video classes even when schools are closed, you could visit your doctor, and of course citizens would have access to all that emergency information and maps described above, real time and two-way.  I could go on and on about this – and I have – read it here. 

Crowdsourcing and Two-Way Communications, Cell Phones

This area is the most ripe for improved technology to “weather the storm”. 

In any emergency – even a minor disaster like a major fire or a pile-up collision – just obtaining and distributing information early and often will have a significant result in managing the problem.   On-duty at any time, the City of Seattle may have 200 firefighters, 350 police officers and several hundred to several thousand other employees.   Yet we also have 600,000 people in the City, each one of which is a possible source  of information.   How could we get many of them, for example, to tell us the snow and ice conditions in their neighborhoods?   Or perhaps to tell us of problems such as clogged storm drains or stuck vehicles?  The Seattle Times actually did this a bit, crowdsourcing snow depths from Facebook. 

How can we “crowd source” such information?   I’m not exactly sure.  Perhaps we could use Facebook apps or Twitter (although not a lot of people use Twitter).  Two-way text messages are possible.   Any one of these solutions would present a whole mass of data which needs to be processed, tagged for reliability, and then presented as useful analytics.    Eventually, of course, there will be whole armies of remote sensors (“the Internet of things”) to collect and report the information.   Perhaps everyone’s cell phone might eventually be a data collector (yes, yes, I’m well aware of privacy concerns).

In the meantime, we should have some way citizens can sign up for alerts about weather or other problems.   Many such systems exist, such as the GovDelivery-powered one used by King County Transportation.   I’m not aware of such a system being used two-way, to crowd-source information from citizens.   There are also plenty of community-notification or “Reverse 911” systems on the market.  The Federal government is developing CMAS, which would automatically alert every cell phone / mobile device in a certain geographical area about an impending problem or disaster. 

Furthermore, during this Seattle snowstorm, many City of Seattle employees – including police and fire chiefs and department heads, used text messages on commercial cellular networks to communicate with their staff and field units.   This continues a tradition of use of text messaging during emergency operations which first came to prominence during Hurricane Katrina.

All of these solutions depend, of course, on reliable cellular networks.  We know during disasters commercial cellular networks can easily be overloaded (example:  2011 Hurricane Irene), calls dropped and cell sites can drop out of service as power outages occur and backup batteries at the sites run out of juice.   Yet, for people without power or land-line Internet, a smartphone with internet is a potential lifesaver and at least a link to the outside world.  I’d like a way to easily collect this information – privately – from the carriers so emergency managers would know the geographies where mobile networks are impacted. 

This leads me, of course, to my final point – that we need a nationwide public safety wireless broadband network.  Such a network would be built using spectrum the Congress and the FCC have set aside for this purpose.  It would only be used by public safety, although – as our Seattle snowstorm underscored, “public safety” must be used broadly to include utilities, transportation and public works – even building departments.  And it would be high speed and resilient, with 4G wireless technology and backup generators, hardened cell sites. 

These are a few of my thoughts on better management, through technology, of future snowstorms and other disasters, large and small, both daily and once-in-a-lifetime ones.   What have I missed?

- Thanks & Turkeys 2010

This week Chief Technology Officer Bill Schrier has a LOT for which to be thankful.   But I also have a few turkeys to carve.

My most significant thanks go to the phenomenal people who work in information technology in local government, especially here at the City of Seattle.    Most City and County CIOs, such as those who are the 60 members of MIX (the Metropolitan Information Exchange)  will agree with me and give thanks for their employees as well.  While some members of the public think government employees are 8 to 5 clock-watching bureaucrats, that’s decidedly NOT true of most employees, especially our technology workers.

This fact slammed home to me again this week – Seattle had a snowstorm.    Two inches.    Those of you in Chicago, Boston or Washington DC are probably laughing.  Two measly inches?  What’s the big deal?  But here in Seattle, because of the uniquenesses of our weather systems/geography and the rarity of snow in the lowlands, it was a real show-stopper.  Monday night many of my employees spent four, five or nine hours commuting home on jammed icy freeways.   I and several of my staff walked home five miles in the snowstorm (video of commuters walking across the West Bridge here).

In Seattle’s Department of Information Technology, we had staff who worked all night Monday, or slept at their workstations Monday night, or stayed in hotels downtown, or turned right around and came back to work Tuesday morning after the long commute home.    They did this because they know the work of a City government and the safety of the people of Seattle depend now, more than ever, on reliable technology:  websites, data networks, e-mail systems and much much more.   For these two hundred dedicated people working in the City of Seattle’s technology department, I give thanks.

(My colleagues elsewhere have similar stories, whether in Houston and Mobile, Alabama, who have suffered through hurricanes, or Los Angeles and Riverside who have suffered through earthquakes, or Chicago and Washington DC, with their snowstorms.)

As I attend conferences and talk to my counterparts across the country, I find similar dedication to keeping the public safe and our governments operational. As just one example, we have twenty cities and states around the nation who have authority from the FCC to build fourth generation wireless networks.  Over the past 11 months I’ve been working with officials from these twenty jurisdictions, as well as the FCC’s Public Safety and Homeland Security Bureau, the Public Safety Communications Research Program of the Department of Commerce, and Homeland Security’s Office of Emergency Communications.   Every one of these agencies and the people involved have been working tirelessly to build a nationwide public safety network, a vision which sprung out of the September 11th World Trade Center disaster.     This year we’ve made real progress, despite a number of hurdles.  Now the first networks are under construction.   For all these dedicated government officials and technical staff, I give thanks.

I also give thanks to the many private companies who are doing extraordinary work with technology – Microsoft and Windows and Office, Google with Android and search, Apple with iPhones and iPads, IBM’s Smart Cities Challenge, and a few more who not only want to make money, but also want to use a significant part of that money make the planet a better place in which to live and work.

Finally, I give thanks for my elected officials – Mayor and City Council – and the department directors running City departments here in Seattle.     This year of the Great Recession they have faced terrible choices with budget shortfalls of $67 million in Seattle.  And precipitously falling tax revenues.  And urgent needs from the public for safety nets for our jobless citizens and the poor and homeless.   My own department’s budget was cut by over 17% and I’ve laid off over 10% of my workforce over the past two years.    These are all tough choices, and they are done in the glare of publicity with many competing demands by constituents for the ever-shrinking pot of money.  But we have a sustainable budget and services going into 2011.  Thank you to the officials who stepped up and made these tough choices.

Now on to the turkeys – at least the ones I’d like to carve and serve.

First are some of our technology vendors, a few of whom have ever increasing appetites for money.   Some of them are resorting to “compliance audits” to make sure we are paying for every last danged software license we are using.  One vendor even demanded to have access to every one of the 11,000 computers at the City of Seattle to see if their software was installed.   Others absolutely refuse to negotiate reduced pricing or flexible maintenance plans.  These few money-grubbing vendors get my “tech turkey” award.

Next there are a few of our public employee unions.   Many public employee unions here in the Seattle area realize we are in an unprecedented recession.   Those unions have willingly forgone raises which were in their contracts, understanding that few workers in the private sector get raises, and many private sector workers have lost their jobs and retirement money.   But a few public sector unions have held out for their contracted raises, which are far larger than inflation.  This, frankly, can make all city and county governments and our workers look greedy and foolish.  The public backlash was evident in our recent elections where few tax increases were passed and many revenue sources were cut.  These few unions get my turkey award as well.

My final turkey award goes to those politicians who want to whip the public into a frenzy about supposed fraud and waste in government, or think we can continue tax cuts, increase defense spending, and balance the budget all at the same time.  How do they think public schools, parks, police and fire departments, child protective services, streets or public health are funded, or how do we pay the dedicated people who provide all those services?   I’ve blogged about this at length before, and will just leave these politicians with my tea-party-turkey award.

All in all, however, at this Thanksgiving of 2010, I’ve got a lot more reasons to give thanks than to carve!

- Tech Terror List 2010

Tech Terror 2010

We are at the end of this quite frightening year of the Great Recession 2010, and at the eve of another frightening Halloween filled with tiny goblins ringing the front door shouting “trick or treat”.  But what are the “real world” goblins knocking at the door, and facing the chief information officer who opens it?

It is time for me to update my 2008 list of nightmares which frighten a CIO. 

Tablet computers (and smartphones). Tablets were on my 2008 list, but are on my 2010 list for an entirely different reason. They’re everywhere. They’re invading! They’re unmanageable.  And every employee wants to use their own. CIOs have long practiced the mantra of standards standards standards.

You need a computer? Yup, we give you a standard HP model with Windows XP, Office 2007 and Anti-Virus loaded on it. You need a smart phone to do your job? Yup, here’s your BlackBerry connected to Outlook and locked down from installing any dangerous applications which present a security threat.

Grinch CIO?

All of a sudden it’s “HELLO Mr. CIO” – the iPhone explodes on the scene, then it’s the iPad, and soon it will be the Windows Phone 7 and Android phones and the RIM Playbook and the Samsung Galaxy. And employees LIKE them and wonder why Mr. CIO is the Grinch and won’t connect them to e-mail and the network.

But of course the Outlook sync doesn’t work exactly right on the iPhone and appointments get dropped. Oh, and someone loses their Android smart phone with all the home and cell numbers of half the police command staff but gee we can’t remotely wipe its contents because installing the remote wipe software is the bureaucratic Sign of the CIO Grinch. Oh, and all of a sudden a public disclosure (FOIA) request comes in and the employee needs to cough up all the documents and messages on their personal iPad, even though some of them are quite personal or even relate to the employee’s personal business or political activity. And oh, gee, by the way, the employee “forgot” to back up all those docs on that personal device, violating not just the public disclosure act but the records retention act as well.

In the meantime, the budget of the IT department has been cut 13.3%, and I’ve laid off 5% of my workforce, but still we’re the Grinch because we can’t support this exotic stew of personal devices.  Arggh!

(I’m convinced we’ll eventually support personal smart phones and tablets, but we need better tools and more staff. For 2010, they remain on my Tech Terror Watch List.)

Cyberterrorists and Malware. There is much new to fear on this front in 2010. There is the Stuxnet worm, apparently written by a nation to infiltrate and damage Iran’s nuclear program, but sophisticated enough to attack many industrial or electrical control systems, and hard to find and eliminate. This is only the tip of the iceberg of a new set of computer viruses and malware written by nation-states to attack each other.

Then there was a rash of Trojan viruses and keystroke loggers which infiltrated some government and school sites.  These viruses stole passwords for financial employees at these firms, and those passwords were used to steal hundreds of thousands of dollars.

And there is the appearance of malware on legitimate websites, so even innocent employees doing their job on the Internet could get their computers infected.  Cyber threats go onto my Terror Watch List.

Stop and Think, before Connecting (and also have a good firewall and anti-virus program!).

Smart Phone Apps. One problem with Smart Phones is that anyone can write an app for them, including criminals, hackers and cyberterrorists. Apple, at least, reviews and tests Apps before allowing them into the iTunes store. Such testing doesn’t happen for BlackBerry or Android apps. I really hope Microsoft does thorough testing on its Windows Phone 7 apps before releasing them into the wild. Smart Phone Apps go onto my Terror Watch List.

Water. And Fire. These remain on my watch list for the same reason as in 2008 – a broken water pipe or a fire in my data center can put it out of commission for a considerable length of time. But this year there is hope – “the cloud”. And no, the Cloud doesn’t rain on the parade of my technology. It means that many of our services and applications might eventually live in the Cloud of servers and storage in distant data centers, much less susceptible to earthquake, fire, water and other disasters.

Speaking of Fire, I have a very recent story to relate. Early on Sunday morning, October 17th, someone started a fire (probably to keep warm, as it is a place homeless are often found) behind a rented City building near 3rd and Main. That fire raced up a conduit burning through fiber and copper cables, bringing down phone and data network services to Seattle’s Fire Administrative Headquarters and main transportation department dispatch center.

The outstanding information technology staff of my department, with support from cabling contractors and Fire/Transportation department staff restored most services within 18 hours, but it illustrates why Fire remains on the Watch List.  And why skilled, dedicated, employees are the best defense against such terror.

Customer expectations. Most terrifying of all is the rise of customer expectations in the midst of the Great Recession, falling IT budgets and reduced staffing. Government employees use computers at home, use tablets and smart phones. They bank online, download apps, text message and use Facebook and blogs. But with reduced technical staff, plus a whole series of requirements like HIPPA and CJIS and the public disclosure act, the CIO Grinch has fewer and fewer resources to meet the expectation that those same tools and applications can all be used at work.

On Halloween, 2010, it is those increased expectations which really terrify me as a CIO.

- Why Don’t Cops just use Cell Phones?

The National Plan

Police officers and firefighters carry $5000 radios.  Local and state governments spend hundreds of millions of dollars to build public safety radio networks.  Yet, today, cell phone networks seem to be everywhere, most people carry a mobile phone and many of us think paying $199 for an iPhone is expensive.  

Why can’t cops and firefighters and emergency medical technicians (EMT) use cell phones like everyone else?   A Washington State legislator from Seattle recently public argued for this approach in his blog.  And, at first, this appears to be a simple way for governments to save a lot of taxpayer dollars.

Here are a few reasons public safety officers need their own dedicated networks:

1. Priority.  Cellular networks do not prioritize their users or traffic.   A teenager’s cell phone has the same priority as a cell phone used by a police officer or, for that matter, the BlackBerry used by President Obama.  We’ve all experienced “no circuits available” or “network busy” when using a cell phone.  When I’m being assaulted or have been injured in an automobile accident or even have had my house burglarized, the last thing I want is to have the network be “busy” so a police officer or EMT couldn’t be dispatched.   Public safety needs dedicated frequencies where police officer sand firefighters have priority and even, perhaps, exclusive rights to for use, without calls being clogged by the public.
2. Reliability.  Seattle’s public safety radio network, part of the larger King County-wide 800 megahertz public safety radio network, handles more than 60,000 police, fire  and emergency medical calls every day.  It operated last year with 99.9994% reliability – that’s about 189 seconds of downtime out of more the than 31 million seconds which composed the year 2009. On the average, only about five out of the 60,000 calls were delayed for any reason, and even then the average delay was about two seconds.  What cell phone network has that kind of reliability?   How many times have you experienced “no service” or “call dropped” with your cell phone?   Do we want firefighters who are reviving a heart attack victim and talking to the emergency room on the radio to all-of-a-sudden have their call dropped?  Or should police officers lose service when drunk drivers clog the roads and bars are closing at 2:00 AM because a cell phone company decides to do maintenance because “no one uses the network then”?
3. Disasters.  Even small disasters cause cell phone networks to collapse.   In Seattle, we’ve had swat team actions or car accidents which have shut down a freeway.   Suddenly cell phone service abruptly ceases in that area because EVERYONE is on their phone.  A few years ago a rifleman was loose and shooting people in Tacoma Mall.  Responding police and EMTs had communications because they had dedicated networks and frequencies, but again cell phone networks were overloaded and down.   In a larger disaster such as an earthquake or hurricane (with associated evacuation of large cities), commercial networks will be overloaded or jammed for days by people trying to escape the affected areas. Do we want police and fire departments – or even transportation, electric utilities and public works departments – to be trying to use those same networks while they are are responding to the disaster? I don’t think so.
4. Talk-around. A key feature of most government-operated networks is something called talk-around or simplex or “walkie-talkie” mode. In this mode, individual radios talk directly to each other, without using a radio or cell tower. This is very important at incident scenes – firefighters commonly use it at the scene of a fire, because the radios will operate at the scene even if there isn’t a tower nearby. But this NEVER a feature of cellular phone networks. If the cell tower is down or out of range, that cell phone in your hands is a useless lump of plastic. But the radios of publicsafety officers still work and will talk to each other even without the tower.
5. Ruggedness. No firefighter in his/her right mind would fight a fire using a cell phone for communications. The heat, water and ruggedness of the environment would quickly destroy the device. Yet most public safety radios will survive being dropped repeatedly on the ground or being immersed in water for 30 minutes or more. No standard cell phone can survive the rigorous work of firefighting or policing.

Are there problems with the current dedicated public safety networks? Absolutely. The use proprietary technologies, for example “Project 25“. Theoretically all “Project 25″ radios work on any “Project 25″ radio system. But only a few of those are deployed around the nation. These proprietary technologies are one reason the radios cost up to $5,000 each.

Representative Carlyle, in his blog, proposes that we deploy “Tetra” radios for public safety. While Tetra is common in some parts of the world, it is not used at all in the United States. This is a dangerous proposal, because it means Tetra networks we buy would not work with the equipment used by any other government or telecommunciations carrier anywhere in the United States. If called to respond to a diaster overseas, we could talk to firefighters in Hong Kong or the police in Ireland, however.

Another problem we face is the small market – the total market for public safety is perhaps 10,000,000 radios which are replaced, say, once every 10 years. On the other hand, the cell phone market is huge – 260 million cell phones replaced every two years in the United States alone. The economies of scale means consumers will have a lot more choice, and their cell phones will be relatively cheap.

So is there some way to reduce the sky-high cost of these dedicated public safety networks while at the same time not endangering cops, firefighters, EMTs and the public in general?

Absolutely. The FCC, in its national broadband plan, and the federal Department of Commerce, with its forward-thinking grant program for broadband, are lighting the way for a new public safety network which will be more robust, national in scope, and interoperable. By “interoperable” I mean the new public safety equipment will probably operate almost anywhere in the nation, wether on a dedicated government network or on a commercial cell phone network. Here are some features of the new networks:

• The FCC and major public safety organizations have called for the new public safety networks to be built using a fourth generation (4G) technology called LTE – long-term evolution. Not coincidently, this is the same technology which will be used by the major cell phone companies Verizon and AT&T when they construct their 4G networks. The commercial networks will operate on different frequencies than the public safety networks, but they will all be built in same general area of the wireless spectrum – the 700 megahertz (MHz) band.
• Because they are all using the same technology (LTE) and are in a similar slice of radio spectrum (700 MHz) potentially they will all interoperate. That means that public safety officers will use the government networks and frequencies when they are within range, but could “roam” to a commercial network if necessary. So cops and firefighters will have the best of both worlds – coverage from dedicated government networks and coverage from multiple private carriers. The FCC is even considering rules which would require the commercial companies to give public safety priority on the commercial LTE networks.
• Because everyone – consumers, cops, firefighters and even general government workers such as transporation and utilities – are all using LTE, constructing the networks can be much cheaper. Commercial telecommunications carriers could put government antennas and equipment at their cell sites, and vice-versa. Perhaps the network equipment at the cell site, or even the central switches could be shared as well. Public safety will still be using its own frequencies and have priority, but could share many other network elements.
• And the radios used by individual public safety officers or placed in police vehicles and fire trucks can be much cheaper as well. Because manufacturers are all making equipment for the same technology – LTE – it could cost just a few hundred dollars. Again, there will be specialized and ruggedized devices for firefighters and others working in punishing environments, but the “innards” – the electronics – will be much less expensive.
• Next, we have to get all first and second resopnders to use the same or common networks. Here in Washington State, for example, we have multiple overlapping and duplicate networks. City and County police and fire in the region have one network, each electric utility (e.g. Seattle City Light) have another network. Transportation departments have their own networks (e.g. Seattle Transportation and Washington State Transportation each have their own separate network). The Washington State Patrol has its own separate network. The State Department of Natural Resources has its own network. Fish and Wildlife has its own network. And federal government agencies (FBI, cutoms and immigration) have their own networks. This is patently stupid and expensive. As we build these new fourth generation LTE networks, we need to build a single network with lots of sites and a lot of redundancy and hardening to withstand disasters. And everyone – first and second responders from all agencies – should use it.
• Finally, and perhaps most importantly, all the networks will be nationally interoperable. The lack of communciations interoperability was a major finding of the Commission which investigated the September 11th World Trade Center attack. But with these new networks, a Seattle police officer’s 4th generation LTE device will also work on New York City’s LTE network or New Mexico’s :LTE network or on any Verizon or AT&T network anywhere in the nation. As disasters happen anywhere in the United States, and first and second responders are rushed to the scene of the disaster, they can take their communications gear with them and it will work.

The City of Seattle is one of a handful (about 20) forward-thinking governments leading the way to deploy these new networks. Seattle’s public safety LTE network, hopefully launched with a federal stimulus grant, will eventually expand throughout the Puget Sound region and across the State of Washington. The State of Oregon also has authority and a grant request to build an LTE network, and we are working with Oregon to make sure our networks work with each other seamlessly.

Is all of this a pipe dream? I don’t think so. A number of public and private companies, governments and telecommunciations carriers and equipment manufacturers are working together to realize it. Many of them are in the Public Safety Alliance. In the Federal government, the FCC is working with the National Institute of Standards and the Departments of Commerce and Homeland security are providing grant funding. It will take a lot of work and many years to realize this network.

But when it is finished, we’ll have public safety networks which work to keep us safe, and consumer networks which work to keep us productive and linked to our friends and families. These networks will be separate yet connected. They will be built from common technologies. And they will be less expensive for taxpayers than the networks we have today.

- PITS Computing

Non-Computing Fads

There are fads and trends in information technology, just like in the world of clothing or hairstyles. One of the latest fads is pie-in-the-sky computing (PITS), otherwise as “cloud computing” or software-as-a-service – SAAS (pronounced as in “sassy”).

But I’ll call it pie-in-the-sky (PITS) computing, just to be different and even a bit contrary.

PITS computing is only the latest in a long line of sea-changes in IT. Electronic data processing (EDP – now there’s an old term) was the very first of these trends, appearing on the scene in the 1950s and 1960s. EDP was a world of punch cards and paper tape. EDP was the era of “glass house” data centers and a computer “priesthood”. Computers were far too expensive and esoteric for normal human beings to comprehend or touch. So there was a “priesthood” of specially anointed and trained computer specialists whose job was the programming, care and feeding of the electronic monsters.

But the development of computing technology continued relentlessly. Along came mainframe computing (green-screen). personal computing, local-area-network computing, client-server computing and Internet or web computing.

Each one of these phases was driven by some significant technological advance. The development of microchips and the Intel 8088 processor, for example, drove the personal computing trend. (Thank you Intel and IBM!) The development of Ethernet standards drove networking which allowed individual computers to talk to each other.

And then computing, of course, became part of the mainstream culture. Any human being in a developed country knows “windows” doesn’t refer to that wonderful device for seeing through walls, the “glass window”, but rather the portal into the world of computers, an operating system developed and marketed by Microsoft. And almost no one thinks of the “web” as a home for spiders or the “net” as a tool for catching fish or butterflies.

In this context, PITS is the latest fad in computing and technology. PITS is driven by the appearance of more-or-less ubiquitous and reliable high speed networking. Networks today, thanks to fiber optic cable, the router/switch revolution (thank you Cisco) and advances in wireless (wi-fi and 3G telecomm networks), are virtually everywhere. Or at least everywhere where human beings live and companies and governments do significant business.

And these networks are reliable. The wired networks almost never go down, although the signal can get weak or strange with wireless. In my house for example, our Wi-Fi network connected to a wired DSL Internet connection has 105 megabits per second of throughput. Yet my commercial telecomm provided cell phone only works at a certain specific spot in the kitchen in front of the microwave!

Most enterprises now operate with giant central servers which store data and applications. At the City of Seattle, for example, we have computer aided dispatch systems which reside on central servers at a “highly secret” police department location. The police data resides there, but cops on the street can access criminal records and license plate information which reside not only in Seattle but also on the other side of the nation or even on another continent.

Our water utility manages pumps and valves and dams and reservoirs across the entire county and up into the Cascade mountains. City Light, our electrical utility, manages an electrical grid which spans the entire state of Washington.

We all routinely use the web to find information and read the news. But we also increasingly use it to store spreadsheets or photos or documents on our own websites or using servers such as Google apps. Microsoft is embracing the trend, with its Office 2010 now available “for free” in a PITS cloud.

So if Microsoft Office can be in a “cloud” somewhere on the Internet, why can’t our payroll system or e-mail system or financial management system be halfway across the State in a data center in Grant County, Washington (next to giant hydroelectric dams to supply the power) or even halfway across the United States, well outside the Seattle earthquake disaster zone?

Of course the applications and data can be almost anywhere. In the past, I’ve been skeptical of PITS / cloud computing because I didn’t trust the networks to stay up in a disaster, and I was concerned about the security of information stored in a non-descript data center in a distant location outside my personal control.

But with today’s reliable networks, the network is not the issue. And major companies like Microsoft or Amazon or Google handle the management and security better than most governments or small businesses. Furthermore, as demonstrated by the World Trade Center disaster, the data could actually reside in multiple different locations around the nation, increasing our ability to withstand a disaster like that 8.0 magnitude earthquake.

It will be a while before we in government embrace PITS, because the loss of control is a big cultural change for governments and many large companies to swallow. Just like people were concerned when their data moved off their desktop computers to a server, and servers moved out of the closet on the same floor to a centralized computer center in the government complex, so it will take us some time to embrace having those computers in an unnamed nondescript but super-secure location, possibly right next to the bunker where Vice-President Dick Cheney hung out after September 11th.

But embrace it we will.

- UASI, Bureaucracy and Terror

Homeland InSecurity

This week I was, in turn, amused, maddened and fasincated by disaster. I had the opportunity to do both leadership and followership as officials from the “Seattle Urban Area” considered projects to submit for grant funding from the fedgov Department of Homeland Insecurity.

This whole process is a look into the little-known culture of “homeland insecurity” which has blossomed over the last 7 years.

Let me say, right at the beginning, that September 11th, 2001, was a horrific event and a real wake-up call for the United States. We’ve been taught the same lesson we learned on December 7th, 1941 – two huge oceans do not make for a nation secure from the enemies of our way of life. My somewhat humorous tone in this blog is not meant to make light of the serious threats we face.

But what is our major response? Pretty typically, we create a giant bureaucracy, slosh a lot of political rhetoric and some money around, and get prepared to fight … the last war.

The major bureaucracy is, of course, the Department of Homeland Security, or Insecurity, as I’ve named it, because it heads an industry and set of programs based upon the fear and feelings of insecurity of the American People. DHS is really a myriad of individual bureaucracies and programs, sometimes working at cross-purposes and often duplicating effort.

But I stray from the subject – the Urban Area Security Initiative (UASI), usually pronounced You-Ah-Zee. In 2009 about $1.8 billion will be granted to urban areas through a series of programs as shown here. The UASI grants are made to “urban areas”, a carefully defined (by the fedgov) set of metropolitan areas which share vulnerabilities and threats. The “Seattle Urban Area” is the City of Seattle, its surrounding King County, and the counties to the north and south. The Seattle urban area is eligible for $11 million, more or less, of that $1.8 billion pie.

Now $11 million may sound like a lot of money, and it is. But think about the Seattle Urban Area – a population of 3 million people, two major ports (Seattle and Tacoma) into which ships with nuclear bombs or bioterrorism agents or chemical weapons could sail, three major military bases (Naval Station Everett, McChord Air Force Base, Fort Lewis), and vulnerability to a potential magnitude 8.0 earthquake. We probably have 12,000 cops and firefighters (Seattle alone has 2,200), each of which needs radios and hazmat gear and training. All of a sudden, $11 million doesn’t sound like a lot, and it isn’t.

Its also a little amazing that places like Vermont and Wyoming – not exactly your typical terrorist or disaster targets – get $6 million each. But that’s the reality of politics in our 2009 world with two senators from every state and a lot of dollars to be allocated.

Each year, a group of officials from the emergency management, law enforcement, firefighting, public health and technology disciplines – the officials responsible to prepare the region for disasters and terrorist events – get together to consider projects for UASI funding. We did that this week in the Seattle Urban Area. We have a strategy to guide us but it is still a give-and-take to determine which projects are most importnat to the region.

I lead the subgroup of technical and operational staff responsible to consider and prioritize interoperable communications projects. These are the networks and systems which allow first, second and third responders to communicate with each other and with their dispatch centers to prepare for and respond to disasters large and small. They include handheld and vehicle-mounted radios, computer and telephone systems, and fiber optic networks.

Small disasters are the ones which occur every day – such as when I fell off my bicycle, shattered my arm and called 911 for help. The large ones are the fires, landslides, earthquakes, and (God forbid) terrorist events which may afflict our region.

Finding projects which prepare us for those disasters and which fit into $11 million is daunting. In the past we’ve procured fireboats and helicopters and personal protective equipment. We’ve found money for training our responders, buying radios and for public outreach and disaster preparedness campaigns. It is hard work to winnow 100 proposed projects and $40 million in needs down to a dozen or so projects with $11 million or so, but we mostly did it this week. In future blog entries I’ll talk about some of those projects – the ones I can reveal without making us more vulnerable to terrorists (and that is most of them).

In the meantime, I’m just proud of the emergency managers, fire and police officials, and technical staff I was able to spend time with this week, prioritizing projects to keep the Seattle urban area safe.

And I’m tired!

- The 108 Degree Data Center

The 108 Degree Data Center

November in Seattle is always cool and rainy and sometimes stormy – windstorms, that is. Seattle’s all time high temperature – for any day of the year – is 100 degrees. That all time high is, of course, outside. But it reached 108 degrees here on Sunday November 16th. Inside a data center. The City of Seattle’s data center.

To make a short blog entry even shorter, I’ll skip to the root cause: a failed power breaker on a pump for the domestic water supply to the building housing the data center. The water supply flows to CRAC (“crack” or computer room air conditioning) units which, in turn, cool the data center. For HomeCity Security reasons, I won’t reveal the actual location of the data center, but let’s just say it is in a downtown 60 story skyscraper which also houses about 3500 office workers during the week. The problem started about noon and was fixed at about 8:00 PM.

The data center holds about 500 servers, storage systems and other equipment. We shut down a lot of servers and many services starting almost immediately. Nevertheless the temperature in the data center rose to that toasty 108 degrees, setting a new record high (sort of) for Seattle.

So why is this notable? For two reasons: the problem and the response.

In terms of the “problem”, let me assure you (especially if you live in Seattle) that cooling problems like this will be rare to non-existent in the future. Years ago we installed a one megawatt generator for backup power. This year we’ve been working a project to install “dry coolers”. These aren’t really “dry”, but the water cooling the data center will flow in a “closed loop” between the new coolers and the center, so we’ll no longer be dependent on external water or power supplies. Unfortunately, the dry coolers don’t come online until January, which is why we went to 108 degrees last Sunday.

But there’s a more general issue here – every city and county government has data centers and servers and vital information. Every area of the country is subject to some sort of a disaster and every government needs to have a backup and recovery plan.

But for what disaster should we prepare?

Here in Seattle, everyone is concerned about the “big one” – a magnitude 8.0 earthquake. While we need to be ready for an major earthquake, we have about one of those “big ones” every 300 years. Much more likely are disasters like last Sunday – a failure of water and cooling, a “meltdown” if you will (non-radioactive, however!). Or perhaps the disaster will be the opposite – too much water from a broken pipe, and a flood drowning those servers. Or – and this also happens in computer centers – a fire followed by (drum roll), a flood as the fire suppression system kicks in. Should we have a plan for “the big one”, that earthquake? Sure. But most of our disaster preparation effort should plan for the much more probable disaster of fire and water.

Finally, any disaster response plan has one element which is vastly more important than any other: people. And, on November 14th, the “people” (employees) of the City of Seattle and its Department of Information Technology performed splendidly. A dozen IT professionals showed up on site within two hours (despite interference from the traffic around a nearby Seahawks football game). The computer center manager – a 44 year employee and true hero Ken Skraban – was on site and immediately in charge. Two employees set up an IT operations center with an incident commander and support staff. Several responded to the data center and shut down servers in an orderly, pre-planned, color coded (red-green-orange-yellow) fashion, with the most critical servers (for example “Blackberry” support) staying up continuously. Server administrators from every major department in City government responded on site.

And when the crisis was past and cool water was again flowing to the “crack” units, those same folks brought all services up in an orderly fashion. And there was not a single call to the help desk on Monday morning as a result of our unanticipated “summer” high.

Disasters happen. Careful planning and skilled, trained staff will always mitigate their effects.

- Tech Nightmares Frighten a CIO

Tech Horror

Sometimes my job as a City CIO keeps me up at night. There are some pretty horrible things which can happen to the technology which keeps City and County governments running. Halloween seems like a perfect time to confront a few of our most frightful fears, and here are a few of mine.

Water. And Fire. Or Fire followed by Water. In my data center. The City of Seattle has multiple data centers, but our main one, constructed in 2001, has well over $15 million of stuff in it. Things like the e-mail servers used by the entire City government, or the disk array holding all our financial data. And about 500 mid-range servers. Our data center is in one of the most modern, earthquake-resistant buildings in town. But my real fear – and much more likely than the predicted 8.0 earthquake – is a fire or a gushing water leak. I guess it’s time to test that disaster recovery plan again!

E-mail Horror

E-mail. Gosh, e-mail is the most important application we have – more important than utility billing systems or computer-aided dispatch or financial management systems. We all get an avalanche of e-mail every day, and the City of Seattle’s great Postini spam filter from Google cleans out most of the viruses and junk mail. But it is really the content of the e-mail which scares me. Like that occasional email which says “hey, we’ve decided to cut your budget for xxx (fill in the blank) by $500,000 but you still get to do the project, on time, with reduced budget” or “oh, hey, Mr. CTO, your Wi-Fi network in the University District is down. Again. And the Mayor has a public meeting there at 3:00 PM”. The only thing more frightening than some of the e-mail messages is arriving in the morning to find that the e-mail system is … ah … “down”. And down HARD!

Tablet Computer

Tablet computers. Ah the great promise of laptops and tablets! You sit at your desk, and it is a desktop computer. You unhook it. You take it to every one of your meetings so you can view documents electronically, and don’t have to print paper to take along. You take notes using Microsoft One-Note on the tablet, rather than writing stuff on paper (and, like me, promptly losing the paper in one of the giant piles in my office). You demonstrate that you are “friendly to the environment” by personally reducing your paper use by storing everything electronically. Then you forget to back the tablet up, you trip on the stairs with the laptop in your hands, and it crashes. Into the wall. Literally.

BlackBerry

BlackBerries. An extraordinary combination of the two most nefarious technologies known to humankind, the cellular telephone and electronic mail. Now you get to be available to your customers 24 hours a day, 7 days a week, 365 days a year. The real killer is discovering, one September day, that one of my last havens where most cell phones and BlackBerries didn’t work – beautiful little Republic, Washington – had been jerked into the modern era (see blog entry Fossils and Technology). The Blackberry worked there! Arggh. The only thing worse than a fully functioning BlackBerry is one which doesn’t work, so you are out of touch! Arrgh!

Mayor’s briefing. You show up to brief the Mayor and his senior staff on your latest new hotshot tech project, hoping to convince them to make a relatively small (less than a million bucks) investment. But, as you walk into the Mayor’s Office, the Dow drops 500 points, Lehman Brothers fails, AIG needs a $85 billion bailout, sales tax revenues drop precipitously right along with consumer confidence, Boeing goes on strike and the room’s technology systems go on the fritz.

The Help Desk. So you call the Help Desk (206-386-1212 for the City) about any one of the problems above, and they fix the problem over the phone. Quickly. Efficiently. Surprisingly well. And you – the Chief Technology Guy – are really frightened, because the problem was “user error” and the user is you!