- 1999, an Odd Odyssey

The Year 2000 Bug

It was just ten short years ago that many of us were preparing to celebrate New Year’s Eve – by working all night!

Anyone over 30 probably still remembers all the information technology work that went into preparing for Year 2000.

I’m going to dredge (!?) up some of my memories in the next few paragraphs, but if you have memories or stories of that frightening December 31, 1999, evening, I invite you to leave them as a comment to this blog entry.

For many of us in Seattle, 1999 was not a good year.

First of all, we had madly been reviewing and fixing our information technology applications and programs and systems for Y2K bugs.

But no one really knew what would happen.  Would buses and trains stop dead due to bugs in their microchips?  Would the electrical grid fail?  Would 911 stop working?

The City of Seattle, like any organization using IT, had very real problems – we knew the accounting/financial database – called SFMS for Seattle Financial Management System – was not ready for Y2K, so we replaced it with an entirely new system.  We also patched up the water utility’s and electrical utility’s billing systems, since another project to replace them was in progress. (That system, now called CCSS for the Consolidated Customer Service System, was implemented in 2001, a year late and $14 million over budget, which is a different story).

The City’s Chief Technology Officer was Lynn Jacobs, and in 1998 she had spread the alarm about Y2K, galvanizing the Mayor, City Council and most departments into action looking for their Y2K bugs.  But by October, 1999, Jacobs had largely checked out due to personal issues, rarely coming to work and exerting virtually no leadership.  So Mayor Schell replaced her with Marty Chakoian, who was, not coincidently, leading the City’s Y2K efforts. There was plenty of consternation among the IT leadership in the City government.

But the outside world was in chaos in 1999 too.

The Seattle Times ran a whole series of articles about the electrical grid and 911 systems and other critical functions, and how we were preparing them for Y2K. Gee, they even talked about potential water systems’ issues with Y2K, even though Seattle’s water reservoirs are high up in the mountains and the basic rule of water and wastewater is “s___ flows downhill” (The s___ stands for “stuff”, of course).

And we had the WTO riots in Seattle in November; Seattle sure appeared to be the anarchy capital of North America, if not the world.

Then on Dec. 14, 1999, a 32-year-old Algerian named Ahmed Ressam was arrested in Port Angeles, Washington, coming across the border from Canada with 100 pounds of powerful explosives in the trunk of his car.  Was he headed to Seattle to detonate the explosives at the base of the Space Needle on New Year’s Eve?  We couldn’t take a chance, so Mayor Paul Schell cancelled the grand New Year’s celebration planned there.

For most of us tech types, and a lot of other folks, it didn’t make any difference, anyway.  We had already planned to be at work instead of celebrating on December 31st.

The City’s Emergency Operations Center was open.  At that time, the EOC was in a crowded basement of Fire Station #2 in the Denny Regrade (it has since been replaced with a $30 million modern facility).  Nevertheless, senior officials from every department hunkered down to see in the millennium in that basement.

My own Department of Information Technology was all of 5 months old – we were created as a separate department on August 1, 1999. Our operations center was in an old stock brokerage (Foster and Marshall) building at 2nd and Columbia, which is now home to the United Way of Seattle. That building was home to the telecommunications division, including the service desk – the rest of the department was in the Dexter Horton building next door. [The Dexter Horton building turned out to be much worse off in the earthquake of 2001, when virtually everyone working there was forced to leave it for a couple weeks due to building damage, but again that's another story.]

City of Seattle IT Staff celebrate Year 2000

On December 31, 1999, we had a whole team of folks who celebrated the beginning of the third millennium* together, watching a quiet, uneventful Seattle 20th Century night turn into a quiet, uneventful and sleepy 21st century* morning.

Was it uneventful due to all our diligency and preparations, or was there never really any problem in the first place?  I don’t know, but I do know I’ll celebrate the end of the decade of the naughts tonight with a bit more enjoyment and a lot less trepidation.

---

*Note: Yes, yes, I do understand the real beginning of the 3rd millennium and the 21st century is January 1,2001. See article here. But, gee, popular culture doesn’t count the years that way, so I took a little tech-journalism-geek liberties with dates in writing this article.

- Higher Tech Policing

Dubuque Police Department

A long time ago in a city far far away I was a street cop. A police officer working the beat. It wasn’t a large city – Dubuque, Iowa – 65,000 people and probably 60 or 70 policemen. Yes “policeMEN”. The first women were hired into the Dubuque PD while I was there, and I – at 5′ 9″ and 170 pounds – was one of the smallest cops on the force.

In those days, technology was not really part of an officer’s life. Times have changed, they REALLY have changed. The Seattle Police Department has just implemented a new Computer Aided Dispatch (CAD) system which is fundamentally altering policing at the City of Seattle – the “SPIDER” project. Technology is now – literally – at the right and left hand of virtually every cop – and firefighter and emergency medical tech.

When I was on the street, my primary technology was the radio in my police cruiser. The voice radio was (and still is) the lifeline for public safety officers on the street. But, in the 1970s, when I walked out of the car, I also walked away from that lifeline. We didn’t have handheld or portable radios, nor did cell phones exist. If there was a problem when we were away from the car, we depended upon each other to “drive by” and check on us (and cops still do that), or on a citizen to use a land-line telephone to notify dispatch. That was scary.

Now police officers carry a handheld radio, and a lapel mike, and every Seattle radio has an emergency button which, when pressed, alerts dispatch center that the officer is in trouble. The emergency alert triggers a display of badge number on the dispatch console. The radios can communicate with officers throughout the region. And automatic vehicle location (AVL) shows the location of every police and fire apparatus in the City. All of this tech doesn’t mean policing is easier or safer than it was in the 1970s – on the contrary, there are new issues and dangers, which I’ll mention a little later.

We did reports by hand, on paper. We filled out index cards for car stops. And every call to police/fire emergency was logged on a card with a timestamp. When we wanted to get information about a license plate or driver’s license, the dispatcher looked up the info in a set of file cards or – this was really high tech in the 1970s – typed the request into a teletype machine for someone in some far city (like Des Moines) to look up on their index cards.

Now, things are much more high tech. First, people call 911 for emergencies. 911 is virtually ubiquitous in the United States, but barely existed in the 1970s. The police call-taker immediately sees the ANI/ALI (automatic number identification / automatic location identification) associated with your number. The call taker immediately enters all the information about the call into the Seattle Police Department’s new CAD (software written by Versaterm). [Fires or emergency medical calls are "hot transferred" from a police call-taker to a fire dispatcher, who enters the information into a Seattle Fire CAD, and you can actually view some real-time information about Fire 911 calls online here].

Dispatchers then dispatch the 911 call to an available police unit. An electronic map shows the location of every 911 call which is in-progress or waiting, the locations of police units and their status (free, working a call, etc.). A double click on a map icon brings up information about the call or the unit. Records management (also by Versaterm) is similarly automated, with reports now written electronically on laptop or in-vehicle computers directly by officers. A wide variety of information (e.g. address) is automatically verified, and the report is uploaded wirelessly.

The state-of-the-art in Seattle Police is even more high tech. Every patrol car has a digital video camera; every car stop is recorded, including the audio of the conversation from a wireless mic carried by the officer. Special license-plate-recognition vehicles (wirelessly connected to national databases) cruise the streets looking for parking scofflaws and stolen cars. Officers with BlackBerrys or their in-car vehicles can easily search for online information – a far cry from that teletype machine.

We are actively working on even higher speed wireless networking in the 700 MHz spectrum, which should allow two-way high-quality video transmissions to/from field units, including video from private security cameras in banks and stores. Fire units already carry electronic versions certain sorts of building plans, but in the future those building plans could be quickly updated to show the locations of hazardous materials or the detailed configuration of a school.

I’m certain high-tech has increased public safety through more rapid sharing of information, and has improved communications and therefore officer safety. This comes at a price, of course, and not just in dollars.   I’m not quite sure how dispatchers and police officers and firefighters stay current with the skills required to dispatch, provide policing, fight fires and provide emergency medical, AND also learn all this technology.  It is a challenge!

And officers today face dangers on the street which I never dreamed of in the 1970s – significant drug use, gangs, potential terrorists, and criminals who specialize themselves in using technology for identity theft, stalking, and crimes against children. I’m glad my experience as a police officer is behind me – I’m not smart enough or quick enough on my feet to face the challenges of the street today. But I hope – by continued wise application of technology – we can make cops, firefighters and the people they serve a bit safer.

- Cyber City Armageddon?

and Loose Laptops Sink Cyber (Security)

Is City-Cyber-Armageddon just around the corner?

Today City governments depend upon technology – more than ever – to operate.  Constituents depend upon the Internet, web, e-mail , cell phones to communicate with their government for information and services.  But, gee, how secure and reliable are these systems, these networks and these communication?

I recently had a non-classified meeting with some fedgov Department of Homeland Security cyber folks, and DHS contractors about potential cyber security tools.  I’m a “geek”, so I love tools and software.  I’m a senior public official, so I also like charts and graphs and statistics.  My meeting had plenty of both tools and statistics.  But I walked away from the meeting ready to move to a mountain cabin “off the grid” and isolated from the world.

Is cybersecurity really a major issue?  What can a municipal government do to improve HomeCity Security?

Is it an issue? I offer the following observations:
•   A laptop computer with records of 26.5 million veterans was stolen from the home of a Veteran’s Administration employee in May 2006 (later recovered).  But these veterans (including me – I’m a retired Army Officer) received letters notifying us of the problem.  The VA also lost records of 1.8 million veterans in February 2007 and covered up other data breaches.  They (that’s “we” for those of us who pay fedgov income tax) paid for a lot of clean-up and credit monitoring.
•   The day after his inauguration, President Obama published a cybersecurity plan and intends – as a top priority – to appoint a national cybersecurity advisor.
•   Within the last few months, Heartland Computer Systems may have lost over 45 million consumer credit card numbers .
•   The nation’s electrical grid is allegedly vulnerable to cyberattack (and my City operates the nation’s ninth largest municipal electric utility with 300,000 customers)
•   Conficker worm may be infecting one million new computers a day

What scares me?

1. Injury to the people who trust the government of the City of Seattle.  The people of Seattle entrust their credit card numbers, their phone numbers, their personal information to my government.  When they call 911, they expect help.  And we’ve had web-based SQL databases compromised by SQL injection attacks, so any constituent visiting those websites receives computer viruses… from us!   If someone is hurt physically or financially or emotionally because we’ve failed to keep the telephone network or their personal information cybersecure, I’ve failed as CTO, and I’ve failed big-time.   I never want to be sending letters like the one I received from the VA.

2.  Damage to the City of Seattle’s reputation.  One reason my government works so well is that the people of Seattle trust us: last November, despite a looming recession, they passed levies to fund more parks, a Pike Place Market renovation, and a $17 billion transit system.  A cyber-incident will damage that special relationship.

3.  Outage of the City’s technology systems. Constituents use technology to report problems and request service from the City.  They call 911 or 684-3000 (utility customer service).  They send e-mail.  The pay bills on the web.  And City employees use technology to coordinate our response – radio systems for public safety, telephone and data networks, electronic mail systems, Windows servers and a 24×7 data center.  I’m proud of 99%+ uptime on those systems to “make technology work for the City.  Cyber incidents endanger those systems.

How can we improve HomeCity Cybersecurity?  Here’s what I’m doing:

1.  Hired a damn fine CISO.  My Chief Information Security Officer, Mike Hamilton, is the best.  Worked for a long time in private industry, came to Seattle ready to give his expertise in public service.  Like all CISO’s, he sees bad guys everywhere.  Unlike many CISO’s, he knows that technology and the Internet are here to stay and we need to take practical measures to make them as secure as possible.

2.  Assemble and train a team of cyber-techies and professional cyber-sleuths.  We have dedicated, skilled IT security professionals scattered throughout City government.  Their departments and agencies spent money to train them, and CISO Hamilton matrix-manages them to patch and secure systems.  We use them as a cyber-incident-management team under Hamilton’s Deputy – David Matthews – to investigate and get to root cause of any potential cybersecurity incident.  They are our best cyber-defense.

3.  Test every doggone Internet-facing application.  Do penetration testing on our Internet connection.  Watch firewall logs.  Apply every Microsoft or Cisco or (fill-in-the-blank technology company) security patch as soon as you can.  No more than five days max from patch release to deployment.

4.  Selectively outsource.  We’ve outsourced management of credit card payments to skilled third parties, rather than “storing and managing our own”.  We can’t outsource accountability, but we can share risk.

5.  Buy some basic tools.  Anti-virus for every computer.  Patch distribution software.  Vulnerability scanning software.  System logging and aggregation software.  Web site blocking software.  Then use it.

6.  Educate, train, harangue and educate again.  The weakest link in every cybersecurity defense is employees.  Employees who transport data from work to home on thumbdrives, potentially losing the data or introducing a new virus or worm.  “Loose lips sink ships” and “Loose laptops sink cyber-security”.  Employees who surf the Internet and hit questionable websites.  We train employees on good security practices, harangue management to enforcement them, and then train again. 

I’m not as concerned about cyber attacks crippling public safety radio systems or the SCADA systems which control the electrical grid and water supply or traffic signal control.  These systems are vulnerable, but have in-depth layers of defense and employees dedicated to protecting them.

I’m concerned about that single lost portable hard drive with social security numbers.  Or that one SQL server database which should be “read only” but is “read-write” and compromised.  Or that employee who goes to a web gambling site and downloads a day-zero cyber virus.

Technology is here to stay. Internet access will only increase.  But we’re working hard to mitigate the vulnerabilities.

And I don’t sleep very well at night.

- Heroes of 9/11 and Tech

Police at WTO Riots in Seattle, 1999

Original post:  23 May 2008
Life is full of coincidences and crossing paths on the journey of life.  I had such a set of coincidences this week.  On Tuesday night I had a couple glasses of wine with an acquaintance of mine, Gino Menchini, who was in town to keynote a conference.   Gino was CIO of the City of New York for a few years early in Michael Bloomberg’s administration.   I first met Gino at a “Large Cities CIOs” conference he hosted in NYC in 2004.
Tonight (May 22nd) the move “Battle in Seattle“ kicked off the Seattle International Film Festival.  This film purports to depict the WTO riots of 1999, which gave Seattle a huge black eye in the media, but also helped galvanize us to improve our preparedness for disasters.
Sunday morning, May 25th, a team of employees of my department – the Department of Information Technology (DoIT) of the City of Seattle, will give up a good chunk of their Memorial Day weekend to upgrade the major telephone switch serving Seattle’s City government to the latest release of the switch’s software.   Many of these are the same folks who helped put the technology into the City of Seattle’s new Emergency Operations Center / Fire Alarm Center (see my blog entry from May 18th).
What do these events have in common?   Just why are they a “coincidence”?
Just this:  Heroism comes in many flavors.   September 11th, unfortunately, created many heroes, most of which are still with us.  Those are folks who supported the City of New York (and also Washington, D.C.) during those difficult days.  Gino was one of them – although he was working as an account executive for Cisco Systems on September 11th, he ran TOWARD the World Trade Center, and “collected” (appropriated?) a lot of technology from Cisco to allow the City of New York’s Emergency Operations Center to be up and functioning within 24 hours of the disaster.   He stayed on duty for weeks after that, helping to direct Cisco’s resources toward the recovery from that disaster.  
Similarly, the WTO riots here in Seattle created a number of heroes, including Assistant Police Chief Jim Pugel, who was a police Captain and in command of the police officers on  the street during that difficult week.   Jim is one of the most caring, unassuming people you’d ever meet.  But he took care of his officers and protected the people of Seattle despite terrible planning by the City (thank you Paul Schell) in preparing for the event.  
In their own ways, those DoIT employees coming in on Sunday to upgrade a telephone switch are also quiet heroes.  While many people are enjoying their Memorial Day weekend, these folks will be in downtown Seattle working.   The City of Seattle’s phone network is up and available over 99.99% of the time, which is really important in disasters and emergencies, because it WORKS when the public telephone network will be overloaded.  As I mentioned, these are the same folks who worked many long hours to install technology in the EOC to make it quite prepared to weather and manage future disasters.  And on September 11th, 2001, they stayed on the job in a skyscraper in downtown Seattle, keeping City of Seattle technology operating, when all the employees of banks and private companies left those skyscrapers and went home in fear.
These are different levels of heroism, but they all require commitment to keeping the people of the United States, the City of New York and the City of Seattle safe.  And these are the quiet heroes, not the folks who get a medal or have their name on the front page. 
I am so proud to know and work with them.

- Two Seattle 911 Centers?

Click to see calls to Seattle 911

Original post:  18 May 2008
We’ve opened a new fire alarm center (FAC) in Seattle – see yesterday’s blog entry.  This center accepts 911 calls for medical emergencies and fires – anything handled by the Seattle Fire Department.   There is a separate 911 center elsewhere in the City for 911 calls for police service and police dispatching.   How do we manage two separate centers?   The answer:  all 911 calls go to police first, and then, if the situation requires the fire department, the caller is “hot transfered” to the FAC.   “Hot transfer” means the police 911 call-taker stays on the line until the fire call-taker picks up the call.
Gee, this seems quite inefficient – we have two separate groups of folks answering 911 calls, two separate buildings, two separate telephone systems, two separate computer systems to enter the information (“computer-aided dispatch”) and so forth.   Isn’t this much more costly for Seattle taxpayers?
A long time ago, I thought so.   Many cities (e.g. Chicago) have a single 911 center where dispatchers and call takers for all three disciplines (police, fire, emergency medical) work.
But the philosophies and requirements have changed, largely after the San Francisco earthquake of 1989 and that event in New York City / the Pentagon on September 11, 2001.   Now almost every City has a backup 911 center – a separate physical location with a whole set of separate systems.   In most cities, this separate location just sits, unused, most days of the year.  It is only activated for testing and during those very rare occasions where some event makes the primary 911 center unusable.   Such events occur more often than one might think – they could include white power (anthrax) scares or power failures (including failure of backup systems), as well as disasters.
In Seattle we’ve chosen a different – and, in my opinion – wiser route.   Both our 911 centers are active and in use every day, 24 hours a day.   We KNOW all the backup systems work because they are constantly in use – we don’t just test the backup center once a year.   Once again Seattle leads – becoming the City most prepared to deal with a disaster.