- Cyber City Armageddon?

and Loose Laptops Sink Cyber (Security)

Is City-Cyber-Armageddon just around the corner?

Today City governments depend upon technology – more than ever – to operate.  Constituents depend upon the Internet, web, e-mail , cell phones to communicate with their government for information and services.  But, gee, how secure and reliable are these systems, these networks and these communication?

I recently had a non-classified meeting with some fedgov Department of Homeland Security cyber folks, and DHS contractors about potential cyber security tools.  I’m a “geek”, so I love tools and software.  I’m a senior public official, so I also like charts and graphs and statistics.  My meeting had plenty of both tools and statistics.  But I walked away from the meeting ready to move to a mountain cabin “off the grid” and isolated from the world.

Is cybersecurity really a major issue?  What can a municipal government do to improve HomeCity Security?

Is it an issue? I offer the following observations:
•   A laptop computer with records of 26.5 million veterans was stolen from the home of a Veteran’s Administration employee in May 2006 (later recovered).  But these veterans (including me – I’m a retired Army Officer) received letters notifying us of the problem.  The VA also lost records of 1.8 million veterans in February 2007 and covered up other data breaches.  They (that’s “we” for those of us who pay fedgov income tax) paid for a lot of clean-up and credit monitoring.
•   The day after his inauguration, President Obama published a cybersecurity plan and intends – as a top priority – to appoint a national cybersecurity advisor.
•   Within the last few months, Heartland Computer Systems may have lost over 45 million consumer credit card numbers .
•   The nation’s electrical grid is allegedly vulnerable to cyberattack (and my City operates the nation’s ninth largest municipal electric utility with 300,000 customers)
•   Conficker worm may be infecting one million new computers a day

What scares me?

1. Injury to the people who trust the government of the City of Seattle.  The people of Seattle entrust their credit card numbers, their phone numbers, their personal information to my government.  When they call 911, they expect help.  And we’ve had web-based SQL databases compromised by SQL injection attacks, so any constituent visiting those websites receives computer viruses… from us!   If someone is hurt physically or financially or emotionally because we’ve failed to keep the telephone network or their personal information cybersecure, I’ve failed as CTO, and I’ve failed big-time.   I never want to be sending letters like the one I received from the VA.

2.  Damage to the City of Seattle’s reputation.  One reason my government works so well is that the people of Seattle trust us: last November, despite a looming recession, they passed levies to fund more parks, a Pike Place Market renovation, and a $17 billion transit system.  A cyber-incident will damage that special relationship.

3.  Outage of the City’s technology systems. Constituents use technology to report problems and request service from the City.  They call 911 or 684-3000 (utility customer service).  They send e-mail.  The pay bills on the web.  And City employees use technology to coordinate our response – radio systems for public safety, telephone and data networks, electronic mail systems, Windows servers and a 24×7 data center.  I’m proud of 99%+ uptime on those systems to “make technology work for the City.  Cyber incidents endanger those systems.

How can we improve HomeCity Cybersecurity?  Here’s what I’m doing:

1.  Hired a damn fine CISO.  My Chief Information Security Officer, Mike Hamilton, is the best.  Worked for a long time in private industry, came to Seattle ready to give his expertise in public service.  Like all CISO’s, he sees bad guys everywhere.  Unlike many CISO’s, he knows that technology and the Internet are here to stay and we need to take practical measures to make them as secure as possible.

2.  Assemble and train a team of cyber-techies and professional cyber-sleuths.  We have dedicated, skilled IT security professionals scattered throughout City government.  Their departments and agencies spent money to train them, and CISO Hamilton matrix-manages them to patch and secure systems.  We use them as a cyber-incident-management team under Hamilton’s Deputy – David Matthews – to investigate and get to root cause of any potential cybersecurity incident.  They are our best cyber-defense.

3.  Test every doggone Internet-facing application.  Do penetration testing on our Internet connection.  Watch firewall logs.  Apply every Microsoft or Cisco or (fill-in-the-blank technology company) security patch as soon as you can.  No more than five days max from patch release to deployment.

4.  Selectively outsource.  We’ve outsourced management of credit card payments to skilled third parties, rather than “storing and managing our own”.  We can’t outsource accountability, but we can share risk.

5.  Buy some basic tools.  Anti-virus for every computer.  Patch distribution software.  Vulnerability scanning software.  System logging and aggregation software.  Web site blocking software.  Then use it.

6.  Educate, train, harangue and educate again.  The weakest link in every cybersecurity defense is employees.  Employees who transport data from work to home on thumbdrives, potentially losing the data or introducing a new virus or worm.  “Loose lips sink ships” and “Loose laptops sink cyber-security”.  Employees who surf the Internet and hit questionable websites.  We train employees on good security practices, harangue management to enforcement them, and then train again. 

I’m not as concerned about cyber attacks crippling public safety radio systems or the SCADA systems which control the electrical grid and water supply or traffic signal control.  These systems are vulnerable, but have in-depth layers of defense and employees dedicated to protecting them.

I’m concerned about that single lost portable hard drive with social security numbers.  Or that one SQL server database which should be “read only” but is “read-write” and compromised.  Or that employee who goes to a web gambling site and downloads a day-zero cyber virus.

Technology is here to stay. Internet access will only increase.  But we’re working hard to mitigate the vulnerabilities.

And I don’t sleep very well at night.

- A Tech Thanksgiving

As many of us sit down to the average American Thanksgiving 3000 calorie meal tomorrow, we’ll be in uncertain and frightening times. But I’m also counting my technology blessings, and here are a few:

1.  I’m thankful for the generosity of the people of Seattle. We’ve asked a lot of them over the years, and they have consistently voted to tax themselves to give our city and region an improved quality of life, for examples:

•   A completely re-built and remodeled Seattle Public Library system, a beautiful central library and 26 branches, including wi-fi in every branch and 1000 computers for public use, all financed with a $196 million levy. This week we have a wonderful new City Librarian in Susan Hildreth, coming to us from the California State Library.

•   A new light-rail line from downtown to the airport, set to open in 2009,  and a just-passed bond $17.9 billion measure to extend that line by 34 miles over the next 20 years

•   A $167 million fire facility levy which, although strapped for cash in times of rising costs, has already seen us build a new state-of-the-high-tech-art emergency operations center and fire alarm center  , a new fireboat and a joint training facility. The technology systems supporting Seattle Fire help them achieve an average four minute response time to calls, and you can even see those calls in real-time on our website.

•   Note: although I’ve highlighted the investments above, Seattle voters also have approved housing levies, parks levies and funding for other projects to improve our quality of life.

2.  I’m thankful for wonderful, dedicated, employees in the City of Seattle and especially those 600 folks who run our information technology across multiple departments. Throw out your old ideas about clock-watching government bureaucrats pushing paper from the in-box to the out-box. These high-tech folks run the electronic mail systems and internal phone network and electronic payment systems and customer service systems which make our City government a truly 24 hour-a-day, 7 day-a-week business. And we have some unique twists such as an online directory of almost all employees to help customers cut through the organization – not many other companies or governments have that: . I’ve blogged before about how diligently and competently these folks respond to disasters large and small, e.g. the 108 degree data center, , Dial Tone comes from God , and Nervous System of a City Government .

3.  I’m thankful for an award-winning City of Seattle web portal www.seattle.gov , twice winning the top city web portal from the Center for Digital government . And also for the Seattle Channel, winner of both Emmys and back-to-back 2007 and 2008 excellence in government programming awards from NATOA

4.  Finally, I’m thankful for great and supportive leadership such as Mayor Greg Nickels who recognizes the efficiency and effectiveness which technology brings to City government by proposing significant technology improvements even in the upcoming lean budget years. And Seattle’s City Council supported that vision by passing the technology portions of his 2009-10 budget with few changes – and those changes were improvements such as a Technology Matching Fund increase and a Citizen Engagement Portal.

Of course this sounds self-serving, because Greg’s my boss and the Council holds the purse strings. But there are hard, solid, initiatives in this budget: a new customer relationship management system, an Outlook/Exchange replacement for an aging e-mail system, an electronic parking guidance system, outage and asset management systems for Seattle City Light, and much more.

5. And, in terms of leadership, we techies can also turn to the federal government and see a new President who knows the importance of broadband and technology to the economy and to making the Federal Government more effective and in touch with people. Everyone in the United States can rejoice and give thanks for that.

You may think I’m a bit Pollyannaish in this blog, and I am, because it is a time to give thanks. But I promise my next blog will be a bit different, as I give you my Recipe for making Technology Turkeys.

- A National CTO?

Which is the National CTO?

Barack Obama states he will appoint the nation’s first Chief Technology Officer (CTO) . And, indeed, his own campaign even has (had?) its own CTO (see CIO-dot-com).  Blogger Robert Scoble recently listed (somewhat tongue-in-cheek) the “A list” of names for the National CTO job.

Vint Cerf (as quoted by Ed Cone in his blog on CIO Insight) worries about “centralizing” technology or technology policy in the Federal government. He correctly points out that a “technology czar” would have about the same level of success as previous administration’s “energy” and “drug” and “fill-in-the-blank” czars.

But what would a “national CTO” actually DO?

Obama’s campaign website lists a potential set of duties. These include:

• More transparency in government – presumably this means the federal government. Chief Geek comment: Yes!
• Development of an interoperable wireless network for first responders. Chief Geek comment: Oh Gawd no. There are so many different groups and bureaucracies trying to do this now, vying for attention and dollars, that we’ve created a mini-first-responder-industrial complex.
• Sharing of best technology practices between government agencies. Chief Geek comment: Well, maybe. The Office of Management and Budget (OMB) of the Bush Adminstration is already and consistently scoring agencies on their management, and specifically the use of electronic government (see the latest scorecard here )

As CTO (aka Chief Geek) for the City of Seattle, I do have an opinion about this (surprise!) .

The City of Seattle does not have a CIO.  To some extent, the title “CTO” instead of CIO is an historical anomaly dating from the time the position was created by the Seattle City Council in the mid-1990s. But I also head a department (Information Technology or DoIT) which largely manages infrastructure. Applications are supported by the individual departments who conduct the business of City government (providing water, electricity, transportation, policing, parks, fire and emergency medical service, etc.).  As CTO, my office provides oversight and standards for the use of technology in City government, but I only directly manage about 215 of the 600 or so IT employees in the government.

In the Fedgov, not even the technology infrastructure of the government can be centralized under a CTO. The Fedgov is just too large and diverse.

I’ve previously written that government generally should not be on the bleeding edge of technology – we should take technologies pioneered and honed by the private sector, and apply them to the business of governing. In the Fedgov this is also true, with the exception of the military and homeland security, who have unique duties which will stretch the envelope of technology in new and different ways from the private sector.

So what would a national CTO actually DO? I suggest:

• Make that blob of the Fedgov more transparent. Absolutely.
• Find technologies and best practices for using technologies pioneered in the private sector and imfuse them into Federal agencies. I’ve previously listed a number of ideas about the use of Web 2.0 tech, for a specific set of examples, in government.
• Push the OMB Scorecard further and deeper with aspects of technology other than “e-gov”. The best way to push agencies to cooperate and interoperate is to score their performance. We do that with project management at the City of Seattle, and it works wonders.
• Where possible, demand, direct and lead Federal agencies to cooperate and consolidate – share web services, share infrastructure, consolidate data centers and so forth.

In terms of national (non-federal-government) leadership by this Federal CTO position, I’m a little more cautious and skeptical. I like Vint Cerf’s idea about an information technology advisory committee (PITAC). But, in general, I’d say the robust set of private technology companies (led by Seattle’s own Microsoft), the University community and the open source Internet community are doing just fine in national and worldwide technology leadership. 

We do have a number of Federal agencies which appropriately regulate or support technology, for example the FCC, the Federal Trade Commission, National Science Foundation and, of course (famously) DARPA.  Most of these agencies could be improved in an administration more technologically enlightened than the present one.

But we don’t really need a federal technology “czar” to “help”.

- A Taxpayer Network Lock Out

San Francisco Locked Out

Original post:  20 July 2008

Terry Childs, a network administrator for the City / County of San Francisco, was arrested last week on four counts of computer fraud. He presently sits in the San Francisco County jail on a $5 million bond. Childs apparently configured the City’s Cisco-based network so he along had the password(s) to control and manage that network. And – seven days after the arrest – the City’s Department of Telecommunications and Information Services is apparently still locked out of its network. The original report of this incident from the San Francisco Chronicle is here. Paul Venezia of Infoworld blogged “insider” information here which he obtained in an e-mail from a SF employee. Although this is an anonymous source, Venezia’s story certain rings true to me. (Note: Although I know Chris Vein, San Francisco CIO, and count him as a friend, I have not discussed this incident with him nor do I have any personal knowledge of the event).

This situation, on the face of it, is both outrageous and troubling. I won’t speculate about why it occurred in San Francisco, other than saying Venezia’s blog has the ring of truth. The larger question: has it happened elsewhere and could in happen again in another public agency or government? And what, if anything, can we do to prevent it?

Has it happened before?
Emphatically and undoubtedly the answer is “yes”. Can I cite a specific example? Not immediately, but there are many many networks – and too many of them are dependent on a single “guru” or talented individual. A couple of caveats are in order here: first, in San Francisco Childs only managed the data communications network of routers and switches – he did NOT have access to applications, databases, and servers. That’s why most City technology functions appear to be working fine. Second, most networks are owned by private companies and businesses. They are NOT in the public eye as the City of San Francisco or the City of Seattle. Security incidents in private networks or even smaller government networks will not be visible to the public or the press.

Could it happen again, elsewhere?
Again, undoubtedly it will. However, I think such an incident in a large network is quite unlikely. Such networks require a number of technical people to cooperatively manage. And the larger the network the more rigorous and formal change management processes are required. Indeed, according to Venezia’s blog, it was a requirement for documentation and change management which might have sent Childs over the edge.

Indicators
Several small points are buried in the news articles: first, Childs allegedly monitored management’s electronic mail. Most technical folks in most organizations have some ability to do this. But most public employees (in my experience) have much higher standards of integrity. And with the availability of e-mail encryption, good security monitoring tools, and teams of employees working together, such monitoring should be rare and declining.
Next, San Francisco recently hired a Chief Information Security Officer (CISO), who was actively investigating, monitoring and instituting stronger security policies. Again, this is another factor which probably led to Child’s discovery and arrest. In my personal experience, CISOs have rigorous integrity and concern for processes and policies which protect agency information from harm.
Finally, Childs appears to have a strong ownership of San Francisco’s fiber-wide-area-network, proud of its construction and reliable operation. These are noble attributes which I find in many public technology employees. He also apparently had a disdain for other administrators, staff and management in the department. This is, thankfully, a rare attribute in my experience.

How can we prevent future occurrences?
Some will suggest conducting “background checks” on employees. These are valuable. We’ve been doing them as a matter of course for five years at the City of Seattle’ DoIT. However background checks merely make sure we’re not hiring employees with a history of convictions for driving while intoxicated or a current set of 100 unpaid parking tickets. And they would not have prevented the Childs’ incident. More importantly, when hiring we need to look for employees who are personable and can work as members of a team. Smart employees can be trained for technical skills. In the distant past (1980s), technology employees were very proud of their programming (“networks”, “systems”, “code”), identified it, and defended it intensely (“there aren’t any bugs in that program – I created it and tested it – are you questioning my technical skills?”) Today we can’t afford that – we need employees who are proud of the technology they control, but who have a life and an identity outside of the work they do. Employees who build reliable systems, but realize that it is not the system which matters, but the fact that the 600,000 people of the City of Seattle are safer and happier because their government uses that technology to better serve them. And we also need to employ “best practices” in technology management, hire Chief Information Security Officers, and have employees and technically-astute management who are diligent with change management processes to keep our technology operating reliably.

A Personal Note
A couple of years ago we at the City of Seattle hired a new network administrator. His managers and I fired him after six weeks on the job. Indeed, we should have fired him after two weeks. He displayed a penchant for trying to hack into network switches rather than collaborate with others on the network team to manage them and administer them. The lessons: teamwork is the most valuable attribute in any public employee! You can train and educate folks to be technologists, administrators and managers. Training for teamwork is much harder – you need to look for it when hiring. Second: don’t hesitate to act on bad behavior. And for this, the management San Francisco’s Department of Telecommunications and Information Services should be commended, even if it was late.

- Seattle’s Toilets make E-Bay

Seattle's Hi-Tech Toilets

Original post:  12 July 2008

Technology makes government vastly more efficient and effective. No cop or utility crew or building inspector could do their jobs without cell phones, radios and computers.
But technology is not necessarily the solution to everything. Seattle spent multi-millions of dollars for public hi-tech toilets in 2004. A noble experiment to try and improve the hygiene of the homeless and people on the street. Now the experiment is ending and you will be able to bid on the toilets on e-bay. See this article in the Seattle Post-Intelligencer and Governing Magazine’s “13th Floor” article here. 
The lesson here? The same lesson those of us working in information technology have learned this lesson hard way over the last 30 years: first of all be clear on the scope and objectives of your project, then re-engineer the business processes, and only after that look for a technology solution. Quite often the solution is not high-tech, but rather changing the business culture or process or the routines and habits of people. And this lesson specifically includes how and where people use the toilet!