Original post: 20 July 2008
Terry Childs, a network administrator for the City / County of San Francisco, was arrested last week on four counts of computer fraud. He presently sits in the San Francisco County jail on a $5 million bond. Childs apparently configured the City’s Cisco-based network so he along had the password(s) to control and manage that network. And – seven days after the arrest – the City’s Department of Telecommunications and Information Services is apparently still locked out of its network. The original report of this incident from the San Francisco Chronicle is here. Paul Venezia of Infoworld blogged “insider” information here which he obtained in an e-mail from a SF employee. Although this is an anonymous source, Venezia’s story certain rings true to me. (Note: Although I know Chris Vein, San Francisco CIO, and count him as a friend, I have not discussed this incident with him nor do I have any personal knowledge of the event).
This situation, on the face of it, is both outrageous and troubling. I won’t speculate about why it occurred in San Francisco, other than saying Venezia’s blog has the ring of truth. The larger question: has it happened elsewhere and could in happen again in another public agency or government? And what, if anything, can we do to prevent it?
Has it happened before?
Emphatically and undoubtedly the answer is “yes”. Can I cite a specific example? Not immediately, but there are many many networks – and too many of them are dependent on a single “guru” or talented individual. A couple of caveats are in order here: first, in San Francisco Childs only managed the data communications network of routers and switches – he did NOT have access to applications, databases, and servers. That’s why most City technology functions appear to be working fine. Second, most networks are owned by private companies and businesses. They are NOT in the public eye as the City of San Francisco or the City of Seattle. Security incidents in private networks or even smaller government networks will not be visible to the public or the press.
Could it happen again, elsewhere?
Again, undoubtedly it will. However, I think such an incident in a large network is quite unlikely. Such networks require a number of technical people to cooperatively manage. And the larger the network the more rigorous and formal change management processes are required. Indeed, according to Venezia’s blog, it was a requirement for documentation and change management which might have sent Childs over the edge.
Several small points are buried in the news articles: first, Childs allegedly monitored management’s electronic mail. Most technical folks in most organizations have some ability to do this. But most public employees (in my experience) have much higher standards of integrity. And with the availability of e-mail encryption, good security monitoring tools, and teams of employees working together, such monitoring should be rare and declining.
Next, San Francisco recently hired a Chief Information Security Officer (CISO), who was actively investigating, monitoring and instituting stronger security policies. Again, this is another factor which probably led to Child’s discovery and arrest. In my personal experience, CISOs have rigorous integrity and concern for processes and policies which protect agency information from harm.
Finally, Childs appears to have a strong ownership of San Francisco’s fiber-wide-area-network, proud of its construction and reliable operation. These are noble attributes which I find in many public technology employees. He also apparently had a disdain for other administrators, staff and management in the department. This is, thankfully, a rare attribute in my experience.
How can we prevent future occurrences?
Some will suggest conducting “background checks” on employees. These are valuable. We’ve been doing them as a matter of course for five years at the City of Seattle’ DoIT. However background checks merely make sure we’re not hiring employees with a history of convictions for driving while intoxicated or a current set of 100 unpaid parking tickets. And they would not have prevented the Childs’ incident. More importantly, when hiring we need to look for employees who are personable and can work as members of a team. Smart employees can be trained for technical skills. In the distant past (1980s), technology employees were very proud of their programming (“networks”, “systems”, “code”), identified it, and defended it intensely (“there aren’t any bugs in that program – I created it and tested it – are you questioning my technical skills?”) Today we can’t afford that – we need employees who are proud of the technology they control, but who have a life and an identity outside of the work they do. Employees who build reliable systems, but realize that it is not the system which matters, but the fact that the 600,000 people of the City of Seattle are safer and happier because their government uses that technology to better serve them. And we also need to employ “best practices” in technology management, hire Chief Information Security Officers, and have employees and technically-astute management who are diligent with change management processes to keep our technology operating reliably.
A Personal Note
A couple of years ago we at the City of Seattle hired a new network administrator. His managers and I fired him after six weeks on the job. Indeed, we should have fired him after two weeks. He displayed a penchant for trying to hack into network switches rather than collaborate with others on the network team to manage them and administer them. The lessons: teamwork is the most valuable attribute in any public employee! You can train and educate folks to be technologists, administrators and managers. Training for teamwork is much harder – you need to look for it when hiring. Second: don’t hesitate to act on bad behavior. And for this, the management San Francisco’s Department of Telecommunications and Information Services should be commended, even if it was late.