- 4 Quick Fixes for the Procurement Dragon

Yes we can!

Almost everyone who deals with government – internally or externally – is frustrated by the “procurement dragon”.  Procurement seem to take forever and are one of the most bureaucracy-laced processes in all of governing.   In these days of innovation and the flourishing of the startup culture, procurement processes seem to be an anachronistic throwback.

Furthermore, the convoluted purchasing process only seems to benefit large corporations who have the legions of attorneys and technical staff to respond to RFPs and negotiate the maze.

Purchasing practice is steeped in a web of local and state laws, regulations and executive orders, so they’re not easy to change.   There are good reasons for the present procurement practices, and I’ll mention those at the end of this post.   But first, can innovation and a culture of agile government survive in the present purchasing jungle?

I suggest four quick fixes, some of which are already in place in many governments:

1. Direct Purchase.   This is a direct purchasing mechanism for small procurements – say procurements under $5000.   This would allow a department director or senior manager to directly purchase a good or service from a company without going through more formal purchasing processes.   A manager might purchase a smartphone app and associated database for use by field crews, or a couple of tablet computers for testing.  There still need to be limits on this mechanism, so I’m not issuing 20 direct purchases to the same company in a year, for example, and to make sure the vendor has a valid business license.
2. Roster and invitation to bid.   With this mechanism, companies would be pre-qualified and put on a roster for bids.   A city, for example, might set up a roster for “web applications”.  Companies who want to be on that roster would provide a minimal amount of information – ownership, business address, business license, etc.   And when the city needs a “web application” for a specific purpose, e.g. to accept photos of graffiti from citizens, it could issue a simple, two-or-three page  “invitation to bid” with its requirements and allow companies on the roster to bid.  Typically these bids would also be restricted to procurements of a certain size, say $50,000.
3. Piggy-back on an existing contract.   This mechanism is already widely used.   If a company already has a contract on the Federal Government’s GSA (General Services Administration) schedule, or the Western States’ Contracting Alliance  (WSCA – commonly called “wisca”), any jurisdiction which joins the alliance and authorizes itself to purchase can purchase at the terms and conditions specified by GSA or WSCA.
4. Credit card.   Most government agencies give their trusted department directors and senior employees credit cards.   These are most often used for travel and similar expenses, but they certainly could be used (depending on local ordinance or law) for small purchases, again, up to a limit of, say, several thousand dollars.
5. Budget.   As an adjunct to these four mechanisms, a city, county or department also needs budget to make the procurement.  Perhaps every department or government should have an “innovation fund”.

Using mechanisms like these, governments could quickly and easily procure innovative technologies, goods and services to help them become more efficient and effective.

Implementing these mechanisms requires a great deal of trust – trust by elected officials in their department directors, and trust by those department directors in their senior managers.    There are many cases where that trust has been abused, for example, by a manager purchasing good/services from friends or by making procurements and receiving kickbacks.  Examples include the controversy which engulfed recently appointed federal CIO Vivek Kundra in 2009, or these Seattle Public Utilities customer service representatives in 2012.  So my “quick fixes” for procurment also require diligent oversight and auditing by the appropriate authorities.

Finally, the present procurement practices in most jurisdictions are not the results of “bureaucrats run wild” with regulations, forms and requirements.   They came into being because of widespread abuse of purchasing in the 19^th Century, where Mayors and other elected officials gave jobs to friends, contracts to cronies and similarly greased their own pockets using the procurement process.

“Good government” advocates instituted reforms such as civil service to protect most employees from the winds of politics, and purchasing laws which required specifications and open competition.   These practices still should be followed for major procurements to keep a “level playing field” for competition for the work.

Over the years, however, city councils and legislatures and county commissions have added twists and turns to procurement, largely to correct past injustices or for social engineering.  Do contracts go to firms owned by white men?  Then let’s add a provision for subcontracts to historically underused businesses (HUBs) – women and minority-owned business.   Are we angered by human rights abuses in ______ (fill in the blank, e.g. Burma, Iran, China, etc.)?  Then let’s add a regulation so we don’t  do any business with a company with business interests or a manufacturing plant in those places.   Are we upset that some companies pollute the air and water with their factories or other facilities?  Then let’s eliminate them from bidding on contracts (or have our pension funds divest themselves of the company’s stock).  Do we want to encourage economic development in our City (county, State, or even the entire United States)?   Then let’s add regulations to give preference to firms headquartered or with operations in those places.

I’m not saying these practices are wrong and should all be eliminated.   I’m pointing out that there are reasons the purchasing process is so complicated, and it will take a lot of thought and careful consideration to “unwind the maze”.

In the meantime, let’s implement the “quick fixes”.

- A Public-Private Radio Network?

Police and Fire radio networks.

They have to work.

All the time

During power outages, hurricanes, earthquakes.

When every other wireless network is dead.

So they have to be built, maintained and operated by government, right?

Or else they cannot be trusted, right?

That’s the way cities, counties, regions, states and local governments have ALWAYS built our radio networks for police, firefighters, emergency medical response, utilities, transportation, public works.

And with good reason.

Historically (by that, I mean “before cell phones”), most radio networks were really unreliable.  They were used to dispatch taxicabs and for citizens’ band radio (“CB”) by amateurs.   But no government would trust such a radio network to dispatch cops or firefighters. Such networks had dead spots, lots of static, and dropped off the air entirely when the electricity failed.

With the rise of commercial cell phone and, later, smart phone networks, such networks became … well … “really unreliable“.   Even today many people are angered and upset by dropped calls, “all circuits busy” and slow-loading (or “never loading”) pages.  And during any large event – a packed stadium for a baseball game, or a major traffic jam, a windstorm or an earthquake, you might as well use your phone as a camera, because you probably won’t get through to make a call.

When you’re being robbed at gunpoint or having a heart attack, do you really want the first responders coming to help YOU to depend on such networks?   That’s why, as I’ve blogged before, “cops don’t use cell phones“.

But building government-owned radio networks is REALLY expensive.  A public safety voice network requires just a handful of sites – say 8 radio sites for Seattle or maybe 30 for all of King County here in Washington State.  However, to rebuild those networks today, and to build the new high-speed data networks for responders’ smart phones, tablets and computers will take dozens – perhaps hundreds of sites to cover the same geography.  And THAT takes hundreds of millions of dollars.

Hello – we’re still in the midst of the Great Recession, right?   Government budgets are pinched left and right – sales tax, income tax, property tax revenues are all falling.   While the private sector is still hiring, many governments are laying off employees.   There are few dollars available for hundred million dollar networks.

Is there a middle way?   Is there some way governments could take advantage of the hundreds of existing cell phone sites developed for commercial networks?  Perhaps a way the commercial networks could take advantage of fiber optic networks and buildings or radio sites owned by government?   And some way we could make the cell phone networks more secure, more resistant to terrorism and natural disasters, and therefore more reliable for public safety use?

Here in Seattle, we think so.

We think we might be able to start with all the assets which taxpayers have already bought and paid for – the fiber and microwave networks, radio sites, backup generators, skilled technology employees, and our existing investments in radios and computers.  Then we would add equipment and cell sites and other assets, along with expertise and innovative ideas from private sector companies – telecommunications carriers, equipment manufacturers and apps developers.  Mashing these together, we might get a private-public partnership which gives consumers and businesses more reliable, faster mobile networks, while giving responders new, state-of-the-art networks at a fraction of the cost of building them from scratch, like we’ve always done before.

That’s the idea behind a request for information (RFI) issued by the City of Seattle several weeks ago  seeking ideas about private-public partnerships for next generation networks.  We need some great pioneering “outside the box” ideas in response to the RFI.

And then, perhaps, we can build a modern, smart, network in the Central Puget Sound which saves everyone money, and works reliably during disasters small (“heart attack”) and large (“earthquake”).

P. S.  All these ideas are not mine.  In fact, to some extent I’ve been hauled kicking and screaming (or maybe shuffling and whimpering) to look for a middle way.   Let’s give credit to Deputy King County Executive Fred Jarrett, United States Chief Technology Officer Aneesh Chopra, elected officials like State Representative Reuven Carlyle and Mr. Stan Wu of the City of Seattle for “coloring outside the lines without falling off the page”.

- Thanks & Turkeys 2010

This week Chief Technology Officer Bill Schrier has a LOT for which to be thankful.   But I also have a few turkeys to carve.

My most significant thanks go to the phenomenal people who work in information technology in local government, especially here at the City of Seattle.    Most City and County CIOs, such as those who are the 60 members of MIX (the Metropolitan Information Exchange)  will agree with me and give thanks for their employees as well.  While some members of the public think government employees are 8 to 5 clock-watching bureaucrats, that’s decidedly NOT true of most employees, especially our technology workers.

This fact slammed home to me again this week – Seattle had a snowstorm.    Two inches.    Those of you in Chicago, Boston or Washington DC are probably laughing.  Two measly inches?  What’s the big deal?  But here in Seattle, because of the uniquenesses of our weather systems/geography and the rarity of snow in the lowlands, it was a real show-stopper.  Monday night many of my employees spent four, five or nine hours commuting home on jammed icy freeways.   I and several of my staff walked home five miles in the snowstorm (video of commuters walking across the West Bridge here).

In Seattle’s Department of Information Technology, we had staff who worked all night Monday, or slept at their workstations Monday night, or stayed in hotels downtown, or turned right around and came back to work Tuesday morning after the long commute home.    They did this because they know the work of a City government and the safety of the people of Seattle depend now, more than ever, on reliable technology:  websites, data networks, e-mail systems and much much more.   For these two hundred dedicated people working in the City of Seattle’s technology department, I give thanks.

(My colleagues elsewhere have similar stories, whether in Houston and Mobile, Alabama, who have suffered through hurricanes, or Los Angeles and Riverside who have suffered through earthquakes, or Chicago and Washington DC, with their snowstorms.)

As I attend conferences and talk to my counterparts across the country, I find similar dedication to keeping the public safe and our governments operational. As just one example, we have twenty cities and states around the nation who have authority from the FCC to build fourth generation wireless networks.  Over the past 11 months I’ve been working with officials from these twenty jurisdictions, as well as the FCC’s Public Safety and Homeland Security Bureau, the Public Safety Communications Research Program of the Department of Commerce, and Homeland Security’s Office of Emergency Communications.   Every one of these agencies and the people involved have been working tirelessly to build a nationwide public safety network, a vision which sprung out of the September 11th World Trade Center disaster.     This year we’ve made real progress, despite a number of hurdles.  Now the first networks are under construction.   For all these dedicated government officials and technical staff, I give thanks.

I also give thanks to the many private companies who are doing extraordinary work with technology – Microsoft and Windows and Office, Google with Android and search, Apple with iPhones and iPads, IBM’s Smart Cities Challenge, and a few more who not only want to make money, but also want to use a significant part of that money make the planet a better place in which to live and work.

Finally, I give thanks for my elected officials – Mayor and City Council – and the department directors running City departments here in Seattle.     This year of the Great Recession they have faced terrible choices with budget shortfalls of $67 million in Seattle.  And precipitously falling tax revenues.  And urgent needs from the public for safety nets for our jobless citizens and the poor and homeless.   My own department’s budget was cut by over 17% and I’ve laid off over 10% of my workforce over the past two years.    These are all tough choices, and they are done in the glare of publicity with many competing demands by constituents for the ever-shrinking pot of money.  But we have a sustainable budget and services going into 2011.  Thank you to the officials who stepped up and made these tough choices.

Now on to the turkeys – at least the ones I’d like to carve and serve.

First are some of our technology vendors, a few of whom have ever increasing appetites for money.   Some of them are resorting to “compliance audits” to make sure we are paying for every last danged software license we are using.  One vendor even demanded to have access to every one of the 11,000 computers at the City of Seattle to see if their software was installed.   Others absolutely refuse to negotiate reduced pricing or flexible maintenance plans.  These few money-grubbing vendors get my “tech turkey” award.

Next there are a few of our public employee unions.   Many public employee unions here in the Seattle area realize we are in an unprecedented recession.   Those unions have willingly forgone raises which were in their contracts, understanding that few workers in the private sector get raises, and many private sector workers have lost their jobs and retirement money.   But a few public sector unions have held out for their contracted raises, which are far larger than inflation.  This, frankly, can make all city and county governments and our workers look greedy and foolish.  The public backlash was evident in our recent elections where few tax increases were passed and many revenue sources were cut.  These few unions get my turkey award as well.

My final turkey award goes to those politicians who want to whip the public into a frenzy about supposed fraud and waste in government, or think we can continue tax cuts, increase defense spending, and balance the budget all at the same time.  How do they think public schools, parks, police and fire departments, child protective services, streets or public health are funded, or how do we pay the dedicated people who provide all those services?   I’ve blogged about this at length before, and will just leave these politicians with my tea-party-turkey award.

All in all, however, at this Thanksgiving of 2010, I’ve got a lot more reasons to give thanks than to carve!

- Cyber City Armageddon?

and Loose Laptops Sink Cyber (Security)

Is City-Cyber-Armageddon just around the corner?

Today City governments depend upon technology – more than ever – to operate.  Constituents depend upon the Internet, web, e-mail , cell phones to communicate with their government for information and services.  But, gee, how secure and reliable are these systems, these networks and these communication?

I recently had a non-classified meeting with some fedgov Department of Homeland Security cyber folks, and DHS contractors about potential cyber security tools.  I’m a “geek”, so I love tools and software.  I’m a senior public official, so I also like charts and graphs and statistics.  My meeting had plenty of both tools and statistics.  But I walked away from the meeting ready to move to a mountain cabin “off the grid” and isolated from the world.

Is cybersecurity really a major issue?  What can a municipal government do to improve HomeCity Security?

Is it an issue? I offer the following observations:
•   A laptop computer with records of 26.5 million veterans was stolen from the home of a Veteran’s Administration employee in May 2006 (later recovered).  But these veterans (including me – I’m a retired Army Officer) received letters notifying us of the problem.  The VA also lost records of 1.8 million veterans in February 2007 and covered up other data breaches.  They (that’s “we” for those of us who pay fedgov income tax) paid for a lot of clean-up and credit monitoring.
•   The day after his inauguration, President Obama published a cybersecurity plan and intends – as a top priority – to appoint a national cybersecurity advisor.
•   Within the last few months, Heartland Computer Systems may have lost over 45 million consumer credit card numbers .
•   The nation’s electrical grid is allegedly vulnerable to cyberattack (and my City operates the nation’s ninth largest municipal electric utility with 300,000 customers)
•   Conficker worm may be infecting one million new computers a day

What scares me?

1. Injury to the people who trust the government of the City of Seattle.  The people of Seattle entrust their credit card numbers, their phone numbers, their personal information to my government.  When they call 911, they expect help.  And we’ve had web-based SQL databases compromised by SQL injection attacks, so any constituent visiting those websites receives computer viruses… from us!   If someone is hurt physically or financially or emotionally because we’ve failed to keep the telephone network or their personal information cybersecure, I’ve failed as CTO, and I’ve failed big-time.   I never want to be sending letters like the one I received from the VA.

2.  Damage to the City of Seattle’s reputation.  One reason my government works so well is that the people of Seattle trust us: last November, despite a looming recession, they passed levies to fund more parks, a Pike Place Market renovation, and a $17 billion transit system.  A cyber-incident will damage that special relationship.

3.  Outage of the City’s technology systems. Constituents use technology to report problems and request service from the City.  They call 911 or 684-3000 (utility customer service).  They send e-mail.  The pay bills on the web.  And City employees use technology to coordinate our response – radio systems for public safety, telephone and data networks, electronic mail systems, Windows servers and a 24×7 data center.  I’m proud of 99%+ uptime on those systems to “make technology work for the City.  Cyber incidents endanger those systems.

How can we improve HomeCity Cybersecurity?  Here’s what I’m doing:

1.  Hired a damn fine CISO.  My Chief Information Security Officer, Mike Hamilton, is the best.  Worked for a long time in private industry, came to Seattle ready to give his expertise in public service.  Like all CISO’s, he sees bad guys everywhere.  Unlike many CISO’s, he knows that technology and the Internet are here to stay and we need to take practical measures to make them as secure as possible.

2.  Assemble and train a team of cyber-techies and professional cyber-sleuths.  We have dedicated, skilled IT security professionals scattered throughout City government.  Their departments and agencies spent money to train them, and CISO Hamilton matrix-manages them to patch and secure systems.  We use them as a cyber-incident-management team under Hamilton’s Deputy – David Matthews – to investigate and get to root cause of any potential cybersecurity incident.  They are our best cyber-defense.

3.  Test every doggone Internet-facing application.  Do penetration testing on our Internet connection.  Watch firewall logs.  Apply every Microsoft or Cisco or (fill-in-the-blank technology company) security patch as soon as you can.  No more than five days max from patch release to deployment.

4.  Selectively outsource.  We’ve outsourced management of credit card payments to skilled third parties, rather than “storing and managing our own”.  We can’t outsource accountability, but we can share risk.

5.  Buy some basic tools.  Anti-virus for every computer.  Patch distribution software.  Vulnerability scanning software.  System logging and aggregation software.  Web site blocking software.  Then use it.

6.  Educate, train, harangue and educate again.  The weakest link in every cybersecurity defense is employees.  Employees who transport data from work to home on thumbdrives, potentially losing the data or introducing a new virus or worm.  “Loose lips sink ships” and “Loose laptops sink cyber-security”.  Employees who surf the Internet and hit questionable websites.  We train employees on good security practices, harangue management to enforcement them, and then train again. 

I’m not as concerned about cyber attacks crippling public safety radio systems or the SCADA systems which control the electrical grid and water supply or traffic signal control.  These systems are vulnerable, but have in-depth layers of defense and employees dedicated to protecting them.

I’m concerned about that single lost portable hard drive with social security numbers.  Or that one SQL server database which should be “read only” but is “read-write” and compromised.  Or that employee who goes to a web gambling site and downloads a day-zero cyber virus.

Technology is here to stay. Internet access will only increase.  But we’re working hard to mitigate the vulnerabilities.

And I don’t sleep very well at night.

- A Tech Thanksgiving

As many of us sit down to the average American Thanksgiving 3000 calorie meal tomorrow, we’ll be in uncertain and frightening times. But I’m also counting my technology blessings, and here are a few:

1.  I’m thankful for the generosity of the people of Seattle. We’ve asked a lot of them over the years, and they have consistently voted to tax themselves to give our city and region an improved quality of life, for examples:

•   A completely re-built and remodeled Seattle Public Library system, a beautiful central library and 26 branches, including wi-fi in every branch and 1000 computers for public use, all financed with a $196 million levy. This week we have a wonderful new City Librarian in Susan Hildreth, coming to us from the California State Library.

•   A new light-rail line from downtown to the airport, set to open in 2009,  and a just-passed bond $17.9 billion measure to extend that line by 34 miles over the next 20 years

•   A $167 million fire facility levy which, although strapped for cash in times of rising costs, has already seen us build a new state-of-the-high-tech-art emergency operations center and fire alarm center  , a new fireboat and a joint training facility. The technology systems supporting Seattle Fire help them achieve an average four minute response time to calls, and you can even see those calls in real-time on our website.

•   Note: although I’ve highlighted the investments above, Seattle voters also have approved housing levies, parks levies and funding for other projects to improve our quality of life.

2.  I’m thankful for wonderful, dedicated, employees in the City of Seattle and especially those 600 folks who run our information technology across multiple departments. Throw out your old ideas about clock-watching government bureaucrats pushing paper from the in-box to the out-box. These high-tech folks run the electronic mail systems and internal phone network and electronic payment systems and customer service systems which make our City government a truly 24 hour-a-day, 7 day-a-week business. And we have some unique twists such as an online directory of almost all employees to help customers cut through the organization – not many other companies or governments have that: . I’ve blogged before about how diligently and competently these folks respond to disasters large and small, e.g. the 108 degree data center, , Dial Tone comes from God , and Nervous System of a City Government .

3.  I’m thankful for an award-winning City of Seattle web portal http://www.seattle.gov , twice winning the top city web portal from the Center for Digital government . And also for the Seattle Channel, winner of both Emmys and back-to-back 2007 and 2008 excellence in government programming awards from NATOA

4.  Finally, I’m thankful for great and supportive leadership such as Mayor Greg Nickels who recognizes the efficiency and effectiveness which technology brings to City government by proposing significant technology improvements even in the upcoming lean budget years. And Seattle’s City Council supported that vision by passing the technology portions of his 2009-10 budget with few changes – and those changes were improvements such as a Technology Matching Fund increase and a Citizen Engagement Portal.

Of course this sounds self-serving, because Greg’s my boss and the Council holds the purse strings. But there are hard, solid, initiatives in this budget: a new customer relationship management system, an Outlook/Exchange replacement for an aging e-mail system, an electronic parking guidance system, outage and asset management systems for Seattle City Light, and much more.

5. And, in terms of leadership, we techies can also turn to the federal government and see a new President who knows the importance of broadband and technology to the economy and to making the Federal Government more effective and in touch with people. Everyone in the United States can rejoice and give thanks for that.

You may think I’m a bit Pollyannaish in this blog, and I am, because it is a time to give thanks. But I promise my next blog will be a bit different, as I give you my Recipe for making Technology Turkeys.

- A National CTO?

Which is the National CTO?

Barack Obama states he will appoint the nation’s first Chief Technology Officer (CTO) . And, indeed, his own campaign even has (had?) its own CTO (see CIO-dot-com).  Blogger Robert Scoble recently listed (somewhat tongue-in-cheek) the “A list” of names for the National CTO job.

Vint Cerf (as quoted by Ed Cone in his blog on CIO Insight) worries about “centralizing” technology or technology policy in the Federal government. He correctly points out that a “technology czar” would have about the same level of success as previous administration’s “energy” and “drug” and “fill-in-the-blank” czars.

But what would a “national CTO” actually DO?

Obama’s campaign website lists a potential set of duties. These include:

• More transparency in government – presumably this means the federal government. Chief Geek comment: Yes!
• Development of an interoperable wireless network for first responders. Chief Geek comment: Oh Gawd no. There are so many different groups and bureaucracies trying to do this now, vying for attention and dollars, that we’ve created a mini-first-responder-industrial complex.
• Sharing of best technology practices between government agencies. Chief Geek comment: Well, maybe. The Office of Management and Budget (OMB) of the Bush Adminstration is already and consistently scoring agencies on their management, and specifically the use of electronic government (see the latest scorecard here )

As CTO (aka Chief Geek) for the City of Seattle, I do have an opinion about this (surprise!) .

The City of Seattle does not have a CIO.  To some extent, the title “CTO” instead of CIO is an historical anomaly dating from the time the position was created by the Seattle City Council in the mid-1990s. But I also head a department (Information Technology or DoIT) which largely manages infrastructure. Applications are supported by the individual departments who conduct the business of City government (providing water, electricity, transportation, policing, parks, fire and emergency medical service, etc.).  As CTO, my office provides oversight and standards for the use of technology in City government, but I only directly manage about 215 of the 600 or so IT employees in the government.

In the Fedgov, not even the technology infrastructure of the government can be centralized under a CTO. The Fedgov is just too large and diverse.

I’ve previously written that government generally should not be on the bleeding edge of technology – we should take technologies pioneered and honed by the private sector, and apply them to the business of governing. In the Fedgov this is also true, with the exception of the military and homeland security, who have unique duties which will stretch the envelope of technology in new and different ways from the private sector.

So what would a national CTO actually DO? I suggest:

• Make that blob of the Fedgov more transparent. Absolutely.
• Find technologies and best practices for using technologies pioneered in the private sector and imfuse them into Federal agencies. I’ve previously listed a number of ideas about the use of Web 2.0 tech, for a specific set of examples, in government.
• Push the OMB Scorecard further and deeper with aspects of technology other than “e-gov”. The best way to push agencies to cooperate and interoperate is to score their performance. We do that with project management at the City of Seattle, and it works wonders.
• Where possible, demand, direct and lead Federal agencies to cooperate and consolidate – share web services, share infrastructure, consolidate data centers and so forth.

In terms of national (non-federal-government) leadership by this Federal CTO position, I’m a little more cautious and skeptical. I like Vint Cerf’s idea about an information technology advisory committee (PITAC). But, in general, I’d say the robust set of private technology companies (led by Seattle’s own Microsoft), the University community and the open source Internet community are doing just fine in national and worldwide technology leadership. 

We do have a number of Federal agencies which appropriately regulate or support technology, for example the FCC, the Federal Trade Commission, National Science Foundation and, of course (famously) DARPA.  Most of these agencies could be improved in an administration more technologically enlightened than the present one.

But we don’t really need a federal technology “czar” to “help”.

- A Taxpayer Network Lock Out

San Francisco Locked Out

Original post:  20 July 2008

Terry Childs, a network administrator for the City / County of San Francisco, was arrested last week on four counts of computer fraud. He presently sits in the San Francisco County jail on a $5 million bond. Childs apparently configured the City’s Cisco-based network so he along had the password(s) to control and manage that network. And – seven days after the arrest – the City’s Department of Telecommunications and Information Services is apparently still locked out of its network. The original report of this incident from the San Francisco Chronicle is here. Paul Venezia of Infoworld blogged “insider” information here which he obtained in an e-mail from a SF employee. Although this is an anonymous source, Venezia’s story certain rings true to me. (Note: Although I know Chris Vein, San Francisco CIO, and count him as a friend, I have not discussed this incident with him nor do I have any personal knowledge of the event).

This situation, on the face of it, is both outrageous and troubling. I won’t speculate about why it occurred in San Francisco, other than saying Venezia’s blog has the ring of truth. The larger question: has it happened elsewhere and could in happen again in another public agency or government? And what, if anything, can we do to prevent it?

Has it happened before?
Emphatically and undoubtedly the answer is “yes”. Can I cite a specific example? Not immediately, but there are many many networks – and too many of them are dependent on a single “guru” or talented individual. A couple of caveats are in order here: first, in San Francisco Childs only managed the data communications network of routers and switches – he did NOT have access to applications, databases, and servers. That’s why most City technology functions appear to be working fine. Second, most networks are owned by private companies and businesses. They are NOT in the public eye as the City of San Francisco or the City of Seattle. Security incidents in private networks or even smaller government networks will not be visible to the public or the press.

Could it happen again, elsewhere?
Again, undoubtedly it will. However, I think such an incident in a large network is quite unlikely. Such networks require a number of technical people to cooperatively manage. And the larger the network the more rigorous and formal change management processes are required. Indeed, according to Venezia’s blog, it was a requirement for documentation and change management which might have sent Childs over the edge.

Indicators
Several small points are buried in the news articles: first, Childs allegedly monitored management’s electronic mail. Most technical folks in most organizations have some ability to do this. But most public employees (in my experience) have much higher standards of integrity. And with the availability of e-mail encryption, good security monitoring tools, and teams of employees working together, such monitoring should be rare and declining.
Next, San Francisco recently hired a Chief Information Security Officer (CISO), who was actively investigating, monitoring and instituting stronger security policies. Again, this is another factor which probably led to Child’s discovery and arrest. In my personal experience, CISOs have rigorous integrity and concern for processes and policies which protect agency information from harm.
Finally, Childs appears to have a strong ownership of San Francisco’s fiber-wide-area-network, proud of its construction and reliable operation. These are noble attributes which I find in many public technology employees. He also apparently had a disdain for other administrators, staff and management in the department. This is, thankfully, a rare attribute in my experience.

How can we prevent future occurrences?
Some will suggest conducting “background checks” on employees. These are valuable. We’ve been doing them as a matter of course for five years at the City of Seattle’ DoIT. However background checks merely make sure we’re not hiring employees with a history of convictions for driving while intoxicated or a current set of 100 unpaid parking tickets. And they would not have prevented the Childs’ incident. More importantly, when hiring we need to look for employees who are personable and can work as members of a team. Smart employees can be trained for technical skills. In the distant past (1980s), technology employees were very proud of their programming (“networks”, “systems”, “code”), identified it, and defended it intensely (“there aren’t any bugs in that program – I created it and tested it – are you questioning my technical skills?”) Today we can’t afford that – we need employees who are proud of the technology they control, but who have a life and an identity outside of the work they do. Employees who build reliable systems, but realize that it is not the system which matters, but the fact that the 600,000 people of the City of Seattle are safer and happier because their government uses that technology to better serve them. And we also need to employ “best practices” in technology management, hire Chief Information Security Officers, and have employees and technically-astute management who are diligent with change management processes to keep our technology operating reliably.

A Personal Note
A couple of years ago we at the City of Seattle hired a new network administrator. His managers and I fired him after six weeks on the job. Indeed, we should have fired him after two weeks. He displayed a penchant for trying to hack into network switches rather than collaborate with others on the network team to manage them and administer them. The lessons: teamwork is the most valuable attribute in any public employee! You can train and educate folks to be technologists, administrators and managers. Training for teamwork is much harder – you need to look for it when hiring. Second: don’t hesitate to act on bad behavior. And for this, the management San Francisco’s Department of Telecommunications and Information Services should be commended, even if it was late.

- Seattle’s Toilets make E-Bay

Seattle's Hi-Tech Toilets

Original post:  12 July 2008

Technology makes government vastly more efficient and effective. No cop or utility crew or building inspector could do their jobs without cell phones, radios and computers.
But technology is not necessarily the solution to everything. Seattle spent multi-millions of dollars for public hi-tech toilets in 2004. A noble experiment to try and improve the hygiene of the homeless and people on the street. Now the experiment is ending and you will be able to bid on the toilets on e-bay. See this article in the Seattle Post-Intelligencer and Governing Magazine’s “13th Floor” article here. 
The lesson here? The same lesson those of us working in information technology have learned this lesson hard way over the last 30 years: first of all be clear on the scope and objectives of your project, then re-engineer the business processes, and only after that look for a technology solution. Quite often the solution is not high-tech, but rather changing the business culture or process or the routines and habits of people. And this lesson specifically includes how and where people use the toilet!