Category Archives: Seattle DoIT

– Hackathoning Government

Update 08-25-10:   I’ll write another post on the results of the Tinkerstorm, but the Tropo Blog has a list of results and awards here.

“Hacking” has a bad connotation. We’re going to change that in Seattle this weekend with the Open Government Hackathon. The Hackathon is the culmination of of Geek Week here in Seattle. Yeah, yeah most of you non-Puget-Sounders think every week is Geek Week in Seattle, complete with nerdish denizens such as Microsofties, Googlers, Socratians, IBMers, Tropons, Amazonians, Pirilloians and now even Facebookers.

Geek Week is the creation of local Geek King Chris Pirillo. I envy Chris, with his 85,326 Twitter followers compared to my mere 2,232 followers. We both come from humble backgrounds in Iowa, and made it to Seattle to seek our futures in the Pacific Northwest.

Chris found his future and part of it is Geek Week and Gnomedex. The 10th Annual Gnomedex “Conference of Inspiration and Influence” is happening on Friday and Saturday, August 20th and 21st.

Socrata - click to see moreAs Gnomedex winds down on Saturday at 5:00 PM, the Open Government Hackathon winds up. The Hackathon is sponsored by two phenomenal local tech companies, Socrata and Tropo. Socrata has made its name making data open and transparent, most notably with data.gov and data.seattle.gov (well, and a few other sites).

Developers will converge on the Edgewater Hotel on (duh) Seattle’s waterfront. They’ll have 24 hours to use government datasets to create interesting applications. At 5:00 PM on Sunday, we’ll be judging applications for those which are most useful, interesting, unique or maybe just cool. There will be a number of prizes – hackers will get codes for some Amazon Web Services usable to deploy and test apps there and other prizes include a Flip HD camera, year membership in Amazon Pro and even an iPad – gee, if Microsoft or Google or Facebook made a nice slate computer, maybe that could be the prize. Perhaps next year!

Tropo - click to see moreRead more about the Hackathon on Chris Pirillo’s blog here, or the Socrata blog here, or the Tropo blog here.

In any case, it will be wonderful to see what sorts of applications the hackers develop, all with the intent not of hacking into government, but rather of making data held by government more accessible to citizens, residents and the people government serves.

Note: I’m especially proud of data.seattle.gov, with over a hundred cool datasets of information like fire department 911 calls, active building permits, and public toilets. That’s an initaitive of Mayor Mike McGinn and the City of Seattle’s Department of Information Technology, and we’ll be adding a lot more data to the site over the next couple of months, including police crime statistics, police 911 calls, and business licenses. You can already view crimes and 911 calls plotted by neighborhood on My Neighborhood Map here, powered by Microsoft’s Bing Maps and the employees of City government. The data feed of this information to data.seattle.gov will be active soon.

4 Comments

Filed under open data, Seattle DoIT

– FUD

Fear, Uncertainty, Doubt

For people who work hard to make government work, we live in frightening, uncertain times.   Even small messages and signals to the people who do the day-to-day work are important.

Recently we had an employee in my department (Department of Information Technology – DoIT, City of Seattle) whose card key was shut off to get to a certain floor after hours. It was inadvertent and an oversight – we were just trying to remove after hours access for anyone who really didn’t need it.  “Enhancing physical security”. 

But this employee immediately became frightened for his job – “are they planning to lay me off?” was the first thought he had.

Even small signals are important. 

I try to smile and greet each employee as I see them walking through the hallways or in work spaces.  I am very intentional about this.

First, I have a genuine respect and admiration for the people in DoIT – and around the City of Seattle – who make government run.  But also I just enjoy talking to people and hearing their stories. I know the first name of every employee in DoIT, and many other IT employees throughout City government, and I’m genuinely concerned about them, their families and their work.

Sometimes I forget, however, and I’m lost in thought, and I walk down the hallway scowling and forgetting to say hello. Employees can interpret that as “the boss is mad at me”, when, really, I’m just thinking about an especially difficult meeting I recent had, or a thorny problem I have to solve.

These are frightening times.

City government revenues are down, positions are being cut, and employees are being laid off. We have more difficulties coming down the road, and there is a significant amount of FUD – fear, uncertainty and doubt in the air. All you have to do is read Publicola, the local scandal sheet (now known as a “blog”) to see the facts and hear the rumors about this.

Yes, I know that I and other department directors will be faced with more cuts and more difficult decisions in the coming months. I am really hoping that the next budget process will be the last time we are cutting and we can stabilize the government after that. I’m a “glass half full” guy.

Nevertheless I lose a lot of sleep and spend a lot of time worrying about these issues and the effects of cuts on employees and their families.  And, even more importantly, on the health and well-being of the 600,000 people who live in Seattle and depend upon their government for safety, utilities and quality of life.

My lost sleep is irrelevant, of course – if I’m not here, the facts of the budget situation are still the same, and the cuts will still come, but it will just be someone else making the decision.

So if I scowl at you as I walk down the hallway, please don’t take it personally. I’m just puzzling over that next difficult decision.

2 Comments

Filed under budget, management of technology, people, Seattle DoIT

– Transclucent to the User

GEM Project Team with Mayor Nickels - click to enlarge

GEM Team with Mayor

On Monday night, December 8th, the Seattle Police Department started to use Microsoft Exchange/Outlook for electronic mail. This culminated moving more than 11,000 City of Seattle employees, over 12,400 e-mailboxes, and 900 BlackBerrys from an older e-mail technology to the Exchange 2007 product. All of it “translucent to the user”.

I’ve previously blogged about project management, and specifically identifying and reducing risks in large technology projects (“the P-I test“). With this entry I’m highlighting somewhat different project management practices.  We used certain techniques to reduce the impact of the technology changes on front-line City workers such as firefighters, accountants, and street maintenance staff.

(In case you think I’m just tooting our own horn, I am, but I’ve also blogged about my biggest project failure and you can read about that here, too!). 

We called this e-mail migration project GEM, for GroupWise to Exchange Migration.

Not only was the project on-time, under-budget and delivering all of its objectives, but there were very few whimpers from most City employees at this major change in their work lives. How was such a change so seamless?  

Electronic mail is, arguably, the most important technology used by workers in almost any company today, whether government or private.  It has supplanted the telephone and even the desktop computer as the key tool for many workers to be productive and efficient. Decisions which might take days or weeks without e-mail can be debated and handled rapidly with e-mail communication. Management of front-line projects (streets, water, electricity), debates and decisions on policies, notification of events, press releases, scheduling, all occur with this tool. Most importantly, it is a primary way for constituents and customers to communicate with City workers and elected officials and the way for those officials to coordinate the City’s response. 

Of course, when anything is this valuable in your life, you are extraordinarily skittish when it is NOT available or about to be significantly changed.  Managing this “culture change” – in the working habits of thousands of City workers – is the elusive key to success in a technology project.

I won’t get into the current debate (war?) about use of internal e-mail versus a hosted service, or whether Google’s g-mail is better or more cost effective than the Microsoft product set. Because e-mail is so important in our work lives, and because many people use Outlook at home (or in a previous job) anyway, it was the right choice for the City of Seattle. Because many e-mail messages are sensitive, and since I have a skilled and dedicated set of employees to manage and operate it, we would not have it hosted or managed elsewhere. Microsoft Exchange/Outlook is an established product, well-supported, used by 65% or so of the organizations in America today.  And many many other applications (purchasing or human resource systems, billing and customer service systems) are written to use Outlook/Exchange for communication.

Here are the elements of success for GEM:

  • Strong executive leadership. Mayor Greg Nickels fully supported this change, and every department director knew it. The nine-member Seattle City Council voted to fund the project ($4.9 million) after considerable, reasoned debate. These elected officials were able to articulate the rationale for making this change. This support helped immensely in cooperation for training, scheduling and acceptance throughout the Government.
  • Strong project leadership. My deputy department director sponsored the project – she has formal and informal ties to many line departments, and she’s managed many brick-and-mortar projects (e.g. building Parks community centers). She chose a strong project director who is a hard-nosed negotiator, and a skilled project manager who pays attention to both people and details.
  • Support. We chose, via competitive bid, a knowledgeable private partner – Avanade – to give us advice, skilled support and knowledge transfer. Avanade had helped many companies with similar conversions in the past, and performed in an outstanding manner for us.
  • Training. We gave employees a chance to purchase Microsoft Office 2007 via the home use program, and 2,000 of them took that chance, thereby learning the product suite at home. A month prior to each department’s conversion, we told them how to prepare, for example, by deleting old e-mail and taking training. We offered training in classes, video and reading material for anyone from heavy e-mail users to people who just needed a refresher on Outlook.
  • Communicate communicate communicate. We told all 12,000 employees at the beginning of 2009 what we planned to do (“to” them!)  One month out from their department’s conversion, we told them how to get trained and ready.  Two weeks out we communicated details via their management chain and via e-mail message. The day before conversion, each employee had a sheet of instructions placed on their chair. The day after conversion, technology staff chosen for their great “deskside manner” walked the halls and cubicles to answer questions and solve problems.  We had a skilled service desk / help desk and a special e-mail contact point. And all along we had a detailed, fact-and-fun-filled internal website with information, training, FAQ’s, and links to more resources.
  • Skilled City employees. We already had a highly competent help desk, capable desktop support staff and experienced engineers supporting servers and storage and messaging system.  We trained and leveraged this skilled and motivated set of employees, coupled with Avanade, to do the technical work on the project.
  • Finally – and perhaps this is most important, we drafted departments into the effort. Each department had at least one and usually a team of people who worked with the GEM project team to customize the training and conversion plan for that department’s unique needs. Police patrol officers use e-mail differently than Parks groundskeepers who are different than budget analysts who are different than electrical utility engineers. These “extended teams” in departments not only participated in the planning, but became natural advocates for overcoming problems and socializing the change in each department.

Leadership, communication, user representation, strong private partner, skilled and motivated technical staff – a GEM of a project, translucent to the users!

1 Comment

Filed under e-mail, management of technology, project management, Seattle DoIT

– My Biggest Failure – Consolidation

IT Consolidation and "Assimilation"

IT Consolidation and "Assimilation"

Everyone likes to tell success stories, particularly if the success occurred under your own leadership. But we all have failures and make mistakes. Few of us like to discuss them. I am writing about my biggest failure as CTO, hoping there is a lesson here for others.

I recently spoke to an IT consolidation retreat in Nashville hosted by the Center for Radical Improvement. In 2005-2006 we tried to partially consolidate information technology in the City of Seattle. It failed. Well, let’s say it was “less than successful”. And here is the “rest of the story”.

City government in Seattle has 11,000 employees, of whom about 550 work directly in technology and 215 of those work in the central Department of Information Technology (DoIT) which I lead. We have three service desks, three different radio systems, four large data centers, and at least six different groups providing server support and desktop support. We have at least five different work management systems, and some unknown number of document management systems.

On the other hand, we are standardized in many ways – a single e-mail system, Windows XP on all desktops, Oracle and SQL Server for databases, a single award-winning web presence at www.seattle.gov, an award-winning municipal TV station, one set of connections to the Internet, a single firewall, and a single financial management system and payroll system.

In 2005 the Mayor and I proposed a consolidation of technology infrastructure employees – about 100 employees would have moved from a dozen departments into the central IT shop – DoIT.

It failed. Why?

1. We did not calculate a return-on-investment and a potential cost savings from the consolidation. Such a cost/benefit analysis is essential to proving the case to elected officials. Furthermore, I promised there would be “no loss of jobs” due to the reorganization. I did this primarily because I felt we could re-deploy employees more efficiently to tackle a whole host of new projects. But I was also hoping to lessen the fear of change which is embedded in any organization, especially government, and especially during an impending organizational change. This is a dilemna – in difficult budget times, consolidation/centralization has a strong return-on-investment (ROI) and good support from elected officials, but the ROI comes from cutting jobs, which has a disastrous effect on morale.   Yet in “good” budget times, when jobs can be preserved, the support from elected and appointed officials is less compelling.

2. I failed my Mayor. Mayor Nickels made the decision for this reorganization. But I didn’t properly engage him – and his senior staff – in supporting and leading the change. Consequently many employees, department directors and others saw this consolidation as “empire building” on my part – an internal grab for power rather than an honest attempt to improve government.   Indeed, a Seattle City Council member openly accused me of “empire building” in a City Council meeting (he later, but privately, apologized for the remark).   That meeting is undoubtedly stored somewhere in the vast video archives of the Seattle Channel.

3. We did not get a consultant. Yes, there are many jokes about consultants. And good organizational consultants are expensive. But the blunt fact is simple: a comprehensive look at consolidation by someone outside the organization – a dispassionate outsider – would have greatly improved the credibility of the change. A good consultant would also complete a detailed cost/benefit analysis.

4. A labor union opposed the change. IT professional employees in the Department of Information Technology (DoIT) employees have decided not to be represented by a union. Many of the employees in other departments (who would be consolidated) are currently represented. Those employees probably would have lost that representation when moving. Although the numbers are small here – 35 to 40 employees in a labor union of 2,500 – there are larger principles at stake.

Generally, I believe in centralization of tech infrastructure functions – networks and data centers and computer operating system support. Certainly, we should have a single financial management system, budget system, and payroll system. Centralized functions are almost always more efficient, effective and secure.

In an organization our size, some applications support should be decentralized, for example, the software used to manage Seattle Parks Department resources and reservations is certainly different from the software used to manage resources for the Seattle Police Department or our electric utility Seattle City Light. Employees in those departments know best how to use technology to support their unique business needs.

Consolidations can be done well, as in Missouri by Bill Bott and Dan Ross, or Teri Takai in Michigan.

But achieving technology consolidation is hard. Although I’m proud of most of the work done under my technology leadership at the City of Seattle, I’ve had a failure or two as well. I hope others can learn from this. I certainly have the “scars of the school of hard knocks” from this experience!

2 Comments

Filed under consolidation, Seattle DoIT

– Tough Times, Tough Decisions

Seattle Technology Budget Cuts

I just finished one of the most difficult tasks a manager can perform – making preliminary decisions on budget cuts for next year. This is a job which is difficult in any line of work, and more so in government for several reasons.

For one thing, there’s an expectation that government is stable and long-term in its operations and its employment. It has to be. Despite the situation with the economy at large, water and electricity have to keep flowing, streets and parks need to be repaired and cleaned, 911 calls answered, cops and firefighters dispatched. Most of this work is at the very base of Maslow’s hierarchy of needs – safety and security. The people who perform these jobs for the public expect to have security in their jobs and the tools they use.

Yet I’ve found most government workers are motivated not by job security or money, but by pride. They – we – are proud of the work they do, and proud to be meeting the most basic needs of the people of our communities. I’ve given employees raises and promotions, but, again and again, I’ve watched their motivations inspired not by more money, but with a kind word or e-mail of appreciation, or being recognized in front of their peers for doing a good job.

Certainly the legal machinery surrounding government – or, really, nowadays, work in any large corporation or bureaucracy – reinforces job security. Civil service regulations, unions, personnel rules, and other legal protections all reinforce the expectation that many jobs in government are “permanent”.

Making budget decisions is hard – we talk in terms of “cutting positions” or “abrogation” or other fancy words. But I know the first name of (almost) every employee in my 200+ person department, and hundreds of others in City government as well. It is hard to separate the name or reality of “services cuts” from the people who do the work and are directly affected.

Almost as difficult is making the decisions about cutting tools and equipment versus positions. It’s one thing to have people to maintain a public safety radio network or operate a computer center. But we also need to provide switches and radios and large-scale computers and space to keep those functions operating.

There are the other jobs of government – the public information officers and municipal TV channel, and support for arts organizations and the library, not to mention feeding the homeless and housing the hungry (and vice versa). All these are important functions, all requiring people, as well as tools and materials and other resources. Finally – and most important, perhaps – are the visible jobs of government – the people who run community centers and libraries, the cops and firefighters, the workers who fill potholes and maintain the electric grid.

Although my job is tough, and my decisions are hard, I don’t envy the elected officials who have to make choices for the government as a whole. Then those elected officials need to explain those choices to voters – many of whom have lost a job or a home themselves. And those explanations often occur during the heat of an election campaign, when emotions and misinformation abound.

Tough times, tough decisions.

Note: A few more details about the budget issues with the City of Seattle’s technology are in a recent Puget Sound Business Journal interview here. The Seattle Department of Information Technology’s budget for 2010 was authorized at $59 million and 216 positions – see DoIT’s and the full City budget here.

Leave a comment

Filed under budget, employees, jobs, Seattle DoIT

– The 108 Degree Data Center

The 108 Degree Data Center

The 108 Degree Data Center

November in Seattle is always cool and rainy and sometimes stormy – windstorms, that is. Seattle’s all time high temperature – for any day of the year – is 100 degrees. That all time high is, of course, outside. But it reached 108 degrees here on Sunday November 16th. Inside a data center. The City of Seattle’s data center.

To make a short blog entry even shorter, I’ll skip to the root cause: a failed power breaker on a pump for the domestic water supply to the building housing the data center. The water supply flows to CRAC (“crack” or computer room air conditioning) units which, in turn, cool the data center. For HomeCity Security reasons, I won’t reveal the actual location of the data center, but let’s just say it is in a downtown 60 story skyscraper which also houses about 3500 office workers during the week. The problem started about noon and was fixed at about 8:00 PM.

The data center holds about 500 servers, storage systems and other equipment. We shut down a lot of servers and many services starting almost immediately. Nevertheless the temperature in the data center rose to that toasty 108 degrees, setting a new record high (sort of) for Seattle.

So why is this notable? For two reasons: the problem and the response.

In terms of the “problem”, let me assure you (especially if you live in Seattle) that cooling problems like this will be rare to non-existent in the future. Years ago we installed a one megawatt generator for backup power. This year we’ve been working a project to install “dry coolers”. These aren’t really “dry”, but the water cooling the data center will flow in a “closed loop” between the new coolers and the center, so we’ll no longer be dependent on external water or power supplies. Unfortunately, the dry coolers don’t come online until January, which is why we went to 108 degrees last Sunday.

But there’s a more general issue here – every city and county government has data centers and servers and vital information. Every area of the country is subject to some sort of a disaster and every government needs to have a backup and recovery plan.

But for what disaster should we prepare?

Here in Seattle, everyone is concerned about the “big one” – a magnitude 8.0 earthquake. While we need to be ready for an major earthquake, we have about one of those “big ones” every 300 years. Much more likely are disasters like last Sunday – a failure of water and cooling, a “meltdown” if you will (non-radioactive, however!). Or perhaps the disaster will be the opposite – too much water from a broken pipe, and a flood drowning those servers. Or – and this also happens in computer centers – a fire followed by (drum roll), a flood as the fire suppression system kicks in. Should we have a plan for “the big one”, that earthquake? Sure. But most of our disaster preparation effort should plan for the much more probable disaster of fire and water.

Finally, any disaster response plan has one element which is vastly more important than any other: people. And, on November 14th, the “people” (employees) of the City of Seattle and its Department of Information Technology performed splendidly. A dozen IT professionals showed up on site within two hours (despite interference from the traffic around a nearby Seahawks football game). The computer center manager – a 44 year employee and true hero Ken Skraban – was on site and immediately in charge. Two employees set up an IT operations center with an incident commander and support staff. Several responded to the data center and shut down servers in an orderly, pre-planned, color coded (red-green-orange-yellow) fashion, with the most critical servers (for example “Blackberry” support) staying up continuously. Server administrators from every major department in City government responded on site.

And when the crisis was past and cool water was again flowing to the “crack” units, those same folks brought all services up in an orderly fashion. And there was not a single call to the help desk on Monday morning as a result of our unanticipated “summer” high.

Disasters happen. Careful planning and skilled, trained staff will always mitigate their effects.

2 Comments

Filed under disaster, people, Seattle DoIT

– Dial Tone comes from God

Dial Tone comes from the City

Dial Tone comes from the City

Or so says the manager of telephone services for the City of Seattle, Stephanie Venrick. What she’s referring to, of course, is that when you pick up a telephone, the dial tone is … well … there. You don’t think about, you just dial. On the other hand, mobile phone users can’t take connectivity for granted. Cell signals come and go, even with companies who promise “more bars in more places” (and they are not talking about building prisons!) Yet we expect the old-fashioned “wired” telephone to deliver dial-tone and connect phone calls day-in, day-out, without fail.

But providing that dial tone is not easy. Stephanie manages a group of about 30 skilled technology people who build, install and maintain the internal City of Seattle phone system of 23 large switches, more than 100 smaller switches, 11,000 phones, 7,000 voicemail boxes and other services such as interactive voice response (Press “1” for this, press “2” for that).

At first thought, you might ask “why does a City government have its own phone system”?  But, as a matter of fact, most large organizations, corporations and public agencies have their own internal telephone systems because it is cheaper and more reliable to operate such systems than to procure services from a public telecommunications company.

For a City government, it’s also a case of disaster preparedness.  The public phone system gets overloaded during earthquakes and on Mothers’ Day and, even, gee, when the Seattle Mariners’ tickets for the World Series go on sale (as if that will ever happen!)  Especially during disasters such as terrorist incidents or earthquakes, the public cell and land-line networks are vastly overloaded.  With the City operating its own telephone network, City functions and facilities can still operate and coordinate our internal response to the disaster.

Doing all of this should be easy, right?  After all, it is basically two telephone sets with copper wire in between – just one step up from the two-tin-cans and string phones we played with as kids?

Alas, just as the two-tin-cans toy for kiddos has been replaced with the high-tech Xbox 360 and Wii, so has delivering basic dial tone been replaced with the marvels of technologies such as fiber optic cable, voice-over-Internet-Protocol (VoIP), and complex automatic call distribution systems.

Today large portions of the City’s phone system rides on the City government’s internal Internet, traveling on the same pathways as public safety radio transmissions and computer-to-computer traffic.

While more complicated, this set of networks gives us quite a bit more flexibility because the City government owns and manages its own services. W ith the IVR (interactive voice response), for example, City customers can get the balance on their electric bill, or pay their water bill or even pay a parking ticket with a credit card. We can highly customize distribution of phone calls, so that customers rapidly reach a city employee/specialist to answer specific questions or render service.

Putting telephone, data, radio all on the same fiber network saves taxpayers a lot of money when you are connecting 11,000 employees to 600,000 Seattle residents scattered across 142 square miles (40% of that being water) with many lakes, rivers, hills and a ship canal to provide additional challenges to making this one of the most “wired” cities.

Yes, Dial Tone does come from God, or at least the City of Seattle, but only with the help of a lot of angels in the guise of the City employees named “Telephone Services”.

P.S.  The City of Seattle is one of the very few governments or corporations to put a phone directory of almost all its employees’ on the web.   Click here to see it.

2 Comments

Filed under Seattle DoIT, telephone