FirstNet’s Scandal and Resurrection

[This version of the post has two updates as noted.]

McClatchy Newspaper’s Greg Gordon just wrote a well-researched investigative article about procurement problems with the nation’s First Responder Network Authority (FirstNet).  The details in the article correspond almost exactly with my mostly second-hand knowledge of the situation.   But I am hoping FirstNet and the nation can, with help, put this episode behind us and proceed to actually building a nationwide wireless broadband network for our brave responders who protect the safety of 320 million Americans.

Sam Ginn

The details of this problem are well-known to insiders and, with Gordon’s article, now to the general public:

1. In February, 2012, Congress creates FirstNet, funds it with $7 billion from sale of spectrum, and directs the appointment of a 15 member Board of Directors. The Board consists of five federal members including the Attorney General and Secretary of Homeland Security, five members from “public safety” agencies, and five members with commercial or industry background.
2. The Secretary of Commerce appoints the Board in August, 2012. The commercial members include wireless industry veterans Sam Ginn and Craig Farrill.   The Secretary appoints Ginn as the Chair of the Board.   Neither Ginn or Farrill have previously worked in government and are unfamiliar with many of the laws, regulations and practices of government agencies.
3. FirstNet, although an “independent agency” under the law, finds itself subordinate to the National Technology and Information Agency (NTIA), and subject to all Federal personnel and procurement regulations. The personnel regulations severely restrict how fast FirstNet can hire full-time staff.
4. Ginn and Farrill are anxious to get the network built as rapidly as possible, just like they’ve built private companies like AirTouch in the past. They use existing federal contracts to hire a set of 35 highly skilled technical staff at large salaries – up to $600,000 a year – to get the network designed.  One of those individuals, Bill D’Agostino, is named the General Manager of FirstNet.  NTIA and the National Institute of Standards (NIST), both agencies in the Department of Commerce, apparently acquiesce to this hiring.
5. Almost all the contract staff are former acquaintances and co-workers of Ginn and Farrill.
6. None of the hiring, the salaries or the details of the staffing contract are known to the public or the public safety community who will be served by FirstNet.
7. The Sheriff calls “foul” on this practice in a public meeting of the FirstNet Board. The Sheriff is Board member Paul Fitzgerald, elected Sheriff of Story County, Iowa.
8. After gnashing of teeth and probably a bit of weeping, the contracts are canceled, the high-paid contractors are terminated and the Inspector General launches an investigation (which still hasn’t been concluded).
9. D’Agostino, Ginn and Farrill resign.
10. In the meantime, FirstNet, under the direction of J. Kennedy, a former cop, firefighter and paramedic, builds a competent staff of over 60 federal employees and other contractors, and gets FirstNet back on track.

Greg Gordon’s article has all the details.   Again, based on all my knowledge and discussions with individuals involved, these details are correct except for two:  First, the Public Safety Advisory Committee  (PSAC) to FirstNet has at 40 members, not 5 as Gordon mentions.  Second, the initial contract for FirstNet staffing was let by a semi-competitive solicitation in late 2012.  this is the solicitation published under the authority of the U.S. Census bureau.   I say “semi-competitive” because competition was limited to an existing set of GSA-pre-qualified contractors, not open to all bidders.  (This paragraph updated from the original post.)

So what’s the truth in this?

I think both Sam Ginn and Craig Farrill are honorable people, recruited by Larry Strickling, Director of NTIA.  Ginn and Farrill took their mission seriously.   They knew they were, essentially, in charge of a start-up company.   They knew getting the network operational was the mission.   And they set out to do it using every bit of their business skill and acumen.  They hired people who they worked with before, and who they knew could do the job.   They did not pay much heed to salaries.  “Damn the torpedoes, full speed ahead.”

What Ginn and Farrill did not know was government.   They did not know how to run public meetings or how to respond to public disclosure requests.  Meetings occurred behind closed doors, begrudgingly televised with 1990s-era video tech.  They probably did not keep all the members of the board (e.g. Sheriff Fitzgerald) in the loop about their activities.   They either did not know about federal competitive procurement regulations or – worse yet – perhaps didn’t care.

There’s also the possibility that Ginn and Farrill were misled – that they thought the law’s statement FirstNet would be an “independent authority” under NTIA truly meant “independent” in the fashion the Tennessee Valley Authority or Bonneville Power Authority are independent.  And that’s independent from Federal Personnel regulations, the Federal Acquisition regulation (FAR) and similar constraints.  And, after they arrived, and tried to be truly independent, the boom was lowered.  (This paragraph added to the original post.)

Worst of all, they did not spend much time consulting their constituents, their future users, the cops and firefighters and other responders who need FirstNet.   They basically ignored and did not use the Public Safety Advisory Committee.

As one example of this, at the first meeting of the Board, on September 25, 2012, Farrill presented a “conceptual architecture” for FirstNet.    Where this architecture originated was a mystery to the hundreds of public safety officials – including me – who had been working on FirstNet and its predecessors for years.   Clearly Farrill was clueless about consulting constituents.

As another example, Sam Ginn famously testified in front of Congress that FirstNet would cover “every square meter” of the United States.   Mr. Ginn, honorable as he is, didn’t know much about testifying to elected officials or making promises.   There are a lot of pretty damned remote, hard-to-reach, “square meters” in the United States, some of them less than 50 miles from my home in Seattle.

Sheriff Fitzgerald

Sheriff Paul Fitzgerald finally became fed up with this lack of consultation with public safety, and came out with a damning indictment of it during the April 23, 2013, Board meeting.   Fitzgerald, like Ginn and Farrill, is an honorable man, elected multiple times to public office, and well-versed in government.   Fitzgerald’s failing was not involving his fellow public safety Board members – Fire Chief Jeff Johnson, Deputy Police Chief Chuck Dowd, and Kevin McGinnis, a paramedic and director of emergency medical services in Maine – in his concerns prior to the meeting.  They were just as startled about his accusations as other Board members.    Most elected officials of City and County Councils and State legislatures know they need at least one other person on their side to second their motions.

Where laws broken and is criminal prosecution in the works?

I doubt it.   Commerce Department Inspector General Todd Zinser is looking into the allegations of illegal or unethical contracting practices.   Perhaps he will find some NTIA or NIST officials bent the law in allowing the high-salary contractors to work on FirstNet.   It certainly is odd (and many of us puzzled over it at the time) that the first solicitation for contractors came from the United States Census!

With the IG’s upcoming report there’s another shoe to drop here, but I hope we don’t waste a lot of time waiting for it.

T. J. Kennedy

Ginn, Farrill and D’Agostino left of their own volition.   Sheriff Paul Fitzgerald and Deputy Chief Chuck Dowd were not reappointed to the board.  (To some extent, I think Sheriff Fitzgerald was punished for blowing the whistle).     These are all honorable people trying to do their very best to support the public safety of the nation.   Like all of us, sometimes they make mistakes.  These key players in this drama are gone, and it’s just the mop-up of the Inspector General’s report which remains to put this scandal to bed.

I see great promise in FirstNet, and a new awakening of purpose under new Board Chair Sue Swenson’s and Acting General Manager T. J. Kennedy’s leadership.

Let’s let them lead, unburdened by the past.

FirstNet Moves into First Gear

FirstNet’s First Gear

The First Responder Network Authority, charged with building a $7 billion nationwide network for responders and now two years old, moved into first gear this week.

In fairness, FirstNet was never stalled or stopped, although it appeared that way when I wrote “Is FirstNet Stalled?” on its two-year birthday, February 22.     Work was going on behind the scenes, and it burst out onto the stage this week:

• A new website appeared, www.firstnet.gov, freed of the clunky National Telecommunications and Information Administration logo and design;
• General manager Bill D’Agostino unveiled the most detailed org chart to date, which showed 40 full-time employees and another 50 or so on the way;
• Leases and office space in Boulder, Colorado, and Reston, Virginia, are virtually complete;
• Ed Parkinson (Director of Government Affairs), David Buchanan (State Plans) and Amanda Hilliard (Outreach) unveiled a “high level” 45 step plan for working with individual states to develop a plan and design for the construction of FirstNet in each state;
• The Strategic Planning process has a bit more detail;
• At least two FirstNet officials established twitter accounts and followed my twitter feed in the last week – and I’ve followed them back.   This indicates a new openness and freedom in how FirstNet staff is operating.  (But I’m not revealing their names in order to prevent the NTIA enforcement apparatus crashing down onto them for violating some obscure policy.)
• It appears, from the slide at right, 
  that FirstNet will support non-mission critical voice, perhaps at the time of launch.

Overall, I’m encouraged.

As the State Point of Contact for Washington (the state, not the place inside the beltway), I especially appreciate the additional information we received this week.  About 70 officials attended a conference in Phoenix for those of us in the western states who are working to prepare our states for FirstNet.    Each state already has a state-and-local-planning grant (SLIGP) for this work.   But many of us were waiting for a “starting gun” to launch our outreach and education efforts.  These efforts will find every potential Firstnet-using agency in our states:  law enforcement, firefighters, emergency medical, transportation, transit, public works, electric and water utilities, schools and everyone else with a public safety mission.

That starting gun is now fired.

We can proceed with that outreach.

State Consultation Process

We also know – and this is new information – that FirstNet will need to collect some additional detail about potential users:  the name of each agency, a point of contact, the number of potential users, the kinds of devices, any existing use of a commercial service and, perhaps, a bit more.  We don’t know the exact nature of the information to collect.  We’ll find out the details when FirstNet comes to our states for an initial meeting, probably sometime this summer.   And we expect there will be a data portal or template to standardize the way the information is collected.

Everything is not, however, sweetness and light.     Potholes and bumps are still sitting on FirstNet’s roadmap  to attain our vision of a nationwide public safety wireless broadband network.

The business plan is still a mystery.

FirstNet officials say there are multiple paths to a viable business plan.   However FirstNet needs to build a network which covers a lot more geography than any commercial network – “every square meter” according to Board Chair Sam Ginn.    It needs to do that with about 5.4 million users, compared to more than 100 million each for Verizon and AT&T, and over 40 million each for T-Mobile and Sprint.   And its per-user subscriber costs need to be comparable to commercial providers, or many public safety agencies cannot afford to switch.   I’ve blogged elsewhere about elements which might constitute a viable business plan, including putting FirstNet in every consumer and business mobile phone, or building sensor networks such as electric utility smartgrid using FirstNet spectrum.

FirstNet has a long way to go to become more engaging and transparent.

• It’s good to see the more detailed org chart, but who are all the full-time employees, with titles and contact information?  Most government organizations have a detailed staff directory (here’s the City of Seattle’s directory of about 10,000 employees and departments and services).
• The new website is a worthy effort and an MVP (no, not “most valuable player” but “minimum viable product”).   Over time, hopefully, it will become timely and engaging, with one or more blogs, twitter feeds and even discussion boards as well as FAQs and a “mythbusters” section similar to what the Texas Department of Public Safety has built.
• If I was in FirstNet senior management, I’d blog or publicize every person FirstNet hired – full-timer or contractor.   I’d publicize the unvarnished (or only slightly varnished) input received at every public meeting.  Every such piece of news – including things which are not flattering – contributes to the desirable image of continuing progress:  a juggernaut moving to fundamentally change and improve public safety in the United States.

There’s still a question of how “independent” FirstNet can become from NTIA.  Andy Seybold feels NTIA called the shots on a recent hiring process.  If FirstNet can achieve some of the transparency objectives I’ve outlined above, you’ll know it is becoming an entrepreneurial startup, not just another federal bureaucracy subject to restrictive, risk-adverse publication and social media policies.

And the staffing challenges remain significant.   FirstNet has hired just a few contractors who are vitally needed to evaluate RFIs, write RFPs and build a design for each state.    But it needs many more, and the task orders have not yet been issued.  The names of the existing hires – as well as the roadmap or even job descriptions to hire additional staff – are shrouded in secrecy.

Overall, however, FirstNet appears to be in first gear.   Just first gear:   we’re not barreling down the public safety broadband highway yet by any means.   You crawl before you walk and run.   And it will take more staff and better plans to get into overdrive.

But at least we appear to be back on the highway.

Is FirstNet Stalled?

The short answer is no.   The longer answer:  maybe.

Today, February 22^nd, is the second anniversary of the Spectrum Act.  Congress passed that law on February 22, 2012.    It created the First Responders’ Network Authority.   The law was the culmination of over a decade of advocacy by many public safety officials who saw the inadequacy of responder communications in the wake of disasters like 9/11, Hurricane Katrina, and many smaller incidents.  In these incidents cops and firefighters and paramedics and other responders found themselves unable to adequately communicate and protect the public.

FirstNet’s mission is grand:  to build the first nationwide public safety communications network for responders, especially first responders to both daily incidents and larger disasters.

Here we are, two years into the ten-year mission authorized by Congress.  It has been a slow start, and lately – over the past 6 months – FirstNet’s progress appears to have either stalled or is undergoing a reboot.

This is very frustrating for those of us in states and cities who are trying our best to evangelize and support FirstNet’s mission.   I’m the FirstNet State Point of Contact (SPOC, commonly pronounced “spock”) here in the Other Washington on the west coast.    I’ve been speaking to groups of public officials and police chiefs and emergency managers and firefighters and other responders in Washington State about FirstNet since May, 2013.

Lately, the mood of the audiences is starting to change.   “Yeah, yeah, we’ve heard you say that before, Bill, but what’s happening now?  Where’s the beef?”

I’m starting to feel a bit like a computer software salesman pushing vaporware.    “Oh yes, that feature will be in our next release slated to come out in 2017”.

So here’s my take on what’s going on Inside-the-Beltway.

1.  Here come the Bureaucrats.   There is one phrase in the Spectrum Act which causes a lot of confusion:  “There is established as an independent authority within the NTIA the ‘First Responder Network Authority’ or ‘FirstNet’” (47 USC 1424 Section 6204).

An “independent authority” “within” a long-established bureaucracy?    What the hell does that mean?   Well, I’m sure lawyers at NTIA and the Departments of Commerce and Homeland Security and the FCC have all been spending thousands of hours trying to figure that out.

I know if I was a head bureaucrat at Commerce or NTIA that I’d interpret it as having another function (or office or directorate or whatever the bureaucratize is) within my organization.   In other words “You report to me, FirstNet. Start acting like all other NTIA offices.”

I suspect there is an epic struggle going on within the Beltway for the control of FirstNet and its $7 billion in funding.  I don’t have direct evidence, but if you look at job descriptions which have been posted, e.g. for the Chief Information Officer, they clearly stated the FirstNet CIO would report to the NTIA CIO on a dotted line and would enforce NTIA information technology policies.  We know FirstNet is subject to all Federal personnel procedures for hiring staff, issuing RFPs and doing procurements.   FirstNet Board members have publicly said it will take them a full year to develop and issue and receive RFPs.

So much for the “independent” part of that law.

2. Contract staffing.    FirstNet’s already had a scandal.   Story County, Iowa, Sheriff and FirstNet Board member Paul Fitzgerald spoke out at the April 23, 2013, Board meeting.  Sheriff Fitzgerald protested, among other things, conflicts of interest between board members and contract technical staff hired to do the real meat-and-potatoes work of designing and building the nationwide network.   I’ve heard – but cannot verify – that some of the contract staff hired in late 2012 and 2013 were paid $300 an hour.

Now hiring contract staff for engineering and technical work at market rates is done all across the federal government.   Federal employee pay scales are compressed and have been kept low for a number of years by Congress.  So hiring outside technical staff is a prudent action.  The allegations of Sheriff Fitzgerald go far beyond just cost, however.   They also relate to the contract vehicle used, how the staff were identified and hired, and more.   And the conflict-of-interest allegations are still open and under investigation by the Inspector General of the Department of Commerce.

But here’s the upshot:   the contract under which the staff were hired expired in October, 2013.  Most of the existing 35 or so contracted staff (who were quite competent, by the way) were laid off.    Three new contracts were established in October.   But as of this writing – four months later – no technical contractors, and only a handful of public relations contractors, have been hired.

How do you create a nationwide design and individual state-specific plans for a wireless network without technical staff?

I suspect #1 above is at play here – the typical reaction of any government bureaucracy to allegations or scandal is to circle the wagons and lay on the rules, regulations, oversight, multiple approvals by multiple levels of officials.     This doesn’t bode well for either the short-term or long-term ability of FirstNet to get the staff support it needs.

3.  Full-Time Staffing.    I think FirstNet has about 25 federal employees working for it.  Their goal, I believe, is to have 100 or more full-time staff to do the work.

Gee, two years into a $7 billion project and only 25 full-time staff have been hired!?    And, frankly, most of those folks are transfers from other federal departments such as Commerce and Homeland Security.   In the Federal personnel system, it is relatively easy to hire and transfer existing federal government employees.   It is much harder to hire from the non-Federal staff – especially folks with on-the-ground responder experience.  Multiple interview panels and layers of human resource review, not to mention background checks and financial disclosure.

There have been a few major hires from the outside – General Manager Bill D’Agostino with commercial/Verizon background, T.J. Kennedy with Utah State patrol background, and Bill Casey formerly of the Boston Police via the FBI.   But key positions go unfilled, such as the CIO and CTO positions.

Despite the difficulties, every full-time person working at FirstNet who I personally know – no matter what their background – is very committed and competent.

But, again, #1 is at play, and at this rate it will be years before FirstNet gets its complete complement of full-time staff.

4.  Stiffing your friends.     Eight cities, regions and states around the country were funded for about $400 million under the Federal stimulus (technically the American Recovery and Reinvestment Act’s Broadband Technology Opportunities Program, ARRA/BTOP, a mouthful) or similar grants to build public safety LTE networks compatible with FirstNet.  Many of these networks were well along – Harris County (Houston) is operational.   Charlotte, North Carolina and Mississippi (statewide) were substantially deployed.  The San Francisco Bay area (BayRICS) was moving rapidly in planning and site development.

But when FirstNet was created in 2012, NTIA abruptly stopped seven of these projects, restricting their construction until FirstNet could review them and authorize them to be completed.  FirstNet started negotiations with them, but in the case of Charlotte and Mississippi, those negotiations have fallen apart and the LTE part of the networks is shut down.    Motorola, vendor on the BayRICS project, was unable to reach accord with FirstNet and gave up its BTOP grant in December, 2013.

I don’t know specifically why each of these negotiations failed.   In some cases I believe it was FirstNet’s refusal to promise to incorporate the local network into its overall nationwide plan.   In other words, FirstNet might actually overbuild the BTOP-funded network in the city, region or state.  Such an overbuild would not give the local agencies time to recoup their investments.  In other cases FirstNet refused, I think, to allow the local network the ability to expand over time and improve coverage in its geographic area, which could hamstring the use of the network by responders.

Each of these jurisdictions invested considerable local funds and political capital, not to mention time and effort, into these projects.    The projects, if completed, would have been showcases for the promise of FirstNet.    More importantly, FirstNet would have created a cadre of mayors, elected officials, Sheriffs, police chiefs, fire chiefs and others singing the praises of public safety broadband.

Maybe the states were asking too much of FirstNet.   Perhaps the lawyers got things tied into legal knots.    Maybe the business plan for funding and operating these networks wasn’t going to work under any circumstance.   All I know is that now, in these jurisdictions, there is simply bitterness over a failed effort and promise.

And there are four jurisdictions which DO have spectrum leases with FirstNet, although their timelines and deliverables are still murky.    Undoubtedly there are lessons to be learned and advocates to be created via those projects.

5.  Overpromise and under-deliver.    We’ve had a number of false starts.  At the very first Board meeting, in September, 2012, member Craig Farrill announced a “conceptual network design”.  Really?   Where was the collaboration with the public safety community before this announcement? At regional meetings in May and June 2013, FirstNet Board members were talking about coming out to states and meeting with Governors within 60 days.    Yeah, right.    In the fall of 2013 we in the states were hoping to have a lot of specifics in terms of materials and data requirements to conduct outreach and education for potential users in our states.   We’re still waiting.   Even minor things like having a viable website at www.firstnet.gov branded for local and state public safety has been promised since summer, 2013.   Today that website has still got the NTIA brand all over it, and is only minimally functional.   Gets us back to #1, I guess.

Still, I’m hopeful.

I’ve listed a whole set of concerns and issues, but I also see some positive signs.

This coming week and in early March, FirstNet and NTIA staff will hold two workshops for the SPOCs and our staff in the Eastern and then the Western U. S.    We’re hoping to see a clear roadmap for the FirstNet’s ahead.    Deputy Manager T. J. Kennedy recently laid out much more detail on how the consultation with states will occur.   FirstNet has published a number of Requests for Information (RFIs) seeking a lot of information from potential vendors and others on a number of aspects of the network ranging from devices to network design to applications and apps stores (although, with the staffing shortfalls mentioned above, I’m not sure who is reading the responses).    General Manager Bill D’Agostino says his plan for the year ahead “will make your head spin”.

There are hundreds of us out here, FirstNet, who still believe in you, believe in the mission, and want to help make it happen.