THE nationwide ground stop Friday could have ended earlier if the contractor who performed the work related to the disruptions at the main air traffic control facilities had located the problem sooner, according to Transportation Secretary Sean Duffy.
Investigators are still working to determine exactly why the newly installed router at the center of it all began blowing up thousands of connection attempts shortly after it was turned on as part of a project to replace a section of old copper wire with fiber optic lines in Arkansas.
Duffy appeared on streaming platform 2WAY’s “The Morning Meeting” Wednesday and said L3Harris failed to quickly enough identify and isolate the router problem that caused the ground shutdown. This contributed to it being the longest ground stop since the attacks of September 11, 2001, because they lasted more than four hours.
“Network monitoring is important,” he said, adding that officials were working to remedy the problem. In the meantime, any new installations of these same routers have been suspended and technicians have disabled the function linked to Friday’s outage.
Experts continue to test equipment related to the issue to determine why this Juniper router behaved differently than those installed in 68 previous projects on the FAA network. Investigators are focusing on whether the router was misconfigured by the experienced technician who performed the work or whether it was defective. The company that made the router said it believed “the device worked as it was configured by the installer.”
Friday’s problems recall 2009 outage
The circumstances of the technical problems that led to last week’s nationwide ground shutdown bear an uncanny resemblance to a 2009 outage caused by a technician doing the same type of work installing new fiber optic lines.
The Department of Transportation’s inspector general said the 2009 disruption that resulted in 820 delayed flights was caused by a Harris technician who misconfigured a router. Harris later became L3Harris in a merger, but that company has operated and maintained the FAA network for decades.
Matthew Hampton, who was deputy inspector general for aviation at the time, said the FAA had long had weak oversight of the contractors it relies on.
“Poor oversight of contractors has been a problem at the FAA, both in the development of the systems and in their deployment, there’s quite a history about that,” Hampton said.
Aviation safety expert Jeff Guzzetti, who worked in the Inspector General’s office for several years, said inadequate oversight of contractors was a problem in the FAA’s latest attempt to overhaul its systems called NextGen.
Normally, the FAA is pretty good at dealing with unexpected challenges and adapting quickly by transferring air traffic control responsibilities to a nearby facility when one location experiences a problem or temporarily slows traffic. But Guzzetti said that wasn’t possible this time because the entire network was hit Friday, including the primary and backup systems.
“The FAA is really good at adapting, improvising and overcoming events. But if the whole system is broken, it makes it very difficult,” said Guzzetti, who has also investigated accidents on behalf of the FAA and the National Transportation Safety Board.
L3Harris has not commented on the ground stop since releasing a statement Saturday pledging to support the FAA’s investigation into what happened. The company declined to comment further Wednesday.
FAA Administrator Bryan Bedford said at another event hosted by The Hill Wednesday morning that he understands his the agency is ultimately responsible for what happened even though it was L3Harris doing the work.
“I take the blame. I take full responsibility for it,” Bedford said. “I wish people would fire the secretary. He has a lot of work to do, but I’m the person they hired to do this. So blame me.”
It’s unclear how much additional money Congress will approve for the overhaul
The FAA has almost doubled its funding request to Congress to $50 billion after Friday’s ground stop to fund the rest of the overhaul of the country’s air traffic control system. Last year, the agency received $12.5 billion to launch the project. Bedford said he understands why members of Congress might have “some indigestion associated with this.”
Several key lawmakers said Tuesday that Congress is unlikely to approve that full amount at this time — especially since the FAA hasn’t spent all of the money approved last year. Duffy said a big part of the reason some of the money hasn’t been spent yet is because the new radars and other systems that will be installed are still under construction, so he hasn’t been able to purchase them yet.
Only about $10 billion of the $50 billion requested would be spent on air traffic control technology. The rest of the money would be spent on other projects, including upgrading runways, airport towers and aging air traffic control centers built in the 1960s.
Other issues are possible when upgrading outdated FAA system
Chris Sununu, president of the trade group Airlines for America, praised Bedford for keeping airlines informed during the ground stop. Sununu said Bedford asked to set up a conference call with all the airline CEOs and operations managers about a half hour after the layover began and briefed them on what he knew so far. Several additional calls occurred throughout the ground stop.
“To be honest, he effectively stayed on the phone with all the CEOs all night,” Sununu said. “We made, I think, three or four calls over the next few hours, to get these folks up and running. And so, as they learned more, as they got a better understanding of what was going on.”
Sununu said the FAA has a difficult task trying to overhaul “what is truly the most robust telecommunications system on the planet while you’re using it 24 hours a day and ensuring the safety of everyone who’s a part of it.” And he said this may not be the last technical problem that arises during the project, because the work the FAA is doing is akin to trying to get your iPhone to talk to your flip phone.
“Can we say ‘oh, you know, is this really the last time we’re going to have a bump like this?’, no, no. Because the technologies that we’re replacing and asking to talk to each other are literally multi-generational,” he said.
____
Associated Press writers Ben Wieder, David Collins and Sudhin Thanawala contributed to this report.
Gn bussni
