The Story Behind the FAA Flight-Plan System Crash

eWeek has an article about Thursday’s crash of the FAA’s flight-plan filing system and the background behind the system. I’m amazed but not surprised that a single router managed to take down the national system. I’ve seen some wild outages caused by a single rogue network device back in my network engineering days.

When the router went offline, only the system maintainer—government telecommunications contractor Harris—knew that the backup card was not immediately available, and that one technician, who hadn't come to work yet that day, had the key to the storage closet where the part was kept. So the FAA had to wait until this technician was able to come to the site in Salt Lake City to replace the faulty card inside the router, reconfigure the software, and get the communications backbone back up and running so that the nation's air traffic could get back to normal.

Interesting, I'm not surprised that, even with the system down, probably costing the country millions of dollars an hour in delays and cancelled flights and with their employers reputation and contract with the government being severely damaged a bunch of systems engineers continued to sit around waiting for the right employee to get there with the key so they could get into the supply closet. But speaking as a mechanical engineer I would have physically forced (ie sledgehammered) my way through the door to fix the problem. Destroying a $100 door to fix a problem of this magnitude an hour or two sooner should be a no-brainer.

You gotta love idiots using buzzwords that they've no idea have a very real technical meaning.

Harris spokesperson Marc Raimondi told eWEEK that people should keep in mind that weather conditions cause most flight delays, and that the FTI system used by the FAA has a very good performance record. "Five nines—maybe even nine nines of efficiency," Raimondi said.

nine nines is one second of downtime every 30 years, or a 30th of a second of downtime in a year! A burst of network lag too small to be noticeable by a user would break that level of performance.

I'm stunned that there was only ONE copy of a key, and tech was allowed to take it offsite.

Nice DR planning. Der.

with practices like that the FAA sure makes me feel safe in the air.

/sarcasm

Sounds like something out of the last season of '24'. Wheres Jack Bouer when you need him!

Originally Posted by qdemn7

I'm stunned that there was only ONE copy of a key, and tech was allowed to take it offsite.

If I was the manager I would have tried to hack the door open with an axe if I have to. If only to reduce the down time by an hour.

yeah, why didnt they just break down the door? door handles easy to replace.

Originally Posted by __hollywood|meow

yeah, why didnt they just break down the door? door handles easy to replace.

I guess because no one had the balls to take a step forward and actually do something about it. Afraid they will get into trouble for breaking a door. What they didn't understand is that they will be a hero if they did break the door and fix the problem.

One word:
Government

Originally Posted by qdemn7

I'm stunned that there was only ONE copy of a key, and tech was allowed to take it offsite.

not only that, but dont all critical systems have auto backup? mainly its in the form of just doubling up your hardware... run 2 routers and if one dies, your bandwidth gets cut in half but at least you're not dead

Originally Posted by Dez Perado

Nice DR planning. Der.

I see this more as a failure to follow Business Continuity guidelines, which in turn meant that they were unable to effectively implement their DR plan. Regardless, a mess.

Originally Posted by SparkedFire

One word:
Government

By Government you mean free market contractor right? They said in the article if it was still run by the FAA that it would have been fixed in minutes...

if it was still run by the FAA that it would have been fixed in minutes...

roflsaurusrex

Originally Posted by davidm71

Sounds like something out of the last season of '24'. Wheres Jack Bouer when you need him!

He's got bigger problems, that's why he can't help the FAA right now. His daughter is behind EVERYTHING!! He doesn't have time to explain, DAMMIT, he'll debrief you when he's done Jack Bauerin' bad guys!!

Don't you know doors are indestructible? Only the proper key card can open it. Games have taught you this...besides the people in charge of the door were probably too weak and stringy to bust down a door. There instrument of destruction would probably bounce and hit them in the face.

Originally Posted by AaronPSU79

Interesting, I'm not surprised that, even with the system down, probably costing the country millions of dollars an hour in delays and cancelled flights and with their employers reputation and contract with the government being severely damaged a bunch of systems engineers continued to sit around waiting for the right employee to get there with the key so they could get into the supply closet. But speaking as a mechanical engineer I would have physically forced (ie sledgehammered) my way through the door to fix the problem. Destroying a $100 door to fix a problem of this magnitude an hour or two sooner should be a no-brainer.

I totally agree, the only way I wouldn't of just cut the lock off is if it was a LOTO, the need of cutting those things usually results in a firing of either the cutter or tagger.

Originally Posted by sc3252

By Government you mean free market contractor right? They said in the article if it was still run by the FAA that it would have been fixed in minutes...

Yea either lowest bid, or no bid contract. Seems those are the only kinds their are, the former being utter crap, and the latter being way overpriced.

Originally Posted by AaronPSU79

Interesting, I'm not surprised that, even with the system down, probably costing the country millions of dollars an hour in delays and cancelled flights and with their employers reputation and contract with the government being severely damaged a bunch of systems engineers continued to sit around waiting for the right employee to get there with the key so they could get into the supply closet. But speaking as a mechanical engineer I would have physically forced (ie sledgehammered) my way through the door to fix the problem. Destroying a $100 door to fix a problem of this magnitude an hour or two sooner should be a no-brainer.

As an ME myself, I would have done the same thing.