Tuesday, September 5, 2006
We would like to thank Prof. Jon A. Krosnick for providing us with our first guest blog. Prof. Krosnick is Professor of Communication, Political Science, and Psychology and Director of the Methods of Analysis Program in the Social Sciences at Stanford University. He was a consultant to the National Aviation Operations Monitoring Service project.
Among the many challenges that face air travelers these days (long lines, weather delays, cancelled flights, and crowded planes), there is something good to think about: passenger deaths in crashes of U.S. commercial jet aiplanes have been much more rare during the last 5 years than at any prior time in modern aviation history, suggesting a dramatic increase in the safety of domestic air travel.
Wonderful news, if it’s true.
But we can’t tell simply by inspecting the number of aircraft accidents per year.
Doing that is like watching coin flips come up heads four times in a row and saying, "The next one’s got to be heads."
Or saying that because Aunt Gertrude has been smoking 2 packs a day for 30 years and hasn’t gotten cancer yet, cigarettes won’t cause her to die prematurely.
The horrific passenger deaths that occur when a plane crashes are only the tip of an iceberg that peeks through the surface of the water extremely rarely. To know whether the risk of a disaster has in fact decreased, we need to know the size and shape of the hidden part of the iceberg.
That hidden part is made up of lots of little incidents that happen every day in the course of air travel and that very, very slightly increase the risk of an airplane accident, just as small changes in Aunt Gertrude’s lung cells increase the likelihood that a cancerous tumor will begin growing sometime later.
Amazingly, detailed information about the frequency of little risk-increasing incidents in commercial air travel exists today, but it has been collected by a federal government program that is quietly being shut down.
After the 1997 White House Commission on Aviation Safety and Security set a goal of reducing the risk of air travel accidents by 80% over the next ten years, federal agencies realized that they had no way to monitor whether progress toward this goal was being achieved.
Airplanes’ "black boxes" and other computers track what planes do physically, but they yield a colossally huge amount of data that can’t be analyzed quickly and inexpensively. And on-board computers can’t see lots of incidents that increase risks, especially incidents involving human behavior in the cockpit and air traffic control tower.
So NASA inaugurated an ambitious program called The National Aviation Operations Monitoring Service. This new monitoring system used the science of survey research to collect reliable data quickly, efficiently, and inexpensively. Every day of every month for years, large random samples of commercial airline pilots have been interviewed in depth and have reported the frequency with which they witnessed many different sorts of risk-increasing incidents.
Nearly 100 types of incidents have been measured: A pilot attempting to talk to an air traffic controller but being unable to reach anyone. A near miss with another plane in the air or on the ground. A pilot failing to follow instructions about where to fly when. A plane failing to avoid dangerous weather while cruising. A piece of on-board equipment failing to work properly. Perceptual confusion by pilots. Passenger disturbances. And many more.
Just as an interview with Aunt Gertrude can tell doctors more useful information than will be revealed by medical tests alone, talking with pilots about what they have witnessed deepens our understanding of what’s been happening in the skies.
More than 24,000 interviews have been conducted. 80% of the contacted commercial pilots agreed to be interviewed, an impressively high rate, because these pilots trust NASA to do high quality work and to protect their confidentiality.
The interviewing method was fine-tuned through years of careful design work and was vetted in public and private meetings with management personnel from airlines, labor unions, and aircraft manufacturers, with the staffs of federal agencies, with airplane and helicopter pilots, and with air traffic controllers.
A few of the numbers generated by this new monitoring system can be compared with measurements of the same incidents made in other, more costly ways (e.g., some equipment failures recorded by on-board computers and near misses tracked by federally mandated reporting systems). The correspondence is very close, reinforcing confidence in the rest of the survey data about incidents that are not tracked in other ways.
Examining the air travel system through the eyes of pilots in this way helps to deepen understanding of what’s been happening in the skies. If NASA had pursued its original plan to develop similar reporting systems for air traffic controllers, flight attendants, and airplane mechanics, we’d have even more insight.
The surveys’ results can be used to determine whether progress is indeed being made toward the White House Commission’s goal of an 80% reduction in risk. But more importantly, the surveys are like canaries in a mine, documenting increases in particular types of risk and allowing air travel professionals to take preventive steps before catastrophes happen. Expanding the program’s mission to collect data on aircraft security might even help reduce the threat of terrorism.
But instead of allowing the program to run as planned through 2009, NASA prematurely shut down interviewing of pilots last January and called off all plans to continue and expand the project, as they shift their priority to putting a man on Mars. The shutdown was coincident with the arrival of a new NASA administrator whose preferences for research seem not to include projects like NAOMS.
No evidence I know of indicates that the NAOMS surveys were somehow a failure or produced misleading or inaccurate data. From all indications, they could not have been more successful.
So from my perspective, the premature shutdown is a big mistake. Many experts who want to increase the safety of everyone boarding an airplane built a great tool that costs relatively little and could help to prevent disasters.
But their work was terminated, and you can’t find out what they learned, even though you paid for it.
Now is the time to bring NAOMS back to life. The only way I can see to do that would be for many Americans to express their support by contacting NASA and Congressional representatives who work on aviation issues. If enough voices speak loudly and in unison, maybe some progress can be made.