| D. Kalinsky Associates |
| Home | Training Courses | Online Learning | Resources | About Us | Contact | Site Map |
Whitepaper / Advanced Course Reading Assignment :
"The Enigmatic Avionic"
"The Enigmatic Avionic"
You could argue that the system I’ll be describing below is not an “avionic” system at all. As I see it, an avionic system
is an aircraft on-board computer system that’s supposed to enhance some aspect of an airplane's flight. The system
I'll be talking about enhances the information provided to travelers flying long distances on passenger aircraft. So, I
think it is indeed an avionic system – even though it’s not flight-critical. But whether or not you agree that it’s an avionic
system, my story is one that’s of interest to both avionics developers and other embedded systems developers alike.
As a professional trainer specializing in embedded software, I often travel overseas. Not long ago I was flying across
the Atlantic to do some conference lectures. I became fascinated by the “moving map” display that is familiar to air
travelers. Here’s roughly what they look like:
is an aircraft on-board computer system that’s supposed to enhance some aspect of an airplane's flight. The system
I'll be talking about enhances the information provided to travelers flying long distances on passenger aircraft. So, I
think it is indeed an avionic system – even though it’s not flight-critical. But whether or not you agree that it’s an avionic
system, my story is one that’s of interest to both avionics developers and other embedded systems developers alike.
As a professional trainer specializing in embedded software, I often travel overseas. Not long ago I was flying across
the Atlantic to do some conference lectures. I became fascinated by the “moving map” display that is familiar to air
travelers. Here’s roughly what they look like:
© Copyright 2010, D. Kalinsky Associates, All Rights Reserved. This page Updated January 1, 2010 |
Although it’s part of the passenger entertainment services of the airplane, it’s quite a complex and sophisticated
embedded computer system involving GPS satellite communication, interaction with other on-board avionic systems,
and an enormous database of geographic data.
Well, on my flight over to Europe that night I was awakened by the plane jerking its way through some choppy air
during the North Atlantic crossing. I switched on the “moving map” display, to see how many hours remained to my
destination. To my chagrin, I discovered that the moving map had crashed! On the display was the moving map’s
equivalent of the infamous “blue screen of death”.
embedded computer system involving GPS satellite communication, interaction with other on-board avionic systems,
and an enormous database of geographic data.
Well, on my flight over to Europe that night I was awakened by the plane jerking its way through some choppy air
during the North Atlantic crossing. I switched on the “moving map” display, to see how many hours remained to my
destination. To my chagrin, I discovered that the moving map had crashed! On the display was the moving map’s
equivalent of the infamous “blue screen of death”.
Instead of a map and a small icon of an airplane, there was a text message displayed saying something like:
I was not too upset by this, since such messages have become all too familiar to me during my long career in the
embedded world. After all, this is a passenger “entertainment” system. But I cringed at the thought of other
passengers seeing this message, particularly non-engineers and children. I could imagine a young child asking a
parent, “Mommy, why is the pilot doing illegal things? The computer says he doesn’t have permission to do these
things? I’m scared!” Or another kid might ask, “What is the airplane dumping? Will we have an emergency
landing? Will they arrest the pilot when we land?” Thankfully, most of the adults and children on board remained
asleep and didn’t respond this way.
After some minutes, the display flickered and the blue screen's content was replaced by new text.
System Error 6752:
Access illegal at address 0x34782592
Permission violation 1729
Minimal register dump
RA 75924 RB 84325 RW34849
I was not too upset by this, since such messages have become all too familiar to me during my long career in the
embedded world. After all, this is a passenger “entertainment” system. But I cringed at the thought of other
passengers seeing this message, particularly non-engineers and children. I could imagine a young child asking a
parent, “Mommy, why is the pilot doing illegal things? The computer says he doesn’t have permission to do these
things? I’m scared!” Or another kid might ask, “What is the airplane dumping? Will we have an emergency
landing? Will they arrest the pilot when we land?” Thankfully, most of the adults and children on board remained
asleep and didn’t respond this way.
After some minutes, the display flickered and the blue screen's content was replaced by new text.
This new screen, still bright blue, announced that the system had started to re-boot. Unfortunately for the
manufacturer of the moving map system, the screen “proudly” announced the name and address of the company
responsible for developing the system that had failed and was now attempting to recover. OK, so now we know who’s
been selling buggy avionic systems that fail in flight on our airliners. Then the screen went on to list the precise
version number and release date of the software that was running in the buggy system. [Mercifully, it didn't list the
names of the programmers involved.] And then it stated something like “Boot-up will complete in 07 min. 34 sec.”
The display updated the boot-up time every few seconds, in a modern user-friendly way.
But this did not detract from the fact that the boot-up time was very long, and that the boot-up time followed another
significant period of time during which the system had gone down and remained down in the middle of a trans-
oceanic flight. Hopefully, the pilots of the plane have a separate and much better navigation system that they use for
actually piloting the plane; and their system does not become unavailable for many minutes at a time in the middle of
a trans-oceanic flight.
I asked myself, “What if this were a moving map navigation system in my wife’s car, when she is trying to
navigate her way through a foreign city like Paris?” Then I quickly forced myself to think about other things.
Eventually, the system re-boot was successful – or so I thought. The moving map screen reappeared, with the small
icon of an airplane still directed Eastward.
manufacturer of the moving map system, the screen “proudly” announced the name and address of the company
responsible for developing the system that had failed and was now attempting to recover. OK, so now we know who’s
been selling buggy avionic systems that fail in flight on our airliners. Then the screen went on to list the precise
version number and release date of the software that was running in the buggy system. [Mercifully, it didn't list the
names of the programmers involved.] And then it stated something like “Boot-up will complete in 07 min. 34 sec.”
The display updated the boot-up time every few seconds, in a modern user-friendly way.
But this did not detract from the fact that the boot-up time was very long, and that the boot-up time followed another
significant period of time during which the system had gone down and remained down in the middle of a trans-
oceanic flight. Hopefully, the pilots of the plane have a separate and much better navigation system that they use for
actually piloting the plane; and their system does not become unavailable for many minutes at a time in the middle of
a trans-oceanic flight.
I asked myself, “What if this were a moving map navigation system in my wife’s car, when she is trying to
navigate her way through a foreign city like Paris?” Then I quickly forced myself to think about other things.
Eventually, the system re-boot was successful – or so I thought. The moving map screen reappeared, with the small
icon of an airplane still directed Eastward.
But the red line showing the path the airplane had taken to get to its current position did not begin at Dallas any
longer. I was sure that my flight had taken off from the Dallas airport the previous afternoon, and hadn’t had any stops
since then. In fact, the location of Dallas didn’t even appear on the map any more. The closest city to Dallas that was
shown was Tulsa. Now, I’m sure Tulsa is a nice city and I’m sure that nice people live there. But what happened to
Dallas?? My flight had taken off from Dallas! Where is Dallas?? The moving map had forgotten about Dallas.
And if I looked at the red line showing the path of my airplane, it was very short. It began very close to the current
location of the airplane icon --- at a spot somewhere in the middle of the North Atlantic Ocean about halfway between
Greenland and Iceland. Now, I was darn sure my flight hadn’t begun there. There are no airports out there in the
middle of the turbulent ocean. I quickly understood that the moving map system had lost its data about the thousands
of flight miles we had traveled before the system failed. After the re-boot, it treated our travel as a totally new flight that
had departed at boot-up time. Dallas was not even a distant memory. . The fact that we’d flown overhead Washington
and New York and Boston, were no longer parts of this computer’s history of the flight.
Then my mind flashed back to the image of my wife driving in Paris. “What if the moving map computer in her car
were to forget vital information like her current destination or her preference for avoiding French autoroutes
( freeways ), while denying her service for many minutes at a time?” Shudder the thought.
As an embedded systems guy, I was captivated by all of this. The technology of an airborne moving map computer is
awesome. The fact that it can fail in mid-flight and stay down for many minutes at a time, is less than awesome. But it
was more interesting than any other in-flight entertainment system I’d ever seen! So next trip overseas, I didn’t even
try to sleep during the flight. I stayed awake all night watching the moving map display for another failure. And sure
enough, as the airplane was heading East over open water between Greenland and Iceland, the moving map failed
again. And next trip to Europe after that, over the open water between Greenland and Iceland, the moving map failed
yet again!
I don’t have the moxie to ask an airline if I can bring on board a debugger or a logic analyzer and tap into their on-
board computer suite during my next flight. [I have a pretty good idea how they’d answer.] But I can start thinking the
embedded engineer’s debugging questions in my own mind:
Well, it's been a couple of years since I wrote this middle-of-the-night list of thoughts about airliner "moving map"
system crashes. I still don't sleep much during overnight oceanic crossings. And as often as not the "moving map"
systems still crash somewhere between Greenland and Iceland, on flights originating from the Western USA. I've
recently discovered that "moving map" systems can also crash on much shorter flights, such as hops from Western
Europe to the Eastern Mediterranean. I've seen "moving map" crashes on different airlines and on airplanes
produced by different manufacturers. It seems that the more I travel, the further I get from pinning down the origin of
the crashes.
And so, whether you think of the airborne moving map display as an avionic system or not, there are broader lessons
to be learned from the way it works, and from the ways it sometimes fails to work.
SUMMARY
This paper has shown how bugs in embedded software can keep people awake nights -- including both embedded
software professionals, and non-technical people just trying to travel safely as they go about their lives. [For more
technical approaches to dealing with faults and crashes of embedded systems and software, attend our advanced
course "Debugging Real-Time Software".]
END.
We hope you've enjoyed this brief whitepaper / course reading assignment. If you found it valuable, you will find
our complete on-site courses to be orders-of-magnitude more valuable.
longer. I was sure that my flight had taken off from the Dallas airport the previous afternoon, and hadn’t had any stops
since then. In fact, the location of Dallas didn’t even appear on the map any more. The closest city to Dallas that was
shown was Tulsa. Now, I’m sure Tulsa is a nice city and I’m sure that nice people live there. But what happened to
Dallas?? My flight had taken off from Dallas! Where is Dallas?? The moving map had forgotten about Dallas.
And if I looked at the red line showing the path of my airplane, it was very short. It began very close to the current
location of the airplane icon --- at a spot somewhere in the middle of the North Atlantic Ocean about halfway between
Greenland and Iceland. Now, I was darn sure my flight hadn’t begun there. There are no airports out there in the
middle of the turbulent ocean. I quickly understood that the moving map system had lost its data about the thousands
of flight miles we had traveled before the system failed. After the re-boot, it treated our travel as a totally new flight that
had departed at boot-up time. Dallas was not even a distant memory. . The fact that we’d flown overhead Washington
and New York and Boston, were no longer parts of this computer’s history of the flight.
Then my mind flashed back to the image of my wife driving in Paris. “What if the moving map computer in her car
were to forget vital information like her current destination or her preference for avoiding French autoroutes
( freeways ), while denying her service for many minutes at a time?” Shudder the thought.
As an embedded systems guy, I was captivated by all of this. The technology of an airborne moving map computer is
awesome. The fact that it can fail in mid-flight and stay down for many minutes at a time, is less than awesome. But it
was more interesting than any other in-flight entertainment system I’d ever seen! So next trip overseas, I didn’t even
try to sleep during the flight. I stayed awake all night watching the moving map display for another failure. And sure
enough, as the airplane was heading East over open water between Greenland and Iceland, the moving map failed
again. And next trip to Europe after that, over the open water between Greenland and Iceland, the moving map failed
yet again!
I don’t have the moxie to ask an airline if I can bring on board a debugger or a logic analyzer and tap into their on-
board computer suite during my next flight. [I have a pretty good idea how they’d answer.] But I can start thinking the
embedded engineer’s debugging questions in my own mind:
- * Has there been a memory leak ? In other words, does the software think it’s running out of memory when
there’s actually useable memory in the computer ?
* Has there been a memory fragmentation problem ? In other words, has the operating system in this
computer chopped up the memory into tiny un-useable pieces ??
* Or perhaps it’s just that the computer has been configured with too little RAM memory ?
* Perhaps it’s not a memory problem at all ? Perhaps it’s an arithmetic overflow that has not been handled
properly by an error handler. For example, the problem seems to occur when the airplane is way out over the
ocean at about its furthest point from a major land mass. Is it doing some calculation based on distance from
the nearest air traffic control station ? Could these calculations overflow or underflow if parameter values get
too large ?
* Perhaps it’s a memory protection problem ? There might be a totally separate program that happens to be
running on the same processor as the moving map, erroneously creating stray pointers that zap the contents
of addresses in the memory of the moving map software ? For example, there might be a Gaelic-to-English
language translation program that kicks into execution when an airplane approaches Ireland’s airspace – and
perhaps it’s got bugs that throw stray pointers and damage other programs ??
* Perhaps it’s ….
Well, it's been a couple of years since I wrote this middle-of-the-night list of thoughts about airliner "moving map"
system crashes. I still don't sleep much during overnight oceanic crossings. And as often as not the "moving map"
systems still crash somewhere between Greenland and Iceland, on flights originating from the Western USA. I've
recently discovered that "moving map" systems can also crash on much shorter flights, such as hops from Western
Europe to the Eastern Mediterranean. I've seen "moving map" crashes on different airlines and on airplanes
produced by different manufacturers. It seems that the more I travel, the further I get from pinning down the origin of
the crashes.
And so, whether you think of the airborne moving map display as an avionic system or not, there are broader lessons
to be learned from the way it works, and from the ways it sometimes fails to work.
SUMMARY
This paper has shown how bugs in embedded software can keep people awake nights -- including both embedded
software professionals, and non-technical people just trying to travel safely as they go about their lives. [For more
technical approaches to dealing with faults and crashes of embedded systems and software, attend our advanced
course "Debugging Real-Time Software".]
END.
We hope you've enjoyed this brief whitepaper / course reading assignment. If you found it valuable, you will find
our complete on-site courses to be orders-of-magnitude more valuable.
| FASTEN SEATBELT WHILE SEATED |
| 4,250 km. to Destination Frankfurt |
| Frankfurt |
| Dallas |
| New York |
| Oslo |
| FASTEN SEATBELT WHILE SEATED |
System Error 6752:
Access illegal at address 0x34782592
Permission violation 1729
Minimal register dump
RA 75924 RB 84325 RW34849
Access illegal at address 0x34782592
Permission violation 1729
Minimal register dump
RA 75924 RB 84325 RW34849
| FASTEN SEATBELT WHILE SEATED |
AIR FLOW ©
Passenger Information System
Developed by: Farmswell Avionics, Inc.
Hwy. 91 at Interstate 274
Cloudyville, PS 99387
Version: 6.8.14 fd, 29-Oct-2005
Boot-up will complete in 07 min. 34 sec.
Passenger Information System
Developed by: Farmswell Avionics, Inc.
Hwy. 91 at Interstate 274
Cloudyville, PS 99387
Version: 6.8.14 fd, 29-Oct-2005
Boot-up will complete in 07 min. 34 sec.
| FASTEN SEATBELT WHILE SEATED |
| 4,250 km. to Destination Frankfurt |
| Frankfurt |
| Tulsa |
| New York |
| Oslo |