Using a real-time operating system (RTOS) such as ThreadX® can go a long way in helping embedded software engineers manage their systems timing requirements and thread behavior. Embedded software developers who are just beginning to use an RTOS may not realize it but there are several potentially disastrous conditions that a developer can get their system into if they don’t carefully architect their software, manage their thread priorities and control how resources are shared in the software. The first and possibly the most well-known issue that developers can find their software in is known as priority inversion.
Priority inversion occurs when a high priority thread is inadvertently preempted by a lower priority thread. Priority inversion typically occurs when a high priority thread shares a resource with a lower priority thread and the low priority thread has locked the resource, leaving the high priority thread waiting for the resource to become available so that it can continue executing its code. When this occurs, the high priority task, which should have access to the CPU is instead yielded because it cannot access the required resource which is held by the The situation is exacerbated when there is more than a single thread between the priority levels of the high priority and low priority threads. As you can imagine, a low priority thread can be preempted by higher priority threads which delays the high priority thread even more. The result is that the priorities for the high and lower priority task become inverted.
Now a priority inversion could be a simple nuisance or it could lead to a catastrophic system failure depending on the application. A priority inversion can cause the system to respond sluggishly or in some cases completely lock-up. A famous example of a priority inversion that occurred in a production embedded system is the Mars Pathfinder mission. Shortly after landing on Mars, the rover began experiencing system resets that interfered with mission operations and prevented retrieving mission critical science data. The cause was a priority inversion that was delaying a critical task from executing, causing the watchdog timer to act and restart the system. (More can be read about this here).
There are several techniques that developers can use to prevent and minimize the impact a priority inversion has on their embedded system. First, analyzing their system using Rate Monotonic Analysis can help developers properly set their thread priorities. See the post on Managing Thread Priorities. RMA will give a good first pass at the priorities but since RMA assumes that threads will not interact with each other, developers need to review what resources are shared and how the threads are synchronized in order to eliminate the possibility that a priority inversion will occur.
In some cases, preventing a priority inversion will not be possible. If a high priority thread is sharing a resource with a lower priority thread, there is a potential that a priority inversion will occur. In these circumstances, we want to minimize the impact a priority inversion can have which brings us to our second option, using priority inheritance. When sharing a resource, developers should use a mutex in order to provide mutual exclusion access to the resource. Mutexes contain a feature known as priority inheritance which can help minimize the timeframe at which the priority inversion occurs.
When the high priority thread is blocked because a mutex is locked by a lower priority thread, the priority inheritance mechanism will temporarily elevate the lower priority thread to have the same priority as the high priority thread. Priority inheritance doesn’t eliminate priority inversion but it can put an upper bound on how long the higher priority thread is blocked. In the worst-case scenario, if the upper bound is within the real-time requirements for the high priority thread, then all deadlines will still be met and the system will behave as expected.
A final option that developers can utilize to help minimize the impact a priority inversion has on the system is to use a ThreadX feature known as preemption threshold. Preemption threshold allows a developer to set the minimum priority level that a thread must have in order to preempt another thread. A developer could set a preemption threshold such that the only thread that can preempt it is the high priority thread which would then help minimize the time that a priority inversion exists. Enabling preemption threshold may slightly increase the RTOS compile size and memory usage in addition to marginally increasing the overhead associated the scheduler.
Priority inversions aren’t the only possible issues that a developer can encounter when designing with a RTOS. In the next post, we discuss another common RTOS bug known as the deadly embrace.
Until next time,
Live long and profit!
Professor_IoT
Hot Tip of the Week
The Renesas Synergy Knowledge base has some great short descriptions for common RTOS terms. If you are ever confused just search the Knowledge Base for the term and you will usually find something helpful.
For example, if you search for Synergy Memory Pool (as shown in the following url) you will get the results seen in the screen shot that follows the url:
https://en-us.knowledgebase.renesas.com/Special:Search?qid=&fpid=230&fpth=&path=&search=Synergy+Memory+Pool
Once you open the article, you can even save it as a PDF using the PDF Icon on the top right side of the article. This can be useful when creating your own library of reference documentation for your design.
If you sign-in you can even leave useful feedback as to how helpful the article was. Cool!