Documenting a real time system

A developer handbook is indispensible if you need to remember what you did a year ago (due to a bug or a new feature to be added) or if you want to bring a junior developer up to speed without baby sitting him. A good handbook should include the following:

1. Introduction: Why does the system exist, what function does it serve?
2. Components of the system, how they communicate with each other (serial, ethernet etc.). Photos of the real system and a mind map denoting peripherals (sensors, actuators) would be handy:



3. Setup: All software and hardware setup procedures, links to software installation folders.
4. Sanity tests to verify that setup works properly. For example, have a test to verify that the system really works in real time by comparing system time with an external time source. A very crude test would be using the timer in your smartphone.
5. Development use cases: How to add a new state to the state machine.
6. Design: 
1. Architecture.
2. Main workflow, especially external input/output.
3. Task priorities and rationale.
4. Design decisions and their reasons. Why is the current design the best one under existing constraints (time, budget, experience)? Why is there no simpler way to satisfy requirements? What were the disadvantages of the alternatives. Examples: Why did you write your own file transfer protocol instead of using an existing one? Why have you not used an OS like VxWorks but choose Micrium? Couldn't you have done it without an OS?
7. Troubleshooting guide for frequent problems.

HWIL development workflow

A typical HWIL development workflow starts with Simulink models and ends with code deployed on hardware with software that is expected to run in real time:

If there are differences between Simulink and Visual Studio runs, most of the time it is due to you forgetting to equalize all inputs. If you are sure about inputs being the same, the remaining sources of difference are details of floating point representations and sometime a block in Simulink generating wrong code. For example Aerospace blockset ECEF2LLA block code results in latitude lagging one time step behind, which does not occur during a Simulink run.

If there are differences between software running on hardware and Visual Studio results, it might be an indication of insufficient stack size allocation or tasks not meeting their deadlines. If you are lucky and have an advanced real time OS like VxWorks, it might throw a segmentation viloation if there is too little stack available. If you don't want to count on luck, you have to test thoroughly, preferably using automated tests.

Stack corruption due to different pointer types

The C programming language allows you to send a float pointer to a function that expects a double pointer, which causes stack corruption. Example code:
When you run it, you will see that the val_f2 is zero (should be 5):

Visual Studio 2015 will only display stack corruption message when you build in debug mode. In release mode, you don't get a message.

If you copy the same code to a cpp file, Visual Studio will use the C++ compiler and it will not build the code, saying that types are incompatible.

Unconventional interfacing

When you have to interface with a program that has a GUI but whose API you don't know, you can use a video camera with image processing if you only need to read data. If you also have to change values in GUI, you can use tools like Macro Scheduler.

Simulink: Passing strings to s-functions

As of R2019b, you can't pass a string (e.g. a file path) to an s-function via string constant block. You can convert a string to an ASCII value array, pass it to s-function, and inside the s-function convert the array back to string. Note that this means you are limited to the ASCII table of characters. You can use the following function to construct string from s-function input:

Verifying that your system runs in real time

Usually custom designed HWIL electronic cards don't have real time clocks independent of the CPU. Your RTOS will calculate time by clock_frequency*time_const. Both these values are part of the card's configuration which are set by the hardware designer before handing the card over to you. If these values are not consistent with each other, your software won't run at real time. You might waste weeks looking for bugs in your software, while the root cause is wrongly configured hardware.

For you own sanity, one of the first tests you should run on hardware is to verify that your software runs in real time. The simplest way is to run for 5 minutes by checking with an independent time source (like the stopwatch app on your phone or Windows system time) and track the simulation time. You cannot rely on time values provided by the RTOS. Time between start and stop should be 5*60 = 300s. If not, contact the person responsible for the card.

After you have established that simulation time and real time are the same, you can start to test the system under different loading conditions to verify that all your software tasks meet their deadlines. If not, you update your software, because at this stage you are sure that the hardware is ok in the real time sense.

Embedding data into code

When working in Windows, you can easily read data from a text file. But when you deploy your code to custom hardware with an operating system that does not have text file reading and parsing functions, it will be easier to embed data in file to arrays in code.

When data is large like Geoid Height Data (721*1440 elements), you cannot embed it into a header file. If you do and try to build in Visual Studio, the build might hang. You have to embed data into a cpp file and expose it in header via extern keyword.

To use two dimensional (2D) data in a function you can use the following example:

Debug C++ file

To debug a C++ file within a Simulink run, you compile the file with the "-g" flag. Consider the following Simulink model:

1. Compile file: mex -g mySFunction.cpp. This will create mySFunction.mexw64 and mySFunction.mexw64.pdb.
2. Open Visual Studio, open mySFunction.cpp
3. Click Debug/Attach to process, select Matlab.exe.
4. Set breakpoint in Visual Studio
5. Run Simulink
6. Code will stop in Visual Studio.