First Step in Safety-Critical Software Development

The first action when developing safety-critical software is to add automatic commit checks for compiler warnings and reject the commit if any warnings are present. Enable the highest warning level and treat all warnings as errors.

Common Visual C++ warnings relevant to safety critical systems:

1. C26451: Arithmetic overflow: Using operator 'op' on a value that may overflow the result (comes from Code Analysis with /analyze). Example: uint64_t c = a + b where a and b are of type uint32_t
2. C4244: Conversion from ‘type1’ to ‘type2’, possible loss of data. Example: int → char or double → float
3. C4018: Signed/unsigned mismatch. Can cause logic bugs and unsafe comparisons.
4. C4701: Potentially uninitialized variable
5. C4715: Not all control paths return a value
6. C4013: 'function' undefined; assuming extern returning int

Of course, there is much more that needs to be done; in this blog post, I just wanted to focus on the first step from the perspective of a tech lead.

UDP vs TCP

When working with real-time systems, it's important to understand how data is sent and received over UDP and TCP. The main reason to use UDP is that it can be 10x faster than TCP, but you have to be aware of its limitations.

UDP (User Datagram Protocol) sends data in discrete packets (called datagrams). Each call to sendto() on the sender side corresponds to exactly one recvfrom() on the receiver side.

• No connection setup or teardown.
• No built-in guarantees about delivery, order, or duplication (intermediate routers may retransmit packets if they think the first one was lost).
• If you call sock.recvfrom(4) and the incoming packet is 9 bytes, you get the first 4 bytes—and the rest are discarded, i.e. you cannot get them with another receive call.

Rough Performance Advantage:

• In short, bursty communications or real-time streaming, UDP can be 2x to 10x faster than TCP due to its lack of handshake, retransmission, and flow control mechanisms.
• In my test case, UDP returned a DNS response in 42 ms, while TCP took 382 ms — nearly 9x faster.

TCP (Transmission Control Protocol) provides a continuous stream of bytes. It breaks your data into segments under the hood, but applications don’t see those packet boundaries.

• Reliable: Guarantees delivery, order, and no duplication (if a duplicate packet is received, TCP automatically discards it).
• Stream-oriented: You send bytes, not messages.
• If you send b"Message 1", and call sock.recv(4), you might receive b"Mess", and then get the rest (b"age 1") in another call.

If the message might get corrupted during the creation phase rather than during transmission over the network, and you want to add a CRC to detect corruption at the application layer, then UDP might be better because it delivers the entire message, including the CRC, in a single recvfrom call.

Longitude convention and octal values

Recently, I encountered a bug in code that handled longitude values, where 33 degrees longitude is typically written as 033. Upon investigation, I discovered that Java was calculating 033 + 1 as 28 instead of 34. This happened because, when we write longitude values like "033" in geographic notation, we mean decimal 33 degrees. However, if we write it directly in Java, C++, or similar languages, it is interpreted as octal 33 (which equals decimal 27) due to the leading zero!

This is a classic example of why it's important to be careful when working with domain-specific number formats - what makes sense in geographic notation can have unexpected behavior when directly translated to programming language syntax!

Verifying Simulink generated C code

I generated embedded C code for the plant in the following MATLAB Simulink model:

To verify that the generated code behaves the same as the Simulink model, I could save the controller outputs to a file, create a C project with a main() function to read those outputs, and feed them to the plant C code at each time step—essentially converting a closed-loop simulation into an open-loop one. This approach would work if I use a single-step numerical integrator/solver like ode1 (Euler method). However, I usually prefer ode4 (Runge-Kutta 4) due to its better accuracy with less computation. Despite RK4 having 4 intermediate computations, it has a global error of O(h^4) and Euler O(h). If RK4 uses a step size of h, Euler would need h^4 to achieve similar accuracy. So, in total Euler would require more function evaluations than RK4—specifically, (1/h^3)/4 times more. For example, if hRK4 = 0.1, Euler would need 250x more computations.

Unfortunately, RK4 is a multi-step method with minor step sizes. Since the Simulink model saves controller outputs only at major steps, reading them from a file to drive the plant might result in divergent behavior—especially if the system has high-frequency dynamics—because controller outputs could differ at minor steps, while reading from the file only accounts for major step outputs.

You can only fully verify the system by integrating the C code into your embedded setup, which generates controller outputs based on real time plant-sensor data. Since testing on an embedded system takes time, to catch issues such as uninitialized variables in the plant code, you can create a C project, build and run your plant model with constant controller inputs. If it compiles and runs without NaN values, you can proceed to the embedded testing phase.

If you want to be more rigorous, you could save the plant outputs together with the controller outputs. In your C project, you read the controller output, feed it to the plant, run it for only one time step, read the Simulink plant output for that time instance from a file, and compare it with the C code plant output. Even this comparison is not perfect and only works for the first time step because Simulink integrator block outputs depend on the system state in the previous time step. To check the whole time span, you also have to save the system state and initialize the system with the correct state at each time step. But this in effect would be equal to running the system for one time step, because you initialize the system, apply a constant control input and check the output.

Linux: Dynamic vs Static Linking

Today, we had a chat with a colleague about whether a C/C++ binary (ELF) built on one Linux distribution would work on another. After some research, I found that default GCC builds are dynamically linked, and the ELF file contains:

1. Your program's code
2. A list of dynamic dependencies (shared libraries) it needs
3. Symbols that need to be resolved at runtime

However, it does not contain the actual shared libraries - those need to be present on the system where you run the program. You can see these dependencies using ldd. For example, a simple C "hello world" program with only a printf() call can be built with gcc hello.c -o hello. ldd hello output:

linux-vdso.so.1 (0x00007fff6a0e1000)

libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdaf6400000)

/lib64/ld-linux-x86-64.so.2 (0x00007fdaf66b3000)

The files size is 15960 bytes. To make is truly independent of any shared library we would build it with gcc -static hello.c -o hello_static. ldd hello_static shows:

not a dynamic executable

The size of hello_static is a whopping 900344 bytes, 56X more than the dynamically linked build.

The good news is that these libraries are present by default on virtually every Linux distribution, so including them in every executable would waste a lot of space. However, you must ensure that C++ version (C++17, C++20, etc.) specific features used in your code are supported by the gcc/g++ version on the target Linux distribution. Of course, the CPU architecture has to be the same too — that goes without saying.

On Windows, C++ libraries come with Visual C++ Redistributable:

Note that  if a Windows PC can successfully load and run a DLL compiled with a specific C++ version, it can also run an EXE compiled with that C++ version because the C++ runtime requirements are the same whether the code is in a DLL or EXE. The only difference is how the code is packaged and loaded, not its runtime requirements.

Music: André Rieu & Mirusia - Ave Maria

C++ std::endl vs "\n"

I have a thread safe print function that uses lock_guard and std::cout and is called from different threads. When I replaced std::endl expressions with "\n", my program stopped printing to console. The main difference is that std::endl does two things: it inserts a newline character AND flushes the output buffer, while "\n" only inserts a newline character. In a multithreaded environment, without flushing, the output might stay in the buffer and not be visible immediately.

In other standard use cases, prefer "\n" because using std::endl in a loop for example can cause performance bottlenecks since it constantly forces the CPU to stop and flush data to the screen. Note that using the single character literal '\n' is slightly faster than the string literal "\n" because the compiler doesn't have to process a null-terminator (\0).

Memory Insights from a Segfault

Recently, a program crashed with segmentation fault (sigsegv). Since segmentation faults can happen for a variety of reasons, it took some time to find out that the root cause was insufficient stack (stack overflow) due to a function allocating extra 16kB for a local variable, float var[4096]. The quick fix was to increase the stack size of the thread calling that function from 32kB to 1MB. 

Another solution is to move var to static storage instead of using the stack by changing the definition to static float var[4096]. Upon closer inspection of the code, I saw that var was just used to copy a static array before sending it to another function. Since that function was not modifying the array, there was no reason for the copy. Removing var removed large stack allocation. 

Problems like this are stressful in the short term but provide an opportunity to review the concept of memory regions. Here's a concise breakdown:

Static Storage

- Memory allocated at program start, lives for entire program duration

- Size must be known at compile time 

- Good for: fixed-size buffers that exist for whole program 

- Example: static uint8_t buffer[1024];

- Zero runtime allocation overhead 

- Can't be resized 

Stack

- Memory allocated/deallocated automatically when entering/leaving scope 

- Very fast allocation/deallocation - Limited size (often few MB) 

- Good for: small-to-medium temporary buffers 

- Example: void foo() { uint8_t temp[1024]; }

- Risk of stack overflow with large allocations

Heap 

- Dynamic runtime allocation 

- More flexible but slower than stack 

- Larger size available 

- Good for: large buffers or unknown sizes 

- Example: auto* buffer = new uint8_t[1024];

- Risk of fragmentation 

- Must manually manage memory 

For embedded systems with fixed-size allocation needs: 

1. Use static storage for long-lived, known-size buffers 

2. Use stack for small temporary buffers 

3. Consider static memory pools instead of raw heap allocation if you need dynamic allocation

In Visual Studio 2022, you can use Build > Run Code Analysis to detect functions with large stack allocations, they will be marked with C6262 excessive stack usage warnings.