Linux: Dynamic vs Static Linking

Today, we had a chat with a colleague about whether a C/C++ binary (ELF) built on one Linux distribution would work on another. After some research, I found that default GCC builds are dynamically linked, and the ELF file contains:

1. Your program's code
2. A list of dynamic dependencies (shared libraries) it needs
3. Symbols that need to be resolved at runtime

However, it does not contain the actual shared libraries - those need to be present on the system where you run the program. You can see these dependencies using ldd. For example, a simple C "hello world" program with only a printf() call can be built with gcc hello.c -o hello. ldd hello output:

linux-vdso.so.1 (0x00007fff6a0e1000)

libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdaf6400000)

/lib64/ld-linux-x86-64.so.2 (0x00007fdaf66b3000)

The files size is 15960 bytes. To make is truly independent of any shared library we would build it with gcc -static hello.c -o hello_static. ldd hello_static shows:

not a dynamic executable

The size of hello_static is a whopping 900344 bytes, 56X more than the dynamically linked build.

The good news is that these libraries are present by default on virtually every Linux distribution, so including them in every executable would waste a lot of space. However, you must ensure that C++ version (C++17, C++20, etc.) specific features used in your code are supported by the gcc/g++ version on the target Linux distribution. Of course, the CPU architecture has to be the same too — that goes without saying.

On Windows, C++ libraries come with Visual C++ Redistributable:

Note that  if a Windows PC can successfully load and run a DLL compiled with a specific C++ version, it can also run an EXE compiled with that C++ version because the C++ runtime requirements are the same whether the code is in a DLL or EXE. The only difference is how the code is packaged and loaded, not its runtime requirements.

Music: André Rieu & Mirusia - Ave Maria

C++ std::endl vs "\n"

I have a thread safe print function that uses lock_guard and std::cout and is called from different threads. When I replaced std::endl expressions with "\n", my program stopped printing to console. The main difference is that std::endl does two things: it inserts a newline character AND flushes the output buffer, while "\n" only inserts a newline character. In a multithreaded environment, without flushing, the output might stay in the buffer and not be visible immediately.

In other standard use cases, prefer "\n" because using std::endl in a loop for example can cause performance bottlenecks since it constantly forces the CPU to stop and flush data to the screen. Note that using the single character literal '\n' is slightly faster than the string literal "\n" because the compiler doesn't have to process a null-terminator (\0).

Memory Insights from a Segfault

Recently, a program crashed with segmentation fault (sigsegv). Since segmentation faults can happen for a variety of reasons, it took some time to find out that the root cause was insufficient stack (stack overflow) due to a function allocating extra 16kB for a local variable, float var[4096]. The quick fix was to increase the stack size of the thread calling that function from 32kB to 1MB. 

Another solution is to move var to static storage instead of using the stack by changing the definition to static float var[4096]. Upon closer inspection of the code, I saw that var was just used to copy a static array before sending it to another function. Since that function was not modifying the array, there was no reason for the copy. Removing var removed large stack allocation. 

Problems like this are stressful in the short term but provide an opportunity to review the concept of memory regions. Here's a concise breakdown:

Static Storage

- Memory allocated at program start, lives for entire program duration

- Size must be known at compile time 

- Good for: fixed-size buffers that exist for whole program 

- Example: static uint8_t buffer[1024];

- Zero runtime allocation overhead 

- Can't be resized 

Stack

- Memory allocated/deallocated automatically when entering/leaving scope 

- Very fast allocation/deallocation - Limited size (often few MB) 

- Good for: small-to-medium temporary buffers 

- Example: void foo() { uint8_t temp[1024]; }

- Risk of stack overflow with large allocations

Heap 

- Dynamic runtime allocation 

- More flexible but slower than stack 

- Larger size available 

- Good for: large buffers or unknown sizes 

- Example: auto* buffer = new uint8_t[1024];

- Risk of fragmentation 

- Must manually manage memory 

For embedded systems with fixed-size allocation needs: 

1. Use static storage for long-lived, known-size buffers 

2. Use stack for small temporary buffers 

3. Consider static memory pools instead of raw heap allocation if you need dynamic allocation

In Visual Studio 2022, you can use Build > Run Code Analysis to detect functions with large stack allocations, they will be marked with C6262 excessive stack usage warnings.

Embedded Software: Simple vs. Complex

Embedded software is integral to modern technology, ranging from simple home appliances to advanced autonomous systems. It can be broadly classified into two categories: simple (Non-OS) and complex (OS-driven) embedded software.

Simple Embedded Software: When simplicity and low cost are priorities and an OS would be overkill

Examples:

1. Power or temperature monitoring systems.
2. Simple applications in household appliances like ovens and washing machines.

Characteristics:

1. Typically designed for applications with few tasks.
2. No operating system necessary.
3. Software interacts directly with the microcontroller’s hardware (registers etc.), forcing rewrites if the hardware changes.

Advantages:

1. Low power consumption, low cost.
2. Can be developed by electronics engineers, no need for computer engineers because basic embedded programming knowledge is sufficient.
3. Deterministic: By sidestepping the complexity of OS schedulers, simple systems achieve predictable performance.
4. Fewer abstraction layers make verification and validation straightforward, which is a huge advantage for safety-critical certification.

Complex Embedded Software: When multi-tasking, file operations and networking necessitate an OS

Examples:

1. IoT devices requiring seamless connectivity.
2. Systems involving advanced sensor integration or navigation.

Characteristics:

1. Runs on an operating system that manages tasks and system resources.
2. Capable of handling multiple tasks and applications simultaneously.
3. Safety-critical certification is difficult. To make it easier, safety-critical parts should be developed as separate, simpler modules.

Advantages:

1. Less competition and higher profit margins, provided that you have a strong technical team.
2. Requires computer engineers to lead the development because of increased software complexity. Besides embedded software courses, related concepts of algorithms, data structures, and operating systems are also a core parts of computer engineering but not electronics engineering.
3. The OS abstracts low-level hardware management, enabling developers to focus on application logic. A POSIX-compliant application, for instance, can run on any POSIX-supporting OS with minimal changes.
4. Easier for new developers to adapt and contribute due to less hardware dependency.
5. A broad range of pre-existing libraries simplifies development.
6. Operating systems provide abstraction layers (e.g., Linux Device Model), allowing drivers to expose standard interfaces while interacting with specific hardware.
7. Simplifies adding new functionality (e.g. telemetry) or adapting to new hardware (e.g. new/different sensors).
8. With minor modifications, software can be tested on a PC, speeding up testing with less effort (no need for electronic cards, power supplies, etc.) and reducing bugs.

Operating systems can also be categorized as either simple (e.g., FreeRTOS) or complex (e.g., real time Linux with ROS) - but let's leave that topic for another blog post.

Music: Peyk - Don Kafa

Serialization

In C++, to serialize simple data, aka plain old data (POD), where the layout in memory is predictable, you can use a char* (byte) buffer:

This method cannot be used for non-POD types (e.g., those with pointers or virtual methods) because their memory layout is not portable. Examples are std::string, std::vector. For such types, you can use std::ostringstream:

Both approaches assume that the serialized data format and endianness match between serialization and deserialization. For more complex cases, use libraries like nlohmann/json for JSON-based serialization and Boost.Serialization for binary/text serialization with more features.

Data Structure Alignment

The C++ compiler aligns data structures to the largest alignment required by any field (8 bytes in the case below, due to double). This ensures faster memory access, as modern CPUs perform better when data is aligned to specific boundaries because it results in single memory word access. As a side effect, sizeof(MyStructure) (40 bytes due to padding) is larger than the sum of individual fields (33 bytes).

Field Offsets and Padding

1. int i1:

   • Requires 4-byte alignment.
   • Starts at offset 0.
   • Takes 4 bytes.
   • The next field, d1, requires 8-byte alighment. Since i1 ends at offset 4, the compiler adds 4 bytes of padding after i1.

2. double d1:

   • Requires 8-byte alignment.
   • Starts at offset 4 + 4 = 8.
   • Takes 8 bytes.

3. char s[9]:

   • Requires no specific alignment (1-byte alignment is sufficient).
   • Starts at offset 16 (immediately after d1).
   • Takes 9 bytes.
   • The next field, int i2, requires 4-byte alignment. Therefore, the compiler adds 3 bytes of padding after s to ensure proper alignment.

4. int i2:

   • Requires 4-byte alignment.
   • Starts at offset 28.
   • Takes 4 bytes.

5. double d2:

   • Requires 8-byte alignment.
   • The next offset must be a multiple of 8. Since i2 ends at offset 32 (already aligned), no padding is required.
   • Starts at offset 32.
   • Takes 8 bytes.

Simulation variable names

In simulation, you have to be precise when talking about a parameter. For example, it is never enough to say "height". You should always say "height with respect to mean sea level, with units in feet". The reason is that height can also be measured from WGS84 ellipsoid or ground (AGL). Every couple of months, I see engineers waste days, sometimes weeks, due to such misunderstandings.

Here is a list that I frequently encounter, with bad and good variable naming:

• height: h - hMSL_ft (height measured from MSL, units in feet)
• time: t - timeFreeFlight_s (time started at free flight start, units in seconds)
• time: currentTime - currentTime_UnixEpoch_s (time started at Unix epoch, units in seconds)
• date: currentDate - currentDate_UTC (date in UTC)
• velocity: v - v_bc_Fn_mps (velocity of body fixed frame Fb wrt ground fixed frame Fc, with components expressed in NED frame, units in m/s)
• speed: v - speed_Mach
• acceleration: a - a_bi_noG_Fb_mps2 (acceleration of Fb wrt inertial frame Fi, without gravity components, expressed in Fb, units in m/s^2)
• Euler angles: euler - euler_Fn2FbRFB321_rpy_rad (321 yaw pitch roll sequence rotated frame based Euler angles that convert a vector in Fn to a vector in Fb, array index order is roll pitch yaw, units in radians)
• Azimuth: az - azimuthTrueNorth_deg (azimuth angle measured from True North, units in degrees)

Music: Peyk - Don Kafa