Showing posts with label safety critical. Show all posts
Showing posts with label safety critical. Show all posts

Thursday, August 6, 2026

The Coldcard incident

A vulnerability in the Coldcard hardware wallet's Bitcoin key generation allowed attackers to steal more than 2,000 BTC (about $120 million). The vulnerability had been introduced by an update on March 2021 and was only exploited a few days ago. The immediate root cause was the code using a less safe/lower entropy random number generator, which allowed the attackers to find wallet addresses and their keys by brute force.

For processes that require high security (like creating bitcoin wallet keys), you have to use cryptographically secure random number generators. On a PC, you might use python's secret module. On small embedded devices, you often don't have access to high-level libraries, so the firmware must implement a secure entropy source itself.

The relevant code depends on a fork of MicroPython maintained by Coldcard (a heavily stripped-down Python implementation for microcontrollers) and on the libngu library, reportedly maintained by Coinkite's CTO. The code expected a function named rng_get(), but Coldcard's implementation exposed differently named functions, so the linker silently selected MicroPython's software PRNG. The problem can be summarized as:

[ Intended Architecture ]
  Entropy Source (STM32 Hardware TRNG) ──► Cryptographically Secure Seed (128-bit)

[ Actual Vulnerable Execution ]
  Silent Check Failure (MICROPY_HW_ENABLE_RNG == 0)
    │
    ▼
  Fallback: MicroPython Software PRNG (Yasmarang)
    │
    ├─► Initialized by: Device Unique ID (UID) + Timer Registers
    └─► Entropy drop: 128-bit (Standard) ──► ~40 bits (Mk2/Mk3)

Because the flawed RNG only had ~40 bits of true entropy, the actual number of possible seeds an attacker had to search was around 2⁴⁰ ≈ 1.1 trillion, large in absolute terms, but small enough to be tractable with modern parallel computing, versus 2¹²⁸ or 2²⁵⁶ for a properly random seed, which is astronomically infeasible.

The general shape of the attack (all 4 steps done offline):

  1. Once the attackers understood the bug (the RNG fell back to a weak software PRNG instead of hardware entropy), they could reconstruct the actual algorithm and its reduced state/seed space rather than treating it as a black box.
  2. Instead of guessing random 256-bit values, the attacker only needs to iterate through the much smaller set of values (1.1 trillion) the flawed generator could actually produce.
  3. For each value, they derive the corresponding Bitcoin address. This uses the standard, public BIP32/BIP39 derivation path (seed → master key → address), completely deterministic and fast to compute.
  4. They compare candidate addresses against the full set of funded/known Bitcoin addresses (a public, downloadable dataset). A match means you've found a real private key controlling real funds.
While iterating through 1.1 trillion values on a single high CPU core could take a year (at 50,000 elliptic-curve (EC) point multiplications per second), a high-end, optimized GPU might do it in about 10 days (at 1 million EC point multiplications per second). Once the search space was precomputed or was being iterated systematically, matching against known funded addresses is comparatively cheap, and the attacker likely had already built the candidate list (private-key-to-address mapping) in advance, which is why the sweep itself (emptying 1,196 addresses in 41 minutes) was so fast and automated.

Details: The weak-entropy path (the software Yasmarang PRNG) only happened because MICROPY_HW_ENABLE_RNG was defined as 0. In libngu (ngu/random.c):

#ifndef MICROPY_HW_ENABLE_RNG
# error "get a HW TRNG plz"
#endif
...
extern uint32_t rng_get(void);      // "I need a function called rng_get()"
#define CHIP_TRNG_32()   rng_get()  // then calls it

This only checks whether the macro exists, not its value. Because it was defined (even as 0), the #error never fired and the build continued. It declares rng_get but does not define (implement) it. It expects whoever builds the final firmware to supply a real function with that exact name. In MicroPython (ports/stm32/rng.c):

#if MICROPY_HW_ENABLE_RNG
// real STM32 hardware TRNG path for rng_get()
#else
// Yasmarang software PRNG fallback for rng_get()
#endif

When the value is 0, the #else branch is taken → the weak software PRNG is compiled in and linked as rng_get(). In Coldcard’s own board RNG files (e.g. stm32/COLDCARD*/rng.c):

#if MICROPY_HW_ENABLE_RNG
#error "this code replaces normal RNG module"
#endif

They intentionally set the macro to 0 so their custom hardware RNG code would be used instead of MicroPython’s. But that custom code never exported the exact rng_get() symbol that libngu expected. Coldcard’s custom RNG code (in files like stm32/COLDCARD/rng.c or stm32/COLDCARD_MK4/rng.c) implements its own, more paranoid hardware TRNG logic. However, it only provided functions under different names, such as:

  • rng_get_or_fault()
  • random_buffer()
  • pyb_rng_get() (the Python-visible version)

It never defined a global function literally named rng_get(). MicroPython’s own ports/stm32/rng.c does define a function named rng_get(). Because MICROPY_HW_ENABLE_RNG was set to 0, the linker picks this weaker version:

uint32_t rng_get(void) {
  return pyb_rng_yasmarang(); // software PRNG
}

When the final firmware binary is built, the linker has to resolve every extern reference.

libngu says: “I need a symbol called rng_get.”

Coldcard’s custom code does not supply a symbol with that exact name.

MicroPython’s code does supply a symbol with that exact name (the software version).

So the linker binds libngu’s call to MicroPython’s software rng_get().

That’s why the custom hardware TRNG code was present in the source tree and even reviewed, yet was never actually used for seed generation. The call path never reached it because the symbol name didn’t match.

If the board-specific RNG file (e.g. stm32/COLDCARD/rng.c or the Mk4 equivalent) had contained a global function literally named rng_get, then the linker would have seen two definitions of the same symbol, would produce a duplicate symbol / multiple definition error and the build would fail.

The solution is to make the ownership of the entropy source explicit and fail-closed, instead of relying on a fragile combination of preprocessor flags and linker symbol resolution. Stop going through libngu / ngu.random for seed generation. Keep the critical path (wallet seed creation) on Coldcard’s own known-good function that already talks directly to the hardware TRNG. Only use ngu.random for non-critical things (if at all).

Another good practice would be to write unit tests to verify the of the pass/key generation. Those tests could look like the following (of course, they should be much more sophisticated than this):

# 1. Generate many seeds using the exact production call path
seeds = set()
n = 50000;
for _ in range(n):
  raw = rng.generate_bytes(32)
  seeds.add(raw)
print(f"len(seeds): {len(seeds)}")
# 2. Basic sanity: should almost never collide
assert len(seeds) == n, "Collisions detected – entropy is far too low"

People are reporting that using widely available AI without access to current events on the internet did not reveal the bug, so this incident is not an obvious coding mistake.

Music: Orhan Gencebay - Hatasız Kul Olmaz

Wednesday, June 10, 2026

Choosing the Right CPU: Desktop vs. Industrial vs. Safety-Critical

We live in an era where a standard desktop processor has 24 cores and clock speeds past 5.5 GHz. Yet, if you walk into an automotive assembly line, you will see computers (PLCs) with processors running 100x slower, and being 10x more expensive than their desktop counterparts.

Why? Because in the world of computing, power is defined entirely by the problem you are trying to solve. We have to look past raw processing speed and examine three distinct operational philosophies: Throughput, Determinism, and Functional Safety.

1. The Desktop CPU

Desktop processors are designed to handle an unpredictable, highly dynamic workload. At any given moment, a desktop CPU might be asked to render a 3D video, compile a massive codebase, manage dozens of browser tabs, or decode high-definition audio.

To excel at this, desktop CPUs use general purpose operating systems like Windows or Linux, which rely on throughput-oriented schedulers. The OS slices up time and distributes it among running applications, trying to give everything a fair share. To squeeze out every drop of performance, the silicon itself relies on microarchitectural optimizations:

  • Out-of-Order Execution: The CPU dynamically rearranges the order of instructions to keep its execution pipelines full.
  • Speculative Execution & Branch Prediction: The processor literally guesses which path a piece of code will take before it even runs, executing the instructions ahead of time to hide latency.
  • Multi-Tiered Caches (L1/L2/L3): Large memory pools sit on the die to prevent the CPU from constantly waiting on slower system RAM.

However, this architecture is inherently non-deterministic. If a background cloud-sync app suddenly demands resources, or if a branch predictor guesses wrong, a task might take 50 milliseconds longer to execute on cycle two than it did on cycle one. In the consumer world, a dropped frame in a video game is an annoyance; in a physical system, a 50ms delay can be catastrophic.

2. The Industrial PLC CPU

Step inside a factory running a high-end programmable logic controller (PLC), like the Siemens SIMATIC S7-1500. Clock speeds range from tens to hundreds of megahertz, and memory is measured in megabytes rather than gigabytes. Yet, these processors are built for a completely opposing goal: Absolute Determinism.

An industrial CPU runs a Real-Time Operating System (RTOS). Instead of a fair share schedule, the RTOS uses strict, unyielding, priority-based cyclic execution. A PLC operates on a continuous loop:

  1. Read Inputs: Snapshot the state of every physical sensor.
  2. Execute Logic: Run the user control code sequentially.
  3. Write Outputs: Instantly update physical actuators, valves, and motors.

To guarantee that this cycle takes exactly the same amount of time down to the microsecond, industrial CPUs strip away the unpredictable optimization layers of desktop chips. There is no speculative execution, no out-of-order execution, and no virtual memory paging. Every task has a strict time budget monitored by a dedicated hardware watchdog timer. If a 1ms motion control loop fails to complete in exactly 1ms, the RTOS catches it, alerts the system, and can trigger a controlled shutdown.

Industrial CPUs are also physically engineered to survive decades in harsh environments. They are decoupled from fragile cooling fans, insulated against severe electromagnetic interference (EMI), and rated to maintain their precise timing clock cycles across extreme temperature swings (e.g., -25°C to +60°C).

3. The Safety-Critical CPU

While an industrial CPU guarantees when a command will execute, a safety-critical processor guarantees the mathematical integrity of the execution itself. Found in drive-by-wire automotive systems, avionics units complying with aerospace standards (like DO-254/DO-178C), or high-speed medical equipment, these processors are certified to standards like ISO 26262 (ASIL-D) or IEC 61508 (SIL-3).

The defining feature of a commercial safety-critical processor (such as the Texas Instruments Hercules™ or Infineon AURIX™ lines) is hardware level redundancy. Instead of using multiple cores to run different applications simultaneously, a safety-critical CPU pairs identical cores into a Dual-Core Lockstep (DCLS) configuration:

  • The Master and the Checker: Two physical hardware cores execute the exact same instruction stream, line-by-line, cycle-by-cycle.
  • Temporal Separation: To ensure that a localized physical event (like a voltage spike or a cosmic ray flipping a bit in memory) doesn't corrupt both cores simultaneously, the second core runs delayed by a micro-interval (typically 2 to 3 clock cycles).
  • Hardware Comparators: Independent hardware logic monitors the internal state and outputs of both cores. If a divergence of even a single bit is detected between the Master and the Checker, the comparator immediately strips power from the actuators or switches the system to a pre-defined, hardware-enforced "safe state."

In this realm, the code is heavily audited, features like Built-In Self-Tests (BIST) continuously sweep memory for corruption, and every single gate on the silicon must be mathematically traceable back to a design requirement.

Music: Ali Baba ve 7 Cüceler (arka jenerik)

Monday, April 14, 2025

First Step in Safety-Critical Software Development

The first action when developing safety-critical software is to add automatic commit checks for compiler warnings and reject the commit if any warnings are present. Enable the highest warning level and treat all warnings as errors.

Common Visual C++ warnings relevant to safety critical systems:

  1. C26451: Arithmetic overflow: Using operator 'op' on a value that may overflow the result (comes from Code Analysis with /analyze). Example: uint64_t c = a + b where a and b are of type uint32_t
  2. C4244: Conversion from ‘type1’ to ‘type2’, possible loss of data. Example: int → char or double → float
  3. C4018: Signed/unsigned mismatch. Can cause logic bugs and unsafe comparisons.
  4. C4701: Potentially uninitialized variable
  5. C4715: Not all control paths return a value
  6. C4013: 'function' undefined; assuming extern returning int
Of course, there is much more that needs to be done; in this blog post, I just wanted to focus on the first step from the perspective of a tech lead.

Wednesday, September 29, 2021

Safety-Critical System Development

Safety-critical system development requires safety certification. Examples for railways are EN 50128 (software) and EN 50129 (hardware). For aerospace there are DO-178C (software) and DO-254 (hardware). Since it it not practically possible to have 100% test coverage for complex systems, these documents require that development processes adhere to practices that minimize risk of catastrophic failure.

Companies with no experience in these standards grossly underestimate time and budget requirements of making the necessary changes for compliance. It takes at least two years to get a company from zero to certified. If the company has the vision to enter the aerospace market, compliance preparations have to be started before any system development contract, because to both change company culture and develop the system at the same time is a sure way to fail.

One way to avoid DO-178C is to just use an electronic circuit, without any software, provided that the requirements are simple enough. For example, an aircraft climate controller consisting of temperature and airflow sensors and fan and valve actuators could be realized with a simple PID controller using only operational amplifiers. Since it has no software and the relatively simple hardware can be tested with 100% coverage, there is no need to demonstrate that company development processes are sound. Hardware still has to comply with DO-254.