Embedded and IoT teams face a very different CI/CD reality than traditional SaaS teams. While most continuous integration and continuous delivery pipelines assume everything can run in the cloud, embedded systems depend on physical hardware, constrained environments, and real world signals.

If you search forums like Reddit (r/embedded, r/devops), Stack Overflow, or vendor communities, the same questions keep appearing:

• How do I run CI tests when I need actual hardware?
• How do I scale hardware testing across teams?
• How do I avoid flaky tests caused by devices?
• Can I still use modern CI/CD tools or do I need something custom?

This guide walks through practical CI/CD strategies used by engineering teams building firmware, IoT platforms, and hardware dependent systems. It focuses on approaches that scale, reduce cost, and maintain reliability.

Why CI/CD is harder for embedded and IoT projects

Unlike pure software systems, embedded pipelines must deal with:

• Physical device availability
• Hardware state and reset issues
• Long flashing and boot cycles
• Non deterministic behavior (timing, sensors, connectivity)
• Limited parallelization

This creates a mismatch with traditional CI/CD platforms that expect fast, stateless, fully virtualized execution.

For engineering leaders, this often results in slower pipelines, higher costs, and fragile automation. The goal is not to force cloud native assumptions onto hardware, but to design a hybrid pipeline.

The core strategy: split your pipeline by test layers

The most effective approach seen across teams is to separate tests into layers, minimizing hardware usage to only what is necessary.

1. Fast feedback layer (no hardware)

Run as much as possible without devices:

• Unit tests for firmware logic
• Static analysis (clang tidy, cppcheck)
• Build validation
• Simulation or emulation (QEMU, Renode)

Example Semaphore pipeline block:

version: v1.0
name: Firmware CI

blocks:
  - name: Build and unit tests
    task:
      jobs:
        - name: Build
          commands:
            - checkout
            - make build
            - make test

This layer should cover 70 to 90 percent of your test surface. It keeps pipelines fast and cost effective.

2. Hardware in the loop testing (HIL)

Only after passing fast checks should jobs use real devices.

Typical setup discussed in forums:

• USB connected device farms
• Network controlled power switches
• Raspberry Pi or similar acting as device controllers

Example structure:

  - name: Hardware tests
    task:
      prologue:
        commands:
          - checkout
      jobs:
        - name: Run on device farm
          commands:
            - ./scripts/flash_device.sh
            - ./scripts/run_integration_tests.sh

With Semaphore, you can integrate self hosted agents that have direct access to hardware, allowing your pipeline to orchestrate physical tests while keeping control centralized.

This hybrid model is critical for teams that have outgrown default CI tools that cannot reliably interface with hardware.

Designing a reliable device farm

A common pain point across discussions is flaky hardware tests. The root cause is usually poor device management.

Best practices:

Isolate devices per job

Avoid shared hardware when possible. If sharing is required, implement locking mechanisms.

Automate reset and recovery

Use:

• Smart power switches
• USB relay boards
• Watchdog scripts

Example reset script:

#!/bin/bash
# Power cycle device
curl http://power-switch.local/off
sleep 2
curl http://power-switch.local/on
sleep 5

Make tests idempotent

Each test run should not depend on previous device state.

Collect logs externally

Stream logs via serial or network so failures can be diagnosed post run.

Orchestrating hardware with CI/CD

One of the biggest questions teams ask is: how do I connect CI/CD pipelines to real devices?

The most robust approach is:

• Use cloud CI/CD for orchestration
• Use self hosted runners or agents for hardware interaction

With Semaphore, this means:

• Running standard pipeline steps in the cloud
• Routing hardware jobs to agents inside your lab

This keeps pipelines fast while maintaining control over physical infrastructure.

Handling test flakiness and timing issues

Forum discussions consistently highlight flaky tests as a major blocker.

Strategies that work:

Add retry logic at the pipeline level

retry:
  limit: 2

Add health checks before tests

Ensure device is reachable and responsive before running tests.

Use time budgets instead of strict timing

Avoid asserting exact timing unless necessary.

Tag and isolate flaky tests

Run them separately to avoid blocking the main pipeline.

Parallelizing hardware testing

Scaling is a key concern for engineering managers.

Options include:

• Expanding device farms
• Sharding tests across devices
• Prioritizing critical test subsets

Example:

jobs:
  - name: Device 1
    commands:
      - ./run_tests.sh --group=1
  - name: Device 2
    commands:
      - ./run_tests.sh --group=2

Semaphore allows parallel job execution, which is especially valuable when coordinating multiple hardware nodes.

Cost optimization strategies

Hardware pipelines can become expensive quickly.

Key tactics:

• Maximize non hardware test coverage
• Use on demand hardware instead of always on
• Optimize pipeline duration
• Avoid rerunning full hardware suites unnecessarily

Engineering teams often switch CI/CD platforms when costs become unpredictable or tied to inefficient pipelines. A system that allows fine grained control over execution and resource usage is critical.

Many teams ask whether they should build their own system.

Build custom only if:

• Your workflow is highly specialized
• You require deep hardware orchestration beyond standard pipelines

Otherwise, modern CI/CD platforms like Semaphore can handle orchestration while letting you customize the hardware layer.

This balance avoids reinventing core CI/CD capabilities while still supporting embedded workflows.

Putting it all together

A production ready embedded CI/CD pipeline typically looks like:

1. Code commit triggers pipeline
2. Build and unit tests run in parallel
3. Simulation tests validate behavior
4. Hardware tests run on device farm via self hosted agents
5. Results aggregated and reported
6. Deployment triggered if all checks pass

This structure aligns with how high performing engineering teams reduce risk while maintaining delivery speed.

FAQs