SemaPLC: A Project-Grounded, Verification-Gated Agent Harness for PLC Code Generation

The Problem: LLMs Generate PLC Code That Looks Good but Fails in the Real World

Large language models can write individual program units for programmable logic controllers (PLCs)—the computers that run factories and power plants. But a crucial gap exists between what an LLM claims is correct and what actually works when that code runs on real hardware. Existing benchmarks test generated code in isolation; they don't check whether it compiles into an actual industrial project or behaves correctly when deployed to a live PLC runtime. This matters enormously. Control logic that passes static checks but crashes at runtime can shut down production lines.

How SemaPLC Works: Three-Layer Verification Instead of LLM Self-Judgment

SemaPLC replaces the typical pattern—where a language model generates code and decides when it's done—with an external verification harness that forces three checkpoints before declaring success.

Layer 1: Specification and Compilation. Generated code must pass syntax rules and compile cleanly within the context of a real PLC project, not in isolation. This catches integration failures that unit-level tests miss.

Layer 2: Static Behavior. Automated static analysis checks whether the logic satisfies the specification without running it—essentially a dry run that verifies expected state transitions.

Layer 3: Dynamic Behavior. The generated code is deployed to an actual PLC runtime alongside reference code, and their execution traces are compared. Only when the generated logic produces the same sequence of outputs and state changes as the known-good reference does the task count as passed. This is the hardest test and the most revealing.

The key insight: all three layers must pass. The harness does not ask the LLM whether it's done; it asks the hardware.

Results: Execution Separates Methods Sharply

On 117 independent-POU (program organization unit) tasks matching prior benchmarks, SemaPLC achieves a 72.6% verified pass rate across seven models—highest reported. But the more telling experiment is the project-context track: 65 tasks where generated code must integrate into a real PLC project and run on real hardware.

Static methods—specification and compilation—show small gaps between approaches. Baselines range from 22.4 to 31.4 on dynamic behavior. SemaPLC reaches 52.2. That tenfold separation on the hardest metric exposes a hard truth: code that compiles and passes static checks can still fail catastrophically at runtime. Traditional scoring hides this failure; only live execution reveals it.

Why This Matters

PLCs are not academic toys. Bugs propagate to assembly lines, chemical processes, and power distribution. The authors argue convincingly that verification-gated automation—where external tools enforce correctness rather than model confidence—is mandatory for safety-critical code generation. SemaPLC shows this approach is practical: it assembles existing compilers, analyzers, and hardware simulators into a completion rule that respects engineering reality. The method is open-sourced, making it reproducible and adoptable in industry workflows where runtime behavior, not benchmark scores, determines whether a deployment succeeds.