Benchmarks, every number#

Every point, interval, timing and command of the benchmark, by scenario and by machine. The same links simulated with comnum and with the Python libraries that offer them — komm, OptiCommPy (with and without numba) and Sionna — with the same parameters and the same number of samples: do the results agree (with the closed form when there is one, with each other otherwise), and how long does each library take? The comparisons page compares the outputs of single blocks value by value; this one runs whole links.

A feature a library does not offer is marked not available, with the reason: no substitute is written for it. The scripts import every library as a user would (the GPL-3.0 code of OptiCommPy is imported by the benchmark scripts only, never copied).

Rules#

Same work. Same parameters, same number of bits, vectors, frames or samples for every library; every library draws its own bits and noise with its own generator and independent seeds, from the same distributions, so the results agree statistically, not bit for bit (the input field of the split-step case and the LLRs of the case “same LLRs” are identical in every library).
Confidence intervals. 95 % intervals: Wilson’s for independent errors; for coded frames and MIMO vectors, whose errors come in bursts, the interval of the mean of the per-frame (per-vector) error counts. A result agrees when every point lies within four standard errors of the closed form (or of comnum, when there is no closed form; points without errors on both sides are skipped).
Timing. time.perf_counter; one warm-up run first, reported apart as first call (it contains the numba compilation, from an empty cache, and the lazy initialisations of PyTorch); then the median of five timed runs. The times are per unit of work (bit, vector, frame, propagation).
Threads. Every library runs in its own process. 1 thread: OMP_NUM_THREADS, OPENBLAS_NUM_THREADS, MKL_NUM_THREADS, VECLIB_MAXIMUM_THREADS, NUMBA_NUM_THREADS and RAYON_NUM_THREADS set to 1, torch.set_num_threads(1), threads=1 for comnum.run_spec and --threads 1 for the binary. All threads: the defaults of every library (every core; threads=0 for comnum). The functions of comnum are single-threaded; its Monte Carlo runner (run_spec, the binary) runs the trials in parallel.
Precision. What every library computes in is stated next to its times: comnum in float64 (float32 on request for the split-step), komm in float64, Sionna in float32 by default, OptiCommPy in float64 except its Gray constellations (complex64) and its LDPC decoder (float32 by default).
Sionna 2.x is built on PyTorch (not TensorFlow); it runs eagerly, under torch.inference_mode(), without torch.compile: on the CPU in the modes 1 thread and all threads (config.device = "cpu": Sionna’s own default is the first CUDA device as soon as it sees one), and on the GPU in the mode GPU (--threads gpu: config.device = "cuda:0", the device every block of Sionna creates its tensors on, the result brought back to the host inside the timed call). OptiCommPy has a CuPy copy of its split-step (optic.models.modelsGPU.ssfm), run in the mode GPU of the split-step scenario, in complex128 and complex64, the transfers to and from the device inside the timed call; its other blocks are CPU only. comnum and komm have no GPU path: in the GPU column they do not run, and the comparison is a library on the device against comnum on the CPU of the same machine.
OptiCommPy without numba means NUMBA_DISABLE_JIT=1: numba’s own switch, which runs every @njit function of OptiCommPy as plain Python (OptiCommPy has no option of its own). Its results are those of the compiled code; it is timed only.

Limits of this comparison#

The CPU of a laptop, and a GPU. Sionna is designed for GPUs, large batches, learning and automatic differentiation. The first set of results runs it on the CPU of the same machine as comnum, eagerly, without torch.compile: its home ground is elsewhere. A second set, when present below, runs the same scripts on a machine with a GPU, Sionna (and OptiCommPy’s split-step) on the device and comnum on that machine’s CPU; comnum has no GPU path, so that comparison is one of a CPU library against an accelerator, which is what a user choosing between the two faces.
Python libraries only. komm, OptiCommPy and Sionna are NumPy, numba and PyTorch code. Specialised C++ libraries (AFF3CT, SIMD decoders) are not timed here: the decoders of comnum are timed against those of AFF3CT, one thread, in comparisons.
Precision. comnum computes in float64, Sionna in float32 by default: the faster library is not always doing less work, but the two are not strictly the same computation.
One machine, cases chosen by comnum. The scenarios are the classical links that several libraries offer; other machines, sizes and cases may rank the libraries differently. The scripts are published to be rerun.
What this benchmark changed in comnum. Over flat Rayleigh fading the first run found the comnum functions 3.4 times slower than Sionna on one thread (293 against 87 ns per bit for BPSK): the linear detector built a dozen 1 × 1 matrices for every symbol. It now has a single-stream path (the same filter, as scalars), with the same error counts at every point, and the binding passes the vectors without a copy.
Why Sionna is fast on a CPU. It processes the whole batch in one call per operation: the cost of Python is paid once per operation, not once per symbol, and the kernels of PyTorch are vectorised C++ working in float32.

Libraries#

Name on this page	What runs	Version
comnum (functions)	comnum from Python: the functions of the comnum package, called block by block	0.2.0
comnum.run_spec	comnum from Python: comnum.run_spec, the parallel Monte Carlo runner of Rust	0.2.0
comnum binary (Rust)	comnum from Rust: the comnum command-line binary on the same specification	0.2.0
komm	komm (NumPy)	0.37.0
OptiCommPy (numba)	OptiCommPy with numba (its @njit kernels compiled)	0.11.0
OptiCommPy (no numba)	OptiCommPy without numba (NUMBA_DISABLE_JIT=1: the same @njit kernels run as plain Python)	0.11.0
Sionna	Sionna PHY (PyTorch, CPU, eager mode)	2.2.0

Machine and versions#

CPU	Apple M6
Cores	12 logical (2 super, 4 performance, 6 efficiency)
Memory	24.0 GB
System	macOS 27.0
Python	3.12.15
Rust	rustc 1.99.1 (58cf8f9c3 2026-10-13)
comnum commit	`f20118f9160d`
Generated	2026-10-30T08:05:23+00:00
comnum	0.2.0
komm	0.37.0
OptiCommPy	0.11.0
sionna	2.2.0
torch	2.15.0
numba	0.68.1
numpy	2.5.4
scipy	1.18.1

Reproducing#

On a quiet machine (the timings are wrong on a loaded one), from the root of the repository:

uv venv --python 3.12 benchmarks/libraries/.venv
source benchmarks/libraries/.venv/bin/activate
uv pip install -r benchmarks/libraries/requirements.txt
maturin develop --release                    # comnum, from this checkout
cd benchmarks/libraries
python run_all.py                            # builds target/release/comnum first
python ../../tools/benchmarks_page.py

python run_all.py --quick runs everything at small sizes in about two minutes, to check that it works (its timings mean nothing). The versions are pinned in requirements.txt; see benchmarks/libraries/README.md.

Uncoded QPSK and 16-QAM over AWGN#

Scenario 1: uncoded Gray QPSK and 16-QAM over AWGN, BER against the exact closed form.

Parameters:

Eb/N0 (dB): 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
bits per point: 10^7 (quick: 2·10^5), in blocks of 10^6 (quick: 10^5)
timing: one block of 10^6 bits (quick: 2·10^5) at Eb/N0 = 6 dB
theory: exact BER of Gray square QAM, Cho & Yoon, IEEE Trans. Commun. 50(7), 2002

Script: benchmarks/libraries/s1_awgn.py.

The script

"""Scenario 1: uncoded Gray QPSK and 16-QAM over AWGN, BER against the exact closed form.

Every library draws its own bits and noise, maps with its own Gray mapper, adds the
noise of its own channel for the Eb/N0 of the point and decides by minimum distance
(Sionna: hard decisions of max-log LLRs, the same decisions). The BER is counted on
the same number of bits per point for every library, in blocks of the same size.
Timing: one block of ``TIMING_BITS`` bits at Eb/N0 = 6 dB (bits, mapping, noise,
decisions, error count), median of 5 runs after a warm-up run.

    python s1_awgn.py --lib comnum --threads 1 [--quick] --out results/_work/s1.json
"""

from __future__ import annotations

import math
import time

import common
import numpy as np
import theory

SCENARIO = "s1_awgn"
TITLE = "Uncoded QPSK and 16-QAM over AWGN"
ORDERS = {"qpsk": 4, "16qam": 16}
ALL = {lib: True for lib in common.LIBRARIES}
CASES = {"qpsk": ALL, "16qam": ALL}
EBN0 = [float(e) for e in range(11)]
TIMING_EBN0 = 6.0
TIMING_TRIALS = 40  # trials of the specification runner per timing run


def sizes(quick: bool) -> dict:
    if quick:
        return {"bits_per_point": 200_000, "block": 100_000, "timing_bits": 200_000, "trials": 10}
    return {
        "bits_per_point": 10_000_000,
        "block": 1_000_000,
        "timing_bits": 1_000_000,
        "trials": 100,
    }


PARAMETERS = {
    "ebn0_db": EBN0,
    "bits_per_point": "10^7 (quick: 2·10^5), in blocks of 10^6 (quick: 10^5)",
    "timing": "one block of 10^6 bits (quick: 2·10^5) at Eb/N0 = 6 dB",
    "theory": "exact BER of Gray square QAM, Cho & Yoon, IEEE Trans. Commun. 50(7), 2002",
}


# ---------------------------------------------------------------------------- chains
# chain(n_bits, ebn0_db) -> bit errors, for one library and one constellation


def chain_comnum(m: int, seed: int):
    import comnum

    c = comnum.Constellation("qam", m).normalized()
    k = int(math.log2(m))
    rng = comnum.Rng(seed)

    def run(n: int, ebn0: float) -> int:
        b = comnum.bits(n, rng)
        y = comnum.awgn_ebn0(c.map_bits(b), ebn0, k, rng)
        return int(np.count_nonzero(c.demap_bits(y) != b))

    return run, "float64"


def chain_komm(m: int, seed: int):
    import komm

    const = komm.QAMConstellation(m)
    lab = komm.ReflectedRectangularLabeling(int(math.log2(m)))
    k = int(math.log2(m))
    es = const.mean_energy()
    rng = np.random.default_rng(seed)

    def run(n: int, ebn0: float) -> int:
        b = rng.integers(0, 2, n)
        x = const.indices_to_symbols(lab.bits_to_indices(b))
        n0 = es / (k * 10 ** (ebn0 / 10))
        y = komm.GaussianChannel(n0, rng=rng).transmit(x)
        bh = lab.indices_to_bits(const.closest_indices(y))
        return int(np.count_nonzero(bh.ravel() != b))

    return run, "float64"


def chain_opticommpy(m: int, seed: int):
    from optic.comm.modulation import demodulateGray, modulateGray
    from optic.models.channels import awgn
    from optic.utils import parameters

    k = int(math.log2(m))
    rng = np.random.default_rng(seed)
    state = {"block": 0}

    def run(n: int, ebn0: float) -> int:
        b = rng.integers(0, 2, n)
        x = modulateGray(b, m, "qam")
        p = parameters()
        p.snr = ebn0 + 10 * math.log10(k)  # Es/N0 (Fs = B = 1)
        p.seed = seed * 1_000_003 + state["block"]  # awgn re-seeds its generator per call
        state["block"] += 1
        y = awgn(x, p)
        bh = demodulateGray(y, m, "qam")
        return int(np.count_nonzero(bh != b))

    return run, "complex64 symbols (its Gray constellation), float64 noise"


def chain_sionna(m: int, seed: int):
    import torch
    from sionna.phy import config
    from sionna.phy.channel import AWGN
    from sionna.phy.mapping import BinarySource, Demapper, Mapper
    from sionna.phy.utils import ebnodb2no

    config.seed = seed
    k = int(math.log2(m))
    source, mapper, channel = BinarySource(), Mapper("qam", k), AWGN()
    demapper = Demapper("maxlog", "qam", k, hard_out=True)

    def run(n: int, ebn0: float) -> int:
        with torch.inference_mode():
            b = source([1, n])
            no = ebnodb2no(ebn0, k, 1.0)
            y = channel(mapper(b), no)
            bh = demapper(y, no)
            return int((bh != b).sum())

    return run, "float32 (config.precision = 'single')"


CHAINS = {
    "comnum": chain_comnum,
    "komm": chain_komm,
    "opticommpy": chain_opticommpy,
    "opticommpy-nojit": chain_opticommpy,
    "sionna": chain_sionna,
}


def run_chains(args, todo: list[str]) -> list[dict]:
    if args.lib == "sionna":
        threading = common.setup_torch(args.threads)
    else:
        threading = {}
    sz = sizes(args.quick)
    out = []
    for case in todo:
        m = ORDERS[case]
        run, precision = CHAINS[args.lib](m, args.seed)
        res = {"case": case, "precision": precision, **threading}
        n = sz["timing_bits"]
        if args.lib == "opticommpy-nojit":
            n = min(n, 100_000)  # plain-Python kernels: a smaller block, the time is per bit
        t0 = time.perf_counter()
        run(n, TIMING_EBN0)  # the first call: JIT compilation (numba), lazy initialisation
        res["first_call_s"] = time.perf_counter() - t0
        if not args.timing_only:
            pts = []
            t0 = time.perf_counter()
            for e in EBN0:
                errors = 0
                for _ in range(sz["bits_per_point"] // sz["block"]):
                    errors += run(sz["block"], e)
                pts.append(
                    common.rate_point(
                        errors, sz["bits_per_point"], float(theory.ber_qam_awgn(m, e)), ebn0_db=e
                    )
                )
            res["points"] = pts
            res["sweep_s"] = time.perf_counter() - t0
        last = {}
        res["timing"] = common.time_it(
            lambda n=n, run=run, last=last: last.__setitem__("e", run(n, TIMING_EBN0))
        )
        res["timing"].update(units=n, unit="bit", workload=f"{n} bits at Eb/N0 = {TIMING_EBN0} dB")
        res["timing_ber"] = last["e"] / n
        out.append(res)
    return out


# ------------------------------------------------------------- specification runner


def spec(m: int, ebn0: list[float], bits: int, trials: int, seed: int) -> dict:
    k = int(math.log2(m))
    return {
        "name": f"bench_awgn_{m}qam",
        "seed": seed,
        "batch": trials,
        "grid": {"ebn0_db": ebn0},
        "link": [
            {
                "type": "source",
                "constellation": {"kind": "qam", "m": m},
                "n_symbols": bits // (k * trials),
            },
            {"type": "awgn_ebn0", "ebn0_db": "$ebn0_db"},
        ],
        "metrics": {"report": ["ber"], "scale": "none"},
        "stop": {"min_trials": trials, "max_trials": trials},
    }


def spec_points(results: list[dict], m: int) -> list[dict]:
    pts = []
    for r in results:
        e = float(r["condition"]["ebn0_db"])
        ber = r["metrics"]["ber"]
        errors = ber["count"] if "count" in ber else ber["errors"]  # run_spec / the binary's report
        pts.append(
            common.rate_point(errors, ber["total"], float(theory.ber_qam_awgn(m, e)), ebn0_db=e)
        )
    return pts


def run_spec_lib(args, todo: list[str], *, rust: bool = False) -> list[dict]:
    """``comnum-spec`` (``comnum.run_spec``), or with ``rust`` ``comnum-rust`` (the binary)."""
    sz = sizes(args.quick)
    n = sz["timing_bits"]
    workload = f"{n} bits at Eb/N0 = {TIMING_EBN0} dB, {TIMING_TRIALS} trials"
    if rust:
        workload += " (process wall time, start-up included)"
    out = []
    for case in todo:
        m = ORDERS[case]
        out.append(
            common.spec_case(
                args,
                {"case": case, "precision": "float64"},
                spec(m, EBN0, sz["bits_per_point"], sz["trials"], args.seed),
                spec(m, [TIMING_EBN0], n, TIMING_TRIALS, args.seed + 1),
                lambda rows, m=m: spec_points(rows, m),
                rust=rust,
                units=n,
                unit="bit",
                workload=workload,
            )
        )
    return out


def run_rust_lib(args, todo: list[str]) -> list[dict]:
    return run_spec_lib(args, todo, rust=True)


RUNNERS = {
    **{lib: run_chains for lib in CHAINS},
    "comnum-spec": run_spec_lib,
    "comnum-rust": run_rust_lib,
}

if __name__ == "__main__":
    common.main(SCENARIO, CASES, RUNNERS)

Support

Case	comnum (functions)	comnum.run_spec	comnum binary (Rust)	komm	OptiCommPy (numba)	OptiCommPy (no numba)	Sionna
QPSK	yes	yes	yes	yes	yes	yes	yes
16-QAM	yes	yes	yes	yes	yes	yes	yes

QPSK: error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	theory	1.90	yes	binomial
comnum.run_spec	float64	theory	2.53	yes	binomial
comnum binary (Rust)	float64	theory	2.53	yes	binomial
komm	float64	theory	2.37	yes	binomial
OptiCommPy (numba)	complex64 symbols (its Gray constellation), float64 noise	theory	3.31	yes	binomial
Sionna	float32 (config.precision = ‘single’)	theory	2.13	yes	binomial

BER and its 95 % interval, every point (10,000,000 bits per point)

Eb/N0 (dB)	theory	comnum (functions)	comnum.run_spec	comnum binary (Rust)	komm	OptiCommPy (numba)	Sionna
0	7.86e-2	7.86e-2 [7.84e-2, 7.88e-2]	7.87e-2 [7.85e-2, 7.89e-2]	7.87e-2 [7.85e-2, 7.89e-2]	7.86e-2 [7.85e-2, 7.88e-2]	7.86e-2 [7.84e-2, 7.88e-2]	7.86e-2 [7.84e-2, 7.88e-2]
1	5.63e-2	5.63e-2 [5.62e-2, 5.65e-2]	5.63e-2 [5.61e-2, 5.64e-2]	5.63e-2 [5.61e-2, 5.64e-2]	5.64e-2 [5.63e-2, 5.65e-2]	5.63e-2 [5.61e-2, 5.64e-2]	5.61e-2 [5.60e-2, 5.63e-2]
2	3.75e-2	3.75e-2 [3.74e-2, 3.76e-2]	3.74e-2 [3.73e-2, 3.76e-2]	3.74e-2 [3.73e-2, 3.76e-2]	3.75e-2 [3.74e-2, 3.77e-2]	3.75e-2 [3.73e-2, 3.76e-2]	3.76e-2 [3.75e-2, 3.77e-2]
3	2.29e-2	2.28e-2 [2.27e-2, 2.29e-2]	2.30e-2 [2.29e-2, 2.31e-2]	2.30e-2 [2.29e-2, 2.31e-2]	2.29e-2 [2.28e-2, 2.29e-2]	2.29e-2 [2.28e-2, 2.30e-2]	2.29e-2 [2.28e-2, 2.30e-2]
4	1.25e-2	1.25e-2 [1.25e-2, 1.26e-2]	1.25e-2 [1.24e-2, 1.26e-2]	1.25e-2 [1.24e-2, 1.26e-2]	1.25e-2 [1.24e-2, 1.26e-2]	1.25e-2 [1.24e-2, 1.26e-2]	1.25e-2 [1.25e-2, 1.26e-2]
5	5.95e-3	5.95e-3 [5.90e-3, 6.00e-3]	5.96e-3 [5.92e-3, 6.01e-3]	5.96e-3 [5.92e-3, 6.01e-3]	5.98e-3 [5.93e-3, 6.03e-3]	5.93e-3 [5.89e-3, 5.98e-3]	5.93e-3 [5.88e-3, 5.98e-3]
6	2.39e-3	2.39e-3 [2.36e-3, 2.42e-3]	2.41e-3 [2.38e-3, 2.44e-3]	2.41e-3 [2.38e-3, 2.44e-3]	2.36e-3 [2.33e-3, 2.39e-3]	2.43e-3 [2.40e-3, 2.46e-3]	2.39e-3 [2.36e-3, 2.42e-3]
7	7.73e-4	7.56e-4 [7.39e-4, 7.73e-4]	7.77e-4 [7.60e-4, 7.95e-4]	7.77e-4 [7.60e-4, 7.95e-4]	7.64e-4 [7.47e-4, 7.81e-4]	8.02e-4 [7.84e-4, 8.20e-4]	7.88e-4 [7.70e-4, 8.05e-4]
8	1.91e-4	1.84e-4 [1.76e-4, 1.92e-4]	1.93e-4 [1.84e-4, 2.01e-4]	1.93e-4 [1.84e-4, 2.01e-4]	1.93e-4 [1.84e-4, 2.01e-4]	1.90e-4 [1.82e-4, 1.99e-4]	1.82e-4 [1.73e-4, 1.90e-4]
9	3.36e-5	3.21e-5 [2.88e-5, 3.58e-5]	3.41e-5 [3.07e-5, 3.79e-5]	3.41e-5 [3.07e-5, 3.79e-5]	3.36e-5 [3.02e-5, 3.74e-5]	3.42e-5 [3.08e-5, 3.80e-5]	3.57e-5 [3.22e-5, 3.96e-5]
10	3.87e-6	4.00e-6 [2.94e-6, 5.45e-6]	3.20e-6 [2.27e-6, 4.52e-6]	3.20e-6 [2.27e-6, 4.52e-6]	2.40e-6 [1.61e-6, 3.57e-6]	4.10e-6 [3.02e-6, 5.56e-6]	3.70e-6 [2.68e-6, 5.10e-6]

16-QAM: error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	theory	2.15	yes	binomial
comnum.run_spec	float64	theory	1.64	yes	binomial
comnum binary (Rust)	float64	theory	1.64	yes	binomial
komm	float64	theory	2.08	yes	binomial
OptiCommPy (numba)	complex64 symbols (its Gray constellation), float64 noise	theory	1.42	yes	binomial
Sionna	float32 (config.precision = ‘single’)	theory	2.41	yes	binomial

BER and its 95 % interval, every point (10,000,000 bits per point)

Eb/N0 (dB)	theory	comnum (functions)	comnum.run_spec	comnum binary (Rust)	komm	OptiCommPy (numba)	Sionna
0	1.41e-1	1.41e-1 [1.41e-1, 1.41e-1]	1.41e-1 [1.41e-1, 1.41e-1]	1.41e-1 [1.41e-1, 1.41e-1]	1.41e-1 [1.41e-1, 1.41e-1]	1.41e-1 [1.41e-1, 1.41e-1]	1.41e-1 [1.41e-1, 1.41e-1]
1	1.19e-1	1.19e-1 [1.19e-1, 1.19e-1]	1.19e-1 [1.19e-1, 1.19e-1]	1.19e-1 [1.19e-1, 1.19e-1]	1.19e-1 [1.19e-1, 1.19e-1]	1.19e-1 [1.19e-1, 1.19e-1]	1.19e-1 [1.19e-1, 1.19e-1]
2	9.77e-2	9.78e-2 [9.76e-2, 9.80e-2]	9.77e-2 [9.75e-2, 9.79e-2]	9.77e-2 [9.75e-2, 9.79e-2]	9.77e-2 [9.75e-2, 9.79e-2]	9.78e-2 [9.76e-2, 9.80e-2]	9.75e-2 [9.73e-2, 9.77e-2]
3	7.75e-2	7.75e-2 [7.74e-2, 7.77e-2]	7.74e-2 [7.73e-2, 7.76e-2]	7.74e-2 [7.73e-2, 7.76e-2]	7.76e-2 [7.74e-2, 7.77e-2]	7.75e-2 [7.73e-2, 7.76e-2]	7.74e-2 [7.73e-2, 7.76e-2]
4	5.86e-2	5.87e-2 [5.85e-2, 5.88e-2]	5.86e-2 [5.84e-2, 5.87e-2]	5.86e-2 [5.84e-2, 5.87e-2]	5.86e-2 [5.85e-2, 5.88e-2]	5.85e-2 [5.84e-2, 5.87e-2]	5.85e-2 [5.84e-2, 5.87e-2]
5	4.19e-2	4.18e-2 [4.17e-2, 4.19e-2]	4.19e-2 [4.17e-2, 4.20e-2]	4.19e-2 [4.17e-2, 4.20e-2]	4.19e-2 [4.18e-2, 4.20e-2]	4.20e-2 [4.19e-2, 4.21e-2]	4.20e-2 [4.19e-2, 4.21e-2]
6	2.79e-2	2.79e-2 [2.78e-2, 2.80e-2]	2.79e-2 [2.78e-2, 2.80e-2]	2.79e-2 [2.78e-2, 2.80e-2]	2.79e-2 [2.78e-2, 2.80e-2]	2.78e-2 [2.77e-2, 2.79e-2]	2.79e-2 [2.78e-2, 2.81e-2]
7	1.70e-2	1.70e-2 [1.69e-2, 1.71e-2]	1.70e-2 [1.69e-2, 1.71e-2]	1.70e-2 [1.69e-2, 1.71e-2]	1.70e-2 [1.69e-2, 1.70e-2]	1.70e-2 [1.69e-2, 1.70e-2]	1.70e-2 [1.69e-2, 1.71e-2]
8	9.25e-3	9.24e-3 [9.18e-3, 9.30e-3]	9.24e-3 [9.18e-3, 9.30e-3]	9.24e-3 [9.18e-3, 9.30e-3]	9.31e-3 [9.25e-3, 9.37e-3]	9.22e-3 [9.16e-3, 9.28e-3]	9.24e-3 [9.19e-3, 9.30e-3]
9	4.39e-3	4.35e-3 [4.30e-3, 4.39e-3]	4.42e-3 [4.38e-3, 4.47e-3]	4.42e-3 [4.38e-3, 4.47e-3]	4.39e-3 [4.35e-3, 4.43e-3]	4.40e-3 [4.36e-3, 4.44e-3]	4.37e-3 [4.33e-3, 4.41e-3]
10	1.75e-3	1.75e-3 [1.73e-3, 1.78e-3]	1.78e-3 [1.75e-3, 1.80e-3]	1.78e-3 [1.75e-3, 1.80e-3]	1.73e-3 [1.71e-3, 1.76e-3]	1.76e-3 [1.73e-3, 1.79e-3]	1.76e-3 [1.73e-3, 1.78e-3]

QPSK: time

Median of 5 runs, ns per bit; workload: 1000000 bits at Eb/N0 = 6.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	Relative	First call
comnum (functions)	float64	7.58	7.56	1.00	0.0198 s
comnum.run_spec	float64	9.60	1.71	1.27	0.00968 s
comnum binary (Rust)	float64	12.9	4.57	1.70	0.0136 s
komm	float64	249	252	32.9	0.253 s
OptiCommPy (numba)	complex64 symbols (its Gray constellation), float64 noise	24.2	15.1	3.19	0.921 s
OptiCommPy (no numba)	complex64 symbols (its Gray constellation), float64 noise	373	373	49.2	0.0376 s
Sionna	float32 (config.precision = ‘single’)	25.2	16.4	3.33	0.0377 s

16-QAM: time

Median of 5 runs, ns per bit; workload: 1000000 bits at Eb/N0 = 6.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	Relative	First call
comnum (functions)	float64	4.44	4.39	1.00	0.00451 s
comnum.run_spec	float64	5.41	1.11	1.22	0.00539 s
comnum binary (Rust)	float64	8.14	3.76	1.83	0.0086 s
komm	float64	136	139	30.6	0.137 s
OptiCommPy (numba)	complex64 symbols (its Gray constellation), float64 noise	20.9	9.85	4.70	0.0213 s
OptiCommPy (no numba)	complex64 symbols (its Gray constellation), float64 noise	192	191	43.2	0.0192 s
Sionna	float32 (config.precision = ‘single’)	32.9	19.1	7.41	0.0362 s

The commands of every run

OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 VECLIB_MAXIMUM_THREADS=1 NUMBA_NUM_THREADS=1 RAYON_NUM_THREADS=1 python s1_awgn.py --lib comnum --threads 1 --work results/_work --out results/_work/s1_awgn_comnum_1.json
 python s1_awgn.py --lib comnum --threads all --work results/_work --out results/_work/s1_awgn_comnum_all.json --timing-only
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 VECLIB_MAXIMUM_THREADS=1 NUMBA_NUM_THREADS=1 RAYON_NUM_THREADS=1 python s1_awgn.py --lib comnum-spec --threads 1 --work results/_work --out results/_work/s1_awgn_comnum-spec_1.json
 python s1_awgn.py --lib comnum-spec --threads all --work results/_work --out results/_work/s1_awgn_comnum-spec_all.json --timing-only
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 VECLIB_MAXIMUM_THREADS=1 NUMBA_NUM_THREADS=1 RAYON_NUM_THREADS=1 python s1_awgn.py --lib comnum-rust --threads 1 --work results/_work --out results/_work/s1_awgn_comnum-rust_1.json
 python s1_awgn.py --lib comnum-rust --threads all --work results/_work --out results/_work/s1_awgn_comnum-rust_all.json --timing-only
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 VECLIB_MAXIMUM_THREADS=1 NUMBA_NUM_THREADS=1 RAYON_NUM_THREADS=1 python s1_awgn.py --lib komm --threads 1 --work results/_work --out results/_work/s1_awgn_komm_1.json
 python s1_awgn.py --lib komm --threads all --work results/_work --out results/_work/s1_awgn_komm_all.json --timing-only
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 VECLIB_MAXIMUM_THREADS=1 NUMBA_NUM_THREADS=1 RAYON_NUM_THREADS=1 python s1_awgn.py --lib opticommpy --threads 1 --work results/_work --out results/_work/s1_awgn_opticommpy_1.json
 python s1_awgn.py --lib opticommpy --threads all --work results/_work --out results/_work/s1_awgn_opticommpy_all.json --timing-only
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 VECLIB_MAXIMUM_THREADS=1 NUMBA_NUM_THREADS=1 RAYON_NUM_THREADS=1 NUMBA_DISABLE_JIT=1 python s1_awgn.py --lib opticommpy-nojit --threads 1 --work results/_work --out results/_work/s1_awgn_opticommpy-nojit_1.json --timing-only
NUMBA_DISABLE_JIT=1 python s1_awgn.py --lib opticommpy-nojit --threads all --work results/_work --out results/_work/s1_awgn_opticommpy-nojit_all.json --timing-only
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 VECLIB_MAXIMUM_THREADS=1 NUMBA_NUM_THREADS=1 RAYON_NUM_THREADS=1 python s1_awgn.py --lib sionna --threads 1 --work results/_work --out results/_work/s1_awgn_sionna_1.json
 python s1_awgn.py --lib sionna --threads all --work results/_work --out results/_work/s1_awgn_sionna_all.json --timing-only

BPSK and QPSK over flat Rayleigh fading#

Scenario 2: BPSK and Gray QPSK over flat Rayleigh fading (one coefficient per symbol, perfectly known at the receiver), BER against the closed form.

Parameters:

Eb/N0 (dB): 0, 5, 10, 15, 20, 25, 30
bits per point: 10^7 (quick: 2·10^5), in blocks of 10^6 (quick: 10^5)
channel: y = h x + w, h ~ CN(0, 1) independent for every symbol, known at the receiver
timing: one block of 10^6 bits (quick: 2·10^5) at Eb/N0 = 10 dB
theory: BPSK: (1 - sqrt(G / (1 + G))) / 2; Gray QPSK: the same per bit (Proakis & Salehi, 5th ed., Sec. 13.3)

Script: benchmarks/libraries/s2_rayleigh.py.

The script

"""Scenario 2: BPSK and Gray QPSK over flat Rayleigh fading (one coefficient per
symbol, perfectly known at the receiver), BER against the closed form.

Every library draws its own bits, channel coefficients h ~ CN(0, 1) and noise, and
detects with its own coherent detector given h (zero forcing over the 1 x 1 channel:
comnum.mimo_detect, Sionna's LinearDetector, the mimo_rx block of the specification).
komm and OptiCommPy have no fading channel. Timing: one block of ``TIMING_BITS`` bits
at Eb/N0 = 10 dB, median of 5 runs after a warm-up run.

    python s2_rayleigh.py --lib sionna --threads 1 [--quick] --out results/_work/s2.json
"""

from __future__ import annotations

import math
import time

import common
import numpy as np
import theory

SCENARIO = "s2_rayleigh"
TITLE = "BPSK and QPSK over flat Rayleigh fading"
ORDERS = {"bpsk": 2, "qpsk": 4}
SUPPORT = {
    "comnum": True,
    "comnum-spec": True,
    "komm": "no fading channel",
    "opticommpy": "no fading channel",
    "opticommpy-nojit": "no fading channel",
    "sionna": True,
}
CASES = {"bpsk": SUPPORT, "qpsk": SUPPORT}
EBN0 = [0.0, 5.0, 10.0, 15.0, 20.0, 25.0, 30.0]
TIMING_EBN0 = 10.0
TIMING_TRIALS = 40

PARAMETERS = {
    "ebn0_db": EBN0,
    "bits_per_point": "10^7 (quick: 2·10^5), in blocks of 10^6 (quick: 10^5)",
    "channel": "y = h x + w, h ~ CN(0, 1) independent for every symbol, known at the receiver",
    "timing": "one block of 10^6 bits (quick: 2·10^5) at Eb/N0 = 10 dB",
    "theory": "BPSK: (1 - sqrt(G / (1 + G))) / 2; Gray QPSK: the same per bit (Proakis & Salehi, 5th ed., Sec. 13.3)",
}


def sizes(quick: bool) -> dict:
    if quick:
        return {"bits_per_point": 200_000, "block": 100_000, "timing_bits": 200_000, "trials": 10}
    return {
        "bits_per_point": 10_000_000,
        "block": 1_000_000,
        "timing_bits": 1_000_000,
        "trials": 100,
    }


def chain_comnum(m: int, seed: int):
    import comnum

    c = comnum.Constellation("psk" if m == 2 else "qam", m).normalized()
    k = int(math.log2(m))
    rng = comnum.Rng(seed)

    def run(n: int, ebn0: float) -> int:
        b = comnum.bits(n, rng)
        x = c.map_bits(b)
        h = comnum.mimo_channels(x.size, 1, 1, rng)
        n0 = 1.0 / (k * 10 ** (ebn0 / 10))
        y = h[:, 0, 0] * x + rng.complex_noise(x.size, n0)
        labels = comnum.mimo_detect(h, y[None, :], c, n0, "zf")
        bh = comnum.unpack_symbols(labels.ravel(), k)
        return int(np.count_nonzero(bh != b))

    return run, "float64"


def chain_sionna(m: int, seed: int):
    import torch
    from sionna.phy import config
    from sionna.phy.channel import FlatFadingChannel
    from sionna.phy.mapping import BinarySource, Mapper
    from sionna.phy.mimo import LinearDetector
    from sionna.phy.utils import ebnodb2no

    config.seed = seed
    k = int(math.log2(m))
    kind = "pam" if m == 2 else "qam"
    source, mapper = BinarySource(), Mapper(kind, k)
    channel = FlatFadingChannel(1, 1, return_channel=True)
    detector = LinearDetector(
        "zf", "bit", "maxlog", constellation_type=kind, num_bits_per_symbol=k, hard_out=True
    )

    def run(n: int, ebn0: float) -> int:
        with torch.inference_mode():
            b = source([n // k, k])
            x = mapper(b)  # [n/k, 1]
            no = ebnodb2no(ebn0, k, 1.0)
            y, h = channel(x, no)
            s = torch.full((n // k, 1, 1), float(no), dtype=y.dtype)
            bh = detector(y, h, s)  # [n/k, 1, k]
            return int((bh.reshape(b.shape) != b).sum())

    return run, "float32 (config.precision = 'single')"


CHAINS = {"comnum": chain_comnum, "sionna": chain_sionna}


def run_chains(args, todo: list[str]) -> list[dict]:
    threading = common.setup_torch(args.threads) if args.lib == "sionna" else {}
    sz = sizes(args.quick)
    out = []
    for case in todo:
        m = ORDERS[case]
        run, precision = CHAINS[args.lib](m, args.seed)
        res = {"case": case, "precision": precision, **threading}
        n = sz["timing_bits"]
        t0 = time.perf_counter()
        run(n, TIMING_EBN0)
        res["first_call_s"] = time.perf_counter() - t0
        if not args.timing_only:
            pts = []
            t0 = time.perf_counter()
            for e in EBN0:
                errors = sum(
                    run(sz["block"], e) for _ in range(sz["bits_per_point"] // sz["block"])
                )
                pts.append(
                    common.rate_point(
                        errors,
                        sz["bits_per_point"],
                        float(theory.ber_qam_rayleigh(m, e)),
                        ebn0_db=e,
                    )
                )
            res["points"] = pts
            res["sweep_s"] = time.perf_counter() - t0
        res["timing"] = common.time_it(lambda n=n, run=run: run(n, TIMING_EBN0))
        res["timing"].update(units=n, unit="bit", workload=f"{n} bits at Eb/N0 = {TIMING_EBN0} dB")
        out.append(res)
    return out


def spec(m: int, ebn0: list[float], bits: int, trials: int, seed: int) -> dict:
    k = int(math.log2(m))
    return {
        "name": f"bench_rayleigh_{m}",
        "seed": seed,
        "batch": trials,
        "grid": {"ebn0_db": ebn0},
        "link": [
            {
                "type": "source",
                "constellation": {"kind": "psk" if m == 2 else "qam", "m": m},
                "n_symbols": bits // (k * trials),
            },
            {"type": "rayleigh", "n_rx": 1, "coherence_samples": 1},
            {"type": "awgn_ebn0", "ebn0_db": "$ebn0_db"},
            {"type": "mimo_rx", "detector": "zf"},
        ],
        "metrics": {"report": ["ber"], "scale": "none"},
        "stop": {"min_trials": trials, "max_trials": trials},
    }


def spec_points(rows: list[dict], m: int) -> list[dict]:
    pts = []
    for row in rows:
        e = float(row["condition"]["ebn0_db"])
        ber = row["metrics"]["ber"]
        pts.append(
            common.rate_point(
                ber["count"], ber["total"], float(theory.ber_qam_rayleigh(m, e)), ebn0_db=e
            )
        )
    return pts


def run_spec_lib(args, todo: list[str]) -> list[dict]:
    sz = sizes(args.quick)
    n = sz["timing_bits"]
    out = []
    for case in todo:
        m = ORDERS[case]
        out.append(
            common.spec_case(
                args,
                {"case": case, "precision": "float64"},
                spec(m, EBN0, sz["bits_per_point"], sz["trials"], args.seed),
                spec(m, [TIMING_EBN0], n, TIMING_TRIALS, args.seed + 1),
                lambda rows, m=m: spec_points(rows, m),
                units=n,
                unit="bit",
                workload=f"{n} bits at Eb/N0 = {TIMING_EBN0} dB, {TIMING_TRIALS} trials",
            )
        )
    return out


RUNNERS = {"comnum": run_chains, "sionna": run_chains, "comnum-spec": run_spec_lib}

if __name__ == "__main__":
    common.main(SCENARIO, CASES, RUNNERS)

Support

Case	comnum (functions)	comnum.run_spec	komm	OptiCommPy (numba)	OptiCommPy (no numba)	Sionna
BPSK	yes	yes	not available: no fading channel	not available: no fading channel	not available: no fading channel	yes
QPSK	yes	yes	not available: no fading channel	not available: no fading channel	not available: no fading channel	yes

BPSK: error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	theory	2.22	yes	binomial
comnum.run_spec	float64	theory	1.78	yes	binomial
Sionna	float32 (config.precision = ‘single’)	theory	2.35	yes	binomial

BER and its 95 % interval, every point (10,000,000 bits per point)

Eb/N0 (dB)	theory	comnum (functions)	comnum.run_spec	Sionna
0	1.46e-1	1.46e-1 [1.46e-1, 1.47e-1]	1.46e-1 [1.46e-1, 1.47e-1]	1.47e-1 [1.46e-1, 1.47e-1]
5	6.42e-2	6.43e-2 [6.41e-2, 6.44e-2]	6.42e-2 [6.41e-2, 6.44e-2]	6.42e-2 [6.40e-2, 6.43e-2]
10	2.33e-2	2.33e-2 [2.32e-2, 2.34e-2]	2.32e-2 [2.32e-2, 2.33e-2]	2.33e-2 [2.32e-2, 2.34e-2]
15	7.72e-3	7.73e-3 [7.67e-3, 7.78e-3]	7.76e-3 [7.71e-3, 7.82e-3]	7.75e-3 [7.69e-3, 7.80e-3]
20	2.48e-3	2.51e-3 [2.48e-3, 2.54e-3]	2.46e-3 [2.43e-3, 2.49e-3]	2.52e-3 [2.49e-3, 2.55e-3]
25	7.89e-4	8.08e-4 [7.91e-4, 8.26e-4]	7.83e-4 [7.66e-4, 8.00e-4]	7.88e-4 [7.71e-4, 8.06e-4]
30	2.50e-4	2.52e-4 [2.42e-4, 2.62e-4]	2.41e-4 [2.31e-4, 2.51e-4]	2.43e-4 [2.34e-4, 2.53e-4]

QPSK: error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	theory	2.08	yes	binomial
comnum.run_spec	float64	theory	1.17	yes	binomial
Sionna	float32 (config.precision = ‘single’)	theory	3.13	yes	binomial

BER and its 95 % interval, every point (10,000,000 bits per point)

Eb/N0 (dB)	theory	comnum (functions)	comnum.run_spec	Sionna
0	1.46e-1	1.46e-1 [1.46e-1, 1.46e-1]	1.46e-1 [1.46e-1, 1.47e-1]	1.46e-1 [1.46e-1, 1.47e-1]
5	6.42e-2	6.41e-2 [6.39e-2, 6.42e-2]	6.42e-2 [6.40e-2, 6.43e-2]	6.42e-2 [6.40e-2, 6.43e-2]
10	2.33e-2	2.34e-2 [2.33e-2, 2.34e-2]	2.33e-2 [2.32e-2, 2.34e-2]	2.31e-2 [2.30e-2, 2.32e-2]
15	7.72e-3	7.77e-3 [7.72e-3, 7.82e-3]	7.73e-3 [7.67e-3, 7.78e-3]	7.67e-3 [7.61e-3, 7.72e-3]
20	2.48e-3	2.51e-3 [2.48e-3, 2.54e-3]	2.50e-3 [2.47e-3, 2.53e-3]	2.46e-3 [2.43e-3, 2.50e-3]
25	7.89e-4	8.02e-4 [7.84e-4, 8.19e-4]	7.81e-4 [7.64e-4, 7.99e-4]	7.92e-4 [7.75e-4, 8.10e-4]
30	2.50e-4	2.53e-4 [2.44e-4, 2.63e-4]	2.46e-4 [2.37e-4, 2.56e-4]	2.51e-4 [2.41e-4, 2.61e-4]

BPSK: time

Median of 5 runs, ns per bit; workload: 1000000 bits at Eb/N0 = 10.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	Relative	First call
comnum (functions)	float64	76.5	77.2	1.00	0.0814 s
comnum.run_spec	float64	99.7	18.0	1.30	0.1 s
Sionna	float32 (config.precision = ‘single’)	87.6	72.6	1.15	0.099 s

QPSK: time

Median of 5 runs, ns per bit; workload: 1000000 bits at Eb/N0 = 10.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	Relative	First call
comnum (functions)	float64	39.1	39.4	1.00	0.0404 s
comnum.run_spec	float64	50.9	9.96	1.30	0.0514 s
Sionna	float32 (config.precision = ‘single’)	63.0	46.7	1.61	0.0628 s

4 x 4 MIMO, 16-QAM, i.i.d. Rayleigh fading: ZF, MMSE and ML detection#

Scenario 3: 4 x 4 spatial multiplexing of Gray 16-QAM over i.i.d. Rayleigh fading (one channel matrix per vector, known at the receiver), detected by zero forcing, MMSE and maximum likelihood.

Parameters:

Eb/N0 (dB): 0, 4, 8, 12, 16
vectors per point: linear detectors and sphere: 10^5 (quick: 5·10^3); exhaustive ML: 2·10^4 (quick: 10^3)
timing: 10^4 vectors at Eb/N0 = 12 dB (exhaustive ML: 10^3; quick: 2·10^3 and 2·10^2)
theory: ZF: 16-QAM over SISO Rayleigh fading (Winters, Salz & Gitlin, IEEE Trans. Commun. 42(2/3/4), 1994)

Script: benchmarks/libraries/s3_mimo.py.

The script

"""Scenario 3: 4 x 4 spatial multiplexing of Gray 16-QAM over i.i.d. Rayleigh fading
(one channel matrix per vector, known at the receiver), detected by zero forcing, MMSE
and maximum likelihood.

Every stream carries unit-energy symbols (no power split between the antennas) and
Eb/N0 is the energy per bit received on one antenna over N0: N0 = 1 / (4 Eb/N0).
Zero forcing over 4 x 4 leaves one diversity branch per stream, so its BER is that of
16-QAM over SISO Rayleigh fading (Winters, Salz & Gitlin 1994); MMSE and ML have no
closed form here and are compared between the libraries. Maximum likelihood is an
exhaustive search over the 16^4 = 65536 vectors in Sionna and in comnum ("ml"), and
the Schnorr-Euchner sphere decoder in comnum ("sphere", the same decisions).
The specification runner splits the power between the antennas (mimo_tx), so it runs
at Eb/N0 + 10 log10(4) dB to have the same SNR. Timing: ``TIMING_VECTORS`` vectors at
Eb/N0 = 12 dB (channel, noise, detection), median of 5 runs after a warm-up run.

    python s3_mimo.py --lib comnum --threads 1 [--quick] --out results/_work/s3.json
"""

from __future__ import annotations

import math
import time

import common
import numpy as np
import theory

SCENARIO = "s3_mimo"
TITLE = "4 x 4 MIMO, 16-QAM, i.i.d. Rayleigh fading: ZF, MMSE and ML detection"
NT = NR = 4
M = 16
K = 4
NO_MIMO = "no MIMO detection"
SUPPORT = {
    "comnum": True,
    "comnum-spec": True,
    "komm": NO_MIMO,
    "opticommpy": "no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)",
    "opticommpy-nojit": "no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)",
    "sionna": True,
}
CASES = {
    "zf": SUPPORT,
    "mmse": SUPPORT,
    "ml": SUPPORT,
    "sphere": {
        **SUPPORT,
        "sionna": "no sphere decoder (its exhaustive ML detector is the case ml)",
    },
}
SPEC_DETECTOR = {"zf": "zf", "mmse": "mmse", "ml": "ml", "sphere": "sphere"}
EBN0 = [0.0, 4.0, 8.0, 12.0, 16.0]
TIMING_EBN0 = 12.0
SPEC_OFFSET_DB = 10 * math.log10(NT)

PARAMETERS = {
    "ebn0_db": EBN0,
    "vectors_per_point": "linear detectors and sphere: 10^5 (quick: 5·10^3); exhaustive ML: 2·10^4 (quick: 10^3)",
    "timing": "10^4 vectors at Eb/N0 = 12 dB (exhaustive ML: 10^3; quick: 2·10^3 and 2·10^2)",
    "theory": "ZF: 16-QAM over SISO Rayleigh fading (Winters, Salz & Gitlin, IEEE Trans. Commun. 42(2/3/4), 1994)",
}


def sizes(quick: bool, case: str) -> dict:
    heavy = case == "ml"
    if quick:
        return {
            "vectors": 1_000 if heavy else 5_000,
            "timing": 200 if heavy else 2_000,
            "blocks": 1,
        }
    return {
        "vectors": 20_000 if heavy else 100_000,
        "timing": 1_000 if heavy else 10_000,
        "blocks": 10,
    }


def theory_ber(case: str, ebn0: float) -> float | None:
    return float(theory.ber_qam_rayleigh(M, ebn0)) if case == "zf" else None


def chain_comnum(case: str, seed: int):
    import comnum

    c = comnum.Constellation("qam", M).normalized()
    rng = comnum.Rng(seed)

    def run(n: int, ebn0: float) -> np.ndarray:
        b = comnum.bits(n * NT * K, rng)
        x = c.map_bits(b).reshape(n, NT)
        h = comnum.mimo_channels(n, NR, NT, rng)
        n0 = 1.0 / (K * 10 ** (ebn0 / 10))
        # y = H x + w (the Python API draws H; the product is NumPy's)
        y = np.matmul(h, x[:, :, None])[:, :, 0] + rng.complex_noise(n * NR, n0).reshape(n, NR)
        labels = comnum.mimo_detect(h, y.T, c, n0, case)  # stream-major (NT, n)
        bh = comnum.unpack_symbols(labels.T.ravel(), K)
        return np.count_nonzero((bh != b).reshape(n, NT * K), axis=1)  # bit errors per vector

    return run, "float64"


def chain_sionna(case: str, seed: int):
    import torch
    from sionna.phy import config
    from sionna.phy.channel import FlatFadingChannel
    from sionna.phy.mapping import BinarySource, Mapper
    from sionna.phy.mimo import LinearDetector, MaximumLikelihoodDetector
    from sionna.phy.utils import ebnodb2no

    config.seed = seed
    source, mapper = BinarySource(), Mapper("qam", K)
    channel = FlatFadingChannel(NT, NR, return_channel=True)
    if case == "ml":
        det = MaximumLikelihoodDetector(
            "bit", "maxlog", NT, constellation_type="qam", num_bits_per_symbol=K, hard_out=True
        )
        chunk = 64  # vectors per call: the search holds 65536 candidates per vector
    else:
        eq = {"zf": "zf", "mmse": "lmmse"}[case]
        det = LinearDetector(
            eq, "bit", "maxlog", constellation_type="qam", num_bits_per_symbol=K, hard_out=True
        )
        chunk = None

    def run(n: int, ebn0: float) -> np.ndarray:
        with torch.inference_mode():
            b = source([n, NT * K])
            x = mapper(b)  # [n, NT]
            no = ebnodb2no(ebn0, K, 1.0)  # Es / (K Eb/N0), Es = 1
            y, h = channel(x, no)
            s = (float(no) * torch.eye(NR, dtype=y.dtype)).expand(n, NR, NR)
            if chunk is None:
                bh = det(y, h, s)
            else:
                bh = torch.cat(
                    [
                        det(y[i : i + chunk], h[i : i + chunk], s[i : i + chunk])
                        for i in range(0, n, chunk)
                    ]
                )
            return common.to_numpy((bh.reshape(b.shape) != b).sum(dim=1))  # bit errors per vector

    return run, "float32 (config.precision = 'single')"


CHAINS = {"comnum": chain_comnum, "sionna": chain_sionna}


def run_chains(args, todo: list[str]) -> list[dict]:
    threading = common.setup_torch(args.threads) if args.lib == "sionna" else {}
    out = []
    for case in todo:
        sz = sizes(args.quick, case)
        run, precision = CHAINS[args.lib](case, args.seed)
        res = {"case": case, "precision": precision, **threading}
        n = sz["timing"]
        t0 = time.perf_counter()
        run(n, TIMING_EBN0)
        res["first_call_s"] = time.perf_counter() - t0
        if not args.timing_only:
            pts = []
            t0 = time.perf_counter()
            for e in EBN0:
                # the 16 bits of a vector share its channel: the interval is per vector
                errs = np.concatenate(
                    [run(sz["vectors"] // sz["blocks"], e) for _ in range(sz["blocks"])]
                )
                pts.append(
                    common.frame_ber_point(
                        errs, NT * K, theory_ber(case, e), ebn0_db=e, vectors=errs.size
                    )
                )
            res["points"] = pts
            res["sweep_s"] = time.perf_counter() - t0
        res["timing"] = common.time_it(lambda n=n, run=run: run(n, TIMING_EBN0))
        res["timing"].update(
            units=n, unit="vector", workload=f"{n} vectors at Eb/N0 = {TIMING_EBN0} dB"
        )
        out.append(res)
    return out


def spec(case: str, ebn0: list[float], vectors: int, trials: int, seed: int) -> dict:
    return {
        "name": f"bench_mimo_{case}",
        "seed": seed,
        "batch": trials,
        "grid": {"ebn0_db": [e + SPEC_OFFSET_DB for e in ebn0]},
        "link": [
            {"type": "source", "constellation": "16-QAM", "n_symbols": NT * vectors // trials},
            {"type": "mimo_tx", "n_tx": NT, "scheme": "spatial"},
            {"type": "rayleigh", "n_rx": NR, "coherence_samples": 1},
            {"type": "awgn_ebn0", "ebn0_db": "$ebn0_db"},
            {"type": "mimo_rx", "detector": SPEC_DETECTOR[case]},
        ],
        "metrics": {"report": ["ber"], "scale": "none"},
        "stop": {"min_trials": trials, "max_trials": trials},
    }


def run_spec_lib(args, todo: list[str]) -> list[dict]:
    trials = 40
    out = []
    for case in todo:
        sz = sizes(args.quick, case)

        def points(rows: list[dict], case=case, vectors=sz["vectors"]) -> list[dict]:
            pts = []
            for e, row in zip(EBN0, rows, strict=True):
                ber = row["metrics"]["ber"]
                pts.append(
                    common.rate_point(
                        ber["count"], ber["total"], theory_ber(case, e), ebn0_db=e, vectors=vectors
                    )
                )
            return pts

        n = sz["timing"]
        out.append(
            common.spec_case(
                args,
                {"case": case, "precision": "float64", "ebn0_offset_db": SPEC_OFFSET_DB},
                spec(case, EBN0, sz["vectors"], trials, args.seed),
                spec(case, [TIMING_EBN0], n, trials, args.seed + 1),
                points,
                units=n,
                unit="vector",
                workload=f"{n} vectors at Eb/N0 = {TIMING_EBN0} dB, {trials} trials",
            )
        )
    return out


RUNNERS = {"comnum": run_chains, "sionna": run_chains, "comnum-spec": run_spec_lib}

if __name__ == "__main__":
    common.main(SCENARIO, CASES, RUNNERS)

Support

Case	comnum (functions)	comnum.run_spec	komm	OptiCommPy (numba)	OptiCommPy (no numba)	Sionna
zero forcing	yes	yes	not available: no MIMO detection	not available: no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)	not available: no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)	yes
MMSE	yes	yes	not available: no MIMO detection	not available: no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)	not available: no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)	yes
ML (exhaustive)	yes	yes	not available: no MIMO detection	not available: no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)	not available: no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)	yes
ML (sphere decoder)	yes	yes	not available: no MIMO detection	not available: no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)	not available: no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)	not available: no sphere decoder (its exhaustive ML detector is the case ml)

zero forcing: error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	theory	1.88	yes	per-frame
comnum.run_spec	float64	theory	2.98	yes	binomial
Sionna	float32 (config.precision = ‘single’)	theory	1.79	yes	per-frame

BER and its 95 % interval, every point (1,600,000 bits per point)

Eb/N0 (dB)	theory	comnum (functions)	comnum.run_spec	Sionna
0	1.98e-1	1.98e-1 [1.97e-1, 1.99e-1]	1.98e-1 [1.97e-1, 1.98e-1]	1.98e-1 [1.97e-1, 1.99e-1]
4	1.20e-1	1.19e-1 [1.18e-1, 1.20e-1]	1.19e-1 [1.19e-1, 1.20e-1]	1.19e-1 [1.19e-1, 1.20e-1]
8	6.20e-2	6.16e-2 [6.10e-2, 6.23e-2]	6.14e-2 [6.10e-2, 6.18e-2]	6.26e-2 [6.19e-2, 6.32e-2]
12	2.83e-2	2.82e-2 [2.78e-2, 2.87e-2]	2.81e-2 [2.79e-2, 2.84e-2]	2.83e-2 [2.78e-2, 2.87e-2]
16	1.20e-2	1.21e-2 [1.18e-2, 1.24e-2]	1.19e-2 [1.17e-2, 1.20e-2]	1.19e-2 [1.16e-2, 1.22e-2]

MMSE: error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	—	—	—	per-frame
comnum.run_spec	float64	comnum	3.63	yes	binomial
Sionna	float32 (config.precision = ‘single’)	comnum	1.57	yes	per-frame

BER and its 95 % interval, every point (1,600,000 bits per point)

Eb/N0 (dB)	comnum (functions)	comnum.run_spec	Sionna
0	1.39e-1 [1.38e-1, 1.40e-1]	1.38e-1 [1.38e-1, 1.39e-1]	1.38e-1 [1.38e-1, 1.39e-1]
4	8.27e-2 [8.21e-2, 8.33e-2]	8.40e-2 [8.36e-2, 8.44e-2]	8.33e-2 [8.28e-2, 8.39e-2]
8	4.31e-2 [4.26e-2, 4.35e-2]	4.29e-2 [4.26e-2, 4.32e-2]	4.32e-2 [4.28e-2, 4.37e-2]
12	1.97e-2 [1.94e-2, 2.00e-2]	1.99e-2 [1.97e-2, 2.01e-2]	1.96e-2 [1.93e-2, 1.99e-2]
16	8.42e-3 [8.20e-3, 8.64e-3]	8.33e-3 [8.19e-3, 8.47e-3]	8.23e-3 [8.02e-3, 8.45e-3]

ML (exhaustive): error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	—	—	—	per-frame
comnum.run_spec	float64	comnum	2.53	yes	binomial
Sionna	float32 (config.precision = ‘single’)	comnum	1.09	yes	per-frame

BER and its 95 % interval, every point (320,000 bits per point)

Eb/N0 (dB)	comnum (functions)	comnum.run_spec	Sionna
0	1.25e-1 [1.23e-1, 1.26e-1]	1.25e-1 [1.24e-1, 1.26e-1]	1.25e-1 [1.23e-1, 1.27e-1]
4	4.03e-2 [3.90e-2, 4.16e-2]	3.99e-2 [3.92e-2, 4.06e-2]	4.05e-2 [3.92e-2, 4.17e-2]
8	4.12e-3 [3.68e-3, 4.56e-3]	4.77e-3 [4.54e-3, 5.02e-3]	4.47e-3 [4.02e-3, 4.93e-3]
12	2.34e-4 [1.24e-4, 3.45e-4]	2.28e-4 [1.81e-4, 2.87e-4]	1.59e-4 [7.61e-5, 2.43e-4]
16	0 [0, 9.38e-6]	0 [0, 1.20e-5]	1.25e-5 [0, 3.70e-5]

ML (sphere decoder): error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	—	—	—	per-frame
comnum.run_spec	float64	comnum	2.84	yes	binomial

BER and its 95 % interval, every point (1,600,000 bits per point)

Eb/N0 (dB)	comnum (functions)	comnum.run_spec
0	1.24e-1 [1.23e-1, 1.25e-1]	1.25e-1 [1.24e-1, 1.25e-1]
4	3.97e-2 [3.92e-2, 4.03e-2]	4.06e-2 [4.03e-2, 4.09e-2]
8	4.44e-3 [4.23e-3, 4.64e-3]	4.62e-3 [4.51e-3, 4.73e-3]
12	1.91e-4 [1.49e-4, 2.32e-4]	1.95e-4 [1.75e-4, 2.18e-4]
16	3.13e-6 [0, 6.37e-6]	1.25e-5 [8.09e-6, 1.93e-5]

zero forcing: time

Median of 5 runs, µs per vector; workload: 10000 vectors at Eb/N0 = 12.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	Relative	First call
comnum (functions)	float64	0.76	0.763	1.00	0.00823 s
comnum.run_spec	float64	0.776	0.175	1.02	0.00794 s
Sionna	float32 (config.precision = ‘single’)	1.06	0.796	1.40	0.0148 s

MMSE: time

Median of 5 runs, µs per vector; workload: 10000 vectors at Eb/N0 = 12.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	Relative	First call
comnum (functions)	float64	0.761	0.76	1.00	0.00759 s
comnum.run_spec	float64	0.778	0.174	1.02	0.00782 s
Sionna	float32 (config.precision = ‘single’)	1.28	0.98	1.68	0.0129 s

ML (exhaustive): time

Median of 5 runs, µs per vector; workload: 1000 vectors at Eb/N0 = 12.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	Relative	First call
comnum (functions)	float64	393	394	1.00	0.393 s
comnum.run_spec	float64	393	105	1	0.398 s
Sionna	float32 (config.precision = ‘single’)	1519	908	3.87	1.57 s

ML (sphere decoder): time

Median of 5 runs, µs per vector; workload: 10000 vectors at Eb/N0 = 12.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	Relative	First call
comnum (functions)	float64	1.02	1.02	1.00	0.0108 s
comnum.run_spec	float64	1.03	0.209	1.01	0.0104 s

Channel coding over BPSK and AWGN#

Scenario 4: channel coding over BPSK and AWGN.

Parameters:

convolutional (133, 171), soft Viterbi: K = 7, (133, 171), k = 1024 + 6 tail bits, soft Viterbi; Eb/N0 = 1, 2, 3, 4 dB
5G NR LDPC (1024, 2048): 5G NR LDPC, k = 1024, n = 2048 (base graph 2, Z = 104), flooding sum-product, 50 iterations; Eb/N0 = 1, 1.5, 2, 2.5 dB
LDPC (1024, 512), same LLRs: PEG (1024, 512), flooding sum-product, 50 iterations, identical LLRs; Eb/N0 = 1.5, 2, 2.5, 3 dB
frames per point: 2000 (quick: 100)
timing: 200 frames (quick: 20) at Eb/N0 = 3 dB (conv) or 2 dB (LDPC); OptiCommPy without numba: 2 frames

Script: benchmarks/libraries/s4_coding.py.

The script

"""Scenario 4: channel coding over BPSK and AWGN.

- ``conv``: the rate-1/2 convolutional code of constraint length 7, generators
  (133, 171) in octal, 1024 information bits per frame terminated by 6 tail bits
  (rate 1024/2060), soft-decision Viterbi decoding. komm counts the taps from the
  least significant bit: the same code is ``ConvolutionalCode([[0o155, 0o117]])``
  there (checked on the impulse response). Reference: the union bound with the
  distance spectrum of the trellis (Viterbi 1971).
- ``nr_ldpc``: the LDPC code of 5G NR (3GPP TS 38.212, Sec. 5.3.2), k = 1024, rate
  matched to n = 2048 (base graph 2 by the rule of Sec. 7.2.2, lifting size 104, the
  first 2Z bits punctured; the same in both libraries), sum-product
  decoding by flooding, 50 iterations at most, in both libraries (comnum's default is
  the layered schedule, which converges in about half the iterations; it is set to
  flooding here so that the error rates compare the same decoder). comnum stops early on
  a codeword, Sionna runs the 50 iterations (its default).
- ``ldpc_same_llr``: one (1024, 512) LDPC code (PEG construction of comnum, variable
  degree 3, seed 1), its parity-check matrix given to the three decoders that accept
  one (comnum, OptiCommPy, Sionna), and the same channel LLRs (drawn by comnum from the
  same seed in every process): flooding sum-product, 50 iterations at most. The
  decoders see identical inputs, so they should take the same decisions frame by
  frame.

Every library uses its own encoder, BPSK mapper, AWGN channel and LLR computation in
``conv`` and ``nr_ldpc`` (independent seeds); Eb/N0 counts the energy per information
bit. Timings: the decoder alone on ``TIMING_FRAMES`` frames of LLRs at the timing
point, and the whole chain (bits, encoding, mapping, noise, LLRs, decoding, error
count) on the same number of frames; median of 5 runs after a warm-up run.

    python s4_coding.py --lib sionna --threads 1 [--quick] --out results/_work/s4.json
"""

from __future__ import annotations

import time

import common
import numpy as np
import theory

SCENARIO = "s4_coding"
TITLE = "Channel coding over BPSK and AWGN"
K_CONV = 1024
CONV_GEN = [0o133, 0o171]
K_NR, N_NR = 1024, 2048
N_PEG, K_PEG = 1024, 512
ITERATIONS = 50
EBN0 = {
    "conv": [1.0, 2.0, 3.0, 4.0],
    "nr_ldpc": [1.0, 1.5, 2.0, 2.5],
    "ldpc_same_llr": [1.5, 2.0, 2.5, 3.0],
}
TIMING_EBN0 = {"conv": 3.0, "nr_ldpc": 2.0, "ldpc_same_llr": 2.0}
SPEC_ONLY_OWN = "the runner draws its own channel (this case feeds the same LLRs to every decoder)"
CASES = {
    "conv": {
        "comnum": True,
        "comnum-spec": True,
        "comnum-rust": True,
        "komm": True,
        "opticommpy": "no convolutional code",
        "opticommpy-nojit": "no convolutional code",
        "sionna": True,
    },
    "nr_ldpc": {
        "comnum": True,
        "comnum-spec": "the specification format has no 5G NR LDPC code (NrLdpc: Python and Rust API)",
        "comnum-rust": "the specification format has no 5G NR LDPC code (NrLdpc: Python and Rust API)",
        "komm": "no LDPC code",
        "opticommpy": "no 5G NR LDPC code (its decoder takes a parity-check matrix: see ldpc_same_llr)",
        "opticommpy-nojit": "no 5G NR LDPC code (its decoder takes a parity-check matrix: see ldpc_same_llr)",
        "sionna": True,
    },
    "ldpc_same_llr": {
        "comnum": True,
        "comnum-spec": SPEC_ONLY_OWN,
        "comnum-rust": SPEC_ONLY_OWN,
        "komm": "no LDPC (belief-propagation) decoder",
        "opticommpy": True,
        "opticommpy-nojit": True,
        "sionna": True,
    },
}

PARAMETERS = {
    "conv": "K = 7, (133, 171), k = 1024 + 6 tail bits, soft Viterbi; Eb/N0 = 1, 2, 3, 4 dB",
    "nr_ldpc": "5G NR LDPC, k = 1024, n = 2048 (base graph 2, Z = 104), flooding sum-product, 50 iterations; Eb/N0 = 1, 1.5, 2, 2.5 dB",
    "ldpc_same_llr": "PEG (1024, 512), flooding sum-product, 50 iterations, identical LLRs; Eb/N0 = 1.5, 2, 2.5, 3 dB",
    "frames_per_point": "2000 (quick: 100)",
    "timing": "200 frames (quick: 20) at Eb/N0 = 3 dB (conv) or 2 dB (LDPC); OptiCommPy without numba: 2 frames",
}


def sizes(quick: bool) -> dict:
    return (
        {"frames": 100, "timing": 20, "block": 50}
        if quick
        else {"frames": 2000, "timing": 200, "block": 200}
    )


def n0_of(ebn0: float, rate: float) -> float:
    """N0 of unit-energy BPSK for an Eb/N0 per information bit (complex noise of
    variance N0, N0/2 on the real axis)."""
    return 1.0 / (rate * 10 ** (ebn0 / 10))


# ---------------------------------------------------------------------- the codecs
# A codec gives: encode(F) -> (info bits [F, k], channel input), llrs(F, ebn0) ->
# (reference bits, LLRs in the library's convention), decode(llrs) -> decided bits
# [F, k_out], and the number k_out of bits compared per frame.


def codec_comnum(case: str, seed: int):
    import comnum

    rng = comnum.Rng(seed)
    bpsk = comnum.Constellation("psk", 2)
    if case == "conv":
        code = comnum.ConvolutionalCode(K_CONV, generators=CONV_GEN, constraint=7)
    elif case == "nr_ldpc":
        code = comnum.NrLdpc(
            K_NR, N_NR, max_iterations=ITERATIONS, rule="sum_product", layered=False
        )
    else:
        return codec_same_llr_comnum(seed)
    k, n = code.k, code.n

    def llrs(f: int, ebn0: float):
        b = comnum.bits(f * k, rng)
        x = bpsk.map_bits(code.encode(b))
        n0 = n0_of(ebn0, k / n)
        y = x + rng.complex_noise(x.size, n0)
        return b.reshape(f, k), bpsk.llr(y, n0)

    def decode(llr):
        return code.decode(llr).reshape(-1, k)

    return llrs, decode, k, "float64"


def peg_code():
    import comnum

    code = comnum.LdpcCode.peg(N_PEG, K_PEG, var_degree=3, seed=1)
    code.set_decoder("sum_product", layered=False, max_iterations=ITERATIONS)
    return code


def same_llrs(seed: int):
    """The codewords and LLRs of ldpc_same_llr: the same in every process."""
    import comnum

    code = peg_code()
    rng = comnum.Rng(seed)
    bpsk = comnum.Constellation("psk", 2)

    def llrs(f: int, ebn0: float):
        b = comnum.bits(f * code.k, rng)
        c = code.encode(b)
        n0 = n0_of(ebn0, code.k / code.n)
        y = bpsk.map_bits(c) + rng.complex_noise(c.size, n0)
        return c.reshape(f, code.n), bpsk.llr(y, n0).reshape(f, code.n)  # positive for a 0

    return code, llrs


def codec_same_llr_comnum(seed: int):
    code, llrs = same_llrs(seed)

    def decode(llr):
        return np.stack([(code.decode_llr(row)[0] < 0).astype(np.uint8) for row in llr])

    return llrs, decode, code.n, "float64"


def codec_opticommpy(case: str, seed: int):
    import scipy.sparse as sp
    from optic.comm.fec import decodeLDPC
    from optic.utils import parameters

    code, llrs = same_llrs(seed)
    rows = code.rows
    h = sp.csr_matrix(
        (
            np.ones(sum(map(len, rows)), dtype=np.uint8),
            (np.repeat(np.arange(len(rows)), list(map(len, rows))), np.concatenate(rows)),
        ),
        shape=(len(rows), code.n),
    )
    p = parameters()
    p.H, p.maxIter, p.alg = h, ITERATIONS, "SPA"

    def decode(llr):
        bits, _out, _fail = decodeLDPC(np.ascontiguousarray(llr.T), p)  # columns are codewords
        return bits.T.astype(np.uint8)

    return llrs, decode, code.n, "float32 (decodeLDPC default prec = np.float32)"


def codec_komm(case: str, seed: int):
    import komm

    # (133, 171) with the current input on the most significant bit = (155, 117) in komm
    cc = komm.ConvolutionalCode([[0o155, 0o117]])
    code = komm.TerminatedConvolutionalCode(cc, num_blocks=K_CONV, mode="zero-termination")
    decoder = komm.ViterbiDecoder(code, input_type="soft")
    bpsk, lab = komm.PAMConstellation(2), komm.NaturalLabeling(1)
    rng = np.random.default_rng(seed)
    k, n = code.dimension, code.length

    def llrs(f: int, ebn0: float):
        b = rng.integers(0, 2, (f, k))
        x = bpsk.indices_to_symbols(lab.bits_to_indices(code.encode(b).ravel()))
        sigma2 = n0_of(ebn0, k / n) / 2  # real channel: N0/2 per dimension
        y = komm.GaussianChannel(sigma2, rng=rng).transmit(x)
        return b, lab.marginalize(bpsk.posteriors(y, sigma2))  # log P(0) / P(1)

    def decode(llr):
        return decoder.decode(llr).reshape(-1, k)

    return llrs, decode, k, "float64"


def codec_sionna(case: str, seed: int):
    import torch
    from sionna.phy import config
    from sionna.phy.channel import AWGN
    from sionna.phy.fec.conv import ConvEncoder, ViterbiDecoder
    from sionna.phy.fec.ldpc import LDPC5GDecoder, LDPC5GEncoder, LDPCBPDecoder
    from sionna.phy.mapping import BinarySource, Demapper, Mapper

    config.seed = seed
    if case == "ldpc_same_llr":
        code, same = same_llrs(seed)
        dec = LDPCBPDecoder(peg_pcm(code), num_iter=ITERATIONS, hard_out=True)

        def decode(llr):
            with torch.inference_mode():
                # Sionna's LLRs are logits, log P(1) / P(0): the opposite sign
                llr_dev = torch.from_numpy(-llr.astype(np.float32)).to(common.TORCH_DEVICE)
                return common.to_numpy(dec(llr_dev)).astype(np.uint8)

        return same, decode, code.n, "float32 (config.precision = 'single')"

    if case == "conv":
        enc = ConvEncoder(rate=0.5, constraint_length=7, terminate=True)
        dec = ViterbiDecoder(encoder=enc, method="soft_llr", return_info_bits=True)
        k, n = K_CONV, 2 * (K_CONV + 6)
    else:
        enc = LDPC5GEncoder(K_NR, N_NR)
        dec = LDPC5GDecoder(enc, num_iter=ITERATIONS, hard_out=True, return_infobits=True)
        k, n = K_NR, N_NR
    source, mapper, channel = BinarySource(), Mapper("pam", 1), AWGN()
    demapper = Demapper("app", "pam", 1)

    def llrs(f: int, ebn0: float):
        with torch.inference_mode():
            b = source([f, k])
            no = n0_of(ebn0, k / n)
            y = channel(mapper(enc(b)), no)
            return b, demapper(y, no)  # logits, log P(1) / P(0)

    def decode(llr):
        with torch.inference_mode():
            return dec(llr)

    return llrs, decode, k, "float32 (config.precision = 'single')"


def peg_pcm(code):
    h = np.zeros((len(code.rows), code.n), dtype=np.float32)
    for i, r in enumerate(code.rows):
        h[i, r] = 1
    return h


CODECS = {
    "comnum": codec_comnum,
    "komm": codec_komm,
    "opticommpy": codec_opticommpy,
    "opticommpy-nojit": codec_opticommpy,
    "sionna": codec_sionna,
}


def as_np(a) -> np.ndarray:
    return common.to_numpy(a)


def run_codecs(args, todo: list[str]) -> list[dict]:
    threading = common.setup_torch(args.threads) if args.lib == "sionna" else {}
    sz = sizes(args.quick)
    out = []
    for case in todo:
        llrs, decode, k_out, precision = CODECS[args.lib](case, args.seed)
        res = {"case": case, "precision": precision, **threading}

        def chain(f: int, ebn0: float, llrs=llrs, decode=decode) -> np.ndarray:
            ref, llr = llrs(f, ebn0)
            return np.count_nonzero(
                as_np(decode(llr)).reshape(f, -1) != as_np(ref).reshape(f, -1), axis=1
            )

        nt = 2 if args.lib == "opticommpy-nojit" else sz["timing"]
        e_t = TIMING_EBN0[case]
        t0 = time.perf_counter()
        chain(min(nt, 2), e_t)
        res["first_call_s"] = time.perf_counter() - t0
        if not args.timing_only:
            pts = []
            t0 = time.perf_counter()
            for e in EBN0[case]:
                errs = np.concatenate(
                    [chain(sz["block"], e) for _ in range(sz["frames"] // sz["block"])]
                )
                bound = (
                    float(theory.conv_union_bound(CONV_GEN, 7, K_CONV / (2 * (K_CONV + 6)), e))
                    if case == "conv"
                    else None
                )
                pt = common.frame_ber_point(errs, k_out, None, ebn0_db=e)
                if bound is not None:
                    pt["union_bound"] = bound
                fe = int(np.count_nonzero(errs))
                pt["bler"] = common.rate_point(fe, errs.size)
                if case == "ldpc_same_llr":
                    pt["errors_per_frame"] = errs.astype(int).tolist()
                pts.append(pt)
            res["points"] = pts
            res["sweep_s"] = time.perf_counter() - t0
            res["bits_per_frame"] = k_out
        _, llr = llrs(nt, e_t)
        res["timing"] = common.time_it(lambda llr=llr, decode=decode: decode(llr))
        res["timing"].update(
            units=nt, unit="frame", workload=f"decoder only, {nt} frames at Eb/N0 = {e_t} dB"
        )
        res["timing_chain"] = common.time_it(lambda nt=nt, e_t=e_t, chain=chain: chain(nt, e_t))
        res["timing_chain"].update(
            units=nt, unit="frame", workload=f"whole chain, {nt} frames at Eb/N0 = {e_t} dB"
        )
        out.append(res)
    return out


# ------------------------------------------------------------- specification runner


def spec(ebn0: list[float], frames: int, trials: int, seed: int) -> dict:
    code = {"type": "conv", "k": K_CONV}
    return {
        "name": "bench_conv",
        "seed": seed,
        "batch": trials,
        "grid": {"ebn0_db": ebn0},
        "link": [
            {"type": "fec_encode", "code": code, "blocks": frames // trials},
            {"type": "map", "constellation": {"kind": "psk", "m": 2}},
            {"type": "awgn_ebn0", "ebn0_db": "$ebn0_db"},
            "demap",
            {"type": "fec_decode", "code": code},
        ],
        "metrics": {"report": ["coded_ber", "bler"]},  # coded_ber: decoded information bits
        "stop": {"min_trials": trials, "max_trials": trials},
    }


def spec_points(rows: list[dict]) -> list[dict]:
    pts = []
    for r in rows:
        e = float(r["condition"]["ebn0_db"])
        m = r["metrics"]
        get = lambda d: d["count"] if "count" in d else d["errors"]  # noqa: E731
        # a binomial interval: the runner keeps the totals, not the errors of every frame
        pt = common.rate_point(get(m["coded_ber"]), m["coded_ber"]["total"], None, ebn0_db=e)
        pt["union_bound"] = float(
            theory.conv_union_bound(CONV_GEN, 7, K_CONV / (2 * (K_CONV + 6)), e)
        )
        pt["bler"] = common.rate_point(get(m["bler"]), m["bler"]["total"])
        pts.append(pt)
    return pts


def run_spec_lib(args, todo: list[str], *, rust: bool = False) -> list[dict]:
    """``comnum-spec`` (``comnum.run_spec``), or with ``rust`` ``comnum-rust`` (the binary)."""
    sz = sizes(args.quick)
    trials = 20
    out = []
    for case in todo:
        nt, e_t = sz["timing"], TIMING_EBN0[case]
        workload = f"whole chain, {nt} frames at Eb/N0 = {e_t} dB, {trials} trials"
        res = common.spec_case(
            args,
            {"case": case, "precision": "float64"},
            spec(EBN0[case], sz["frames"], trials, args.seed),
            spec([e_t], nt, trials, args.seed + 1),
            spec_points,
            rust=rust,
            key="timing_chain",
            units=nt,
            unit="frame",
            workload=workload + (" (process wall time)" if rust else ""),
        )
        if not args.timing_only:
            res["bits_per_frame"] = K_CONV
        out.append(res)
    return out


def run_rust_lib(args, todo: list[str]) -> list[dict]:
    return run_spec_lib(args, todo, rust=True)


def postprocess(merged: dict, work) -> dict:
    """ldpc_same_llr: the decoders saw the same LLRs; compare their errors frame by frame
    with those of comnum (same number of bit errors in a frame, same frame errors)."""
    per_lib = {}
    for run in merged["runs"]:
        if run.get("threads") != "1":
            continue
        for case in run.get("cases", []):
            if case["case"] == "ldpc_same_llr" and "points" in case:
                per_lib[run["library"]] = case["points"]
    base = per_lib.get("comnum")
    rows = []
    for lib, pts in per_lib.items():
        if lib == "comnum" or base is None:
            continue
        for p, q in zip(pts, base, strict=True):
            a, b = np.array(p["errors_per_frame"]), np.array(q["errors_per_frame"])
            rows.append(
                {
                    "library": lib,
                    "ebn0_db": p["ebn0_db"],
                    "frames": int(a.size),
                    "same_bit_errors": int(np.count_nonzero(a == b)),
                    "same_frame_outcome": int(np.count_nonzero((a > 0) == (b > 0))),
                }
            )
    return {"same_llr_vs_comnum": rows}


RUNNERS = {
    **{lib: run_codecs for lib in CODECS},
    "comnum-spec": run_spec_lib,
    "comnum-rust": run_rust_lib,
}

if __name__ == "__main__":
    common.main(SCENARIO, CASES, RUNNERS)

Support

Case	comnum (functions)	comnum.run_spec	comnum binary (Rust)	komm	OptiCommPy (numba)	OptiCommPy (no numba)	Sionna
convolutional (133, 171), soft Viterbi	yes	yes	yes	yes	not available: no convolutional code	not available: no convolutional code	yes
5G NR LDPC (1024, 2048)	yes	not available: the specification format has no 5G NR LDPC code (NrLdpc: Python and Rust API)	not available: the specification format has no 5G NR LDPC code (NrLdpc: Python and Rust API)	not available: no LDPC code	not available: no 5G NR LDPC code (its decoder takes a parity-check matrix: see ldpc_same_llr)	not available: no 5G NR LDPC code (its decoder takes a parity-check matrix: see ldpc_same_llr)	yes
LDPC (1024, 512), same LLRs	yes	not available: the runner draws its own channel (this case feeds the same LLRs to every decoder)	not available: the runner draws its own channel (this case feeds the same LLRs to every decoder)	not available: no LDPC (belief-propagation) decoder	yes	yes	yes

convolutional (133, 171), soft Viterbi: error rates

Library	Precision	Against	max abs(z), BER	max abs(z), FER	Agrees	BER interval
comnum (functions)	float64	—	—	—	—	per-frame
comnum.run_spec	float64	comnum	1.35	1.33	yes	binomial
comnum binary (Rust)	float64	comnum	1.35	1.33	yes	binomial
komm	float64	comnum	0.974	1.81	yes	per-frame
Sionna	float32 (config.precision = ‘single’)	comnum	1.41	2.53	yes	per-frame

BER and its 95 % interval, every point (2,048,000 bits per point)

Eb/N0 (dB)	union bound	comnum (functions)	comnum.run_spec	comnum binary (Rust)	komm	Sionna
1	3.05e-1	4.02e-2 [3.91e-2, 4.13e-2]	4.02e-2 [3.99e-2, 4.05e-2]	4.02e-2 [3.99e-2, 4.05e-2]	4.05e-2 [3.94e-2, 4.17e-2]	4.03e-2 [3.91e-2, 4.14e-2]
2	1.65e-2	4.97e-3 [4.64e-3, 5.30e-3]	5.16e-3 [5.06e-3, 5.26e-3]	5.16e-3 [5.06e-3, 5.26e-3]	5.21e-3 [4.86e-3, 5.55e-3]	5.32e-3 [4.97e-3, 5.67e-3]
3	6.27e-4	3.31e-4 [2.59e-4, 4.02e-4]	2.98e-4 [2.76e-4, 3.23e-4]	2.98e-4 [2.76e-4, 3.23e-4]	3.71e-4 [3.04e-4, 4.38e-4]	3.64e-4 [2.94e-4, 4.34e-4]
4	2.02e-5	2.54e-5 [8.46e-6, 4.23e-5]	1.32e-5 [9.06e-6, 1.92e-5]	1.32e-5 [9.06e-6, 1.92e-5]	2.00e-5 [3.66e-6, 3.64e-5]	3.08e-5 [9.04e-6, 5.25e-5]

Frame error rate and its 95 % interval (2,000 frames per point)

Eb/N0 (dB)	comnum (functions)	comnum.run_spec	comnum binary (Rust)	komm	Sionna
1	9.73e-1 [9.66e-1, 9.80e-1]	9.77e-1 [9.69e-1, 9.83e-1]	9.77e-1 [9.69e-1, 9.83e-1]	9.82e-1 [9.75e-1, 9.87e-1]	9.85e-1 [9.79e-1, 9.89e-1]
2	5.05e-1 [4.84e-1, 5.27e-1]	4.84e-1 [4.63e-1, 5.06e-1]	4.84e-1 [4.63e-1, 5.06e-1]	5.01e-1 [4.79e-1, 5.23e-1]	5.13e-1 [4.91e-1, 5.35e-1]
3	5.95e-2 [5.00e-2, 7.07e-2]	6.05e-2 [5.09e-2, 7.18e-2]	6.05e-2 [5.09e-2, 7.18e-2]	7.15e-2 [6.10e-2, 8.36e-2]	6.70e-2 [5.69e-2, 7.88e-2]
4	6.00e-3 [3.44e-3, 1.05e-2]	5.00e-3 [2.72e-3, 9.18e-3]	5.00e-3 [2.72e-3, 9.18e-3]	5.00e-3 [2.72e-3, 9.18e-3]	5.50e-3 [3.07e-3, 9.82e-3]

5G NR LDPC (1024, 2048): error rates

Library	Precision	Against	max abs(z), BER	max abs(z), FER	Agrees	BER interval
comnum (functions)	float64	—	—	—	—	per-frame
Sionna	float32 (config.precision = ‘single’)	comnum	0.893	1.21	yes	per-frame

BER and its 95 % interval, every point (2,048,000 bits per point)

Eb/N0 (dB)	comnum (functions)	Sionna
1	8.79e-3 [7.46e-3, 1.01e-2]	9.30e-3 [7.94e-3, 1.07e-2]
1.5	3.47e-5 [0, 1.03e-4]	9.38e-5 [0, 2.04e-4]
2	0 [0, 1.46e-6]	0 [0, 1.46e-6]
2.5	0 [0, 1.46e-6]	0 [0, 1.46e-6]

Frame error rate and its 95 % interval (2,000 frames per point)

Eb/N0 (dB)	comnum (functions)	Sionna
1	9.40e-2 [8.20e-2, 1.08e-1]	1.05e-1 [9.28e-2, 1.20e-1]
1.5	5.00e-4 [8.83e-5, 2.83e-3]	1.50e-3 [5.10e-4, 4.40e-3]
2	0 [0, 1.92e-3]	0 [0, 1.92e-3]
2.5	0 [0, 1.92e-3]	0 [0, 1.92e-3]

LDPC (1024, 512), same LLRs: error rates

Library	Precision	Against	max abs(z), BER	max abs(z), FER	Agrees	BER interval
comnum (functions)	float64	—	—	—	—	per-frame
OptiCommPy (numba)	float32 (decodeLDPC default prec = np.float32)	comnum	0.076	0	yes	per-frame
Sionna	float32 (config.precision = ‘single’)	comnum	0.0906	0.0381	yes	per-frame

BER and its 95 % interval, every point (2,048,000 bits per point)

Eb/N0 (dB)	comnum (functions)	OptiCommPy (numba)	Sionna
1.5	1.53e-2 [1.39e-2, 1.66e-2]	1.53e-2 [1.40e-2, 1.67e-2]	1.54e-2 [1.40e-2, 1.67e-2]
2	1.37e-3 [9.56e-4, 1.78e-3]	1.37e-3 [9.55e-4, 1.78e-3]	1.37e-3 [9.57e-4, 1.78e-3]
2.5	0 [0, 1.46e-6]	0 [0, 1.46e-6]	0 [0, 1.46e-6]
3	0 [0, 1.46e-6]	0 [0, 1.46e-6]	0 [0, 1.46e-6]

Frame error rate and its 95 % interval (2,000 frames per point)

Eb/N0 (dB)	comnum (functions)	OptiCommPy (numba)	Sionna
1.5	2.21e-1 [2.03e-1, 2.40e-1]	2.21e-1 [2.03e-1, 2.40e-1]	2.21e-1 [2.04e-1, 2.40e-1]
2	2.30e-2 [1.73e-2, 3.05e-2]	2.30e-2 [1.73e-2, 3.05e-2]	2.30e-2 [1.73e-2, 3.05e-2]
2.5	0 [0, 1.92e-3]	0 [0, 1.92e-3]	0 [0, 1.92e-3]
3	0 [0, 1.92e-3]	0 [0, 1.92e-3]	0 [0, 1.92e-3]

Same LLRs: the decisions frame by frame

The decoders received identical LLRs. Over all the points: the frames where a library makes as many bit errors as comnum, and the frames it decodes (or fails to decode) as comnum does.

Library	Points	Frames	Same bit errors	Same frame outcome
OptiCommPy (numba)	4	8000	7996 (99.95 %)	8000 (100.00 %)
Sionna	4	8000	7980 (99.75 %)	7999 (99.99 %)

convolutional (133, 171), soft Viterbi: decoder time

Median of 5 runs, ms per frame; workload: decoder only, 200 frames at Eb/N0 = 3.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	Relative	First call
comnum (functions)	float64	0.276	0.277	1.00	0.000821 s
komm	float64	0.566	0.564	2.05	0.0188 s
Sionna	float32 (config.precision = ‘single’)	1.13	0.916	4.10	0.0305 s

5G NR LDPC (1024, 2048): decoder time

Median of 5 runs, ms per frame; workload: decoder only, 200 frames at Eb/N0 = 2.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	Relative	First call
comnum (functions)	float64	1.70	1.68	1.00	0.00361 s
Sionna	float32 (config.precision = ‘single’)	7.74	4.23	4.54	0.0213 s

LDPC (1024, 512), same LLRs: decoder time

Median of 5 runs, ms per frame; workload: decoder only, 200 frames at Eb/N0 = 2.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	Relative	First call
comnum (functions)	float64	0.316	0.323	1.00	0.000748 s
OptiCommPy (numba)	float32 (decodeLDPC default prec = np.float32)	1.47	4.53	4.66	1.47 s
OptiCommPy (no numba)	float32 (decodeLDPC default prec = np.float32)	42.4	42.0	134	0.082 s
Sionna	float32 (config.precision = ‘single’)	2.29	1.26	7.25	0.00648 s

convolutional (133, 171), soft Viterbi: whole-chain time

Median of 5 runs, ms per frame; workload: whole chain, 200 frames at Eb/N0 = 3.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	Relative	First call
comnum (functions)	float64	0.325	0.325	1.00	0.000821 s
comnum.run_spec	float64	0.324	0.0513	0.997	0.0651 s
comnum binary (Rust)	float64	0.336	0.0688	1.03	0.0696 s
komm	float64	0.687	0.687	2.11	0.0188 s
Sionna	float32 (config.precision = ‘single’)	1.23	1.01	3.79	0.0305 s

5G NR LDPC (1024, 2048): whole-chain time

Median of 5 runs, ms per frame; workload: whole chain, 200 frames at Eb/N0 = 2.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	Relative	First call
comnum (functions)	float64	1.75	1.74	1.00	0.00361 s
Sionna	float32 (config.precision = ‘single’)	7.84	4.28	4.48	0.0213 s

LDPC (1024, 512), same LLRs: whole-chain time

Median of 5 runs, ms per frame; workload: whole chain, 200 frames at Eb/N0 = 2.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	Relative	First call
comnum (functions)	float64	0.356	0.363	1.00	0.000748 s
OptiCommPy (numba)	float32 (decodeLDPC default prec = np.float32)	1.44	4.31	4.04	1.47 s
OptiCommPy (no numba)	float32 (decodeLDPC default prec = np.float32)	37.0	36.5	104	0.082 s
Sionna	float32 (config.precision = ‘single’)	2.35	1.32	6.60	0.00648 s

The commands of every run

OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 VECLIB_MAXIMUM_THREADS=1 NUMBA_NUM_THREADS=1 RAYON_NUM_THREADS=1 python s4_coding.py --lib comnum --threads 1 --work results/_work --out results/_work/s4_coding_comnum_1.json
 python s4_coding.py --lib comnum --threads all --work results/_work --out results/_work/s4_coding_comnum_all.json --timing-only
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 VECLIB_MAXIMUM_THREADS=1 NUMBA_NUM_THREADS=1 RAYON_NUM_THREADS=1 python s4_coding.py --lib comnum-spec --threads 1 --work results/_work --out results/_work/s4_coding_comnum-spec_1.json
 python s4_coding.py --lib comnum-spec --threads all --work results/_work --out results/_work/s4_coding_comnum-spec_all.json --timing-only
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 VECLIB_MAXIMUM_THREADS=1 NUMBA_NUM_THREADS=1 RAYON_NUM_THREADS=1 python s4_coding.py --lib comnum-rust --threads 1 --work results/_work --out results/_work/s4_coding_comnum-rust_1.json
 python s4_coding.py --lib comnum-rust --threads all --work results/_work --out results/_work/s4_coding_comnum-rust_all.json --timing-only
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 VECLIB_MAXIMUM_THREADS=1 NUMBA_NUM_THREADS=1 RAYON_NUM_THREADS=1 python s4_coding.py --lib komm --threads 1 --work results/_work --out results/_work/s4_coding_komm_1.json
 python s4_coding.py --lib komm --threads all --work results/_work --out results/_work/s4_coding_komm_all.json --timing-only
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 VECLIB_MAXIMUM_THREADS=1 NUMBA_NUM_THREADS=1 RAYON_NUM_THREADS=1 python s4_coding.py --lib opticommpy --threads 1 --work results/_work --out results/_work/s4_coding_opticommpy_1.json
 python s4_coding.py --lib opticommpy --threads all --work results/_work --out results/_work/s4_coding_opticommpy_all.json --timing-only
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 VECLIB_MAXIMUM_THREADS=1 NUMBA_NUM_THREADS=1 RAYON_NUM_THREADS=1 NUMBA_DISABLE_JIT=1 python s4_coding.py --lib opticommpy-nojit --threads 1 --work results/_work --out results/_work/s4_coding_opticommpy-nojit_1.json --timing-only
NUMBA_DISABLE_JIT=1 python s4_coding.py --lib opticommpy-nojit --threads all --work results/_work --out results/_work/s4_coding_opticommpy-nojit_all.json --timing-only
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 VECLIB_MAXIMUM_THREADS=1 NUMBA_NUM_THREADS=1 RAYON_NUM_THREADS=1 python s4_coding.py --lib sionna --threads 1 --work results/_work --out results/_work/s4_coding_sionna_1.json
 python s4_coding.py --lib sionna --threads all --work results/_work --out results/_work/s4_coding_sionna_all.json --timing-only

Split-step Fourier propagation, 16-QAM, 32 GBd, 10 x 80 km#

Scenario 5: single-polarisation split-step Fourier propagation of 32 GBd 16-QAM over 10 x 80 km of standard single-mode fibre, a fixed step of 1 km (80 steps per span), noiseless amplification at the end of every span (gain e^{alpha L}).

Parameters:

fibre: alpha = 0.2 dB/km, D = 17 ps/nm/km, gamma = 1.3 /W/km, fc = 193.1 THz
link: 10 x 80 km, ideal (noiseless) amplification after every span
step: 1 km (800 steps); reference: comnum at 0.05 km (16000 steps)
signal: 16-QAM, 32 GBd, RRC roll-off 0.1, 4 samples per symbol, 4 dBm; 2^14 symbols (quick: 2^11)
timing: one propagation over the 10 spans

Script: benchmarks/libraries/s5_ssfm.py.

The script

"""Scenario 5: single-polarisation split-step Fourier propagation of 32 GBd 16-QAM over
10 x 80 km of standard single-mode fibre, a fixed step of 1 km (80 steps per span),
noiseless amplification at the end of every span (gain e^{alpha L}).

The input field is the same for every library: 16-QAM symbols of comnum (seed 1),
shaped by a root-raised cosine (roll-off 0.1, 4 samples per symbol, 128 GS/s) and
scaled to 4 dBm; it is drawn again, identically, by every process. Every library
propagates it with its own split-step solver (symmetric steps, nonlinear phase over
the whole step in all three); run_all.py compares the output fields (NMSE) with that
of comnum in double precision and with a step-converged reference (comnum, 0.05 km).

Sionna's equation is the complex conjugate of the one of comnum and OptiCommPy
(dE/dz = ... + j beta_2/2 d^2E/dt^2 - j gamma |E|^2 E): its field is compared in their
convention, conj(sionna(conj(x))), and the NMSE of the plain output is reported too.
Sionna propagates one span per call (SSFM) and amplifies with its EDFA of noise figure
0 (no noise). OptiCommPy's ssfm is NumPy only (no numba kernel): its two variants run
the same code. In the mode gpu, OptiCommPy runs its CuPy solver
(optic.models.modelsGPU.ssfm: the same steps, every array on the device; the input is
copied to the device and the output back to the host inside the timed call) in
complex128 and in complex64. Timing: one propagation over the 10 spans, median of 5
runs after a warm-up run.

    python s5_ssfm.py --lib opticommpy --threads 1 [--quick] --out results/_work/s5.json
"""

from __future__ import annotations

import math
import time
from pathlib import Path

import common
import numpy as np

SCENARIO = "s5_ssfm"
TITLE = "Split-step Fourier propagation, 16-QAM, 32 GBd, 10 x 80 km"
CASES = {
    "ssfm": {
        "comnum": True,
        "komm": "no fibre model",
        "opticommpy": True,
        "opticommpy-nojit": True,
        "sionna": True,
    }
}
# The libraries with a GPU path in this scenario (run_all.py: the mode gpu).
GPU_LIBS = {"sionna", "opticommpy"}
ALPHA_DB_KM, D_PS_NM_KM, GAMMA, FC = 0.2, 17.0, 1.3, 193.1e12
SPAN_KM, N_SPANS, STEP_KM = 80.0, 10, 1.0
REF_STEP_KM = 0.05
SYMBOL_RATE, SPS = 32e9, 4
FS = SYMBOL_RATE * SPS
POWER_DBM = 4.0

PARAMETERS = {
    "fibre": "alpha = 0.2 dB/km, D = 17 ps/nm/km, gamma = 1.3 /W/km, fc = 193.1 THz",
    "link": "10 x 80 km, ideal (noiseless) amplification after every span",
    "step": "1 km (800 steps); reference: comnum at 0.05 km (16000 steps)",
    "signal": "16-QAM, 32 GBd, RRC roll-off 0.1, 4 samples per symbol, 4 dBm; 2^14 symbols (quick: 2^11)",
    "timing": "one propagation over the 10 spans",
}


def n_symbols(quick: bool) -> int:
    return 2**11 if quick else 2**14


def input_field(quick: bool) -> np.ndarray:
    import comnum

    rng = comnum.Rng(1)
    c = comnum.Constellation("qam", 16).normalized()
    _, s = comnum.symbols(c, n_symbols(quick), rng)
    h = comnum.pulse_shape("rrc", SPS, n_taps=1025, rolloff=0.1)
    x = comnum.pnorm(comnum.fir_filter(h, comnum.upsample(s, SPS)))
    return x * math.sqrt(comnum.dbm2w(POWER_DBM))


def beta2_ps2_km() -> float:
    c = 299_792_458.0
    lam = c / FC
    return -(D_PS_NM_KM * 1e-6) * lam**2 / (2 * math.pi * c) * 1e24 * 1e3  # s^2/m -> ps^2/km


def prop_comnum(precision: str, step_km: float = STEP_KM):
    import comnum

    f = comnum.FiberParams(
        alpha_db_km=ALPHA_DB_KM,
        d_ps_nm_km=D_PS_NM_KM,
        gamma_w_km=GAMMA,
        span_km=SPAN_KM,
        n_spans=N_SPANS,
        fc_hz=FC,
    )
    link = comnum.Ssfm(f, step_km=step_km, amplification="ideal")
    return lambda x: link.propagate(x, FS, precision=precision)


def prop_opticommpy(gpu: bool = False, prec=np.complex128):
    """OptiCommPy's split-step: the NumPy solver, or its CuPy copy on the GPU
    (``optic.models.modelsGPU.ssfm``, which takes and returns NumPy arrays: the
    transfers to and from the device are inside the timed call)."""
    if gpu:
        from optic.models.modelsGPU import ssfm
    else:
        from optic.models.channels import ssfm
    from optic.utils import parameters

    p = parameters()
    p.Ltotal, p.Lspan, p.hz = SPAN_KM * N_SPANS, SPAN_KM, STEP_KM
    p.alpha, p.D, p.gamma, p.Fc, p.Fs = ALPHA_DB_KM, D_PS_NM_KM, GAMMA, FC, FS
    p.amp, p.prgsBar, p.prec = "ideal", False, prec
    return lambda x: ssfm(x, p)


def prop_sionna(precision: str):
    import torch
    from sionna.phy.channel.optical import EDFA, SSFM

    alpha = ALPHA_DB_KM / (10 * math.log10(math.e))  # 1/km
    span = SSFM(
        alpha=alpha,
        beta_2=beta2_ps2_km(),
        f_c=FC,
        gamma=GAMMA,
        length=SPAN_KM,
        n_ssfm=round(SPAN_KM / STEP_KM),
        sample_duration=1e12 / FS,  # ps
        t_norm=1e-12,
        with_amplification=False,
        precision=precision,
    )
    amp = EDFA(g=math.exp(alpha * SPAN_KM), f=0.0, f_c=FC, dt=1 / FS, precision=precision)
    cdtype = torch.complex64 if precision == "single" else torch.complex128

    def run(x: np.ndarray) -> np.ndarray:
        with torch.inference_mode():
            u = torch.from_numpy(x).to(device=common.TORCH_DEVICE, dtype=cdtype)
            for _ in range(N_SPANS):
                u = amp(span(u))
            return common.to_numpy(u)

    return run


def run_lib(args, todo: list[str]) -> list[dict]:
    if args.lib == "sionna":
        threading = common.setup_torch(args.threads)
    elif args.lib == "opticommpy" and args.threads == "gpu":
        threading = common.setup_cupy()
    else:
        threading = {}
    x = input_field(args.quick)
    work = Path(args.work)
    work.mkdir(parents=True, exist_ok=True)
    if args.lib == "comnum":
        variants = {"f64": (prop_comnum("f64"), "float64"), "f32": (prop_comnum("f32"), "float32")}
    elif args.lib == "sionna":
        variants = {
            "single": (prop_sionna("single"), "float32 (precision = 'single', the default)"),
            "double": (prop_sionna("double"), "float64 (precision = 'double')"),
        }
    elif args.lib == "opticommpy" and args.threads == "gpu":
        import cupy as cp

        variants = {
            "f64": (prop_opticommpy(True, cp.complex128), "float64 (prec = cp.complex128)"),
            "f32": (prop_opticommpy(True, cp.complex64), "float32 (prec = cp.complex64)"),
        }
    else:
        variants = {"f64": (prop_opticommpy(), "float64 (prec = np.complex128)")}
    out = []
    for name, (prop, precision) in variants.items():
        res = {
            "case": "ssfm",
            "variant": name,
            "precision": precision,
            "samples": int(x.size),
            **threading,
        }
        if args.lib == "sionna":
            fwd = lambda prop=prop: np.conj(prop(np.conj(x)))  # noqa: E731
        else:
            fwd = lambda prop=prop: prop(x)  # noqa: E731
        t0 = time.perf_counter()
        y = fwd()
        res["first_call_s"] = time.perf_counter() - t0
        if not args.timing_only:
            np.save(work / f"{SCENARIO}_{args.lib}_{name}.npy", np.asarray(y, dtype=np.complex128))
            if args.lib == "sionna":
                np.save(
                    work / f"{SCENARIO}_{args.lib}_{name}_plain.npy",
                    np.asarray(prop(x), dtype=np.complex128),
                )
        res["timing"] = common.time_it(fwd)
        res["timing"].update(
            units=1,
            unit="propagation",
            workload=f"{x.size} samples, {N_SPANS} x {SPAN_KM:g} km, {STEP_KM:g} km steps",
        )
        out.append(res)
    if args.lib == "comnum" and not args.timing_only:
        t0 = time.perf_counter()
        np.save(work / f"{SCENARIO}_reference.npy", prop_comnum("f64", REF_STEP_KM)(x))
        out.append(
            {
                "case": "ssfm",
                "variant": "reference",
                "precision": "float64",
                "step_km": REF_STEP_KM,
                "wall_s": time.perf_counter() - t0,
            }
        )
    return out


def nmse(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.sum(np.abs(a - b) ** 2) / np.sum(np.abs(b) ** 2))


def postprocess(merged: dict, work: Path) -> dict:
    """The NMSE of every output field against comnum (f64, 1 km) and the reference."""
    fields = {}
    for path in sorted(work.glob(f"{SCENARIO}_*.npy")):
        fields[path.stem.removeprefix(f"{SCENARIO}_")] = np.load(path)
    base, ref = fields.get("comnum_f64"), fields.get("reference")
    rows = []
    for name, y in fields.items():
        if name == "reference":
            continue
        rows.append(
            {
                "field": name,
                "nmse_vs_comnum_f64": nmse(y, base) if base is not None else None,
                "nmse_vs_reference": nmse(y, ref) if ref is not None else None,
            }
        )
    return {"nmse": rows, "reference_step_km": REF_STEP_KM}


RUNNERS = {"comnum": run_lib, "opticommpy": run_lib, "opticommpy-nojit": run_lib, "sionna": run_lib}

if __name__ == "__main__":
    common.main(SCENARIO, CASES, RUNNERS)

Support

Case	comnum (functions)	komm	OptiCommPy (numba)	OptiCommPy (no numba)	Sionna
split-step Fourier	yes	not available: no fibre model	yes	yes	yes

Output fields

NMSE of the output field, sum |a - b|^2 / sum |b|^2, against comnum in float64 at the same step and against the step-converged reference (comnum, 0.05 km). The NMSE against the reference is the error of the 1 km step itself.

Field	NMSE vs comnum (float64, 1 km)	NMSE vs reference
comnum, float32	6.35e-9	6.14e-8
comnum, float64	0	3.69e-8
OptiCommPy, float64	5.72e-26	3.69e-8
Sionna, float64	4.83e-26	3.69e-8
Sionna, float64, its own convention (not conjugated)	2.03e0	2.03e0
Sionna, float32	1.02e-8	7.34e-8
Sionna, float32, its own convention (not conjugated)	2.03e0	2.03e0

split-step Fourier: time

Median of 5 runs, s per propagation; workload: 65536 samples, 10 x 80 km, 1 km steps. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	Relative	First call
comnum (functions), f64	float64	0.556	0.554	1.00	0.561 s
comnum (functions), f32	float32	0.365	0.364	0.656	0.365 s
OptiCommPy (numba), f64	float64 (prec = np.complex128)	1.28	1.28	2.31	1.29 s
OptiCommPy (no numba), f64	float64 (prec = np.complex128)	1.28	1.28	2.30	1.29 s
Sionna, single	float32 (precision = ‘single’, the default)	1.00	0.872	1.80	0.999 s
Sionna, double	float64 (precision = ‘double’)	1.30	1.11	2.34	1.29 s

On another machine: h100 (NVIDIA H100 80GB HBM3)#

The same scripts, run on a second machine (run_all.py --results results/h100): Sionna (and, for the split-step, OptiCommPy) on its GPU in the column GPU, and every library on its CPU in the other columns. The Relative column stays the 1-thread time over that of comnum (functions) on 1 thread of this machine.

This machine is a RunPod Secure Cloud pod with one NVIDIA H100 SXM (80 GB HBM3), the PyTorch template (torch 2.13, CUDA 13.0): a container with a CPU quota on a shared host.

Machine and versions#

CPU	Intel(R) Xeon(R) Platinum 8480+
Cores	224 logical; 24 available to the process (a container’s CPU quota: the thread pools of the mode all threads are sized to it)
Memory	2015.3 GB
GPU	NVIDIA H100 80GB HBM3, 81559 MiB, driver 595.91.07
System	Ubuntu 24.04.5 LTS
Python	3.12.3
Rust	—
comnum commit	`45a7c78f80fc`
Generated	2026-11-06T17:48:24+00:00
comnum	0.3.1
komm	not installed
OptiCommPy	0.11.0
sionna	2.2.0
torch	2.13.0+cu130
numba	0.68.1
numpy	2.5.2
scipy	1.18.1

Uncoded QPSK and 16-QAM over AWGN#

Scenario 1: uncoded Gray QPSK and 16-QAM over AWGN, BER against the exact closed form.

Parameters:

Eb/N0 (dB): 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
bits per point: 10^7 (quick: 2·10^5), in blocks of 10^6 (quick: 10^5)
timing: one block of 10^6 bits (quick: 2·10^5) at Eb/N0 = 6 dB
theory: exact BER of Gray square QAM, Cho & Yoon, IEEE Trans. Commun. 50(7), 2002

Script: benchmarks/libraries/s1_awgn.py.

The script

"""Scenario 1: uncoded Gray QPSK and 16-QAM over AWGN, BER against the exact closed form.

Every library draws its own bits and noise, maps with its own Gray mapper, adds the
noise of its own channel for the Eb/N0 of the point and decides by minimum distance
(Sionna: hard decisions of max-log LLRs, the same decisions). The BER is counted on
the same number of bits per point for every library, in blocks of the same size.
Timing: one block of ``TIMING_BITS`` bits at Eb/N0 = 6 dB (bits, mapping, noise,
decisions, error count), median of 5 runs after a warm-up run.

    python s1_awgn.py --lib comnum --threads 1 [--quick] --out results/_work/s1.json
"""

from __future__ import annotations

import math
import time

import common
import numpy as np
import theory

SCENARIO = "s1_awgn"
TITLE = "Uncoded QPSK and 16-QAM over AWGN"
ORDERS = {"qpsk": 4, "16qam": 16}
ALL = {lib: True for lib in common.LIBRARIES}
CASES = {"qpsk": ALL, "16qam": ALL}
EBN0 = [float(e) for e in range(11)]
TIMING_EBN0 = 6.0
TIMING_TRIALS = 40  # trials of the specification runner per timing run


def sizes(quick: bool) -> dict:
    if quick:
        return {"bits_per_point": 200_000, "block": 100_000, "timing_bits": 200_000, "trials": 10}
    return {
        "bits_per_point": 10_000_000,
        "block": 1_000_000,
        "timing_bits": 1_000_000,
        "trials": 100,
    }


PARAMETERS = {
    "ebn0_db": EBN0,
    "bits_per_point": "10^7 (quick: 2·10^5), in blocks of 10^6 (quick: 10^5)",
    "timing": "one block of 10^6 bits (quick: 2·10^5) at Eb/N0 = 6 dB",
    "theory": "exact BER of Gray square QAM, Cho & Yoon, IEEE Trans. Commun. 50(7), 2002",
}


# ---------------------------------------------------------------------------- chains
# chain(n_bits, ebn0_db) -> bit errors, for one library and one constellation


def chain_comnum(m: int, seed: int):
    import comnum

    c = comnum.Constellation("qam", m).normalized()
    k = int(math.log2(m))
    rng = comnum.Rng(seed)

    def run(n: int, ebn0: float) -> int:
        b = comnum.bits(n, rng)
        y = comnum.awgn_ebn0(c.map_bits(b), ebn0, k, rng)
        return int(np.count_nonzero(c.demap_bits(y) != b))

    return run, "float64"


def chain_komm(m: int, seed: int):
    import komm

    const = komm.QAMConstellation(m)
    lab = komm.ReflectedRectangularLabeling(int(math.log2(m)))
    k = int(math.log2(m))
    es = const.mean_energy()
    rng = np.random.default_rng(seed)

    def run(n: int, ebn0: float) -> int:
        b = rng.integers(0, 2, n)
        x = const.indices_to_symbols(lab.bits_to_indices(b))
        n0 = es / (k * 10 ** (ebn0 / 10))
        y = komm.GaussianChannel(n0, rng=rng).transmit(x)
        bh = lab.indices_to_bits(const.closest_indices(y))
        return int(np.count_nonzero(bh.ravel() != b))

    return run, "float64"


def chain_opticommpy(m: int, seed: int):
    from optic.comm.modulation import demodulateGray, modulateGray
    from optic.models.channels import awgn
    from optic.utils import parameters

    k = int(math.log2(m))
    rng = np.random.default_rng(seed)
    state = {"block": 0}

    def run(n: int, ebn0: float) -> int:
        b = rng.integers(0, 2, n)
        x = modulateGray(b, m, "qam")
        p = parameters()
        p.snr = ebn0 + 10 * math.log10(k)  # Es/N0 (Fs = B = 1)
        p.seed = seed * 1_000_003 + state["block"]  # awgn re-seeds its generator per call
        state["block"] += 1
        y = awgn(x, p)
        bh = demodulateGray(y, m, "qam")
        return int(np.count_nonzero(bh != b))

    return run, "complex64 symbols (its Gray constellation), float64 noise"


def chain_sionna(m: int, seed: int):
    import torch
    from sionna.phy import config
    from sionna.phy.channel import AWGN
    from sionna.phy.mapping import BinarySource, Demapper, Mapper
    from sionna.phy.utils import ebnodb2no

    config.seed = seed
    k = int(math.log2(m))
    source, mapper, channel = BinarySource(), Mapper("qam", k), AWGN()
    demapper = Demapper("maxlog", "qam", k, hard_out=True)

    def run(n: int, ebn0: float) -> int:
        with torch.inference_mode():
            b = source([1, n])
            no = ebnodb2no(ebn0, k, 1.0)
            y = channel(mapper(b), no)
            bh = demapper(y, no)
            return int((bh != b).sum())

    return run, "float32 (config.precision = 'single')"


CHAINS = {
    "comnum": chain_comnum,
    "komm": chain_komm,
    "opticommpy": chain_opticommpy,
    "opticommpy-nojit": chain_opticommpy,
    "sionna": chain_sionna,
}


def run_chains(args, todo: list[str]) -> list[dict]:
    if args.lib == "sionna":
        threading = common.setup_torch(args.threads)
    else:
        threading = {}
    sz = sizes(args.quick)
    out = []
    for case in todo:
        m = ORDERS[case]
        run, precision = CHAINS[args.lib](m, args.seed)
        res = {"case": case, "precision": precision, **threading}
        n = sz["timing_bits"]
        if args.lib == "opticommpy-nojit":
            n = min(n, 100_000)  # plain-Python kernels: a smaller block, the time is per bit
        t0 = time.perf_counter()
        run(n, TIMING_EBN0)  # the first call: JIT compilation (numba), lazy initialisation
        res["first_call_s"] = time.perf_counter() - t0
        if not args.timing_only:
            pts = []
            t0 = time.perf_counter()
            for e in EBN0:
                errors = 0
                for _ in range(sz["bits_per_point"] // sz["block"]):
                    errors += run(sz["block"], e)
                pts.append(
                    common.rate_point(
                        errors, sz["bits_per_point"], float(theory.ber_qam_awgn(m, e)), ebn0_db=e
                    )
                )
            res["points"] = pts
            res["sweep_s"] = time.perf_counter() - t0
        last = {}
        res["timing"] = common.time_it(
            lambda n=n, run=run, last=last: last.__setitem__("e", run(n, TIMING_EBN0))
        )
        res["timing"].update(units=n, unit="bit", workload=f"{n} bits at Eb/N0 = {TIMING_EBN0} dB")
        res["timing_ber"] = last["e"] / n
        out.append(res)
    return out


# ------------------------------------------------------------- specification runner


def spec(m: int, ebn0: list[float], bits: int, trials: int, seed: int) -> dict:
    k = int(math.log2(m))
    return {
        "name": f"bench_awgn_{m}qam",
        "seed": seed,
        "batch": trials,
        "grid": {"ebn0_db": ebn0},
        "link": [
            {
                "type": "source",
                "constellation": {"kind": "qam", "m": m},
                "n_symbols": bits // (k * trials),
            },
            {"type": "awgn_ebn0", "ebn0_db": "$ebn0_db"},
        ],
        "metrics": {"report": ["ber"], "scale": "none"},
        "stop": {"min_trials": trials, "max_trials": trials},
    }


def spec_points(results: list[dict], m: int) -> list[dict]:
    pts = []
    for r in results:
        e = float(r["condition"]["ebn0_db"])
        ber = r["metrics"]["ber"]
        errors = ber["count"] if "count" in ber else ber["errors"]  # run_spec / the binary's report
        pts.append(
            common.rate_point(errors, ber["total"], float(theory.ber_qam_awgn(m, e)), ebn0_db=e)
        )
    return pts


def run_spec_lib(args, todo: list[str], *, rust: bool = False) -> list[dict]:
    """``comnum-spec`` (``comnum.run_spec``), or with ``rust`` ``comnum-rust`` (the binary)."""
    sz = sizes(args.quick)
    n = sz["timing_bits"]
    workload = f"{n} bits at Eb/N0 = {TIMING_EBN0} dB, {TIMING_TRIALS} trials"
    if rust:
        workload += " (process wall time, start-up included)"
    out = []
    for case in todo:
        m = ORDERS[case]
        out.append(
            common.spec_case(
                args,
                {"case": case, "precision": "float64"},
                spec(m, EBN0, sz["bits_per_point"], sz["trials"], args.seed),
                spec(m, [TIMING_EBN0], n, TIMING_TRIALS, args.seed + 1),
                lambda rows, m=m: spec_points(rows, m),
                rust=rust,
                units=n,
                unit="bit",
                workload=workload,
            )
        )
    return out


def run_rust_lib(args, todo: list[str]) -> list[dict]:
    return run_spec_lib(args, todo, rust=True)


RUNNERS = {
    **{lib: run_chains for lib in CHAINS},
    "comnum-spec": run_spec_lib,
    "comnum-rust": run_rust_lib,
}

if __name__ == "__main__":
    common.main(SCENARIO, CASES, RUNNERS)

Support

Case	comnum (functions)	comnum.run_spec	OptiCommPy (numba)	Sionna
QPSK	yes	yes	yes	yes
16-QAM	yes	yes	yes	yes

QPSK: error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	theory	1.90	yes	binomial
comnum.run_spec	float64	theory	2.53	yes	binomial
OptiCommPy (numba)	complex64 symbols (its Gray constellation), float64 noise	theory	3.31	yes	binomial
Sionna	float32 (config.precision = ‘single’)	theory	2.13	yes	binomial

BER and its 95 % interval, every point (10,000,000 bits per point)

Eb/N0 (dB)	theory	comnum (functions)	comnum.run_spec	OptiCommPy (numba)	Sionna
0	7.86e-2	7.86e-2 [7.84e-2, 7.88e-2]	7.87e-2 [7.85e-2, 7.89e-2]	7.86e-2 [7.84e-2, 7.88e-2]	7.86e-2 [7.84e-2, 7.88e-2]
1	5.63e-2	5.63e-2 [5.62e-2, 5.65e-2]	5.63e-2 [5.61e-2, 5.64e-2]	5.63e-2 [5.61e-2, 5.64e-2]	5.61e-2 [5.60e-2, 5.63e-2]
2	3.75e-2	3.75e-2 [3.74e-2, 3.76e-2]	3.74e-2 [3.73e-2, 3.76e-2]	3.75e-2 [3.73e-2, 3.76e-2]	3.76e-2 [3.75e-2, 3.77e-2]
3	2.29e-2	2.28e-2 [2.27e-2, 2.29e-2]	2.30e-2 [2.29e-2, 2.31e-2]	2.29e-2 [2.28e-2, 2.30e-2]	2.29e-2 [2.28e-2, 2.30e-2]
4	1.25e-2	1.25e-2 [1.25e-2, 1.26e-2]	1.25e-2 [1.24e-2, 1.26e-2]	1.25e-2 [1.24e-2, 1.26e-2]	1.25e-2 [1.25e-2, 1.26e-2]
5	5.95e-3	5.95e-3 [5.90e-3, 6.00e-3]	5.96e-3 [5.92e-3, 6.01e-3]	5.93e-3 [5.89e-3, 5.98e-3]	5.93e-3 [5.88e-3, 5.98e-3]
6	2.39e-3	2.39e-3 [2.36e-3, 2.42e-3]	2.41e-3 [2.38e-3, 2.44e-3]	2.43e-3 [2.40e-3, 2.46e-3]	2.39e-3 [2.36e-3, 2.42e-3]
7	7.73e-4	7.56e-4 [7.39e-4, 7.73e-4]	7.77e-4 [7.60e-4, 7.95e-4]	8.02e-4 [7.84e-4, 8.20e-4]	7.88e-4 [7.70e-4, 8.05e-4]
8	1.91e-4	1.84e-4 [1.76e-4, 1.92e-4]	1.93e-4 [1.84e-4, 2.01e-4]	1.90e-4 [1.82e-4, 1.99e-4]	1.82e-4 [1.73e-4, 1.90e-4]
9	3.36e-5	3.21e-5 [2.88e-5, 3.58e-5]	3.41e-5 [3.07e-5, 3.79e-5]	3.42e-5 [3.08e-5, 3.80e-5]	3.57e-5 [3.22e-5, 3.96e-5]
10	3.87e-6	4.00e-6 [2.94e-6, 5.45e-6]	3.20e-6 [2.27e-6, 4.52e-6]	4.10e-6 [3.02e-6, 5.56e-6]	3.70e-6 [2.68e-6, 5.10e-6]

16-QAM: error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	theory	2.15	yes	binomial
comnum.run_spec	float64	theory	1.64	yes	binomial
OptiCommPy (numba)	complex64 symbols (its Gray constellation), float64 noise	theory	1.42	yes	binomial
Sionna	float32 (config.precision = ‘single’)	theory	2.41	yes	binomial

BER and its 95 % interval, every point (10,000,000 bits per point)

Eb/N0 (dB)	theory	comnum (functions)	comnum.run_spec	OptiCommPy (numba)	Sionna
0	1.41e-1	1.41e-1 [1.41e-1, 1.41e-1]	1.41e-1 [1.41e-1, 1.41e-1]	1.41e-1 [1.41e-1, 1.41e-1]	1.41e-1 [1.41e-1, 1.41e-1]
1	1.19e-1	1.19e-1 [1.19e-1, 1.19e-1]	1.19e-1 [1.19e-1, 1.19e-1]	1.19e-1 [1.19e-1, 1.19e-1]	1.19e-1 [1.19e-1, 1.19e-1]
2	9.77e-2	9.78e-2 [9.76e-2, 9.80e-2]	9.77e-2 [9.75e-2, 9.79e-2]	9.78e-2 [9.76e-2, 9.80e-2]	9.75e-2 [9.73e-2, 9.77e-2]
3	7.75e-2	7.75e-2 [7.74e-2, 7.77e-2]	7.74e-2 [7.73e-2, 7.76e-2]	7.75e-2 [7.73e-2, 7.76e-2]	7.74e-2 [7.73e-2, 7.76e-2]
4	5.86e-2	5.87e-2 [5.85e-2, 5.88e-2]	5.86e-2 [5.84e-2, 5.87e-2]	5.85e-2 [5.84e-2, 5.87e-2]	5.85e-2 [5.84e-2, 5.87e-2]
5	4.19e-2	4.18e-2 [4.17e-2, 4.19e-2]	4.19e-2 [4.17e-2, 4.20e-2]	4.20e-2 [4.19e-2, 4.21e-2]	4.20e-2 [4.19e-2, 4.21e-2]
6	2.79e-2	2.79e-2 [2.78e-2, 2.80e-2]	2.79e-2 [2.78e-2, 2.80e-2]	2.78e-2 [2.77e-2, 2.79e-2]	2.79e-2 [2.78e-2, 2.81e-2]
7	1.70e-2	1.70e-2 [1.69e-2, 1.71e-2]	1.70e-2 [1.69e-2, 1.71e-2]	1.70e-2 [1.69e-2, 1.70e-2]	1.70e-2 [1.69e-2, 1.71e-2]
8	9.25e-3	9.24e-3 [9.18e-3, 9.30e-3]	9.24e-3 [9.18e-3, 9.30e-3]	9.22e-3 [9.16e-3, 9.28e-3]	9.24e-3 [9.19e-3, 9.30e-3]
9	4.39e-3	4.35e-3 [4.30e-3, 4.39e-3]	4.42e-3 [4.38e-3, 4.47e-3]	4.40e-3 [4.36e-3, 4.44e-3]	4.37e-3 [4.33e-3, 4.41e-3]
10	1.75e-3	1.75e-3 [1.73e-3, 1.78e-3]	1.78e-3 [1.75e-3, 1.80e-3]	1.76e-3 [1.73e-3, 1.79e-3]	1.76e-3 [1.73e-3, 1.78e-3]

QPSK: time

Median of 5 runs, ns per bit; workload: 1000000 bits at Eb/N0 = 6.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	36.1	36.9	—	1.00	0.0419 s
comnum.run_spec	float64	34.0	6.32	—	0.942	0.0345 s
OptiCommPy (numba)	complex64 symbols (its Gray constellation), float64 noise	86.7	50.3	—	2.40	2.94 s
Sionna	float32 (config.precision = ‘single’)	98.2	17.7	0.548	2.72	0.116 s

16-QAM: time

Median of 5 runs, ns per bit; workload: 1000000 bits at Eb/N0 = 6.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	19.5	20.0	—	1.00	0.0205 s
comnum.run_spec	float64	19.9	4.25	—	1.02	0.0219 s
OptiCommPy (numba)	complex64 symbols (its Gray constellation), float64 noise	87.5	33.7	—	4.48	0.0959 s
Sionna	float32 (config.precision = ‘single’)	172	23.1	0.702	8.82	0.206 s

BPSK and QPSK over flat Rayleigh fading#

Scenario 2: BPSK and Gray QPSK over flat Rayleigh fading (one coefficient per symbol, perfectly known at the receiver), BER against the closed form.

Parameters:

Eb/N0 (dB): 0, 5, 10, 15, 20, 25, 30
bits per point: 10^7 (quick: 2·10^5), in blocks of 10^6 (quick: 10^5)
channel: y = h x + w, h ~ CN(0, 1) independent for every symbol, known at the receiver
timing: one block of 10^6 bits (quick: 2·10^5) at Eb/N0 = 10 dB
theory: BPSK: (1 - sqrt(G / (1 + G))) / 2; Gray QPSK: the same per bit (Proakis & Salehi, 5th ed., Sec. 13.3)

Script: benchmarks/libraries/s2_rayleigh.py.

The script

"""Scenario 2: BPSK and Gray QPSK over flat Rayleigh fading (one coefficient per
symbol, perfectly known at the receiver), BER against the closed form.

Every library draws its own bits, channel coefficients h ~ CN(0, 1) and noise, and
detects with its own coherent detector given h (zero forcing over the 1 x 1 channel:
comnum.mimo_detect, Sionna's LinearDetector, the mimo_rx block of the specification).
komm and OptiCommPy have no fading channel. Timing: one block of ``TIMING_BITS`` bits
at Eb/N0 = 10 dB, median of 5 runs after a warm-up run.

    python s2_rayleigh.py --lib sionna --threads 1 [--quick] --out results/_work/s2.json
"""

from __future__ import annotations

import math
import time

import common
import numpy as np
import theory

SCENARIO = "s2_rayleigh"
TITLE = "BPSK and QPSK over flat Rayleigh fading"
ORDERS = {"bpsk": 2, "qpsk": 4}
SUPPORT = {
    "comnum": True,
    "comnum-spec": True,
    "komm": "no fading channel",
    "opticommpy": "no fading channel",
    "opticommpy-nojit": "no fading channel",
    "sionna": True,
}
CASES = {"bpsk": SUPPORT, "qpsk": SUPPORT}
EBN0 = [0.0, 5.0, 10.0, 15.0, 20.0, 25.0, 30.0]
TIMING_EBN0 = 10.0
TIMING_TRIALS = 40

PARAMETERS = {
    "ebn0_db": EBN0,
    "bits_per_point": "10^7 (quick: 2·10^5), in blocks of 10^6 (quick: 10^5)",
    "channel": "y = h x + w, h ~ CN(0, 1) independent for every symbol, known at the receiver",
    "timing": "one block of 10^6 bits (quick: 2·10^5) at Eb/N0 = 10 dB",
    "theory": "BPSK: (1 - sqrt(G / (1 + G))) / 2; Gray QPSK: the same per bit (Proakis & Salehi, 5th ed., Sec. 13.3)",
}


def sizes(quick: bool) -> dict:
    if quick:
        return {"bits_per_point": 200_000, "block": 100_000, "timing_bits": 200_000, "trials": 10}
    return {
        "bits_per_point": 10_000_000,
        "block": 1_000_000,
        "timing_bits": 1_000_000,
        "trials": 100,
    }


def chain_comnum(m: int, seed: int):
    import comnum

    c = comnum.Constellation("psk" if m == 2 else "qam", m).normalized()
    k = int(math.log2(m))
    rng = comnum.Rng(seed)

    def run(n: int, ebn0: float) -> int:
        b = comnum.bits(n, rng)
        x = c.map_bits(b)
        h = comnum.mimo_channels(x.size, 1, 1, rng)
        n0 = 1.0 / (k * 10 ** (ebn0 / 10))
        y = h[:, 0, 0] * x + rng.complex_noise(x.size, n0)
        labels = comnum.mimo_detect(h, y[None, :], c, n0, "zf")
        bh = comnum.unpack_symbols(labels.ravel(), k)
        return int(np.count_nonzero(bh != b))

    return run, "float64"


def chain_sionna(m: int, seed: int):
    import torch
    from sionna.phy import config
    from sionna.phy.channel import FlatFadingChannel
    from sionna.phy.mapping import BinarySource, Mapper
    from sionna.phy.mimo import LinearDetector
    from sionna.phy.utils import ebnodb2no

    config.seed = seed
    k = int(math.log2(m))
    kind = "pam" if m == 2 else "qam"
    source, mapper = BinarySource(), Mapper(kind, k)
    channel = FlatFadingChannel(1, 1, return_channel=True)
    detector = LinearDetector(
        "zf", "bit", "maxlog", constellation_type=kind, num_bits_per_symbol=k, hard_out=True
    )

    def run(n: int, ebn0: float) -> int:
        with torch.inference_mode():
            b = source([n // k, k])
            x = mapper(b)  # [n/k, 1]
            no = ebnodb2no(ebn0, k, 1.0)
            y, h = channel(x, no)
            s = torch.full((n // k, 1, 1), float(no), dtype=y.dtype)
            bh = detector(y, h, s)  # [n/k, 1, k]
            return int((bh.reshape(b.shape) != b).sum())

    return run, "float32 (config.precision = 'single')"


CHAINS = {"comnum": chain_comnum, "sionna": chain_sionna}


def run_chains(args, todo: list[str]) -> list[dict]:
    threading = common.setup_torch(args.threads) if args.lib == "sionna" else {}
    sz = sizes(args.quick)
    out = []
    for case in todo:
        m = ORDERS[case]
        run, precision = CHAINS[args.lib](m, args.seed)
        res = {"case": case, "precision": precision, **threading}
        n = sz["timing_bits"]
        t0 = time.perf_counter()
        run(n, TIMING_EBN0)
        res["first_call_s"] = time.perf_counter() - t0
        if not args.timing_only:
            pts = []
            t0 = time.perf_counter()
            for e in EBN0:
                errors = sum(
                    run(sz["block"], e) for _ in range(sz["bits_per_point"] // sz["block"])
                )
                pts.append(
                    common.rate_point(
                        errors,
                        sz["bits_per_point"],
                        float(theory.ber_qam_rayleigh(m, e)),
                        ebn0_db=e,
                    )
                )
            res["points"] = pts
            res["sweep_s"] = time.perf_counter() - t0
        res["timing"] = common.time_it(lambda n=n, run=run: run(n, TIMING_EBN0))
        res["timing"].update(units=n, unit="bit", workload=f"{n} bits at Eb/N0 = {TIMING_EBN0} dB")
        out.append(res)
    return out


def spec(m: int, ebn0: list[float], bits: int, trials: int, seed: int) -> dict:
    k = int(math.log2(m))
    return {
        "name": f"bench_rayleigh_{m}",
        "seed": seed,
        "batch": trials,
        "grid": {"ebn0_db": ebn0},
        "link": [
            {
                "type": "source",
                "constellation": {"kind": "psk" if m == 2 else "qam", "m": m},
                "n_symbols": bits // (k * trials),
            },
            {"type": "rayleigh", "n_rx": 1, "coherence_samples": 1},
            {"type": "awgn_ebn0", "ebn0_db": "$ebn0_db"},
            {"type": "mimo_rx", "detector": "zf"},
        ],
        "metrics": {"report": ["ber"], "scale": "none"},
        "stop": {"min_trials": trials, "max_trials": trials},
    }


def spec_points(rows: list[dict], m: int) -> list[dict]:
    pts = []
    for row in rows:
        e = float(row["condition"]["ebn0_db"])
        ber = row["metrics"]["ber"]
        pts.append(
            common.rate_point(
                ber["count"], ber["total"], float(theory.ber_qam_rayleigh(m, e)), ebn0_db=e
            )
        )
    return pts


def run_spec_lib(args, todo: list[str]) -> list[dict]:
    sz = sizes(args.quick)
    n = sz["timing_bits"]
    out = []
    for case in todo:
        m = ORDERS[case]
        out.append(
            common.spec_case(
                args,
                {"case": case, "precision": "float64"},
                spec(m, EBN0, sz["bits_per_point"], sz["trials"], args.seed),
                spec(m, [TIMING_EBN0], n, TIMING_TRIALS, args.seed + 1),
                lambda rows, m=m: spec_points(rows, m),
                units=n,
                unit="bit",
                workload=f"{n} bits at Eb/N0 = {TIMING_EBN0} dB, {TIMING_TRIALS} trials",
            )
        )
    return out


RUNNERS = {"comnum": run_chains, "sionna": run_chains, "comnum-spec": run_spec_lib}

if __name__ == "__main__":
    common.main(SCENARIO, CASES, RUNNERS)

Support

Case	comnum (functions)	comnum.run_spec	OptiCommPy (numba)	Sionna
BPSK	yes	yes	not available: no fading channel	yes
QPSK	yes	yes	not available: no fading channel	yes

BPSK: error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	theory	2.22	yes	binomial
comnum.run_spec	float64	theory	1.78	yes	binomial
Sionna	float32 (config.precision = ‘single’)	theory	2.35	yes	binomial

BER and its 95 % interval, every point (10,000,000 bits per point)

Eb/N0 (dB)	theory	comnum (functions)	comnum.run_spec	Sionna
0	1.46e-1	1.46e-1 [1.46e-1, 1.47e-1]	1.46e-1 [1.46e-1, 1.47e-1]	1.47e-1 [1.46e-1, 1.47e-1]
5	6.42e-2	6.43e-2 [6.41e-2, 6.44e-2]	6.42e-2 [6.41e-2, 6.44e-2]	6.42e-2 [6.40e-2, 6.43e-2]
10	2.33e-2	2.33e-2 [2.32e-2, 2.34e-2]	2.32e-2 [2.32e-2, 2.33e-2]	2.33e-2 [2.32e-2, 2.34e-2]
15	7.72e-3	7.73e-3 [7.67e-3, 7.78e-3]	7.76e-3 [7.71e-3, 7.82e-3]	7.75e-3 [7.70e-3, 7.80e-3]
20	2.48e-3	2.51e-3 [2.48e-3, 2.54e-3]	2.46e-3 [2.43e-3, 2.49e-3]	2.52e-3 [2.49e-3, 2.55e-3]
25	7.89e-4	8.08e-4 [7.91e-4, 8.26e-4]	7.83e-4 [7.66e-4, 8.00e-4]	7.88e-4 [7.71e-4, 8.06e-4]
30	2.50e-4	2.52e-4 [2.42e-4, 2.62e-4]	2.41e-4 [2.31e-4, 2.51e-4]	2.43e-4 [2.34e-4, 2.53e-4]

QPSK: error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	theory	2.08	yes	binomial
comnum.run_spec	float64	theory	1.17	yes	binomial
Sionna	float32 (config.precision = ‘single’)	theory	3.13	yes	binomial

BER and its 95 % interval, every point (10,000,000 bits per point)

Eb/N0 (dB)	theory	comnum (functions)	comnum.run_spec	Sionna
0	1.46e-1	1.46e-1 [1.46e-1, 1.46e-1]	1.46e-1 [1.46e-1, 1.47e-1]	1.46e-1 [1.46e-1, 1.47e-1]
5	6.42e-2	6.41e-2 [6.39e-2, 6.42e-2]	6.42e-2 [6.40e-2, 6.43e-2]	6.42e-2 [6.40e-2, 6.43e-2]
10	2.33e-2	2.34e-2 [2.33e-2, 2.34e-2]	2.33e-2 [2.32e-2, 2.34e-2]	2.31e-2 [2.30e-2, 2.32e-2]
15	7.72e-3	7.77e-3 [7.72e-3, 7.82e-3]	7.73e-3 [7.67e-3, 7.78e-3]	7.67e-3 [7.61e-3, 7.72e-3]
20	2.48e-3	2.51e-3 [2.48e-3, 2.54e-3]	2.50e-3 [2.47e-3, 2.53e-3]	2.46e-3 [2.43e-3, 2.50e-3]
25	7.89e-4	8.02e-4 [7.84e-4, 8.19e-4]	7.81e-4 [7.64e-4, 7.99e-4]	7.92e-4 [7.75e-4, 8.10e-4]
30	2.50e-4	2.53e-4 [2.44e-4, 2.63e-4]	2.46e-4 [2.37e-4, 2.56e-4]	2.51e-4 [2.41e-4, 2.61e-4]

BPSK: time

Median of 5 runs, ns per bit; workload: 1000000 bits at Eb/N0 = 10.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	273	249	—	1.00	0.264 s
comnum.run_spec	float64	291	20.7	—	1.07	0.305 s
Sionna	float32 (config.precision = ‘single’)	327	196	8.11	1.20	0.348 s

QPSK: time

Median of 5 runs, ns per bit; workload: 1000000 bits at Eb/N0 = 10.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	133	136	—	1.00	0.124 s
comnum.run_spec	float64	151	10.9	—	1.14	0.154 s
Sionna	float32 (config.precision = ‘single’)	217	106	4.46	1.64	0.215 s

4 x 4 MIMO, 16-QAM, i.i.d. Rayleigh fading: ZF, MMSE and ML detection#

Scenario 3: 4 x 4 spatial multiplexing of Gray 16-QAM over i.i.d. Rayleigh fading (one channel matrix per vector, known at the receiver), detected by zero forcing, MMSE and maximum likelihood.

Parameters:

Eb/N0 (dB): 0, 4, 8, 12, 16
vectors per point: linear detectors and sphere: 10^5 (quick: 5·10^3); exhaustive ML: 2·10^4 (quick: 10^3)
timing: 10^4 vectors at Eb/N0 = 12 dB (exhaustive ML: 10^3; quick: 2·10^3 and 2·10^2)
theory: ZF: 16-QAM over SISO Rayleigh fading (Winters, Salz & Gitlin, IEEE Trans. Commun. 42(2/3/4), 1994)

Script: benchmarks/libraries/s3_mimo.py.

The script

"""Scenario 3: 4 x 4 spatial multiplexing of Gray 16-QAM over i.i.d. Rayleigh fading
(one channel matrix per vector, known at the receiver), detected by zero forcing, MMSE
and maximum likelihood.

Every stream carries unit-energy symbols (no power split between the antennas) and
Eb/N0 is the energy per bit received on one antenna over N0: N0 = 1 / (4 Eb/N0).
Zero forcing over 4 x 4 leaves one diversity branch per stream, so its BER is that of
16-QAM over SISO Rayleigh fading (Winters, Salz & Gitlin 1994); MMSE and ML have no
closed form here and are compared between the libraries. Maximum likelihood is an
exhaustive search over the 16^4 = 65536 vectors in Sionna and in comnum ("ml"), and
the Schnorr-Euchner sphere decoder in comnum ("sphere", the same decisions).
The specification runner splits the power between the antennas (mimo_tx), so it runs
at Eb/N0 + 10 log10(4) dB to have the same SNR. Timing: ``TIMING_VECTORS`` vectors at
Eb/N0 = 12 dB (channel, noise, detection), median of 5 runs after a warm-up run.

    python s3_mimo.py --lib comnum --threads 1 [--quick] --out results/_work/s3.json
"""

from __future__ import annotations

import math
import time

import common
import numpy as np
import theory

SCENARIO = "s3_mimo"
TITLE = "4 x 4 MIMO, 16-QAM, i.i.d. Rayleigh fading: ZF, MMSE and ML detection"
NT = NR = 4
M = 16
K = 4
NO_MIMO = "no MIMO detection"
SUPPORT = {
    "comnum": True,
    "comnum-spec": True,
    "komm": NO_MIMO,
    "opticommpy": "no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)",
    "opticommpy-nojit": "no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)",
    "sionna": True,
}
CASES = {
    "zf": SUPPORT,
    "mmse": SUPPORT,
    "ml": SUPPORT,
    "sphere": {
        **SUPPORT,
        "sionna": "no sphere decoder (its exhaustive ML detector is the case ml)",
    },
}
SPEC_DETECTOR = {"zf": "zf", "mmse": "mmse", "ml": "ml", "sphere": "sphere"}
EBN0 = [0.0, 4.0, 8.0, 12.0, 16.0]
TIMING_EBN0 = 12.0
SPEC_OFFSET_DB = 10 * math.log10(NT)

PARAMETERS = {
    "ebn0_db": EBN0,
    "vectors_per_point": "linear detectors and sphere: 10^5 (quick: 5·10^3); exhaustive ML: 2·10^4 (quick: 10^3)",
    "timing": "10^4 vectors at Eb/N0 = 12 dB (exhaustive ML: 10^3; quick: 2·10^3 and 2·10^2)",
    "theory": "ZF: 16-QAM over SISO Rayleigh fading (Winters, Salz & Gitlin, IEEE Trans. Commun. 42(2/3/4), 1994)",
}


def sizes(quick: bool, case: str) -> dict:
    heavy = case == "ml"
    if quick:
        return {
            "vectors": 1_000 if heavy else 5_000,
            "timing": 200 if heavy else 2_000,
            "blocks": 1,
        }
    return {
        "vectors": 20_000 if heavy else 100_000,
        "timing": 1_000 if heavy else 10_000,
        "blocks": 10,
    }


def theory_ber(case: str, ebn0: float) -> float | None:
    return float(theory.ber_qam_rayleigh(M, ebn0)) if case == "zf" else None


def chain_comnum(case: str, seed: int):
    import comnum

    c = comnum.Constellation("qam", M).normalized()
    rng = comnum.Rng(seed)

    def run(n: int, ebn0: float) -> np.ndarray:
        b = comnum.bits(n * NT * K, rng)
        x = c.map_bits(b).reshape(n, NT)
        h = comnum.mimo_channels(n, NR, NT, rng)
        n0 = 1.0 / (K * 10 ** (ebn0 / 10))
        # y = H x + w (the Python API draws H; the product is NumPy's)
        y = np.matmul(h, x[:, :, None])[:, :, 0] + rng.complex_noise(n * NR, n0).reshape(n, NR)
        labels = comnum.mimo_detect(h, y.T, c, n0, case)  # stream-major (NT, n)
        bh = comnum.unpack_symbols(labels.T.ravel(), K)
        return np.count_nonzero((bh != b).reshape(n, NT * K), axis=1)  # bit errors per vector

    return run, "float64"


def chain_sionna(case: str, seed: int):
    import torch
    from sionna.phy import config
    from sionna.phy.channel import FlatFadingChannel
    from sionna.phy.mapping import BinarySource, Mapper
    from sionna.phy.mimo import LinearDetector, MaximumLikelihoodDetector
    from sionna.phy.utils import ebnodb2no

    config.seed = seed
    source, mapper = BinarySource(), Mapper("qam", K)
    channel = FlatFadingChannel(NT, NR, return_channel=True)
    if case == "ml":
        det = MaximumLikelihoodDetector(
            "bit", "maxlog", NT, constellation_type="qam", num_bits_per_symbol=K, hard_out=True
        )
        chunk = 64  # vectors per call: the search holds 65536 candidates per vector
    else:
        eq = {"zf": "zf", "mmse": "lmmse"}[case]
        det = LinearDetector(
            eq, "bit", "maxlog", constellation_type="qam", num_bits_per_symbol=K, hard_out=True
        )
        chunk = None

    def run(n: int, ebn0: float) -> np.ndarray:
        with torch.inference_mode():
            b = source([n, NT * K])
            x = mapper(b)  # [n, NT]
            no = ebnodb2no(ebn0, K, 1.0)  # Es / (K Eb/N0), Es = 1
            y, h = channel(x, no)
            s = (float(no) * torch.eye(NR, dtype=y.dtype)).expand(n, NR, NR)
            if chunk is None:
                bh = det(y, h, s)
            else:
                bh = torch.cat(
                    [
                        det(y[i : i + chunk], h[i : i + chunk], s[i : i + chunk])
                        for i in range(0, n, chunk)
                    ]
                )
            return common.to_numpy((bh.reshape(b.shape) != b).sum(dim=1))  # bit errors per vector

    return run, "float32 (config.precision = 'single')"


CHAINS = {"comnum": chain_comnum, "sionna": chain_sionna}


def run_chains(args, todo: list[str]) -> list[dict]:
    threading = common.setup_torch(args.threads) if args.lib == "sionna" else {}
    out = []
    for case in todo:
        sz = sizes(args.quick, case)
        run, precision = CHAINS[args.lib](case, args.seed)
        res = {"case": case, "precision": precision, **threading}
        n = sz["timing"]
        t0 = time.perf_counter()
        run(n, TIMING_EBN0)
        res["first_call_s"] = time.perf_counter() - t0
        if not args.timing_only:
            pts = []
            t0 = time.perf_counter()
            for e in EBN0:
                # the 16 bits of a vector share its channel: the interval is per vector
                errs = np.concatenate(
                    [run(sz["vectors"] // sz["blocks"], e) for _ in range(sz["blocks"])]
                )
                pts.append(
                    common.frame_ber_point(
                        errs, NT * K, theory_ber(case, e), ebn0_db=e, vectors=errs.size
                    )
                )
            res["points"] = pts
            res["sweep_s"] = time.perf_counter() - t0
        res["timing"] = common.time_it(lambda n=n, run=run: run(n, TIMING_EBN0))
        res["timing"].update(
            units=n, unit="vector", workload=f"{n} vectors at Eb/N0 = {TIMING_EBN0} dB"
        )
        out.append(res)
    return out


def spec(case: str, ebn0: list[float], vectors: int, trials: int, seed: int) -> dict:
    return {
        "name": f"bench_mimo_{case}",
        "seed": seed,
        "batch": trials,
        "grid": {"ebn0_db": [e + SPEC_OFFSET_DB for e in ebn0]},
        "link": [
            {"type": "source", "constellation": "16-QAM", "n_symbols": NT * vectors // trials},
            {"type": "mimo_tx", "n_tx": NT, "scheme": "spatial"},
            {"type": "rayleigh", "n_rx": NR, "coherence_samples": 1},
            {"type": "awgn_ebn0", "ebn0_db": "$ebn0_db"},
            {"type": "mimo_rx", "detector": SPEC_DETECTOR[case]},
        ],
        "metrics": {"report": ["ber"], "scale": "none"},
        "stop": {"min_trials": trials, "max_trials": trials},
    }


def run_spec_lib(args, todo: list[str]) -> list[dict]:
    trials = 40
    out = []
    for case in todo:
        sz = sizes(args.quick, case)

        def points(rows: list[dict], case=case, vectors=sz["vectors"]) -> list[dict]:
            pts = []
            for e, row in zip(EBN0, rows, strict=True):
                ber = row["metrics"]["ber"]
                pts.append(
                    common.rate_point(
                        ber["count"], ber["total"], theory_ber(case, e), ebn0_db=e, vectors=vectors
                    )
                )
            return pts

        n = sz["timing"]
        out.append(
            common.spec_case(
                args,
                {"case": case, "precision": "float64", "ebn0_offset_db": SPEC_OFFSET_DB},
                spec(case, EBN0, sz["vectors"], trials, args.seed),
                spec(case, [TIMING_EBN0], n, trials, args.seed + 1),
                points,
                units=n,
                unit="vector",
                workload=f"{n} vectors at Eb/N0 = {TIMING_EBN0} dB, {trials} trials",
            )
        )
    return out


RUNNERS = {"comnum": run_chains, "sionna": run_chains, "comnum-spec": run_spec_lib}

if __name__ == "__main__":
    common.main(SCENARIO, CASES, RUNNERS)

Support

Case	comnum (functions)	comnum.run_spec	OptiCommPy (numba)	Sionna
zero forcing	yes	yes	not available: no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)	yes
MMSE	yes	yes	not available: no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)	yes
ML (exhaustive)	yes	yes	not available: no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)	yes
ML (sphere decoder)	yes	yes	not available: no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)	not available: no sphere decoder (its exhaustive ML detector is the case ml)

zero forcing: error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	theory	1.88	yes	per-frame
comnum.run_spec	float64	theory	2.98	yes	binomial
Sionna	float32 (config.precision = ‘single’)	theory	1.79	yes	per-frame

BER and its 95 % interval, every point (1,600,000 bits per point)

Eb/N0 (dB)	theory	comnum (functions)	comnum.run_spec	Sionna
0	1.98e-1	1.98e-1 [1.97e-1, 1.99e-1]	1.98e-1 [1.97e-1, 1.98e-1]	1.98e-1 [1.97e-1, 1.99e-1]
4	1.20e-1	1.19e-1 [1.18e-1, 1.20e-1]	1.19e-1 [1.19e-1, 1.20e-1]	1.19e-1 [1.19e-1, 1.20e-1]
8	6.20e-2	6.16e-2 [6.10e-2, 6.23e-2]	6.14e-2 [6.10e-2, 6.18e-2]	6.26e-2 [6.19e-2, 6.32e-2]
12	2.83e-2	2.82e-2 [2.78e-2, 2.87e-2]	2.81e-2 [2.79e-2, 2.84e-2]	2.83e-2 [2.78e-2, 2.87e-2]
16	1.20e-2	1.21e-2 [1.18e-2, 1.24e-2]	1.19e-2 [1.17e-2, 1.20e-2]	1.19e-2 [1.16e-2, 1.22e-2]

MMSE: error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	—	—	—	per-frame
comnum.run_spec	float64	comnum	3.63	yes	binomial
Sionna	float32 (config.precision = ‘single’)	comnum	1.57	yes	per-frame

BER and its 95 % interval, every point (1,600,000 bits per point)

Eb/N0 (dB)	comnum (functions)	comnum.run_spec	Sionna
0	1.39e-1 [1.38e-1, 1.40e-1]	1.38e-1 [1.38e-1, 1.39e-1]	1.38e-1 [1.38e-1, 1.39e-1]
4	8.27e-2 [8.21e-2, 8.33e-2]	8.40e-2 [8.36e-2, 8.44e-2]	8.33e-2 [8.28e-2, 8.39e-2]
8	4.31e-2 [4.26e-2, 4.35e-2]	4.29e-2 [4.26e-2, 4.32e-2]	4.32e-2 [4.28e-2, 4.37e-2]
12	1.97e-2 [1.94e-2, 2.00e-2]	1.99e-2 [1.97e-2, 2.01e-2]	1.96e-2 [1.93e-2, 1.99e-2]
16	8.42e-3 [8.20e-3, 8.64e-3]	8.33e-3 [8.19e-3, 8.47e-3]	8.23e-3 [8.02e-3, 8.45e-3]

ML (exhaustive): error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	—	—	—	per-frame
comnum.run_spec	float64	comnum	2.53	yes	binomial
Sionna	float32 (config.precision = ‘single’)	comnum	1.09	yes	per-frame

BER and its 95 % interval, every point (320,000 bits per point)

Eb/N0 (dB)	comnum (functions)	comnum.run_spec	Sionna
0	1.25e-1 [1.23e-1, 1.26e-1]	1.25e-1 [1.24e-1, 1.26e-1]	1.25e-1 [1.23e-1, 1.27e-1]
4	4.03e-2 [3.90e-2, 4.16e-2]	3.99e-2 [3.92e-2, 4.06e-2]	4.05e-2 [3.92e-2, 4.17e-2]
8	4.12e-3 [3.68e-3, 4.56e-3]	4.77e-3 [4.54e-3, 5.02e-3]	4.47e-3 [4.02e-3, 4.93e-3]
12	2.34e-4 [1.24e-4, 3.45e-4]	2.28e-4 [1.81e-4, 2.87e-4]	1.59e-4 [7.61e-5, 2.43e-4]
16	0 [0, 9.38e-6]	0 [0, 1.20e-5]	1.25e-5 [0, 3.70e-5]

ML (sphere decoder): error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	—	—	—	per-frame
comnum.run_spec	float64	comnum	2.84	yes	binomial

BER and its 95 % interval, every point (1,600,000 bits per point)

Eb/N0 (dB)	comnum (functions)	comnum.run_spec
0	1.24e-1 [1.23e-1, 1.25e-1]	1.25e-1 [1.24e-1, 1.25e-1]
4	3.97e-2 [3.92e-2, 4.03e-2]	4.06e-2 [4.03e-2, 4.09e-2]
8	4.44e-3 [4.23e-3, 4.64e-3]	4.62e-3 [4.51e-3, 4.73e-3]
12	1.91e-4 [1.49e-4, 2.32e-4]	1.95e-4 [1.75e-4, 2.18e-4]
16	3.13e-6 [0, 6.37e-6]	1.25e-5 [8.09e-6, 1.93e-5]

zero forcing: time

Median of 5 runs, µs per vector; workload: 10000 vectors at Eb/N0 = 12.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	2.13	2.11	—	1.00	0.0241 s
comnum.run_spec	float64	2.28	0.355	—	1.07	0.0209 s
Sionna	float32 (config.precision = ‘single’)	5.09	1.86	0.143	2.40	0.0513 s

MMSE: time

Median of 5 runs, µs per vector; workload: 10000 vectors at Eb/N0 = 12.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	2.16	2.12	—	1.00	0.0265 s
comnum.run_spec	float64	2.42	0.341	—	1.12	0.0231 s
Sionna	float32 (config.precision = ‘single’)	4.34	2.41	0.158	2.01	0.052 s

ML (exhaustive): time

Median of 5 runs, µs per vector; workload: 1000 vectors at Eb/N0 = 12.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	969	984	—	1.00	0.985 s
comnum.run_spec	float64	1015	55.8	—	1.05	0.986 s
Sionna	float32 (config.precision = ‘single’)	13203	2919	61.8	13.6	13.1 s

ML (sphere decoder): time

Median of 5 runs, µs per vector; workload: 10000 vectors at Eb/N0 = 12.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	2.65	2.67	—	1.00	0.0283 s
comnum.run_spec	float64	3.17	0.435	—	1.20	0.0299 s

Channel coding over BPSK and AWGN#

Scenario 4: channel coding over BPSK and AWGN.

Parameters:

convolutional (133, 171), soft Viterbi: K = 7, (133, 171), k = 1024 + 6 tail bits, soft Viterbi; Eb/N0 = 1, 2, 3, 4 dB
5G NR LDPC (1024, 2048): 5G NR LDPC, k = 1024, n = 2048 (base graph 2, Z = 104), flooding sum-product, 50 iterations; Eb/N0 = 1, 1.5, 2, 2.5 dB
LDPC (1024, 512), same LLRs: PEG (1024, 512), flooding sum-product, 50 iterations, identical LLRs; Eb/N0 = 1.5, 2, 2.5, 3 dB
frames per point: 2000 (quick: 100)
timing: 200 frames (quick: 20) at Eb/N0 = 3 dB (conv) or 2 dB (LDPC); OptiCommPy without numba: 2 frames

Script: benchmarks/libraries/s4_coding.py.

The script

"""Scenario 4: channel coding over BPSK and AWGN.

- ``conv``: the rate-1/2 convolutional code of constraint length 7, generators
  (133, 171) in octal, 1024 information bits per frame terminated by 6 tail bits
  (rate 1024/2060), soft-decision Viterbi decoding. komm counts the taps from the
  least significant bit: the same code is ``ConvolutionalCode([[0o155, 0o117]])``
  there (checked on the impulse response). Reference: the union bound with the
  distance spectrum of the trellis (Viterbi 1971).
- ``nr_ldpc``: the LDPC code of 5G NR (3GPP TS 38.212, Sec. 5.3.2), k = 1024, rate
  matched to n = 2048 (base graph 2 by the rule of Sec. 7.2.2, lifting size 104, the
  first 2Z bits punctured; the same in both libraries), sum-product
  decoding by flooding, 50 iterations at most, in both libraries (comnum's default is
  the layered schedule, which converges in about half the iterations; it is set to
  flooding here so that the error rates compare the same decoder). comnum stops early on
  a codeword, Sionna runs the 50 iterations (its default).
- ``ldpc_same_llr``: one (1024, 512) LDPC code (PEG construction of comnum, variable
  degree 3, seed 1), its parity-check matrix given to the three decoders that accept
  one (comnum, OptiCommPy, Sionna), and the same channel LLRs (drawn by comnum from the
  same seed in every process): flooding sum-product, 50 iterations at most. The
  decoders see identical inputs, so they should take the same decisions frame by
  frame.

Every library uses its own encoder, BPSK mapper, AWGN channel and LLR computation in
``conv`` and ``nr_ldpc`` (independent seeds); Eb/N0 counts the energy per information
bit. Timings: the decoder alone on ``TIMING_FRAMES`` frames of LLRs at the timing
point, and the whole chain (bits, encoding, mapping, noise, LLRs, decoding, error
count) on the same number of frames; median of 5 runs after a warm-up run.

    python s4_coding.py --lib sionna --threads 1 [--quick] --out results/_work/s4.json
"""

from __future__ import annotations

import time

import common
import numpy as np
import theory

SCENARIO = "s4_coding"
TITLE = "Channel coding over BPSK and AWGN"
K_CONV = 1024
CONV_GEN = [0o133, 0o171]
K_NR, N_NR = 1024, 2048
N_PEG, K_PEG = 1024, 512
ITERATIONS = 50
EBN0 = {
    "conv": [1.0, 2.0, 3.0, 4.0],
    "nr_ldpc": [1.0, 1.5, 2.0, 2.5],
    "ldpc_same_llr": [1.5, 2.0, 2.5, 3.0],
}
TIMING_EBN0 = {"conv": 3.0, "nr_ldpc": 2.0, "ldpc_same_llr": 2.0}
SPEC_ONLY_OWN = "the runner draws its own channel (this case feeds the same LLRs to every decoder)"
CASES = {
    "conv": {
        "comnum": True,
        "comnum-spec": True,
        "comnum-rust": True,
        "komm": True,
        "opticommpy": "no convolutional code",
        "opticommpy-nojit": "no convolutional code",
        "sionna": True,
    },
    "nr_ldpc": {
        "comnum": True,
        "comnum-spec": "the specification format has no 5G NR LDPC code (NrLdpc: Python and Rust API)",
        "comnum-rust": "the specification format has no 5G NR LDPC code (NrLdpc: Python and Rust API)",
        "komm": "no LDPC code",
        "opticommpy": "no 5G NR LDPC code (its decoder takes a parity-check matrix: see ldpc_same_llr)",
        "opticommpy-nojit": "no 5G NR LDPC code (its decoder takes a parity-check matrix: see ldpc_same_llr)",
        "sionna": True,
    },
    "ldpc_same_llr": {
        "comnum": True,
        "comnum-spec": SPEC_ONLY_OWN,
        "comnum-rust": SPEC_ONLY_OWN,
        "komm": "no LDPC (belief-propagation) decoder",
        "opticommpy": True,
        "opticommpy-nojit": True,
        "sionna": True,
    },
}

PARAMETERS = {
    "conv": "K = 7, (133, 171), k = 1024 + 6 tail bits, soft Viterbi; Eb/N0 = 1, 2, 3, 4 dB",
    "nr_ldpc": "5G NR LDPC, k = 1024, n = 2048 (base graph 2, Z = 104), flooding sum-product, 50 iterations; Eb/N0 = 1, 1.5, 2, 2.5 dB",
    "ldpc_same_llr": "PEG (1024, 512), flooding sum-product, 50 iterations, identical LLRs; Eb/N0 = 1.5, 2, 2.5, 3 dB",
    "frames_per_point": "2000 (quick: 100)",
    "timing": "200 frames (quick: 20) at Eb/N0 = 3 dB (conv) or 2 dB (LDPC); OptiCommPy without numba: 2 frames",
}


def sizes(quick: bool) -> dict:
    return (
        {"frames": 100, "timing": 20, "block": 50}
        if quick
        else {"frames": 2000, "timing": 200, "block": 200}
    )


def n0_of(ebn0: float, rate: float) -> float:
    """N0 of unit-energy BPSK for an Eb/N0 per information bit (complex noise of
    variance N0, N0/2 on the real axis)."""
    return 1.0 / (rate * 10 ** (ebn0 / 10))


# ---------------------------------------------------------------------- the codecs
# A codec gives: encode(F) -> (info bits [F, k], channel input), llrs(F, ebn0) ->
# (reference bits, LLRs in the library's convention), decode(llrs) -> decided bits
# [F, k_out], and the number k_out of bits compared per frame.


def codec_comnum(case: str, seed: int):
    import comnum

    rng = comnum.Rng(seed)
    bpsk = comnum.Constellation("psk", 2)
    if case == "conv":
        code = comnum.ConvolutionalCode(K_CONV, generators=CONV_GEN, constraint=7)
    elif case == "nr_ldpc":
        code = comnum.NrLdpc(
            K_NR, N_NR, max_iterations=ITERATIONS, rule="sum_product", layered=False
        )
    else:
        return codec_same_llr_comnum(seed)
    k, n = code.k, code.n

    def llrs(f: int, ebn0: float):
        b = comnum.bits(f * k, rng)
        x = bpsk.map_bits(code.encode(b))
        n0 = n0_of(ebn0, k / n)
        y = x + rng.complex_noise(x.size, n0)
        return b.reshape(f, k), bpsk.llr(y, n0)

    def decode(llr):
        return code.decode(llr).reshape(-1, k)

    return llrs, decode, k, "float64"


def peg_code():
    import comnum

    code = comnum.LdpcCode.peg(N_PEG, K_PEG, var_degree=3, seed=1)
    code.set_decoder("sum_product", layered=False, max_iterations=ITERATIONS)
    return code


def same_llrs(seed: int):
    """The codewords and LLRs of ldpc_same_llr: the same in every process."""
    import comnum

    code = peg_code()
    rng = comnum.Rng(seed)
    bpsk = comnum.Constellation("psk", 2)

    def llrs(f: int, ebn0: float):
        b = comnum.bits(f * code.k, rng)
        c = code.encode(b)
        n0 = n0_of(ebn0, code.k / code.n)
        y = bpsk.map_bits(c) + rng.complex_noise(c.size, n0)
        return c.reshape(f, code.n), bpsk.llr(y, n0).reshape(f, code.n)  # positive for a 0

    return code, llrs


def codec_same_llr_comnum(seed: int):
    code, llrs = same_llrs(seed)

    def decode(llr):
        return np.stack([(code.decode_llr(row)[0] < 0).astype(np.uint8) for row in llr])

    return llrs, decode, code.n, "float64"


def codec_opticommpy(case: str, seed: int):
    import scipy.sparse as sp
    from optic.comm.fec import decodeLDPC
    from optic.utils import parameters

    code, llrs = same_llrs(seed)
    rows = code.rows
    h = sp.csr_matrix(
        (
            np.ones(sum(map(len, rows)), dtype=np.uint8),
            (np.repeat(np.arange(len(rows)), list(map(len, rows))), np.concatenate(rows)),
        ),
        shape=(len(rows), code.n),
    )
    p = parameters()
    p.H, p.maxIter, p.alg = h, ITERATIONS, "SPA"

    def decode(llr):
        bits, _out, _fail = decodeLDPC(np.ascontiguousarray(llr.T), p)  # columns are codewords
        return bits.T.astype(np.uint8)

    return llrs, decode, code.n, "float32 (decodeLDPC default prec = np.float32)"


def codec_komm(case: str, seed: int):
    import komm

    # (133, 171) with the current input on the most significant bit = (155, 117) in komm
    cc = komm.ConvolutionalCode([[0o155, 0o117]])
    code = komm.TerminatedConvolutionalCode(cc, num_blocks=K_CONV, mode="zero-termination")
    decoder = komm.ViterbiDecoder(code, input_type="soft")
    bpsk, lab = komm.PAMConstellation(2), komm.NaturalLabeling(1)
    rng = np.random.default_rng(seed)
    k, n = code.dimension, code.length

    def llrs(f: int, ebn0: float):
        b = rng.integers(0, 2, (f, k))
        x = bpsk.indices_to_symbols(lab.bits_to_indices(code.encode(b).ravel()))
        sigma2 = n0_of(ebn0, k / n) / 2  # real channel: N0/2 per dimension
        y = komm.GaussianChannel(sigma2, rng=rng).transmit(x)
        return b, lab.marginalize(bpsk.posteriors(y, sigma2))  # log P(0) / P(1)

    def decode(llr):
        return decoder.decode(llr).reshape(-1, k)

    return llrs, decode, k, "float64"


def codec_sionna(case: str, seed: int):
    import torch
    from sionna.phy import config
    from sionna.phy.channel import AWGN
    from sionna.phy.fec.conv import ConvEncoder, ViterbiDecoder
    from sionna.phy.fec.ldpc import LDPC5GDecoder, LDPC5GEncoder, LDPCBPDecoder
    from sionna.phy.mapping import BinarySource, Demapper, Mapper

    config.seed = seed
    if case == "ldpc_same_llr":
        code, same = same_llrs(seed)
        dec = LDPCBPDecoder(peg_pcm(code), num_iter=ITERATIONS, hard_out=True)

        def decode(llr):
            with torch.inference_mode():
                # Sionna's LLRs are logits, log P(1) / P(0): the opposite sign
                llr_dev = torch.from_numpy(-llr.astype(np.float32)).to(common.TORCH_DEVICE)
                return common.to_numpy(dec(llr_dev)).astype(np.uint8)

        return same, decode, code.n, "float32 (config.precision = 'single')"

    if case == "conv":
        enc = ConvEncoder(rate=0.5, constraint_length=7, terminate=True)
        dec = ViterbiDecoder(encoder=enc, method="soft_llr", return_info_bits=True)
        k, n = K_CONV, 2 * (K_CONV + 6)
    else:
        enc = LDPC5GEncoder(K_NR, N_NR)
        dec = LDPC5GDecoder(enc, num_iter=ITERATIONS, hard_out=True, return_infobits=True)
        k, n = K_NR, N_NR
    source, mapper, channel = BinarySource(), Mapper("pam", 1), AWGN()
    demapper = Demapper("app", "pam", 1)

    def llrs(f: int, ebn0: float):
        with torch.inference_mode():
            b = source([f, k])
            no = n0_of(ebn0, k / n)
            y = channel(mapper(enc(b)), no)
            return b, demapper(y, no)  # logits, log P(1) / P(0)

    def decode(llr):
        with torch.inference_mode():
            return dec(llr)

    return llrs, decode, k, "float32 (config.precision = 'single')"


def peg_pcm(code):
    h = np.zeros((len(code.rows), code.n), dtype=np.float32)
    for i, r in enumerate(code.rows):
        h[i, r] = 1
    return h


CODECS = {
    "comnum": codec_comnum,
    "komm": codec_komm,
    "opticommpy": codec_opticommpy,
    "opticommpy-nojit": codec_opticommpy,
    "sionna": codec_sionna,
}


def as_np(a) -> np.ndarray:
    return common.to_numpy(a)


def run_codecs(args, todo: list[str]) -> list[dict]:
    threading = common.setup_torch(args.threads) if args.lib == "sionna" else {}
    sz = sizes(args.quick)
    out = []
    for case in todo:
        llrs, decode, k_out, precision = CODECS[args.lib](case, args.seed)
        res = {"case": case, "precision": precision, **threading}

        def chain(f: int, ebn0: float, llrs=llrs, decode=decode) -> np.ndarray:
            ref, llr = llrs(f, ebn0)
            return np.count_nonzero(
                as_np(decode(llr)).reshape(f, -1) != as_np(ref).reshape(f, -1), axis=1
            )

        nt = 2 if args.lib == "opticommpy-nojit" else sz["timing"]
        e_t = TIMING_EBN0[case]
        t0 = time.perf_counter()
        chain(min(nt, 2), e_t)
        res["first_call_s"] = time.perf_counter() - t0
        if not args.timing_only:
            pts = []
            t0 = time.perf_counter()
            for e in EBN0[case]:
                errs = np.concatenate(
                    [chain(sz["block"], e) for _ in range(sz["frames"] // sz["block"])]
                )
                bound = (
                    float(theory.conv_union_bound(CONV_GEN, 7, K_CONV / (2 * (K_CONV + 6)), e))
                    if case == "conv"
                    else None
                )
                pt = common.frame_ber_point(errs, k_out, None, ebn0_db=e)
                if bound is not None:
                    pt["union_bound"] = bound
                fe = int(np.count_nonzero(errs))
                pt["bler"] = common.rate_point(fe, errs.size)
                if case == "ldpc_same_llr":
                    pt["errors_per_frame"] = errs.astype(int).tolist()
                pts.append(pt)
            res["points"] = pts
            res["sweep_s"] = time.perf_counter() - t0
            res["bits_per_frame"] = k_out
        _, llr = llrs(nt, e_t)
        res["timing"] = common.time_it(lambda llr=llr, decode=decode: decode(llr))
        res["timing"].update(
            units=nt, unit="frame", workload=f"decoder only, {nt} frames at Eb/N0 = {e_t} dB"
        )
        res["timing_chain"] = common.time_it(lambda nt=nt, e_t=e_t, chain=chain: chain(nt, e_t))
        res["timing_chain"].update(
            units=nt, unit="frame", workload=f"whole chain, {nt} frames at Eb/N0 = {e_t} dB"
        )
        out.append(res)
    return out


# ------------------------------------------------------------- specification runner


def spec(ebn0: list[float], frames: int, trials: int, seed: int) -> dict:
    code = {"type": "conv", "k": K_CONV}
    return {
        "name": "bench_conv",
        "seed": seed,
        "batch": trials,
        "grid": {"ebn0_db": ebn0},
        "link": [
            {"type": "fec_encode", "code": code, "blocks": frames // trials},
            {"type": "map", "constellation": {"kind": "psk", "m": 2}},
            {"type": "awgn_ebn0", "ebn0_db": "$ebn0_db"},
            "demap",
            {"type": "fec_decode", "code": code},
        ],
        "metrics": {"report": ["coded_ber", "bler"]},  # coded_ber: decoded information bits
        "stop": {"min_trials": trials, "max_trials": trials},
    }


def spec_points(rows: list[dict]) -> list[dict]:
    pts = []
    for r in rows:
        e = float(r["condition"]["ebn0_db"])
        m = r["metrics"]
        get = lambda d: d["count"] if "count" in d else d["errors"]  # noqa: E731
        # a binomial interval: the runner keeps the totals, not the errors of every frame
        pt = common.rate_point(get(m["coded_ber"]), m["coded_ber"]["total"], None, ebn0_db=e)
        pt["union_bound"] = float(
            theory.conv_union_bound(CONV_GEN, 7, K_CONV / (2 * (K_CONV + 6)), e)
        )
        pt["bler"] = common.rate_point(get(m["bler"]), m["bler"]["total"])
        pts.append(pt)
    return pts


def run_spec_lib(args, todo: list[str], *, rust: bool = False) -> list[dict]:
    """``comnum-spec`` (``comnum.run_spec``), or with ``rust`` ``comnum-rust`` (the binary)."""
    sz = sizes(args.quick)
    trials = 20
    out = []
    for case in todo:
        nt, e_t = sz["timing"], TIMING_EBN0[case]
        workload = f"whole chain, {nt} frames at Eb/N0 = {e_t} dB, {trials} trials"
        res = common.spec_case(
            args,
            {"case": case, "precision": "float64"},
            spec(EBN0[case], sz["frames"], trials, args.seed),
            spec([e_t], nt, trials, args.seed + 1),
            spec_points,
            rust=rust,
            key="timing_chain",
            units=nt,
            unit="frame",
            workload=workload + (" (process wall time)" if rust else ""),
        )
        if not args.timing_only:
            res["bits_per_frame"] = K_CONV
        out.append(res)
    return out


def run_rust_lib(args, todo: list[str]) -> list[dict]:
    return run_spec_lib(args, todo, rust=True)


def postprocess(merged: dict, work) -> dict:
    """ldpc_same_llr: the decoders saw the same LLRs; compare their errors frame by frame
    with those of comnum (same number of bit errors in a frame, same frame errors)."""
    per_lib = {}
    for run in merged["runs"]:
        if run.get("threads") != "1":
            continue
        for case in run.get("cases", []):
            if case["case"] == "ldpc_same_llr" and "points" in case:
                per_lib[run["library"]] = case["points"]
    base = per_lib.get("comnum")
    rows = []
    for lib, pts in per_lib.items():
        if lib == "comnum" or base is None:
            continue
        for p, q in zip(pts, base, strict=True):
            a, b = np.array(p["errors_per_frame"]), np.array(q["errors_per_frame"])
            rows.append(
                {
                    "library": lib,
                    "ebn0_db": p["ebn0_db"],
                    "frames": int(a.size),
                    "same_bit_errors": int(np.count_nonzero(a == b)),
                    "same_frame_outcome": int(np.count_nonzero((a > 0) == (b > 0))),
                }
            )
    return {"same_llr_vs_comnum": rows}


RUNNERS = {
    **{lib: run_codecs for lib in CODECS},
    "comnum-spec": run_spec_lib,
    "comnum-rust": run_rust_lib,
}

if __name__ == "__main__":
    common.main(SCENARIO, CASES, RUNNERS)

Support

Case	comnum (functions)	comnum.run_spec	OptiCommPy (numba)	Sionna
convolutional (133, 171), soft Viterbi	yes	yes	not available: no convolutional code	yes
5G NR LDPC (1024, 2048)	yes	not available: the specification format has no 5G NR LDPC code (NrLdpc: Python and Rust API)	not available: no 5G NR LDPC code (its decoder takes a parity-check matrix: see ldpc_same_llr)	yes
LDPC (1024, 512), same LLRs	yes	not available: the runner draws its own channel (this case feeds the same LLRs to every decoder)	yes	yes

convolutional (133, 171), soft Viterbi: error rates

Library	Precision	Against	max abs(z), BER	max abs(z), FER	Agrees	BER interval
comnum (functions)	float64	—	—	—	—	per-frame
comnum.run_spec	float64	comnum	1.35	1.33	yes	binomial
Sionna	float32 (config.precision = ‘single’)	comnum	1.41	2.53	yes	per-frame

BER and its 95 % interval, every point (2,048,000 bits per point)

Eb/N0 (dB)	union bound	comnum (functions)	comnum.run_spec	Sionna
1	3.05e-1	4.02e-2 [3.91e-2, 4.13e-2]	4.02e-2 [3.99e-2, 4.05e-2]	4.03e-2 [3.91e-2, 4.14e-2]
2	1.65e-2	4.97e-3 [4.64e-3, 5.30e-3]	5.16e-3 [5.06e-3, 5.26e-3]	5.32e-3 [4.97e-3, 5.67e-3]
3	6.27e-4	3.31e-4 [2.59e-4, 4.02e-4]	2.98e-4 [2.76e-4, 3.23e-4]	3.64e-4 [2.94e-4, 4.34e-4]
4	2.02e-5	2.54e-5 [8.46e-6, 4.23e-5]	1.32e-5 [9.06e-6, 1.92e-5]	3.08e-5 [9.04e-6, 5.25e-5]

Frame error rate and its 95 % interval (2,000 frames per point)

Eb/N0 (dB)	comnum (functions)	comnum.run_spec	Sionna
1	9.73e-1 [9.66e-1, 9.80e-1]	9.77e-1 [9.69e-1, 9.83e-1]	9.85e-1 [9.79e-1, 9.89e-1]
2	5.05e-1 [4.84e-1, 5.27e-1]	4.84e-1 [4.63e-1, 5.06e-1]	5.13e-1 [4.91e-1, 5.35e-1]
3	5.95e-2 [5.00e-2, 7.07e-2]	6.05e-2 [5.09e-2, 7.18e-2]	6.70e-2 [5.69e-2, 7.88e-2]
4	6.00e-3 [3.44e-3, 1.05e-2]	5.00e-3 [2.72e-3, 9.18e-3]	5.50e-3 [3.07e-3, 9.82e-3]

5G NR LDPC (1024, 2048): error rates

Library	Precision	Against	max abs(z), BER	max abs(z), FER	Agrees	BER interval
comnum (functions)	float64	—	—	—	—	per-frame
Sionna	float32 (config.precision = ‘single’)	comnum	0.893	1.21	yes	per-frame

BER and its 95 % interval, every point (2,048,000 bits per point)

Eb/N0 (dB)	comnum (functions)	Sionna
1	8.79e-3 [7.46e-3, 1.01e-2]	9.31e-3 [7.94e-3, 1.07e-2]
1.5	3.47e-5 [0, 1.03e-4]	9.38e-5 [0, 2.04e-4]
2	0 [0, 1.46e-6]	0 [0, 1.46e-6]
2.5	0 [0, 1.46e-6]	0 [0, 1.46e-6]

Frame error rate and its 95 % interval (2,000 frames per point)

Eb/N0 (dB)	comnum (functions)	Sionna
1	9.40e-2 [8.20e-2, 1.08e-1]	1.05e-1 [9.28e-2, 1.20e-1]
1.5	5.00e-4 [8.83e-5, 2.83e-3]	1.50e-3 [5.10e-4, 4.40e-3]
2	0 [0, 1.92e-3]	0 [0, 1.92e-3]
2.5	0 [0, 1.92e-3]	0 [0, 1.92e-3]

LDPC (1024, 512), same LLRs: error rates

Library	Precision	Against	max abs(z), BER	max abs(z), FER	Agrees	BER interval
comnum (functions)	float64	—	—	—	—	per-frame
OptiCommPy (numba)	float32 (decodeLDPC default prec = np.float32)	comnum	0.076	0	yes	per-frame
Sionna	float32 (config.precision = ‘single’)	comnum	0.0795	0.0381	yes	per-frame

BER and its 95 % interval, every point (2,048,000 bits per point)

Eb/N0 (dB)	comnum (functions)	OptiCommPy (numba)	Sionna
1.5	1.53e-2 [1.39e-2, 1.66e-2]	1.53e-2 [1.40e-2, 1.67e-2]	1.53e-2 [1.40e-2, 1.67e-2]
2	1.37e-3 [9.56e-4, 1.78e-3]	1.37e-3 [9.55e-4, 1.78e-3]	1.37e-3 [9.56e-4, 1.78e-3]
2.5	0 [0, 1.46e-6]	0 [0, 1.46e-6]	0 [0, 1.46e-6]
3	0 [0, 1.46e-6]	0 [0, 1.46e-6]	0 [0, 1.46e-6]

Frame error rate and its 95 % interval (2,000 frames per point)

Eb/N0 (dB)	comnum (functions)	OptiCommPy (numba)	Sionna
1.5	2.21e-1 [2.03e-1, 2.40e-1]	2.21e-1 [2.03e-1, 2.40e-1]	2.21e-1 [2.04e-1, 2.40e-1]
2	2.30e-2 [1.73e-2, 3.05e-2]	2.30e-2 [1.73e-2, 3.05e-2]	2.30e-2 [1.73e-2, 3.05e-2]
2.5	0 [0, 1.92e-3]	0 [0, 1.92e-3]	0 [0, 1.92e-3]
3	0 [0, 1.92e-3]	0 [0, 1.92e-3]	0 [0, 1.92e-3]

Same LLRs: the decisions frame by frame

The decoders received identical LLRs. Over all the points: the frames where a library makes as many bit errors as comnum, and the frames it decodes (or fails to decode) as comnum does.

Library	Points	Frames	Same bit errors	Same frame outcome
OptiCommPy (numba)	4	8000	7996 (99.95 %)	8000 (100.00 %)
Sionna	4	8000	7981 (99.76 %)	7999 (99.99 %)

convolutional (133, 171), soft Viterbi: decoder time

Median of 5 runs, ms per frame; workload: decoder only, 200 frames at Eb/N0 = 3.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	0.71	0.703	—	1.00	0.00199 s
Sionna	float32 (config.precision = ‘single’)	3.55	2.85	0.8	5.00	0.0987 s

5G NR LDPC (1024, 2048): decoder time

Median of 5 runs, ms per frame; workload: decoder only, 200 frames at Eb/N0 = 2.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	7.81	7.79	—	1.00	0.016 s
Sionna	float32 (config.precision = ‘single’)	32.8	4.73	0.135	4.20	0.0476 s

LDPC (1024, 512), same LLRs: decoder time

Median of 5 runs, ms per frame; workload: decoder only, 200 frames at Eb/N0 = 2.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	1.51	1.51	—	1.00	0.00307 s
OptiCommPy (numba)	float32 (decodeLDPC default prec = np.float32)	7.88	6.99	—	5.23	5.34 s
Sionna	float32 (config.precision = ‘single’)	6.39	0.873	0.114	4.24	0.0188 s

convolutional (133, 171), soft Viterbi: whole-chain time

Median of 5 runs, ms per frame; workload: whole chain, 200 frames at Eb/N0 = 3.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	0.896	0.895	—	1.00	0.00199 s
comnum.run_spec	float64	0.882	0.0723	—	0.984	0.177 s
Sionna	float32 (config.precision = ‘single’)	3.84	3.12	1.31	4.29	0.0987 s

5G NR LDPC (1024, 2048): whole-chain time

Median of 5 runs, ms per frame; workload: whole chain, 200 frames at Eb/N0 = 2.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	7.96	8.20	—	1.00	0.016 s
Sionna	float32 (config.precision = ‘single’)	33.8	4.79	0.132	4.24	0.0476 s

LDPC (1024, 512), same LLRs: whole-chain time

Median of 5 runs, ms per frame; workload: whole chain, 200 frames at Eb/N0 = 2.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	1.59	1.55	—	1.00	0.00307 s
OptiCommPy (numba)	float32 (decodeLDPC default prec = np.float32)	7.46	6.36	—	4.68	5.34 s
Sionna	float32 (config.precision = ‘single’)	6.62	1.09	0.26	4.15	0.0188 s

Split-step Fourier propagation, 16-QAM, 32 GBd, 10 x 80 km#

Scenario 5: single-polarisation split-step Fourier propagation of 32 GBd 16-QAM over 10 x 80 km of standard single-mode fibre, a fixed step of 1 km (80 steps per span), noiseless amplification at the end of every span (gain e^{alpha L}).

Parameters:

fibre: alpha = 0.2 dB/km, D = 17 ps/nm/km, gamma = 1.3 /W/km, fc = 193.1 THz
link: 10 x 80 km, ideal (noiseless) amplification after every span
step: 1 km (800 steps); reference: comnum at 0.05 km (16000 steps)
signal: 16-QAM, 32 GBd, RRC roll-off 0.1, 4 samples per symbol, 4 dBm; 2^14 symbols (quick: 2^11)
timing: one propagation over the 10 spans

Script: benchmarks/libraries/s5_ssfm.py.

The script

"""Scenario 5: single-polarisation split-step Fourier propagation of 32 GBd 16-QAM over
10 x 80 km of standard single-mode fibre, a fixed step of 1 km (80 steps per span),
noiseless amplification at the end of every span (gain e^{alpha L}).

The input field is the same for every library: 16-QAM symbols of comnum (seed 1),
shaped by a root-raised cosine (roll-off 0.1, 4 samples per symbol, 128 GS/s) and
scaled to 4 dBm; it is drawn again, identically, by every process. Every library
propagates it with its own split-step solver (symmetric steps, nonlinear phase over
the whole step in all three); run_all.py compares the output fields (NMSE) with that
of comnum in double precision and with a step-converged reference (comnum, 0.05 km).

Sionna's equation is the complex conjugate of the one of comnum and OptiCommPy
(dE/dz = ... + j beta_2/2 d^2E/dt^2 - j gamma |E|^2 E): its field is compared in their
convention, conj(sionna(conj(x))), and the NMSE of the plain output is reported too.
Sionna propagates one span per call (SSFM) and amplifies with its EDFA of noise figure
0 (no noise). OptiCommPy's ssfm is NumPy only (no numba kernel): its two variants run
the same code. In the mode gpu, OptiCommPy runs its CuPy solver
(optic.models.modelsGPU.ssfm: the same steps, every array on the device; the input is
copied to the device and the output back to the host inside the timed call) in
complex128 and in complex64. Timing: one propagation over the 10 spans, median of 5
runs after a warm-up run.

    python s5_ssfm.py --lib opticommpy --threads 1 [--quick] --out results/_work/s5.json
"""

from __future__ import annotations

import math
import time
from pathlib import Path

import common
import numpy as np

SCENARIO = "s5_ssfm"
TITLE = "Split-step Fourier propagation, 16-QAM, 32 GBd, 10 x 80 km"
CASES = {
    "ssfm": {
        "comnum": True,
        "komm": "no fibre model",
        "opticommpy": True,
        "opticommpy-nojit": True,
        "sionna": True,
    }
}
# The libraries with a GPU path in this scenario (run_all.py: the mode gpu).
GPU_LIBS = {"sionna", "opticommpy"}
ALPHA_DB_KM, D_PS_NM_KM, GAMMA, FC = 0.2, 17.0, 1.3, 193.1e12
SPAN_KM, N_SPANS, STEP_KM = 80.0, 10, 1.0
REF_STEP_KM = 0.05
SYMBOL_RATE, SPS = 32e9, 4
FS = SYMBOL_RATE * SPS
POWER_DBM = 4.0

PARAMETERS = {
    "fibre": "alpha = 0.2 dB/km, D = 17 ps/nm/km, gamma = 1.3 /W/km, fc = 193.1 THz",
    "link": "10 x 80 km, ideal (noiseless) amplification after every span",
    "step": "1 km (800 steps); reference: comnum at 0.05 km (16000 steps)",
    "signal": "16-QAM, 32 GBd, RRC roll-off 0.1, 4 samples per symbol, 4 dBm; 2^14 symbols (quick: 2^11)",
    "timing": "one propagation over the 10 spans",
}


def n_symbols(quick: bool) -> int:
    return 2**11 if quick else 2**14


def input_field(quick: bool) -> np.ndarray:
    import comnum

    rng = comnum.Rng(1)
    c = comnum.Constellation("qam", 16).normalized()
    _, s = comnum.symbols(c, n_symbols(quick), rng)
    h = comnum.pulse_shape("rrc", SPS, n_taps=1025, rolloff=0.1)
    x = comnum.pnorm(comnum.fir_filter(h, comnum.upsample(s, SPS)))
    return x * math.sqrt(comnum.dbm2w(POWER_DBM))


def beta2_ps2_km() -> float:
    c = 299_792_458.0
    lam = c / FC
    return -(D_PS_NM_KM * 1e-6) * lam**2 / (2 * math.pi * c) * 1e24 * 1e3  # s^2/m -> ps^2/km


def prop_comnum(precision: str, step_km: float = STEP_KM):
    import comnum

    f = comnum.FiberParams(
        alpha_db_km=ALPHA_DB_KM,
        d_ps_nm_km=D_PS_NM_KM,
        gamma_w_km=GAMMA,
        span_km=SPAN_KM,
        n_spans=N_SPANS,
        fc_hz=FC,
    )
    link = comnum.Ssfm(f, step_km=step_km, amplification="ideal")
    return lambda x: link.propagate(x, FS, precision=precision)


def prop_opticommpy(gpu: bool = False, prec=np.complex128):
    """OptiCommPy's split-step: the NumPy solver, or its CuPy copy on the GPU
    (``optic.models.modelsGPU.ssfm``, which takes and returns NumPy arrays: the
    transfers to and from the device are inside the timed call)."""
    if gpu:
        from optic.models.modelsGPU import ssfm
    else:
        from optic.models.channels import ssfm
    from optic.utils import parameters

    p = parameters()
    p.Ltotal, p.Lspan, p.hz = SPAN_KM * N_SPANS, SPAN_KM, STEP_KM
    p.alpha, p.D, p.gamma, p.Fc, p.Fs = ALPHA_DB_KM, D_PS_NM_KM, GAMMA, FC, FS
    p.amp, p.prgsBar, p.prec = "ideal", False, prec
    return lambda x: ssfm(x, p)


def prop_sionna(precision: str):
    import torch
    from sionna.phy.channel.optical import EDFA, SSFM

    alpha = ALPHA_DB_KM / (10 * math.log10(math.e))  # 1/km
    span = SSFM(
        alpha=alpha,
        beta_2=beta2_ps2_km(),
        f_c=FC,
        gamma=GAMMA,
        length=SPAN_KM,
        n_ssfm=round(SPAN_KM / STEP_KM),
        sample_duration=1e12 / FS,  # ps
        t_norm=1e-12,
        with_amplification=False,
        precision=precision,
    )
    amp = EDFA(g=math.exp(alpha * SPAN_KM), f=0.0, f_c=FC, dt=1 / FS, precision=precision)
    cdtype = torch.complex64 if precision == "single" else torch.complex128

    def run(x: np.ndarray) -> np.ndarray:
        with torch.inference_mode():
            u = torch.from_numpy(x).to(device=common.TORCH_DEVICE, dtype=cdtype)
            for _ in range(N_SPANS):
                u = amp(span(u))
            return common.to_numpy(u)

    return run


def run_lib(args, todo: list[str]) -> list[dict]:
    if args.lib == "sionna":
        threading = common.setup_torch(args.threads)
    elif args.lib == "opticommpy" and args.threads == "gpu":
        threading = common.setup_cupy()
    else:
        threading = {}
    x = input_field(args.quick)
    work = Path(args.work)
    work.mkdir(parents=True, exist_ok=True)
    if args.lib == "comnum":
        variants = {"f64": (prop_comnum("f64"), "float64"), "f32": (prop_comnum("f32"), "float32")}
    elif args.lib == "sionna":
        variants = {
            "single": (prop_sionna("single"), "float32 (precision = 'single', the default)"),
            "double": (prop_sionna("double"), "float64 (precision = 'double')"),
        }
    elif args.lib == "opticommpy" and args.threads == "gpu":
        import cupy as cp

        variants = {
            "f64": (prop_opticommpy(True, cp.complex128), "float64 (prec = cp.complex128)"),
            "f32": (prop_opticommpy(True, cp.complex64), "float32 (prec = cp.complex64)"),
        }
    else:
        variants = {"f64": (prop_opticommpy(), "float64 (prec = np.complex128)")}
    out = []
    for name, (prop, precision) in variants.items():
        res = {
            "case": "ssfm",
            "variant": name,
            "precision": precision,
            "samples": int(x.size),
            **threading,
        }
        if args.lib == "sionna":
            fwd = lambda prop=prop: np.conj(prop(np.conj(x)))  # noqa: E731
        else:
            fwd = lambda prop=prop: prop(x)  # noqa: E731
        t0 = time.perf_counter()
        y = fwd()
        res["first_call_s"] = time.perf_counter() - t0
        if not args.timing_only:
            np.save(work / f"{SCENARIO}_{args.lib}_{name}.npy", np.asarray(y, dtype=np.complex128))
            if args.lib == "sionna":
                np.save(
                    work / f"{SCENARIO}_{args.lib}_{name}_plain.npy",
                    np.asarray(prop(x), dtype=np.complex128),
                )
        res["timing"] = common.time_it(fwd)
        res["timing"].update(
            units=1,
            unit="propagation",
            workload=f"{x.size} samples, {N_SPANS} x {SPAN_KM:g} km, {STEP_KM:g} km steps",
        )
        out.append(res)
    if args.lib == "comnum" and not args.timing_only:
        t0 = time.perf_counter()
        np.save(work / f"{SCENARIO}_reference.npy", prop_comnum("f64", REF_STEP_KM)(x))
        out.append(
            {
                "case": "ssfm",
                "variant": "reference",
                "precision": "float64",
                "step_km": REF_STEP_KM,
                "wall_s": time.perf_counter() - t0,
            }
        )
    return out


def nmse(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.sum(np.abs(a - b) ** 2) / np.sum(np.abs(b) ** 2))


def postprocess(merged: dict, work: Path) -> dict:
    """The NMSE of every output field against comnum (f64, 1 km) and the reference."""
    fields = {}
    for path in sorted(work.glob(f"{SCENARIO}_*.npy")):
        fields[path.stem.removeprefix(f"{SCENARIO}_")] = np.load(path)
    base, ref = fields.get("comnum_f64"), fields.get("reference")
    rows = []
    for name, y in fields.items():
        if name == "reference":
            continue
        rows.append(
            {
                "field": name,
                "nmse_vs_comnum_f64": nmse(y, base) if base is not None else None,
                "nmse_vs_reference": nmse(y, ref) if ref is not None else None,
            }
        )
    return {"nmse": rows, "reference_step_km": REF_STEP_KM}


RUNNERS = {"comnum": run_lib, "opticommpy": run_lib, "opticommpy-nojit": run_lib, "sionna": run_lib}

if __name__ == "__main__":
    common.main(SCENARIO, CASES, RUNNERS)

Support

Case	comnum (functions)	OptiCommPy (numba)	Sionna
split-step Fourier	yes	yes	yes

Output fields

NMSE of the output field, sum |a - b|^2 / sum |b|^2, against comnum in float64 at the same step and against the step-converged reference (comnum, 0.05 km). The NMSE against the reference is the error of the 1 km step itself.

Field	NMSE vs comnum (float64, 1 km)	NMSE vs reference
comnum, float32	1.14e-8	7.45e-8
comnum, float64	0	3.69e-8
OptiCommPy, float64	2.24e-26	3.69e-8
Sionna, float64	4.17e-26	3.69e-8
Sionna, float64, its own convention (not conjugated)	2.03e0	2.03e0
Sionna, float32	2.61e-8	1.05e-7
Sionna, float32, its own convention (not conjugated)	2.03e0	2.03e0

split-step Fourier: time

Median of 5 runs, s per propagation; workload: 65536 samples, 10 x 80 km, 1 km steps. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions), f64	float64	1.21	1.20	—	1.00	1.21 s
comnum (functions), f32	float32	0.627	0.637	—	0.519	0.673 s
OptiCommPy (numba), f64	float64 (prec = np.complex128)	3.56	3.65	0.0929	2.95	3.87 s
OptiCommPy (numba), f32	float32 (prec = cp.complex64)	—	—	0.0938	—	1.78 s
Sionna, single	float32 (precision = ‘single’, the default)	7.31	1.12	0.167	6.05	7.39 s
Sionna, double	float64 (precision = ‘double’)	6.99	2.26	0.171	5.79	7.37 s

On another machine: rtx4090 (NVIDIA GeForce RTX 4090)#

The same scripts, run on a second machine (run_all.py --results results/rtx4090): Sionna (and, for the split-step, OptiCommPy) on its GPU in the column GPU, and every library on its CPU in the other columns. The Relative column stays the 1-thread time over that of comnum (functions) on 1 thread of this machine.

This machine is a RunPod Secure Cloud pod with one NVIDIA GeForce RTX 4090 (24 GB), the PyTorch template (torch 2.13, CUDA 13.0): a container with a CPU quota on a shared host.

Machine and versions#

CPU	AMD EPYC 7K62 48-Core Processor
Cores	96 logical; 11 available to the process (a container’s CPU quota: the thread pools of the mode all threads are sized to it)
Memory	251.5 GB
GPU	NVIDIA GeForce RTX 4090, 24564 MiB, driver 595.91.07
System	Ubuntu 24.04.5 LTS
Python	3.12.3
Rust	—
comnum commit	`45a7c78f80fc`
Generated	2026-11-06T17:38:20+00:00
comnum	0.3.1
komm	not installed
OptiCommPy	0.11.0
sionna	2.2.0
torch	2.13.0+cu130
numba	0.68.1
numpy	2.5.2
scipy	1.18.1

Uncoded QPSK and 16-QAM over AWGN#

Scenario 1: uncoded Gray QPSK and 16-QAM over AWGN, BER against the exact closed form.

Parameters:

Eb/N0 (dB): 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
bits per point: 10^7 (quick: 2·10^5), in blocks of 10^6 (quick: 10^5)
timing: one block of 10^6 bits (quick: 2·10^5) at Eb/N0 = 6 dB
theory: exact BER of Gray square QAM, Cho & Yoon, IEEE Trans. Commun. 50(7), 2002

Script: benchmarks/libraries/s1_awgn.py.

The script

"""Scenario 1: uncoded Gray QPSK and 16-QAM over AWGN, BER against the exact closed form.

Every library draws its own bits and noise, maps with its own Gray mapper, adds the
noise of its own channel for the Eb/N0 of the point and decides by minimum distance
(Sionna: hard decisions of max-log LLRs, the same decisions). The BER is counted on
the same number of bits per point for every library, in blocks of the same size.
Timing: one block of ``TIMING_BITS`` bits at Eb/N0 = 6 dB (bits, mapping, noise,
decisions, error count), median of 5 runs after a warm-up run.

    python s1_awgn.py --lib comnum --threads 1 [--quick] --out results/_work/s1.json
"""

from __future__ import annotations

import math
import time

import common
import numpy as np
import theory

SCENARIO = "s1_awgn"
TITLE = "Uncoded QPSK and 16-QAM over AWGN"
ORDERS = {"qpsk": 4, "16qam": 16}
ALL = {lib: True for lib in common.LIBRARIES}
CASES = {"qpsk": ALL, "16qam": ALL}
EBN0 = [float(e) for e in range(11)]
TIMING_EBN0 = 6.0
TIMING_TRIALS = 40  # trials of the specification runner per timing run


def sizes(quick: bool) -> dict:
    if quick:
        return {"bits_per_point": 200_000, "block": 100_000, "timing_bits": 200_000, "trials": 10}
    return {
        "bits_per_point": 10_000_000,
        "block": 1_000_000,
        "timing_bits": 1_000_000,
        "trials": 100,
    }


PARAMETERS = {
    "ebn0_db": EBN0,
    "bits_per_point": "10^7 (quick: 2·10^5), in blocks of 10^6 (quick: 10^5)",
    "timing": "one block of 10^6 bits (quick: 2·10^5) at Eb/N0 = 6 dB",
    "theory": "exact BER of Gray square QAM, Cho & Yoon, IEEE Trans. Commun. 50(7), 2002",
}


# ---------------------------------------------------------------------------- chains
# chain(n_bits, ebn0_db) -> bit errors, for one library and one constellation


def chain_comnum(m: int, seed: int):
    import comnum

    c = comnum.Constellation("qam", m).normalized()
    k = int(math.log2(m))
    rng = comnum.Rng(seed)

    def run(n: int, ebn0: float) -> int:
        b = comnum.bits(n, rng)
        y = comnum.awgn_ebn0(c.map_bits(b), ebn0, k, rng)
        return int(np.count_nonzero(c.demap_bits(y) != b))

    return run, "float64"


def chain_komm(m: int, seed: int):
    import komm

    const = komm.QAMConstellation(m)
    lab = komm.ReflectedRectangularLabeling(int(math.log2(m)))
    k = int(math.log2(m))
    es = const.mean_energy()
    rng = np.random.default_rng(seed)

    def run(n: int, ebn0: float) -> int:
        b = rng.integers(0, 2, n)
        x = const.indices_to_symbols(lab.bits_to_indices(b))
        n0 = es / (k * 10 ** (ebn0 / 10))
        y = komm.GaussianChannel(n0, rng=rng).transmit(x)
        bh = lab.indices_to_bits(const.closest_indices(y))
        return int(np.count_nonzero(bh.ravel() != b))

    return run, "float64"


def chain_opticommpy(m: int, seed: int):
    from optic.comm.modulation import demodulateGray, modulateGray
    from optic.models.channels import awgn
    from optic.utils import parameters

    k = int(math.log2(m))
    rng = np.random.default_rng(seed)
    state = {"block": 0}

    def run(n: int, ebn0: float) -> int:
        b = rng.integers(0, 2, n)
        x = modulateGray(b, m, "qam")
        p = parameters()
        p.snr = ebn0 + 10 * math.log10(k)  # Es/N0 (Fs = B = 1)
        p.seed = seed * 1_000_003 + state["block"]  # awgn re-seeds its generator per call
        state["block"] += 1
        y = awgn(x, p)
        bh = demodulateGray(y, m, "qam")
        return int(np.count_nonzero(bh != b))

    return run, "complex64 symbols (its Gray constellation), float64 noise"


def chain_sionna(m: int, seed: int):
    import torch
    from sionna.phy import config
    from sionna.phy.channel import AWGN
    from sionna.phy.mapping import BinarySource, Demapper, Mapper
    from sionna.phy.utils import ebnodb2no

    config.seed = seed
    k = int(math.log2(m))
    source, mapper, channel = BinarySource(), Mapper("qam", k), AWGN()
    demapper = Demapper("maxlog", "qam", k, hard_out=True)

    def run(n: int, ebn0: float) -> int:
        with torch.inference_mode():
            b = source([1, n])
            no = ebnodb2no(ebn0, k, 1.0)
            y = channel(mapper(b), no)
            bh = demapper(y, no)
            return int((bh != b).sum())

    return run, "float32 (config.precision = 'single')"


CHAINS = {
    "comnum": chain_comnum,
    "komm": chain_komm,
    "opticommpy": chain_opticommpy,
    "opticommpy-nojit": chain_opticommpy,
    "sionna": chain_sionna,
}


def run_chains(args, todo: list[str]) -> list[dict]:
    if args.lib == "sionna":
        threading = common.setup_torch(args.threads)
    else:
        threading = {}
    sz = sizes(args.quick)
    out = []
    for case in todo:
        m = ORDERS[case]
        run, precision = CHAINS[args.lib](m, args.seed)
        res = {"case": case, "precision": precision, **threading}
        n = sz["timing_bits"]
        if args.lib == "opticommpy-nojit":
            n = min(n, 100_000)  # plain-Python kernels: a smaller block, the time is per bit
        t0 = time.perf_counter()
        run(n, TIMING_EBN0)  # the first call: JIT compilation (numba), lazy initialisation
        res["first_call_s"] = time.perf_counter() - t0
        if not args.timing_only:
            pts = []
            t0 = time.perf_counter()
            for e in EBN0:
                errors = 0
                for _ in range(sz["bits_per_point"] // sz["block"]):
                    errors += run(sz["block"], e)
                pts.append(
                    common.rate_point(
                        errors, sz["bits_per_point"], float(theory.ber_qam_awgn(m, e)), ebn0_db=e
                    )
                )
            res["points"] = pts
            res["sweep_s"] = time.perf_counter() - t0
        last = {}
        res["timing"] = common.time_it(
            lambda n=n, run=run, last=last: last.__setitem__("e", run(n, TIMING_EBN0))
        )
        res["timing"].update(units=n, unit="bit", workload=f"{n} bits at Eb/N0 = {TIMING_EBN0} dB")
        res["timing_ber"] = last["e"] / n
        out.append(res)
    return out


# ------------------------------------------------------------- specification runner


def spec(m: int, ebn0: list[float], bits: int, trials: int, seed: int) -> dict:
    k = int(math.log2(m))
    return {
        "name": f"bench_awgn_{m}qam",
        "seed": seed,
        "batch": trials,
        "grid": {"ebn0_db": ebn0},
        "link": [
            {
                "type": "source",
                "constellation": {"kind": "qam", "m": m},
                "n_symbols": bits // (k * trials),
            },
            {"type": "awgn_ebn0", "ebn0_db": "$ebn0_db"},
        ],
        "metrics": {"report": ["ber"], "scale": "none"},
        "stop": {"min_trials": trials, "max_trials": trials},
    }


def spec_points(results: list[dict], m: int) -> list[dict]:
    pts = []
    for r in results:
        e = float(r["condition"]["ebn0_db"])
        ber = r["metrics"]["ber"]
        errors = ber["count"] if "count" in ber else ber["errors"]  # run_spec / the binary's report
        pts.append(
            common.rate_point(errors, ber["total"], float(theory.ber_qam_awgn(m, e)), ebn0_db=e)
        )
    return pts


def run_spec_lib(args, todo: list[str], *, rust: bool = False) -> list[dict]:
    """``comnum-spec`` (``comnum.run_spec``), or with ``rust`` ``comnum-rust`` (the binary)."""
    sz = sizes(args.quick)
    n = sz["timing_bits"]
    workload = f"{n} bits at Eb/N0 = {TIMING_EBN0} dB, {TIMING_TRIALS} trials"
    if rust:
        workload += " (process wall time, start-up included)"
    out = []
    for case in todo:
        m = ORDERS[case]
        out.append(
            common.spec_case(
                args,
                {"case": case, "precision": "float64"},
                spec(m, EBN0, sz["bits_per_point"], sz["trials"], args.seed),
                spec(m, [TIMING_EBN0], n, TIMING_TRIALS, args.seed + 1),
                lambda rows, m=m: spec_points(rows, m),
                rust=rust,
                units=n,
                unit="bit",
                workload=workload,
            )
        )
    return out


def run_rust_lib(args, todo: list[str]) -> list[dict]:
    return run_spec_lib(args, todo, rust=True)


RUNNERS = {
    **{lib: run_chains for lib in CHAINS},
    "comnum-spec": run_spec_lib,
    "comnum-rust": run_rust_lib,
}

if __name__ == "__main__":
    common.main(SCENARIO, CASES, RUNNERS)

Support

Case	comnum (functions)	comnum.run_spec	OptiCommPy (numba)	Sionna
QPSK	yes	yes	yes	yes
16-QAM	yes	yes	yes	yes

QPSK: error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	theory	1.90	yes	binomial
comnum.run_spec	float64	theory	2.53	yes	binomial
OptiCommPy (numba)	complex64 symbols (its Gray constellation), float64 noise	theory	3.31	yes	binomial
Sionna	float32 (config.precision = ‘single’)	theory	2.13	yes	binomial

BER and its 95 % interval, every point (10,000,000 bits per point)

Eb/N0 (dB)	theory	comnum (functions)	comnum.run_spec	OptiCommPy (numba)	Sionna
0	7.86e-2	7.86e-2 [7.84e-2, 7.88e-2]	7.87e-2 [7.85e-2, 7.89e-2]	7.86e-2 [7.84e-2, 7.88e-2]	7.86e-2 [7.84e-2, 7.88e-2]
1	5.63e-2	5.63e-2 [5.62e-2, 5.65e-2]	5.63e-2 [5.61e-2, 5.64e-2]	5.63e-2 [5.61e-2, 5.64e-2]	5.61e-2 [5.60e-2, 5.63e-2]
2	3.75e-2	3.75e-2 [3.74e-2, 3.76e-2]	3.74e-2 [3.73e-2, 3.76e-2]	3.75e-2 [3.73e-2, 3.76e-2]	3.76e-2 [3.75e-2, 3.77e-2]
3	2.29e-2	2.28e-2 [2.27e-2, 2.29e-2]	2.30e-2 [2.29e-2, 2.31e-2]	2.29e-2 [2.28e-2, 2.30e-2]	2.29e-2 [2.28e-2, 2.30e-2]
4	1.25e-2	1.25e-2 [1.25e-2, 1.26e-2]	1.25e-2 [1.24e-2, 1.26e-2]	1.25e-2 [1.24e-2, 1.26e-2]	1.25e-2 [1.25e-2, 1.26e-2]
5	5.95e-3	5.95e-3 [5.90e-3, 6.00e-3]	5.96e-3 [5.92e-3, 6.01e-3]	5.93e-3 [5.89e-3, 5.98e-3]	5.93e-3 [5.88e-3, 5.98e-3]
6	2.39e-3	2.39e-3 [2.36e-3, 2.42e-3]	2.41e-3 [2.38e-3, 2.44e-3]	2.43e-3 [2.40e-3, 2.46e-3]	2.39e-3 [2.36e-3, 2.42e-3]
7	7.73e-4	7.56e-4 [7.39e-4, 7.73e-4]	7.77e-4 [7.60e-4, 7.95e-4]	8.02e-4 [7.84e-4, 8.20e-4]	7.88e-4 [7.70e-4, 8.05e-4]
8	1.91e-4	1.84e-4 [1.76e-4, 1.92e-4]	1.93e-4 [1.84e-4, 2.01e-4]	1.90e-4 [1.82e-4, 1.99e-4]	1.82e-4 [1.73e-4, 1.90e-4]
9	3.36e-5	3.21e-5 [2.88e-5, 3.58e-5]	3.41e-5 [3.07e-5, 3.79e-5]	3.42e-5 [3.08e-5, 3.80e-5]	3.57e-5 [3.22e-5, 3.96e-5]
10	3.87e-6	4.00e-6 [2.94e-6, 5.45e-6]	3.20e-6 [2.27e-6, 4.52e-6]	4.10e-6 [3.02e-6, 5.56e-6]	3.70e-6 [2.68e-6, 5.10e-6]

16-QAM: error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	theory	2.15	yes	binomial
comnum.run_spec	float64	theory	1.64	yes	binomial
OptiCommPy (numba)	complex64 symbols (its Gray constellation), float64 noise	theory	1.42	yes	binomial
Sionna	float32 (config.precision = ‘single’)	theory	2.41	yes	binomial

BER and its 95 % interval, every point (10,000,000 bits per point)

Eb/N0 (dB)	theory	comnum (functions)	comnum.run_spec	OptiCommPy (numba)	Sionna
0	1.41e-1	1.41e-1 [1.41e-1, 1.41e-1]	1.41e-1 [1.41e-1, 1.41e-1]	1.41e-1 [1.41e-1, 1.41e-1]	1.41e-1 [1.41e-1, 1.41e-1]
1	1.19e-1	1.19e-1 [1.19e-1, 1.19e-1]	1.19e-1 [1.19e-1, 1.19e-1]	1.19e-1 [1.19e-1, 1.19e-1]	1.19e-1 [1.19e-1, 1.19e-1]
2	9.77e-2	9.78e-2 [9.76e-2, 9.80e-2]	9.77e-2 [9.75e-2, 9.79e-2]	9.78e-2 [9.76e-2, 9.80e-2]	9.75e-2 [9.73e-2, 9.77e-2]
3	7.75e-2	7.75e-2 [7.74e-2, 7.77e-2]	7.74e-2 [7.73e-2, 7.76e-2]	7.75e-2 [7.73e-2, 7.76e-2]	7.74e-2 [7.73e-2, 7.76e-2]
4	5.86e-2	5.87e-2 [5.85e-2, 5.88e-2]	5.86e-2 [5.84e-2, 5.87e-2]	5.85e-2 [5.84e-2, 5.87e-2]	5.85e-2 [5.84e-2, 5.87e-2]
5	4.19e-2	4.18e-2 [4.17e-2, 4.19e-2]	4.19e-2 [4.17e-2, 4.20e-2]	4.20e-2 [4.19e-2, 4.21e-2]	4.20e-2 [4.19e-2, 4.21e-2]
6	2.79e-2	2.79e-2 [2.78e-2, 2.80e-2]	2.79e-2 [2.78e-2, 2.80e-2]	2.78e-2 [2.77e-2, 2.79e-2]	2.79e-2 [2.78e-2, 2.81e-2]
7	1.70e-2	1.70e-2 [1.69e-2, 1.71e-2]	1.70e-2 [1.69e-2, 1.71e-2]	1.70e-2 [1.69e-2, 1.70e-2]	1.70e-2 [1.69e-2, 1.71e-2]
8	9.25e-3	9.24e-3 [9.18e-3, 9.30e-3]	9.24e-3 [9.18e-3, 9.30e-3]	9.22e-3 [9.16e-3, 9.28e-3]	9.24e-3 [9.19e-3, 9.30e-3]
9	4.39e-3	4.35e-3 [4.30e-3, 4.39e-3]	4.42e-3 [4.38e-3, 4.47e-3]	4.40e-3 [4.36e-3, 4.44e-3]	4.37e-3 [4.33e-3, 4.41e-3]
10	1.75e-3	1.75e-3 [1.73e-3, 1.78e-3]	1.78e-3 [1.75e-3, 1.80e-3]	1.76e-3 [1.73e-3, 1.79e-3]	1.76e-3 [1.73e-3, 1.78e-3]

QPSK: time

Median of 5 runs, ns per bit; workload: 1000000 bits at Eb/N0 = 6.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	50.4	50.7	—	1.00	0.0588 s
comnum.run_spec	float64	42.2	8.56	—	0.836	0.0421 s
OptiCommPy (numba)	complex64 symbols (its Gray constellation), float64 noise	105	73.0	—	2.09	3.3 s
Sionna	float32 (config.precision = ‘single’)	123	48.4	0.748	2.44	0.148 s

16-QAM: time

Median of 5 runs, ns per bit; workload: 1000000 bits at Eb/N0 = 6.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	27.1	27.2	—	1.00	0.0273 s
comnum.run_spec	float64	29.0	5.16	—	1.07	0.0271 s
OptiCommPy (numba)	complex64 symbols (its Gray constellation), float64 noise	100	51.0	—	3.71	0.108 s
Sionna	float32 (config.precision = ‘single’)	219	63.1	0.832	8.10	0.25 s

BPSK and QPSK over flat Rayleigh fading#

Scenario 2: BPSK and Gray QPSK over flat Rayleigh fading (one coefficient per symbol, perfectly known at the receiver), BER against the closed form.

Parameters:

Eb/N0 (dB): 0, 5, 10, 15, 20, 25, 30
bits per point: 10^7 (quick: 2·10^5), in blocks of 10^6 (quick: 10^5)
channel: y = h x + w, h ~ CN(0, 1) independent for every symbol, known at the receiver
timing: one block of 10^6 bits (quick: 2·10^5) at Eb/N0 = 10 dB
theory: BPSK: (1 - sqrt(G / (1 + G))) / 2; Gray QPSK: the same per bit (Proakis & Salehi, 5th ed., Sec. 13.3)

Script: benchmarks/libraries/s2_rayleigh.py.

The script

"""Scenario 2: BPSK and Gray QPSK over flat Rayleigh fading (one coefficient per
symbol, perfectly known at the receiver), BER against the closed form.

Every library draws its own bits, channel coefficients h ~ CN(0, 1) and noise, and
detects with its own coherent detector given h (zero forcing over the 1 x 1 channel:
comnum.mimo_detect, Sionna's LinearDetector, the mimo_rx block of the specification).
komm and OptiCommPy have no fading channel. Timing: one block of ``TIMING_BITS`` bits
at Eb/N0 = 10 dB, median of 5 runs after a warm-up run.

    python s2_rayleigh.py --lib sionna --threads 1 [--quick] --out results/_work/s2.json
"""

from __future__ import annotations

import math
import time

import common
import numpy as np
import theory

SCENARIO = "s2_rayleigh"
TITLE = "BPSK and QPSK over flat Rayleigh fading"
ORDERS = {"bpsk": 2, "qpsk": 4}
SUPPORT = {
    "comnum": True,
    "comnum-spec": True,
    "komm": "no fading channel",
    "opticommpy": "no fading channel",
    "opticommpy-nojit": "no fading channel",
    "sionna": True,
}
CASES = {"bpsk": SUPPORT, "qpsk": SUPPORT}
EBN0 = [0.0, 5.0, 10.0, 15.0, 20.0, 25.0, 30.0]
TIMING_EBN0 = 10.0
TIMING_TRIALS = 40

PARAMETERS = {
    "ebn0_db": EBN0,
    "bits_per_point": "10^7 (quick: 2·10^5), in blocks of 10^6 (quick: 10^5)",
    "channel": "y = h x + w, h ~ CN(0, 1) independent for every symbol, known at the receiver",
    "timing": "one block of 10^6 bits (quick: 2·10^5) at Eb/N0 = 10 dB",
    "theory": "BPSK: (1 - sqrt(G / (1 + G))) / 2; Gray QPSK: the same per bit (Proakis & Salehi, 5th ed., Sec. 13.3)",
}


def sizes(quick: bool) -> dict:
    if quick:
        return {"bits_per_point": 200_000, "block": 100_000, "timing_bits": 200_000, "trials": 10}
    return {
        "bits_per_point": 10_000_000,
        "block": 1_000_000,
        "timing_bits": 1_000_000,
        "trials": 100,
    }


def chain_comnum(m: int, seed: int):
    import comnum

    c = comnum.Constellation("psk" if m == 2 else "qam", m).normalized()
    k = int(math.log2(m))
    rng = comnum.Rng(seed)

    def run(n: int, ebn0: float) -> int:
        b = comnum.bits(n, rng)
        x = c.map_bits(b)
        h = comnum.mimo_channels(x.size, 1, 1, rng)
        n0 = 1.0 / (k * 10 ** (ebn0 / 10))
        y = h[:, 0, 0] * x + rng.complex_noise(x.size, n0)
        labels = comnum.mimo_detect(h, y[None, :], c, n0, "zf")
        bh = comnum.unpack_symbols(labels.ravel(), k)
        return int(np.count_nonzero(bh != b))

    return run, "float64"


def chain_sionna(m: int, seed: int):
    import torch
    from sionna.phy import config
    from sionna.phy.channel import FlatFadingChannel
    from sionna.phy.mapping import BinarySource, Mapper
    from sionna.phy.mimo import LinearDetector
    from sionna.phy.utils import ebnodb2no

    config.seed = seed
    k = int(math.log2(m))
    kind = "pam" if m == 2 else "qam"
    source, mapper = BinarySource(), Mapper(kind, k)
    channel = FlatFadingChannel(1, 1, return_channel=True)
    detector = LinearDetector(
        "zf", "bit", "maxlog", constellation_type=kind, num_bits_per_symbol=k, hard_out=True
    )

    def run(n: int, ebn0: float) -> int:
        with torch.inference_mode():
            b = source([n // k, k])
            x = mapper(b)  # [n/k, 1]
            no = ebnodb2no(ebn0, k, 1.0)
            y, h = channel(x, no)
            s = torch.full((n // k, 1, 1), float(no), dtype=y.dtype)
            bh = detector(y, h, s)  # [n/k, 1, k]
            return int((bh.reshape(b.shape) != b).sum())

    return run, "float32 (config.precision = 'single')"


CHAINS = {"comnum": chain_comnum, "sionna": chain_sionna}


def run_chains(args, todo: list[str]) -> list[dict]:
    threading = common.setup_torch(args.threads) if args.lib == "sionna" else {}
    sz = sizes(args.quick)
    out = []
    for case in todo:
        m = ORDERS[case]
        run, precision = CHAINS[args.lib](m, args.seed)
        res = {"case": case, "precision": precision, **threading}
        n = sz["timing_bits"]
        t0 = time.perf_counter()
        run(n, TIMING_EBN0)
        res["first_call_s"] = time.perf_counter() - t0
        if not args.timing_only:
            pts = []
            t0 = time.perf_counter()
            for e in EBN0:
                errors = sum(
                    run(sz["block"], e) for _ in range(sz["bits_per_point"] // sz["block"])
                )
                pts.append(
                    common.rate_point(
                        errors,
                        sz["bits_per_point"],
                        float(theory.ber_qam_rayleigh(m, e)),
                        ebn0_db=e,
                    )
                )
            res["points"] = pts
            res["sweep_s"] = time.perf_counter() - t0
        res["timing"] = common.time_it(lambda n=n, run=run: run(n, TIMING_EBN0))
        res["timing"].update(units=n, unit="bit", workload=f"{n} bits at Eb/N0 = {TIMING_EBN0} dB")
        out.append(res)
    return out


def spec(m: int, ebn0: list[float], bits: int, trials: int, seed: int) -> dict:
    k = int(math.log2(m))
    return {
        "name": f"bench_rayleigh_{m}",
        "seed": seed,
        "batch": trials,
        "grid": {"ebn0_db": ebn0},
        "link": [
            {
                "type": "source",
                "constellation": {"kind": "psk" if m == 2 else "qam", "m": m},
                "n_symbols": bits // (k * trials),
            },
            {"type": "rayleigh", "n_rx": 1, "coherence_samples": 1},
            {"type": "awgn_ebn0", "ebn0_db": "$ebn0_db"},
            {"type": "mimo_rx", "detector": "zf"},
        ],
        "metrics": {"report": ["ber"], "scale": "none"},
        "stop": {"min_trials": trials, "max_trials": trials},
    }


def spec_points(rows: list[dict], m: int) -> list[dict]:
    pts = []
    for row in rows:
        e = float(row["condition"]["ebn0_db"])
        ber = row["metrics"]["ber"]
        pts.append(
            common.rate_point(
                ber["count"], ber["total"], float(theory.ber_qam_rayleigh(m, e)), ebn0_db=e
            )
        )
    return pts


def run_spec_lib(args, todo: list[str]) -> list[dict]:
    sz = sizes(args.quick)
    n = sz["timing_bits"]
    out = []
    for case in todo:
        m = ORDERS[case]
        out.append(
            common.spec_case(
                args,
                {"case": case, "precision": "float64"},
                spec(m, EBN0, sz["bits_per_point"], sz["trials"], args.seed),
                spec(m, [TIMING_EBN0], n, TIMING_TRIALS, args.seed + 1),
                lambda rows, m=m: spec_points(rows, m),
                units=n,
                unit="bit",
                workload=f"{n} bits at Eb/N0 = {TIMING_EBN0} dB, {TIMING_TRIALS} trials",
            )
        )
    return out


RUNNERS = {"comnum": run_chains, "sionna": run_chains, "comnum-spec": run_spec_lib}

if __name__ == "__main__":
    common.main(SCENARIO, CASES, RUNNERS)

Support

Case	comnum (functions)	comnum.run_spec	OptiCommPy (numba)	Sionna
BPSK	yes	yes	not available: no fading channel	yes
QPSK	yes	yes	not available: no fading channel	yes

BPSK: error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	theory	2.22	yes	binomial
comnum.run_spec	float64	theory	1.78	yes	binomial
Sionna	float32 (config.precision = ‘single’)	theory	2.35	yes	binomial

BER and its 95 % interval, every point (10,000,000 bits per point)

Eb/N0 (dB)	theory	comnum (functions)	comnum.run_spec	Sionna
0	1.46e-1	1.46e-1 [1.46e-1, 1.47e-1]	1.46e-1 [1.46e-1, 1.47e-1]	1.47e-1 [1.46e-1, 1.47e-1]
5	6.42e-2	6.43e-2 [6.41e-2, 6.44e-2]	6.42e-2 [6.41e-2, 6.44e-2]	6.42e-2 [6.40e-2, 6.43e-2]
10	2.33e-2	2.33e-2 [2.32e-2, 2.34e-2]	2.32e-2 [2.32e-2, 2.33e-2]	2.33e-2 [2.32e-2, 2.34e-2]
15	7.72e-3	7.73e-3 [7.67e-3, 7.78e-3]	7.76e-3 [7.71e-3, 7.82e-3]	7.75e-3 [7.70e-3, 7.80e-3]
20	2.48e-3	2.51e-3 [2.48e-3, 2.54e-3]	2.46e-3 [2.43e-3, 2.49e-3]	2.52e-3 [2.49e-3, 2.55e-3]
25	7.89e-4	8.08e-4 [7.91e-4, 8.26e-4]	7.83e-4 [7.66e-4, 8.00e-4]	7.88e-4 [7.71e-4, 8.06e-4]
30	2.50e-4	2.52e-4 [2.42e-4, 2.62e-4]	2.41e-4 [2.31e-4, 2.51e-4]	2.43e-4 [2.34e-4, 2.53e-4]

QPSK: error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	theory	2.08	yes	binomial
comnum.run_spec	float64	theory	1.17	yes	binomial
Sionna	float32 (config.precision = ‘single’)	theory	3.13	yes	binomial

BER and its 95 % interval, every point (10,000,000 bits per point)

Eb/N0 (dB)	theory	comnum (functions)	comnum.run_spec	Sionna
0	1.46e-1	1.46e-1 [1.46e-1, 1.46e-1]	1.46e-1 [1.46e-1, 1.47e-1]	1.46e-1 [1.46e-1, 1.47e-1]
5	6.42e-2	6.41e-2 [6.39e-2, 6.42e-2]	6.42e-2 [6.40e-2, 6.43e-2]	6.42e-2 [6.40e-2, 6.43e-2]
10	2.33e-2	2.34e-2 [2.33e-2, 2.34e-2]	2.33e-2 [2.32e-2, 2.34e-2]	2.31e-2 [2.30e-2, 2.32e-2]
15	7.72e-3	7.77e-3 [7.72e-3, 7.82e-3]	7.73e-3 [7.67e-3, 7.78e-3]	7.67e-3 [7.61e-3, 7.72e-3]
20	2.48e-3	2.51e-3 [2.48e-3, 2.54e-3]	2.50e-3 [2.47e-3, 2.53e-3]	2.46e-3 [2.43e-3, 2.50e-3]
25	7.89e-4	8.02e-4 [7.84e-4, 8.19e-4]	7.81e-4 [7.64e-4, 7.99e-4]	7.92e-4 [7.75e-4, 8.10e-4]
30	2.50e-4	2.53e-4 [2.44e-4, 2.63e-4]	2.46e-4 [2.37e-4, 2.56e-4]	2.51e-4 [2.41e-4, 2.61e-4]

BPSK: time

Median of 5 runs, ns per bit; workload: 1000000 bits at Eb/N0 = 10.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	303	295	—	1.00	0.317 s
comnum.run_spec	float64	322	48.9	—	1.06	0.347 s
Sionna	float32 (config.precision = ‘single’)	437	487	8.61	1.44	0.447 s

QPSK: time

Median of 5 runs, ns per bit; workload: 1000000 bits at Eb/N0 = 10.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	156	151	—	1.00	0.156 s
comnum.run_spec	float64	166	25.5	—	1.06	0.162 s
Sionna	float32 (config.precision = ‘single’)	289	298	3.67	1.86	0.32 s

4 x 4 MIMO, 16-QAM, i.i.d. Rayleigh fading: ZF, MMSE and ML detection#

Scenario 3: 4 x 4 spatial multiplexing of Gray 16-QAM over i.i.d. Rayleigh fading (one channel matrix per vector, known at the receiver), detected by zero forcing, MMSE and maximum likelihood.

Parameters:

Eb/N0 (dB): 0, 4, 8, 12, 16
vectors per point: linear detectors and sphere: 10^5 (quick: 5·10^3); exhaustive ML: 2·10^4 (quick: 10^3)
timing: 10^4 vectors at Eb/N0 = 12 dB (exhaustive ML: 10^3; quick: 2·10^3 and 2·10^2)
theory: ZF: 16-QAM over SISO Rayleigh fading (Winters, Salz & Gitlin, IEEE Trans. Commun. 42(2/3/4), 1994)

Script: benchmarks/libraries/s3_mimo.py.

The script

"""Scenario 3: 4 x 4 spatial multiplexing of Gray 16-QAM over i.i.d. Rayleigh fading
(one channel matrix per vector, known at the receiver), detected by zero forcing, MMSE
and maximum likelihood.

Every stream carries unit-energy symbols (no power split between the antennas) and
Eb/N0 is the energy per bit received on one antenna over N0: N0 = 1 / (4 Eb/N0).
Zero forcing over 4 x 4 leaves one diversity branch per stream, so its BER is that of
16-QAM over SISO Rayleigh fading (Winters, Salz & Gitlin 1994); MMSE and ML have no
closed form here and are compared between the libraries. Maximum likelihood is an
exhaustive search over the 16^4 = 65536 vectors in Sionna and in comnum ("ml"), and
the Schnorr-Euchner sphere decoder in comnum ("sphere", the same decisions).
The specification runner splits the power between the antennas (mimo_tx), so it runs
at Eb/N0 + 10 log10(4) dB to have the same SNR. Timing: ``TIMING_VECTORS`` vectors at
Eb/N0 = 12 dB (channel, noise, detection), median of 5 runs after a warm-up run.

    python s3_mimo.py --lib comnum --threads 1 [--quick] --out results/_work/s3.json
"""

from __future__ import annotations

import math
import time

import common
import numpy as np
import theory

SCENARIO = "s3_mimo"
TITLE = "4 x 4 MIMO, 16-QAM, i.i.d. Rayleigh fading: ZF, MMSE and ML detection"
NT = NR = 4
M = 16
K = 4
NO_MIMO = "no MIMO detection"
SUPPORT = {
    "comnum": True,
    "comnum-spec": True,
    "komm": NO_MIMO,
    "opticommpy": "no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)",
    "opticommpy-nojit": "no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)",
    "sionna": True,
}
CASES = {
    "zf": SUPPORT,
    "mmse": SUPPORT,
    "ml": SUPPORT,
    "sphere": {
        **SUPPORT,
        "sionna": "no sphere decoder (its exhaustive ML detector is the case ml)",
    },
}
SPEC_DETECTOR = {"zf": "zf", "mmse": "mmse", "ml": "ml", "sphere": "sphere"}
EBN0 = [0.0, 4.0, 8.0, 12.0, 16.0]
TIMING_EBN0 = 12.0
SPEC_OFFSET_DB = 10 * math.log10(NT)

PARAMETERS = {
    "ebn0_db": EBN0,
    "vectors_per_point": "linear detectors and sphere: 10^5 (quick: 5·10^3); exhaustive ML: 2·10^4 (quick: 10^3)",
    "timing": "10^4 vectors at Eb/N0 = 12 dB (exhaustive ML: 10^3; quick: 2·10^3 and 2·10^2)",
    "theory": "ZF: 16-QAM over SISO Rayleigh fading (Winters, Salz & Gitlin, IEEE Trans. Commun. 42(2/3/4), 1994)",
}


def sizes(quick: bool, case: str) -> dict:
    heavy = case == "ml"
    if quick:
        return {
            "vectors": 1_000 if heavy else 5_000,
            "timing": 200 if heavy else 2_000,
            "blocks": 1,
        }
    return {
        "vectors": 20_000 if heavy else 100_000,
        "timing": 1_000 if heavy else 10_000,
        "blocks": 10,
    }


def theory_ber(case: str, ebn0: float) -> float | None:
    return float(theory.ber_qam_rayleigh(M, ebn0)) if case == "zf" else None


def chain_comnum(case: str, seed: int):
    import comnum

    c = comnum.Constellation("qam", M).normalized()
    rng = comnum.Rng(seed)

    def run(n: int, ebn0: float) -> np.ndarray:
        b = comnum.bits(n * NT * K, rng)
        x = c.map_bits(b).reshape(n, NT)
        h = comnum.mimo_channels(n, NR, NT, rng)
        n0 = 1.0 / (K * 10 ** (ebn0 / 10))
        # y = H x + w (the Python API draws H; the product is NumPy's)
        y = np.matmul(h, x[:, :, None])[:, :, 0] + rng.complex_noise(n * NR, n0).reshape(n, NR)
        labels = comnum.mimo_detect(h, y.T, c, n0, case)  # stream-major (NT, n)
        bh = comnum.unpack_symbols(labels.T.ravel(), K)
        return np.count_nonzero((bh != b).reshape(n, NT * K), axis=1)  # bit errors per vector

    return run, "float64"


def chain_sionna(case: str, seed: int):
    import torch
    from sionna.phy import config
    from sionna.phy.channel import FlatFadingChannel
    from sionna.phy.mapping import BinarySource, Mapper
    from sionna.phy.mimo import LinearDetector, MaximumLikelihoodDetector
    from sionna.phy.utils import ebnodb2no

    config.seed = seed
    source, mapper = BinarySource(), Mapper("qam", K)
    channel = FlatFadingChannel(NT, NR, return_channel=True)
    if case == "ml":
        det = MaximumLikelihoodDetector(
            "bit", "maxlog", NT, constellation_type="qam", num_bits_per_symbol=K, hard_out=True
        )
        chunk = 64  # vectors per call: the search holds 65536 candidates per vector
    else:
        eq = {"zf": "zf", "mmse": "lmmse"}[case]
        det = LinearDetector(
            eq, "bit", "maxlog", constellation_type="qam", num_bits_per_symbol=K, hard_out=True
        )
        chunk = None

    def run(n: int, ebn0: float) -> np.ndarray:
        with torch.inference_mode():
            b = source([n, NT * K])
            x = mapper(b)  # [n, NT]
            no = ebnodb2no(ebn0, K, 1.0)  # Es / (K Eb/N0), Es = 1
            y, h = channel(x, no)
            s = (float(no) * torch.eye(NR, dtype=y.dtype)).expand(n, NR, NR)
            if chunk is None:
                bh = det(y, h, s)
            else:
                bh = torch.cat(
                    [
                        det(y[i : i + chunk], h[i : i + chunk], s[i : i + chunk])
                        for i in range(0, n, chunk)
                    ]
                )
            return common.to_numpy((bh.reshape(b.shape) != b).sum(dim=1))  # bit errors per vector

    return run, "float32 (config.precision = 'single')"


CHAINS = {"comnum": chain_comnum, "sionna": chain_sionna}


def run_chains(args, todo: list[str]) -> list[dict]:
    threading = common.setup_torch(args.threads) if args.lib == "sionna" else {}
    out = []
    for case in todo:
        sz = sizes(args.quick, case)
        run, precision = CHAINS[args.lib](case, args.seed)
        res = {"case": case, "precision": precision, **threading}
        n = sz["timing"]
        t0 = time.perf_counter()
        run(n, TIMING_EBN0)
        res["first_call_s"] = time.perf_counter() - t0
        if not args.timing_only:
            pts = []
            t0 = time.perf_counter()
            for e in EBN0:
                # the 16 bits of a vector share its channel: the interval is per vector
                errs = np.concatenate(
                    [run(sz["vectors"] // sz["blocks"], e) for _ in range(sz["blocks"])]
                )
                pts.append(
                    common.frame_ber_point(
                        errs, NT * K, theory_ber(case, e), ebn0_db=e, vectors=errs.size
                    )
                )
            res["points"] = pts
            res["sweep_s"] = time.perf_counter() - t0
        res["timing"] = common.time_it(lambda n=n, run=run: run(n, TIMING_EBN0))
        res["timing"].update(
            units=n, unit="vector", workload=f"{n} vectors at Eb/N0 = {TIMING_EBN0} dB"
        )
        out.append(res)
    return out


def spec(case: str, ebn0: list[float], vectors: int, trials: int, seed: int) -> dict:
    return {
        "name": f"bench_mimo_{case}",
        "seed": seed,
        "batch": trials,
        "grid": {"ebn0_db": [e + SPEC_OFFSET_DB for e in ebn0]},
        "link": [
            {"type": "source", "constellation": "16-QAM", "n_symbols": NT * vectors // trials},
            {"type": "mimo_tx", "n_tx": NT, "scheme": "spatial"},
            {"type": "rayleigh", "n_rx": NR, "coherence_samples": 1},
            {"type": "awgn_ebn0", "ebn0_db": "$ebn0_db"},
            {"type": "mimo_rx", "detector": SPEC_DETECTOR[case]},
        ],
        "metrics": {"report": ["ber"], "scale": "none"},
        "stop": {"min_trials": trials, "max_trials": trials},
    }


def run_spec_lib(args, todo: list[str]) -> list[dict]:
    trials = 40
    out = []
    for case in todo:
        sz = sizes(args.quick, case)

        def points(rows: list[dict], case=case, vectors=sz["vectors"]) -> list[dict]:
            pts = []
            for e, row in zip(EBN0, rows, strict=True):
                ber = row["metrics"]["ber"]
                pts.append(
                    common.rate_point(
                        ber["count"], ber["total"], theory_ber(case, e), ebn0_db=e, vectors=vectors
                    )
                )
            return pts

        n = sz["timing"]
        out.append(
            common.spec_case(
                args,
                {"case": case, "precision": "float64", "ebn0_offset_db": SPEC_OFFSET_DB},
                spec(case, EBN0, sz["vectors"], trials, args.seed),
                spec(case, [TIMING_EBN0], n, trials, args.seed + 1),
                points,
                units=n,
                unit="vector",
                workload=f"{n} vectors at Eb/N0 = {TIMING_EBN0} dB, {trials} trials",
            )
        )
    return out


RUNNERS = {"comnum": run_chains, "sionna": run_chains, "comnum-spec": run_spec_lib}

if __name__ == "__main__":
    common.main(SCENARIO, CASES, RUNNERS)

Support

Case	comnum (functions)	comnum.run_spec	OptiCommPy (numba)	Sionna
zero forcing	yes	yes	not available: no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)	yes
MMSE	yes	yes	not available: no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)	yes
ML (exhaustive)	yes	yes	not available: no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)	yes
ML (sphere decoder)	yes	yes	not available: no MIMO detection (only the 2 x 2 adaptive equalisers of polarisation multiplexing)	not available: no sphere decoder (its exhaustive ML detector is the case ml)

zero forcing: error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	theory	1.88	yes	per-frame
comnum.run_spec	float64	theory	2.98	yes	binomial
Sionna	float32 (config.precision = ‘single’)	theory	1.79	yes	per-frame

BER and its 95 % interval, every point (1,600,000 bits per point)

Eb/N0 (dB)	theory	comnum (functions)	comnum.run_spec	Sionna
0	1.98e-1	1.98e-1 [1.97e-1, 1.99e-1]	1.98e-1 [1.97e-1, 1.98e-1]	1.98e-1 [1.97e-1, 1.99e-1]
4	1.20e-1	1.19e-1 [1.18e-1, 1.20e-1]	1.19e-1 [1.19e-1, 1.20e-1]	1.19e-1 [1.19e-1, 1.20e-1]
8	6.20e-2	6.16e-2 [6.10e-2, 6.23e-2]	6.14e-2 [6.10e-2, 6.18e-2]	6.26e-2 [6.19e-2, 6.32e-2]
12	2.83e-2	2.82e-2 [2.78e-2, 2.87e-2]	2.81e-2 [2.79e-2, 2.84e-2]	2.83e-2 [2.78e-2, 2.87e-2]
16	1.20e-2	1.21e-2 [1.18e-2, 1.24e-2]	1.19e-2 [1.17e-2, 1.20e-2]	1.19e-2 [1.16e-2, 1.22e-2]

MMSE: error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	—	—	—	per-frame
comnum.run_spec	float64	comnum	3.63	yes	binomial
Sionna	float32 (config.precision = ‘single’)	comnum	1.57	yes	per-frame

BER and its 95 % interval, every point (1,600,000 bits per point)

Eb/N0 (dB)	comnum (functions)	comnum.run_spec	Sionna
0	1.39e-1 [1.38e-1, 1.40e-1]	1.38e-1 [1.38e-1, 1.39e-1]	1.38e-1 [1.38e-1, 1.39e-1]
4	8.27e-2 [8.21e-2, 8.33e-2]	8.40e-2 [8.36e-2, 8.44e-2]	8.33e-2 [8.28e-2, 8.39e-2]
8	4.31e-2 [4.26e-2, 4.35e-2]	4.29e-2 [4.26e-2, 4.32e-2]	4.32e-2 [4.28e-2, 4.37e-2]
12	1.97e-2 [1.94e-2, 2.00e-2]	1.99e-2 [1.97e-2, 2.01e-2]	1.96e-2 [1.93e-2, 1.99e-2]
16	8.42e-3 [8.20e-3, 8.64e-3]	8.33e-3 [8.19e-3, 8.47e-3]	8.23e-3 [8.02e-3, 8.45e-3]

ML (exhaustive): error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	—	—	—	per-frame
comnum.run_spec	float64	comnum	2.53	yes	binomial
Sionna	float32 (config.precision = ‘single’)	comnum	1.09	yes	per-frame

BER and its 95 % interval, every point (320,000 bits per point)

Eb/N0 (dB)	comnum (functions)	comnum.run_spec	Sionna
0	1.25e-1 [1.23e-1, 1.26e-1]	1.25e-1 [1.24e-1, 1.26e-1]	1.25e-1 [1.23e-1, 1.27e-1]
4	4.03e-2 [3.90e-2, 4.16e-2]	3.99e-2 [3.92e-2, 4.06e-2]	4.05e-2 [3.92e-2, 4.17e-2]
8	4.12e-3 [3.68e-3, 4.56e-3]	4.77e-3 [4.54e-3, 5.02e-3]	4.47e-3 [4.02e-3, 4.93e-3]
12	2.34e-4 [1.24e-4, 3.45e-4]	2.28e-4 [1.81e-4, 2.87e-4]	1.59e-4 [7.61e-5, 2.43e-4]
16	0 [0, 9.38e-6]	0 [0, 1.20e-5]	1.25e-5 [0, 3.70e-5]

ML (sphere decoder): error rates

Library	Precision	Against	max abs(z), BER	Agrees	BER interval
comnum (functions)	float64	—	—	—	per-frame
comnum.run_spec	float64	comnum	2.84	yes	binomial

BER and its 95 % interval, every point (1,600,000 bits per point)

Eb/N0 (dB)	comnum (functions)	comnum.run_spec
0	1.24e-1 [1.23e-1, 1.25e-1]	1.25e-1 [1.24e-1, 1.25e-1]
4	3.97e-2 [3.92e-2, 4.03e-2]	4.06e-2 [4.03e-2, 4.09e-2]
8	4.44e-3 [4.23e-3, 4.64e-3]	4.62e-3 [4.51e-3, 4.73e-3]
12	1.91e-4 [1.49e-4, 2.32e-4]	1.95e-4 [1.75e-4, 2.18e-4]
16	3.13e-6 [0, 6.37e-6]	1.25e-5 [8.09e-6, 1.93e-5]

zero forcing: time

Median of 5 runs, µs per vector; workload: 10000 vectors at Eb/N0 = 12.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	2.99	2.87	—	1.00	0.0328 s
comnum.run_spec	float64	2.51	0.451	—	0.839	0.0274 s
Sionna	float32 (config.precision = ‘single’)	5.49	2.24	0.162	1.84	0.0569 s

MMSE: time

Median of 5 runs, µs per vector; workload: 10000 vectors at Eb/N0 = 12.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	3.21	2.87	—	1.00	0.0306 s
comnum.run_spec	float64	2.53	0.488	—	0.789	0.0362 s
Sionna	float32 (config.precision = ‘single’)	4.66	2.61	0.189	1.45	0.0545 s

ML (exhaustive): time

Median of 5 runs, µs per vector; workload: 1000 vectors at Eb/N0 = 12.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	1315	1335	—	1.00	1.59 s
comnum.run_spec	float64	1306	169	—	0.993	1.35 s
Sionna	float32 (config.precision = ‘single’)	19059	5567	71.3	14.5	19 s

ML (sphere decoder): time

Median of 5 runs, µs per vector; workload: 10000 vectors at Eb/N0 = 12.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	3.27	3.36	—	1.00	0.0359 s
comnum.run_spec	float64	3.37	0.641	—	1.03	0.0309 s

Channel coding over BPSK and AWGN#

Scenario 4: channel coding over BPSK and AWGN.

Parameters:

convolutional (133, 171), soft Viterbi: K = 7, (133, 171), k = 1024 + 6 tail bits, soft Viterbi; Eb/N0 = 1, 2, 3, 4 dB
5G NR LDPC (1024, 2048): 5G NR LDPC, k = 1024, n = 2048 (base graph 2, Z = 104), flooding sum-product, 50 iterations; Eb/N0 = 1, 1.5, 2, 2.5 dB
LDPC (1024, 512), same LLRs: PEG (1024, 512), flooding sum-product, 50 iterations, identical LLRs; Eb/N0 = 1.5, 2, 2.5, 3 dB
frames per point: 2000 (quick: 100)
timing: 200 frames (quick: 20) at Eb/N0 = 3 dB (conv) or 2 dB (LDPC); OptiCommPy without numba: 2 frames

Script: benchmarks/libraries/s4_coding.py.

The script

"""Scenario 4: channel coding over BPSK and AWGN.

- ``conv``: the rate-1/2 convolutional code of constraint length 7, generators
  (133, 171) in octal, 1024 information bits per frame terminated by 6 tail bits
  (rate 1024/2060), soft-decision Viterbi decoding. komm counts the taps from the
  least significant bit: the same code is ``ConvolutionalCode([[0o155, 0o117]])``
  there (checked on the impulse response). Reference: the union bound with the
  distance spectrum of the trellis (Viterbi 1971).
- ``nr_ldpc``: the LDPC code of 5G NR (3GPP TS 38.212, Sec. 5.3.2), k = 1024, rate
  matched to n = 2048 (base graph 2 by the rule of Sec. 7.2.2, lifting size 104, the
  first 2Z bits punctured; the same in both libraries), sum-product
  decoding by flooding, 50 iterations at most, in both libraries (comnum's default is
  the layered schedule, which converges in about half the iterations; it is set to
  flooding here so that the error rates compare the same decoder). comnum stops early on
  a codeword, Sionna runs the 50 iterations (its default).
- ``ldpc_same_llr``: one (1024, 512) LDPC code (PEG construction of comnum, variable
  degree 3, seed 1), its parity-check matrix given to the three decoders that accept
  one (comnum, OptiCommPy, Sionna), and the same channel LLRs (drawn by comnum from the
  same seed in every process): flooding sum-product, 50 iterations at most. The
  decoders see identical inputs, so they should take the same decisions frame by
  frame.

Every library uses its own encoder, BPSK mapper, AWGN channel and LLR computation in
``conv`` and ``nr_ldpc`` (independent seeds); Eb/N0 counts the energy per information
bit. Timings: the decoder alone on ``TIMING_FRAMES`` frames of LLRs at the timing
point, and the whole chain (bits, encoding, mapping, noise, LLRs, decoding, error
count) on the same number of frames; median of 5 runs after a warm-up run.

    python s4_coding.py --lib sionna --threads 1 [--quick] --out results/_work/s4.json
"""

from __future__ import annotations

import time

import common
import numpy as np
import theory

SCENARIO = "s4_coding"
TITLE = "Channel coding over BPSK and AWGN"
K_CONV = 1024
CONV_GEN = [0o133, 0o171]
K_NR, N_NR = 1024, 2048
N_PEG, K_PEG = 1024, 512
ITERATIONS = 50
EBN0 = {
    "conv": [1.0, 2.0, 3.0, 4.0],
    "nr_ldpc": [1.0, 1.5, 2.0, 2.5],
    "ldpc_same_llr": [1.5, 2.0, 2.5, 3.0],
}
TIMING_EBN0 = {"conv": 3.0, "nr_ldpc": 2.0, "ldpc_same_llr": 2.0}
SPEC_ONLY_OWN = "the runner draws its own channel (this case feeds the same LLRs to every decoder)"
CASES = {
    "conv": {
        "comnum": True,
        "comnum-spec": True,
        "comnum-rust": True,
        "komm": True,
        "opticommpy": "no convolutional code",
        "opticommpy-nojit": "no convolutional code",
        "sionna": True,
    },
    "nr_ldpc": {
        "comnum": True,
        "comnum-spec": "the specification format has no 5G NR LDPC code (NrLdpc: Python and Rust API)",
        "comnum-rust": "the specification format has no 5G NR LDPC code (NrLdpc: Python and Rust API)",
        "komm": "no LDPC code",
        "opticommpy": "no 5G NR LDPC code (its decoder takes a parity-check matrix: see ldpc_same_llr)",
        "opticommpy-nojit": "no 5G NR LDPC code (its decoder takes a parity-check matrix: see ldpc_same_llr)",
        "sionna": True,
    },
    "ldpc_same_llr": {
        "comnum": True,
        "comnum-spec": SPEC_ONLY_OWN,
        "comnum-rust": SPEC_ONLY_OWN,
        "komm": "no LDPC (belief-propagation) decoder",
        "opticommpy": True,
        "opticommpy-nojit": True,
        "sionna": True,
    },
}

PARAMETERS = {
    "conv": "K = 7, (133, 171), k = 1024 + 6 tail bits, soft Viterbi; Eb/N0 = 1, 2, 3, 4 dB",
    "nr_ldpc": "5G NR LDPC, k = 1024, n = 2048 (base graph 2, Z = 104), flooding sum-product, 50 iterations; Eb/N0 = 1, 1.5, 2, 2.5 dB",
    "ldpc_same_llr": "PEG (1024, 512), flooding sum-product, 50 iterations, identical LLRs; Eb/N0 = 1.5, 2, 2.5, 3 dB",
    "frames_per_point": "2000 (quick: 100)",
    "timing": "200 frames (quick: 20) at Eb/N0 = 3 dB (conv) or 2 dB (LDPC); OptiCommPy without numba: 2 frames",
}


def sizes(quick: bool) -> dict:
    return (
        {"frames": 100, "timing": 20, "block": 50}
        if quick
        else {"frames": 2000, "timing": 200, "block": 200}
    )


def n0_of(ebn0: float, rate: float) -> float:
    """N0 of unit-energy BPSK for an Eb/N0 per information bit (complex noise of
    variance N0, N0/2 on the real axis)."""
    return 1.0 / (rate * 10 ** (ebn0 / 10))


# ---------------------------------------------------------------------- the codecs
# A codec gives: encode(F) -> (info bits [F, k], channel input), llrs(F, ebn0) ->
# (reference bits, LLRs in the library's convention), decode(llrs) -> decided bits
# [F, k_out], and the number k_out of bits compared per frame.


def codec_comnum(case: str, seed: int):
    import comnum

    rng = comnum.Rng(seed)
    bpsk = comnum.Constellation("psk", 2)
    if case == "conv":
        code = comnum.ConvolutionalCode(K_CONV, generators=CONV_GEN, constraint=7)
    elif case == "nr_ldpc":
        code = comnum.NrLdpc(
            K_NR, N_NR, max_iterations=ITERATIONS, rule="sum_product", layered=False
        )
    else:
        return codec_same_llr_comnum(seed)
    k, n = code.k, code.n

    def llrs(f: int, ebn0: float):
        b = comnum.bits(f * k, rng)
        x = bpsk.map_bits(code.encode(b))
        n0 = n0_of(ebn0, k / n)
        y = x + rng.complex_noise(x.size, n0)
        return b.reshape(f, k), bpsk.llr(y, n0)

    def decode(llr):
        return code.decode(llr).reshape(-1, k)

    return llrs, decode, k, "float64"


def peg_code():
    import comnum

    code = comnum.LdpcCode.peg(N_PEG, K_PEG, var_degree=3, seed=1)
    code.set_decoder("sum_product", layered=False, max_iterations=ITERATIONS)
    return code


def same_llrs(seed: int):
    """The codewords and LLRs of ldpc_same_llr: the same in every process."""
    import comnum

    code = peg_code()
    rng = comnum.Rng(seed)
    bpsk = comnum.Constellation("psk", 2)

    def llrs(f: int, ebn0: float):
        b = comnum.bits(f * code.k, rng)
        c = code.encode(b)
        n0 = n0_of(ebn0, code.k / code.n)
        y = bpsk.map_bits(c) + rng.complex_noise(c.size, n0)
        return c.reshape(f, code.n), bpsk.llr(y, n0).reshape(f, code.n)  # positive for a 0

    return code, llrs


def codec_same_llr_comnum(seed: int):
    code, llrs = same_llrs(seed)

    def decode(llr):
        return np.stack([(code.decode_llr(row)[0] < 0).astype(np.uint8) for row in llr])

    return llrs, decode, code.n, "float64"


def codec_opticommpy(case: str, seed: int):
    import scipy.sparse as sp
    from optic.comm.fec import decodeLDPC
    from optic.utils import parameters

    code, llrs = same_llrs(seed)
    rows = code.rows
    h = sp.csr_matrix(
        (
            np.ones(sum(map(len, rows)), dtype=np.uint8),
            (np.repeat(np.arange(len(rows)), list(map(len, rows))), np.concatenate(rows)),
        ),
        shape=(len(rows), code.n),
    )
    p = parameters()
    p.H, p.maxIter, p.alg = h, ITERATIONS, "SPA"

    def decode(llr):
        bits, _out, _fail = decodeLDPC(np.ascontiguousarray(llr.T), p)  # columns are codewords
        return bits.T.astype(np.uint8)

    return llrs, decode, code.n, "float32 (decodeLDPC default prec = np.float32)"


def codec_komm(case: str, seed: int):
    import komm

    # (133, 171) with the current input on the most significant bit = (155, 117) in komm
    cc = komm.ConvolutionalCode([[0o155, 0o117]])
    code = komm.TerminatedConvolutionalCode(cc, num_blocks=K_CONV, mode="zero-termination")
    decoder = komm.ViterbiDecoder(code, input_type="soft")
    bpsk, lab = komm.PAMConstellation(2), komm.NaturalLabeling(1)
    rng = np.random.default_rng(seed)
    k, n = code.dimension, code.length

    def llrs(f: int, ebn0: float):
        b = rng.integers(0, 2, (f, k))
        x = bpsk.indices_to_symbols(lab.bits_to_indices(code.encode(b).ravel()))
        sigma2 = n0_of(ebn0, k / n) / 2  # real channel: N0/2 per dimension
        y = komm.GaussianChannel(sigma2, rng=rng).transmit(x)
        return b, lab.marginalize(bpsk.posteriors(y, sigma2))  # log P(0) / P(1)

    def decode(llr):
        return decoder.decode(llr).reshape(-1, k)

    return llrs, decode, k, "float64"


def codec_sionna(case: str, seed: int):
    import torch
    from sionna.phy import config
    from sionna.phy.channel import AWGN
    from sionna.phy.fec.conv import ConvEncoder, ViterbiDecoder
    from sionna.phy.fec.ldpc import LDPC5GDecoder, LDPC5GEncoder, LDPCBPDecoder
    from sionna.phy.mapping import BinarySource, Demapper, Mapper

    config.seed = seed
    if case == "ldpc_same_llr":
        code, same = same_llrs(seed)
        dec = LDPCBPDecoder(peg_pcm(code), num_iter=ITERATIONS, hard_out=True)

        def decode(llr):
            with torch.inference_mode():
                # Sionna's LLRs are logits, log P(1) / P(0): the opposite sign
                llr_dev = torch.from_numpy(-llr.astype(np.float32)).to(common.TORCH_DEVICE)
                return common.to_numpy(dec(llr_dev)).astype(np.uint8)

        return same, decode, code.n, "float32 (config.precision = 'single')"

    if case == "conv":
        enc = ConvEncoder(rate=0.5, constraint_length=7, terminate=True)
        dec = ViterbiDecoder(encoder=enc, method="soft_llr", return_info_bits=True)
        k, n = K_CONV, 2 * (K_CONV + 6)
    else:
        enc = LDPC5GEncoder(K_NR, N_NR)
        dec = LDPC5GDecoder(enc, num_iter=ITERATIONS, hard_out=True, return_infobits=True)
        k, n = K_NR, N_NR
    source, mapper, channel = BinarySource(), Mapper("pam", 1), AWGN()
    demapper = Demapper("app", "pam", 1)

    def llrs(f: int, ebn0: float):
        with torch.inference_mode():
            b = source([f, k])
            no = n0_of(ebn0, k / n)
            y = channel(mapper(enc(b)), no)
            return b, demapper(y, no)  # logits, log P(1) / P(0)

    def decode(llr):
        with torch.inference_mode():
            return dec(llr)

    return llrs, decode, k, "float32 (config.precision = 'single')"


def peg_pcm(code):
    h = np.zeros((len(code.rows), code.n), dtype=np.float32)
    for i, r in enumerate(code.rows):
        h[i, r] = 1
    return h


CODECS = {
    "comnum": codec_comnum,
    "komm": codec_komm,
    "opticommpy": codec_opticommpy,
    "opticommpy-nojit": codec_opticommpy,
    "sionna": codec_sionna,
}


def as_np(a) -> np.ndarray:
    return common.to_numpy(a)


def run_codecs(args, todo: list[str]) -> list[dict]:
    threading = common.setup_torch(args.threads) if args.lib == "sionna" else {}
    sz = sizes(args.quick)
    out = []
    for case in todo:
        llrs, decode, k_out, precision = CODECS[args.lib](case, args.seed)
        res = {"case": case, "precision": precision, **threading}

        def chain(f: int, ebn0: float, llrs=llrs, decode=decode) -> np.ndarray:
            ref, llr = llrs(f, ebn0)
            return np.count_nonzero(
                as_np(decode(llr)).reshape(f, -1) != as_np(ref).reshape(f, -1), axis=1
            )

        nt = 2 if args.lib == "opticommpy-nojit" else sz["timing"]
        e_t = TIMING_EBN0[case]
        t0 = time.perf_counter()
        chain(min(nt, 2), e_t)
        res["first_call_s"] = time.perf_counter() - t0
        if not args.timing_only:
            pts = []
            t0 = time.perf_counter()
            for e in EBN0[case]:
                errs = np.concatenate(
                    [chain(sz["block"], e) for _ in range(sz["frames"] // sz["block"])]
                )
                bound = (
                    float(theory.conv_union_bound(CONV_GEN, 7, K_CONV / (2 * (K_CONV + 6)), e))
                    if case == "conv"
                    else None
                )
                pt = common.frame_ber_point(errs, k_out, None, ebn0_db=e)
                if bound is not None:
                    pt["union_bound"] = bound
                fe = int(np.count_nonzero(errs))
                pt["bler"] = common.rate_point(fe, errs.size)
                if case == "ldpc_same_llr":
                    pt["errors_per_frame"] = errs.astype(int).tolist()
                pts.append(pt)
            res["points"] = pts
            res["sweep_s"] = time.perf_counter() - t0
            res["bits_per_frame"] = k_out
        _, llr = llrs(nt, e_t)
        res["timing"] = common.time_it(lambda llr=llr, decode=decode: decode(llr))
        res["timing"].update(
            units=nt, unit="frame", workload=f"decoder only, {nt} frames at Eb/N0 = {e_t} dB"
        )
        res["timing_chain"] = common.time_it(lambda nt=nt, e_t=e_t, chain=chain: chain(nt, e_t))
        res["timing_chain"].update(
            units=nt, unit="frame", workload=f"whole chain, {nt} frames at Eb/N0 = {e_t} dB"
        )
        out.append(res)
    return out


# ------------------------------------------------------------- specification runner


def spec(ebn0: list[float], frames: int, trials: int, seed: int) -> dict:
    code = {"type": "conv", "k": K_CONV}
    return {
        "name": "bench_conv",
        "seed": seed,
        "batch": trials,
        "grid": {"ebn0_db": ebn0},
        "link": [
            {"type": "fec_encode", "code": code, "blocks": frames // trials},
            {"type": "map", "constellation": {"kind": "psk", "m": 2}},
            {"type": "awgn_ebn0", "ebn0_db": "$ebn0_db"},
            "demap",
            {"type": "fec_decode", "code": code},
        ],
        "metrics": {"report": ["coded_ber", "bler"]},  # coded_ber: decoded information bits
        "stop": {"min_trials": trials, "max_trials": trials},
    }


def spec_points(rows: list[dict]) -> list[dict]:
    pts = []
    for r in rows:
        e = float(r["condition"]["ebn0_db"])
        m = r["metrics"]
        get = lambda d: d["count"] if "count" in d else d["errors"]  # noqa: E731
        # a binomial interval: the runner keeps the totals, not the errors of every frame
        pt = common.rate_point(get(m["coded_ber"]), m["coded_ber"]["total"], None, ebn0_db=e)
        pt["union_bound"] = float(
            theory.conv_union_bound(CONV_GEN, 7, K_CONV / (2 * (K_CONV + 6)), e)
        )
        pt["bler"] = common.rate_point(get(m["bler"]), m["bler"]["total"])
        pts.append(pt)
    return pts


def run_spec_lib(args, todo: list[str], *, rust: bool = False) -> list[dict]:
    """``comnum-spec`` (``comnum.run_spec``), or with ``rust`` ``comnum-rust`` (the binary)."""
    sz = sizes(args.quick)
    trials = 20
    out = []
    for case in todo:
        nt, e_t = sz["timing"], TIMING_EBN0[case]
        workload = f"whole chain, {nt} frames at Eb/N0 = {e_t} dB, {trials} trials"
        res = common.spec_case(
            args,
            {"case": case, "precision": "float64"},
            spec(EBN0[case], sz["frames"], trials, args.seed),
            spec([e_t], nt, trials, args.seed + 1),
            spec_points,
            rust=rust,
            key="timing_chain",
            units=nt,
            unit="frame",
            workload=workload + (" (process wall time)" if rust else ""),
        )
        if not args.timing_only:
            res["bits_per_frame"] = K_CONV
        out.append(res)
    return out


def run_rust_lib(args, todo: list[str]) -> list[dict]:
    return run_spec_lib(args, todo, rust=True)


def postprocess(merged: dict, work) -> dict:
    """ldpc_same_llr: the decoders saw the same LLRs; compare their errors frame by frame
    with those of comnum (same number of bit errors in a frame, same frame errors)."""
    per_lib = {}
    for run in merged["runs"]:
        if run.get("threads") != "1":
            continue
        for case in run.get("cases", []):
            if case["case"] == "ldpc_same_llr" and "points" in case:
                per_lib[run["library"]] = case["points"]
    base = per_lib.get("comnum")
    rows = []
    for lib, pts in per_lib.items():
        if lib == "comnum" or base is None:
            continue
        for p, q in zip(pts, base, strict=True):
            a, b = np.array(p["errors_per_frame"]), np.array(q["errors_per_frame"])
            rows.append(
                {
                    "library": lib,
                    "ebn0_db": p["ebn0_db"],
                    "frames": int(a.size),
                    "same_bit_errors": int(np.count_nonzero(a == b)),
                    "same_frame_outcome": int(np.count_nonzero((a > 0) == (b > 0))),
                }
            )
    return {"same_llr_vs_comnum": rows}


RUNNERS = {
    **{lib: run_codecs for lib in CODECS},
    "comnum-spec": run_spec_lib,
    "comnum-rust": run_rust_lib,
}

if __name__ == "__main__":
    common.main(SCENARIO, CASES, RUNNERS)

Support

Case	comnum (functions)	comnum.run_spec	OptiCommPy (numba)	Sionna
convolutional (133, 171), soft Viterbi	yes	yes	not available: no convolutional code	yes
5G NR LDPC (1024, 2048)	yes	not available: the specification format has no 5G NR LDPC code (NrLdpc: Python and Rust API)	not available: no 5G NR LDPC code (its decoder takes a parity-check matrix: see ldpc_same_llr)	yes
LDPC (1024, 512), same LLRs	yes	not available: the runner draws its own channel (this case feeds the same LLRs to every decoder)	yes	yes

convolutional (133, 171), soft Viterbi: error rates

Library	Precision	Against	max abs(z), BER	max abs(z), FER	Agrees	BER interval
comnum (functions)	float64	—	—	—	—	per-frame
comnum.run_spec	float64	comnum	1.35	1.33	yes	binomial
Sionna	float32 (config.precision = ‘single’)	comnum	1.41	2.53	yes	per-frame

BER and its 95 % interval, every point (2,048,000 bits per point)

Eb/N0 (dB)	union bound	comnum (functions)	comnum.run_spec	Sionna
1	3.05e-1	4.02e-2 [3.91e-2, 4.13e-2]	4.02e-2 [3.99e-2, 4.05e-2]	4.03e-2 [3.91e-2, 4.14e-2]
2	1.65e-2	4.97e-3 [4.64e-3, 5.30e-3]	5.16e-3 [5.06e-3, 5.26e-3]	5.32e-3 [4.97e-3, 5.67e-3]
3	6.27e-4	3.31e-4 [2.59e-4, 4.02e-4]	2.98e-4 [2.76e-4, 3.23e-4]	3.64e-4 [2.94e-4, 4.34e-4]
4	2.02e-5	2.54e-5 [8.46e-6, 4.23e-5]	1.32e-5 [9.06e-6, 1.92e-5]	3.08e-5 [9.04e-6, 5.25e-5]

Frame error rate and its 95 % interval (2,000 frames per point)

Eb/N0 (dB)	comnum (functions)	comnum.run_spec	Sionna
1	9.73e-1 [9.66e-1, 9.80e-1]	9.77e-1 [9.69e-1, 9.83e-1]	9.85e-1 [9.79e-1, 9.89e-1]
2	5.05e-1 [4.84e-1, 5.27e-1]	4.84e-1 [4.63e-1, 5.06e-1]	5.13e-1 [4.91e-1, 5.35e-1]
3	5.95e-2 [5.00e-2, 7.07e-2]	6.05e-2 [5.09e-2, 7.18e-2]	6.70e-2 [5.69e-2, 7.88e-2]
4	6.00e-3 [3.44e-3, 1.05e-2]	5.00e-3 [2.72e-3, 9.18e-3]	5.50e-3 [3.07e-3, 9.82e-3]

5G NR LDPC (1024, 2048): error rates

Library	Precision	Against	max abs(z), BER	max abs(z), FER	Agrees	BER interval
comnum (functions)	float64	—	—	—	—	per-frame
Sionna	float32 (config.precision = ‘single’)	comnum	0.893	1.21	yes	per-frame

BER and its 95 % interval, every point (2,048,000 bits per point)

Eb/N0 (dB)	comnum (functions)	Sionna
1	8.79e-3 [7.46e-3, 1.01e-2]	9.30e-3 [7.94e-3, 1.07e-2]
1.5	3.47e-5 [0, 1.03e-4]	9.38e-5 [0, 2.04e-4]
2	0 [0, 1.46e-6]	0 [0, 1.46e-6]
2.5	0 [0, 1.46e-6]	0 [0, 1.46e-6]

Frame error rate and its 95 % interval (2,000 frames per point)

Eb/N0 (dB)	comnum (functions)	Sionna
1	9.40e-2 [8.20e-2, 1.08e-1]	1.05e-1 [9.28e-2, 1.20e-1]
1.5	5.00e-4 [8.83e-5, 2.83e-3]	1.50e-3 [5.10e-4, 4.40e-3]
2	0 [0, 1.92e-3]	0 [0, 1.92e-3]
2.5	0 [0, 1.92e-3]	0 [0, 1.92e-3]

LDPC (1024, 512), same LLRs: error rates

Library	Precision	Against	max abs(z), BER	max abs(z), FER	Agrees	BER interval
comnum (functions)	float64	—	—	—	—	per-frame
OptiCommPy (numba)	float32 (decodeLDPC default prec = np.float32)	comnum	0.076	0	yes	per-frame
Sionna	float32 (config.precision = ‘single’)	comnum	0.0846	0.0381	yes	per-frame

BER and its 95 % interval, every point (2,048,000 bits per point)

Eb/N0 (dB)	comnum (functions)	OptiCommPy (numba)	Sionna
1.5	1.53e-2 [1.39e-2, 1.66e-2]	1.53e-2 [1.40e-2, 1.67e-2]	1.53e-2 [1.40e-2, 1.67e-2]
2	1.37e-3 [9.56e-4, 1.78e-3]	1.37e-3 [9.55e-4, 1.78e-3]	1.37e-3 [9.58e-4, 1.78e-3]
2.5	0 [0, 1.46e-6]	0 [0, 1.46e-6]	0 [0, 1.46e-6]
3	0 [0, 1.46e-6]	0 [0, 1.46e-6]	0 [0, 1.46e-6]

Frame error rate and its 95 % interval (2,000 frames per point)

Eb/N0 (dB)	comnum (functions)	OptiCommPy (numba)	Sionna
1.5	2.21e-1 [2.03e-1, 2.40e-1]	2.21e-1 [2.03e-1, 2.40e-1]	2.21e-1 [2.04e-1, 2.40e-1]
2	2.30e-2 [1.73e-2, 3.05e-2]	2.30e-2 [1.73e-2, 3.05e-2]	2.30e-2 [1.73e-2, 3.05e-2]
2.5	0 [0, 1.92e-3]	0 [0, 1.92e-3]	0 [0, 1.92e-3]
3	0 [0, 1.92e-3]	0 [0, 1.92e-3]	0 [0, 1.92e-3]

Same LLRs: the decisions frame by frame

The decoders received identical LLRs. Over all the points: the frames where a library makes as many bit errors as comnum, and the frames it decodes (or fails to decode) as comnum does.

Library	Points	Frames	Same bit errors	Same frame outcome
OptiCommPy (numba)	4	8000	7996 (99.95 %)	8000 (100.00 %)
Sionna	4	8000	7984 (99.80 %)	7999 (99.99 %)

convolutional (133, 171), soft Viterbi: decoder time

Median of 5 runs, ms per frame; workload: decoder only, 200 frames at Eb/N0 = 3.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	0.75	0.756	—	1.00	0.0021 s
Sionna	float32 (config.precision = ‘single’)	4.14	3.85	1.26	5.52	0.171 s

5G NR LDPC (1024, 2048): decoder time

Median of 5 runs, ms per frame; workload: decoder only, 200 frames at Eb/N0 = 2.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	8.53	8.28	—	1.00	0.0173 s
Sionna	float32 (config.precision = ‘single’)	47.3	12.5	0.171	5.55	0.0652 s

LDPC (1024, 512), same LLRs: decoder time

Median of 5 runs, ms per frame; workload: decoder only, 200 frames at Eb/N0 = 2.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	1.54	1.62	—	1.00	0.0032 s
OptiCommPy (numba)	float32 (decodeLDPC default prec = np.float32)	7.45	5.00	—	4.83	5.59 s
Sionna	float32 (config.precision = ‘single’)	8.59	2.14	0.172	5.57	0.0275 s

convolutional (133, 171), soft Viterbi: whole-chain time

Median of 5 runs, ms per frame; workload: whole chain, 200 frames at Eb/N0 = 3.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	0.967	0.966	—	1.00	0.0021 s
comnum.run_spec	float64	0.966	0.202	—	0.999	0.192 s
Sionna	float32 (config.precision = ‘single’)	4.65	4.34	2.05	4.81	0.171 s

5G NR LDPC (1024, 2048): whole-chain time

Median of 5 runs, ms per frame; workload: whole chain, 200 frames at Eb/N0 = 2.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	8.72	8.69	—	1.00	0.0173 s
Sionna	float32 (config.precision = ‘single’)	47.8	12.6	0.181	5.47	0.0652 s

LDPC (1024, 512), same LLRs: whole-chain time

Median of 5 runs, ms per frame; workload: whole chain, 200 frames at Eb/N0 = 2.0 dB. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions)	float64	1.64	1.68	—	1.00	0.0032 s
OptiCommPy (numba)	float32 (decodeLDPC default prec = np.float32)	7.08	4.91	—	4.32	5.59 s
Sionna	float32 (config.precision = ‘single’)	9.09	2.24	0.357	5.55	0.0275 s

Split-step Fourier propagation, 16-QAM, 32 GBd, 10 x 80 km#

Scenario 5: single-polarisation split-step Fourier propagation of 32 GBd 16-QAM over 10 x 80 km of standard single-mode fibre, a fixed step of 1 km (80 steps per span), noiseless amplification at the end of every span (gain e^{alpha L}).

Parameters:

fibre: alpha = 0.2 dB/km, D = 17 ps/nm/km, gamma = 1.3 /W/km, fc = 193.1 THz
link: 10 x 80 km, ideal (noiseless) amplification after every span
step: 1 km (800 steps); reference: comnum at 0.05 km (16000 steps)
signal: 16-QAM, 32 GBd, RRC roll-off 0.1, 4 samples per symbol, 4 dBm; 2^14 symbols (quick: 2^11)
timing: one propagation over the 10 spans

Script: benchmarks/libraries/s5_ssfm.py.

The script

"""Scenario 5: single-polarisation split-step Fourier propagation of 32 GBd 16-QAM over
10 x 80 km of standard single-mode fibre, a fixed step of 1 km (80 steps per span),
noiseless amplification at the end of every span (gain e^{alpha L}).

The input field is the same for every library: 16-QAM symbols of comnum (seed 1),
shaped by a root-raised cosine (roll-off 0.1, 4 samples per symbol, 128 GS/s) and
scaled to 4 dBm; it is drawn again, identically, by every process. Every library
propagates it with its own split-step solver (symmetric steps, nonlinear phase over
the whole step in all three); run_all.py compares the output fields (NMSE) with that
of comnum in double precision and with a step-converged reference (comnum, 0.05 km).

Sionna's equation is the complex conjugate of the one of comnum and OptiCommPy
(dE/dz = ... + j beta_2/2 d^2E/dt^2 - j gamma |E|^2 E): its field is compared in their
convention, conj(sionna(conj(x))), and the NMSE of the plain output is reported too.
Sionna propagates one span per call (SSFM) and amplifies with its EDFA of noise figure
0 (no noise). OptiCommPy's ssfm is NumPy only (no numba kernel): its two variants run
the same code. In the mode gpu, OptiCommPy runs its CuPy solver
(optic.models.modelsGPU.ssfm: the same steps, every array on the device; the input is
copied to the device and the output back to the host inside the timed call) in
complex128 and in complex64. Timing: one propagation over the 10 spans, median of 5
runs after a warm-up run.

    python s5_ssfm.py --lib opticommpy --threads 1 [--quick] --out results/_work/s5.json
"""

from __future__ import annotations

import math
import time
from pathlib import Path

import common
import numpy as np

SCENARIO = "s5_ssfm"
TITLE = "Split-step Fourier propagation, 16-QAM, 32 GBd, 10 x 80 km"
CASES = {
    "ssfm": {
        "comnum": True,
        "komm": "no fibre model",
        "opticommpy": True,
        "opticommpy-nojit": True,
        "sionna": True,
    }
}
# The libraries with a GPU path in this scenario (run_all.py: the mode gpu).
GPU_LIBS = {"sionna", "opticommpy"}
ALPHA_DB_KM, D_PS_NM_KM, GAMMA, FC = 0.2, 17.0, 1.3, 193.1e12
SPAN_KM, N_SPANS, STEP_KM = 80.0, 10, 1.0
REF_STEP_KM = 0.05
SYMBOL_RATE, SPS = 32e9, 4
FS = SYMBOL_RATE * SPS
POWER_DBM = 4.0

PARAMETERS = {
    "fibre": "alpha = 0.2 dB/km, D = 17 ps/nm/km, gamma = 1.3 /W/km, fc = 193.1 THz",
    "link": "10 x 80 km, ideal (noiseless) amplification after every span",
    "step": "1 km (800 steps); reference: comnum at 0.05 km (16000 steps)",
    "signal": "16-QAM, 32 GBd, RRC roll-off 0.1, 4 samples per symbol, 4 dBm; 2^14 symbols (quick: 2^11)",
    "timing": "one propagation over the 10 spans",
}


def n_symbols(quick: bool) -> int:
    return 2**11 if quick else 2**14


def input_field(quick: bool) -> np.ndarray:
    import comnum

    rng = comnum.Rng(1)
    c = comnum.Constellation("qam", 16).normalized()
    _, s = comnum.symbols(c, n_symbols(quick), rng)
    h = comnum.pulse_shape("rrc", SPS, n_taps=1025, rolloff=0.1)
    x = comnum.pnorm(comnum.fir_filter(h, comnum.upsample(s, SPS)))
    return x * math.sqrt(comnum.dbm2w(POWER_DBM))


def beta2_ps2_km() -> float:
    c = 299_792_458.0
    lam = c / FC
    return -(D_PS_NM_KM * 1e-6) * lam**2 / (2 * math.pi * c) * 1e24 * 1e3  # s^2/m -> ps^2/km


def prop_comnum(precision: str, step_km: float = STEP_KM):
    import comnum

    f = comnum.FiberParams(
        alpha_db_km=ALPHA_DB_KM,
        d_ps_nm_km=D_PS_NM_KM,
        gamma_w_km=GAMMA,
        span_km=SPAN_KM,
        n_spans=N_SPANS,
        fc_hz=FC,
    )
    link = comnum.Ssfm(f, step_km=step_km, amplification="ideal")
    return lambda x: link.propagate(x, FS, precision=precision)


def prop_opticommpy(gpu: bool = False, prec=np.complex128):
    """OptiCommPy's split-step: the NumPy solver, or its CuPy copy on the GPU
    (``optic.models.modelsGPU.ssfm``, which takes and returns NumPy arrays: the
    transfers to and from the device are inside the timed call)."""
    if gpu:
        from optic.models.modelsGPU import ssfm
    else:
        from optic.models.channels import ssfm
    from optic.utils import parameters

    p = parameters()
    p.Ltotal, p.Lspan, p.hz = SPAN_KM * N_SPANS, SPAN_KM, STEP_KM
    p.alpha, p.D, p.gamma, p.Fc, p.Fs = ALPHA_DB_KM, D_PS_NM_KM, GAMMA, FC, FS
    p.amp, p.prgsBar, p.prec = "ideal", False, prec
    return lambda x: ssfm(x, p)


def prop_sionna(precision: str):
    import torch
    from sionna.phy.channel.optical import EDFA, SSFM

    alpha = ALPHA_DB_KM / (10 * math.log10(math.e))  # 1/km
    span = SSFM(
        alpha=alpha,
        beta_2=beta2_ps2_km(),
        f_c=FC,
        gamma=GAMMA,
        length=SPAN_KM,
        n_ssfm=round(SPAN_KM / STEP_KM),
        sample_duration=1e12 / FS,  # ps
        t_norm=1e-12,
        with_amplification=False,
        precision=precision,
    )
    amp = EDFA(g=math.exp(alpha * SPAN_KM), f=0.0, f_c=FC, dt=1 / FS, precision=precision)
    cdtype = torch.complex64 if precision == "single" else torch.complex128

    def run(x: np.ndarray) -> np.ndarray:
        with torch.inference_mode():
            u = torch.from_numpy(x).to(device=common.TORCH_DEVICE, dtype=cdtype)
            for _ in range(N_SPANS):
                u = amp(span(u))
            return common.to_numpy(u)

    return run


def run_lib(args, todo: list[str]) -> list[dict]:
    if args.lib == "sionna":
        threading = common.setup_torch(args.threads)
    elif args.lib == "opticommpy" and args.threads == "gpu":
        threading = common.setup_cupy()
    else:
        threading = {}
    x = input_field(args.quick)
    work = Path(args.work)
    work.mkdir(parents=True, exist_ok=True)
    if args.lib == "comnum":
        variants = {"f64": (prop_comnum("f64"), "float64"), "f32": (prop_comnum("f32"), "float32")}
    elif args.lib == "sionna":
        variants = {
            "single": (prop_sionna("single"), "float32 (precision = 'single', the default)"),
            "double": (prop_sionna("double"), "float64 (precision = 'double')"),
        }
    elif args.lib == "opticommpy" and args.threads == "gpu":
        import cupy as cp

        variants = {
            "f64": (prop_opticommpy(True, cp.complex128), "float64 (prec = cp.complex128)"),
            "f32": (prop_opticommpy(True, cp.complex64), "float32 (prec = cp.complex64)"),
        }
    else:
        variants = {"f64": (prop_opticommpy(), "float64 (prec = np.complex128)")}
    out = []
    for name, (prop, precision) in variants.items():
        res = {
            "case": "ssfm",
            "variant": name,
            "precision": precision,
            "samples": int(x.size),
            **threading,
        }
        if args.lib == "sionna":
            fwd = lambda prop=prop: np.conj(prop(np.conj(x)))  # noqa: E731
        else:
            fwd = lambda prop=prop: prop(x)  # noqa: E731
        t0 = time.perf_counter()
        y = fwd()
        res["first_call_s"] = time.perf_counter() - t0
        if not args.timing_only:
            np.save(work / f"{SCENARIO}_{args.lib}_{name}.npy", np.asarray(y, dtype=np.complex128))
            if args.lib == "sionna":
                np.save(
                    work / f"{SCENARIO}_{args.lib}_{name}_plain.npy",
                    np.asarray(prop(x), dtype=np.complex128),
                )
        res["timing"] = common.time_it(fwd)
        res["timing"].update(
            units=1,
            unit="propagation",
            workload=f"{x.size} samples, {N_SPANS} x {SPAN_KM:g} km, {STEP_KM:g} km steps",
        )
        out.append(res)
    if args.lib == "comnum" and not args.timing_only:
        t0 = time.perf_counter()
        np.save(work / f"{SCENARIO}_reference.npy", prop_comnum("f64", REF_STEP_KM)(x))
        out.append(
            {
                "case": "ssfm",
                "variant": "reference",
                "precision": "float64",
                "step_km": REF_STEP_KM,
                "wall_s": time.perf_counter() - t0,
            }
        )
    return out


def nmse(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.sum(np.abs(a - b) ** 2) / np.sum(np.abs(b) ** 2))


def postprocess(merged: dict, work: Path) -> dict:
    """The NMSE of every output field against comnum (f64, 1 km) and the reference."""
    fields = {}
    for path in sorted(work.glob(f"{SCENARIO}_*.npy")):
        fields[path.stem.removeprefix(f"{SCENARIO}_")] = np.load(path)
    base, ref = fields.get("comnum_f64"), fields.get("reference")
    rows = []
    for name, y in fields.items():
        if name == "reference":
            continue
        rows.append(
            {
                "field": name,
                "nmse_vs_comnum_f64": nmse(y, base) if base is not None else None,
                "nmse_vs_reference": nmse(y, ref) if ref is not None else None,
            }
        )
    return {"nmse": rows, "reference_step_km": REF_STEP_KM}


RUNNERS = {"comnum": run_lib, "opticommpy": run_lib, "opticommpy-nojit": run_lib, "sionna": run_lib}

if __name__ == "__main__":
    common.main(SCENARIO, CASES, RUNNERS)

Support

Case	comnum (functions)	OptiCommPy (numba)	Sionna
split-step Fourier	yes	yes	yes

Output fields

NMSE of the output field, sum |a - b|^2 / sum |b|^2, against comnum in float64 at the same step and against the step-converged reference (comnum, 0.05 km). The NMSE against the reference is the error of the 1 km step itself.

Field	NMSE vs comnum (float64, 1 km)	NMSE vs reference
comnum, float32	1.14e-8	7.45e-8
comnum, float64	0	3.69e-8
OptiCommPy, float64	2.24e-26	3.69e-8
Sionna, float64	3.55e-25	3.69e-8
Sionna, float64, its own convention (not conjugated)	2.03e0	2.03e0
Sionna, float32	2.15e-8	2.45e-8
Sionna, float32, its own convention (not conjugated)	2.03e0	2.03e0

split-step Fourier: time

Median of 5 runs, s per propagation; workload: 65536 samples, 10 x 80 km, 1 km steps. Relative: the 1-thread time over that of comnum (functions) on 1 thread.

Library	Precision	1 thread	all threads	GPU	Relative	First call
comnum (functions), f64	float64	1.29	1.31	—	1.00	1.31 s
comnum (functions), f32	float32	0.614	0.619	—	0.476	0.625 s
OptiCommPy (numba), f64	float64 (prec = np.complex128)	4.54	4.51	0.144	3.52	5.28 s
OptiCommPy (numba), f32	float32 (prec = cp.complex64)	—	—	0.144	—	0.155 s
Sionna, single	float32 (precision = ‘single’, the default)	5.36	1.51	0.277	4.16	5.39 s
Sionna, double	float64 (precision = ‘double’)	6.80	5.65	0.272	5.28	6.4 s