Comparisons with other libraries#

comnum gathers blocks that usually live in separate tools: channel codes, wireless waveforms and channels, fibre propagation, optical devices, receiver DSP, recordings. Each domain has its reference implementations, and comnum is compared with them on the same inputs. The outputs of the tools are recorded once (tests/oracle/, validation/aff3ct/), with the version that produced them, so that the tests run without the tools; none of their code is copied (see THIRD_PARTY.md). The maturity page lists, module by module, the tests named below.

Agreeing with a tool is evidence, not proof: two implementations of a standard can share a misreading. Where a tool and comnum differ, the difference is traced to the standard or the paper, and documented.

Library

What is compared

Agreement

Differences found

OptiCommPy 0.11 (GPL-3.0)

Gray mapping and demapping, closed-form error rates, pulse shaping and filters, resampling, linear fibre, split-step Fourier and Manakov propagation, WDM grid and SNR, Mach–Zehnder and IQ modulators, photodiodes, optical hybrid and coherent receiver, adaptive equalisers (CMA, RDE, NLMS, DD-LMS, 2×2), carrier recovery (fourth power, Viterbi–Viterbi, blind phase search, DD-PLL), Gardner clock recovery, BER, SNR, EVM, MI and GMI (tests/oracle/test_oracle.py)

to machine precision for the deterministic blocks (10⁻⁸ relative on the propagated fields); statistically for the Monte Carlo estimators

the BER of BPSK from theoryBER is twice the exact value; its edc limits the SNR of a noiseless link to 23 dB; its edfa re-seeds its generator on every call; the trapezoidal nonlinear phase of manakovSSF smooths the cross-phase modulation of channels that walk off within a step (45 % of the XPM of Chiang et al. 1996 with steps of 0.5 km) and its 0.02 rad per step leaves 0.45 dB of error on WDM links, so the Manakov solver of comnum uses other defaults (docs/adr/0013-manakov-defaults.md; all the fibre comparisons: Fibre propagation: how the split step is validated)

Optilux Apr 2021 (GPL-3.0, under Octave)

split-step Fourier propagation of 16-QAM at 32 GBd over 10 × 80 km with 1 km steps, and one span with and without the Kerr effect, on the field of the benchmark scenario (tests/oracle/test_optilux_oracle.py)

to machine precision without the Kerr effect (NMSE 3·10⁻²⁸); 1.2·10⁻¹⁰ after one span and 1.5·10⁻⁸ after ten with it

the nonlinear step: Optilux applies the phase over the effective length of the step, comnum over its length; at 1 km Optilux’s field is the closer to the step-converged solution (1.1·10⁻⁸ against 3.7·10⁻⁸). Optilux writes the equation with the opposite sign of the imaginary unit, which the recording undoes

GNPy 3.0 (BSD-3-Clause)

Raman power profiles with co- and counter-propagating pumps, their spontaneous noise, the generalised GN model with power profiles (tests/oracle/test_gnpy_oracle.py)

0.02 dB on the NLI; 0.004 dB on the profiles at a 5 m step

GNPy integrates by Euler’s method (first order in its step); at its default tolerances it underestimates the NLI of counter-pumped spans by about 1 dB

AFF3CT v4.7.1 (MIT)

bit and frame error rates of convolutional (Viterbi), LTE turbo (log-MAP and max-log-MAP), LDPC (sum-product, min-sum variants, flooding and layered), polar (SC, CA-SCL), Reed–Solomon and BCH codes, the short LDPC code of DVB-S2 (validation/tests/coding/aff3ct.rs)

within the confidence intervals of both simulations

none on the decoders; the polar construction is that of comnum on both sides

Sionna (Apache-2.0), its test vectors

the CRCs of 3GPP (CRC6 to CRC24C); convolutional codes of rates 1/2, 1/3, 1/4, encoded and Viterbi-decoded; the LTE turbo code at K = 40, 112, 168, 432, encoded and decoded at 0 dB; the 5G LDPC code with rate matching (28 sizes, 64 to 8448 bits, both base graphs) and the 5G polar code of the uplink control information (tests/oracle/test_sionna_vectors.py)

bit for bit, the decisions included (errors and all)

none; the vectors brought the full QPP table of LTE (188 block sizes) to TurboCode.lte; its tables of the LDPC base graphs and of the polar sequence are identical to those extracted from TS 38.212

py3gpp 0.6 (AGPL-3.0)

5G NR: PSS, SSS, PBCH DMRS and their positions, the PDSCH DMRS of 42 configurations, the OFDM modulation of a subframe (tests/oracle/test_nr_oracle.py)

bit for bit for sequences and positions; 10⁻¹⁴ for the waveforms

py3gpp lengthens the cyclic prefix of every slot instead of every half subframe (its subframes no longer last 1 ms from 60 kHz); its DMRS table adds a symbol to 8- and 9-symbol allocations without additional position

sigmf-python 1.x (LGPL-3.0)

SigMF recordings both ways: the 28 sample formats, several channels, captures, annotations, archives (tests/test_sigmf.py)

identical samples and metadata; the recordings of comnum pass its validation

sigmf-python scales 32-bit integers in float32 (an error of 3·10⁻⁸)

Other open implementations of the standards can serve the same way, through their outputs: srsRAN and OpenAirInterface (LTE, 5G NR), the gr-dtv module of GNU Radio (DVB-T, DVB-T2) and gr-ieee802-11 (Wi-Fi). Their licences (GPL, AGPL and others) forbid copying their code into comnum, which is MIT/Apache-2.0, but not comparing with their results.

Decoding throughput against AFF3CT#

The decoders of comnum timed against those of AFF3CT v4.7.1 (commit d4d715e, the version of the recorded error rates, built with clang 21 and -march=native, NEON) on the seventeen configurations of validation/aff3ct/: one thread, the same code, the same \(E_b/N_0\) (the middle of the recorded sweep), a budget of 8 s per point, the decoder alone (AFF3CT: the task decode_siho of --sim-stats; comnum: the decode_block of the code on frames of LLRs drawn beforehand), in Mb/s of the bits AFF3CT counts (information bits, or coded bits for the LDPC codes it runs with --sim-coded). AFF3CT runs its STD/NAIVE implementations, as in the recordings: its inter-frame SIMD decoders (FAST, AVX, NEON) are faster still on the polar and LDPC codes. One run per point, on an Apple M6; tools/speed_aff3ct.py and validation/src/bin/speed_fec.rs reproduce it, the raw results are in benchmarks/aff3ct/.

Code and decoder

\(E_b/N_0\) (dB)

AFF3CT (Mb/s)

comnum (Mb/s)

comnum / AFF3CT

comnum, whole chain (Mb/s)

BCH (63, 45), t = 3, hard decisions

5.0

70.23

142.71

2.03

31.95

BCH (127, 92), t = 5

5.0

51.49

107.91

2.10

33.01

BCH (255, 223), t = 4

5.5

78.69

178.68

2.27

43.12

RS (15, 11), t = 2

5.5

349.52

119.32

0.34

35.23

RS (63, 55), t = 4

5.75

417.44

238.38

0.57

57.08

RS (255, 239), t = 8

6.25

441.83

255.39

0.58

62.28

convolutional K = 7, 64 bits, soft Viterbi

2.5

3.77

3.93

1.04

3.18

LDPC PEG (576, 288), flooding SPA, 50 it.

2.0

2.52

2.79

1.11

2.79

LDPC PEG (576, 288), flooding OMS 0.15, 20 it.

2.25

19.79

7.51

0.38

6.39

LDPC PEG (576, 288), layered NMS 0.75, 20 it.

2.0

23.58

10.69

0.45

8.53

LDPC PEG (648, 540), layered SPA, 20 it.

3.75

13.18

14.28

1.08

10.84

LDPC DVB-S2 (16200, 7200), layered SPA, 50 it.

0.7

0.92

1.13

1.23

1.14

polar (256, 128), SC

3.0

38.40

12.80

0.33

9.69

polar (1024, 512), SC

2.5

34.38

13.23

0.38

9.93

polar (256, 128) + CRC-11, SCL L = 8

2.0

1.72

2.14

1.24

2.00

turbo LTE K = 40, log-MAP, 8 it.

2.5

0.27

0.21

0.78

0.21

turbo LTE K = 1024, max-log-MAP, 6 it.

1.0

9.65

1.93

0.20

1.81

comnum is level with AFF3CT on the sum-product, list and Viterbi decoders and twice as fast on the BCH codes; it is 2 to 3 times slower where AFF3CT has specialised paths (the min-sum LDPC rules, which here share the code of the sum-product; the successive cancellation of polar codes, which AFF3CT prunes on the rate-0, rate-1 and repetition nodes; the Reed–Solomon arithmetic) and 5 times slower on the max-log-MAP turbo decoder of 1024 bits (a scalar BCJR in double precision). The whole chain of comnum (bits, encoding, BPSK, noise, LLRs, decoding, error count) is also given: for the short codes the channel dominates it.

Deviations from OptiCommPy#

The blocks follow the layout, names and conventions of OptiCommPy, so that a user of one finds the same blocks in the other, and the oracle tests compare them to machine precision. The deviations are deliberate and tested as such:

  • the declarative metrics compute MI and GMI with the least-squares channel gain; comnum.monte_carlo_mi() and comnum.monte_carlo_gmi() keep OptiCommPy’s power normalisation (which underestimates the rates by 0.06 bit at 0 dB);

  • comnum.evm() follows calcEVM, which normalises before the rotation, unlike fastBERcalc;

  • comnum.theory_ber() of BPSK is the exact \(Q(\sqrt{2E_b/N_0})\); OptiCommPy’s general PSK formula doubles it;

  • comnum.monte_carlo_mi() snaps the normalised transmitted symbols to the constellation, so that the estimate never exceeds \(\log_2 M\), and both estimators return \(\log_2 M\) without noise instead of 0/0;

  • comnum.pulse_shape() uses a symmetric time grid, equal to OptiCommPy’s for an even length; an odd length puts the taps on the sampling instants, while OptiCommPy’s odd grid is shifted by about half a sample;

  • the clock recovery computes in double precision; OptiCommPy stores its output in complex64;

  • with shaped symbols, comnum.theory_gmi() is \(H(X) - \sum_k H(B_k|Y)\), the rate of bit-metric decoding and the limit of the Monte Carlo estimators; OptiCommPy’s theoryGMI sums the bit-wise informations \(\sum_k I(B_k;Y)\), which can exceed the entropy of the source (both agree for uniform symbols);

  • with a seed, OptiCommPy’s edfa re-seeds its generator on every call, so that manakovSSF adds the same ASE to both polarisations and after every span; the comparisons of noisy links leave its seed unset, and comnum.ManakovSsfm draws independent noise;

  • OptiCommPy’s edc limits the SNR of a linear, noiseless link to 23 dB; the oracle tests compensate dispersion with its linear fibre of opposite dispersion.