comnum.sim#

Monte Carlo campaigns of Python trials, on the Rust runner.

The trials of a Python scenario run on the runner of the Rust crate comnum-sim, with one thread (the Python code is called under the GIL): the vocabulary (conditions, trials, records, stopping rules), the per-trial random streams, the statistics, the statuses, the labels of the conditions and the CSV and JSON loggers are those of the Rust campaigns, so a scenario written in Python draws exactly what its Rust port draws and reports the same numbers. For long campaigns, write the link with the blocks of the registry (a native comnum.Chain runs on all cores).

Vocabulary:

a condition is a dict of named values; grid() builds the Cartesian product of axes (the last axis varies fastest);
a trial is a function trial(condition, rng, record[, state]) that draws only from rng and reports metrics in the Record: error counts (Record.errors(), estimated as an ErrorRate with a Wilson interval; ErrorRate.interval() gives the other intervals of a proportion, and interval_coverage() and interval_length() their exact coverage and expected length) or values (Record.value(), estimated as a Mean with a normal interval);
trial t of condition i draws from Rng.stream(seed, name, i, t), so the results depend neither on the batch size nor on the order of execution;
a stopping rule (Stop) is checked after every batch of trials: Stop.min_errors("ber", 200) | Stop.max_trials(10_000), or the stop mapping of a specification.

Example

>>> import comnum
>>> from comnum import sim
>>> c = comnum.Constellation("qam", 4).normalized()
>>> def trial(cond, rng, rec):
...     b = comnum.bits(2 * 1000, rng)
...     y = comnum.awgn_ebn0(c.map_bits(b), cond["ebn0_db"], 2, rng)
...     rec.errors("ber", int((c.demap_bits(y) != b).sum()), b.size)
>>> res = sim.run(trial, sim.grid(ebn0_db=[4.0, 6.0]), name="qpsk",
...               stop=sim.Stop.min_errors("ber", 100) | sim.Stop.max_trials(500))
>>> [r.status for r in res]
['completed', 'completed']
>>> lo, hi = res[0].metrics["ber"].interval(0.999)
>>> bool(lo < comnum.theory_ber(4, 4.0, "qam") < hi)
True

Functions

`grid`(**axes)	The Cartesian product of the axes, the last one varying fastest (the `grid` of a specification).
`run`(trial, conditions, *, name[, seed, ...])	Run a scenario on every condition, on the Rust runner (one thread).

Classes

`ConditionResult`()	The result of one condition of a campaign (`comnum_sim::runner::ConditionResult`).
`ErrorRate`([errors, total])	An error rate: errors out of trials, with a Wilson score interval (`comnum_sim::stats::ErrorRate`).
`Mean`()	A running mean and variance (Welford), merged exactly (Chan et al.), with a normal interval (`comnum_sim::stats::Mean`).
`Record`([trial])	The metrics of one trial, or merged over many (`comnum_sim::scenario::Record`): `errors()` counts errors (an error rate), `value()` adds values (a mean).
`Stop`()	A stopping rule, checked after every batch of trials of a condition; rules combine with `\|` (any of them) and `&` (all of them).

class comnum.sim.ConditionResult#

The result of one condition of a campaign (comnum_sim::runner::ConditionResult).

row(confidence=0.95)#

A flat row, as the CSV logger writes it (comnum_sim::runner::ConditionResult::row).

Parameters:: confidence (float) – the level of the intervals.
Returns:: index, the condition (its values as given), status, trials, wall_s, and for every metric m: m, m_lo, m_hi, m_count, m_total.
Return type:: dict
Raises:: ValueError – for a value that has no Python equivalent (not expected from the runner).

to_dict(confidence=0.95)#

The result as a dict, the format of comnum.run_spec().

Parameters:: confidence (float) – the level of the intervals.
Returns:: index, condition, status, trials, wall_s and metrics ({name: {value, lo, hi, kind, count, total}}).
Return type:: dict
Raises:: ValueError – for a value that has no Python equivalent (not expected from the runner).

condition#

The condition (as given).

Type:: dict

index#

Index of the condition in the grid.

Type:: int

metrics#

The metrics merged over the trials.

Type:: dict[str, ErrorRate | Mean]

status#

completed, stopped, skipped or dev_run.

Type:: str
Type:: How it ended

trials#

Trials run.

Type:: int

wall_s#

Wall time of the condition, setup included (s).

Type:: float

class comnum.sim.ErrorRate(errors=0, total=0)#

An error rate: errors out of trials, with a Wilson score interval (comnum_sim::stats::ErrorRate).

Parameters:

errors (int) – errors counted.
total (int) – trials counted.

Examples

>>> from comnum.sim import ErrorRate
>>> r = ErrorRate(3, 1000)
>>> r.value, [round(v, 5) for v in r.interval(0.95)]
(0.003, [0.00102, 0.00878])

add(errors, total)#

Count errors.

Parameters:

errors (int) – errors among the new trials.
total (int) – new trials.

interval(confidence=0.95, method='wilson')#

The confidence interval of the rate ((0, 1) without trials).

The Wilson score interval by default, valid down to zero errors; method picks another interval of Brown, Cai & DasGupta (2001): "wald" (eq. (1)), "agresti_coull" (eq. (5)), "jeffreys" (eqs. (6)-(8)) or "clopper_pearson" (Sec. 4.2.1). The Wald and Agresti-Coull ends may leave [0, 1], as the paper defines them.

Parameters:

confidence (float) – the level of the interval.
method (str) – the interval.

Returns:

the bounds.

Return type:

tuple[float, float]

Raises:

ValueError – for an unknown method or a level outside (0, 1).

Example

>>> from comnum import sim
>>> lo, hi = sim.ErrorRate(3, 7).interval(method="jeffreys")
>>> round(lo, 3), round(hi, 3)
(0.139, 0.765)

merge(other)#

Add the counts of another estimate.

Parameters:: other (ErrorRate) – the other estimate.

relative_precision(confidence=0.95)#

Half-width of the interval relative to the rate (inf without errors).

Parameters:: confidence (float) – the level of the interval.
Returns:: the relative half-width.
Return type:: float

count#

Number of errors (the count of the stopping rules).

Type:: int

errors#

Errors counted.

Type:: int

total#

Trials counted.

Type:: int

value#

The rate (nan without trials).

Type:: float

class comnum.sim.Mean#

A running mean and variance (Welford), merged exactly (Chan et al.), with a normal interval (comnum_sim::stats::Mean).

Examples

>>> from comnum.sim import Mean
>>> m = Mean()
>>> for x in [1.0, 2.0, 3.0]:
...     m.add(x)
>>> m.value, m.std, m.count
(2.0, 1.0, 3)

add(x)#

Add one value.

Parameters:: x (float) – the value.

interval(confidence=0.95)#

The normal interval of the mean, mean ± z·std/√n (unbounded with fewer than two values).

Parameters:: confidence (float) – the level of the interval.
Returns:: the bounds.
Return type:: tuple[float, float]

merge(other)#

Merge the values of another estimate.

Parameters:: other (Mean) – the other estimate.

relative_precision(confidence=0.95)#

Half-width of the interval relative to the mean (inf for a zero mean).

Parameters:: confidence (float) – the level of the interval.
Returns:: the relative half-width.
Return type:: float

count#

Number of values.

Type:: int

sem#

Standard error of the mean, std/√n.

Type:: float

std#

Sample standard deviation.

Type:: float

value#

The mean (nan without values).

Type:: float

variance#

Unbiased sample variance (nan with fewer than two values).

Type:: float

class comnum.sim.Record(trial=0)#

The metrics of one trial, or merged over many (comnum_sim::scenario::Record): errors() counts errors (an error rate), value() adds values (a mean).

Parameters:: trial (int) – the index of the trial of its condition (0 when merged), for the blocks that save or label a trial.

Examples

>>> from comnum.sim import Record
>>> rec = Record()
>>> rec.errors("ber", 3, 100)
>>> rec.value("snr_db", 9.5)
>>> rec["ber"].value, sorted(rec.metrics)
(0.03, ['ber', 'snr_db'])

errors(name, errors, total)#

Count errors of an error rate.

Parameters:

name (str) – the error rate.
errors (int) – errors among the events.
total (int) – events.

Raises:

TypeError – if name is a mean.

merge(other)#

Merge the metrics of another record.

Parameters:: other (Record) – the other record.
Raises:: TypeError – for a metric that is a rate in one and a mean in the other.

value(name, x)#

Add a value to a mean (non-finite values are ignored).

Parameters:

name (str) – the mean.
x (float) – the value.

Raises:

TypeError – if name is an error rate.

metrics#

The metrics, sorted by name (copies).

Type:: dict[str, ErrorRate | Mean]

trial#

The index of the trial of its condition (0 when merged).

Type:: int

class comnum.sim.Stop#

A stopping rule, checked after every batch of trials of a condition; rules combine with | (any of them) and & (all of them).

The rules of the Rust runner (comnum_sim::stopping::Stop): max_trials(), min_trials(), min_errors(), precision() and max_time(). A rule must be bounded (contain max_trials or max_time on every branch of an &). from_spec() parses the stop mapping of a specification.

Examples

>>> from comnum.sim import Stop
>>> Stop.min_errors("ber", 100) | Stop.max_trials(1000)
(errors(ber) ≥ 100 or max_trials(1000))
>>> Stop.from_spec({"min_errors": {"ber": 100}, "max_trials": 1000}).bounded
True

static from_spec(stop)#

The rule of the stop mapping of a specification, parsed as the Rust runner parses it: the targets (min_errors, precision, with min_trials) or the caps (max_trials, max_time_s); min_trials alone caps the trials, and an empty mapping gives 100 trials.

Parameters:: stop (dict) – the mapping.
Returns:: the rule.
Return type:: Stop
Raises:: ValueError – for an unknown key or a value out of its domain (the message names it).

Examples

>>> from comnum.sim import Stop
>>> Stop.from_spec({"min_trials": 50})
max_trials(50)

static max_time(seconds)#

Stop after an amount of wall time on the condition (machine-dependent).

Parameters:: seconds (float) – the time (s).
Returns:: the rule.
Return type:: Stop
Raises:: ValueError – if seconds is negative, infinite or NaN.

static max_trials(n)#

Stop after a number of trials.

Parameters:: n (int) – the number of trials.
Returns:: the rule.
Return type:: Stop

static min_errors(metric, errors)#

Stop after a number of errors of an error rate.

Parameters:

metric (str) – the error rate.
errors (int) – the number of errors.

Returns:

the rule.

Return type:

Stop

static min_trials(n)#

Never stop before a number of trials (combine with &).

Parameters:: n (int) – the number of trials.
Returns:: the rule.
Return type:: Stop

static precision(metric, relative, *, confidence=0.95)#

Stop when the confidence interval of a metric is within a relative half-width of its value.

Parameters:

metric (str) – the metric.
relative (float) – the relative half-width.
confidence (float) – the level of the interval.

Returns:

the rule.

Return type:

Stop

to_spec(*, confidence=0.95)#

The stop mapping of a specification whose rule is this one (the inverse of from_spec()): a specification stops when all its targets (min_errors, precision, with min_trials) are met, or any of its caps (max_trials, max_time_s).

Parameters:: confidence (float) – the confidence level of the specification, that of every precision target.
Returns:: the mapping.
Return type:: dict
Raises:: ValueError – for a rule the format cannot write (targets combined by |, a cap inside an &, a precision target at another confidence level).

Examples

>>> from comnum.sim import Stop
>>> (Stop.min_errors("ber", 100) | Stop.max_trials(1000)).to_spec()
{'min_errors': {'ber': 100}, 'max_trials': 1000}

bounded#

Whether the rule can stop (a rule made only of lower bounds runs forever).

Type:: bool

comnum.sim.grid(**axes: Any) → list[dict[str, Any]]#

The Cartesian product of the axes, the last one varying fastest (the grid of a specification).

Parameters:: **axes – the values of every variable: a list, {"from": a, "to": b, "step": s} or {"linspace": [start, stop, count]}.
Returns:: the conditions.
Return type:: list[dict]

Example

>>> from comnum.sim import grid
>>> grid(m=[4, 16], snr_db=[0.0, 10.0])
[{'m': 4, 'snr_db': 0.0}, {'m': 4, 'snr_db': 10.0}, {'m': 16, 'snr_db': 0.0}, {'m': 16, 'snr_db': 10.0}]

comnum.sim.interval_coverage(method, n, p, confidence=0.95)#

The exact coverage probability of a binomial interval, P(L(X) <= p <= U(X)) for X ~ Bin(n, p): the sum of the binomial probabilities of the counts whose interval contains p (Brown, Cai & DasGupta 2001, Sec. 2), no simulation.

Parameters:

method (str) – "wald", "wilson", "agresti_coull", "jeffreys" or "clopper_pearson" (see ErrorRate.interval()).
n (int) – the trials.
p (float) – the probability, in [0, 1].
confidence (float) – the nominal level of the interval.

Returns:

the coverage.

Return type:

float

Raises:

ValueError – for an unknown method, p outside [0, 1] or a level outside (0, 1).

Example

>>> from comnum import sim
>>> round(sim.interval_coverage("wald", 10, 0.5), 3)  # Brown et al., Table 1
0.891

comnum.sim.interval_length(method, n, p, confidence=0.95)#

The expected length of a binomial interval, sum_k (U(k) - L(k)) P(X = k) for X ~ Bin(n, p) (Brown, Cai & DasGupta 2001, Sec. 3.3), with the ends of ErrorRate.interval().

Parameters:

method (str) – the interval (see interval_coverage()).
n (int) – the trials.
p (float) – the probability, in [0, 1].
confidence (float) – the nominal level of the interval.

Returns:

the expected length.

Return type:

float

Raises:

ValueError – for an unknown method, p outside [0, 1] or a level outside (0, 1).

comnum.sim.run(trial: Callable[[...], None], conditions: Iterable[Mapping[str, Any]], *, name: str, seed: int = 0, batch: int = 32, stop: Stop | Mapping[str, Any] | None = None, setup: Callable[[Mapping[str, Any]], Any] | None = None, confidence: float = 0.95, progress: bool = False, csv_path: str | Path | None = None, json_path: str | Path | None = None) → list[ConditionResult]#

Run a scenario on every condition, on the Rust runner (one thread).

Parameters:

trial (callable) – trial(condition, rng, record), or trial(condition, rng, record, state) when setup is given.
conditions (list[dict]) – the conditions, for instance from grid().
name (str) – the scenario name (it enters the seeds).
seed (int) – the base seed.
batch (int) – trials between two checks of the stopping rule.
stop (Stop | dict | None) – the stopping rule, or the stop mapping of a specification (default: 100 trials); it must be bounded.
setup (callable | None) – setup(condition) -> state, run once per condition.
confidence (float) – level of the intervals of the reports.
progress (bool) – print one line per condition on the standard error.
csv_path (str | os.PathLike | None) – write one row per condition to this CSV file (the columns of the Rust logger).
json_path (str | os.PathLike | None) – write the report of the run to this JSON file.

Returns:

one result per condition, in order.

Return type:

list[ConditionResult]

Raises:

ValueError – for a stopping rule that can run forever. An exception of a trial or of the setup is raised again with a note naming the trial and the condition.

comnum.sim.z_score(confidence)#

The two-sided normal quantile of a confidence level (1.96 for 0.95).

Parameters:: confidence (float) – the level, in (0, 1).
Returns:: the quantile z.
Return type:: float
Raises:: ValueError – for a level outside (0, 1).