Validation Report Artifacts

NextStat produces two categories of validation artifacts: Apex2 JSON reports (shipped) and a unified validation report pack (JSON + PDF) via nextstat validation-report.

Quick Start

# Generate JSON validation report
nextstat validation-report \
  --apex2 tmp/apex2_master_report.json \
  --workspace workspace.json \
  --out validation_report.json \
  --deterministic

# Also generate a 7-page audit-ready PDF
nextstat validation-report \
  --apex2 tmp/apex2_master_report.json \
  --workspace workspace.json \
  --out validation_report.json \
  --pdf validation_report.pdf \
  --deterministic

# JSON-only mode (no PDF, no matplotlib dependency)
nextstat validation-report \
  --apex2 tmp/apex2_master_report.json \
  --workspace workspace.json \
  --out validation_report.json \
  --deterministic

# Single entrypoint (Apex2 + JSON + PDF in one command)
make validation-pack

Schema

The output follows validation_report_v1 schema. Retrieve it via:

nextstat config schema --name validation_report_v1

Report Structure

Section	Content
dataset_fingerprint	SHA-256 of workspace JSON, channel/bin/sample counts, observation summary (total, min, max)
model_spec	POI, parameter names/bounds/constraints, interpolation codes, objective, optimizer
environment	NextStat version, Python version, platform, Rust toolchain, pyhf version, determinism settings
apex2_summary	SHA-256 of master report, per-suite pass/fail status, worst-case deltas, overall verdict
regulated_review	Data handling notes, intended use, scope, limitations, risk-based assurance matrix

Apex2 Validation Suites

The Apex2 master runner aggregates results from all validation suites into a single deterministic JSON:

Suite	What it validates
pyhf	NLL + expected_data parity vs pyhf at init/random/POI-varied points
histfactory_golden	Deterministic regression vs pre-recorded reference values
regression_golden	OLS, logistic, Poisson, NegBin coefficient/SE/NLL parity vs R/statsmodels
survival	Contract tests + Cox PH parity vs statsmodels (Efron/Breslow)
pharma_reference	Analytic PK reference + NLME fit smoke tests
timeseries	Kalman filter/smoother/EM validation
nuts_quality	Divergence rate, R-hat, ESS, E-BFMI floors
sbc	Simulation-based calibration rank uniformity (NUTS)
bias_pulls	Toy-based μ̂ bias and pull width checks
root	3-way profile scan comparison (ROOT/RooFit vs pyhf vs NextStat)

PDF Output (7 pages)

Cover — schema version, overall pass/fail, deterministic flag, workspace + Apex2 SHA-256
Dataset Fingerprint — workspace hash, channels, bin counts, observation summary
Model Spec — POI, parameter table, interpolation codes, optimizer settings
Apex2 Summary — per-suite pass/fail table with worst-case deltas
Environment — NextStat/Python/pyhf versions, platform, determinism settings
Regulated Review — intended use, scope, limitations, risk-based assurance
Footer — "Generated by nextstat validation-report (OSS baseline)"

PDF rendering requires matplotlib (install via pip install "nextstat[viz]"). In deterministic mode, PDF metadata timestamps are pinned for reproducible output.

Manifest & Signing

The validation pack produces a validation_pack_manifest.json with SHA-256 hashes and file sizes for all pack artifacts. This manifest serves as a convenient index for replication and optional cryptographic signing.

validation_pack_manifest.json — SHA-256 + sizes for all core pack files
Optional: --sign-openssl-key / --sign-openssl-pub flags produce Ed25519 signatures
Outputs: .sha256, .sha256.bin, .json.sig, and (optionally) .pub.pem files
Verify: openssl pkeyutl -verify -pubin -inkey ... -rawin -in ... -sigfile ...

# Sign the manifest with Ed25519
bash validation-pack/render_validation_pack.sh \
  --out-dir tmp/validation_pack \
  --workspace workspace.json \
  --deterministic \
  --sign-openssl-key tmp/manifest_priv.pem \
  --sign-openssl-pub tmp/manifest_pub.pem

Target Consumers

HEP — analysis preservation, reinterpretation readiness (RECAST / pyhf ecosystem)
Pharma — IQ/OQ/PQ validation packs for regulatory submission support (21 CFR Part 11)
FinTech — model risk management documentation (SR 11-7 / SS1/23 model inventory)

Open-Core Boundary

The OSS baseline provides validation_report.json (machine-readable, deterministic) and validation_report.pdf (matplotlib-rendered). Enterprise extensions (commercial license) may add branded PDF templates, digital signatures, audit trail integration, and automated scheduling.

Trust-Building Use

Publish validation_report.json + PDF for each tagged release (or weekly CI)
Link from: homepage footer (Trust), pricing page (Compliance), enterprise/security page
Attach to outbound emails for regulated prospects (IQ/OQ/PQ, SR 11-7 style review)
Use the JSON schema to make claims auditable: "pass/fail + worst-case deltas + hashes"

Report System Public Benchmarks CLI Reference Trust Offensive (Blog)