Skip to content

0.4.2 audit and repeatable verification

This iteration starts from 246bae9 (published 0.4.1). The baseline's 332 non-live tests passed, but targeted adverse-input checks reproduced the gaps below. A passing suite is evidence for its tested contracts, not proof that all failure modes have been found.

Finding Repair and regression evidence
Missing explicit config silently returned defaults config.not_found; test_config_boundaries.py
Config inspection exposed gateway and nested secret keys Recursive key/custom-field masking; same test file
Unknown inspection selector returned empty success CLI exit 2; same test file
Zero sampling fraction selected every record; invalid arguments escaped validation Count/fraction validation; test_dataset_boundaries.py
Negative split fractions duplicated records and disagreed with manifest Finite, nonnegative normalized partitions; same test file
External record/metadata/lineage and callback mutation changed immutable source Deep snapshots at mutable boundaries; same test file
Nested/shared context exit reset another entry/task's token Task-local token stacks; test_context_schedule_boundaries.py
Replaced schedule ran its old handler; rejected replacement changed working handler Validate before mutation, discard old handler; same test file
Infinite intervals/poll periods were accepted Finite timing checks; same test file
Doctor live probes skipped config credentials and retained clients Resolved credentials, bounded probes and runtime cleanup; test_diagnostics.py

Regression tests were run failing before implementation and passing afterward. The new local diagnostic feature exercises generation, embedding, a tool and cited retrieval without using configured cloud providers or telemetry exporters.

Repeatable repository loop

uv sync --locked --no-default-groups --group dev
make audit
# .audit/core.json and .audit/core-logs/*.log

The 14 core checks cover the lock, lint, formatting, types, core tests, offline doctor smoke, strict docs, release metadata, build, external wheel installation and four behavioral examples. CI runs this same profile on Linux/macOS and Python 3.11–3.13 and uploads evidence even on failure. Reports/logs must be in a Git-ignored directory or outside the repository. Missing tools, failures and timeouts are recorded per check; later checks still run. Inspect individual logs for details.

uv sync --locked --no-default-groups --group dev --all-extras
# Install system FFmpeg and Tesseract first.
make audit-optional
# .audit/optional.json and .audit/optional-logs/*.log

Optional audit explicitly checks CPU-test prerequisites before running all non-live tests; missing prerequisites fail its report. This command preserves the existing environment. Thread bounds, Torch-backed Keras and offline Hugging Face settings are enforced in child checks. Live Docker/service QA remains separately available through scripts/verify_services.py --wheel <path>.

Reports include start/end commit, dirty state and content fingerprint (tracked plus nonignored untracked source), per-check source states, selected commands, logs and durations. An in-progress/interrupted report is unhealthy; unrun checks are explicit. Any source change observed before/after a check permanently invalidates otherwise passing checks, even if a later check restores the original tree. Edits restored entirely inside one command cannot be detected by endpoint fingerprints; run release QA on a stable CI checkout. Each check has a finite timeout (default 600 seconds); on timeout/interruption its process group is terminated on the supported POSIX CI platforms. The CLI routes SIGINT, SIGTERM and SIGHUP through cleanup, restoring previous signal handlers afterward. An uncatchable SIGKILL or machine crash cannot run Python cleanup. The runner does not install packages or silently repair source. Inspect the failure, add a failing regression, fix it, and run the affected checks followed by the selected full profile again.

QA and release boundary

Local macOS arm64/Python 3.13 validation: core audit passed all 14 checks; optional CPU audit passed 396 tests with 11 live tests deselected. Four upstream warnings concern Starlette/httpx, PEFT's Conv1D setting and MPS pinning. These do not establish paid-provider availability, GPU correctness or production throughput. Ruff, strict mypy (190 source files), strict MkDocs, workflow lint, fresh wheel installation and the resolved production dependency vulnerability audit also passed. Independent review completed its five-defect fix pass; exact-source CI is required before merge/publication.

Current publication status is recorded by the 0.4.2 GitHub release and PyPI version, not inferred from this source-preparation record. The release workflow separately requires passing exact-main CI, CodeQL, Docs and Benchmarks, verified wheel/sdist hashes, fresh installation and publisher/GitHub success. See publishing.

Independent review and dispositions

The fresh read-only branch review covered 246bae9..07c8b36. It found five Important issues and no Critical/Minor issues. All entered one regression fix pass: bounded ambient context variables (10,000 completed-scope probe), source changes restored between checks, SIGTERM process-group cleanup/restored handlers, invalid UTF-8 diagnostic reports, and ordinary/stepped/reversed/empty dataset slices with isolated records. Eight new cases failed before the repairs; the combined focused suite then passed 59 tests. Final full QA follows below in the GitHub PR record; no second reviewer was used to substitute for these regressions.

The reviewer also set aside the following boundaries. These decisions retain explicit limits rather than claiming unsupported behavior:

  • Fingerprints cannot detect edits restored entirely within one command; detect and latch every observed boundary change. Cost if wrong: a transient edit can make local evidence incomplete; release QA uses a stable checkout.
  • Secret values under arbitrary unrecognized keys require mask_fields; key-based masking is not a secret classifier. Cost if wrong: an unlisted secret could appear in inspection output.
  • Wrong-type sampling arguments such as string fractions are outside the typed numeric API. Cost if wrong: untyped callers receive Python type errors instead of a structured validation error.
  • AIRE_CONFIG selection remains unsupported; use an explicit configuration path or documented discovery. Cost if wrong: a caller expecting that undocumented environment variable may load defaults instead.
  • Hosted-provider availability and production load remain unverified by local review; verify actual publication separately through the planned workflow and public-byte/install checks. Cost if wrong: an untested hosted/load scenario may fail; a release is never inferred from source metadata.