Skip to content

Testing Guide

Running Tests

Warning

Running all tests simultaneously may cause errors due to resource constraints. It is recommended to run per file or small groups.

Running Unit Tests

python -m pytest tests/unit/core/test_block.py -v

Running Integration Tests

python -m pytest tests/integration/test_data_flow.py -v

Running Scenario Tests

python -m pytest tests/scenarios/test_recovery.py -v

Running Benchmark Tests

python -m pytest tests --benchmark-only -v --benchmark-save=benchmark_report
python -m pytest tests --benchmark-only -v --benchmark-histogram=benchmark_report

Running All Tests

for test_file in $(find tests/unit tests/integration tests/scenarios -name 'test_*.py' | sort); do
    python -m pytest "$test_file" -v || break
done

Consensus integration, 2PC, data-flow and recovery tests use per-case temporary SQLite databases and journals. They do not require the PostgreSQL service from the application's .env; dedicated live-backend tests exercise PostgreSQL.

Required CI gates

CI installs uv.lock with uv sync --frozen --extra dev and runs test files sequentially. A separate PostgreSQL 16/Redis 7.4 job provisions isolated databases and separate audit-manifest roles, then uses --fail-on-skip for its required backend contracts. Local backend tests remain optional when their service URLs are absent; IPFS daemon checks need a separate running daemon.

The required ty gate checks the entire hierachain/ package. The older mypy ignore settings do not apply to it. Run the same check locally:

uv run --frozen --extra dev ty check hierachain/

Durable Throughput Benchmark in Docker

Run the benchmark from the current source tree with PostgreSQL 16, 1 CPU, 1 GiB RAM, and durable journal writes enabled:

docker compose -f docker/docker-compose.benchmark.yml up --build --abort-on-container-exit

Override the workload with BENCHMARK_EVENTS, BENCHMARK_BATCH_SIZE, BENCHMARK_CPUS, and BENCHMARK_MEMORY. Journal fsync always stays enabled. Run down -v before a clean-database comparison; clean any Compose leftovers with:

docker compose -f docker/docker-compose.benchmark.yml down -v

Docker-specific Runtime Tests

Run deterministic checks for the image, native dependencies, journal replay, and PostgreSQL adapter wiring without starting the four-node cluster:

docker compose -f docker/docker-compose.test.yml \
  --profile docker-test run --rm docker-tests

The profile starts an isolated PostgreSQL 16 service and stores the append-only journal under /app/data. Journal fsync is always enabled. Override DOCKER_TEST_CPUS or DOCKER_TEST_MEMORY to change the container limit.

Stress Testing

Docker Stress Testing

Production stress requires HRC_API_KEY in the root .env or environment, using a key already provisioned in HRC_API_KEYS_SOURCE_FILE with chains, events, and proofs permissions (or all). Never paste the key into logs or reports. The launcher checks that a key is configured before deployment; pytest checks authenticated chain access on every node before running tests. Missing keys, 401/403/429 responses, and unreachable nodes stop the session. HTTP and WebSocket clients use the same key and HRC_API_KEY_NAME (default X-API-Key).

The Docker source stage exports dependency constraints from uv.lock; the production stage installs the wheel and test tools under those constraints and runs pip check. Rebuild the image after changing the lockfile.

After source changes, rebuild the wheel and image. The wheel build clears generated build/ so deleted modules cannot survive in setuptools' cache:

bash -c 'source docker/lib/common.sh; build_wheel'
docker build --target production -t hierachain:latest -f docker/Dockerfile .

Once the cluster uses the new image, run with --reuse to preserve its deployment and volumes. The default stress command redeploys and removes volumes. Check an individual file before running the full suite:

docker compose --env-file .env -f docker/docker-compose.yml --profile stress-test run --rm stress-tester python -m pytest docker/stress/test_real_network.py -v
bash docker/hierachain.sh stress docker --reuse

Crypto fixtures sign events and blocks and provide a trusted public key. Their benchmarks assert verification succeeds. For local crypto-only checks without network preflight, set REAL_REQUESTS=false; this does not validate live services.

--duration controls tests that read TEST_DURATION; it is not a time limit for the entire suite. The full tsunami flood still requests 5,000 events. Its send phase has a separate STRESS_TIMEOUT budget (default 120 seconds), after node readiness and chain creation. Workers stop at the deadline, queued batches are cancelled, and active HTTP calls use the remaining budget without flood retries. Incomplete runs report timed_out and unattempted_events and fail acceptance. The launcher shows each test result, short failure tracebacks, and the ten slowest tests. Live logs, captured console logs, and timed thread stack dumps are hidden by default; INFO logs remain captured for the HTML report, alongside the XML results.

STRESS_TIMEOUT=120 bash docker/hierachain.sh stress docker --reuse --duration 15

An HTTP 200 from /api/ledger/health confirms liveness; /api/ledger/ready checks hierarchy recovery. Stress readiness uses /api/ledger/ready without falling back to liveness; node polling shares its timeout across requests, without automatic HTTP retries. Chain setup requires every configured node, and poison setup probes each node once. Failed hierarchy bootstrap closes its storage pool, coordinator journal, and any started sub-chains. Concurrent bootstrap requests receive HTTP 503 immediately; failed attempts have a five-second retry cooldown. The signed event envelope permits its top-level sender field while business content still follows terminology checks; signature validation remains required by the admin API. Ordering startup validates persisted block links before replaying the journal. A Chain link BROKEN error requires recovery from trusted data or an explicitly approved reset of disposable test data. Rebuilding an image does not repair persisted blocks. Sub-chain startup waits for journal replay to finish, without cancelling a valid backlog after ten seconds; processor failure or shutdown still aborts bootstrap. Sub-chain rehydration updates local state without rewinding the active orderer's block index or cache.

Sub-chain blocks complete consensus, respect its minimum block interval, and receive the trusted header signature before the orderer saves them. The consumer validates and applies these committed blocks without changing their index, hash, events or signature, and never writes them again. Consensus or storage failure cannot advance the commit queue or block index. Concurrent consumer and flush calls drain the queue in order. HTTP consensus throughput checks wait up to 120 seconds for every accepted business event to commit, excluding the PoA consensus event in each new block. Throughput includes this drain time, so subsequent restart tests inherit no backlog from a passing throughput test. Recovery still requires the restarted primary to become ledger-ready within 60 seconds; they exercise a generic PoA chain and do not prove BFT protocol view changes.

IPFS stress setup creates the generic chain through the shared ledger client and requires success on every target node. Events supply entity_id directly to POST /api/ledger/chains/{chain_name}/events; no separate entity registration endpoint is required.

Run stress tests in Docker containers with 4 HieraChain nodes (1 CPU, 1GiB RAM each):

  • Build and run stress tests with HTML report:

    docker compose -f docker/docker-compose.yml --profile stress-test run --rm stress-tester python -m pytest docker/stress/ -v --html=/app/log/report/stress_test_report.html --self-contained-html

  • Run real network stress tests (sends actual HTTP requests to nodes):

    docker compose -f docker/docker-compose.yml --profile stress-test run --rm stress-tester python -m pytest docker/stress/test_real_network.py -v -s

  • Run without HTML report:

    docker compose -f docker/docker-compose.yml --profile stress-test run --rm stress-tester

  • Stop and clean up containers:

    docker compose -f docker/docker-compose.yml down --remove-orphans

Reports are saved to log/report/ directory.

Kubernetes Stress Testing

The unified wrapper targets an existing Kubernetes cluster. For local OrbStack:

orbctl start k8s
bash docker/hierachain.sh stress k8s --duration 15
bash docker/hierachain.sh stress k8s --reuse --duration 15

The first command builds and deploys four SQLite-backed nodes, four private IPFS pods, Redis and the gateway. Deployment preserves existing PVCs and keys; K8s identities are generated separately from Docker identities. Workloads and all node/IPFS pods must become ready before testing. Other clusters must have hierachain:latest loaded or available in their image registry.

Pytest runs in the cluster through kubectl exec, targets all four node DNS names and verifies their API readiness before collection. The configured IPFS API must also be reachable; otherwise preflight fails. HTML and JUnit reports are copied to log/report/. The wrapper removes the test runner and returns pytest failures without a Docker fallback. --duration controls load test duration; the full suite includes longer chaos and recovery checks. Docker-only checks may skip on K8s; review the recorded reasons.

These manifests use HRC_ENV=test, disable authentication and give test node pods NET_ADMIN for network fault injection. The runner has namespace-scoped permission to get/delete and exec into the four node pods. Use them only for isolated testing. setup k8s --force and down k8s delete the namespace and its PVCs; setup k8s preserves data by default.

Developer Scripts

The scripts/ directory contains utility tools.

Static Analysis

# Run default
python -m scripts.static_analysis

# Export results to file
python -m scripts.static_analysis --output analysis_report.json

Security Auditing & Code Scanning

Automated security checks across codebase and dependencies:

  • Bandit (SAST security analysis for Python code):

    # Full security scan across all severity levels
uv run --frozen --extra dev bandit -r hierachain/

# Scan Medium and High severity issues only
uv run --frozen --extra dev bandit -r hierachain/ -ll

  • pip-audit (Vulnerability & CVE scanner for packages):

    # Scan all installed dependencies
uv run pip-audit

# Strict mode (fail on any vulnerability)
uv run pip-audit --strict

  • Bandit CI gate (high-severity findings block the job):

    uv run --frozen --extra dev bandit -r hierachain/ -lll

The Bandit CI job also stores a full JSON report, including medium- and low-severity findings. The existing CodeQL workflow provides additional Python security analysis. Bandit replaces the local Semgrep dependency; its rules do not provide identical coverage. Review findings against the source before treating them as confirmed vulnerabilities.

Benchmarking

  • Hashing Performance (Compare Merkle tree hash vs JSON):

    python scripts/benchmark_hashing.py

  • Throughput Benchmark (Measure event processing throughput):

    python scripts/benchmark_throughput.py --events 1000 --batch-size 100

Storage Verification

  • Verify Storage Persistence (Test local storage durability):

    python scripts/verify_storage.py

Pytest Configuration (excerpt from pyproject.toml)

  • testpaths = ["tests/unit", "tests/integration", "tests/scenarios"]
  • python_files = "test_*.py"
  • python_classes = "Test*"
  • python_functions = "test_*"

  • Markers:

    • critical, high, medium, low
    • integration, recovery, stress

Running by Marker Example

python -m pytest tests/integration/test_data_flow.py -v -m critical
python -m pytest tests/integration/test_data_flow.py -v -m integration

Testing Principles

  • Focus on public API behavior of the module.
  • Test edge cases and error scenarios.
  • Keep tests independent, runnable by marker.

Test Layout Suggestions

  • Unit: test individual classes/functions (core, security, storage...).
  • Integration: test end-to-end flows via API Ledger/business.
  • Scenarios: business scenarios (e.g. create sub-chain → write event → submit proof → trace entity).