Benchmarks

​

Running Benchmarks

​

Prerequisites

• CUDA-capable NVIDIA GPU (compute capability 7.0+; development device: RTX PRO 3000 Blackwell, SM120)
• CUDA Toolkit 13.x
• Decision-DNNF knowledge compiler (for exact inference benchmarks that use external compilation)
• Sufficient GPU memory (4GB minimum, 12GB recommended for neural-symbolic training)

​

Quick Start

# Run all benchmarks
cargo bench

# Run specific benchmark suite
cargo bench -p xlog-gpu    # Transitive closure, joins, aggregation
cargo bench -p xlog-prob   # Exact inference, Monte Carlo
cargo bench -p xlog-stats  # Statistics manager
cargo bench -p xlog-solve  # SAT solver

# Run specific benchmark group
cargo bench -p xlog-gpu -- tc_benches
cargo bench -p xlog-gpu -- join_benches

# Generate HTML report
cargo bench -- --save-baseline baseline_name
cargo bench -- --baseline baseline_name

​

Environment Variables

​

Benchmark Categories

​

GPU Logic Benchmarks (xlog-gpu)

​

Transitive Closure

• tc_chain: depth 100, 500, 1000, 2000
• tc_random: 10K, 100K, 1M edges
• tc_dense: K_{100,100}, K_{200,200}, K_{500,500}

​

Hash Join Throughput

• Cardinalities: 10Kx10K to 1Mx100K
• Key ranges: 100 (high selectivity) to 100K (low selectivity)
• Multi-way: 10K, 50K, 100K rows per relation

​

Aggregation

• 100K rows with 1K groups
• 100K rows with 10K groups
• 1M rows with 10K groups
• 1M rows with 100K groups

​

WCOJ Triangle (default-on adaptive, xlog-integration)

# Default matrix (~25 cells; few minutes)
cargo bench -p xlog-integration --bench wcoj_triangle_bench

# Full matrix adds 100K + 250K rows per relation (slow)
WCOJ_BENCH_FULL=1 cargo bench -p xlog-integration --bench wcoj_triangle_bench

• Timed region = Executor::execute_plan only. Driven via b.iter_custom(...) so the per-iteration loop is owned by the harness. Each cell builds ONE long-lived Executor; put_relation uploads + store.remove("tri") cleanup live OUTSIDE the timed region. The long-lived Executor is required so the executor’s cached wcoj_triangle_stream (OnceLock<StreamId>) is acquired exactly once per cell and reused — a fresh Executor per iteration would drain the runtime’s StreamPool (cap 16, grow-only) past iteration 16.
• Each (width, fixture, size) cell pre-runs an untimed correctness check: gate=Some(false) (binary-join) and gate=Some(true) (WCOJ) must produce identical row sets (host-side dedup of fixtures aligns the two paths to set semantics). Counter delta is also asserted inside iter_custom: gate=true must increment by iters over the loop, gate=false must increment by 0 — a silent fallback anywhere in the hot loop fails the bench.
• Bench-only: the StreamPool cap is bumped to 1024 in make_provider (production default 16). The bench has many short-lived correctness-check executors that each acquire one stream; production runs at 16 because each long-lived process has one provider with one cached stream.
• Baseline numbers, adaptive default-on acceptance, phase-timing evidence, and the post-layout-fast-path results are indexed in the WCOJ bench baseline evidence bundle. Default-on adaptive WCOJ for eligible non-recursive triangle rules ships with XLOG_DISABLE_WCOJ_TRIANGLE=1 as the hard kill switch; the WCOJ subsystem now covers triangles, cost-aware planning, recursive/SCC integration, and K-clique coverage — see the WCOJ architecture guide.

​

Probabilistic Benchmarks (xlog-prob)

​

Exact Inference (Decision-DNNF)

• Path lengths: 5, 10, 15, 20, 25 nodes
• Grid sizes: 3x3, 4x4, 5x5, 6x6
• Bayesian: 10, 20, 30, 50 variables

​

Monte Carlo Inference

• Sample counts: 1K, 5K, 10K, 50K, 100K
• AD counts: 10, 50, 100, 500, 1000
• Path lengths: 10, 25, 50, 100, 200
• Grid sizes: 5x5, 10x10, 15x15, 20x20
• Bayesian: 50, 100, 200, 500 variables

​

Statistics Manager Benchmarks (xlog-stats)

​

Solver Benchmarks (xlog-solve)

​

Methodology

​

Measurement Approach

​

Throughput Calculation

Throughput = Elements / Time

Elements vary by benchmark type:
- Transitive closure: input edges
- Joins: total input rows (left + right)
- Aggregation: input rows
- Exact inference: circuit variables
- MC inference: samples × variables

​

Warm-up Protocol

1. PTX modules are compiled and cached
2. Memory pools are initialized
3. CUDA context is established

​

Reproducibility

• LCG with fixed seed (no system entropy)
• Same seed produces identical graphs

​

Baseline Metrics

Status of the tables below (audited 2026-06-10): the Transitive Closure, Hash Join, Exact Inference, and Monte Carlo tables are aspirational targets, not measured results. No published in-repo run backs them — the Criterion harnesses exist (crates/xlog-gpu/benches/, crates/xlog-prob/benches/) but their output is git-ignored and no baseline has been committed. Do not cite these numbers as evidence. Measured, source-backed results in this repo are: the WCOJ super-hub speedups (10.5×–33.8×, docs/evidence/2026-05-01-wcoj-bench-baseline/) and the neural-symbolic cache ablation below (2.74×, CI-backed, measured 2026-02-18).

​

Transitive Closure (targets — unmeasured)

​

Hash Join (targets — unmeasured)

​

Exact Inference (targets — unmeasured)

​

Monte Carlo (targets — unmeasured)

​

Neural-Symbolic Training

​

Interpreting Results

​

Criterion Output

tc_random/edges/100K_edges
                        time:   [12.345 ms 12.456 ms 12.567 ms]
                        thrpt:  [7.9567 Melem/s 8.0283 Melem/s 8.1003 Melem/s]
                 change: [-2.5% -1.2% +0.1%] (p = 0.12 > 0.10)
                        No change in performance detected.

​

Performance Regression Detection

1. change lower bound > +5%
2. p < 0.10

​

Common Issues

​

CI Integration

​

GitHub Actions Workflow

# .github/workflows/bench.yml
name: Benchmarks

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  benchmark:
    runs-on: [self-hosted, gpu]
    steps:
      - uses: actions/checkout@v4

      - name: Run benchmarks
        run: cargo bench --no-fail-fast -- --save-baseline pr_${{ github.sha }}

      - name: Compare to main
        if: github.event_name == 'pull_request'
        run: |
          git fetch origin main
          git checkout origin/main
          cargo bench --no-fail-fast -- --save-baseline main_baseline
          git checkout -
          cargo bench --no-fail-fast -- --baseline main_baseline --load-baseline pr_${{ github.sha }}

      - name: Upload results
        uses: actions/upload-artifact@v4
        with:
          name: benchmark-results
          path: target/criterion/

​

Regression Alerts

• 10% regression vs main branch

• Statistical significance p < 0.05

​

Benchmark History

• target/criterion/ (local)
• GitHub Actions artifacts (CI)

​

Contributing Benchmarks

​

Adding a New Benchmark

1. Create benchmark function:

fn bench_new_feature(c: &mut Criterion) {
    let mut group = c.benchmark_group("new_feature");
    group.sample_size(10);

    for size in [100, 1000, 10000].iter() {
        group.throughput(Throughput::Elements(*size as u64));
        group.bench_with_input(
            BenchmarkId::new("size", size),
            size,
            |b, &size| {
                b.iter(|| {
                    // Benchmark code here
                    black_box(operation(size))
                });
            },
        );
    }

    group.finish();
}

2. Add to criterion group:

criterion_group!(
    name = my_benches;
    config = Criterion::default();
    targets = bench_new_feature
);

criterion_main!(my_benches);

3. Add to Cargo.toml:

[[bench]]
name = "my_bench"
harness = false

[dev-dependencies]
criterion = "0.5"

​

Benchmark Guidelines

​

Review Checklist

• Benchmark measures meaningful operation
• Throughput metric is appropriate
• Parameters cover realistic range
• Handles missing GPU gracefully
• Documentation updated

​

See Also

• Architecture — System design
• Roadmap — Development plans