Guides

OPT-125m sample: GPU profiling → instruction simulation → request replay

Full-request result#

254.45 ms real H100 → 262.14 ms simulated H100 (+3.02%), mean of steady repeats 1–3 for the 32-token request. Mean TTFT is 8.63 ms real versus 9.00 ms simulated (+4.3%); mean ITL is 7.93 ms versus 8.17 ms (+3.06%). The native A100 reference is 222.08 ms real → 229.58 ms simulated SM80_A100 (+3.38%). The H200 counterfactual is 262.16 ms, with no measured H200 baseline or accuracy error. The tables below give the per-repeat request numbers and a compact kernel-error breakdown.

These headline values use the precision of the OPT-125m results draft; detailed tables below retain the committed artifacts' precision and distinguish the original verification run from the later Sail sweep. The ~3 µs (~6,000-cycle) per-kernel floor is the inferred main source of tiny-kernel over-prediction; calibrating it offers the largest GPU-accuracy payoff.

Each steady repeat#

Where the error comes from#

The profiler measures unprofiled request/step timing and Nsight kernel timing separately. NVBit traces four steps (prefill and decode at contexts ≈ 8, 16, 32); Accel-Sim simulates 31 distinct kernels, then propagates durations to 17,088 steady kernel launches. Replay swaps in simulated kernel durations while retaining measured host residuals, gaps and unmatched GPU activity.

Kernel means compare the 712 aligned traced occurrences, not full-request-weighted decodes; group roles are inferred from kernel names. Interpretation: a ~3 µs (~6,000-cycle) per-kernel floor makes tiny kernels 1.5–2.3× too slow. With 178 kernels per step, small elementwise, layer norm, KV-write and split-K kernels account for about 200 µs of the ~240 µs step over-prediction (derived). Calibrating that floor is the main accuracy opportunity.

Across architectures#

H100 traces run on H100/H200 (sm_90); older architectures are rejected for mismatch. H200 differs here only in memory-controller count (40 → 48), yielding +0.02 ms per request. Inference: this small workload does not separate the configs; the simplified H200 model is not a validated physical H200.

Scope and evidence#

This analysis describes the H100 e2e-plan-v1 sample and native A100 e2e-a100-plan-v1 follow-up, not a generic model benchmark. Unless explicitly marked A100, the original sections below describe H100. The source is simulator@ed337cd. Measured values are computed from:

• /home/ubuntu/runs/phases/e2e-plan-v1/: run.json, workload.json, corrected_timeline.json, trace_alignment.json, phase logs/step logs, sim/kernels.csv, sim/summary.json, sim_durations.json, sim/compare.json.
• Committed hardware sweep: inventory, per-kernel/step/request CSVs, H100/H200 summaries and comparisons.
• Native A100 profiling and the complete Modal reference, with the secondary partial Sail evidence kept separate.

Absolute paths identify the collected evidence on the execution VM, not portable dependencies of Cascade. The hardware-sweep artifacts are committed; the large profiling run is retained in the run store. See the run-directory contract for file ownership and semantics. Numbers marked derived are arithmetic on these artifacts; inferred means interpretation, not an additional measurement.

A100 is complete: native profiling used f190c99 on Modal A100-80GB with local-disk tracing. The authoritative SM80_A100 simulation is the complete Modal run at accelsim-traces:hw-sweep/e2e-a100-plan-v1-sm80-v1/. Its real baseline was byte-identical to the collected A100 timeline. No H100 traces are substituted for native A100 SASS.

Workload and environment#

There are 32 steps/request: one prefill and 31 decode, not 33 steps. Repeat-local traced steps are 0, 2, 10, 26, at contexts 6, 8, 16, 32. The requested plan contains context 1, but this six-token prompt has no such decode context. All 128 timing steps exist; the three steady repeats have 96 profiled steps and 17,088 kernels (178 per step). Repeat 1 supplies the four instruction-traced steps: 712 aligned occurrences.

Pipeline cost#

These are stage wall seconds from the original run, not the later Sail sweep. The trace stage includes both spinlock-detection passes and the NVBit trace process. Compilation/image build, scheduling and downloads are excluded.

NVBit step envelopes come from traced/trace_steps.json monotonic timestamps:

The sum is 1515.010 seconds (derived), not the full 1738.577-second stage. Spinlock pass 0 and pass 1 step-log spans, from first step start to last step end, are 50.968 and 44.285 seconds. These spans exclude process/engine startup; they are not complete pass wall timings. Trace-process step span is 1541.794 seconds, also excluding its startup/teardown.

The logs explicitly report engine initialization (profile/KV-cache/warmup) of 85.80 seconds in baseline, 2.00 in profiled, 13.72 in spinlock pass 0, 13.68 in pass 1 and 13.78 in tracing. These overlap the stage totals and must not be added again. Baseline weight loading alone is only 0.41 seconds. Inference: engine initialization and cold/warm caches are significant; the difference between processes does not isolate any one cache's effect.

Local-disk optimization and artifact compatibility#

The old trace logs show TRACES_FOLDER/working paths under /__modal/volumes/.../phases/e2e-plan-v1/traced; the approximately 375-second decode envelopes above are measured on that path. This alone does not prove the fraction attributable to volume I/O. The separate earlier local-disk comparison mentioned in the task is not in the source artifacts above, so no measured speedup factor is asserted here.

Profiler commit 0b54e42 now runs both spinlock passes, tracing and post-processing under /tmp/cascade-trace-*/traced, then copies the result once into the run directory, measuring stage_wall_s.trace_collect. Failures also collect diagnostic files before scratch cleanup. This commit was tested and pushed before the A100 launch; f190c99 adds --gpu and builds NVBit with ARCH=all rather than an H100-only binary.

No bundle was introduced. The collected directory preserves the existing traced/traces/kernel-*.trace.xz, kernelslist.g and processed .tracez layout, plus traced/trace_steps.json, metrics, spinlock data and logs. The pipeline's post-processor provides the existing compressed trace format; collection is a directory copy, not tar/zstd. No extraction command or app ingestion change is needed. The separate legacy streaming S3 path is unchanged. The A100 run confirms 624 raw and 624 processed traces; collection took 3.912637 s.

Real timeline anatomy#

The following means cover three steady prefills and 93 steady decodes. host_ns = baseline_wall_ns − gpu_busy_ns is a residual decomposition: it is not independently instrumented exclusive CPU time. CUDA API time can overlap GPU work and is not an additional summand.

Decode gap-before mean/p50 is 0.036950/0.035911 ms. First-step gaps are null, not an extra request boundary delay. Request endpoint fields retain signed post-last-step intervals; they must not be replaced by zero.

Walls + gaps are not the whole envelope: the contract also preserves pre-first-step and signed post-last-step terms. ITL p50 is absent in the original request metrics, so none is invented.

Real kernel mix per step#

Counts and summed kernel durations below are averaged over all steady steps of each kind, not just the four instruction samples. Names are classified with results/hw-sweep/report.py::group: KV/attention names, nvjet GEMM grid-Y values 9/3/12/6/66 for QKV/out/up/down/lm_head, and elementwise/layer-norm/reduction names. Role labels are inferred, not NVTX annotations. Kernel sums need not equal step GPU busy (unmatched residual activity and the busy-time aggregation are different quantities).

Accel-Sim by kernel group#

These tables use the later Sail hardware sweep, where both configs ran 31 representatives with verification disabled. Means are over the same 712 aligned occurrences (one prefill and three sampled decodes); they are not weighted by the full request's 31 decodes. Real values are matched by (repeat, step, seq), not approximate name matching. cycles / MHz = µs. Error is signed (mean simulated µs / mean real µs − 1) × 100, not MAPE. The reference is Nsight-measured kernel time, not independently measured unprofiled per-kernel time; only step/request baselines are unprofiled.

H200 predictions are counterfactual, with no accuracy error shown because no real H200 baseline exists. The shipped H100/H200 configs both have 132 SMs, identical clocks and simple DRAM modeling; memory-controller count changes from 40 to 48. Near-identical request predictions do not imply physical H100/H200 equivalence.

Coverage, dedup verification and context scaling#

Both original and Sail replay cover 17,088/17,088 steady kernels (100%); each captured request has 5696/5696. Measured GPU-time coverage is 98.13%; the unmatched measured remainder is retained, not modeled by Accel-Sim. Warm-up has no comparable profiled kernels. Coverage is not an accuracy score.

The original has 31 header groups, 31 representatives plus 62 verification executions. The sum of individual simulation wall times is 2866.019 seconds, versus 718.709 orchestration seconds because kernels execute concurrently. Do not sum reused dedup-row wall fields as additional CPU executions.

Dedup keys contain launch/header properties, not full instruction/memory trace contents. Original verification therefore matters:

The largest mismatch is UID 6573 versus representative 6217 (indexSelectSmallIndex), not attention or a large GEMM. It occurs between different prefill/decode input contexts sharing a header. This demonstrates that header identity is not cycle identity; it does not quantify error for every unverified group member. A zero reference did not produce nonzero verification cycles in this sample.

First-layer original duration records show:

Thus this pipeline's output is flat across sampled decode contexts because it reuses representative cycles. This is not an independent observation of physical/context-invariant attention. The original decode attention/KV verification comparisons have zero difference, but those two checked members do not establish arbitrary context scaling. In the verification-free Sail sweep, all sampled decode groups satisfy identical propagation, so enabling propagate=interp actually yields zero interpolation labels. Inference: independent context simulations and a stronger cache/dedup key are required before making scaling claims; the current results do not provide them.

Replay errors and eager-mode dilution#

cascade simulate replaces modeled kernel durations while retaining host residuals, gaps, unmatched GPU activity and request envelopes. Both original and later runs use the same real H100 baseline. Positive signed errors below mean simulation is longer; the original step/GPU mean errors are all positive.

Original H100 p50/p95 absolute step errors are 240.540/242.402 µs.

Later sweep GPU busy decode/prefill mean is 816.980/822.679 µs for H100 and 817.620/823.136 µs for H200. Predicted E2E repeats 1/2/3 are 261.133918/262.302298/262.987898 ms and 261.154211/262.322591/263.008191 ms respectively.

Derived explanation: only about 7.29% of the mean decode step is GPU busy (0.575611 / 7.894492); about 92.71% is the retained host residual. Adding roughly 240 µs to GPU busy can therefore be a roughly 42% GPU error but only 3% step/E2E error. Eager-mode request agreement is a weak validation of GPU model fidelity. It is not evidence that a graph-mode or compute-dominated workload would have the same small request error. Keeping host/gaps fixed also makes the counterfactual insensitive to hardware-dependent host changes.

Architecture compatibility#

Trace binary version is 90; the stream contains Hopper-specific kernels, including FlashAttnFwdSm90 and SM90 GEMMs. H100 and H200 are the valid registered native targets. Older configs are rejected by the Sail app's architecture gate; preliminary pre-Sail probes also failed (for example, A100 hit the specialized-unit scheduler assertion). These are not valid A100 predictions. A real A100 run must choose native kernels and collect its own SM80 traces and unprofiled baseline.

The pinned and checked upstream-dev trees have no Blackwell/SM100/B200/B100/ GB200 configuration. H200 is the only shipped Hopper variant beyond H100. No H100-number-edited approximation was created. See the sweep inventory for all 12 tested and nine deprecated config files and their SM/clock evidence.

Bottlenecks and optimization order#

The lm_head is only one of 178 kernels/step and about 30.5 µs of real decode GPU time, yet its trace-driven CPU simulation takes hundreds of seconds. Inference: these jobs are likely critical-path drivers of the original 719-second orchestration, but CSVs do not store execution overlap intervals, so an exact critical-path decomposition is not available. The later Sail sweep likewise spends 773.080 s (H100) / 745.353 s (H200) in its single lm_head representative, compared with total walls 804.429/777.060 s. These CPU wall measurements must never be confused with modeled GPU latency.

Native A100 complete reference#

The same workload ran on NVIDIA A100-SXM4-80GB with vLLM 0.21.0, eager execution, 32 output tokens, four repeats and trace_steps=plan. All 624 trace headers identify SM80; alignment has zero unmatched records. Conversion uses the shipped SM80_A100 clock, 1410 MHz, not the profiled median clock of 1155 MHz.

The complete Modal run executed 33 representatives for 624 aligned occurrences with zero failures/timeouts, taking 565.086033 s orchestration wall. Verification was disabled: provenance is 33 exact, 591 dedup, 14,100 identical and zero interp. Coverage is 14,724/14,724 steady kernels (100%) and 98.2570% measured GPU time; unmodeled residual activity remains measured. Step-wall MAPE is 3.394%, but GPU-busy MAPE is 24.141%. Retained eager host time dilutes the request error; +3.38% E2E is not a sub-4% GPU-model accuracy claim.

Instrumented step envelopes were 625.564963/869.278675/813.347379/786.331915 s for local steps 0/2/10/26 (contexts 6/8/16/32). These do not reproduce the earlier 52-s local-disk anecdote. Different GPUs/native kernels prevent a controlled I/O-speedup claim; the remaining instrumentation cost has not been isolated.

Reproduce the A100 comparison#

Profiler (already includes both pushed fixes):

modal run accelsim_trace.py::e2e_main --gpu A100-80GB \
  --run-name e2e-a100-plan-v1 --model facebook/opt-125m \
  --prompt 'Hello, my name is' --max-tokens 32 --repeats 4 \
  --trace-steps plan --trace-repeat 1
modal volume get accelsim-traces /phases/e2e-a100-plan-v1 \
  /home/ubuntu/runs/phases

The committed Sail driver reproduces the secondary experiment through the deployed app. It ingests native SM80 traces and uses dedup=True, verify_dedup=2 and propagate="interp", saving results separately under sail-partial. Cleanup terminates main/child guests and deletes the checkpoint. It does not overwrite the complete Modal reference.

Then, from the simulator checkout:

RUN=/home/ubuntu/runs/phases/e2e-a100-plan-v1
SIM=results/a100-plan-v1/modal-complete
gzip -dk "$SIM/sim_durations.json.gz"
uv run cascade simulate "$RUN/corrected_timeline.json" \
  --durations "$SIM/sim_durations.json" --output "$SIM/simulated_timeline.json"
uv run cascade compare "$RUN/corrected_timeline.json" \
  "$SIM/simulated_timeline.json" --output "$SIM/compare.json"

These commands run from the simulator checkout at the pinned source commit. Both native A100 replays compare against the real A100 baseline, never the H100 baseline.