# Cascade docs

> Cascade traces an ML workload once on a real GPU and simulates it on many GPU configs at once with Accel-Sim.


# Introduction

Page: index.html

Cascade takes a trace of an ML workload captured on one real GPU and simulates it on other GPU configs, many copies at once. You trace once, then ask how the same kernels run on an H100, an H200 or a dozen replicas of each.

## How it fits together

Cascade has three parts. Each part runs in a different place, and they talk over S3 and plain HTTP.

Diagram: The profiler uploads to the app's S3 endpoint. The app forks one simulator per GPU config replica. Simulators send heartbeats and logs back to the app.

- **Profiler.** Runs your workload, such as a `vLLM` model, on a Modal H100 under Accel-Sim's NVBit tracer. It uploads each kernel's trace to the Cascade app while tracing is still running.
- **Cascade app.** One Sail sandbox that receives traces over S3, gets them ready to simulate, launches runs, and shows everything in a web UI and an HTTP API.
- **Simulators.** Each simulator is a fork of the app's sandbox, so it starts with the trace already on disk. It runs Accel-Sim against one GPU config and reports progress back to the app.

> Every workspace includes a sample run. Profiling a workload takes about 30 minutes, so each app opens with a finished one marked Sample: Alessio's facebook/opt-125m trace from an H100 and its Accel-Sim run on `SM90_H100`, with real step timing to compare against. Its results are real, recorded before your workspace existed, and you can't rerun it because its 415 MB of instruction traces aren't bundled. See [samples/opt125m-h100-e2e](https://github.com/atoniolo76/cascade/tree/amani/samples/opt125m-h100-e2e). Set `CASCADE_SAMPLES=false` to start without it.

## Worked example: OPT-125m

The same eager workload generates 32 tokens from `Hello, my name is`, with four repeats (one warm-up). H100 and A100 each have their own native traces and real unprofiled baseline; the H200 prediction replays the H100 traces and has no measured H200 baseline.

| Hardware / config | Real mean E2E | Simulated mean E2E | Error |
|---|---|---|---|
| H100 / SM90_H100 | 254.45 ms | 262.14 ms | +3.02% |
| H200 / SM90_H200 | — | 262.16 ms | — (counterfactual) |
| A100 / SM80_A100 | 222.08 ms | 229.58 ms | **+3.38%** |

The A100 result is the complete Modal SM80_A100 reference, not the partial Sail attempt. Small request errors retain measured eager host time and are not equivalent to small GPU-model errors. See the [full sample analysis](sample-analysis-opt125m.html) for timings, coverage and limitations.

To ask about a different prompt length, output length or concurrent arrival pattern, use the [request model and scheduler](request-simulation.html). The Request simulator UI and `/api/scenarios` build estimates from a finished sim; `/v1` estimates each request alone. Predictions outside the six-token prefill / decode-context 7–37 calibration range are **unvalidated extrapolations, not measured results**.

## Where to go next

- [Get started](../start.html): Create your own Cascade app on Sail from the browser with a workspace name and a Sail API key. No clone.
- [Quickstart](quickstart.html): Run the app on your laptop with the stand-in simulator and finish a run in a few minutes.
- [Concepts](concepts.html): Traces, runs, sims and GPU configs, their state machines, and how forking works.
- [Trace a workload](trace-a-workload.html): Capture a vLLM model on a Modal H100 and stream it into your app.
- [API reference](api.html): Every route, with real responses captured from a running server.
- [Bring your own simulator](simulator.html): The contract a program follows to run as a Cascade simulator.
- [Types](types.html): Every request and response model, generated from `cascade/models.py`.


# Quickstart

Page: quickstart.html

> To run Cascade on Sail without cloning anything, use [Get started](../start.html): name a workspace, paste a Sail API key, and open your app a few minutes later. This page runs the app on your laptop instead.

In this quickstart, we run the Cascade app on your laptop, upload a small stand-in trace, simulate it on two copies of the `SM86_RTX3070` config, and download the results.

The simulators here are `cascade worker --replay`, which replays a recorded Accel-Sim log of BabelStream on an RTX 3070 instead of running Accel-Sim. Every number you see comes from that log, not from the trace you upload. The point is to see the whole loop work before you deploy for real.

## Before you start

You need these on your machine.

- `git` and [uv](https://docs.astral.sh/uv/). uv installs Python 3.12 for you.
- `rclone`, which serves the app's S3 endpoint. On macOS, run `brew install rclone`.
- Node 22, only if you want the web UI. Everything else works without it.

## Run the loop

1. **Clone and install.**

   ```bash
   git clone https://github.com/atoniolo76/cascade.git
   cd cascade
   uv sync
   ```

   To get the web UI as well, build it once. The build lands in `cascade/static/`, which the app serves.

   ```bash
   (cd web && npm ci && npm run build)
   ```

2. **Start the app.** This runs the API and UI on port 8000 and the S3 endpoint on port 9000. Leave it running and open a second terminal for the next steps.

   ```bash
   export CASCADE_TOKEN=dev-token
   CASCADE_DATA_DIR=./data \
   CASCADE_ACCELSIM_DIR=$PWD/tests/fixtures/accelsim \
   CASCADE_SIM_CMD="$PWD/.venv/bin/cascade worker --replay $PWD/tests/fixtures/logs/babelstream_sm86_rtx3070.log" \
   uv run cascade serve
   ```

   When the app is up, the log ends with this line.

   ```text
   INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
   ```

3. **Make a stand-in trace.** The replaying worker only checks that `traces/kernelslist.g` exists, so one line is enough. A real trace comes from the profiler. See [Trace a workload](trace-a-workload.html).

   ```bash
   export CASCADE_TOKEN=dev-token CASCADE_URL=http://127.0.0.1:8000
   mkdir -p demo/traces
   echo kernel-1.traceg > demo/traces/kernelslist.g
   ```

4. **Upload it.** `scripts/upload_trace.py` uploads a directory the same way the profiler does and sends `metadata.json` last. The two `--meta` flags tell the app which GPU architecture the trace is for and which config to suggest.

   ```bash
   uv run python scripts/upload_trace.py demo --run-name demo \
     --meta trace_gpu_arch=sm_86 --meta suggested_sim_config=SM86_RTX3070
   ```

   You see each file go up.

   ```text
   uploaded traces/kernelslist.g (16 bytes)
   uploaded metadata.json (1 files listed) to s3://cascade/accelsim-traces/demo/
   ```

5. **Wait for the trace to be ready.** The app scans the bucket every 2 seconds. Because the upload already has `kernelslist.g`, it skips post-processing.

   ```bash
   curl -s -H "Authorization: Bearer $CASCADE_TOKEN" "$CASCADE_URL/api/traces/demo"
   ```

   Repeat until the response has `"state":"ready"`. It takes a few seconds.

6. **Launch a run.** This asks for two sims on `SM86_RTX3070` and saves the run's id.

   ```bash
   RUN=$(curl -s -X POST "$CASCADE_URL/api/runs" \
     -H "Authorization: Bearer $CASCADE_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"trace_id": "demo", "entries": [{"gpu_config": "SM86_RTX3070", "count": 2}]}' \
     | python3 -c 'import json, sys; print(json.load(sys.stdin)["id"])')
   echo $RUN
   ```

7. **Watch it finish.** Each sim replays about 10,000 log lines and finishes in a few seconds.

   ```bash
   curl -s -H "Authorization: Bearer $CASCADE_TOKEN" "$CASCADE_URL/api/runs/$RUN" \
     | python3 -c 'import json, sys; r = json.load(sys.stdin); print(r["state"], r["sim_counts"])'
   ```

   When both sims are done, you see this.

   ```text
   succeeded {'succeeded': 2}
   ```

   If you built the UI, open [http://127.0.0.1:8000](http://127.0.0.1:8000), paste `dev-token`, and open the run to see the same thing live.

8. **Download the results.** This saves one JSON file with the run, the trace, every sim's per-kernel stats, and the comparison.

   ```bash
   curl -s -OJ -H "Authorization: Bearer $CASCADE_TOKEN" "$CASCADE_URL/api/runs/$RUN/results"
   ```

   The file is named `<run id>-results.json`. [Reading results](reading-results.html) explains what's in it.

To stop, press Ctrl+C in the first terminal. The app stops rclone too. Your traces and runs stay in `./data` for next time.

## Next steps

- [Concepts](concepts.html) explains what a trace, a run and a sim are, and how their states move.
- [Deploy the app](deploy.html) runs the same app on Sail, where each sim is its own sandbox.
- [Bring your own simulator](simulator.html) replaces the replay with Accel-Sim or anything else.


# Use with AI agents

Page: agents.html

Paste one prompt into your agent and it can work your Cascade app for you. It reads your traces, launches runs on the configs that fit, watches them finish, and explains the results.

**Setup prompt**Tells your agent how to authenticate, which routes to call, and how to report results.

## What an agent can do

The agent works through the same [HTTP API](api.html) as the web UI. With the prompt, it can do these things.

- Check traces. It lists them, waits for an upload to become `ready`, and reports a trace that `failed`.
- Launch runs. It picks the GPU configs whose `arch` matches the trace's `trace_gpu_arch` and asks before it starts more than 4 sims.
- Watch runs. It polls the run until every sim has finished and tells you the progress as it goes.
- Pull results and comparisons. It reads per-kernel times from `/kernels`, real against simulated time from `/comparison`, and the full stats from `/results` when you ask for them.
- Explain the numbers. It reports simulated time per config, the error against real hardware when the trace has real timing, the worst kernels by readable name, and why any sim failed.

The agent can't trace a workload. Tracing runs on a real GPU under the Modal profiler, outside the API. [Trace a workload](trace-a-workload.html) covers that part. Once the trace is in your app, the agent takes it from there.

> A default deploy runs the stand-in simulator, which replays one recorded Accel-Sim run of OPT-125m on an H100 for every trace. The prompt tells the agent to say so when the numbers look like the stand-in's, so a report never passes them off as a simulation of your trace.

## Set it up

1. **Put the app's URL and token in the agent's environment.** Run this in the terminal you start the agent from, then start the agent. The prompt tells the agent to read the token from `$CASCADE_TOKEN` and never print it.

   ```bash
   export CASCADE_URL=https://<your-app>
   export CASCADE_TOKEN=<your token>
   ```

   The URL and token are the ones [Get started](../start.html) shows when your workspace is ready, or the `app:` and `token:` lines `cascade deploy` prints. Locally the URL is `http://127.0.0.1:8000`.

2. **Give the agent the prompt.** Copy the [setup prompt](agents.html#the-setup-prompt) and paste it into the chat, or save it where your agent reads instructions, as [Set up your tool](agents.html#set-up-your-tool) shows.

3. **Ask for what you want.** For example:

   ```text
   Which of my traces are ready?
   Simulate the newest trace on every config that matches it, one sim each, and compare them.
   Why did sims fail in my last run?
   ```

## The setup prompt

The build generates the API, state and type sections of this prompt from `landing/docs-src/endpoints.py` and `cascade/models.py`, the same sources as the [API reference](api.html). It always matches the API these docs describe. If you saved a copy, copy it again after you update Cascade.

agent-prompt.md:

````
# Cascade

You are working with a Cascade app. Cascade takes a trace of a GPU workload, captured once on a real GPU, and simulates it with Accel-Sim on other GPU configs, many sims at once. You drive the app over its HTTP API. You can check traces, launch runs, watch them, pull results and comparisons, and explain the numbers to the user.

You cannot capture a trace. Tracing runs the workload on a real GPU under Cascade's Modal profiler, outside this API. When the user wants a new trace, tell them to run the profiler (the "Trace a workload" page of the Cascade docs) and then wait for the trace to appear in the app.

Words used below:

- A **trace** is one profiler run. Its `id` is the profiler's run name.
- A **GPU config** is one of Accel-Sim's tested configs, such as `SM90_H100`.
- A **run** is one trace plus a list of entries. Each entry names a GPU config and a `count`.
- A **sim** is one simulator working through the trace on one config. Sims of one entry differ only in `replica`, numbered from 0.

## Connect

The app's URL and token come from two environment variables.

- `CASCADE_URL` is the app's base URL with no trailing slash, such as `http://127.0.0.1:8000`.
- `CASCADE_TOKEN` is the bearer token the app was started with.

Check that both are set without printing them:

```bash
test -n "$CASCADE_URL" && test -n "$CASCADE_TOKEN" && echo set
```

If either is unset, ask the user to export both in the terminal they start you from, and to start you again. If the user gives you the URL or the token in the conversation instead, write them once to a file only they can read, and load that file at the start of every command. Variables you export in one command may be gone in the next. Writing the file is the one command that holds the token's value. When you finish the task, delete it with `rm ~/.cascade.env` and tell the user, unless they ask you to keep it.

```bash
(umask 077 && printf 'export CASCADE_URL=%s CASCADE_TOKEN=%s\n' '<url>' '<token>' > ~/.cascade.env)
. ~/.cascade.env && curl -sS --fail-with-body -H "Authorization: Bearer $CASCADE_TOKEN" "$CASCADE_URL/api/traces"
```

Send the token on every request as an `Authorization: Bearer` header. Write every request in this form, with the variables unexpanded, so the token never appears in a command or its output:

```bash
curl -sS --fail-with-body -H "Authorization: Bearer $CASCADE_TOKEN" "$CASCADE_URL/api/traces"
```

For a request with a JSON body, add `-X POST -H "Content-Type: application/json" -d '<json>'`. When the status is 400 or more, `--fail-with-body` makes curl exit with code 22, print `The requested URL returned error: <status>` and still print the JSON body. Parse responses with `jq` or `python3`, and aggregate large responses in the command instead of printing them whole.

Check the connection before anything else.

1. `GET /api/health` needs no token. Expect `"ok": true` and `"auth": true`. `workspace` names the workspace, or is `null`. Anything else means `CASCADE_URL` is wrong or the app is down. `"auth": false` means the app runs without a token check, which is fine.
2. `GET /api/traces` with the token. A `401` means the token is wrong. Stop and ask the user to check `CASCADE_TOKEN`.

The connection is good when both calls succeed.

## Procedure

Follow these steps to simulate a trace and report on it. When the user asks about an existing run, start at step 5 with its `id`. `GET /api/runs` lists runs newest first.

1. **Find the trace.** Call `GET /api/traces`. Keep the traces whose `state` is `ready`. Match the user's words against `id`, `model`, `layers` and `source_gpu`. If more than one trace matches, list each with its `id`, `model`, `layers`, `trace_gpu_arch`, `kernel_count` and `created_at`, and ask which one. If the trace the user means is `uploading` or `processing`, poll `GET /api/traces/{trace_id}` every 10 seconds until it is `ready`. If it is `failed`, report its `error` and stop. This step is done when you hold one `ready` trace and its `trace_gpu_arch`.
2. **Pick the configs.** Call `GET /api/gpu-configs`. Keep only the configs whose `arch` equals the trace's `trace_gpu_arch`. A trace can only run on configs of its own architecture. Use the configs the user named. If they named none, use the trace's `suggested_sim_config` when it is set, and otherwise every matching config. If a config the user named has another `arch`, say so and offer the matching ones. This step is done when every config you will use has the trace's `arch`.
3. **Size the run.** Each entry's `count`, from 1 to 64, is how many replicas run that config. Use 1 per config unless the user asks for more. Suggest 3 when the user wants to know whether the numbers are stable. Add up the counts. If the total is more than 4 sims, show the user the entries and the total and wait for them to agree. On Sail, every sim is its own forked sandbox, and each one costs money. This step is done when you have the entries and, above 4 sims, the user's yes.
4. **Launch the run.** Call `POST /api/runs` with `{"trace_id": "<trace id>", "entries": [{"gpu_config": "<name>", "count": 1}]}`. A `201` returns the run. Save its `id`. Launch once. If the call fails with a network error or a `5xx`, call `GET /api/runs` and look for a new run on this trace before you try again.
5. **Watch the run.** Poll `GET /api/runs/{run_id}` every 10 seconds until `state` is anything but `active`. Between polls, tell the user the counts in `sim_counts` and each sim's `kernels_done` out of `kernels_total`. If `kernels_done` passes `kernels_total`, give both numbers as they are, cap any percentage at 100, and treat it as a stand-in sign (see Rules). A `lost` sim can come back, so keep waiting. A sim that stays `lost` for 10 times the heartbeat timeout, 600 seconds by default, becomes `failed`. A real Accel-Sim run can take hours. If the run is still `active` after about 5 minutes, give the user the run `id` and the progress, and offer to check again later. `GET /api/runs/{run_id}/events` streams the same changes as server-sent events, but the stream never closes, so polling is simpler. To read the stream anyway, add `-N --max-time 30` to curl. This step is done when the run's `state` is `succeeded`, `partial`, `failed` or `cancelled`.
6. **Fetch the results.** Call `GET /api/runs/{run_id}/kernels` for every sim's per-kernel times and `GET /api/runs/{run_id}/comparison` for real against simulated time per kernel. A `KernelStat` names only its `sim_id`. Join it to the run's `sims` by `id` to get its `gpu_config`, `replica` and `state`. For each sim that did not succeed, read `error`, `exit_code` and `log_tail` from the run. If `log_tail` does not explain the failure, read the end of `GET /api/sims/{sim_id}/log`, then search the whole log with `grep -inE 'error|abort|assert|fault|exception'`. If nothing explains it, report that the log does not show a cause, and give the `exit_code`, the signal it implies (134 is SIGABRT, 137 is SIGKILL, 139 is SIGSEGV), and the last kernel the sim reached. Call `GET /api/runs/{run_id}/results` only when the user asks for the full Accel-Sim stats. It carries about 30 KB per kernel. This step is done when you hold the kernels, the comparison and the reason for every sim that did not succeed.
7. **Report.** Write the report as the next section describes.

## Report

Lead with one line that gives the run `id`, the trace `id`, the run `state`, and the sim counts. Then cover these points, in this order.

1. **Simulated time per config.** For each sim, add up `sim_time_ns` over its kernels from `GET /api/runs/{run_id}/kernels`. For each config, give the median of those totals over the replicas that succeeded, and the lowest and highest total when there is more than one replica. A sim that did not succeed has only the kernels it finished, so leave it out of the totals. If no replica of a config succeeded, say so. Name the fastest config and give every other config as a multiple of it. When configs are within 1% of each other, say they tie instead of naming a winner. When the numbers come from the stand-in, say the configs cannot be compared from this run. Also give `latest.gpu_tot_sim_cycle` and `latest.gpu_tot_ipc` per config, from the run's sims. This computes the totals:

```bash
curl -sS --fail-with-body -H "Authorization: Bearer $CASCADE_TOKEN" "$CASCADE_URL/api/runs/<run id>" > /tmp/cascade-run.json
curl -sS --fail-with-body -H "Authorization: Bearer $CASCADE_TOKEN" "$CASCADE_URL/api/runs/<run id>/kernels" > /tmp/cascade-kernels.json
python3 -c '
import json, statistics
run = json.load(open("/tmp/cascade-run.json"))
sims = {s["id"]: s for s in run["sims"]}
totals = {}
for k in json.load(open("/tmp/cascade-kernels.json")):
    totals[k["sim_id"]] = totals.get(k["sim_id"], 0) + (k["sim_time_ns"] or 0)
configs = {}
for sim, total in totals.items():
    if sims[sim]["state"] == "succeeded":
        configs.setdefault(sims[sim]["gpu_config"], []).append(total)
for name, values in sorted(configs.items()):
    print(name, "median", statistics.median(values), "lowest", min(values), "highest", max(values), "replicas", len(values))
'
```
2. **Accuracy against real hardware.** When the comparison has `has_real_timing` true, compute the mean absolute percentage error (MAPE) for each config. Use the rows where both `real_ns` and `sim_ns` for that config are set. MAPE is the mean of `abs(sim_ns - real_ns) / real_ns * 100`. Say how many rows went into it. A row's `sim_ns` is the median over every replica that finished that kernel, including replicas that failed later. When `has_real_timing` is false, say that the trace has no real timing yet, so the report covers simulated time only.
3. **Worst kernels.** Name up to 5 kernels. With real timing, pick those with the largest absolute percentage error. Without it, pick those with the largest share of simulated time, and say how much their time differs between configs. Name each kernel in readable form. Demangle `kernel_name` with `c++filt -n -p <kernel_name>` when `c++filt` is installed, so `_Z11copy_kernelIdEvPKT_PS0_` becomes `copy_kernel<double>`. The `-n` matters on macOS, where `c++filt` without it returns the name unchanged. Without `c++filt`, give the identifier inside the mangled name, such as `copy_kernel`. Give the `kernel_index` too.
4. **Failed, lost and cancelled sims.** List every sim whose `state` is not `succeeded`, with its `id`, `gpu_config`, `replica`, `state`, `error`, `exit_code` and the lines of `log_tail` that show the cause.

When the user wants the simulator's own files, list them with `GET /api/runs/{run_id}/files`, download one with `GET /api/sims/{sim_id}/files/{name}`, and fetch a Perfetto trace of the run with `GET /api/runs/{run_id}/perfetto`.

Give times in the largest unit that keeps the number above 1, such as 5.2 µs or 3.1 ms, and call them simulated. Take every number from an API response. When a value is `null`, say that it is missing. If the user asks what a number means, explain it from the field descriptions under "Types" below.

## Rules

- Keep the token secret. Refer to it only as `$CASCADE_TOKEN`, and never print, echo or repeat its value.
- Launch a run only when the user asked for one. Above 4 sims in total, get the user's yes to the entries first.
- Cancel a run only when the user asks. `POST /api/runs/{run_id}/cancel` stops every sim that has not finished. It is safe to repeat.
- Use `GET /api/runs/{run_id}/kernels` for per-kernel numbers. Use `GET /api/runs/{run_id}/results` only for the full stats maps.
- Check whether the numbers come from Cascade's stand-in simulator, unless the user has said the app runs a real Accel-Sim image. A default deploy runs the stand-in, `cascade worker --replay`, which replays one recorded Accel-Sim run of `facebook/opt-125m` on SM90_H100 (20 vLLM kernels), whatever the trace. A local dev server may replay a BabelStream log instead, whose kernels are named init, copy, mul, add, triad and dot. The signs are the same kernels for every trace (`nvjet_sm90_hsh_64x8_64x16_4x1_v_bz_bias_TNT`, `vllm::reshape_and_cache_flash_kernel`, `cutlass::device_kernel` and `at::native::vectorized_elementwise_kernel` on a default deploy), OPT-125m kernels for a trace of another model, and `kernels_done` different from `kernels_total`. When you see these signs, say at the top of the report that the numbers come from the stand-in and do not describe the trace.
- Never call `POST /api/sims/{sim_id}/heartbeat`, `POST /api/sims/{sim_id}/logs`, `POST /api/sims/{sim_id}/step-timeline` or `PUT /api/sims/{sim_id}/files/{name}`. Those routes are for simulators, and a heartbeat changes a sim's state.
- Lists come back whole, with no pagination.

## Errors

Every error is JSON with a `detail` field.

- **401.** The token is missing or wrong. Stop and ask the user to check `CASCADE_TOKEN`. Do not try other tokens.
- **404.** When `detail` names a trace, run or sim, that id does not exist. List again with `GET /api/traces` or `GET /api/runs` to find the right one. When `detail` is just `Not Found`, the path itself is wrong. Check it against the routes below.
- **409.** `POST /api/runs` named a trace that is not `ready`. `detail` gives its state. If the state is `uploading` or `processing`, wait until the trace is `ready` and launch again. If it is `failed`, report the trace's `error`.
- **422.** When `detail` is a sentence, a `gpu_config` is unknown or has an `arch` other than the trace's `trace_gpu_arch`. Call `GET /api/gpu-configs` and use matching configs. When `detail` is a list, the body is invalid: `entries` is empty, or a `count` is outside 1 to 64. Each item's `loc` and `msg` say what to fix.
- **5xx or a network error.** Retry a `GET` up to 3 times, a few seconds apart. Before you retry `POST /api/runs`, check `GET /api/runs` for the run, so you never launch it twice.

## API

Every route takes the bearer token except `GET /api/health`. Bodies are JSON, and timestamps are ISO 8601 in UTC.

| method | path | what it does |
|---|---|---|
| GET | `/api/health` | Liveness check. Needs no token. |
| GET | `/api/traces` | Every trace, newest first. |
| GET | `/api/traces/{trace_id}` | One trace with its metadata and kernels. |
| GET | `/api/gpu-configs` | The GPU configs a run can use. |
| POST | `/api/runs` | Launch sims for a ready trace. |
| GET | `/api/runs` | Every run, newest first. |
| GET | `/api/runs/{run_id}` | One run with its trace and every sim. |
| POST | `/api/runs/{run_id}/cancel` | Cancel every unfinished sim. |
| GET | `/api/runs/{run_id}/comparison` | Per-kernel real time against simulated time per config. |
| GET | `/api/runs/{run_id}/kernels` | Every sim's per-kernel stats, without the full stats maps. |
| GET | `/api/runs/{run_id}/step-timelines` | Each sim's predicted step timeline and its comparison with the real run. |
| GET | `/api/runs/{run_id}/results` | One JSON file with the run, trace, every sim's stats and the comparison. |
| GET | `/api/sims/{sim_id}/results` | One sim's config, per-kernel stats and totals. |
| GET | `/api/sims/{sim_id}/log` | The sim's full raw output. |
| GET | `/api/runs/{run_id}/files` | Every output file the run's sims uploaded. |
| GET | `/api/runs/{run_id}/perfetto` | The run's simulated kernels as a Chrome trace JSON for ui.perfetto.dev. |
| GET | `/api/sims/{sim_id}/files` | The output files one sim uploaded. |
| GET | `/api/sims/{sim_id}/files/{name}` | One output file, as the sim uploaded it. |
| GET | `/api/runs/{run_id}/events` | Server-sent events for one run. |

### GET /api/health

Check health. Returns `{"ok": true, "auth": true, "workspace": ...}` while the app process is serving requests. `auth` is `false` only when the app runs with `CASCADE_NO_AUTH=1`, a local-dev mode where no token is needed and the web UI skips its token screen. `workspace` is `CASCADE_WORKSPACE`, the name the web UI shows in its rail, or `null`. Health does not check the database, the S3 endpoint or the launcher.

- Auth: No token needed.
- Returns: `200` with `{"ok": true, "auth": true, "workspace": "<name or null>"}`.

### GET /api/traces

List traces. Lists every trace the app has registered, in every state, newest first by `created_at`. A trace appears as soon as its first file lands in the bucket, so this list also shows uploads in progress.

- Returns: `200` with a list of `TraceSummary`.

### GET /api/traces/{trace_id}

Get a trace. Returns one trace with the parsed `metadata.json` and every kernel launch read from the trace's `stats_ctx_*` files. `kernels` stays empty until the trace is `ready`.

- Path parameter `trace_id` (str): The trace's `id`, which is the profiler's run name.
- Returns: `200` with a `TraceDetail`.
- `404`: No trace has this id.

### GET /api/gpu-configs

List GPU configs. Lists the GPU configs the app found at startup under `$CASCADE_ACCELSIM_DIR/gpu-simulator/gpgpu-sim/configs/tested-cfgs/*/gpgpusim.config`. The app derives `arch` from the directory name (`SM90_H100` is `sm_90`, `SM7_QV100` is `sm_70`) and reads `core_clock_mhz` from the first value of `-gpgpu_clock_domains`. A directory whose name or clock it can't read is skipped.

- Returns: `200` with a list of `GpuConfig`, sorted by name.

### POST /api/runs

Create a run. Creates a run and starts launching its sims in the background. Each entry fans out to `count` sims, numbered by `replica` from 0. The response comes back before any sim has launched, so every sim is `pending` or `launching`. Follow progress with `GET /api/runs/{run_id}/events` or by polling `GET /api/runs/{run_id}`.

- Body: `RunCreate`.
- Returns: `201` with a `RunDetail`.
- `404`: `trace_id` names no trace.
- `409`: The trace is not `ready`, or is a sample, which has no instruction traces.
- `422`: An entry names an unknown `gpu_config`, or a config whose `arch` differs from the trace's `trace_gpu_arch`.
- `422`: The body fails validation: `entries` is empty, or a `count` is outside 1 to 64.

### GET /api/runs

List runs. Lists every run, newest first, with per-state sim counts. Use `GET /api/runs/{run_id}` for the sims themselves.

- Returns: `200` with a list of `RunSummary`.

### GET /api/runs/{run_id}

Get a run. Returns the run, its trace, and every sim with its state, progress, latest metrics and last 20 log lines.

- Path parameter `run_id` (str): The run's `id`, such as `run-9f56e171af08`.
- Returns: `200` with a `RunDetail`.
- `404`: No run has this id.

### POST /api/runs/{run_id}/cancel

Cancel a run. Moves every sim that has not finished to `cancelled` and asks the launcher to stop it. Sims that already finished keep their state. The next heartbeat from a cancelled sim gets `cancel: true`. Calling it again changes nothing, so it is safe to retry. On a sample run it changes nothing either, since its sim already `succeeded`.

- Path parameter `run_id` (str): The run's `id`, such as `run-9f56e171af08`.
- Returns: `200` with the `RunDetail` after cancelling.
- `404`: No run has this id.

### GET /api/runs/{run_id}/comparison

Compare real and simulated time. Lines up kernels by order. Row `i` holds the `i`th kernel the sims finished and the `i`th row of the trace's real timing, sorted by `kernel_id`. For each config, `sim_ns` is the median `sim_time_ns` over that config's replicas, or `null` if no replica has finished that kernel. `real_ns` is `null` until the trace has real timing, which the profiler does not capture yet.

- Path parameter `run_id` (str): The run's `id`, such as `run-9f56e171af08`.
- Returns: `200` with a `Comparison`.
- `404`: No run has this id.

### GET /api/runs/{run_id}/kernels

List a run's kernel timings. Returns a `KernelStat` for every kernel every sim in the run has finished, ordered by config, replica, then kernel. Each `stats` map is empty to keep the response small; the full maps are in the results downloads. Laying each sim's kernels end to end by `sim_time_ns` gives its timeline, which is what the app's timeline and Compare views draw.

- Path parameter `run_id` (str): The run's `id`, such as `run-9f56e171af08`.
- Returns: `200` with a list of `KernelStat`.
- `404`: No run has this id.

### GET /api/runs/{run_id}/step-timelines

List a run's step timing. Returns a `SimStepTimeline` for every sim in the run that has sent step timing, ordered by config, then replica. A sim sends one after it succeeds on a trace whose `has_step_timeline` is true, so the list is empty until then, and stays empty for other traces. `simulated` is a full `simulated_timeline.json`, with every kernel of every step, so for a real model it can be several megabytes per sim. Per-kind step means are in `simulated.summary.per_kind`, coverage is in `simulated.summary.simulation.coverage`, and per-step and per-request errors against the real run are in `compare`.

- Path parameter `run_id` (str): The run's `id`, such as `run-9f56e171af08`.
- Returns: `200` with a list of `SimStepTimeline`.
- `404`: No run has this id.

### GET /api/runs/{run_id}/results

Download run results. Returns everything about a run as one JSON download, with `Content-Disposition: attachment; filename="<run_id>-results.json"`. Each sim's `kernels` carry the full `stats` map, every `key = value` line Accel-Sim printed for that kernel, which is about 30 KB per kernel.

- Path parameter `run_id` (str): The run's `id`, such as `run-9f56e171af08`.
- Returns: `200` with a `RunResults`.
- `404`: No run has this id.

### GET /api/sims/{sim_id}/results

Download sim results. Returns one sim's results as a JSON download named `<sim_id>-results.json`. `totals` is the sim's latest `SimMetrics`.

- Path parameter `sim_id` (str): The sim's `id`, such as `sim-6b4c91e71437`. Find it in `RunDetail.sims`.
- Returns: `200` with a `SimResults`.
- `404`: No sim has this id.

### GET /api/sims/{sim_id}/log

Download a sim's log. Streams every line the simulator sent, in `seq` order, as `text/plain` named `<sim_id>.log`.

- Path parameter `sim_id` (str): The sim's `id`, such as `sim-6b4c91e71437`. Find it in `RunDetail.sims`.
- Returns: `200` with the log as plain text.
- `404`: No sim has this id.

### GET /api/runs/{run_id}/files

List a run's output files. Lists the files every sim in the run uploaded, ordered by config, replica, then name. Each `RunFile` names its sim, so one call covers the run. `cascade worker` uploads a sim's files just before its final heartbeat, so the list grows as sims finish. Fetch one with `GET /api/sims/{sim_id}/files/{name}`.

- Path parameter `run_id` (str): The run's `id`, such as `run-9f56e171af08`.
- Returns: `200` with a list of `RunFile`.
- `404`: No run has this id.

### GET /api/runs/{run_id}/perfetto

Download a Perfetto trace. Builds a Chrome trace JSON from the run's parsed kernel stats and returns it with `Content-Disposition: attachment; filename="<run_id>-perfetto.json"`. Each config is a process, each replica is a thread in it, and each finished kernel is one slice. The app builds it from the stats it parsed, so every run has one, whether or not the simulator wrote its own. Reading results describes the layout.

- Path parameter `run_id` (str): The run's `id`, such as `run-9f56e171af08`.
- Returns: `200` with a Chrome trace JSON object: `traceEvents`, `displayTimeUnit` and `otherData`.
- `404`: No run has this id.

### GET /api/sims/{sim_id}/files

List a sim's output files. Lists one sim's uploaded files, sorted by name. The list is empty until the simulator uploads something.

- Path parameter `sim_id` (str): The sim's `id`, such as `sim-6b4c91e71437`. Find it in `RunDetail.sims`.
- Returns: `200` with a list of `SimFile`.
- `404`: No sim has this id.

### GET /api/sims/{sim_id}/files/{name}

Download an output file. Returns the file as a download named `name`, with the file's `content_type`. A `.gz` file comes back as `application/gzip` with no `Content-Encoding`, so it stays compressed.

- Path parameter `sim_id` (str): The sim's `id`, such as `sim-6b4c91e71437`. Find it in `RunDetail.sims`.
- Path parameter `name` (str): The file's name, such as `kernel-stats.csv`. Letters, digits, `.`, `_` and `-`, 1 to 128 characters, and not `.` or `..`.
- Returns: `200` with the file's bytes.
- `404`: No sim has this id, or the sim has no file with this name.

### GET /api/runs/{run_id}/events

Stream run events. Opens a `text/event-stream`. The first event is always a `snapshot`. After that the stream sends `sim`, `log` and `kernel` events as they happen, and a `: keepalive` comment after 15 seconds without one. The stream does not close when the run finishes.

- Auth: Bearer token, or the token as the `token` query parameter.
- Path parameter `run_id` (str): The run's `id`, such as `run-9f56e171af08`.
- Query parameter `token` (str, optional): Your `CASCADE_TOKEN`. Use it when you can't set headers, as with `EventSource`. If present, it is checked instead of the header.
- Returns: `200` with a `text/event-stream` of `RunEvent`, one JSON object per `data:` line.
- `404`: No run has this id.

## States

### Trace states

A trace's `state`. Runs can only launch on a `ready` trace.

- `uploading`: Files are arriving. `kernel_count` and `bytes` grow on each scan, every 2 seconds. Next: `processing` or `failed`.
- `processing`: The upload is complete. The app is running the trace post-processor, unless the upload already has `traces/kernelslist.g`. Next: `ready` or `failed`.
- `ready`: Post-processing produced `traces/kernelslist.g`. Runs can launch. Terminal.
- `failed`: `metadata.json` has `status: "failed"`, or post-processing failed. `error` says which. Terminal.

### Sim states

A sim's `state`. The app ignores any move not listed here.

- `pending`: Created with the run. Waiting for the launcher. Next: `launching`, `failed` or `cancelled`.
- `launching`: The launcher is starting the sim's sandbox or process. Next: `running`, `lost`, `succeeded`, `failed` or `cancelled`.
- `running`: The sim sent a `running` heartbeat. Next: `lost`, `succeeded`, `failed` or `cancelled`.
- `lost`: No heartbeat for `CASCADE_HEARTBEAT_TIMEOUT_S`. A later heartbeat brings it back. Next: `running`, `succeeded`, `failed` or `cancelled`.
- `succeeded`: The sim sent a `succeeded` heartbeat. Terminal.
- `failed`: The sim sent a `failed` heartbeat, the launcher failed, or it stayed `lost` for 10 times the timeout. Terminal.
- `cancelled`: The run was cancelled before this sim finished. Terminal.

### Run states

`RunState` is not stored. `run_state()` in `cascade/models.py` derives it from the run's sim states each time a run is read.

- `active`: At least one sim has not finished.
- `succeeded`: Every sim succeeded.
- `partial`: Every sim finished, at least one succeeded, and at least one did not.
- `failed`: Every sim failed.
- `cancelled`: Every sim finished, at least one was cancelled, and none succeeded.

## Types

These are the request and response bodies, from `cascade/models.py`. A field written `X | None` can be `null`. `TraceState`, `SimState` and `RunState` are under States.

### GpuConfig

- `name` (`str`): Accel-Sim tested-cfgs directory name, e.g. "SM90_H100".
- `arch` (`str`): For example, "sm_90".
- `core_clock_mhz` (`float`): First value of -gpgpu_clock_domains in gpgpusim.config.

### KernelLaunch

One kernel launch from the trace's stats_ctx_* file (original hardware).

- `kernel_id` (`int`): Matches kernel-<N> in the trace filename.
- `name` (`str`): Mangled kernel name.
- `grid` (`tuple[int, int, int]`): Grid size, x, y and z.
- `block` (`tuple[int, int, int]`): Block size, x, y and z.
- `instructions` (`int | None`): Instructions the tracer counted. `null` if the kernel's trace was cut short.

### TraceSummary

- `id` (`str`): The profiler's run_name.
- `state` (`TraceState`): Where the trace is in its lifecycle.
- `error` (`str | None`): Why the trace failed. `null` unless `state` is `failed`.
- `created_at` (`datetime`): When the first file of the trace landed.
- `updated_at` (`datetime`): When the app last changed this trace.
- `model` (`str | None`): `metadata.json` `model`, such as `facebook/opt-125m`.
- `layers` (`list[str]`): `metadata.json` `layers`. Empty if missing or not a list.
- `source_gpu` (`str | None`): metadata.json "gpu".
- `trace_gpu_arch` (`str`): metadata.json "trace_gpu_arch", default "sm_90".
- `suggested_sim_config` (`str | None`): `metadata.json` `suggested_sim_config`. The UI preselects it on the New run screen.
- `real_gpu_config` (`str | None`): metadata.json: config of the GPU it was captured on.
- `sample_sim_configs` (`list[str]`): metadata.json: preselected.
- `sample_set` (`str | None`): metadata.json: traces compared side by side.
- `kernel_count` (`int`): Traced kernels seen so far.
- `bytes` (`int`): Bytes on disk under the trace dir.
- `has_real_timing` (`bool`): True when the trace has real per-kernel timings (`real_timing.json`). The profiler does not write them yet.
- `has_step_timeline` (`bool`): corrected_timeline.json + kernel_mapping.json are in the trace dir.
- `sample` (`bool`): True for the sample trace that `cascade serve` seeds from `samples/` (see Introduction), unless `CASCADE_SAMPLES` is false. It is `ready` but has no instruction traces, so it can't start a run.

### TraceDetail

Every field of `TraceSummary`, plus these.

- `metadata` (`dict | None`): The trace's `metadata.json`, parsed. `null` until it arrives.
- `kernels` (`list[KernelLaunch]`): Kernel launches from the trace's `stats_ctx_*` files, by `kernel_id`. Filled in when the trace becomes `ready`.

### RunEntry

- `gpu_config` (`str`): A `GpuConfig.name` whose `arch` matches the trace's `trace_gpu_arch`.
- `count` (`int`): How many sims run this config. Allowed: at least 1, at most 64.

### RunCreate

- `trace_id` (`str`): A trace in state `ready`.
- `entries` (`list[RunEntry]`): One entry per config. Each fans out to `count` sims. Allowed: at least 1 item.

### RunSummary

- `id` (`str`): Generated, `run-` and 12 hex digits.
- `trace_id` (`str`): The trace the run simulates.
- `created_at` (`datetime`): When the run was created.
- `state` (`RunState`): Derived from the sims. See `RunState`.
- `entries` (`list[RunEntry]`): The entries the run was created with.
- `sim_counts` (`dict[SimState, int]`): How many sims are in each state. States with no sims are left out.
- `sample` (`bool`): True for the sample run seeded with the sample trace. Its results were recorded when the run was made, not in this workspace, and cancelling it changes nothing.
- `replay` (`bool`): Replays a sample's recorded Accel-Sim results.

### RunDetail

Every field of `RunSummary`, plus these.

- `trace` (`TraceSummary`): The run's trace.
- `sims` (`list[SimSummary]`): Every sim, sorted by `gpu_config` and then `replica`.

### SimSummary

- `id` (`str`): Generated, `sim-` and 12 hex digits. The simulator receives it as `CASCADE_SIM_ID`.
- `run_id` (`str`): The run this sim belongs to.
- `gpu_config` (`str`): The config this sim simulates. The simulator receives it as `CASCADE_GPU_CONFIG`.
- `replica` (`int`): 0-based index within its {gpu_config, count} entry.
- `state` (`SimState`): Where the sim is in its lifecycle.
- `error` (`str | None`): Why the sim failed or went `lost`, from a heartbeat or from the app. A `running` heartbeat clears it.
- `launcher_ref` (`str | None`): Sailbox id or local pid.
- `created_at` (`datetime`): When the run was created.
- `started_at` (`datetime | None`): The first `running` heartbeat.
- `finished_at` (`datetime | None`): When the sim reached a terminal state.
- `last_heartbeat_at` (`datetime | None`): The last heartbeat. `null` before the first one.
- `exit_code` (`int | None`): From the final heartbeat, when the simulator sent one.
- `kernels_done` (`int`): Kernel stats blocks parsed from the log so far.
- `kernels_total` (`int`): The trace's kernel count when the run was created.
- `current_kernel` (`str | None`): The kernel named by the last `launching kernel name:` line in the log.
- `latest` (`SimMetrics`): The most recent totals parsed from the log.
- `log_tail` (`list[str]`): Last 20 lines.

### SimMetrics

- `gpu_tot_sim_cycle` (`float | None`): Simulated cycles so far, over every kernel.
- `gpu_tot_sim_insn` (`float | None`): Simulated instructions so far, over every kernel.
- `gpu_tot_ipc` (`float | None`): Instructions per cycle so far, over every kernel.
- `sim_rate_ips` (`float | None`): gpgpu_simulation_rate, instructions/sec.

### KernelStat

One per-kernel stats block parsed from Accel-Sim output.

- `sim_id` (`str`): The sim that ran this kernel.
- `kernel_index` (`int`): 0-based order of completion within the sim.
- `kernel_name` (`str`): The block's `kernel_name`.
- `kernel_launch_uid` (`int | None`): The block's `kernel_launch_uid`.
- `gpu_sim_cycle` (`int | None`): Cycles this kernel took on the simulated GPU.
- `gpu_sim_insn` (`int | None`): Instructions this kernel ran.
- `gpu_ipc` (`float | None`): This kernel's instructions per cycle.
- `sim_time_ns` (`float | None`): gpu_sim_cycle / core_clock_mhz * 1000.
- `stats` (`dict[str, float | str]`): Every "key = value" line in the block.

### ComparisonRow

- `kernel_index` (`int`): Row position. Kernels are matched by order.
- `kernel_name` (`str`): From real timing if present, otherwise from the first sim that finished this kernel.
- `real_ns` (`float | None`): `duration_ns` from real timing. `null` without real timing.
- `sim_ns` (`dict[str, float | None]`): gpu_config -> median sim_time_ns over replicas.

### Comparison

- `has_real_timing` (`bool`): False until the trace has real timing.
- `configs` (`list[str]`): The run's configs, in entry order, without repeats.
- `rows` (`list[ComparisonRow]`): One row per kernel position.

### SimResults

- `sim` (`SimSummary`): The sim as `SimSummary`.
- `trace_id` (`str`): The trace the sim ran.
- `gpu_config` (`GpuConfig`): The full config. `arch` is `unknown` and the clock is 0 if the config disappeared since the run started.
- `kernels` (`list[KernelStat]`): Every kernel the sim finished, with the full `stats` map.
- `totals` (`SimMetrics`): Same as `sim.latest`.

### RunResults

- `run` (`RunSummary`): The run without its trace and sims, which are below.
- `trace` (`TraceDetail`): The full trace.
- `sims` (`list[SimResults]`): One entry per sim.
- `comparison` (`Comparison`): Same as `GET /api/runs/{run_id}/comparison`.

### SimStepTimeline

- `sim_id` (`str`): The sim that sent it.
- `gpu_config` (`str`): The sim's config.
- `replica` (`int`): The sim's replica.
- `simulated` (`dict`): As sent in `StepTimelineUpload.simulated`.
- `compare` (`dict | None`): As sent in `StepTimelineUpload.compare`.

### SimFile

An output file a sim uploaded with PUT /api/sims/{id}/files/{name}.

- `name` (`str`): Matches FILE_NAME_PATTERN; Accel-Sim's name, plus .gz when the worker gzipped it.
- `size` (`int`): Bytes as stored.
- `content_type` (`str`): From the name: `text/csv` for `.csv`, `text/plain` for `.txt` and `.log`, `application/json` for `.json`, `application/gzip` for `.gz`, and otherwise a guess, falling back to `application/octet-stream`.
- `created_at` (`datetime`): When this version was stored; a re-put replaces the file.

### RunFile

Every field of `SimFile`, plus these.

- `sim_id` (`str`): The sim that uploaded the file.
- `gpu_config` (`str`): The sim's config.
- `replica` (`int`): The sim's replica.

### RunEvent

`RunEvent` is `SnapshotEvent | SimEvent | LogEvent | KernelEvent`. One event on the run event stream. The `type` field says which model it is.

### SnapshotEvent

- `type` (`Literal["snapshot"]`): Always `snapshot`.
- `run` (`RunDetail`): The run as it is at connect time.
- `kernels` (`list[KernelStat]`): Every sim's stats with `stats` emptied; full stats are ~30 KB each.

### SimEvent

- `type` (`Literal["sim"]`): Always `sim`.
- `sim` (`SimSummary`): The sim after a change: state, heartbeat, progress or metrics.

### LogEvent

- `type` (`Literal["log"]`): Always `log`.
- `sim_id` (`str`): The sim the lines came from.
- `lines` (`list[str]`): The new chunk, as the simulator sent it.

### KernelEvent

- `type` (`Literal["kernel"]`): Always `kernel`.
- `stat` (`KernelStat`): A kernel that just finished, with the full `stats` map.
````

The same prompt is at [agent-prompt.md](agent-prompt.md) as plain Markdown, and at [cascade-skill/SKILL.md](cascade-skill/SKILL.md) as a skill with `name` and `description` frontmatter.

## Set up your tool

Each tool below can take the prompt pasted into a chat. To keep it across sessions, save it as a file the tool reads. The commands copy the files from a Cascade checkout. Run them from your project's root, and set `CASCADE` to the checkout's path.

### Claude Code

Save the prompt as a skill. Claude Code loads it when a request matches the skill's description. Use `.claude/skills/` in your project, or `~/.claude/skills/` to have it in every project.

```bash
mkdir -p .claude/skills/cascade
cp "$CASCADE/landing/docs/cascade-skill/SKILL.md" .claude/skills/cascade/SKILL.md
```

To load it in every session instead, append the prompt to your project's `CLAUDE.md`.

```bash
cat "$CASCADE/landing/docs/agent-prompt.md" >> CLAUDE.md
```

### Codex

Codex reads `AGENTS.md` in your project's root, and `~/.codex/AGENTS.md` for every project. Append the prompt to either one.

```bash
cat "$CASCADE/landing/docs/agent-prompt.md" >> AGENTS.md
```

### Cursor

Save the prompt as a project rule in `.cursor/rules/`. This rule has a description, so the agent can pull it in when a request is about Cascade. Cursor's docs list the other ways a rule can apply.

```bash
mkdir -p .cursor/rules
cat > .cursor/rules/cascade.mdc <<'EOF'
---
description: Drive the Cascade app API for traces, runs and simulation results
alwaysApply: false
---

EOF
cat "$CASCADE/landing/docs/agent-prompt.md" >> .cursor/rules/cascade.mdc
```

### Other agents

Paste the prompt at the start of the conversation. If your agent loads skills from `SKILL.md` folders, use [cascade-skill/SKILL.md](cascade-skill/SKILL.md). Otherwise, save [agent-prompt.md](agent-prompt.md) wherever your agent reads project instructions. Check your agent's docs for where that is.

## Docs for agents

These files give an agent the docs without the site around them. The build writes them next to the pages.

- [llms.txt](llms.txt) lists every docs page with a one-line description, in the llms.txt convention.
- [llms-full.txt](llms-full.txt) has every page as Markdown in one file, in the order of the sidebar.
- [agent-prompt.md](agent-prompt.md) is the setup prompt above.
- [cascade-skill/SKILL.md](cascade-skill/SKILL.md) is the setup prompt as a skill.


# Concepts

Page: concepts.html

Cascade has four nouns. A trace is what you captured. A GPU config is what you simulate it on. A run asks for some number of sims per config. A sim is one simulator process working through the trace.

## Trace

A trace is one profiler run: Accel-Sim SASS traces for every kernel the traced layers launched, plus the profiler's `metadata.json`. Its id is the profiler's run name, such as `20261107-120000__facebook_opt-125m__model.decoder.layers.10.self_attn`.

The app keeps each trace as plain files at `$CASCADE_DATA_DIR/s3/cascade/accelsim-traces/<trace id>/`, because `rclone serve s3` writes uploaded objects to disk as files. Sims read the trace from the same path on their own copy of that disk.

A trace records the architecture it was captured on in `trace_gpu_arch`. An H100 trace contains Hopper-only instructions, so it can only run on `sm_90` configs. If `metadata.json` has no `trace_gpu_arch`, the app assumes `sm_90`, which is what the H100 profiler produces.

### Trace states

Diagram: Trace states, from cascade/ingest.py

- `uploading` starts when the first file of the trace appears in the bucket.
- `processing` starts when the upload is complete. [Traces](traces.html#when-an-upload-counts-as-complete) explains how the app decides that.
- `ready` means `traces/kernelslist.g` exists and runs can launch.
- `failed` means `metadata.json` says `status: "failed"`, or post-processing failed.

## GPU config

A GPU config is one of Accel-Sim's tested configs, a directory under `gpu-simulator/gpgpu-sim/configs/tested-cfgs/` such as `SM90_H100`. The app reads two things from each one: the architecture, from the directory name, and the core clock, from the first value of `-gpgpu_clock_domains` in `gpgpusim.config`. The clock turns simulated cycles into simulated time.

## Run

A run is one trace plus a list of entries. Each entry names a config and a count from 1 to 64, so `[{"gpu_config": "SM90_H100", "count": 4}, {"gpu_config": "SM90_H200", "count": 2}]` asks for six sims. The app doesn't store a run's state. It derives the run's state from its sims every time you read it.

| sim states | run state |
|---|---|
| `running`, `succeeded` | `active` |
| `succeeded`, `succeeded` | `succeeded` |
| `succeeded`, `failed` | `partial` |
| `failed`, `failed` | `failed` |
| `cancelled`, `failed` | `cancelled` |
| `cancelled`, `succeeded` | `partial` |
| `cancelled`, `cancelled` | `cancelled` |

## Sim

A sim is one simulator working on one config. Sims of the same entry differ only in `replica`, numbered from 0. Running several replicas of a config tells you whether its numbers are stable.

The app and the sim share ownership of the sim's state. The app moves sims to `launching`, `lost` and `cancelled`, and to `failed` when a launch fails or a sim stays lost too long. Heartbeats from the sim move it to `running`, `succeeded` or `failed`.

### Sim states

Diagram: Sim states, drawn from SIM_TRANSITIONS in cascade/models.py

The diagram is drawn from `SIM_TRANSITIONS` in `cascade/models.py`, which is the only list of allowed moves. Any move outside it is ignored. `failed` and `cancelled` can be reached from every state that hasn't finished, so they are listed under the diagram instead of drawn.

- A sim with no heartbeat for `CASCADE_HEARTBEAT_TIMEOUT_S` seconds, 60 by default, becomes `lost`. The app checks every 5 seconds.
- A `lost` sim that sends another heartbeat goes back to `running`, or straight to `succeeded` or `failed` if that's what it reports.
- A sim that stays `lost` for 10 times the timeout, 600 seconds by default, becomes `failed` with the error `no heartbeat for 600s`, and the app stops its sandbox. Every run finishes eventually.

## How a run launches

When you create a run, the app writes the run and all of its sims as `pending`, returns, and launches in the background. Up to 8 launches run at once.

On Sail (`cascade serve --sims sail`, which the deploy script runs), launching works like this.

1. The app checkpoints its own sandbox once per run, as `cascade-<run id>`. A checkpoint saves the disk, so it includes the ready trace. If a checkpoint with that name already exists, the app reuses it, so a restart mid-launch doesn't take a second one.
2. For each sim, the app starts a new sandbox named `cascade-<sim id>` from that checkpoint with `restore_memory=False`. The fork boots fresh from the saved disk instead of cloning the app's running process.
3. In the fork, the app runs `CASCADE_SIM_CMD` in the background with the sim's environment, and passes the sim id as the call's idempotency key. The command's output goes to `$CASCADE_DATA_DIR/work/sims/<sim id>.out` on the fork's disk.
4. The sim's `launcher_ref` is the fork's sandbox id. When the sim finishes or is cancelled, the app terminates that sandbox.

Every fork of a run starts from the same checkpoint, so every sim reads an identical trace, and the trace is never copied over the network.

Locally (`cascade serve --sims local`, the default), the app runs `CASCADE_SIM_CMD` as a subprocess in its own process group, with the same environment, and `launcher_ref` is the process id. To stop it, the app sends `SIGTERM` to the group and `SIGKILL` after 5 seconds.

> The Sail launcher was written against the Sail SDK 0.17.2 source and is covered by tests with a fake SDK. It hasn't run against the live Sail service yet.

## How sims report back

Sims push and the app listens. A sim needs no public address of its own. It reaches the app at `CASCADE_MAIN_URL` and sends two kinds of request.

- A heartbeat every few seconds, with its state. The reply tells it whether it has been cancelled.
- Its raw output in numbered chunks. The app parses Accel-Sim's per-kernel stats out of the lines itself, so a simulator never has to.

The app stores every chunk, so you can download a sim's full log at any point. It fans every change out to browsers over server-sent events. [Bring your own simulator](simulator.html) has the full contract.


# Deploy the app

Page: deploy.html

A workspace is your own Cascade app in your Sail account: the Sail app `cascade-<name>` with one sandbox, `cascade-main`, that runs the app and forks a sandbox per sim. Create it from the browser, or with `cascade deploy` from a checkout.

## Get started in the browser

Open [Get started](../start.html), pick a workspace name, and paste a Sail API key from the [Sail dashboard](https://app.sailresearch.com). The page shows each step as it happens and ends with an Open my workspace link that logs you in, the token, the S3 endpoint and ready-made upload commands. Nothing to clone.

- The key creates the workspace in your Sail account. Cascade doesn't store it. It is copied into the workspace, because the app needs it to checkpoint itself and fork sims.
- The workspace boots the published image, `ghcr.io/atoniolo76/cascade:latest`, with the stand-in simulator. Set another public image under advanced.
- Manage, at the bottom of the page, shows a workspace's status, pauses and resumes it, and deletes it.

## Deploy from the command line

`cascade deploy` does the same from a checkout. It is safe to rerun: it finds the workspace's sandbox, resumes it if it's paused, and prints the same URLs and token again.

```bash
git clone https://github.com/atoniolo76/cascade.git
cd cascade
export SAIL_API_KEY=...
uv run --extra sail cascade deploy --workspace acme-lab
```

When the app answers, it prints what you need.

```text
cascade-acme-lab: starting Cascade
cascade-acme-lab: healthy
sailbox:  <sailbox id>
app:      https://<app url>
s3:       https://<s3 url>  (bucket cascade, prefix accelsim-traces, path-style, unsigned)
token:    <token>
open:     https://<app url>/#token=<token>
logs:     /data/cascade-serve.log inside the sailbox
```

| output | use it for |
|---|---|
| app (port 8000) | The web UI and the API. The `open:` link logs the browser in with the token. |
| s3 (port 9000) | Pass it to the profiler as `--s3-endpoint`. Traces upload here, unsigned and path-style. |
| token | The `CASCADE_TOKEN` for the UI, the API and your agent. Set `CASCADE_TOKEN` before the first deploy to choose it. |

## Deploy with a simulator image

To simulate your traces instead of replaying the stand-in, build the app on top of a simulator image and tell it where things are inside it. `--from-source` builds this checkout's `Dockerfile` on Sail, which takes a few minutes the first time.

```bash
uv run --extra sail cascade deploy --workspace acme-lab --from-source \
  --base-image <simulator-image> \
  --sim-cmd "/app/.venv/bin/cascade worker" \
  --accelsim-dir <path to accel-sim in that image>
```

`cascade worker` runs `<accelsim-dir>/gpu-simulator/bin/release/accel-sim.out` on the trace with the sim's config. If the image keeps the post-processor somewhere other than `<accelsim-dir>/util/tracer_nvbit/tracer_tool/traces-processing/post-traces-processing`, add `--postprocess-cmd`. [Bring your own simulator](simulator.html#what-the-image-needs) lists what the image needs. The simulator settings are written once, when the workspace is first created. To change them, delete the workspace and deploy again.

> Without `--base-image`, `--sim-cmd` and `--accelsim-dir`, every sim runs `cascade worker --replay`, which replays a real Accel-Sim run of `facebook/opt-125m` on SM90_H100 (20 vLLM kernels, from `results/20261108-021007-opt125m-h100-accelsim/`). It doesn't simulate your trace, and its numbers are the same for every trace. The image is plain `ubuntu:22.04`, so it has no post-processor. Upload traces that already contain `traces/kernelslist.g`, or they fail in `processing`.

## Flags

| flag | default | what it sets |
|---|---|---|
| `--workspace` | required | 3 to 32 lowercase letters, digits and hyphens. The Sail app is `cascade-<workspace>`, and the UI shows the name in its rail. |
| `--image` | `ghcr.io/atoniolo76/cascade:latest` | The public app image to boot. |
| `--from-source` | off | Build this checkout's `Dockerfile` on Sail instead of pulling `--image`. |
| `--base-image` | `ubuntu:22.04` | The Dockerfile's `BASE_IMAGE`. Only with `--from-source`. |
| `--sim-cmd` | `cascade worker --replay` on the OPT-125m H100 log | `CASCADE_SIM_CMD`, the command each sim runs. |
| `--accelsim-dir` | `/app/tests/fixtures/accelsim` | `CASCADE_ACCELSIM_DIR`. The default holds real Accel-Sim configs but no simulator. |
| `--postprocess-cmd` | derived from `--accelsim-dir` | `CASCADE_POSTPROCESS_CMD`. |
| `--disk-gib` | Sail's default | The app sandbox's disk limit. Sims fork this disk, so size it for your traces. |

## What's in the image

The `Dockerfile` builds the web UI with Node 22, then layers the app onto `BASE_IMAGE`: rclone 1.76.0, uv with Python 3.12, the app at `/app`, and the Accel-Sim configs and logs from `tests/fixtures`. CI publishes it to `ghcr.io/atoniolo76/cascade` on every push to `amani` that changes the app. The app sandbox and every sim fork run this one image. Sail doesn't run the image's `CMD`, so the deploy writes the app's settings, including your `SAIL_API_KEY`, to `/data/cascade.env` (mode 600) and starts `cascade serve --sims sail` with them.

The sandbox has auto-sleep turned off, so the app keeps running while sims report in. Pause it from Manage on [Get started](../start.html) when you aren't using it.

## Configuration

`cascade serve` reads these environment variables. The table is generated from `cascade/settings.py`.

| variable | default | what it sets |
|---|---|---|
| `CASCADE_TOKEN` | required | The shared bearer token. The UI and every sim send it. The app refuses to start without it unless `CASCADE_NO_AUTH` is set. |
| `CASCADE_NO_AUTH` | false | Local development only. Set to `1` to turn off the token check: the API accepts any request, `/api/health` reports `auth: false`, and the web UI skips its token screen. Never set it on a deployment someone else can reach. |
| `CASCADE_ACCELSIM_DIR` | `/opt/accel-sim-framework` | Accel-Sim checkout. The app reads GPU configs from `gpu-simulator/gpgpu-sim/configs/tested-cfgs/*/gpgpusim.config` under it. |
| `CASCADE_OPENAI_WAIT_S` | `3600` | Seconds `/v1` and `/api/scenarios` wait for base-simulation step timing. A timeout returns 504 with the run/sim IDs; it does not cancel the simulation. |
| `CASCADE_S3_BUCKET` | `cascade` | The bucket the profiler uploads to. It is a directory under `$CASCADE_DATA_DIR/s3/`. |
| `CASCADE_DATA_DIR` | `/data` | Where the app keeps `cascade.db`, the S3 bucket directory `s3/`, sims' output files in `files/`, and scratch files in `work/`. |
| `CASCADE_SIM_CMD` | none | The command each sim runs. On Sail it runs through a shell. Locally it is split into arguments and run without one. Without it, every sim fails at launch. |
| `CASCADE_POSTPROCESS_CMD` | `$CASCADE_ACCELSIM_DIR/util/tracer_nvbit/tracer_tool/traces-processing/post-traces-processing` | Accel-Sim's trace post-processor. The app runs it once per trace as `<cmd> <trace>/traces -j <cpus>`. |
| `CASCADE_HEARTBEAT_TIMEOUT_S` | `60` | Seconds without a heartbeat before a sim becomes `lost`. After 10 times this, a `lost` sim becomes `failed`. |
| `CASCADE_RUN_S3` | `true` | Whether `cascade serve` starts `rclone serve s3` next to the app. Set `false` to run the S3 endpoint yourself. |
| `CASCADE_WORKSPACE` | none | The workspace's name. `/api/health` returns it and the web UI shows it in the rail. `cascade deploy` and the Get started page set it. |
| `CASCADE_SAMPLES` | `true` | Whether `cascade serve` seeds the bundled sample trace and run (`samples/`) into the workspace on startup. Set `false` to start empty. A sample already seeded stays. |
| `CASCADE_PUBLIC_URL` | `http://127.0.0.1:8000` | The URL sims use to reach the app. Each sim gets it as `CASCADE_MAIN_URL`. |
| `CASCADE_S3_PREFIX` | `accelsim-traces` | The key prefix traces live under. Each trace is `<prefix>/<run_name>/`. |
| `CASCADE_S3_PUBLIC_URL` | none | The S3 endpoint the profiler uploads to. `/api/health` returns it as `s3_endpoint` and the UI fills it into the send-a-trace command. `cascade deploy` sets it; a local `cascade serve` derives it from `CASCADE_PUBLIC_URL`'s host and `--s3-port`. |

`cascade serve` takes `--host` (default `0.0.0.0`), `--port` (default 8000), `--s3-port` (default 9000) and `--sims`. `--sims local`, the default, runs each sim as a subprocess of the app. `--sims sail` forks a Sail sandbox per sim.

The S3 endpoint accepts anonymous reads and writes. Anyone with its URL can upload a trace. The app's API and UI need the token.


# Trace a workload

Page: trace-a-workload.html

Profile GPU timings and selected NVBit steps in your own Modal app; the package does not run CPU simulation.

```python
# app.py
import modal
from cascade_profiler import Workload, register

app = modal.App("my-profiler")
runs = modal.Volume.from_name("my-profiler-runs", create_if_missing=True)
hf = modal.Volume.from_name("my-hf-cache", create_if_missing=True)
prof = register(app, runs, gpu="H100!", hf_cache=hf)

@app.local_entrypoint()
def main():
    wl = Workload("facebook/opt-125m", ["Hello, my name is"], max_tokens=32, repeats=4)
    run = prof.profile.remote(wl.to_dict(), trace_steps=[0, 5])   # GPU: baseline -> nsys -> NVBit on steps 0,5
    print(run["run_name"], run["stage_wall_s"])
    print(f"modal volume get my-profiler-runs {run['run_name']} ./{run['run_name']}")
```

Optional S3 output: replace the registration above with:

```python
prof = register(app, runs, gpu="H100!", hf_cache=hf,
    s3={"endpoint": "https://s3.example.com", "bucket": "my-runs", "prefix": "profiles",
        "region": "auto", "addressing_style": "path"},
    secrets=[modal.Secret.from_name("my-s3-credentials")])
```

Put `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and optionally `AWS_SESSION_TOKEN` in that app-owned secret or the GPU runtime environment, never in destination options or files. The finished run uploads to `s3://my-runs/profiles/<run_name>/`, retaining the volume copy; `run.json` and `traced/metadata.json` record the destination and URI.

## Run and download

Use Python 3.12 locally: `register()` serializes Modal functions, so client and container Python minor versions must match.

```bash
pip install "cascade-profiler[modal] @ git+https://github.com/atoniolo76/cascade@profiler"
modal setup
modal run app.py
```

The example prints the `modal volume get` command to download the run. The first invocation builds the GPU image; later runs reuse it.

The GPU function runs baseline (no profiler), nsys capture, then NVBit tracing of the selected steps. Outputs include `corrected_timeline.json`, `trace_alignment.json` and `traced/traces/`; traced-run timings are not used for reconciliation. S3 uploads happen at the end, with `run.json` last; `keep_nsys_rep=True` is rejected with S3 output to avoid exporting an unscrubbed environment capture.

See the [OPT-125m sample analysis](sample-analysis-opt125m.html) for measured costs, replay errors and limitations; its CPU simulation uses our own tooling, not the profiling package.

## Legacy streaming into Cascade

For the app's streaming trace layout, use `accelsim_trace.py` from `validate-accelsim-modal` (commit `b348d7d`), not the package example above. Pass the endpoint from [Deploy the app](deploy.html): bucket `cascade`, path-style addressing.

```bash
modal run accelsim_trace.py --model facebook/opt-125m --list-layers
modal run accelsim_trace.py \
  --model facebook/opt-125m \
  --layers model.decoder.layers.10.self_attn \
  --prompt "Hello, my name is" \
  --max-tokens 4 \
  --s3-endpoint https://<your-s3-endpoint> \
  --s3-bucket cascade \
  --s3-addressing-style path
```

Choose layer paths from `--list-layers`. `--s3-prefix` defaults to `accelsim-traces` and must match the app's `CASCADE_S3_PREFIX`; `--s3-region` defaults to `auto`. Use `--run-name` for an explicit identifier. `--spinlock-handling` defaults to `mark_region`; alternatives are `fast_forward` and `none`.

This legacy CLI checks bucket access before tracing, streams closed kernel files and deletes the local copies. It also uploads logs, spinlock data and `metadata.json`, including on failure, under `s3://cascade/accelsim-traces/<run_name>/`. The app post-processes this layout; see [Traces](traces.html). Package uploads instead retain the full profiling run layout.

## Upload a legacy trace you already have

`scripts/upload_trace.py` sends `metadata.json` last with `status` and a `files` list. Use `--meta key=value` for trace metadata, such as the GPU architecture (the app defaults to `sm_90` for H100):

```bash
uv run python scripts/upload_trace.py ./my-run --run-name my-run \
  --endpoint https://<your-s3-endpoint> --meta trace_gpu_arch=sm_80
```

Add `--status failed` to mark the trace failed without processing it.


# OPT-125m sample: GPU profiling → instruction simulation → request replay

Page: sample-analysis-opt125m.html

## Full-request result

**254.45 ms real H100 → 262.14 ms simulated H100 (+3.02%)**, mean of steady repeats 1–3 for the 32-token request. Mean TTFT is 8.63 ms real versus 9.00 ms simulated (+4.3%); mean ITL is 7.93 ms versus 8.17 ms (+3.06%). The native **A100 reference is 222.08 ms real → 229.58 ms simulated SM80_A100 (+3.38%)**. The H200 counterfactual is 262.16 ms, with no measured H200 baseline or accuracy error. The tables below give the per-repeat request numbers and a compact kernel-error breakdown.

These headline values use the precision of the OPT-125m results draft; detailed tables below retain the committed artifacts' precision and distinguish the original verification run from the later Sail sweep. The ~3 µs (~6,000-cycle) per-kernel floor is the inferred main source of tiny-kernel over-prediction; calibrating it offers the largest GPU-accuracy payoff.

| Request metric (mean) | Real H100 | Simulated H100 | Signed error |
|---|---|---|---|
| **Full request (e2e)** | **254.45 ms** | **262.14 ms** | **+3.02%** |
| Time to first token (TTFT) | 8.63 ms | 9.00 ms | +4.3% |
| Inter-token latency (ITL) | 7.93 ms | 8.17 ms | +3.06% |

### Each steady repeat

| Repeat | Real e2e ms | H100 sim ms | Error | Real / sim TTFT ms | Real / sim ITL mean ms |
|---|---|---|---|---|---|
| 1 | 253.435 | 261.134 | +7.699 ms (+3.04%) | 8.636 / 8.987 | 7.897 / 8.141 |
| 2 | 254.613 | 262.302 | +7.689 ms (+3.02%) | 8.383 / 8.710 | 7.943 / 8.187 |
| 3 | 255.308 | 262.988 | +7.680 ms (+3.01%) | 8.865 / 9.302 | 7.950 / 8.191 |

### Where the error comes from

The profiler measures unprofiled request/step timing and Nsight kernel timing separately. NVBit traces four steps (prefill and decode at contexts ≈ 8, 16, 32); Accel-Sim simulates 31 distinct kernels, then propagates durations to 17,088 steady kernel launches. Replay swaps in simulated kernel durations while retaining measured host residuals, gaps and unmatched GPU activity.

| Kernel group | Real mean µs | H100 sim µs | Signed error |
|---|---|---|---|
| QKV GEMM | 4.69 | 4.94 | +5.4% |
| Attention | 6.33 | 6.86 | +8.5% |
| Out-proj GEMM | 4.08 | 4.46 | +9.3% |
| MLP up GEMM | 4.88 | 5.47 | +12.1% |
| MLP down GEMM | 5.01 | 5.19 | +3.6% |
| lm_head GEMM | 30.62 | 26.26 | −14.2% |
| KV-cache write | 2.05 | 4.08 | +98.7% |
| Split-K reduction | 1.94 | 3.67 | +89.2% |
| Layer norm | 3.40 | 5.10 | +49.9% |
| Small elementwise | 1.39 | 3.23 | +133.3% |

Kernel means compare the 712 aligned traced occurrences, not full-request-weighted decodes; group roles are inferred from kernel names. **Interpretation:** a ~3 µs (~6,000-cycle) per-kernel floor makes tiny kernels 1.5–2.3× too slow. With 178 kernels per step, small elementwise, layer norm, KV-write and split-K kernels account for about 200 µs of the ~240 µs step over-prediction (derived). Calibrating that floor is the main accuracy opportunity.

> **Small request error is not small GPU error.** Decode GPU busy is 575.6 µs real versus 817.0 µs simulated (+41.9%, about +42%). In eager mode, about 93% of step wall time is the retained host residual (a derived residual, not independently measured CPU time). That dilutes GPU error to about +3% e2e; CUDA graphs could move request error toward ~42% unless the floor is calibrated.

### Across architectures

| Hardware / config | Full request mean | Comparison / status |
|---|---|---|
| Real H100 | 254.45 ms | Measured unprofiled reference |
| SM90_H100 simulation | 262.14 ms | +3.02% vs real H100 |
| SM90_H200 simulation | 262.16 ms | —; counterfactual, no measured H200 baseline |
| Real A100 | 222.08 ms | Measured native unprofiled reference |
| SM80_A100 simulation, complete Modal | 229.58 ms | **+3.38% vs real A100** |
| B200 / Blackwell | — | No config exists in Accel-Sim |

H100 traces run on H100/H200 (sm_90); older architectures are rejected for mismatch. H200 differs here only in memory-controller count (40 → 48), yielding +0.02 ms per request. **Inference:** this small workload does not separate the configs; the simplified H200 model is not a validated physical H200.

> Coverage is 100%, not proof of accuracy: dedup verification found up to 39.7% cycle disagreement in one small-kernel group. Attention and KV-write cycles are reused across short contexts, so this is not a context-scaling result.

## Scope and evidence

This analysis describes the **H100 `e2e-plan-v1` sample and native A100 `e2e-a100-plan-v1` follow-up**, not a generic model benchmark. Unless explicitly marked A100, the original sections below describe H100. The source is [simulator@ed337cd](https://github.com/atoniolo76/cascade/blob/ed337cd1ccad643058bfdf25eef570bfba872792/docs/sample-analysis-opt125m.md). Measured values are computed from:

- `/home/ubuntu/runs/phases/e2e-plan-v1/`: `run.json`, `workload.json`, `corrected_timeline.json`, `trace_alignment.json`, phase logs/step logs, `sim/kernels.csv`, `sim/summary.json`, `sim_durations.json`, `sim/compare.json`.
- Committed [hardware sweep](https://github.com/atoniolo76/cascade/blob/8cbcd04d46daa2c57089193c76f3c25584d485cc/results/hw-sweep/README.md): inventory, per-kernel/step/request CSVs, H100/H200 summaries and comparisons.
- Native A100 profiling and the [complete Modal reference](https://github.com/atoniolo76/cascade/blob/ed337cd1ccad643058bfdf25eef570bfba872792/results/a100-plan-v1/modal-complete/tables.md), with the [secondary partial Sail evidence](https://github.com/atoniolo76/cascade/blob/ed337cd1ccad643058bfdf25eef570bfba872792/results/a100-plan-v1/sail-partial/tables.md) kept separate.

Absolute paths identify the collected evidence on the execution VM, not portable dependencies of Cascade. The hardware-sweep artifacts are committed; the large profiling run is retained in the run store. See the [run-directory contract](https://github.com/atoniolo76/cascade/blob/8cbcd04d46daa2c57089193c76f3c25584d485cc/docs/run-dir-contract.md) for file ownership and semantics. Numbers marked **derived** are arithmetic on these artifacts; **inferred** means interpretation, not an additional measurement.

**A100 is complete:** native profiling used `f190c99` on Modal A100-80GB with local-disk tracing. The authoritative SM80_A100 simulation is the complete Modal run at `accelsim-traces:hw-sweep/e2e-a100-plan-v1-sm80-v1/`. Its real baseline was byte-identical to the collected A100 timeline. No H100 traces are substituted for native A100 SASS.

## Workload and environment

| Item | Saved value |
|---|---|
| Model / architecture | `facebook/opt-125m` / `OPTForCausalLM` |
| Prompt | `Hello, my name is` (six scheduled prefill tokens) |
| Generation | 32 output tokens, four repeats; repeat 0 is warm-up |
| Execution | eager (`enforce_eager=true`), async scheduling off |
| Model shape | 12 layers, hidden 768, 12 attention/KV heads, head dimension 64 |
| Numeric format / parallelism | float16, tensor parallel size 1, no quantization |
| vLLM / measured GPU | 0.21.0 / NVIDIA H100 80GB HBM3 |
| Profiled median SM / memory clocks | 1980 MHz / 2619 MHz |
| Accel-Sim / GPGPU-Sim pins | `7ba54de` / `4195dd4` |
| Simulation core clocks | H100 and H200: 1980 MHz, read from config |

There are **32 steps/request: one prefill and 31 decode**, not 33 steps. Repeat-local traced steps are 0, 2, 10, 26, at contexts 6, 8, 16, 32. The requested plan contains context 1, but this six-token prompt has no such decode context. All 128 timing steps exist; the three steady repeats have 96 profiled steps and 17,088 kernels (178 per step). Repeat 1 supplies the four instruction-traced steps: 712 aligned occurrences.

## Pipeline cost

These are stage wall seconds from the **original** run, not the later Sail sweep. The trace stage includes both spinlock-detection passes and the NVBit trace process. Compilation/image build, scheduling and downloads are excluded.

| Stage | Wall seconds | Share of summed stages (derived) |
|---|---|---|
| Baseline process | 123.582 | 4.67% |
| Nsight-profiled process | 30.140 | 1.14% |
| NVBit + spinlock processes | 1738.577 | 65.73% |
| Trace post-processing | 34.171 | 1.29% |
| Accel-Sim orchestration | 718.709 | 27.17% |
| Total of the above | 2645.179 | 100% |

NVBit step envelopes come from `traced/trace_steps.json` monotonic timestamps:

| Local step | Context | Instrumented step seconds |
|---|---|---|
| 0 | 6 | 390.943 |
| 2 | 8 | 375.588 |
| 10 | 16 | 375.926 |
| 26 | 32 | 372.553 |

The sum is 1515.010 seconds (derived), not the full 1738.577-second stage. Spinlock pass 0 and pass 1 step-log spans, from first step start to last step end, are 50.968 and 44.285 seconds. These spans **exclude process/engine startup**; they are not complete pass wall timings. Trace-process step span is 1541.794 seconds, also excluding its startup/teardown.

The logs explicitly report engine initialization (profile/KV-cache/warmup) of 85.80 seconds in baseline, 2.00 in profiled, 13.72 in spinlock pass 0, 13.68 in pass 1 and 13.78 in tracing. These overlap the stage totals and must not be added again. Baseline weight loading alone is only 0.41 seconds. **Inference:** engine initialization and cold/warm caches are significant; the difference between processes does not isolate any one cache's effect.

### Local-disk optimization and artifact compatibility

The old trace logs show `TRACES_FOLDER`/working paths under `/__modal/volumes/.../phases/e2e-plan-v1/traced`; the approximately 375-second decode envelopes above are measured on that path. This alone does not prove the fraction attributable to volume I/O. The separate earlier local-disk comparison mentioned in the task is not in the source artifacts above, so no measured speedup factor is asserted here.

Profiler commit `0b54e42` now runs both spinlock passes, tracing and post-processing under `/tmp/cascade-trace-*/traced`, then copies the result once into the run directory, measuring `stage_wall_s.trace_collect`. Failures also collect diagnostic files before scratch cleanup. This commit was tested and pushed **before** the A100 launch; `f190c99` adds `--gpu` and builds NVBit with `ARCH=all` rather than an H100-only binary.

**No bundle was introduced.** The collected directory preserves the existing `traced/traces/kernel-*.trace.xz`, `kernelslist.g` and processed `.tracez` layout, plus `traced/trace_steps.json`, metrics, spinlock data and logs. The pipeline's post-processor provides the existing compressed trace format; collection is a directory copy, not tar/zstd. No extraction command or app ingestion change is needed. The separate legacy streaming S3 path is unchanged. The A100 run confirms 624 raw and 624 processed traces; collection took 3.912637 s.

## Real timeline anatomy

The following means cover three steady prefills and 93 steady decodes. `host_ns = baseline_wall_ns − gpu_busy_ns` is a **residual decomposition**: it is not independently instrumented exclusive CPU time. CUDA API time can overlap GPU work and is not an additional summand.

| Step kind | N | Baseline wall mean ms | Wall p50 ms | GPU busy mean ms | Residual host mean ms | CUDA API mean ms |
|---|---|---|---|---|---|---|
| Prefill | 3 | 8.498641 | 8.458830 | 0.615279 | 7.883362 | 1.637063 |
| Decode | 93 | 7.894492 | 7.848008 | 0.575611 | 7.318881 | 1.656008 |

Decode gap-before mean/p50 is 0.036950/0.035911 ms. First-step gaps are null, not an extra request boundary delay. Request endpoint fields retain signed post-last-step intervals; they must not be replaced by zero.

| Steady repeat | Real TTFT ms | Real ITL mean ms | Real E2E ms | Sum walls ms | Sum gaps ms |
|---|---|---|---|---|---|
| 1 | 8.636344 | 7.896743 | 253.435384 | 252.179409 | 1.158236 |
| 2 | 8.382869 | 7.942919 | 254.613360 | 253.426680 | 1.109083 |
| 3 | 8.864829 | 7.949773 | 255.307791 | 254.077620 | 1.169005 |

Walls + gaps are not the whole envelope: the contract also preserves pre-first-step and signed post-last-step terms. ITL p50 is absent in the original request metrics, so none is invented.

### Real kernel mix per step

Counts and summed kernel durations below are averaged over **all steady steps of each kind**, not just the four instruction samples. Names are classified with `results/hw-sweep/report.py::group`: KV/attention names, `nvjet` GEMM grid-Y values 9/3/12/6/66 for QKV/out/up/down/lm_head, and elementwise/layer-norm/reduction names. Role labels are **inferred**, not NVTX annotations. Kernel sums need not equal step GPU busy (unmatched residual activity and the busy-time aggregation are different quantities).

| Group | Kernels/step | Prefill summed GPU µs | Decode summed GPU µs |
|---|---|---|---|
| QKV GEMM | 12 | 56.682 | 55.910 |
| Attention | 12 | 95.584 | 69.689 |
| KV write | 12 | 26.645 | 23.904 |
| Out-proj | 12 | 49.258 | 49.028 |
| MLP up GEMM | 12 | 58.900 | 58.622 |
| MLP down GEMM | 12 | 60.288 | 59.908 |
| MLP down split-K reduction | 12 | 24.096 | 23.163 |
| lm_head | 1 | 30.773 | 30.474 |
| Small elementwise | 63 | 88.585 | 86.520 |
| Layer norm | 25 | 85.151 | 85.398 |
| Other setup/index/reduction | 5 | 28.533 | 22.196 |
| Total count | 178 | — | — |

## Accel-Sim by kernel group

These tables use the **later Sail hardware sweep**, where both configs ran 31 representatives with verification disabled. Means are over the same 712 aligned occurrences (one prefill and three sampled decodes); they are not weighted by the full request's 31 decodes. Real values are matched by `(repeat, step, seq)`, not approximate name matching. `cycles / MHz = µs`. Error is signed `(mean simulated µs / mean real µs − 1) × 100`, not MAPE. The reference is Nsight-measured kernel time, not independently measured unprofiled per-kernel time; only step/request baselines are unprofiled.

| Group | Real mean µs | H100 mean cycles | H100 µs | H100 error % | H200 mean cycles | H200 µs | H200 error (no real baseline) |
|---|---|---|---|---|---|---|---|
| QKV GEMM | 4.689312 | 9786.000 | 4.942424 | +5.398 | 9771.000 | 4.934848 | — |
| Attention | 6.326604 | 13589.750 | 6.863510 | +8.486 | 13590.250 | 6.863763 | — |
| KV write | 2.052000 | 8073.750 | 4.077652 | +98.716 | 8073.250 | 4.077399 | — |
| Out-proj | 4.079292 | 8832.000 | 4.460606 | +9.348 | 8831.000 | 4.460101 | — |
| MLP up GEMM | 4.875271 | 10821.000 | 5.465152 | +12.099 | 10684.000 | 5.395960 | — |
| MLP down GEMM | 5.007292 | 10273.000 | 5.188384 | +3.617 | 10282.000 | 5.192929 | — |
| Split-K reduction | 1.940000 | 7269.000 | 3.671212 | +89.238 | 7269.000 | 3.671212 | — |
| lm_head | 30.623500 | 52002.000 | 26.263636 | −14.237 | 55296.000 | 27.927273 | — |
| Small elementwise | 1.385139 | 6397.849 | 3.231237 | +133.279 | 6397.044 | 3.230830 | — |
| Layer norm | 3.404100 | 10103.750 | 5.102904 | +49.905 | 10099.750 | 5.100884 | — |
| Other setup/index/reduction | 4.753550 | 13534.600 | 6.835657 | +43.801 | 13486.800 | 6.811515 | — |

H200 predictions are **counterfactual**, with no accuracy error shown because no real H200 baseline exists. The shipped H100/H200 configs both have 132 SMs, identical clocks and simple DRAM modeling; memory-controller count changes from 40 to 48. Near-identical request predictions do not imply physical H100/H200 equivalence.

| Native workload / modeled config | Real baseline | Sim wall s | Unique executions / traced occurrences | Failures / timeouts |
|---|---|---|---|---|
| H100 traces / H100, original verification run | H100 | 718.709 | 93 / 712 | 0 / 0 |
| H100 traces / H100, Sail sweep | H100 | 804.429 | 31 / 712 | 0 / 0 |
| H100 traces / H200, Sail sweep | H100 counterfactual only | 777.060 | 31 / 712 | 0 / 0 |
| Native A100 / SM80_A100, complete Modal | A100 | 565.086 | 33 / 624 | 0 / 0 |
| Native A100 / SM80_A100, secondary partial Sail | A100 | 1124.050 | 99 / 624 | 0 / 4 aligned timeouts (3 executions) |

## Coverage, dedup verification and context scaling

Both original and Sail replay cover **17,088/17,088 steady kernels (100%)**; each captured request has 5696/5696. Measured GPU-time coverage is **98.13%**; the unmatched measured remainder is retained, not modeled by Accel-Sim. Warm-up has no comparable profiled kernels. Coverage is not an accuracy score.

| Run | Exact | Dedup | Identical propagation | Interpolation | Verification |
|---|---|---|---|---|---|
| Original H100 | 93 | 619 | 15206 | 1170 | 2 extra members/group |
| Sail H100 | 31 | 681 | 16376 | 0 | disabled |
| Sail H200 | 31 | 681 | 16376 | 0 | disabled |

The original has 31 header groups, 31 representatives plus 62 verification executions. The sum of individual simulation wall times is 2866.019 seconds, versus 718.709 orchestration seconds because kernels execute concurrently. Do not sum reused dedup-row wall fields as additional CPU executions.

Dedup keys contain launch/header properties, **not full instruction/memory trace contents**. Original verification therefore matters:

| Group / representative UID | Maximum cycle difference | Maximum relative difference |
|---|---|---|
| Index-select setup, 6217 | 4426 | 39.734% |
| Small elementwise, 6379 | 604 | 6.404% |
| Prefill layer norm, 6221 | 621 | 6.145% |
| QKV GEMM, 6222 | 360 | 3.679% |
| Out-proj, 6226 | 201 | 2.276% |
| Decode layer norm, 6577 | 213 | 2.108% |
| MLP down GEMM, 6231 | 127 | 1.236% |
| MLP up GEMM, 6229 | 132 | 1.220% |

The largest mismatch is UID 6573 versus representative 6217 (`indexSelectSmallIndex`), not attention or a large GEMM. It occurs between different prefill/decode input contexts sharing a header. This demonstrates that header identity is not cycle identity; it does not quantify error for every unverified group member. A zero reference did not produce nonzero verification cycles in this sample.

First-layer original duration records show:

| Local step / context | KV cycles | KV provenance | Attention cycles | Attention provenance |
|---|---|---|---|---|
| 0 / 6 | 8229 | exact | 14501 | exact |
| 2 / 8 | 8022 | exact | 13286 | exact |
| 10 / 16 | 8022 | dedup:6579 | 13286 | dedup:6580 |
| 26 / 32 | 8022 | dedup:6579 | 13286 | dedup:6580 |

Thus this pipeline's output is flat across sampled decode contexts **because it reuses representative cycles**. This is not an independent observation of physical/context-invariant attention. The original decode attention/KV verification comparisons have zero difference, but those two checked members do not establish arbitrary context scaling. In the verification-free Sail sweep, all sampled decode groups satisfy identical propagation, so enabling `propagate=interp` actually yields **zero interpolation labels**. Inference: independent context simulations and a stronger cache/dedup key are required before making scaling claims; the current results do not provide them.

## Replay errors and eager-mode dilution

`cascade simulate` replaces modeled kernel durations while retaining host residuals, gaps, unmatched GPU activity and request envelopes. Both original and later runs use the same real H100 baseline. Positive signed errors below mean simulation is longer; the original step/GPU mean errors are all positive.

| Run | Mean abs step error µs / % | Mean abs GPU-busy error µs / % |
|---|---|---|
| Original H100 | 239.587 / 3.031 | 239.587 / 41.551 |
| Sail H100 | 240.307 / 3.040 | 240.307 / 41.676 |
| Sail H200 counterfactual | — (no real H200 baseline) | — |

Original H100 p50/p95 absolute step errors are 240.540/242.402 µs.

| Repeat | Original TTFT error ms | ITL mean error ms | E2E error ms | Sail H100 E2E error ms | H200 error (no real baseline) |
|---|---|---|---|---|---|
| 1 | +0.350397 | +0.243503 | +7.675474 | +7.698534 | — |
| 2 | +0.326948 | +0.243288 | +7.665878 | +7.688938 | — |
| 3 | +0.436559 | +0.239997 | +7.657047 | +7.680107 | — |

Later sweep GPU busy decode/prefill mean is 816.980/822.679 µs for H100 and 817.620/823.136 µs for H200. Predicted E2E repeats 1/2/3 are 261.133918/262.302298/262.987898 ms and 261.154211/262.322591/263.008191 ms respectively.

**Derived explanation:** only about 7.29% of the mean decode step is GPU busy (0.575611 / 7.894492); about 92.71% is the retained host residual. Adding roughly 240 µs to GPU busy can therefore be a roughly 42% GPU error but only 3% step/E2E error. Eager-mode request agreement is a weak validation of GPU model fidelity. It is not evidence that a graph-mode or compute-dominated workload would have the same small request error. Keeping host/gaps fixed also makes the counterfactual insensitive to hardware-dependent host changes.

## Architecture compatibility

Trace binary version is 90; the stream contains Hopper-specific kernels, including `FlashAttnFwdSm90` and SM90 GEMMs. H100 and H200 are the valid registered native targets. Older configs are rejected by the Sail app's architecture gate; preliminary pre-Sail probes also failed (for example, A100 hit the specialized-unit scheduler assertion). These are **not valid A100 predictions**. A real A100 run must choose native kernels and collect its own SM80 traces and unprofiled baseline.

The pinned and checked upstream-dev trees have no Blackwell/SM100/B200/B100/ GB200 configuration. H200 is the only shipped Hopper variant beyond H100. No H100-number-edited approximation was created. See the sweep inventory for all 12 tested and nine deprecated config files and their SM/clock evidence.

## Bottlenecks and optimization order

| Priority / proposal | Measured motivation | Status / qualification |
|---|---|---|
| 1. Local-disk trace and post-process, collect once | H100 NVBit 1738.577 s, 65.73% of stage sum; decode envelopes 372.553–375.926 s on volume | Implemented/pushed `0b54e42`; A100 local-disk decode envelopes 786.332–869.279 s, collection 3.913 s. Different native kernels/hardware: not a controlled speedup comparison. No bundle or S3 protocol change. |
| 2. Trace one instance per distinct kernel, using NVBit kernel-range limits | 712 traced occurrences collapse to 31 header groups | Proposal only. Need a prepass/range mapping and alignment preserving omitted identities; headers are not sufficient proof of semantic identity. |
| 3. Cache spinlock detection | Two step spans of 50.968/44.285 s plus initialization, every trace invocation | Proposal only. Cache must include tool/binary/model/input compatibility; do not disable required spinlock handling blindly. |
| 4. CTA sampling for large kernels | Original lm_head executions 459.661/454.428/397.684 s; next slowest 46.701 s | Proposal only. Needs estimator/error validation, especially persistent scheduling and memory contention. |
| 5. Persistent simulator cache keyed by trace hash + config | Repeated H100/H200 sweep recomputes 31 representatives/config | Proposal only. Include simulator/build/trace-parser versions, complete effective config and auxiliary config files; no header-only identity. |
| 6. Fewer verification copies after validation | 93 versus 31 executions; verification lm_head copies include the 459.661/397.684 s jobs | Proposal/trade-off. Disabling verification hides observed 39.734% mismatch and changes propagation; not an accuracy-preserving optimization by default. |
| 7. Warm caches / amortize engine init | Baseline engine init 85.80 s versus 2.00 s profiled, weights only 0.41 s | Proposal. Difference suggests warming effects but is not a controlled measurement of cache speedup. |

The lm_head is only one of 178 kernels/step and about 30.5 µs of real decode GPU time, yet its trace-driven CPU simulation takes hundreds of seconds. **Inference:** these jobs are likely critical-path drivers of the original 719-second orchestration, but CSVs do not store execution overlap intervals, so an exact critical-path decomposition is not available. The later Sail sweep likewise spends 773.080 s (H100) / 745.353 s (H200) in its single lm_head representative, compared with total walls 804.429/777.060 s. These CPU wall measurements must never be confused with modeled GPU latency.

## Native A100 complete reference

The same workload ran on NVIDIA A100-SXM4-80GB with vLLM 0.21.0, eager execution, 32 output tokens, four repeats and `trace_steps=plan`. All 624 trace headers identify SM80; alignment has zero unmatched records. Conversion uses the shipped SM80_A100 clock, 1410 MHz, not the profiled median clock of 1155 MHz.

| Steady repeat | Real A100 E2E ms | Complete simulated E2E ms |
|---|---|---|
| 1 | 222.428868 | 229.943469 |
| 2 | 221.967065 | 229.440732 |
| 3 | 221.853422 | 229.355994 |
| Mean | 222.083118 | 229.580065 (+3.375739%) |

The complete Modal run executed 33 representatives for 624 aligned occurrences with zero failures/timeouts, taking 565.086033 s orchestration wall. Verification was disabled: provenance is 33 exact, 591 dedup, 14,100 identical and zero interp. Coverage is 14,724/14,724 steady kernels (100%) and 98.2570% measured GPU time; unmodeled residual activity remains measured. Step-wall MAPE is 3.394%, but GPU-busy MAPE is 24.141%. Retained eager host time dilutes the request error; +3.38% E2E is not a sub-4% GPU-model accuracy claim.

| A100 profiling stage | Wall seconds |
|---|---|
| Baseline process | 123.845410 |
| Nsight-profiled process | 29.751735 |
| NVBit + spinlock processes | 3271.468301 |
| Post-processing | 62.159358 |
| Collection from /tmp to volume | 3.912637 |

Instrumented step envelopes were 625.564963/869.278675/813.347379/786.331915 s for local steps 0/2/10/26 (contexts 6/8/16/32). These do not reproduce the earlier 52-s local-disk anecdote. Different GPUs/native kernels prevent a controlled I/O-speedup claim; the remaining instrumentation cost has not been isolated.

> **The Sail experiment is secondary and partial.** It used verify_dedup=2 and eight processes, but three lm_head executions hit the 900-s cap (four aligned timeouts after dedup expansion). Its 92.86% measured GPU-time coverage retains all 96 steady lm_head durations, so its apparently smaller E2E error is not a complete prediction. Verification found 39.378776% / 4171 cycles disagreement in a GEMVX group shared by inferred out-projection and MLP-down roles, distinct from H100's index-select discrepancy. Both Sail guests were terminated and the checkpoint deleted. No replacement simulation was launched.

## Reproduce the A100 comparison

Profiler (already includes both pushed fixes):

sh:

```
modal run accelsim_trace.py::e2e_main --gpu A100-80GB \
  --run-name e2e-a100-plan-v1 --model facebook/opt-125m \
  --prompt 'Hello, my name is' --max-tokens 32 --repeats 4 \
  --trace-steps plan --trace-repeat 1
modal volume get accelsim-traces /phases/e2e-a100-plan-v1 \
  /home/ubuntu/runs/phases
```

The committed [Sail driver](https://github.com/atoniolo76/cascade/blob/ed337cd1ccad643058bfdf25eef570bfba872792/results/a100-plan-v1/drive_sail.py) reproduces the secondary experiment through the deployed app. It ingests native SM80 traces and uses dedup=True, verify_dedup=2 and propagate="interp", saving results separately under `sail-partial`. Cleanup terminates main/child guests and deletes the checkpoint. It does not overwrite the complete Modal reference.

Then, from the simulator checkout:

sh:

```
RUN=/home/ubuntu/runs/phases/e2e-a100-plan-v1
SIM=results/a100-plan-v1/modal-complete
gzip -dk "$SIM/sim_durations.json.gz"
uv run cascade simulate "$RUN/corrected_timeline.json" \
  --durations "$SIM/sim_durations.json" --output "$SIM/simulated_timeline.json"
uv run cascade compare "$RUN/corrected_timeline.json" \
  "$SIM/simulated_timeline.json" --output "$SIM/compare.json"
```

These commands run from the simulator checkout at the pinned source commit. Both native A100 replays compare against the real A100 baseline, never the H100 baseline.


# Request simulation and scheduling

Page: request-simulation.html

A finished instruction simulation is a hardware performance profile, not a measurement of every possible request. Cascade uses that profile to estimate new prompt/output sizes and to run a discrete-event, vLLM-v1-like request scheduler. No model inference or new GPU calibration is performed by this request layer.

> **Outside-calibration predictions are unvalidated.** The bundled OPT-125m profiles have one prefill size, **6 tokens**, and decode contexts **7–37**. Longer prompts, longer contexts and concurrent/chunked batches have not been validated by new GPU runs. Do not interpret a plausible latency or throughput number as measured performance or a proven accuracy bound.

## Choose the right interface

- **Request simulator UI:** open the sidebar entry and choose a ready trace and supported GPU config, or use the link on a finished run/sim to prefill both. Enter prompt/output tokens, request count and arrival rate; expand the advanced scheduler assumptions if needed. Results include percentile cards, sortable/paginated request metrics, step charts, calibration and provenance.
- **`POST /api/scenarios`:** simulate explicit arrivals or a shorthand workload with continuous batching. It uses the usual API bearer authentication. Results are transient, not persisted as new inference runs.
- **`POST /v1/completions` or `/v1/chat/completions`:** estimate one submitted request with the same performance profile. **`/v1` never batches across HTTP requests**, even when calls arrive simultaneously.

The request-sizing targets are exactly `SM90_H100`, `SM90_H200` and `SM80_A100`, and the chosen config must be installed in the app's GPU-config registry. Trace/config architectures must match: H100 traces can use H100/H200; native A100 traces are required for A100. Other targets, quantized models and tensor-parallel models are unsupported. This is a single-GPU, dense-model approximation, not a general vLLM emulator.

## Build a profile from a finished sim

1. Resolve the trace by exact ID or by model name (newest ready trace), then its GPU config. The shared resolver first reuses a successful sim with posted step timing; otherwise it attaches to a pending/launching/running sim, or launches a normal one-sim base run. `fresh: true` requests a new base run.
2. Wait for base timing up to `CASCADE_OPENAI_WAIT_S`, default 3600 seconds. A timeout is 504, with the base run/sim IDs; it does not cancel the run. Failed sims or success without a step timeline return 502. The UI links matching active simulations while waiting.
3. Combine the stored real `corrected_timeline.json` metadata with supplied `run.json`/`workload.json` and the stored simulated timeline. `build_profile` requires steady, single-request repeats with one unchunked prefill each and a common profiled prompt/output shape.
4. Classify kernels, sum simulated durations by group, and take medians over repeats for each kind/context. Record separate prefill and decode host residuals (`step wall − GPU busy`), unattributed GPU busy, inter-step gaps, pre-first-step envelopes and signed post-last-step envelopes.
5. Cache the resulting `PerfProfile` in process memory by `sim_id`, up to 128 profiles. Different request sizes reuse the base sim; scenario results themselves are not cached or saved. Only an absent/fresh base creates a normal run visible in the existing UI.

Bundled samples reuse finished simulations by default. With `fresh: true`, or when no reusable base exists, they launch a run that **replays recorded Accel-Sim results**, marked `replay: true`; configs without recorded results return 422. Instruction traces are not bundled, so replay does not perform new instruction simulation or GPU calibration. The CPU Accel-Sim execution remains in the **simulator**, never the profiler. Request sizing is a separate pure-Python performance model over finished timing; it does not generate or replay new-shaped SASS traces.

## Kernel groups and roofline floors

Grouping is heuristic, based on kernel names and launch shape. Model dimensions supply hidden/FFN widths, layers, heads/KV heads, vocabulary and dtype byte width. A missing FFN width is assumed to be `4 × hidden`; missing OPT vocabulary is assumed to be 50,272. Such fallbacks appear in the assumptions list.

| Group | Identification / approximate work |
|---|---|
| `linear` | GEMM/GEMV, cuBLAS/CUTLASS/xmma and related matmuls. FLOPs ≈ 2 × non-embedding parameters × total query tokens; bytes include body weights and activation traffic. |
| `attention` | Flash/attention/paged attention kernels. FLOPs ≈ 4 × layers × hidden × Σ(query tokens × context); KV-read bytes scale with the sum of per-sequence contexts. |
| `kv_write` | Identifiable reshape/cache writes, separated from attention; bytes scale with query tokens × KV bytes/token. |
| `lm_head` | Explicit name, or the final linear launch when its grid is at least twice the median linear grid. FLOPs ≈ 2 × hidden × vocabulary × sampling sequences, plus projection-weight bytes. |
| `sampler` | Sampling/top-k/argmax/softmax-like work, proportional to vocabulary and sampling sequences, not all prompt tokens. |
| `elementwise` | Remaining operations, including reductions and layer norms; approximate memory traffic scales with query tokens × hidden × layers. |

For a batch, the model receives per-sequence query-token and context vectors, not merely a total token count. LM-head/sampler work counts only sequences that sample on this step: ordinary decode or a completed prefill. Intermediate prefill chunks omit those groups.

```text
roofline(group, shape) = max(FLOPs / (0.7 × peak compute),
                            bytes / (0.8 × peak bandwidth))
predicted group time = max(baseline group floor, anchored roofline time)
step wall = assumed host(kind) + sum(predicted group times) + residual GPU busy
```

The roofline is anchored downward if needed so its calibrated-shape value cannot exceed the corresponding simulated group floor. That anchoring is not a fitted scaling law. Decode floors interpolate between available baseline contexts and clamp at endpoints outside their range; the prefill floor comes from its single recorded size. Only the roofline term grows beyond the range. Linear activations assume 8 memory passes and elementwise work 16 passes; the dense FFN is ungated with two projections, and biases/norm parameters are omitted.

Matching calibrated group times reproduce their simulated floors exactly. Full request estimates use median components rather than replaying each repeat, so they need not equal the base timeline's mean E2E exactly. Baseline floors and retained eager host time can dominate: a nearly flat predicted ITL is **not evidence that long contexts are free**.

### Datasheet assumptions

| Config | Dense BF16/FP16 TFLOPS | HBM TB/s | Memory GB |
|---|---|---|---|
| `SM90_H100` | 989 | 3.35 | 80 |
| `SM90_H200` | 989 | 4.8 | 141 |
| `SM80_A100` | 312 | 2.039 | 80 |

These are NVIDIA SXM datasheet values ([H100](https://www.nvidia.com/en-us/data-center/h100/), [H200](https://www.nvidia.com/en-us/data-center/h200/), [A100](https://www.nvidia.com/en-us/data-center/a100/)), not achieved measurements. The **0.7 compute** and **0.8 bandwidth** fractions are named assumptions, not independently calibrated efficiencies. Datasheet memory GB/TB use decimal units. H200 remains counterfactual: there is no measured H200 baseline or accuracy error.

## Read provenance before the numbers

| Label | Meaning |
|---|---|
| `measured` | Single-request shape matches a stored simulated step: prefill at the profiled size, or decode at an available context. It does **not** mean independently measured on physical hardware. |
| `interpolated` | A single-request decode context lies between available calibration endpoints, without an exact stored point. |
| `extrapolated:roofline` | Outside the profiled shape/range, or a multi-sequence/chunked shape. An unvalidated model prediction. |
| `assumed-host` | Host timing is held at the profiled kind median. Every step carries this alongside its GPU provenance label. |

Counts describe steps, not kernels or a probability of correctness. The sample's 7–37 decode-context envelope comes from its finished timeline; only prefill 6 and decode contexts around 8/16/32 were instruction-traced. Upstream deduplication/context interpolation can supply many of the stored durations. Those points are not independent context-scaling measurements. Neither within-range labels nor 100% replay coverage validate new batching behavior. See the [sample analysis](sample-analysis-opt125m.html) for base-run accuracy and dedup limitations.

## Scheduler semantics

1. **Arrivals and idle time.** Arrivals enter a waiting queue in stable order. If no request is runnable, advance to the next arrival rather than consuming fake GPU steps.
2. **Running first, decode first.** Schedule running decode requests (one query token each) before partial prefills. Respect the sequence and batched-token budgets; a mixed step uses prefill-kind host/gap assumptions if it contains any prefill work.
3. **FCFS admission and chunked prefill.** Admit waiting requests first-come, first-served as budgets/KV space permit. With chunking enabled, split prompts across steps to fit the token budget. Without it, a prompt larger than the budget is rejected.
4. **KV blocks and recompute preemption.** Allocate blocks as context crosses block boundaries. Under memory pressure, preempt the most recently admitted running request, free its blocks and place it at the front of waiting. Recompute its prompt plus already generated prefix; retain its emitted tokens and do not emit them twice. It cannot be readmitted in the same step.
5. **Time and emissions.** Use the batch GPU estimate plus kind-specific host time, profiled inter-step gaps and one pre-step envelope per busy period. Tokens emit at actual step ends. Completion retains the signed post-step envelope; completed requests free their blocks.

TTFT and E2E start at arrival; queue time ends at first GPU-step start. Per-request ITL is computed from successive token timestamps. Aggregate p50 ITL pools those intervals, percentiles use linear interpolation, and output throughput divides total emitted tokens by first-arrival-to-last-completion time. A single request at time zero agrees with `simulate_request` within 1e-6 relative in the consistency tests. Zero output still performs prefill, with null TTFT/ITL; one output has no ITL interval.

### Defaults and KV capacity

| Setting | Default / assumption |
|---|---|
| `max_num_seqs` | 256 |
| `max_num_batched_tokens` | 8192 |
| `block_size` | 16 tokens |
| `enable_chunked_prefill` | true |
| `gpu_memory_utilization` | Supplied run/profile value, otherwise 0.8 |
| `num_kv_blocks` | Estimated, unless explicitly overridden |

> **Resolved vLLM scheduler defaults and KV-block capacity were not captured in the sample traces.** These settings are scheduler assumptions, not a claim to reproduce the exact vLLM deployment configuration. Memory utilization is distinct from the roofline's bandwidth-efficiency fraction, even though both can be 0.8.

```text
KV bytes/token = 2 × layers × KV heads × head dimension × dtype bytes
weight bytes = (dense body parameters + hidden × vocabulary) × dtype bytes
KV blocks = floor((GPU memory bytes × utilization − weight bytes − 1 GiB)
                  / (block_size × KV bytes/token))
```

The embedding/projection weights are assumed tied. The fixed activation reserve is **1 GiB = 1,073,741,824 bytes**; fragmentation is ignored. An explicit `num_kv_blocks` overrides the estimate and can intentionally model a smaller budget. Impossible isolation capacity, or an unchunked recompute prefix exceeding the token budget, returns an error rather than looping indefinitely.

## Scenario API example

This example reuses the H100 bundled sample and submits eight simultaneous requests, each with 512 prompt tokens and 128 output tokens. It performs no new Accel-Sim execution.

```bash
curl "$CASCADE_URL/api/scenarios" \
  -H "Authorization: Bearer $CASCADE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "trace_id": "sample__facebook_opt-125m__h100-e2e",
    "gpu_config": "SM90_H100",
    "num_requests": 8,
    "prompt_tokens": 512,
    "output_tokens": 128,
    "arrival_rate_rps": 0,
    "scheduler": {"max_num_seqs": 256, "max_num_batched_tokens": 8192}
  }'
```

Rounded response excerpt from the request-simulator example; omitted fields include per-request metrics, per-step records, resolved scheduler and the full assumption list:

json:

```
{
  "trace_id": "sample__facebook_opt-125m__h100-e2e",
  "gpu_config": "SM90_H100",
  "run_id": "run-sample-opt125m-h100-e2e",
  "sim_id": "sim-sample-opt125m-h100-e2e",
  "cached": true,
  "aggregate": {
    "p50_ttft_ms": 9.799,
    "p99_ttft_ms": 9.799,
    "p50_itl_ms": 8.125,
    "p50_e2e_ms": 1041.514,
    "p99_e2e_ms": 1041.514,
    "throughput_output_tokens_s": 983.184
  },
  "steps_total": 128,
  "steps_returned": 128,
  "steps_downsampled": false,
  "provenance_counts": {
    "measured": 0,
    "interpolated": 0,
    "extrapolated:roofline": 128,
    "assumed-host": 128
  },
  "calibration_range": {"prefill_tokens": [6, 6], "decode_context": [7, 37]}
}
```

Provide exactly one selector, `trace_id` or `model`. Shorthand requires `num_requests`, `prompt_tokens` and `output_tokens`. Null/zero/omitted arrival rate means all arrive at zero; a positive rate produces deterministic arrival `i / arrival_rate_rps`, not a Poisson process. For explicit requests, replace the shorthand fields with:

json:

```
{
  "trace_id": "sample__facebook_opt-125m__h100-e2e",
  "requests": [
    {"id": "a", "arrival_s": 0, "prompt_tokens": 512, "output_tokens": 128},
    {"id": "b", "arrival_s": 0.5, "prompt_tokens": 512, "output_tokens": 128}
  ]
}
```

Do not mix explicit requests with shorthand fields. Omitted IDs become `request-<index>` and must remain unique. Prompts must be positive integers, outputs nonnegative integers, and prompt plus output must fit the supplied model limit (2048 if absent). Invalid bodies return 422. Service safety limits are **4096 requests**, **1,000,000 total prompt/output tokens**, **100,000 executed steps**, and finite arrivals within 0–1e9 seconds. These are not vLLM limits.

Above 5000 steps, the response uniformly downsamples records while retaining first/last and original `step_index`; `steps_total`, `steps_returned`, `steps_downsampled` and `steps_sampling` disclose it. Aggregate/request metrics and provenance counts always use all executed steps. The UI may further subsample chart points; it does not recalculate the metrics from the plotted subset.

## OpenAI-compatible single-request example

```bash
curl "$CASCADE_URL/v1/completions" \
  -H "Authorization: Bearer $CASCADE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "sample__facebook_opt-125m__h100-e2e",
    "gpu_config": "SM90_H100",
    "prompt": "Hello, my name is",
    "max_tokens": 32,
    "seed": 42
  }'
```

With the OPT tokenizer reporting six prompt tokens, the requested-shape model predicts **8.946 ms TTFT, 8.125 ms mean ITL and 260.593 ms E2E** (rounded). These request-layer estimates use median profile components; they are not the base replay's per-repeat means. Check `usage.prompt_tokens` and `cascade.prompt_tokens_source` before comparing with this six-token example.

- `cascade.timing` contains requested-size TTFT, mean/p50 ITL, E2E, token counts, calibration, assumptions and provenance. `cascade.profiled` separately retains base-workload real/simulated metrics and its comparison summary. `profile_match` remains a prompt/output match flag, not proof of accuracy.
- Both completions and chat accept `max_tokens` or `max_completion_tokens` (conflicting values are rejected). The default is the trace workload's `max_tokens`. Timing uses that resolved count, not the base workload's output count.
- Prompt usage and timing use the same tokenizer count. If a GPT2/OPT model lacks `tokenizer.json`, its byte-level BPE tokenizer is built from `vocab.json`, `merges.txt` and tokenizer configuration (including special tokens), then cached. OPT's configured BOS `</s>` (ID 2) is included in the six-token example above. Chat applies a usable cached chat template, otherwise joins message contents with newlines. If the tokenizer is unavailable, the deterministic estimate is `max(1, ceil(UTF-8 bytes / 4))`, flagged `estimated:utf8-bytes/4`; the prompt above then estimates five tokens, not six. Other sources are `tokenizer`, `tokenizer:chat_template` and `tokenizer:joined`.
- Response text is uniformly sampled non-special vocabulary IDs, seedable with `seed`; unavailable tokenizers use flagged `tok<i>` placeholders. This is not semantic text generation. Completion usage counts sampled IDs, which decoded text may re-tokenize differently.
- With `stream: true`, final chunks carry the same timing/profiled structure after progress comments and content deltas; chunks are not paced like actual inference. Concurrent `/v1` calls can share a base sim but **never share a scheduler batch**.

## Reported predictions, not new calibration

Standalone request estimates from the bundled H100/A100 profiles; times are ms and ITL is mean. All rows other than the matching 6/32 shape use unvalidated roofline extrapolation; host time is assumed throughout.

| GPU | Prompt / output | TTFT | Mean ITL | E2E |
|---|---|---|---|---|
| SM90_H100 | 6 / 32 | 8.946 | 8.125 | 260.593 |
| SM90_H100 | 128 / 128 | 8.946 | 8.125 | 1040.662 |
| SM90_H100 | 1024 / 256 | 8.956 | 8.125 | 2080.732 |
| SM90_H100 | 1800 / 200 | 9.228 | 8.125 | 1625.977 |
| SM80_A100 | 6 / 32 | 8.190 | 7.139 | 229.340 |
| SM80_A100 | 128 / 128 | 8.190 | 7.139 | 914.664 |
| SM80_A100 | 1024 / 256 | 8.398 | 7.139 | 1828.637 |
| SM80_A100 | 1800 / 200 | 9.371 | 7.139 | 1429.838 |

H100 scheduler estimates for **512 prompt / 128 output** tokens per request, all arriving at zero with default scheduler assumptions:

| Concurrent requests | p50 TTFT ms | p50 ITL ms | p50 E2E ms | Output tok/s |
|---|---|---|---|---|
| 1 | 8.946 | 8.125 | 1040.662 | 123.00 |
| 8 | 9.799 | 8.125 | 1041.514 | 983.18 |
| 64 | 33.824 | 8.549 | 1122.326 | 7194.53 |

Estimated default H100 KV capacity is 106,267 blocks; those three examples do not preempt. A separate deterministic stress test with three 8/8 requests, block size 4 and only 4 KV blocks finishes all requests with exactly eight output tokens each, preemptions [0, 3, 2], in 23 steps. This checks scheduler bookkeeping, not fidelity to a real memory-pressure run.

> **No new GPU prompt/context/concurrency sweep has calibrated these curves.** Flat ITL, higher estimated batching throughput, within-range provenance or successful recomputation tests do not validate real vLLM performance. The [recorded H100/A100 replay errors](sample-analysis-opt125m.html) apply to the original workload only, not to the arbitrary request sizes or scheduler results on this page.


# Traces

Page: traces.html

The app watches its bucket directory. A trace appears on the **Traces** screen as soon as its first file lands, and it's ready once the upload is complete and post-processed.

## Follow a trace

Open **Traces** in the UI, or poll [List traces](api-traces.html#list-traces). Each trace moves through these states.

| value | meaning |
|---|---|
| `uploading` | Files are arriving. `kernel_count` and `bytes` grow on each scan, every 2 seconds. |
| `processing` | The upload is complete. The app is running the trace post-processor, unless the upload already has `traces/kernelslist.g`. |
| `ready` | Post-processing produced `traces/kernelslist.g`. Runs can launch. |
| `failed` | `metadata.json` has `status: "failed"`, or post-processing failed. `error` says which. |

Open a trace to see what it came from: the model, the layers, the prompts, the source GPU, the suggested config, and every traced kernel with its grid and block size. The kernel list fills in when the trace becomes `ready`.

## When an upload counts as complete

The app scans `$CASCADE_DATA_DIR/s3/<bucket>/<prefix>/` every 2 seconds. It skips dotfiles, which covers rclone's in-flight `.rclone_temp_put_*` objects. The upload is complete when `metadata.json` exists and one of these holds.

- `metadata.json` has a `files` list, and every listed file is on disk with the listed size in bytes. `scripts/upload_trace.py` writes this list.
- There is no `files` list, every `traces/kernelslist_ctx_*` has a matching `traces/stats_ctx_*`, and the file tree hasn't changed since the previous scan. This is the case for the streaming profiler, which uploads `metadata.json` before its `traces/` files.

If `metadata.json` has `status: "failed"`, the trace goes to `failed` straight away, without processing.

## Post-processing

The profiler uploads raw traces. Accel-Sim reads processed ones, `traces/kernelslist.g` and one `.tracez` per kernel. So the app runs the post-processor once per trace.

```bash
$CASCADE_POSTPROCESS_CMD <trace dir>/traces -j <cpu count>
```

Its output goes to `$CASCADE_DATA_DIR/work/postprocess/<trace id>.log`. If it exits non-zero, the trace fails with the exit code and the last 5 lines of that log. If it succeeds but leaves no `kernelslist.g`, the trace fails too.

An upload that already contains `traces/kernelslist.g` skips post-processing. The quickstart's stand-in trace works this way.

If the app restarts during post-processing, the trace stays `processing` and the post-processor runs again on the next start.

## Where the trace's fields come from

| field | source |
|---|---|
| `id` | The directory name under the prefix, which is the profiler's `--run-name`. |
| `model`, `layers` | `metadata.json`. |
| `source_gpu` | `metadata.json` `gpu`. |
| `trace_gpu_arch` | `metadata.json` `trace_gpu_arch`, or `sm_90` if it's missing. |
| `suggested_sim_config` | `metadata.json` `suggested_sim_config`. The profiler doesn't set it, so the New run screen preselects the first compatible config instead. |
| `kernel_count` | While uploading, the raw `kernel-*.trace` and `.trace.xz` files seen. Once ready, the kernel lines in `kernelslist.g`. |
| `bytes` | Every file under the trace's directory. |
| `kernels` | The rows of `traces/stats_ctx_*`, or `stats.csv` from older tracers. |
| `has_real_timing` | Whether `real_timing.json`, or the file `metadata.json` names in `real_timing`, holds a list of rows with integer `kernel_id`s. |
| `has_step_timeline` | Whether the trace directory holds both `corrected_timeline.json` and `kernel_mapping.json`. The app rechecks on every scan, so it can turn true after the trace is `ready`. See [Step timing](simulator.html#step-timing). |
| `sample` | True only for the sample trace `cascade serve` seeds from `samples/` (see [Introduction](index.html)). It is `ready` but has no instruction traces, so it can't start a run. |

## Disk space

Traces grow with the layers, tokens and forward passes you trace. A single small layer is tens of MB. Whole models with long prompts reach hundreds of GB. Processed files sit next to the raw ones, so plan for about 2.2 times the uploaded size on the app's disk, and pass `--disk-gib` when you deploy.


# Runs

Page: runs.html

A run simulates one ready trace on a list of GPU configs, with a count for each. Every count is its own sim, and every sim is its own sandbox.

## Launch a run in the UI

1. Open a `ready` trace and choose **New run**.
2. Add a row per config and set its count, from 1 to 64. The trace's `suggested_sim_config` is preselected. Without one, the first compatible config is.
3. Choose **Launch 6 simulations**. The button counts the sims for you. The run screen opens and updates live.

Configs for a different architecture than the trace's `trace_gpu_arch` are disabled. An H100 trace contains Hopper-only instructions, so it runs only on `sm_90` configs such as `SM90_H100` and `SM90_H200`. To model another architecture, trace on that GPU.

## Launch a run over the API

[Create a run](api-runs.html#create-run) takes the trace id and the entries.

```bash
curl -X POST "$CASCADE_URL/api/runs" \
  -H "Authorization: Bearer $CASCADE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"trace_id": "demo", "entries": [{"gpu_config": "SM90_H100", "count": 4}, {"gpu_config": "SM90_H200", "count": 2}]}'
```

The app checks the request before creating anything.

- The trace must exist (`404`), be `ready`, and not be a sample (`409`).
- Every `gpu_config` must be one [List GPU configs](api-runs.html#list-gpu-configs) returns, with the trace's architecture (`422`).
- `entries` can't be empty and every `count` is 1 to 64 (`422`).

The response is the new run with every sim `pending` or `launching`. Launching happens in the background. [Concepts](concepts.html#how-a-run-launches) walks through it.

## Follow a run

The run screen shows a card per sim with its state, how long since its last heartbeat, kernels done out of the trace's kernel count, the kernel it's on, its latest metrics, and the last lines of its log. Over the API, read [Get a run](api-runs.html#get-run) or subscribe to [Stream run events](api-events.html).

The run's own state comes from its sims.

| sim states | run state |
|---|---|
| `running`, `succeeded` | `active` |
| `succeeded`, `succeeded` | `succeeded` |
| `succeeded`, `failed` | `partial` |
| `failed`, `failed` | `failed` |
| `cancelled`, `failed` | `cancelled` |
| `cancelled`, `succeeded` | `partial` |
| `cancelled`, `cancelled` | `cancelled` |

## Cancel a run

On the run screen, choose **Cancel run** and then **Confirm cancel**, or call [Cancel a run](api-runs.html#cancel-run). Every sim that hasn't finished becomes `cancelled`, the app stops its sandbox, and the simulator's next heartbeat gets `cancel: true`. Sims that already finished keep their results. Cancelling twice is harmless.

## When a sim goes quiet

A sim with no heartbeat for 60 seconds shows as `lost`. It goes back to `running` when it reports again. A sim still `lost` after 600 seconds becomes `failed` and its sandbox is stopped. Both times scale with `CASCADE_HEARTBEAT_TIMEOUT_S`.

## After a restart

The app keeps everything in SQLite at `$CASCADE_DATA_DIR/cascade.db`. When it starts again, it launches any sims still `pending`. Sims on Sail run in their own sandboxes, so they keep going and their heartbeats land again once the app is back.


# Reading results

Page: reading-results.html

Every number Cascade shows comes from Accel-Sim's own output. Sims send raw stdout, and the app parses one stats block per finished kernel out of it.

## Metrics on a sim card

Each sim card shows its latest totals. They update every time the sim sends a log chunk with a new kernel block.

| metric | Accel-Sim line | meaning |
|---|---|---|
| Cycles | `gpu_tot_sim_cycle` | Simulated GPU cycles so far, over every finished kernel. |
| IPC | `gpu_tot_ipc` | Instructions per cycle over the whole simulated GPU so far. |
| Sim rate | `gpgpu_simulation_rate` | How fast the simulator itself runs, in simulated instructions per wall-clock second. |
| Kernels | `kernel_name` blocks | Kernel blocks parsed so far, out of the trace's kernel count. |

In the examples on these pages, one sim finished with `gpu_tot_sim_cycle` 76713, `gpu_tot_ipc` 346.2289 and a sim rate of 510774 instructions per second. Those numbers come from the recorded BabelStream log that the stand-in simulator replays.

## Simulated time per kernel

Accel-Sim counts cycles. To turn cycles into time, the app divides by the config's core clock, the first value of `-gpgpu_clock_domains` in its `gpgpusim.config`.

```text
sim_time_ns = gpu_sim_cycle / core_clock_mhz * 1000
```

For example, the first kernel in the captured run took 6174 cycles on `SM86_RTX3070`, whose core clock is 1132 MHz, so its `sim_time_ns` is 6174 / 1132 * 1000 = 5454.06.

## Real against simulated

The comparison panel lines kernels up by position. Row `i` is the `i`th kernel the sims finished, and for each config it shows the median `sim_time_ns` over that config's replicas. With several replicas, one odd sim can't skew the row.

When a trace has real timings from the original GPU, row `i` also shows the `i`th real kernel by `kernel_id`, and the panel shows each config's relative error per kernel and its mean absolute error over all kernels. The app reads them from `real_timing.json` in the trace, or the file `metadata.json` names in `real_timing`, as a list of `{kernel_id, kernel_name, duration_ns, grid, block}`.

> The profiler doesn't capture real timings yet. Until it does, `has_real_timing` is false, `real_ns` is `null` on every row, and the panel shows simulated time only.

## Outputs

A simulator can upload the files it writes. `cascade worker` uploads every file Accel-Sim created or changed in its run directory, such as `kernel-stats.csv`, `gpgpu_inst_stats.txt` and `exit-status.txt`, and then its own log as `accelsim.log.gz`. It does this when the simulator exits, succeeded or failed, before the final heartbeat. A cancelled sim uploads nothing.

The run screen's **Outputs** card groups the files by config, then by replica, with each file's size. Select a sim card, and its panel lists the same sim's files under **Files**. Every file has a download button. A `.csv`, `.txt` or `.log` file up to 16 MiB, gzipped or not, also has a preview button that shows its first 200 lines.

The card has two more buttons. **Perfetto JSON** downloads the run's [Perfetto trace](reading-results.html#the-perfetto-trace). **Open in Perfetto** opens ui.perfetto.dev in a new tab and hands it the same trace, so there is no file to save and open. Both stay disabled until a sim has finished a kernel.

In the examples on these pages, each sim's output files come from a recorded OPT-125m run on `SM90_H100`, which the stand-in copies in with `--replay-outputs`. They don't match the BabelStream log the sim replays.

## The Perfetto trace

The app builds the trace from the kernel stats it parsed, so every run has one, whether or not the simulator wrote its own. It is a Chrome trace JSON file.

- Each config is one process, named `<config> · Accel-Sim simulated GPU`.
- Each replica is one thread in its config's process, named `replica <n> · <sim id>`.
- Each finished kernel is one slice, named `k<uid> <readable name>`, such as `k2 copy_kernel`. The readable name is the one the run screen shows.
- A sim's slices are laid end to end in kernel order. A slice lasts the kernel's `sim_time_ns`, and the time axis is simulated microseconds. A kernel without `sim_time_ns` is left out.
- Each slice's args hold `kernel_name` as recorded, `kernel_index`, `uid`, `gpu_sim_cycle`, `gpu_sim_insn`, `ipc` and `sim_time_ns`. `start_cycle` and `end_cycle` count the sim's cycles before and after the kernel.
- When the trace has real timing, one more process, `Real hardware · <source_gpu>`, holds the measured kernels the same way.

Each slice includes the config's launch latency. Laying kernels end to end leaves out host time, copies and any overlap between kernels.

## Downloads

| download | where | contents |
|---|---|---|
| Run results | **Download results** on the run screen, or [the API](api-runs.html#get-run-results) | One [`RunResults`](types.html#runresults) JSON file: the run, the trace, every sim's per-kernel stats, and the comparison. |
| Sim results | **Results** on a sim card, or [the API](api-runs.html#get-sim-results) | One [`SimResults`](types.html#simresults) file: the sim, its config, its kernels and its totals. |
| Log | **Log** on a sim card, or [the API](api-runs.html#get-sim-log) | The sim's full raw output, as plain text. |
| Output file | The **Outputs** card, **Files** in a sim's panel, or [the API](api-runs.html#get-sim-file) | One file as the sim uploaded it. A `.gz` file stays compressed. |
| Perfetto JSON | **Perfetto JSON** on the **Outputs** card, or [the API](api-runs.html#get-run-perfetto) | The run's simulated kernels as a Chrome trace JSON, named `<run_id>-perfetto.json`. |

Each kernel in a results file has a `stats` map with every `key = value` line in its Accel-Sim block, about 950 keys and 30 KB per kernel. Numbers are parsed as floats, and anything else stays a string. The live run screen leaves `stats` out to stay small. Use the downloads for it.

## How the parser reads a log

A kernel's block starts at a `kernel_name = ...` line and ends at the next `gpgpu_simulation_time`, `launching kernel name:`, `Processing kernel`, `Header info loaded` or `GPGPU-Sim: *** exit detected` line, or at the next `kernel_name`. A block still open when the sim finishes is closed then. The `launching kernel name: X uid: N` line sets the sim's `current_kernel`.

The app stores the raw log as well as the parsed stats. After a restart, it rebuilds a running sim's parser by replaying that sim's stored log, so parsing picks up where it stopped.


# Bring your own simulator

Page: simulator.html

Cascade runs whatever command is in `CASCADE_SIM_CMD` inside each sim. The app ships one, `cascade worker`, which runs Accel-Sim. Any other program that follows this contract can be the simulator: a patched build, or something else entirely.

## The contract

1. Read your identity from the environment.

   | variable | value |
   |---|---|
   | `CASCADE_SIM_ID` | This sim's id, such as `sim-6b4c91e71437`. |
   | `CASCADE_MAIN_URL` | The app's URL, from `CASCADE_PUBLIC_URL`. |
   | `CASCADE_TOKEN` | The bearer token for every request. |
   | `CASCADE_GPU_CONFIG` | The config to simulate, such as `SM90_H100`. |
   | `CASCADE_TRACE_DIR` | The trace's directory. The processed trace is at `$CASCADE_TRACE_DIR/traces/kernelslist.g`. |
   | `CASCADE_ACCELSIM_DIR` | The app's `CASCADE_ACCELSIM_DIR`, where Accel-Sim and its configs live. |

2. Send `POST /api/sims/{id}/heartbeat` with `{"state": "running"}` right away, and again every few seconds. If the reply has `"cancel": true`, stop and exit.

3. Send your output as `POST /api/sims/{id}/logs` with `{"seq": n, "lines": [...]}`, counting `seq` up from 0. Send Accel-Sim's stdout as it is. The app parses the per-kernel stats out of it.

4. Optionally, upload the files your simulator wrote with `PUT /api/sims/{id}/files/{name}`, one file per call, with the raw bytes as the body. They show up under the run's **Outputs**.

5. Finish with one heartbeat of `succeeded` or `failed`, with your `exit_code`, and an `error` if you failed. Send every log chunk and file before it.

[Simulator endpoints](api-simulator.html) has the full request and response for each call.

## Retries are safe

Every call is safe to repeat, so retry any `5xx` or network error with backoff. The app ignores a log chunk whose `seq` it already has, a heartbeat that repeats the current state only refreshes the heartbeat time, and a file put again replaces the earlier one. Don't retry a `4xx`. It means the token, the body or the file name is wrong, the file is too big, or the sim doesn't exist.

Keep heartbeats going while you wait on retries. After 60 seconds of silence the sim shows as `lost`, and after 600 seconds it fails and the app stops its sandbox.

## The built-in worker

`cascade worker` follows this contract for Accel-Sim. To use it, deploy with `--sim-cmd "/app/.venv/bin/cascade worker"`.

- It runs `$CASCADE_ACCELSIM_DIR/gpu-simulator/bin/release/accel-sim.out -trace $CASCADE_TRACE_DIR/traces/kernelslist.g` with the sim's `gpgpusim.config` and `trace.config`. It runs in a fresh directory, `$TMPDIR/cascade-worker/<sim id>/`, with a copy of every file in both config directories, because some configs name their aux files by relative path.
- It keeps the full log in that directory as `accelsim.log`.
- If the binary, a config or `kernelslist.g` is missing, it sends a `failed` heartbeat with exit code 2 and an error that names the missing path.
- It sends a chunk every 200 lines or 0.5 seconds, a heartbeat every 5 seconds, and every chunk before its final heartbeat.
- After the simulator exits, succeeded or failed, it uploads its [output files](simulator.html#output-files), then sends the final heartbeat.
- `cascade worker --replay <log>` replays a recorded Accel-Sim log instead of running Accel-Sim. `--delay` sets the pause every 200 lines, 0.05 seconds by default. `--replay-outputs <dir>` copies the directory's files, except `run.log` and `setup.log`, into the run directory as if Accel-Sim wrote them, so they upload like real output. This is the stand-in simulator. A default deploy replays a recorded OPT-125m run and its outputs.

## Output files

Before it starts the simulator, `cascade worker` notes every file in its run directory, which is the copied configs. After the simulator exits, it uploads each regular file in that directory that the simulator created or changed, in name order, and then its own log as `accelsim.log.gz`. It skips symlinks and subdirectories. A cancelled sim uploads nothing.

- A text file over 1 MiB is gzipped as it uploads and stored as `<name>.gz`.
- The app keeps up to 64 MiB per file and 256 MiB per sim. The worker skips a file over a limit, and a name the app would refuse, with one log line each.
- A failed upload is retried like every other call. If it still fails, the worker writes a log line, and the sim's state doesn't change.

A custom simulator uploads with the same call. [Upload an output file](api-simulator.html#put-sim-file) has the name rules and errors.

## Step timing

A trace can carry two more files next to `metadata.json`. When both are there, the trace's `has_step_timeline` is true. The app checks on every scan, so they can arrive after the trace is ready.

- `corrected_timeline.json`, the profiler's per-step timeline of the real run.
- `kernel_mapping.json`, a list of `{"uid", "name", "repeat", "step", "seq"}`. `uid` and `name` name a kernel in the sim's log. `repeat`, `step` and `seq` name the kernel in the corrected timeline that the sim's time replaces.

After a sim succeeds on such a trace, `cascade worker` runs the step-timing model in `cascade/sim/` on its own log. It imports each mapped kernel's cycles with the config's core clock, rebuilds every step with `cascade simulate`'s defaults, and compares the result with the corrected timeline using `cascade compare`. Then it sends both results to [Send step timing](api-simulator.html#post-step-timeline), before its final heartbeat. If either file is missing, it writes one log line and skips this. If the model fails, for example because the mapping names a kernel the log doesn't have, the sim still succeeds, nothing is sent, and the error goes to the log.

## A complete client

To run a simulator other than Accel-Sim, write your own client. This one runs any command, forwards its stdout as log chunks, sends a heartbeat every 5 seconds, stops the command if the run is cancelled, and reports the exit code. It uses only the Python standard library.

cascade_sim.py:

```
"""Run a simulator command as a Cascade sim: send heartbeats and forward its stdout.

CASCADE_SIM_CMD="python3 cascade_sim.py accel-sim.out -trace ... -config ..."
"""

import json
import os
import subprocess
import sys
import threading
import time
import urllib.error
import urllib.request

BASE = f"{os.environ['CASCADE_MAIN_URL']}/api/sims/{os.environ['CASCADE_SIM_ID']}"
TOKEN = os.environ["CASCADE_TOKEN"]
cancelled = threading.Event()


def post(path, body):
    data = json.dumps(body).encode()
    headers = {"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"}
    for attempt in range(6):
        req = urllib.request.Request(f"{BASE}/{path}", data, headers, method="POST")
        try:
            with urllib.request.urlopen(req, timeout=10) as resp:
                raw = resp.read()
                return json.loads(raw) if raw else None
        except urllib.error.HTTPError as e:
            if e.code < 500:
                raise
        except (urllib.error.URLError, TimeoutError):
            pass
        time.sleep(2**attempt)
    raise RuntimeError(f"POST {path} kept failing")


def heartbeat(state, exit_code=None, error=None):
    ack = post("heartbeat", {"state": state, "exit_code": exit_code, "error": error})
    if ack["cancel"]:
        cancelled.set()


def keep_beating(proc):
    while proc.poll() is None:
        try:
            heartbeat("running")
        except Exception as e:
            print(f"heartbeat failed: {e}", file=sys.stderr)
        if cancelled.is_set():
            proc.terminate()
        time.sleep(5)


def main():
    heartbeat("running")
    if cancelled.is_set():
        return 0
    proc = subprocess.Popen(
        sys.argv[1:], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, errors="replace"
    )
    threading.Thread(target=keep_beating, args=(proc,), daemon=True).start()
    seq, batch, sent_at = 0, [], time.monotonic()
    for line in proc.stdout:
        batch.append(line.rstrip("\n"))
        if len(batch) >= 200 or time.monotonic() - sent_at > 1:
            post("logs", {"seq": seq, "lines": batch})
            seq, batch, sent_at = seq + 1, [], time.monotonic()
    if batch:
        post("logs", {"seq": seq, "lines": batch})
    code = proc.wait()
    if cancelled.is_set():
        return 0
    if code == 0:
        heartbeat("succeeded", exit_code=0)
    else:
        heartbeat("failed", exit_code=code, error=f"simulator exited with {code}")
    return code


if __name__ == "__main__":
    sys.exit(main())
```

Put it in your simulator image and point `CASCADE_SIM_CMD` at it, followed by your command. On Sail, the app runs the command through a shell inside the fork, so `$VARIABLES` such as `$CASCADE_GPU_CONFIG` expand there. Quote them with single quotes so your own shell leaves them alone. The local launcher splits the command into arguments with `shlex` and runs it without a shell, so they don't expand locally.

## What the image needs

The app is layered on top of your image with the Dockerfile's `BASE_IMAGE` argument, so the app and every sim run the same image. It needs these things.

| what | where the app looks |
|---|---|
| Accel-Sim's GPU configs | `$CASCADE_ACCELSIM_DIR/gpu-simulator/gpgpu-sim/configs/tested-cfgs/*/gpgpusim.config`. Each directory becomes a GPU config. |
| The trace post-processor | `CASCADE_POSTPROCESS_CMD`, by default `$CASCADE_ACCELSIM_DIR/util/tracer_nvbit/tracer_tool/traces-processing/post-traces-processing`. The app runs it on the main sandbox, not in sims. |
| Accel-Sim | `$CASCADE_ACCELSIM_DIR/gpu-simulator/bin/release/accel-sim.out`, and each config's `trace.config` under `$CASCADE_ACCELSIM_DIR/gpu-simulator/configs/tested-cfgs/`. `cascade worker` runs it. |
| Your simulator command | `CASCADE_SIM_CMD`, a shell command, such as `/app/.venv/bin/cascade worker`. Sims run it with the environment above. |
| A shell and `apt-get` | The Dockerfile installs rclone and uv with `apt-get`, so the base must be Debian or Ubuntu. |

Pin the simulator to the version the profiler traced with. The profiler at `b348d7d` builds the tracer from Accel-Sim commit `7ba54ded`. The app's notes name Accel-Sim tag `v2.0.1` for the simulator.

## Where output goes

Anything your command prints to stdout or stderr is also saved to `$CASCADE_DATA_DIR/work/sims/<sim id>.out` on the sim's disk. Only what you send to `/logs` and `/files` reaches the app, and only the log gets parsed. On Sail, the sim's disk is gone when its fork ends, so upload any file you want to keep.


# API overview and auth

Page: api.html

Everything the UI does goes through this HTTP API. It serves JSON over the app's URL, the one [Get started](../start.html) opens, which `cascade deploy` prints as `app:`.

> Every response on these pages was captured from a running server, by `landing/docs-src/capture_examples.py`. The sims in that run replayed a recorded Accel-Sim log of BabelStream on an RTX 3070, so the cycles, IPC and kernel names come from that log, not from the uploaded trace. That's also why a finished sim shows `kernels_done` 6 against `kernels_total` 4: the log has six kernels and the uploaded trace has four. Long lists and maps are trimmed, and each cut is marked with a `/* ... trimmed */` comment.

## Base URL

The samples read the app's URL and your token from the environment.

```bash
export CASCADE_URL=https://<your-app>
export CASCADE_TOKEN=<your token>
```

Locally, the app's URL is `http://127.0.0.1:8000`.

## Authentication

Send the token the app was started with as a bearer token on every request.

```bash
curl -H "Authorization: Bearer $CASCADE_TOKEN" "$CASCADE_URL/api/traces"
```

A missing or wrong token gets `401` with a `WWW-Authenticate: Bearer` header. Two routes differ.

- [Check health](api.html#health) needs no token.
- [Stream run events](api-events.html#run-events) also accepts the token as `?token=`, because the browser's `EventSource` can't send headers.

There is one token per app, shared by the UI, the profiler and every sim. The app compares it in constant time.

## Conventions

- Request and response bodies are JSON. Every [type](types.html) is generated from `cascade/models.py`.
- Timestamps are ISO 8601 in UTC, such as `2026-11-08T03:05:34.001386Z`.
- A trace's id is the profiler's run name. Runs and sims get generated ids, `run-` or `sim-` and 12 hex digits.
- Lists come back whole. There is no pagination.
- Errors are JSON with a `detail` field. See [Errors](errors.html).

## Routes

| method | path | what it does |
|---|---|---|
| GET | [`/api/health`](api.html#health) | Liveness check. Needs no token. |
| GET | [`/api/traces`](api-traces.html#list-traces) | Every trace, newest first. |
| GET | [`/api/traces/{trace_id}`](api-traces.html#get-trace) | One trace with its metadata and kernels. |
| GET | [`/api/gpu-configs`](api-runs.html#list-gpu-configs) | The GPU configs a run can use. |
| POST | [`/api/runs`](api-runs.html#create-run) | Launch sims for a ready trace. |
| GET | [`/api/runs`](api-runs.html#list-runs) | Every run, newest first. |
| GET | [`/api/runs/{run_id}`](api-runs.html#get-run) | One run with its trace and every sim. |
| POST | [`/api/runs/{run_id}/cancel`](api-runs.html#cancel-run) | Cancel every unfinished sim. |
| GET | [`/api/runs/{run_id}/comparison`](api-runs.html#get-comparison) | Per-kernel real time against simulated time per config. |
| GET | [`/api/runs/{run_id}/kernels`](api-runs.html#get-run-kernels) | Every sim's per-kernel stats, without the full stats maps. |
| GET | [`/api/runs/{run_id}/step-timelines`](api-runs.html#get-step-timelines) | Each sim's predicted step timeline and its comparison with the real run. |
| GET | [`/api/runs/{run_id}/results`](api-runs.html#get-run-results) | One JSON file with the run, trace, every sim's stats and the comparison. |
| GET | [`/api/sims/{sim_id}/results`](api-runs.html#get-sim-results) | One sim's config, per-kernel stats and totals. |
| GET | [`/api/sims/{sim_id}/log`](api-runs.html#get-sim-log) | The sim's full raw output. |
| GET | [`/api/runs/{run_id}/files`](api-runs.html#list-run-files) | Every output file the run's sims uploaded. |
| GET | [`/api/runs/{run_id}/perfetto`](api-runs.html#get-run-perfetto) | The run's simulated kernels as a Chrome trace JSON for ui.perfetto.dev. |
| GET | [`/api/sims/{sim_id}/files`](api-runs.html#list-sim-files) | The output files one sim uploaded. |
| GET | [`/api/sims/{sim_id}/files/{name}`](api-runs.html#get-sim-file) | One output file, as the sim uploaded it. |
| GET | [`/api/runs/{run_id}/events`](api-events.html#run-events) | Server-sent events for one run. |
| POST | [`/api/sims/{sim_id}/heartbeat`](api-simulator.html#heartbeat) | Report a sim's state. The reply says whether to stop. |
| POST | [`/api/sims/{sim_id}/logs`](api-simulator.html#logs) | Append a chunk of simulator output. |
| POST | [`/api/sims/{sim_id}/step-timeline`](api-simulator.html#post-step-timeline) | Store a sim's predicted step timeline and its comparison. |
| PUT | [`/api/sims/{sim_id}/files/{name}`](api-simulator.html#put-sim-file) | Store one of the simulator's output files. |

Any other path under `/api/` returns `404` with `{"detail": "Not Found"}`, without checking the token. Paths outside `/api/` serve the web UI.

## Check health

`GET /api/health` (no auth)

Returns `{"ok": true, "auth": true, "workspace": ...}` while the app process is serving requests. `auth` is `false` only when the app runs with `CASCADE_NO_AUTH=1`, a local-dev mode where no token is needed and the web UI skips its token screen. `workspace` is `CASCADE_WORKSPACE`, the name the web UI shows in its rail, or `null`. Health does not check the database, the S3 endpoint or the launcher.

### Returns

`200` with `{"ok": true, "auth": true, "workspace": "<name or null>"}`.

### Example

```bash
curl "$CASCADE_URL/api/health"
```

```python
import os

import httpx

resp = httpx.get(f"{os.environ['CASCADE_URL']}/api/health")
resp.raise_for_status()
print(resp.json())
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/health`);
console.log(await resp.json());
```

Response · 200 OK · application/json:

```json
{
  "ok": true,
  "auth": true,
  "workspace": "acme-lab",
  "s3_endpoint": "http://127.0.0.1:9020"
}
```


# Traces

Page: api-traces.html

Traces arrive over S3, not over this API. These routes read what the app has registered. To add a trace, upload it to the S3 endpoint as described in [Trace a workload](trace-a-workload.html).

## List traces

`GET /api/traces` (bearer token)

Lists every trace the app has registered, in every state, newest first by `created_at`. A trace appears as soon as its first file lands in the bucket, so this list also shows uploads in progress.

### Returns

`200` with a list of [`TraceSummary`](types.html#tracesummary).

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |

### Example

```bash
curl "$CASCADE_URL/api/traces" \
  -H "Authorization: Bearer $CASCADE_TOKEN"
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.get(f"{os.environ['CASCADE_URL']}/api/traces", headers=headers)
resp.raise_for_status()
print(resp.json())
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/traces`, {
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}` },
});
console.log(await resp.json());
```

Response · 200 OK · application/json:

```json
[
  {
    "id": "demo-failed",
    "state": "failed",
    "error": "profiler reported status 'failed' in metadata.json",
    "created_at": "2026-11-08T08:59:02.590922Z",
    "updated_at": "2026-11-08T08:59:02.592166Z",
    "model": null,
    "layers": [],
    "source_gpu": null,
    "trace_gpu_arch": "sm_86",
    "suggested_sim_config": null,
    "kernel_count": 0,
    "bytes": 16650759,
    "has_real_timing": false,
    "has_step_timeline": false,
    "sample": false
  },
  {
    "id": "demo",
    "state": "ready",
    "error": null,
    "created_at": "2026-11-08T08:59:02.587743Z",
    "updated_at": "2026-11-08T08:59:02.592073Z",
    "model": null,
    "layers": [],
    "source_gpu": null,
    "trace_gpu_arch": "sm_86",
    "suggested_sim_config": "SM86_RTX3070",
    "kernel_count": 20,
    "bytes": 16890250,
    "has_real_timing": false,
    "has_step_timeline": true,
    "sample": false
  },
  {
    "id": "sample__facebook_opt-125m__h100-e2e",
    "state": "ready",
    "error": null,
    "created_at": "2026-11-08T06:19:57.515017Z",
    "updated_at": "2026-11-08T06:19:57.515017Z",
    "model": "facebook/opt-125m",
    "layers": [
      "step 0 (prefill)",
      "step 2 (decode)",
      "step 10 (decode)",
      "step 26 (decode)"
    ],
    "source_gpu": "NVIDIA H100 80GB HBM3",
    "trace_gpu_arch": "sm_90",
    "suggested_sim_config": "SM90_H100",
    "kernel_count": 712,
    "bytes": 13065268,
    "has_real_timing": true,
    "has_step_timeline": true,
    "sample": true
  }
]
```

## Get a trace

`GET /api/traces/{trace_id}` (bearer token)

Returns one trace with the parsed `metadata.json` and every kernel launch read from the trace's `stats_ctx_*` files. `kernels` stays empty until the trace is `ready`.

### Path parameters

| name | type | description |
|---|---|---|
| `trace_id` | `str` | The trace's `id`, which is the profiler's run name. |

### Returns

`200` with a [`TraceDetail`](types.html#tracedetail).

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |
| [`404`](errors.html#e404) | No trace has this id. |

### Example

```bash
curl "$CASCADE_URL/api/traces/demo" \
  -H "Authorization: Bearer $CASCADE_TOKEN"
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.get(f"{os.environ['CASCADE_URL']}/api/traces/demo", headers=headers)
resp.raise_for_status()
print(resp.json())
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/traces/demo`, {
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}` },
});
console.log(await resp.json());
```

Response · 200 OK · application/json:

```json
{
  "id": "demo",
  "state": "ready",
  "error": null,
  "created_at": "2026-11-08T08:59:02.587743Z",
  "updated_at": "2026-11-08T08:59:02.592073Z",
  "model": null,
  "layers": [],
  "source_gpu": null,
  "trace_gpu_arch": "sm_86",
  "suggested_sim_config": "SM86_RTX3070",
  "kernel_count": 20,
  "bytes": 16890250,
  "has_real_timing": false,
  "has_step_timeline": true,
  "sample": false,
  "metadata": {
    "status": "ok",
    "files": [
      {
        "key": "accelsim-traces/demo/configs/config_hopper_islip.icnt",
        "bytes": 1352
      },
      {
        "key": "accelsim-traces/demo/configs/gpgpusim.config",
        "bytes": 6028
      },
      {
        "key": "accelsim-traces/demo/configs/trace.config",
        "bytes": 1085
      }
      /* 24 more items trimmed */
    ],
    "trace_gpu_arch": "sm_86",
    "suggested_sim_config": "SM86_RTX3070"
  },
  "kernels": []
}
```

404 response · 404 Not Found · application/json:

```json
{
  "detail": "trace nope not found"
}
```


# Runs

Page: api-runs.html

A run simulates one ready trace on a list of GPU configs. These routes create and cancel runs, read their sims, and download what the sims produced. [Runs](runs.html) covers the same ground as a guide.

## List GPU configs

`GET /api/gpu-configs` (bearer token)

Lists the GPU configs the app found at startup under `$CASCADE_ACCELSIM_DIR/gpu-simulator/gpgpu-sim/configs/tested-cfgs/*/gpgpusim.config`. The app derives `arch` from the directory name (`SM90_H100` is `sm_90`, `SM7_QV100` is `sm_70`) and reads `core_clock_mhz` from the first value of `-gpgpu_clock_domains`. A directory whose name or clock it can't read is skipped.

### Returns

`200` with a list of [`GpuConfig`](types.html#gpuconfig), sorted by name.

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |

### Example

```bash
curl "$CASCADE_URL/api/gpu-configs" \
  -H "Authorization: Bearer $CASCADE_TOKEN"
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.get(f"{os.environ['CASCADE_URL']}/api/gpu-configs", headers=headers)
resp.raise_for_status()
print(resp.json())
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/gpu-configs`, {
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}` },
});
console.log(await resp.json());
```

Response · 200 OK · application/json:

```json
[
  {
    "name": "SM7_QV100",
    "arch": "sm_70",
    "core_clock_mhz": 1132.0
  },
  {
    "name": "SM80_A100",
    "arch": "sm_80",
    "core_clock_mhz": 1410.0
  },
  {
    "name": "SM86_RTX3070",
    "arch": "sm_86",
    "core_clock_mhz": 1132.0
  },
  {
    "name": "SM90_H100",
    "arch": "sm_90",
    "core_clock_mhz": 1980.0
  },
  {
    "name": "SM90_H200",
    "arch": "sm_90",
    "core_clock_mhz": 1980.0
  }
]
```

## Create a run

`POST /api/runs` (bearer token)

Creates a run and starts launching its sims in the background. Each entry fans out to `count` sims, numbered by `replica` from 0. The response comes back before any sim has launched, so every sim is `pending` or `launching`. Follow progress with [Stream run events](api-events.html#run-events) or by polling [Get a run](api-runs.html#get-run).

### Request body

JSON, [RunCreate](types.html#runcreate).

| field | type | description |
|---|---|---|
| `trace_id` | `str` | A trace in state `ready`. |
| `entries` | `list[RunEntry]` | One entry per config. Each fans out to `count` sims. (at least 1 item) |

Each [RunEntry](types.html#runentry) has these fields.

| field | type | description |
|---|---|---|
| `gpu_config` | `str` | A `GpuConfig.name` whose `arch` matches the trace's `trace_gpu_arch`. |
| `count` | `int` | How many sims run this config. (at least 1, at most 64) |

### Returns

`201` with a [`RunDetail`](types.html#rundetail).

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |
| [`404`](errors.html#e404) | `trace_id` names no trace. |
| [`409`](errors.html#e409) | The trace is not `ready`, or is a sample, which has no instruction traces. |
| [`422`](errors.html#e422) | An entry names an unknown `gpu_config`, or a config whose `arch` differs from the trace's `trace_gpu_arch`. |
| [`422`](errors.html#e422) | The body fails validation: `entries` is empty, or a `count` is outside 1 to 64. |

### Example

```bash
curl -X POST "$CASCADE_URL/api/runs" \
  -H "Authorization: Bearer $CASCADE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"trace_id": "demo", "entries": [{"gpu_config": "SM86_RTX3070", "count": 2}]}'
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.post(f"{os.environ['CASCADE_URL']}/api/runs", headers=headers, json={"trace_id": "demo", "entries": [{"gpu_config": "SM86_RTX3070", "count": 2}]})
resp.raise_for_status()
print(resp.json())
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/runs`, {
  method: "POST",
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}`, "Content-Type": "application/json" },
  body: JSON.stringify({"trace_id": "demo", "entries": [{"gpu_config": "SM86_RTX3070", "count": 2}]}),
});
console.log(await resp.json());
```

Response · 201 Created · application/json:

```json
{
  "id": "run-210d75c008c4",
  "trace_id": "demo",
  "created_at": "2026-11-08T08:59:03.024065Z",
  "state": "active",
  "entries": [
    {
      "gpu_config": "SM86_RTX3070",
      "count": 2
    }
  ],
  "sim_counts": {
    "pending": 2
  },
  "sample": false,
  "trace": {
    "id": "demo",
    "state": "ready",
    "error": null,
    "created_at": "2026-11-08T08:59:02.587743Z",
    "updated_at": "2026-11-08T08:59:02.592073Z",
    "model": null,
    "layers": [],
    "source_gpu": null,
    "trace_gpu_arch": "sm_86",
    "suggested_sim_config": "SM86_RTX3070",
    "kernel_count": 20,
    "bytes": 16890250,
    "has_real_timing": false,
    "has_step_timeline": true,
    "sample": false
  },
  "sims": [
    {
      "id": "sim-1047ddbdf5b1",
      "run_id": "run-210d75c008c4",
      "gpu_config": "SM86_RTX3070",
      "replica": 0,
      "state": "pending",
      "error": null,
      "launcher_ref": null,
      "created_at": "2026-11-08T08:59:03.024065Z",
      "started_at": null,
      "finished_at": null,
      "last_heartbeat_at": null,
      "exit_code": null,
      "kernels_done": 0,
      "kernels_total": 20,
      "current_kernel": null,
      "latest": {
        "gpu_tot_sim_cycle": null,
        "gpu_tot_sim_insn": null,
        "gpu_tot_ipc": null,
        "sim_rate_ips": null
      },
      "log_tail": []
    },
    {
      "id": "sim-e8f63b6b3bed",
      "run_id": "run-210d75c008c4",
      "gpu_config": "SM86_RTX3070",
      "replica": 1,
      "state": "pending",
      "error": null,
      "launcher_ref": null,
      "created_at": "2026-11-08T08:59:03.024065Z",
      "started_at": null,
      "finished_at": null,
      "last_heartbeat_at": null,
      "exit_code": null,
      "kernels_done": 0,
      "kernels_total": 20,
      "current_kernel": null,
      "latest": {
        "gpu_tot_sim_cycle": null,
        "gpu_tot_sim_insn": null,
        "gpu_tot_ipc": null,
        "sim_rate_ips": null
      },
      "log_tail": []
    }
  ]
}
```

409 response · 409 Conflict · application/json:

```json
{
  "detail": "trace demo-failed is failed, not ready"
}
```

422 response, arch mismatch · 422 Unprocessable Entity · application/json:

```json
{
  "detail": "SM90_H100 is sm_90 but trace demo is sm_86"
}
```

422 response, validation · 422 Unprocessable Entity · application/json:

```json
{
  "detail": [
    {
      "type": "greater_than_equal",
      "loc": [
        "body",
        "entries",
        0,
        "count"
      ],
      "msg": "Input should be greater than or equal to 1",
      "input": 0,
      "ctx": {
        "ge": 1
      }
    }
  ]
}
```

## List runs

`GET /api/runs` (bearer token)

Lists every run, newest first, with per-state sim counts. Use [Get a run](api-runs.html#get-run) for the sims themselves.

### Returns

`200` with a list of [`RunSummary`](types.html#runsummary).

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |

### Example

```bash
curl "$CASCADE_URL/api/runs" \
  -H "Authorization: Bearer $CASCADE_TOKEN"
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.get(f"{os.environ['CASCADE_URL']}/api/runs", headers=headers)
resp.raise_for_status()
print(resp.json())
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/runs`, {
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}` },
});
console.log(await resp.json());
```

Response · 200 OK · application/json:

```json
[
  {
    "id": "run-210d75c008c4",
    "trace_id": "demo",
    "created_at": "2026-11-08T08:59:03.024065Z",
    "state": "succeeded",
    "entries": [
      {
        "gpu_config": "SM86_RTX3070",
        "count": 2
      }
    ],
    "sim_counts": {
      "succeeded": 2
    },
    "sample": false
  },
  {
    "id": "run-sample-opt125m-h100-e2e",
    "trace_id": "sample__facebook_opt-125m__h100-e2e",
    "created_at": "2026-11-08T06:50:10Z",
    "state": "succeeded",
    "entries": [
      {
        "gpu_config": "SM90_H100",
        "count": 1
      }
    ],
    "sim_counts": {
      "succeeded": 1
    },
    "sample": true
  }
]
```

## Get a run

`GET /api/runs/{run_id}` (bearer token)

Returns the run, its trace, and every sim with its state, progress, latest metrics and last 20 log lines.

### Path parameters

| name | type | description |
|---|---|---|
| `run_id` | `str` | The run's `id`, such as `run-9f56e171af08`. |

### Returns

`200` with a [`RunDetail`](types.html#rundetail).

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |
| [`404`](errors.html#e404) | No run has this id. |

### Example

```bash
curl "$CASCADE_URL/api/runs/run-210d75c008c4" \
  -H "Authorization: Bearer $CASCADE_TOKEN"
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.get(f"{os.environ['CASCADE_URL']}/api/runs/run-210d75c008c4", headers=headers)
resp.raise_for_status()
print(resp.json())
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/runs/run-210d75c008c4`, {
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}` },
});
console.log(await resp.json());
```

Response · 200 OK · application/json:

```json
{
  "id": "run-210d75c008c4",
  "trace_id": "demo",
  "created_at": "2026-11-08T08:59:03.024065Z",
  "state": "succeeded",
  "entries": [
    {
      "gpu_config": "SM86_RTX3070",
      "count": 2
    }
  ],
  "sim_counts": {
    "succeeded": 2
  },
  "sample": false,
  "trace": {
    "id": "demo",
    "state": "ready",
    "error": null,
    "created_at": "2026-11-08T08:59:02.587743Z",
    "updated_at": "2026-11-08T08:59:02.592073Z",
    "model": null,
    "layers": [],
    "source_gpu": null,
    "trace_gpu_arch": "sm_86",
    "suggested_sim_config": "SM86_RTX3070",
    "kernel_count": 20,
    "bytes": 16890250,
    "has_real_timing": false,
    "has_step_timeline": true,
    "sample": false
  },
  "sims": [
    {
      "id": "sim-1047ddbdf5b1",
      "run_id": "run-210d75c008c4",
      "gpu_config": "SM86_RTX3070",
      "replica": 0,
      "state": "succeeded",
      "error": null,
      "launcher_ref": "64418",
      "created_at": "2026-11-08T08:59:03.024065Z",
      "started_at": "2026-11-08T08:59:03.224424Z",
      "finished_at": "2026-11-08T08:59:06.130355Z",
      "last_heartbeat_at": "2026-11-08T08:59:06.130355Z",
      "exit_code": 0,
      "kernels_done": 6,
      "kernels_total": 20,
      "current_kernel": "_Z10dot_kernelIdEvPKT_S2_PS0_i",
      "latest": {
        "gpu_tot_sim_cycle": 76713.0,
        "gpu_tot_sim_insn": 26560256.0,
        "gpu_tot_ipc": 346.2289,
        "sim_rate_ips": 510774.0
      },
      "log_tail": [
        "Reply_Network_in_buffer_full_per_cycle =       0.0000",
        "Reply_Network_in_buffer_avg_util =       0.0000",
        "Reply_Network_out_buffer_full_per_cycle =       0.0000"
        /* 17 more items trimmed */
      ]
    },
    {
      "id": "sim-e8f63b6b3bed",
      "run_id": "run-210d75c008c4",
      "gpu_config": "SM86_RTX3070",
      "replica": 1,
      "state": "succeeded",
      "error": null,
      "launcher_ref": "64419",
      "created_at": "2026-11-08T08:59:03.024065Z",
      "started_at": "2026-11-08T08:59:03.224173Z",
      "finished_at": "2026-11-08T08:59:06.164567Z",
      "last_heartbeat_at": "2026-11-08T08:59:06.164567Z",
      "exit_code": 0,
      "kernels_done": 6,
      "kernels_total": 20,
      "current_kernel": "_Z10dot_kernelIdEvPKT_S2_PS0_i",
      "latest": {
        "gpu_tot_sim_cycle": 76713.0,
        "gpu_tot_sim_insn": 26560256.0,
        "gpu_tot_ipc": 346.2289,
        "sim_rate_ips": 510774.0
      },
      "log_tail": [
        "Reply_Network_in_buffer_full_per_cycle =       0.0000",
        "Reply_Network_in_buffer_avg_util =       0.0000",
        "Reply_Network_out_buffer_full_per_cycle =       0.0000"
        /* 17 more items trimmed */
      ]
    }
  ]
}
```

404 response · 404 Not Found · application/json:

```json
{
  "detail": "run run-000000000000 not found"
}
```

## Cancel a run

`POST /api/runs/{run_id}/cancel` (bearer token)

Moves every sim that has not finished to `cancelled` and asks the launcher to stop it. Sims that already finished keep their state. The next heartbeat from a cancelled sim gets `cancel: true`. Calling it again changes nothing, so it is safe to retry. On a sample run it changes nothing either, since its sim already `succeeded`.

### Path parameters

| name | type | description |
|---|---|---|
| `run_id` | `str` | The run's `id`, such as `run-9f56e171af08`. |

### Returns

`200` with the [`RunDetail`](types.html#rundetail) after cancelling.

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |
| [`404`](errors.html#e404) | No run has this id. |

### Example

```bash
curl -X POST "$CASCADE_URL/api/runs/run-b42541e17665/cancel" \
  -H "Authorization: Bearer $CASCADE_TOKEN"
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.post(f"{os.environ['CASCADE_URL']}/api/runs/run-b42541e17665/cancel", headers=headers)
resp.raise_for_status()
print(resp.json())
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/runs/run-b42541e17665/cancel`, {
  method: "POST",
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}` },
});
console.log(await resp.json());
```

Response · 200 OK · application/json:

```json
{
  "id": "run-b42541e17665",
  "trace_id": "demo",
  "created_at": "2026-11-08T08:59:06.669870Z",
  "state": "cancelled",
  "entries": [
    {
      "gpu_config": "SM86_RTX3070",
      "count": 1
    }
  ],
  "sim_counts": {
    "cancelled": 1
  },
  "sample": false,
  "trace": {
    "id": "demo",
    "state": "ready",
    "error": null,
    "created_at": "2026-11-08T08:59:02.587743Z",
    "updated_at": "2026-11-08T08:59:02.592073Z",
    "model": null,
    "layers": [],
    "source_gpu": null,
    "trace_gpu_arch": "sm_86",
    "suggested_sim_config": "SM86_RTX3070",
    "kernel_count": 20,
    "bytes": 16890250,
    "has_real_timing": false,
    "has_step_timeline": true,
    "sample": false
  },
  "sims": [
    {
      "id": "sim-57c51b464fc9",
      "run_id": "run-b42541e17665",
      "gpu_config": "SM86_RTX3070",
      "replica": 0,
      "state": "cancelled",
      "error": null,
      "launcher_ref": "64556",
      "created_at": "2026-11-08T08:59:06.669870Z",
      "started_at": null,
      "finished_at": "2026-11-08T08:59:06.673285Z",
      "last_heartbeat_at": null,
      "exit_code": null,
      "kernels_done": 0,
      "kernels_total": 20,
      "current_kernel": null,
      "latest": {
        "gpu_tot_sim_cycle": null,
        "gpu_tot_sim_insn": null,
        "gpu_tot_ipc": null,
        "sim_rate_ips": null
      },
      "log_tail": []
    }
  ]
}
```

## Compare real and simulated time

`GET /api/runs/{run_id}/comparison` (bearer token)

Lines up kernels by order. Row `i` holds the `i`th kernel the sims finished and the `i`th row of the trace's real timing, sorted by `kernel_id`. For each config, `sim_ns` is the median `sim_time_ns` over that config's replicas, or `null` if no replica has finished that kernel. `real_ns` is `null` until the trace has real timing, which the profiler does not capture yet.

### Path parameters

| name | type | description |
|---|---|---|
| `run_id` | `str` | The run's `id`, such as `run-9f56e171af08`. |

### Returns

`200` with a [`Comparison`](types.html#comparison).

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |
| [`404`](errors.html#e404) | No run has this id. |

### Example

```bash
curl "$CASCADE_URL/api/runs/run-210d75c008c4/comparison" \
  -H "Authorization: Bearer $CASCADE_TOKEN"
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.get(f"{os.environ['CASCADE_URL']}/api/runs/run-210d75c008c4/comparison", headers=headers)
resp.raise_for_status()
print(resp.json())
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/runs/run-210d75c008c4/comparison`, {
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}` },
});
console.log(await resp.json());
```

Response · 200 OK · application/json:

```json
{
  "has_real_timing": false,
  "configs": [
    "SM86_RTX3070"
  ],
  "rows": [
    {
      "kernel_index": 0,
      "kernel_name": "_Z11init_kernelIdEvPT_S1_S1_S0_S0_S0_",
      "real_ns": null,
      "sim_ns": {
        "SM86_RTX3070": 5454.063604240282
      }
    },
    {
      "kernel_index": 1,
      "kernel_name": "_Z11copy_kernelIdEvPKT_PS0_",
      "real_ns": null,
      "sim_ns": {
        "SM86_RTX3070": 5232.332155477032
      }
    },
    {
      "kernel_index": 2,
      "kernel_name": "_Z10mul_kernelIdEvPT_PKS0_",
      "real_ns": null,
      "sim_ns": {
        "SM86_RTX3070": 5491.166077738516
      }
    }
    /* 3 more items trimmed */
  ]
}
```

## List a run's kernel timings

`GET /api/runs/{run_id}/kernels` (bearer token)

Returns a [`KernelStat`](types.html#kernelstat) for every kernel every sim in the run has finished, ordered by config, replica, then kernel. Each `stats` map is empty to keep the response small; the full maps are in the results downloads. Laying each sim's kernels end to end by `sim_time_ns` gives its timeline, which is what the app's timeline and Compare views draw.

### Path parameters

| name | type | description |
|---|---|---|
| `run_id` | `str` | The run's `id`, such as `run-9f56e171af08`. |

### Returns

`200` with a list of [`KernelStat`](types.html#kernelstat).

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |
| [`404`](errors.html#e404) | No run has this id. |

### Example

```bash
curl "$CASCADE_URL/api/runs/run-210d75c008c4/kernels" \
  -H "Authorization: Bearer $CASCADE_TOKEN"
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.get(f"{os.environ['CASCADE_URL']}/api/runs/run-210d75c008c4/kernels", headers=headers)
resp.raise_for_status()
print(resp.json())
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/runs/run-210d75c008c4/kernels`, {
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}` },
});
console.log(await resp.json());
```

Response · 200 OK · application/json:

```json
[
  {
    "sim_id": "sim-1047ddbdf5b1",
    "kernel_index": 0,
    "kernel_name": "_Z11init_kernelIdEvPT_S1_S1_S0_S0_S0_",
    "kernel_launch_uid": 1,
    "gpu_sim_cycle": 6174,
    "gpu_sim_insn": 19456,
    "gpu_ipc": 3.1513,
    "sim_time_ns": 5454.063604240282,
    "stats": {}
  },
  {
    "sim_id": "sim-1047ddbdf5b1",
    "kernel_index": 1,
    "kernel_name": "_Z11copy_kernelIdEvPKT_PS0_",
    "kernel_launch_uid": 2,
    "gpu_sim_cycle": 5923,
    "gpu_sim_insn": 11264,
    "gpu_ipc": 1.9017,
    "sim_time_ns": 5232.332155477032,
    "stats": {}
  },
  {
    "sim_id": "sim-1047ddbdf5b1",
    "kernel_index": 2,
    "kernel_name": "_Z10mul_kernelIdEvPT_PKS0_",
    "kernel_launch_uid": 3,
    "gpu_sim_cycle": 6216,
    "gpu_sim_insn": 12288,
    "gpu_ipc": 1.9768,
    "sim_time_ns": 5491.166077738516,
    "stats": {}
  }
  /* 9 more items trimmed */
]
```

## List a run's step timing

`GET /api/runs/{run_id}/step-timelines` (bearer token)

Returns a [`SimStepTimeline`](types.html#simsteptimeline) for every sim in the run that has sent step timing, ordered by config, then replica. A sim sends one after it succeeds on a trace whose `has_step_timeline` is true, so the list is empty until then, and stays empty for other traces. `simulated` is a full `simulated_timeline.json`, with every kernel of every step, so for a real model it can be several megabytes per sim. Per-kind step means are in `simulated.summary.per_kind`, coverage is in `simulated.summary.simulation.coverage`, and per-step and per-request errors against the real run are in `compare`.

### Path parameters

| name | type | description |
|---|---|---|
| `run_id` | `str` | The run's `id`, such as `run-9f56e171af08`. |

### Returns

`200` with a list of [`SimStepTimeline`](types.html#simsteptimeline).

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |
| [`404`](errors.html#e404) | No run has this id. |

### Example

```bash
curl "$CASCADE_URL/api/runs/run-210d75c008c4/step-timelines" \
  -H "Authorization: Bearer $CASCADE_TOKEN"
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.get(f"{os.environ['CASCADE_URL']}/api/runs/run-210d75c008c4/step-timelines", headers=headers)
resp.raise_for_status()
print(resp.json())
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/runs/run-210d75c008c4/step-timelines`, {
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}` },
});
console.log(await resp.json());
```

Response · 200 OK · application/json:

```json
[
  {
    "sim_id": "sim-1047ddbdf5b1",
    "gpu_config": "SM86_RTX3070",
    "replica": 0,
    "simulated": {
      "meta": {
        "run_name": "offline-h100-derived-fixture",
        "model": "facebook/opt-125m",
        "prompt": "Hello, my name is",
        "max_tokens": 4,
        "repeats": 1,
        "exec_mode": "eager",
        "gpu_name": "NVIDIA H100 80GB HBM3",
        "sm_clock_mhz_median": 1980,
        "mem_clock_mhz": 3106,
        "vllm_version": "fixture-unknown",
        "layerwise_nvtx": false,
        "fixture_provenance": "Actual nsys kernel names/dimensions/durations and simulator cycles; synthetic phase assignment, baseline/profiled walls, queue/API values and rescaled gaps. Not a real corrected_timeline capture. Synthetic consistent absolute step/request timestamps and baseline inter-step gaps added for gap-aware replay."
      },
      "steps": [
        {
          "step": -1,
          "repeat": -1,
          "warmup": true,
          "kind": "prefill",
          "num_reqs": 1,
          "num_scheduled_tokens": 6,
          "baseline_wall_ns": 9000000,
          "profiled_wall_ns": null,
          "gpu_busy_ns": null,
          "cuda_api_ns": null,
          "host_ns": null,
          "kernels": [],
          "baseline_wall_start_ns": 100000000,
          "baseline_wall_end_ns": 109000000,
          "baseline_gap_before_ns": null,
          "profiled_wall_start_ns": null,
          "profiled_wall_end_ns": null,
          "profiled_gap_before_ns": null
        },
        {
          "step": 0,
          "repeat": 0,
          "warmup": false,
          "kind": "prefill",
          "num_reqs": 1,
          "num_scheduled_tokens": 6,
          "baseline_wall_ns": 6507260.692579505,
          "profiled_wall_ns": 6507260.692579505,
          "gpu_busy_ns": 544095.6925795053,
          "cuda_api_ns": null,
          "host_ns": 5963165,
          "kernels": [
            {
              "seq": 536,
              "name": "_ZN8cublasLt19splitKreduce_kernelILi32ELi16Eif6__halffS1_Lb0EfS1_S1_Lb1ELb1ELb0ELb0EEEvNS_18cublasSplitKParamsIT4_EEPKT2_PKT8_PT7_PT3_PKS3_SG_PKT9_S7_PSH_PvlPS3_PiSM_SM_SG_SG_SG_SG_SG_",
              "grid": "24x1x1",
              "block": "32x16x1",
              "duration_ns": 2016,
              "start_offset_ns": 108088.07402640952
            },
            {
              "seq": 537,
              "name": "_ZN2at6native29vectorized_elementwise_kernelILi8ENS0_15CUDAFunctor_addIN3c104HalfEEESt5arrayIPcLm3EEEEviT0_T1_",
              "grid": "5x1x1",
              "block": "128x1x1",
              "duration_ns": 1376,
              "start_offset_ns": 150410.8374180178
            }
            /* 150 more items trimmed */
          ],
          "baseline_wall_start_ns": 1000200000,
          "baseline_wall_end_ns": 1006707260.6925795,
          "baseline_gap_before_ns": null,
          "profiled_wall_start_ns": 1000200000,
          "profiled_wall_end_ns": 1006707260.6925795,
          "profiled_gap_before_ns": null
        }
        /* 3 more items trimmed */
      ],
      "requests": [
        {
          "repeat": -1,
          "warmup": true,
          "queued_ms": 0,
          "ttft_ms": 9,
          "itl_ms_mean": null,
          "itl_ms_p50": null,
          "e2e_ms": 9,
          "num_output_tokens": 1
        },
        {
          "repeat": 0,
          "warmup": false,
          "queued_ms": 0.2,
          "ttft_ms": 6.707260692579508,
          "itl_ms_mean": 5.79187032508834,
          "itl_ms_p50": 5.65,
          "e2e_ms": 24.082871667844532,
          "num_output_tokens": 4,
          "arrival_ns": 1000000000,
          "pre_first_step_ns": 200000,
          "post_last_step_ns": 0,
          "first_step_start_ns": 1000200000,
          "last_step_end_ns": 1024082871.6678445,
          "first_token_ns": 1006707260.6925795,
          "last_token_ns": 1024082871.6678445
        }
      ],
      "summary": {
        "simulation": {
          "host_model": "scaled-gaps",
          "host_fit": {},
          "coverage": {
            "kernels": 683,
            "simulated_kernels": 6,
            "kernel_fraction": 0.008784773060029283,
            "gpu_time_fraction": 0.011172026136868988,
            "measured_gpu_ns": 2228423.0,
            "covered_measured_gpu_ns": 24896.0,
            "unmodeled_gpu_ns": 0.0,
            "steps": [
              {
                "repeat": 0,
                "step": 0,
                "kernels": 152,
                "simulated_kernels": 5,
                "measured_gpu_ns": 536835,
                "simulated_coverage_gpu_ns": 20128.0,
                "unmodeled_gpu_ns": 0,
                "measured_kernel_overlap_ns": 0.0
              },
              {
                "repeat": 0,
                "step": 1,
                "kernels": 177,
                "simulated_kernels": 1,
                "measured_gpu_ns": 564801,
                "simulated_coverage_gpu_ns": 4768.0,
                "unmodeled_gpu_ns": 0,
                "measured_kernel_overlap_ns": 0.0
              }
              /* 2 more items trimmed */
            ]
          },
          "request_assumption": "one request per repeat; queued (or arrival) + pre/steps/inter-step gaps/post",
          "gap_model": "measured",
          "gap_fit": {}
        },
        "per_kind": {
          "prefill": {
            "steps": 1,
            "wall_ns_mean": 6507260.692579505,
            "gpu_busy_ns_mean": 544095.6925795053
          },
          "decode": {
            "steps": 3,
            "wall_ns_mean": 4791870.325088339,
            "gpu_busy_ns_mean": 575732.9917550059,
            "gap_before_ns": {
              "n": 3,
              "mean": 1000000,
              "p50": 1000000,
              "p95": 1180000.0
            }
          }
        }
      }
    },
    "compare": {
      "steps": [
        {
          "repeat": 0,
          "step": 0,
          "kind": "prefill",
          "wall_ns": {
            "real": 6500000,
            "simulated": 6507260.692579505,
            "error": 7260.692579505034,
            "error_pct": 0.1117029627616159
          },
          "gpu_busy_ns": {
            "real": 536835,
            "simulated": 544095.6925795053,
            "error": 7260.692579505267,
            "error_pct": 1.3524998518176472
          }
        },
        {
          "repeat": 0,
          "step": 1,
          "kind": "decode",
          "wall_ns": {
            "real": 4790000,
            "simulated": 4825610.975265018,
            "error": 35610.97526501771,
            "error_pct": 0.7434441600212466
          },
          "gpu_busy_ns": {
            "real": 564801,
            "simulated": 600411.9752650177,
            "error": 35610.97526501771,
            "error_pct": 6.3050481966246
          }
        }
        /* 2 more items trimmed */
      ],
      "requests": [
        {
          "repeat": 0,
          "ttft_ms": {
            "real": 6.7,
            "simulated": 6.707260692579508,
            "error": 0.00726069257950801,
            "error_pct": 0.10836854596280611
          },
          "itl_ms_mean": {
            "real": 5.78,
            "simulated": 5.79187032508834,
            "error": 0.011870325088339406,
            "error_pct": 0.20536894616504162
          },
          "itl_ms_p50": {
            "real": 5.65,
            "simulated": 5.65,
            "error": 0.0,
            "error_pct": 0.0
          },
          "e2e_ms": {
            "real": 24.04,
            "simulated": 24.082871667844532,
            "error": 0.042871667844533334,
            "error_pct": 0.1783347248108708
          }
        }
      ],
      "summary": {
        "wall_ns": {
          "count": 4,
          "mean_error": 10717.916961130686,
          "mean_abs_error": 10717.916961130686,
          "p50_abs_error": 3630.346289752517,
          "p95_abs_error": 31358.4328621908,
          "mean_error_pct": 0.2137867806957156,
          "mean_abs_error_pct": 0.2137867806957156,
          "p50_abs_error_pct": 0.05585148138080795,
          "p95_abs_error_pct": 0.6486829804323018
        },
        "gpu_busy_ns": {
          "count": 4,
          "mean_error": 10717.916961130744,
          "mean_abs_error": 10717.916961130744,
          "p50_abs_error": 3630.3462897526333,
          "p95_abs_error": 31358.432862190835,
          "mean_error_pct": 1.914387012110562,
          "mean_abs_error_pct": 1.914387012110562,
          "p50_abs_error_pct": 0.6762499259088236,
          "p95_abs_error_pct": 5.562165944903556
        }
      }
    }
  },
  {
    "sim_id": "sim-e8f63b6b3bed",
    "gpu_config": "SM86_RTX3070",
    "replica": 1,
    "simulated": {
      "meta": {
        "run_name": "offline-h100-derived-fixture",
        "model": "facebook/opt-125m",
        "prompt": "Hello, my name is",
        "max_tokens": 4,
        "repeats": 1,
        "exec_mode": "eager",
        "gpu_name": "NVIDIA H100 80GB HBM3",
        "sm_clock_mhz_median": 1980,
        "mem_clock_mhz": 3106,
        "vllm_version": "fixture-unknown",
        "layerwise_nvtx": false,
        "fixture_provenance": "Actual nsys kernel names/dimensions/durations and simulator cycles; synthetic phase assignment, baseline/profiled walls, queue/API values and rescaled gaps. Not a real corrected_timeline capture. Synthetic consistent absolute step/request timestamps and baseline inter-step gaps added for gap-aware replay."
      },
      "steps": [
        {
          "step": -1,
          "repeat": -1,
          "warmup": true,
          "kind": "prefill",
          "num_reqs": 1,
          "num_scheduled_tokens": 6,
          "baseline_wall_ns": 9000000,
          "profiled_wall_ns": null,
          "gpu_busy_ns": null,
          "cuda_api_ns": null,
          "host_ns": null,
          "kernels": [],
          "baseline_wall_start_ns": 100000000,
          "baseline_wall_end_ns": 109000000,
          "baseline_gap_before_ns": null,
          "profiled_wall_start_ns": null,
          "profiled_wall_end_ns": null,
          "profiled_gap_before_ns": null
        },
        {
          "step": 0,
          "repeat": 0,
          "warmup": false,
          "kind": "prefill",
          "num_reqs": 1,
          "num_scheduled_tokens": 6,
          "baseline_wall_ns": 6507260.692579505,
          "profiled_wall_ns": 6507260.692579505,
          "gpu_busy_ns": 544095.6925795053,
          "cuda_api_ns": null,
          "host_ns": 5963165,
          "kernels": [
            {
              "seq": 536,
              "name": "_ZN8cublasLt19splitKreduce_kernelILi32ELi16Eif6__halffS1_Lb0EfS1_S1_Lb1ELb1ELb0ELb0EEEvNS_18cublasSplitKParamsIT4_EEPKT2_PKT8_PT7_PT3_PKS3_SG_PKT9_S7_PSH_PvlPS3_PiSM_SM_SG_SG_SG_SG_SG_",
              "grid": "24x1x1",
              "block": "32x16x1",
              "duration_ns": 2016,
              "start_offset_ns": 108088.07402640952
            },
            {
              "seq": 537,
              "name": "_ZN2at6native29vectorized_elementwise_kernelILi8ENS0_15CUDAFunctor_addIN3c104HalfEEESt5arrayIPcLm3EEEEviT0_T1_",
              "grid": "5x1x1",
              "block": "128x1x1",
              "duration_ns": 1376,
              "start_offset_ns": 150410.8374180178
            }
            /* 150 more items trimmed */
          ],
          "baseline_wall_start_ns": 1000200000,
          "baseline_wall_end_ns": 1006707260.6925795,
          "baseline_gap_before_ns": null,
          "profiled_wall_start_ns": 1000200000,
          "profiled_wall_end_ns": 1006707260.6925795,
          "profiled_gap_before_ns": null
        }
        /* 3 more items trimmed */
      ],
      "requests": [
        {
          "repeat": -1,
          "warmup": true,
          "queued_ms": 0,
          "ttft_ms": 9,
          "itl_ms_mean": null,
          "itl_ms_p50": null,
          "e2e_ms": 9,
          "num_output_tokens": 1
        },
        {
          "repeat": 0,
          "warmup": false,
          "queued_ms": 0.2,
          "ttft_ms": 6.707260692579508,
          "itl_ms_mean": 5.79187032508834,
          "itl_ms_p50": 5.65,
          "e2e_ms": 24.082871667844532,
          "num_output_tokens": 4,
          "arrival_ns": 1000000000,
          "pre_first_step_ns": 200000,
          "post_last_step_ns": 0,
          "first_step_start_ns": 1000200000,
          "last_step_end_ns": 1024082871.6678445,
          "first_token_ns": 1006707260.6925795,
          "last_token_ns": 1024082871.6678445
        }
      ],
      "summary": {
        "simulation": {
          "host_model": "scaled-gaps",
          "host_fit": {},
          "coverage": {
            "kernels": 683,
            "simulated_kernels": 6,
            "kernel_fraction": 0.008784773060029283,
            "gpu_time_fraction": 0.011172026136868988,
            "measured_gpu_ns": 2228423.0,
            "covered_measured_gpu_ns": 24896.0,
            "unmodeled_gpu_ns": 0.0,
            "steps": [
              {
                "repeat": 0,
                "step": 0,
                "kernels": 152,
                "simulated_kernels": 5,
                "measured_gpu_ns": 536835,
                "simulated_coverage_gpu_ns": 20128.0,
                "unmodeled_gpu_ns": 0,
                "measured_kernel_overlap_ns": 0.0
              },
              {
                "repeat": 0,
                "step": 1,
                "kernels": 177,
                "simulated_kernels": 1,
                "measured_gpu_ns": 564801,
                "simulated_coverage_gpu_ns": 4768.0,
                "unmodeled_gpu_ns": 0,
                "measured_kernel_overlap_ns": 0.0
              }
              /* 2 more items trimmed */
            ]
          },
          "request_assumption": "one request per repeat; queued (or arrival) + pre/steps/inter-step gaps/post",
          "gap_model": "measured",
          "gap_fit": {}
        },
        "per_kind": {
          "prefill": {
            "steps": 1,
            "wall_ns_mean": 6507260.692579505,
            "gpu_busy_ns_mean": 544095.6925795053
          },
          "decode": {
            "steps": 3,
            "wall_ns_mean": 4791870.325088339,
            "gpu_busy_ns_mean": 575732.9917550059,
            "gap_before_ns": {
              "n": 3,
              "mean": 1000000,
              "p50": 1000000,
              "p95": 1180000.0
            }
          }
        }
      }
    },
    "compare": {
      "steps": [
        {
          "repeat": 0,
          "step": 0,
          "kind": "prefill",
          "wall_ns": {
            "real": 6500000,
            "simulated": 6507260.692579505,
            "error": 7260.692579505034,
            "error_pct": 0.1117029627616159
          },
          "gpu_busy_ns": {
            "real": 536835,
            "simulated": 544095.6925795053,
            "error": 7260.692579505267,
            "error_pct": 1.3524998518176472
          }
        },
        {
          "repeat": 0,
          "step": 1,
          "kind": "decode",
          "wall_ns": {
            "real": 4790000,
            "simulated": 4825610.975265018,
            "error": 35610.97526501771,
            "error_pct": 0.7434441600212466
          },
          "gpu_busy_ns": {
            "real": 564801,
            "simulated": 600411.9752650177,
            "error": 35610.97526501771,
            "error_pct": 6.3050481966246
          }
        }
        /* 2 more items trimmed */
      ],
      "requests": [
        {
          "repeat": 0,
          "ttft_ms": {
            "real": 6.7,
            "simulated": 6.707260692579508,
            "error": 0.00726069257950801,
            "error_pct": 0.10836854596280611
          },
          "itl_ms_mean": {
            "real": 5.78,
            "simulated": 5.79187032508834,
            "error": 0.011870325088339406,
            "error_pct": 0.20536894616504162
          },
          "itl_ms_p50": {
            "real": 5.65,
            "simulated": 5.65,
            "error": 0.0,
            "error_pct": 0.0
          },
          "e2e_ms": {
            "real": 24.04,
            "simulated": 24.082871667844532,
            "error": 0.042871667844533334,
            "error_pct": 0.1783347248108708
          }
        }
      ],
      "summary": {
        "wall_ns": {
          "count": 4,
          "mean_error": 10717.916961130686,
          "mean_abs_error": 10717.916961130686,
          "p50_abs_error": 3630.346289752517,
          "p95_abs_error": 31358.4328621908,
          "mean_error_pct": 0.2137867806957156,
          "mean_abs_error_pct": 0.2137867806957156,
          "p50_abs_error_pct": 0.05585148138080795,
          "p95_abs_error_pct": 0.6486829804323018
        },
        "gpu_busy_ns": {
          "count": 4,
          "mean_error": 10717.916961130744,
          "mean_abs_error": 10717.916961130744,
          "p50_abs_error": 3630.3462897526333,
          "p95_abs_error": 31358.432862190835,
          "mean_error_pct": 1.914387012110562,
          "mean_abs_error_pct": 1.914387012110562,
          "p50_abs_error_pct": 0.6762499259088236,
          "p95_abs_error_pct": 5.562165944903556
        }
      }
    }
  }
]
```

## Download run results

`GET /api/runs/{run_id}/results` (bearer token)

Returns everything about a run as one JSON download, with `Content-Disposition: attachment; filename="<run_id>-results.json"`. Each sim's `kernels` carry the full `stats` map, every `key = value` line Accel-Sim printed for that kernel, which is about 30 KB per kernel.

### Path parameters

| name | type | description |
|---|---|---|
| `run_id` | `str` | The run's `id`, such as `run-9f56e171af08`. |

### Returns

`200` with a [`RunResults`](types.html#runresults).

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |
| [`404`](errors.html#e404) | No run has this id. |

### Example

```bash
curl -OJ "$CASCADE_URL/api/runs/run-210d75c008c4/results" \
  -H "Authorization: Bearer $CASCADE_TOKEN"
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.get(f"{os.environ['CASCADE_URL']}/api/runs/run-210d75c008c4/results", headers=headers)
resp.raise_for_status()
open("run-210d75c008c4-results.json", "wb").write(resp.content)
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/runs/run-210d75c008c4/results`, {
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}` },
});
const data = await resp.json();
```

Response · 200 OK · application/json:

```json
{
  "run": {
    "id": "run-210d75c008c4",
    "trace_id": "demo",
    "created_at": "2026-11-08T08:59:03.024065Z",
    "state": "succeeded",
    "entries": [
      {
        "gpu_config": "SM86_RTX3070",
        "count": 2
      }
    ],
    "sim_counts": {
      "succeeded": 2
    },
    "sample": false
  },
  "trace": {
    "id": "demo",
    "state": "ready",
    "error": null,
    "created_at": "2026-11-08T08:59:02.587743Z",
    "updated_at": "2026-11-08T08:59:02.592073Z",
    "model": null,
    "layers": [],
    "source_gpu": null,
    "trace_gpu_arch": "sm_86",
    "suggested_sim_config": "SM86_RTX3070",
    "kernel_count": 20,
    "bytes": 16890250,
    "has_real_timing": false,
    "has_step_timeline": true,
    "sample": false,
    "metadata": {
      "status": "ok",
      "files": [
        {
          "key": "accelsim-traces/demo/configs/config_hopper_islip.icnt",
          "bytes": 1352
        },
        {
          "key": "accelsim-traces/demo/configs/gpgpusim.config",
          "bytes": 6028
        }
        /* 25 more items trimmed */
      ],
      "trace_gpu_arch": "sm_86",
      "suggested_sim_config": "SM86_RTX3070"
    },
    "kernels": []
  },
  "sims": [
    {
      "sim": {
        "id": "sim-1047ddbdf5b1",
        "run_id": "run-210d75c008c4",
        "gpu_config": "SM86_RTX3070",
        "replica": 0,
        "state": "succeeded",
        "error": null,
        "launcher_ref": "64418",
        "created_at": "2026-11-08T08:59:03.024065Z",
        "started_at": "2026-11-08T08:59:03.224424Z",
        "finished_at": "2026-11-08T08:59:06.130355Z",
        "last_heartbeat_at": "2026-11-08T08:59:06.130355Z",
        "exit_code": 0,
        "kernels_done": 6,
        "kernels_total": 20,
        "current_kernel": "_Z10dot_kernelIdEvPKT_S2_PS0_i",
        "latest": {
          "gpu_tot_sim_cycle": 76713.0,
          "gpu_tot_sim_insn": 26560256.0,
          "gpu_tot_ipc": 346.2289,
          "sim_rate_ips": 510774.0
        },
        "log_tail": [
          "Reply_Network_in_buffer_full_per_cycle =       0.0000",
          "Reply_Network_in_buffer_avg_util =       0.0000",
          "Reply_Network_out_buffer_full_per_cycle =       0.0000"
          /* 17 more items trimmed */
        ]
      },
      "trace_id": "demo",
      "gpu_config": {
        "name": "SM86_RTX3070",
        "arch": "sm_86",
        "core_clock_mhz": 1132.0
      },
      "kernels": [
        {
          "sim_id": "sim-1047ddbdf5b1",
          "kernel_index": 0,
          "kernel_name": "_Z11init_kernelIdEvPT_S1_S1_S0_S0_S0_",
          "kernel_launch_uid": 1,
          "gpu_sim_cycle": 6174,
          "gpu_sim_insn": 19456,
          "gpu_ipc": 3.1513,
          "sim_time_ns": 5454.063604240282,
          "stats": {
            "kernel_name": "_Z11init_kernelIdEvPT_S1_S1_S0_S0_S0_",
            "kernel_launch_uid": 1.0,
            "gpu_sim_cycle": 6174.0,
            "gpu_sim_insn": 19456.0
            /* 945 more keys trimmed */
          }
        },
        {
          "sim_id": "sim-1047ddbdf5b1",
          "kernel_index": 1,
          "kernel_name": "_Z11copy_kernelIdEvPKT_PS0_",
          "kernel_launch_uid": 2,
          "gpu_sim_cycle": 5923,
          "gpu_sim_insn": 11264,
          "gpu_ipc": 1.9017,
          "sim_time_ns": 5232.332155477032,
          "stats": {
            "kernel_name": "_Z11copy_kernelIdEvPKT_PS0_",
            "kernel_launch_uid": 2.0,
            "gpu_sim_cycle": 5923.0,
            "gpu_sim_insn": 11264.0
            /* 950 more keys trimmed */
          }
        }
        /* 4 more items trimmed */
      ],
      "totals": {
        "gpu_tot_sim_cycle": 76713.0,
        "gpu_tot_sim_insn": 26560256.0,
        "gpu_tot_ipc": 346.2289,
        "sim_rate_ips": 510774.0
      }
    },
    {
      "sim": {
        "id": "sim-e8f63b6b3bed",
        "run_id": "run-210d75c008c4",
        "gpu_config": "SM86_RTX3070",
        "replica": 1,
        "state": "succeeded",
        "error": null,
        "launcher_ref": "64419",
        "created_at": "2026-11-08T08:59:03.024065Z",
        "started_at": "2026-11-08T08:59:03.224173Z",
        "finished_at": "2026-11-08T08:59:06.164567Z",
        "last_heartbeat_at": "2026-11-08T08:59:06.164567Z",
        "exit_code": 0,
        "kernels_done": 6,
        "kernels_total": 20,
        "current_kernel": "_Z10dot_kernelIdEvPKT_S2_PS0_i",
        "latest": {
          "gpu_tot_sim_cycle": 76713.0,
          "gpu_tot_sim_insn": 26560256.0,
          "gpu_tot_ipc": 346.2289,
          "sim_rate_ips": 510774.0
        },
        "log_tail": [
          "Reply_Network_in_buffer_full_per_cycle =       0.0000",
          "Reply_Network_in_buffer_avg_util =       0.0000",
          "Reply_Network_out_buffer_full_per_cycle =       0.0000"
          /* 17 more items trimmed */
        ]
      },
      "trace_id": "demo",
      "gpu_config": {
        "name": "SM86_RTX3070",
        "arch": "sm_86",
        "core_clock_mhz": 1132.0
      },
      "kernels": [
        {
          "sim_id": "sim-e8f63b6b3bed",
          "kernel_index": 0,
          "kernel_name": "_Z11init_kernelIdEvPT_S1_S1_S0_S0_S0_",
          "kernel_launch_uid": 1,
          "gpu_sim_cycle": 6174,
          "gpu_sim_insn": 19456,
          "gpu_ipc": 3.1513,
          "sim_time_ns": 5454.063604240282,
          "stats": {
            "kernel_name": "_Z11init_kernelIdEvPT_S1_S1_S0_S0_S0_",
            "kernel_launch_uid": 1.0,
            "gpu_sim_cycle": 6174.0,
            "gpu_sim_insn": 19456.0
            /* 945 more keys trimmed */
          }
        },
        {
          "sim_id": "sim-e8f63b6b3bed",
          "kernel_index": 1,
          "kernel_name": "_Z11copy_kernelIdEvPKT_PS0_",
          "kernel_launch_uid": 2,
          "gpu_sim_cycle": 5923,
          "gpu_sim_insn": 11264,
          "gpu_ipc": 1.9017,
          "sim_time_ns": 5232.332155477032,
          "stats": {
            "kernel_name": "_Z11copy_kernelIdEvPKT_PS0_",
            "kernel_launch_uid": 2.0,
            "gpu_sim_cycle": 5923.0,
            "gpu_sim_insn": 11264.0
            /* 950 more keys trimmed */
          }
        }
        /* 4 more items trimmed */
      ],
      "totals": {
        "gpu_tot_sim_cycle": 76713.0,
        "gpu_tot_sim_insn": 26560256.0,
        "gpu_tot_ipc": 346.2289,
        "sim_rate_ips": 510774.0
      }
    }
  ],
  "comparison": {
    "has_real_timing": false,
    "configs": [
      "SM86_RTX3070"
    ],
    "rows": [
      {
        "kernel_index": 0,
        "kernel_name": "_Z11init_kernelIdEvPT_S1_S1_S0_S0_S0_",
        "real_ns": null,
        "sim_ns": {
          "SM86_RTX3070": 5454.063604240282
        }
      },
      {
        "kernel_index": 1,
        "kernel_name": "_Z11copy_kernelIdEvPKT_PS0_",
        "real_ns": null,
        "sim_ns": {
          "SM86_RTX3070": 5232.332155477032
        }
      }
      /* 4 more items trimmed */
    ]
  }
}
```

## Download sim results

`GET /api/sims/{sim_id}/results` (bearer token)

Returns one sim's results as a JSON download named `<sim_id>-results.json`. `totals` is the sim's latest [`SimMetrics`](types.html#simmetrics).

### Path parameters

| name | type | description |
|---|---|---|
| `sim_id` | `str` | The sim's `id`, such as `sim-6b4c91e71437`. Find it in `RunDetail.sims`. |

### Returns

`200` with a [`SimResults`](types.html#simresults).

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |
| [`404`](errors.html#e404) | No sim has this id. |

### Example

```bash
curl -OJ "$CASCADE_URL/api/sims/sim-1047ddbdf5b1/results" \
  -H "Authorization: Bearer $CASCADE_TOKEN"
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.get(f"{os.environ['CASCADE_URL']}/api/sims/sim-1047ddbdf5b1/results", headers=headers)
resp.raise_for_status()
open("sim-1047ddbdf5b1-results.json", "wb").write(resp.content)
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/sims/sim-1047ddbdf5b1/results`, {
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}` },
});
const data = await resp.json();
```

Response · 200 OK · application/json:

```json
{
  "sim": {
    "id": "sim-1047ddbdf5b1",
    "run_id": "run-210d75c008c4",
    "gpu_config": "SM86_RTX3070",
    "replica": 0,
    "state": "succeeded",
    "error": null,
    "launcher_ref": "64418",
    "created_at": "2026-11-08T08:59:03.024065Z",
    "started_at": "2026-11-08T08:59:03.224424Z",
    "finished_at": "2026-11-08T08:59:06.130355Z",
    "last_heartbeat_at": "2026-11-08T08:59:06.130355Z",
    "exit_code": 0,
    "kernels_done": 6,
    "kernels_total": 20,
    "current_kernel": "_Z10dot_kernelIdEvPKT_S2_PS0_i",
    "latest": {
      "gpu_tot_sim_cycle": 76713.0,
      "gpu_tot_sim_insn": 26560256.0,
      "gpu_tot_ipc": 346.2289,
      "sim_rate_ips": 510774.0
    },
    "log_tail": [
      "Reply_Network_in_buffer_full_per_cycle =       0.0000",
      "Reply_Network_in_buffer_avg_util =       0.0000",
      "Reply_Network_out_buffer_full_per_cycle =       0.0000"
      /* 17 more items trimmed */
    ]
  },
  "trace_id": "demo",
  "gpu_config": {
    "name": "SM86_RTX3070",
    "arch": "sm_86",
    "core_clock_mhz": 1132.0
  },
  "kernels": [
    {
      "sim_id": "sim-1047ddbdf5b1",
      "kernel_index": 0,
      "kernel_name": "_Z11init_kernelIdEvPT_S1_S1_S0_S0_S0_",
      "kernel_launch_uid": 1,
      "gpu_sim_cycle": 6174,
      "gpu_sim_insn": 19456,
      "gpu_ipc": 3.1513,
      "sim_time_ns": 5454.063604240282,
      "stats": {
        "kernel_name": "_Z11init_kernelIdEvPT_S1_S1_S0_S0_S0_",
        "kernel_launch_uid": 1.0,
        "gpu_sim_cycle": 6174.0,
        "gpu_sim_insn": 19456.0
        /* 945 more keys trimmed */
      }
    },
    {
      "sim_id": "sim-1047ddbdf5b1",
      "kernel_index": 1,
      "kernel_name": "_Z11copy_kernelIdEvPKT_PS0_",
      "kernel_launch_uid": 2,
      "gpu_sim_cycle": 5923,
      "gpu_sim_insn": 11264,
      "gpu_ipc": 1.9017,
      "sim_time_ns": 5232.332155477032,
      "stats": {
        "kernel_name": "_Z11copy_kernelIdEvPKT_PS0_",
        "kernel_launch_uid": 2.0,
        "gpu_sim_cycle": 5923.0,
        "gpu_sim_insn": 11264.0
        /* 950 more keys trimmed */
      }
    }
    /* 4 more items trimmed */
  ],
  "totals": {
    "gpu_tot_sim_cycle": 76713.0,
    "gpu_tot_sim_insn": 26560256.0,
    "gpu_tot_ipc": 346.2289,
    "sim_rate_ips": 510774.0
  }
}
```

## Download a sim's log

`GET /api/sims/{sim_id}/log` (bearer token)

Streams every line the simulator sent, in `seq` order, as `text/plain` named `<sim_id>.log`.

### Path parameters

| name | type | description |
|---|---|---|
| `sim_id` | `str` | The sim's `id`, such as `sim-6b4c91e71437`. Find it in `RunDetail.sims`. |

### Returns

`200` with the log as plain text.

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |
| [`404`](errors.html#e404) | No sim has this id. |

### Example

```bash
curl -OJ "$CASCADE_URL/api/sims/sim-1047ddbdf5b1/log" \
  -H "Authorization: Bearer $CASCADE_TOKEN"
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.get(f"{os.environ['CASCADE_URL']}/api/sims/sim-1047ddbdf5b1/log", headers=headers)
resp.raise_for_status()
open("sim-1047ddbdf5b1.log", "wb").write(resp.content)
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/sims/sim-1047ddbdf5b1/log`, {
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}` },
});
const text = await resp.text();
```

Response · 200 OK · text/plain:

```
GPGPU-Sim version 4.2.0 (build gpgpu-sim_git-commit-13c67115070dc2f0876254a790d0238073ca364a-modified_590.0) configured with AccelWattch.

----------------------------------------------------------------------------
INFO - If you only care about PTX execution, ignore this message. GPGPU-Sim supports PTX execution in modern CUDA.
If you want to run PTXPLUS (sm_1x SASS) with a modern card configuration - set the envronment variable
$PTXAS_CUDA_INSTALL_PATH to point a CUDA version compabible with your card configurations (i.e. 8+ for PASCAL, 9+ for VOLTA etc..)
For example: "export $PTXAS_CUDA_INSTALL_PATH=/usr/local/cuda-9.1"

The following text describes why:
If you are using PTXPLUS, only sm_1x is supported and it requires that the app and simulator binaries are compiled in CUDA 4.2 or less.
The simulator requires it since CUDA headers desribe struct sizes in the exec which change from gen to gen.
The apps require 4.2 because new versions of CUDA tools have dropped parsing support for generating sm_1x
/* 10679 more lines trimmed */
```

## List a run's output files

`GET /api/runs/{run_id}/files` (bearer token)

Lists the files every sim in the run uploaded, ordered by config, replica, then name. Each [`RunFile`](types.html#runfile) names its sim, so one call covers the run. `cascade worker` uploads a sim's files just before its final heartbeat, so the list grows as sims finish. Fetch one with [Download an output file](api-runs.html#get-sim-file).

### Path parameters

| name | type | description |
|---|---|---|
| `run_id` | `str` | The run's `id`, such as `run-9f56e171af08`. |

### Returns

`200` with a list of [`RunFile`](types.html#runfile).

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |
| [`404`](errors.html#e404) | No run has this id. |

### Example

```bash
curl "$CASCADE_URL/api/runs/run-210d75c008c4/files" \
  -H "Authorization: Bearer $CASCADE_TOKEN"
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.get(f"{os.environ['CASCADE_URL']}/api/runs/run-210d75c008c4/files", headers=headers)
resp.raise_for_status()
print(resp.json())
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/runs/run-210d75c008c4/files`, {
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}` },
});
console.log(await resp.json());
```

Response · 200 OK · application/json:

```json
[
  {
    "name": "accelsim.log.gz",
    "size": 37105,
    "content_type": "application/gzip",
    "created_at": "2026-11-08T08:59:06.128163Z",
    "sim_id": "sim-1047ddbdf5b1",
    "gpu_config": "SM86_RTX3070",
    "replica": 0
  },
  {
    "name": "exit-status.txt",
    "size": 2,
    "content_type": "text/plain",
    "created_at": "2026-11-08T08:59:06.120587Z",
    "sim_id": "sim-1047ddbdf5b1",
    "gpu_config": "SM86_RTX3070",
    "replica": 0
  },
  {
    "name": "gpgpu_inst_stats.txt",
    "size": 133,
    "content_type": "text/plain",
    "created_at": "2026-11-08T08:59:06.121677Z",
    "sim_id": "sim-1047ddbdf5b1",
    "gpu_config": "SM86_RTX3070",
    "replica": 0
  },
  {
    "name": "kernel-stats.csv",
    "size": 3829,
    "content_type": "text/csv",
    "created_at": "2026-11-08T08:59:06.122701Z",
    "sim_id": "sim-1047ddbdf5b1",
    "gpu_config": "SM86_RTX3070",
    "replica": 0
  }
  /* 8 more items trimmed */
]
```

## Download a Perfetto trace

`GET /api/runs/{run_id}/perfetto` (bearer token)

Builds a Chrome trace JSON from the run's parsed kernel stats and returns it with `Content-Disposition: attachment; filename="<run_id>-perfetto.json"`. Each config is a process, each replica is a thread in it, and each finished kernel is one slice. The app builds it from the stats it parsed, so every run has one, whether or not the simulator wrote its own. [Reading results](reading-results.html#the-perfetto-trace) describes the layout.

### Path parameters

| name | type | description |
|---|---|---|
| `run_id` | `str` | The run's `id`, such as `run-9f56e171af08`. |

### Returns

`200` with a Chrome trace JSON object: `traceEvents`, `displayTimeUnit` and `otherData`.

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |
| [`404`](errors.html#e404) | No run has this id. |

### Example

```bash
curl -OJ "$CASCADE_URL/api/runs/run-210d75c008c4/perfetto" \
  -H "Authorization: Bearer $CASCADE_TOKEN"
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.get(f"{os.environ['CASCADE_URL']}/api/runs/run-210d75c008c4/perfetto", headers=headers)
resp.raise_for_status()
open("run-210d75c008c4-perfetto.json", "wb").write(resp.content)
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/runs/run-210d75c008c4/perfetto`, {
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}` },
});
const data = await resp.json();
```

Response · 200 OK · application/json:

```json
{
  "traceEvents": [
    {
      "ph": "M",
      "pid": 1,
      "name": "process_name",
      "args": {
        "name": "SM86_RTX3070 · Accel-Sim simulated GPU"
      }
    },
    {
      "ph": "M",
      "pid": 1,
      "name": "process_sort_index",
      "args": {
        "sort_index": 1
      }
    },
    {
      "ph": "M",
      "pid": 1,
      "name": "thread_name",
      "args": {
        "name": "replica 0 · sim-1047ddbdf5b1"
      },
      "tid": 1
    },
    {
      "ph": "M",
      "pid": 1,
      "name": "thread_sort_index",
      "args": {
        "sort_index": 1
      },
      "tid": 1
    },
    {
      "ph": "X",
      "pid": 1,
      "tid": 1,
      "cat": "kernel",
      "name": "k1 init_kernel",
      "ts": 0.0,
      "dur": 5.454,
      "args": {
        "kernel_index": 0,
        "uid": 1,
        "kernel_name": "_Z11init_kernelIdEvPT_S1_S1_S0_S0_S0_",
        "gpu_sim_cycle": 6174,
        "gpu_sim_insn": 19456,
        "ipc": 3.1513,
        "sim_time_ns": 5454.063604240282,
        "start_cycle": 0,
        "end_cycle": 6174
      }
    },
    {
      "ph": "X",
      "pid": 1,
      "tid": 1,
      "cat": "kernel",
      "name": "k2 copy_kernel",
      "ts": 5.454,
      "dur": 5.232,
      "args": {
        "kernel_index": 1,
        "uid": 2,
        "kernel_name": "_Z11copy_kernelIdEvPKT_PS0_",
        "gpu_sim_cycle": 5923,
        "gpu_sim_insn": 11264,
        "ipc": 1.9017,
        "sim_time_ns": 5232.332155477032,
        "start_cycle": 6174,
        "end_cycle": 12097
      }
    }
    /* 12 more items trimmed */
  ],
  "displayTimeUnit": "ns",
  "otherData": {
    "run_id": "run-210d75c008c4",
    "trace_id": "demo",
    "time_axis": "simulated time, microseconds; each sim's kernels laid end to end",
    "slice_duration": "sim_time_ns = gpu_sim_cycle / core clock (MHz) * 1000",
    "kernel_scope": "sequential kernels, each including the configured launch latency; excludes host time and memcpy"
  }
}
```

## List a sim's output files

`GET /api/sims/{sim_id}/files` (bearer token)

Lists one sim's uploaded files, sorted by name. The list is empty until the simulator uploads something.

### Path parameters

| name | type | description |
|---|---|---|
| `sim_id` | `str` | The sim's `id`, such as `sim-6b4c91e71437`. Find it in `RunDetail.sims`. |

### Returns

`200` with a list of [`SimFile`](types.html#simfile).

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |
| [`404`](errors.html#e404) | No sim has this id. |

### Example

```bash
curl "$CASCADE_URL/api/sims/sim-1047ddbdf5b1/files" \
  -H "Authorization: Bearer $CASCADE_TOKEN"
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.get(f"{os.environ['CASCADE_URL']}/api/sims/sim-1047ddbdf5b1/files", headers=headers)
resp.raise_for_status()
print(resp.json())
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/sims/sim-1047ddbdf5b1/files`, {
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}` },
});
console.log(await resp.json());
```

Response · 200 OK · application/json:

```json
[
  {
    "name": "accelsim.log.gz",
    "size": 37105,
    "content_type": "application/gzip",
    "created_at": "2026-11-08T08:59:06.128163Z"
  },
  {
    "name": "exit-status.txt",
    "size": 2,
    "content_type": "text/plain",
    "created_at": "2026-11-08T08:59:06.120587Z"
  },
  {
    "name": "gpgpu_inst_stats.txt",
    "size": 133,
    "content_type": "text/plain",
    "created_at": "2026-11-08T08:59:06.121677Z"
  },
  {
    "name": "kernel-stats.csv",
    "size": 3829,
    "content_type": "text/csv",
    "created_at": "2026-11-08T08:59:06.122701Z"
  },
  {
    "name": "opt125m_H100_perfetto.json.gz",
    "size": 104763,
    "content_type": "application/gzip",
    "created_at": "2026-11-08T08:59:06.123623Z"
  },
  {
    "name": "perf_counter.csv.gz",
    "size": 474373,
    "content_type": "application/gzip",
    "created_at": "2026-11-08T08:59:06.124946Z"
  }
]
```

## Download an output file

`GET /api/sims/{sim_id}/files/{name}` (bearer token)

Returns the file as a download named `name`, with the file's `content_type`. A `.gz` file comes back as `application/gzip` with no `Content-Encoding`, so it stays compressed.

### Path parameters

| name | type | description |
|---|---|---|
| `sim_id` | `str` | The sim's `id`, such as `sim-6b4c91e71437`. Find it in `RunDetail.sims`. |
| `name` | `str` | The file's name, such as `kernel-stats.csv`. Letters, digits, `.`, `_` and `-`, 1 to 128 characters, and not `.` or `..`. |

### Returns

`200` with the file's bytes.

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |
| [`404`](errors.html#e404) | No sim has this id, or the sim has no file with this name. |

### Example

```bash
curl -OJ "$CASCADE_URL/api/sims/sim-1047ddbdf5b1/files/kernel-stats.csv" \
  -H "Authorization: Bearer $CASCADE_TOKEN"
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.get(f"{os.environ['CASCADE_URL']}/api/sims/sim-1047ddbdf5b1/files/kernel-stats.csv", headers=headers)
resp.raise_for_status()
open("kernel-stats.csv", "wb").write(resp.content)
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/sims/sim-1047ddbdf5b1/files/kernel-stats.csv`, {
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}` },
});
const text = await resp.text();
```

Response · 200 OK · text/csv:

```
uid,name,gpu_sim_cycle,gpu_sim_insn,ipc
1,nvjet_sm90_hsh_64x8_64x16_4x1_v_bz_bias_TNT,10120,4765148,470.86442687747035
2,_ZN4vllm30reshape_and_cache_flash_kernelIttLNS_18Fp8KVCacheDataTypeE0EEEvPKT_S4_PT0_S6_PKlllllliiiPKfSA_i,7429,395584,53.24862027190739
3,_ZN7cutlass13device_kernelIN5flash20enable_sm90_or_laterINS1_16FlashAttnFwdSm90INS1_25CollectiveMainloopFwdSm90ILi2EN4cute5tupleIJNS5_1CILi1EEES8_S8_EEENS6_IJNS7_ILi64EEENS7_ILi192EEESA_EEELi64ENS_6half_tEfNS_4arch4Sm90ELb1ELb0ELb0ELb1ELb1ELb0ELb0ELb1ELb1ELb1ELb0ELb0ENS_10bfloat16_tELi1EEENS1_21CollectiveEpilogueFwdINS6_IJSA_SA_SB_EEES9_SD_SF_Li128ELb1ELb1ELb0ELb0ELi1EEENS1_36VarlenDynamicPersistentTileSchedulerILi64ELi192ELi128ELi128ELb0ELb1ELb1ELb1ELb0ELb1EEEEEEEEEvNT_6ParamsE,14142,5542420,391.9120350728327
4,_ZN2at6native29vectorized_elementwise_kernelILi4ENS0_11FillFunctorIiEESt5arrayIPcLm1EEEEviT0_T1_,5345,3078,0.5758652946679139
5,nvjet_sm90_hsh_64x8_64x16_4x1_v_bz_bias_TNT,8998,1588148,176.50011113580797
6,nvjet_sm90_hsh_64x8_64x16_4x1_v_bz_bias_TNT,8634,4762652,551.6159369932824
7,_ZN4vllm30reshape_and_cache_flash_kernelIttLNS_18Fp8KVCacheDataTypeE0EEEvPKT_S4_PT0_S6_PKlllllliiiPKfSA_i,7026,65760,9.359521776259607
8,_ZN7cutlass13device_kernelIN5flash20enable_sm90_or_laterINS1_16FlashAttnFwdSm90INS1_25CollectiveMainloopFwdSm90ILi2EN4cute5tupleIJNS5_1CILi1EEES8_S8_EEENS6_IJNS7_ILi64EEENS7_ILi192EEESA_EEELi64ENS_6half_tEfNS_4arch4Sm90ELb0ELb0ELb0ELb1ELb1ELb0ELb0ELb1ELb1ELb1ELb0ELb0ENS_10bfloat16_tELi1EEENS1_21CollectiveEpilogueFwdINS6_IJSA_SA_SB_EEES9_SD_SF_Li128ELb1ELb1ELb0ELb0ELi1EEENS1_36VarlenDynamicPersistentTileSchedulerILi64ELi192ELi128ELi128ELb0ELb1ELb1ELb0ELb0ELb1EEEEEEEEEvNT_6ParamsE,12302,5247952,426.59339944724434
9,_ZN2at6native29vectorized_elementwise_kernelILi4ENS0_11FillFunctorIiEESt5arrayIPcLm1EEEEviT0_T1_,5348,3078,0.5755422587883321
10,nvjet_sm90_hsh_64x8_64x16_4x1_v_bz_bias_TNT,8424,1588020,188.51139601139602
11,nvjet_sm90_hsh_64x8_64x16_4x1_v_bz_bias_TNT,8621,4766492,552.8931678459576
/* 9 more lines trimmed */
```

404 response, no such file · 404 Not Found · application/json:

```json
{
  "detail": "sim sim-1047ddbdf5b1 has no file 'nope.txt'"
}
```


# Live events (SSE)

Page: api-events.html

The run screen stays live through one server-sent event stream per run. You can subscribe to the same stream from a script.

## Stream run events

`GET /api/runs/{run_id}/events` (bearer token or ?token=)

Opens a `text/event-stream`. The first event is always a `snapshot`. After that the stream sends `sim`, `log` and `kernel` events as they happen, and a `: keepalive` comment after 15 seconds without one. The stream does not close when the run finishes.

### Path parameters

| name | type | description |
|---|---|---|
| `run_id` | `str` | The run's `id`, such as `run-9f56e171af08`. |

### Query parameters

| name | type | description |
|---|---|---|
| `token` optional | `str` | Your `CASCADE_TOKEN`. Use it when you can't set headers, as with `EventSource`. If present, it is checked instead of the header. |

### Returns

`200` with a `text/event-stream` of [`RunEvent`](types.html#runevent), one JSON object per `data:` line.

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |
| [`404`](errors.html#e404) | No run has this id. |

### Example

```bash
curl -N -H "Authorization: Bearer $CASCADE_TOKEN" \
  "$CASCADE_URL/api/runs/run-210d75c008c4/events"
```

```python
import json
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
url = f"{os.environ['CASCADE_URL']}/api/runs/run-210d75c008c4/events"
with httpx.stream("GET", url, headers=headers, timeout=None) as resp:
    for line in resp.iter_lines():
        if line.startswith("data: "):
            event = json.loads(line[6:])
            print(event["type"])
```

```javascript
// EventSource cannot send headers, so pass the token as a query parameter.
const url = `${CASCADE_URL}/api/runs/run-210d75c008c4/events?token=${encodeURIComponent(CASCADE_TOKEN)}`;
const events = new EventSource(url);
events.onmessage = (msg) => {
  const event = JSON.parse(msg.data);
  console.log(event.type);
};
```

## Wire format

Each event is one `data:` line holding one JSON object, followed by a blank line. The stream uses no `event:` or `id:` fields, so read the object's `type` to tell events apart.

```text
data: {"type":"snapshot","run":{"id":"run-9f56e171af08","trace_id":"demo", ...}, "kernels":[]}

data: {"type":"sim","sim":{"id":"sim-70f1ae5fc7c6","state":"running", ...}}

: keepalive
```

The response sets `Cache-Control: no-cache` and `X-Accel-Buffering: no`, so proxies pass events through as they happen.

## Events

These events come from the captured run. Each shows the first event of its kind that the stream sent.

### snapshot

Always first. It holds the whole [`RunDetail`](types.html#rundetail) as it is at connect time and every finished [`KernelStat`](types.html#kernelstat) so far, with each `stats` map emptied to keep the event small. Full stats come in live `kernel` events and in the results downloads. In this capture the stream connected right after the run was created, so both sims are still `launching` and no kernel has finished.

data: snapshot event:

```
{
  "type": "snapshot",
  "run": {
    "id": "run-210d75c008c4",
    "trace_id": "demo",
    "created_at": "2026-11-08T08:59:03.024065Z",
    "state": "active",
    "entries": [
      {
        "gpu_config": "SM86_RTX3070",
        "count": 2
      }
    ],
    "sim_counts": {
      "launching": 2
    },
    "sample": false,
    "trace": {
      "id": "demo",
      "state": "ready",
      "error": null,
      "created_at": "2026-11-08T08:59:02.587743Z",
      "updated_at": "2026-11-08T08:59:02.592073Z",
      "model": null,
      "layers": [],
      "source_gpu": null,
      "trace_gpu_arch": "sm_86",
      "suggested_sim_config": "SM86_RTX3070",
      "kernel_count": 20,
      "bytes": 16890250,
      "has_real_timing": false,
      "has_step_timeline": true,
      "sample": false
    },
    "sims": [
      {
        "id": "sim-1047ddbdf5b1",
        "run_id": "run-210d75c008c4",
        "gpu_config": "SM86_RTX3070",
        "replica": 0,
        "state": "launching",
        "error": null,
        "launcher_ref": "64418",
        "created_at": "2026-11-08T08:59:03.024065Z",
        "started_at": null,
        "finished_at": null,
        "last_heartbeat_at": null,
        "exit_code": null,
        "kernels_done": 0,
        "kernels_total": 20,
        "current_kernel": null,
        "latest": {
          "gpu_tot_sim_cycle": null,
          "gpu_tot_sim_insn": null,
          "gpu_tot_ipc": null,
          "sim_rate_ips": null
        },
        "log_tail": []
      },
      {
        "id": "sim-e8f63b6b3bed",
        "run_id": "run-210d75c008c4",
        "gpu_config": "SM86_RTX3070",
        "replica": 1,
        "state": "launching",
        "error": null,
        "launcher_ref": "64419",
        "created_at": "2026-11-08T08:59:03.024065Z",
        "started_at": null,
        "finished_at": null,
        "last_heartbeat_at": null,
        "exit_code": null,
        "kernels_done": 0,
        "kernels_total": 20,
        "current_kernel": null,
        "latest": {
          "gpu_tot_sim_cycle": null,
          "gpu_tot_sim_insn": null,
          "gpu_tot_ipc": null,
          "sim_rate_ips": null
        },
        "log_tail": []
      }
    ]
  },
  "kernels": []
}
```

### sim

A sim changed: its state, its heartbeat time, its progress or its latest metrics. It carries the whole [`SimSummary`](types.html#simsummary), so replace your copy of that sim.

data: sim event:

```
{
  "type": "sim",
  "sim": {
    "id": "sim-e8f63b6b3bed",
    "run_id": "run-210d75c008c4",
    "gpu_config": "SM86_RTX3070",
    "replica": 1,
    "state": "running",
    "error": null,
    "launcher_ref": "64419",
    "created_at": "2026-11-08T08:59:03.024065Z",
    "started_at": "2026-11-08T08:59:03.224173Z",
    "finished_at": null,
    "last_heartbeat_at": "2026-11-08T08:59:03.224173Z",
    "exit_code": null,
    "kernels_done": 0,
    "kernels_total": 20,
    "current_kernel": null,
    "latest": {
      "gpu_tot_sim_cycle": null,
      "gpu_tot_sim_insn": null,
      "gpu_tot_ipc": null,
      "sim_rate_ips": null
    },
    "log_tail": []
  }
}
```

The same event when a sim finishes.

data: sim event, sim finished:

```
{
  "type": "sim",
  "sim": {
    "id": "sim-1047ddbdf5b1",
    "run_id": "run-210d75c008c4",
    "gpu_config": "SM86_RTX3070",
    "replica": 0,
    "state": "succeeded",
    "error": null,
    "launcher_ref": "64418",
    "created_at": "2026-11-08T08:59:03.024065Z",
    "started_at": "2026-11-08T08:59:03.224424Z",
    "finished_at": "2026-11-08T08:59:06.130355Z",
    "last_heartbeat_at": "2026-11-08T08:59:06.130355Z",
    "exit_code": 0,
    "kernels_done": 6,
    "kernels_total": 20,
    "current_kernel": "_Z10dot_kernelIdEvPKT_S2_PS0_i",
    "latest": {
      "gpu_tot_sim_cycle": 76713.0,
      "gpu_tot_sim_insn": 26560256.0,
      "gpu_tot_ipc": 346.2289,
      "sim_rate_ips": 510774.0
    },
    "log_tail": [
      "Reply_Network_in_buffer_full_per_cycle =       0.0000",
      "Reply_Network_in_buffer_avg_util =       0.0000",
      "Reply_Network_out_buffer_full_per_cycle =       0.0000"
      /* 17 more items trimmed */
    ]
  }
}
```

### log

New output from one sim, exactly as the simulator sent it in one chunk.

data: log event:

```
{
  "type": "log",
  "sim_id": "sim-e8f63b6b3bed",
  "lines": [
    "GPGPU-Sim version 4.2.0 (build gpgpu-sim_git-commit-13c67115070dc2f0876254a790d0238073ca364a-modified_590.0) configured with AccelWattch.",
    "",
    "----------------------------------------------------------------------------"
    /* 197 more items trimmed */
  ]
}
```

### kernel

A kernel finished on one sim. The `stat` carries the full `stats` map.

data: kernel event:

```
{
  "type": "kernel",
  "stat": {
    "sim_id": "sim-1047ddbdf5b1",
    "kernel_index": 0,
    "kernel_name": "_Z11init_kernelIdEvPT_S1_S1_S0_S0_S0_",
    "kernel_launch_uid": 1,
    "gpu_sim_cycle": 6174,
    "gpu_sim_insn": 19456,
    "gpu_ipc": 3.1513,
    "sim_time_ns": 5454.063604240282,
    "stats": {
      "kernel_name": "_Z11init_kernelIdEvPT_S1_S1_S0_S0_S0_",
      "kernel_launch_uid": 1.0,
      "gpu_sim_cycle": 6174.0,
      "gpu_sim_insn": 19456.0,
      "gpu_ipc": 3.1513,
      "gpu_tot_sim_cycle": 6174.0
      /* 943 more keys trimmed */
    }
  }
}
```

## Order and reconnects

- Events arrive in the order the app committed them, and the snapshot never overlaps an event that follows it. Apply the snapshot, then apply each event on top.
- For one log chunk the order is `log`, then any `kernel` events it completed, then `sim`.
- The app keeps up to 2000 unsent events per subscriber. A subscriber that falls further behind is disconnected. Reconnect and start again from the new snapshot. `EventSource` reconnects by itself.
- The stream stays open after the run finishes. Close it when the run's state is no longer `active`.


# Simulator endpoints

Page: api-simulator.html

A simulator sends a heartbeat every few seconds and a log chunk whenever it has output. It can also send step timing and upload its output files before its final heartbeat. Every call is safe to retry. [Bring your own simulator](simulator.html) puts them together into a working client.

## Send a heartbeat

`POST /api/sims/{sim_id}/heartbeat` (bearer token)

Records that the sim is alive and, if `state` differs, moves the sim to it when `SIM_TRANSITIONS` allows the move. A `running` heartbeat brings a `lost` sim back. A heartbeat for a sim that already finished changes nothing and returns its final state. Send one every few seconds. A sim with no heartbeat for `CASCADE_HEARTBEAT_TIMEOUT_S` (60 by default) becomes `lost`.

### Path parameters

| name | type | description |
|---|---|---|
| `sim_id` | `str` | The sim's `id`, such as `sim-6b4c91e71437`. Find it in `RunDetail.sims`. |

### Request body

JSON, [Heartbeat](types.html#heartbeat).

| field | type | description |
|---|---|---|
| `state` | `Literal["running", "succeeded", "failed"]` | What the simulator reports. Send `running` while working and `succeeded` or `failed` once at the end. |
| `exit_code` | `int \| None` | Your process's exit code. Send it with the final heartbeat. (default `null`) |
| `error` | `str \| None` | A short reason, with a `failed` heartbeat. (default `null`) |

### Returns

`200` with a [`HeartbeatAck`](types.html#heartbeatack). If `cancel` is `true`, stop simulating and exit.

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |
| [`404`](errors.html#e404) | No sim has this id. |
| [`422`](errors.html#e422) | `state` is not `running`, `succeeded` or `failed`. |

### Example

```bash
curl -X POST "$CASCADE_URL/api/sims/sim-1047ddbdf5b1/heartbeat" \
  -H "Authorization: Bearer $CASCADE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"state": "running"}'
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.post(f"{os.environ['CASCADE_URL']}/api/sims/sim-1047ddbdf5b1/heartbeat", headers=headers, json={"state": "running"})
resp.raise_for_status()
print(resp.json())
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/sims/sim-1047ddbdf5b1/heartbeat`, {
  method: "POST",
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}`, "Content-Type": "application/json" },
  body: JSON.stringify({"state": "running"}),
});
console.log(await resp.json());
```

Response · 200 OK · application/json:

```json
{
  "state": "succeeded",
  "cancel": false
}
```

Response to a cancelled sim · 200 OK · application/json:

```json
{
  "state": "cancelled",
  "cancel": true
}
```

404 response · 404 Not Found · application/json:

```json
{
  "detail": "sim sim-000000000000 not found"
}
```

## Send log lines

`POST /api/sims/{sim_id}/logs` (bearer token)

Appends `lines` to the sim's log and parses any finished Accel-Sim kernel blocks out of them. Number chunks from 0 and add one each time. The app stores the first chunk it gets for each `seq` and ignores repeats, so a retry after a timeout is safe. Send lines without their trailing newline.

### Path parameters

| name | type | description |
|---|---|---|
| `sim_id` | `str` | The sim's `id`, such as `sim-6b4c91e71437`. Find it in `RunDetail.sims`. |

### Request body

JSON, [LogChunk](types.html#logchunk).

| field | type | description |
|---|---|---|
| `seq` | `int` | Per-sim, increasing; a repeated seq is ignored. (at least 0) |
| `lines` | `list[str]` | Lines of simulator stdout, without trailing newlines. |

### Returns

`204` with no body, whether the chunk was new or a repeat.

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |
| [`404`](errors.html#e404) | No sim has this id. |
| [`422`](errors.html#e422) | `seq` is negative or `lines` is not a list of strings. |

### Example

```bash
curl -X POST "$CASCADE_URL/api/sims/sim-1047ddbdf5b1/logs" \
  -H "Authorization: Bearer $CASCADE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"seq": 0, "lines": ["replayed line"]}'
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.post(f"{os.environ['CASCADE_URL']}/api/sims/sim-1047ddbdf5b1/logs", headers=headers, json={"seq": 0, "lines": ["replayed line"]})
resp.raise_for_status()
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/sims/sim-1047ddbdf5b1/logs`, {
  method: "POST",
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}`, "Content-Type": "application/json" },
  body: JSON.stringify({"seq": 0, "lines": ["replayed line"]}),
});
```

Response · 204 No Content:

```
(empty body)
```

## Send step timing

`POST /api/sims/{sim_id}/step-timeline` (bearer token)

Stores the output of `cascade simulate` and `cascade compare` for this sim. `cascade worker` sends it after a successful sim on a trace with step timing, before its final heartbeat. Sending again replaces the earlier upload, so a retry is safe. The app checks only that `simulated` has a `steps` list and a `summary` object and stores both objects as they are.

### Path parameters

| name | type | description |
|---|---|---|
| `sim_id` | `str` | The sim's `id`, such as `sim-6b4c91e71437`. Find it in `RunDetail.sims`. |

### Request body

JSON, [StepTimelineUpload](types.html#steptimelineupload).

| field | type | description |
|---|---|---|
| `simulated` | `dict` | `cascade simulate` output (simulated_timeline.json), with steps and summary. |
| `compare` | `dict \| None` | `cascade compare` output (compare.json) against the real timeline. |

### Returns

`204` with no body.

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |
| [`404`](errors.html#e404) | No sim has this id. |
| [`422`](errors.html#e422) | `simulated` lacks a `steps` list or a `summary` object, or `compare` is neither an object nor `null`. |

### Example

```bash
curl -X POST "$CASCADE_URL/api/sims/sim-57c51b464fc9/step-timeline" \
  -H "Authorization: Bearer $CASCADE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"simulated": {"meta": {"run_name": "empty"}, "steps": [], "requests": [], "summary": {"simulation": {"host_model": "scaled-gaps", "host_fit": {}, "coverage": {"kernels": 0, "simulated_kernels": 0, "kernel_fraction": 0, "gpu_time_fraction": 0, "measured_gpu_ns": 0.0, "covered_measured_gpu_ns": 0.0, "unmodeled_gpu_ns": 0.0, "steps": []}, "request_assumption": "one request per repeat; queue + prefill/decode walls"}, "per_kind": {}}}, "compare": {"steps": [], "requests": [], "summary": {"wall_ns": {"count": 0, "mean_error": null, "mean_abs_error": null, "p50_abs_error": null, "p95_abs_error": null, "mean_error_pct": null, "mean_abs_error_pct": null, "p50_abs_error_pct": null, "p95_abs_error_pct": null}, "gpu_busy_ns": {"count": 0, "mean_error": null, "mean_abs_error": null, "p50_abs_error": null, "p95_abs_error": null, "mean_error_pct": null, "mean_abs_error_pct": null, "p50_abs_error_pct": null, "p95_abs_error_pct": null}}}}'
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}"}
resp = httpx.post(f"{os.environ['CASCADE_URL']}/api/sims/sim-57c51b464fc9/step-timeline", headers=headers, json={"simulated": {"meta": {"run_name": "empty"}, "steps": [], "requests": [], "summary": {"simulation": {"host_model": "scaled-gaps", "host_fit": {}, "coverage": {"kernels": 0, "simulated_kernels": 0, "kernel_fraction": 0, "gpu_time_fraction": 0, "measured_gpu_ns": 0.0, "covered_measured_gpu_ns": 0.0, "unmodeled_gpu_ns": 0.0, "steps": []}, "request_assumption": "one request per repeat; queue + prefill/decode walls"}, "per_kind": {}}}, "compare": {"steps": [], "requests": [], "summary": {"wall_ns": {"count": 0, "mean_error": None, "mean_abs_error": None, "p50_abs_error": None, "p95_abs_error": None, "mean_error_pct": None, "mean_abs_error_pct": None, "p50_abs_error_pct": None, "p95_abs_error_pct": None}, "gpu_busy_ns": {"count": 0, "mean_error": None, "mean_abs_error": None, "p50_abs_error": None, "p95_abs_error": None, "mean_error_pct": None, "mean_abs_error_pct": None, "p50_abs_error_pct": None, "p95_abs_error_pct": None}}}})
resp.raise_for_status()
```

```javascript
const resp = await fetch(`${CASCADE_URL}/api/sims/sim-57c51b464fc9/step-timeline`, {
  method: "POST",
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}`, "Content-Type": "application/json" },
  body: JSON.stringify({"simulated": {"meta": {"run_name": "empty"}, "steps": [], "requests": [], "summary": {"simulation": {"host_model": "scaled-gaps", "host_fit": {}, "coverage": {"kernels": 0, "simulated_kernels": 0, "kernel_fraction": 0, "gpu_time_fraction": 0, "measured_gpu_ns": 0.0, "covered_measured_gpu_ns": 0.0, "unmodeled_gpu_ns": 0.0, "steps": []}, "request_assumption": "one request per repeat; queue + prefill/decode walls"}, "per_kind": {}}}, "compare": {"steps": [], "requests": [], "summary": {"wall_ns": {"count": 0, "mean_error": null, "mean_abs_error": null, "p50_abs_error": null, "p95_abs_error": null, "mean_error_pct": null, "mean_abs_error_pct": null, "p50_abs_error_pct": null, "p95_abs_error_pct": null}, "gpu_busy_ns": {"count": 0, "mean_error": null, "mean_abs_error": null, "p50_abs_error": null, "p95_abs_error": null, "mean_error_pct": null, "mean_abs_error_pct": null, "p50_abs_error_pct": null, "p95_abs_error_pct": null}}}}),
});
```

Response · 204 No Content:

```
(empty body)
```

## Upload an output file

`PUT /api/sims/{sim_id}/files/{name}` (bearer token)

Stores the request body as the sim's file `name`, in `$CASCADE_DATA_DIR/files/<sim_id>/`. Putting the same name again replaces the file, so a retry is safe. `cascade worker` uploads every file the simulator wrote, then its own log as `accelsim.log.gz`, before its final heartbeat.

### Path parameters

| name | type | description |
|---|---|---|
| `sim_id` | `str` | The sim's `id`, such as `sim-6b4c91e71437`. Find it in `RunDetail.sims`. |
| `name` | `str` | The file's name, such as `kernel-stats.csv`. Letters, digits, `.`, `_` and `-`, 1 to 128 characters, and not `.` or `..`. |

### Request body

The file's raw bytes, sent as `Content-Type: application/octet-stream`. Not JSON and not a form.

### Returns

`200` with the stored [`SimFile`](types.html#simfile).

### Errors

| status | when |
|---|---|
| [`401`](errors.html#e401) | The `Authorization` header is missing, is not `Bearer <token>`, or the token is wrong. |
| [`404`](errors.html#e404) | No sim has this id. |
| [`413`](errors.html#e413) | The file is over 64 MiB. |
| [`413`](errors.html#e413) | The sim's files would pass 256 MiB in all. The file a put replaces doesn't count. |
| [`422`](errors.html#e422) | `name` has a character other than letters, digits, `.`, `_` and `-`, is over 128 characters, or is `.` or `..`. |

### Example

```bash
curl -X PUT "$CASCADE_URL/api/sims/sim-57c51b464fc9/files/kernel-stats.csv" \
  -H "Authorization: Bearer $CASCADE_TOKEN" \
  -H "Content-Type: application/octet-stream" \
  --data-binary @kernel-stats.csv
```

```python
import os

import httpx

headers = {"Authorization": f"Bearer {os.environ['CASCADE_TOKEN']}", "Content-Type": "application/octet-stream"}
resp = httpx.put(f"{os.environ['CASCADE_URL']}/api/sims/sim-57c51b464fc9/files/kernel-stats.csv", headers=headers, content=open("kernel-stats.csv", "rb").read())
resp.raise_for_status()
print(resp.json())
```

```javascript
import { readFile } from "node:fs/promises";

const resp = await fetch(`${CASCADE_URL}/api/sims/sim-57c51b464fc9/files/kernel-stats.csv`, {
  method: "PUT",
  headers: { Authorization: `Bearer ${CASCADE_TOKEN}`, "Content-Type": "application/octet-stream" },
  body: await readFile("kernel-stats.csv"),
});
console.log(await resp.json());
```

Response · 200 OK · application/json:

```json
{
  "name": "kernel-stats.csv",
  "size": 3829,
  "content_type": "text/csv",
  "created_at": "2026-11-08T08:59:06.686299Z"
}
```

413 response, file too big · 413 Content Too Large · application/json:

```json
{
  "detail": "file is over the 64 MiB per-file limit"
}
```

422 response, bad name · 422 Unprocessable Entity · application/json:

```json
{
  "detail": "file name 'my file.txt' must match ^(?!\\.\\.?$)[A-Za-z0-9._-]{1,128}$"
}
```


# Types

Page: types.html

This page is generated from `cascade/models.py` each time the docs build, so it matches the code. The web UI mirrors the same types in `web/src/api/types.ts`.

Types are Python annotations as written in the source. `X | None` means the field is always present and may be `null`. `datetime` is an ISO 8601 string in UTC. `tuple[int, int, int]` is a JSON array of three integers.

## States

### `TraceState`

| value | meaning |
|---|---|
| `uploading` | Files are arriving. `kernel_count` and `bytes` grow on each scan, every 2 seconds. |
| `processing` | The upload is complete. The app is running the trace post-processor, unless the upload already has `traces/kernelslist.g`. |
| `ready` | Post-processing produced `traces/kernelslist.g`. Runs can launch. |
| `failed` | `metadata.json` has `status: "failed"`, or post-processing failed. `error` says which. |

### `SimState`

| value | meaning |
|---|---|
| `pending` | Created with the run. Waiting for the launcher. |
| `launching` | The launcher is starting the sim's sandbox or process. |
| `running` | The sim sent a `running` heartbeat. |
| `lost` | No heartbeat for `CASCADE_HEARTBEAT_TIMEOUT_S`. A later heartbeat brings it back. |
| `succeeded` | The sim sent a `succeeded` heartbeat. Terminal. |
| `failed` | The sim sent a `failed` heartbeat, the launcher failed, or it stayed `lost` for 10 times the timeout. Terminal. |
| `cancelled` | The run was cancelled before this sim finished. Terminal. |

### `RunState`

cascade/models.py:

```
RunState = Literal["active", "succeeded", "partial", "failed", "cancelled"]
```

| value | meaning |
|---|---|
| `active` | At least one sim has not finished. |
| `succeeded` | Every sim succeeded. |
| `partial` | Every sim finished, at least one succeeded, and at least one did not. |
| `failed` | Every sim failed. |
| `cancelled` | Every sim finished, at least one was cancelled, and none succeeded. |

[`RunState`](types.html#runstate) is not stored. `run_state()` in `cascade/models.py` derives it from the run's sim states each time a run is read.

## Traces

### `GpuConfig`

| field | type | description |
|---|---|---|
| `name` | `str` | Accel-Sim tested-cfgs directory name, e.g. "SM90_H100". |
| `arch` | `str` | For example, "sm_90". |
| `core_clock_mhz` | `float` | First value of -gpgpu_clock_domains in gpgpusim.config. |

### `KernelLaunch`

One kernel launch from the trace's stats_ctx_* file (original hardware).

| field | type | description |
|---|---|---|
| `kernel_id` | `int` | Matches kernel-<N> in the trace filename. |
| `name` | `str` | Mangled kernel name. |
| `grid` | `tuple[int, int, int]` | Grid size, x, y and z. |
| `block` | `tuple[int, int, int]` | Block size, x, y and z. |
| `instructions` | `int \| None` | Instructions the tracer counted. `null` if the kernel's trace was cut short. |

### `TraceSummary`

| field | type | description |
|---|---|---|
| `id` | `str` | The profiler's run_name. |
| `state` | `TraceState` | Where the trace is in [its lifecycle](concepts.html#trace-states). |
| `error` | `str \| None` | Why the trace failed. `null` unless `state` is `failed`. |
| `created_at` | `datetime` | When the first file of the trace landed. |
| `updated_at` | `datetime` | When the app last changed this trace. |
| `model` | `str \| None` | `metadata.json` `model`, such as `facebook/opt-125m`. |
| `layers` | `list[str]` | `metadata.json` `layers`. Empty if missing or not a list. |
| `source_gpu` | `str \| None` | metadata.json "gpu". |
| `trace_gpu_arch` | `str` | metadata.json "trace_gpu_arch", default "sm_90". |
| `suggested_sim_config` | `str \| None` | `metadata.json` `suggested_sim_config`. The UI preselects it on the New run screen. |
| `real_gpu_config` | `str \| None` | metadata.json: config of the GPU it was captured on. (default `null`) |
| `sample_sim_configs` | `list[str]` | metadata.json: preselected. (default `[]`) |
| `sample_set` | `str \| None` | metadata.json: traces compared side by side. (default `null`) |
| `kernel_count` | `int` | Traced kernels seen so far. |
| `bytes` | `int` | Bytes on disk under the trace dir. |
| `has_real_timing` | `bool` | True when the trace has real per-kernel timings (`real_timing.json`). The profiler does not write them yet. |
| `has_step_timeline` | `bool` | corrected_timeline.json + kernel_mapping.json are in the trace dir. |
| `sample` | `bool` | True for the sample trace that `cascade serve` seeds from `samples/` (see [Introduction](index.html)), unless `CASCADE_SAMPLES` is false. It is `ready` but has no instruction traces, so it can't start a run. |

### `TraceDetail`

Every field of [TraceSummary](types.html#tracesummary), plus these.

| field | type | description |
|---|---|---|
| `metadata` | `dict \| None` | The trace's `metadata.json`, parsed. `null` until it arrives. |
| `kernels` | `list[KernelLaunch]` | Kernel launches from the trace's `stats_ctx_*` files, by `kernel_id`. Filled in when the trace becomes `ready`. |

## Runs and sims

### `RunEntry`

| field | type | description |
|---|---|---|
| `gpu_config` | `str` | A `GpuConfig.name` whose `arch` matches the trace's `trace_gpu_arch`. |
| `count` | `int` | How many sims run this config. (at least 1, at most 64) |

### `RunCreate`

| field | type | description |
|---|---|---|
| `trace_id` | `str` | A trace in state `ready`. |
| `entries` | `list[RunEntry]` | One entry per config. Each fans out to `count` sims. (at least 1 item) |

### `RunSummary`

| field | type | description |
|---|---|---|
| `id` | `str` | Generated, `run-` and 12 hex digits. |
| `trace_id` | `str` | The trace the run simulates. |
| `created_at` | `datetime` | When the run was created. |
| `state` | `RunState` | Derived from the sims. See [`RunState`](types.html#runstate). |
| `entries` | `list[RunEntry]` | The entries the run was created with. |
| `sim_counts` | `dict[SimState, int]` | How many sims are in each state. States with no sims are left out. |
| `sample` | `bool` | True for the sample run seeded with the sample trace. Its results were recorded when the run was made, not in this workspace, and cancelling it changes nothing. |
| `replay` | `bool` | Replays a sample's recorded Accel-Sim results. (default `false`) |

### `RunDetail`

Every field of [RunSummary](types.html#runsummary), plus these.

| field | type | description |
|---|---|---|
| `trace` | `TraceSummary` | The run's trace. |
| `sims` | `list[SimSummary]` | Every sim, sorted by `gpu_config` and then `replica`. |

### `SimSummary`

| field | type | description |
|---|---|---|
| `id` | `str` | Generated, `sim-` and 12 hex digits. The simulator receives it as `CASCADE_SIM_ID`. |
| `run_id` | `str` | The run this sim belongs to. |
| `gpu_config` | `str` | The config this sim simulates. The simulator receives it as `CASCADE_GPU_CONFIG`. |
| `replica` | `int` | 0-based index within its {gpu_config, count} entry. |
| `state` | `SimState` | Where the sim is in [its lifecycle](concepts.html#sim-states). |
| `error` | `str \| None` | Why the sim failed or went `lost`, from a heartbeat or from the app. A `running` heartbeat clears it. |
| `launcher_ref` | `str \| None` | Sailbox id or local pid. |
| `created_at` | `datetime` | When the run was created. |
| `started_at` | `datetime \| None` | The first `running` heartbeat. |
| `finished_at` | `datetime \| None` | When the sim reached a terminal state. |
| `last_heartbeat_at` | `datetime \| None` | The last heartbeat. `null` before the first one. |
| `exit_code` | `int \| None` | From the final heartbeat, when the simulator sent one. |
| `kernels_done` | `int` | Kernel stats blocks parsed from the log so far. |
| `kernels_total` | `int` | The trace's kernel count when the run was created. |
| `current_kernel` | `str \| None` | The kernel named by the last `launching kernel name:` line in the log. |
| `latest` | `SimMetrics` | The most recent totals parsed from the log. |
| `log_tail` | `list[str]` | Last 20 lines. |

### `SimMetrics`

| field | type | description |
|---|---|---|
| `gpu_tot_sim_cycle` | `float \| None` | Simulated cycles so far, over every kernel. (default `null`) |
| `gpu_tot_sim_insn` | `float \| None` | Simulated instructions so far, over every kernel. (default `null`) |
| `gpu_tot_ipc` | `float \| None` | Instructions per cycle so far, over every kernel. (default `null`) |
| `sim_rate_ips` | `float \| None` | gpgpu_simulation_rate, instructions/sec. (default `null`) |

## Results

### `KernelStat`

One per-kernel stats block parsed from Accel-Sim output.

| field | type | description |
|---|---|---|
| `sim_id` | `str` | The sim that ran this kernel. |
| `kernel_index` | `int` | 0-based order of completion within the sim. |
| `kernel_name` | `str` | The block's `kernel_name`. |
| `kernel_launch_uid` | `int \| None` | The block's `kernel_launch_uid`. |
| `gpu_sim_cycle` | `int \| None` | Cycles this kernel took on the simulated GPU. |
| `gpu_sim_insn` | `int \| None` | Instructions this kernel ran. |
| `gpu_ipc` | `float \| None` | This kernel's instructions per cycle. |
| `sim_time_ns` | `float \| None` | gpu_sim_cycle / core_clock_mhz * 1000. |
| `stats` | `dict[str, float \| str]` | Every "key = value" line in the block. |

### `ComparisonRow`

| field | type | description |
|---|---|---|
| `kernel_index` | `int` | Row position. Kernels are matched by order. |
| `kernel_name` | `str` | From real timing if present, otherwise from the first sim that finished this kernel. |
| `real_ns` | `float \| None` | `duration_ns` from real timing. `null` without real timing. |
| `sim_ns` | `dict[str, float \| None]` | gpu_config -> median sim_time_ns over replicas. |

### `Comparison`

| field | type | description |
|---|---|---|
| `has_real_timing` | `bool` | False until the trace has real timing. |
| `configs` | `list[str]` | The run's configs, in entry order, without repeats. |
| `rows` | `list[ComparisonRow]` | One row per kernel position. |

### `SimResults`

| field | type | description |
|---|---|---|
| `sim` | `SimSummary` | The sim as [`SimSummary`](types.html#simsummary). |
| `trace_id` | `str` | The trace the sim ran. |
| `gpu_config` | `GpuConfig` | The full config. `arch` is `unknown` and the clock is 0 if the config disappeared since the run started. |
| `kernels` | `list[KernelStat]` | Every kernel the sim finished, with the full `stats` map. |
| `totals` | `SimMetrics` | Same as `sim.latest`. |

### `RunResults`

| field | type | description |
|---|---|---|
| `run` | `RunSummary` | The run without its trace and sims, which are below. |
| `trace` | `TraceDetail` | The full trace. |
| `sims` | `list[SimResults]` | One entry per sim. |
| `comparison` | `Comparison` | Same as [Compare real and simulated time](api-runs.html#get-comparison). |

### `SimStepTimeline`

| field | type | description |
|---|---|---|
| `sim_id` | `str` | The sim that sent it. |
| `gpu_config` | `str` | The sim's config. |
| `replica` | `int` | The sim's replica. |
| `simulated` | `dict` | As sent in `StepTimelineUpload.simulated`. |
| `compare` | `dict \| None` | As sent in `StepTimelineUpload.compare`. |

### `SimFile`

An output file a sim uploaded with PUT /api/sims/{id}/files/{name}.

| field | type | description |
|---|---|---|
| `name` | `str` | Matches FILE_NAME_PATTERN; Accel-Sim's name, plus .gz when the worker gzipped it. |
| `size` | `int` | Bytes as stored. |
| `content_type` | `str` | From the name: `text/csv` for `.csv`, `text/plain` for `.txt` and `.log`, `application/json` for `.json`, `application/gzip` for `.gz`, and otherwise a guess, falling back to `application/octet-stream`. |
| `created_at` | `datetime` | When this version was stored; a re-put replaces the file. |

### `RunFile`

Every field of [SimFile](types.html#simfile), plus these.

| field | type | description |
|---|---|---|
| `sim_id` | `str` | The sim that uploaded the file. |
| `gpu_config` | `str` | The sim's config. |
| `replica` | `int` | The sim's replica. |

## Simulator calls

### `Heartbeat`

| field | type | description |
|---|---|---|
| `state` | `Literal["running", "succeeded", "failed"]` | What the simulator reports. Send `running` while working and `succeeded` or `failed` once at the end. |
| `exit_code` | `int \| None` | Your process's exit code. Send it with the final heartbeat. (default `null`) |
| `error` | `str \| None` | A short reason, with a `failed` heartbeat. (default `null`) |

### `HeartbeatAck`

| field | type | description |
|---|---|---|
| `state` | `SimState` | The sim's state after this heartbeat. |
| `cancel` | `bool` | True when main cancelled this sim; the simulator must stop. |

### `LogChunk`

| field | type | description |
|---|---|---|
| `seq` | `int` | Per-sim, increasing; a repeated seq is ignored. (at least 0) |
| `lines` | `list[str]` | Lines of simulator stdout, without trailing newlines. |

### `StepTimelineUpload`

What a sim posts after step timing: `cascade simulate` and `cascade compare` output.

| field | type | description |
|---|---|---|
| `simulated` | `dict` | `cascade simulate` output (simulated_timeline.json), with steps and summary. |
| `compare` | `dict \| None` | `cascade compare` output (compare.json) against the real timeline. |

## Events

### `RunEvent`

cascade/models.py:

```
RunEvent = SnapshotEvent | SimEvent | LogEvent | KernelEvent
```

One event on the [run event stream](api-events.html). The `type` field says which model it is.

### `SnapshotEvent`

| field | type | description |
|---|---|---|
| `type` | `Literal["snapshot"]` | Always `snapshot`. (default `"snapshot"`) |
| `run` | `RunDetail` | The run as it is at connect time. |
| `kernels` | `list[KernelStat]` | Every sim's stats with `stats` emptied; full stats are ~30 KB each. |

### `SimEvent`

| field | type | description |
|---|---|---|
| `type` | `Literal["sim"]` | Always `sim`. (default `"sim"`) |
| `sim` | `SimSummary` | The sim after a change: state, heartbeat, progress or metrics. |

### `LogEvent`

| field | type | description |
|---|---|---|
| `type` | `Literal["log"]` | Always `log`. (default `"log"`) |
| `sim_id` | `str` | The sim the lines came from. |
| `lines` | `list[str]` | The new chunk, as the simulator sent it. |

### `KernelEvent`

| field | type | description |
|---|---|---|
| `type` | `Literal["kernel"]` | Always `kernel`. (default `"kernel"`) |
| `stat` | `KernelStat` | A kernel that just finished, with the full `stats` map. |

## Sim transitions

`SIM_TRANSITIONS` lists every allowed move. The app ignores any other move, whether it comes from a heartbeat or from the app itself. The comment above it in the source says who makes each move.

> `Allowed sim transitions. Main owns pending/launching/cancelled/lost; heartbeats own the rest.`

| from ↓ to → | `pending` | `launching` | `running` | `lost` | `succeeded` | `failed` | `cancelled` |
|---|---|---|---|---|---|---|---|
| `pending` | · | ✓ | · | · | · | ✓ | ✓ |
| `launching` | · | · | ✓ | ✓ | ✓ | ✓ | ✓ |
| `running` | · | · | · | ✓ | ✓ | ✓ | ✓ |
| `lost` | · | · | ✓ | · | ✓ | ✓ | ✓ |
| `succeeded` | · | · | · | · | · | · | · |
| `failed` | · | · | · | · | · | · | · |
| `cancelled` | · | · | · | · | · | · | · |

`succeeded`, `failed` and `cancelled` are `TERMINAL_SIM_STATES`. Nothing leaves them. [Concepts](concepts.html#sim-states) draws the same table as a diagram.

## Run state

`run_state()` derives a run's state from its sims' states. These rows are computed by calling it during the build.

| sim states | run state |
|---|---|
| `running`, `succeeded` | `active` |
| `succeeded`, `succeeded` | `succeeded` |
| `succeeded`, `failed` | `partial` |
| `failed`, `failed` | `failed` |
| `cancelled`, `failed` | `cancelled` |
| `cancelled`, `succeeded` | `partial` |
| `cancelled`, `cancelled` | `cancelled` |


# Errors

Page: errors.html

Errors come back as JSON with a `detail` field. For errors the app raises, `detail` is a sentence. For a body that fails validation, it's a list with one entry per problem.

## 401 Unauthorized

The `Authorization` header is missing, doesn't start with `Bearer `, or carries the wrong token. On the event stream, a wrong `?token=` gets the same error. The response has a `WWW-Authenticate: Bearer` header.

Response · 401 Unauthorized · application/json:

```json
{
  "detail": "missing or wrong token"
}
```

## 404 Not Found

The trace, run, sim or sim's file in the path doesn't exist, or `trace_id` in a new run names no trace. `detail` names what's missing.

Response · 404 Not Found · application/json:

```json
{
  "detail": "run run-000000000000 not found"
}
```

A path under `/api/` that no route serves gets a plain `Not Found`, without a token check.

Response · 404 Not Found · application/json:

```json
{
  "detail": "Not Found"
}
```

## 409 Conflict

Only [Create a run](api-runs.html#create-run) returns `409`, when the trace exists but isn't `ready`, or is the bundled sample, which can't be simulated again. Wait for it to finish processing, or check its `error` if it `failed`.

Response · 409 Conflict · application/json:

```json
{
  "detail": "trace demo-failed is failed, not ready"
}
```

## 413 Content Too Large

Only [Upload an output file](api-simulator.html#put-sim-file) returns `413`, when the file is over 64 MiB or the sim's files would pass 256 MiB in all. The app reads the whole body before it answers, so the client gets the `413` instead of a dropped connection.

Response · 413 Content Too Large · application/json:

```json
{
  "detail": "file is over the 64 MiB per-file limit"
}
```

## 422 Unprocessable Entity

`422` has two shapes. When a run asks for a config the app doesn't have, or one for another architecture, or a file upload has a name the app refuses, `detail` is a sentence.

Response · 422 Unprocessable Entity · application/json:

```json
{
  "detail": "unknown gpu_config 'SM99_X'"
}
```

Response · 422 Unprocessable Entity · application/json:

```json
{
  "detail": "SM90_H100 is sm_90 but trace demo is sm_86"
}
```

Response · 422 Unprocessable Entity · application/json:

```json
{
  "detail": "file name 'my file.txt' must match ^(?!\\.\\.?$)[A-Za-z0-9._-]{1,128}$"
}
```

When the body doesn't match its model, `detail` is a list. Each entry has a `type`, a `loc` path into the request, a `msg`, the `input` it rejected, and sometimes a `ctx` with the limit.

Response · 422 Unprocessable Entity · application/json:

```json
{
  "detail": [
    {
      "type": "greater_than_equal",
      "loc": [
        "body",
        "entries",
        0,
        "count"
      ],
      "msg": "Input should be greater than or equal to 1",
      "input": 0,
      "ctx": {
        "ge": 1
      }
    }
  ]
}
```

## Server errors

The app doesn't return `5xx` on purpose. A `5xx` means something failed inside the app, such as the disk filling up. Simulators should retry `5xx` and network errors with backoff, and stop on any `4xx`.


# Troubleshooting

Page: troubleshooting.html

Each section starts from what you see. Most problems show their cause in a trace's `error`, a sim's `error`, or a log file on the app's disk.

## The trace is stuck in uploading

- The profiler sends `metadata.json` and the `traces/kernelslist_ctx_*` and `stats_ctx_*` files after tracing ends. Check that the Modal run finished.
- If `metadata.json` has a `files` list, one listed file may be missing or a different size than listed. Compare the list with what's under `$CASCADE_DATA_DIR/s3/cascade/accelsim-traces/<trace id>/`.
- Without a `files` list, the app also waits for a scan in which no file changed. A trace that keeps changing stays `uploading`.

## The trace failed in processing

Read the trace's `error`.

- `post-processor not found` means `CASCADE_POSTPROCESS_CMD` isn't on the image. A deploy without `--base-image` has no post-processor, so only uploads that already include `traces/kernelslist.g` work there.
- `post-processing exited <code>` ends with the last lines of the post-processor's log. The whole log is at `$CASCADE_DATA_DIR/work/postprocess/<trace id>.log`.
- `profiler reported status 'failed'` means tracing failed. The profiler's logs are in the trace's directory.

## The profiler can't reach the bucket

Use the S3 endpoint URL, not the app URL, with `--s3-bucket cascade` and `--s3-addressing-style path`. The profiler checks the bucket before it starts tracing, so this fails fast.

## The profiler has no --s3-endpoint flag

You're running the older profiler at the root of the app branch. Use `accelsim_trace.py` from the `validate-accelsim-modal` branch, commit `b348d7d`.

## The token is rejected

Use the exact `CASCADE_TOKEN` the app was started with. If you deployed without setting one, rerun `cascade deploy --workspace <name>` (or create the same workspace again on [Get started](../start.html)). It finds the running workspace and prints its token again. Check for a trailing newline or space if you pasted it.

## A config is disabled or the run gets 422

The config's architecture differs from the trace's `trace_gpu_arch`. A trace only runs on configs of the GPU architecture it was captured on, so trace on that GPU instead. If a config you expect isn't listed at all, check that its `gpgpusim.config` is under `$CASCADE_ACCELSIM_DIR/gpu-simulator/gpgpu-sim/configs/tested-cfgs/` and has a `-gpgpu_clock_domains` line, then restart the app.

## A sim shows lost

The sim hasn't sent a heartbeat for `CASCADE_HEARTBEAT_TIMEOUT_S`. It recovers by itself if it reports again. If it doesn't, read its output at `$CASCADE_DATA_DIR/work/sims/<sim id>.out` on its sandbox. After 10 times the timeout it fails with `no heartbeat for 600s`.

## Every sim fails at launch

- `launch: CASCADE_SIM_CMD is not set` means the app has no simulator command.
- `launcher prepare:` means the app couldn't checkpoint its own sandbox. Check that `SAIL_API_KEY` is set in the app's environment.
- Locally, `could not start` means the first word of `CASCADE_SIM_CMD` isn't an executable path.

## A sim fails right away with kernelslist.g not found

The trace has no processed `traces/kernelslist.g`. The stand-in simulator and Accel-Sim both need it. Check that the trace reached `ready` through post-processing, not through a partial upload.

## The event stream goes quiet

The stream sends `: keepalive` every 15 seconds when nothing happens. If even that stops, a proxy may be buffering. The app sets `X-Accel-Buffering: no`, but other proxies may need their own setting. If your client fell more than 2000 events behind, the app closed the stream. Reconnect to get a fresh snapshot.


# Limits

Page: limits.html

These are the limits as of this version, from the code and the open work around it.

## Not built yet

- A default deploy runs the stand-in simulator, `cascade worker --replay`, unless you pass `--base-image`, `--sim-cmd` and `--accelsim-dir`. Its numbers come from a recorded log, not from your trace.
- The profiler doesn't capture real hardware timing (`real_timing.json`) yet, so the comparison shows simulated time only.
- Each sim models one GPU. Making many simulated GPUs act as one system is the next step, not something Cascade does yet.
- The Sail launcher is written against the Sail SDK 0.17.2 source and tested with a fake SDK. It hasn't run against the live Sail service.
- The S3 endpoint accepts anonymous uploads. Anyone with its URL can add a trace.

## Fixed limits

- Each entry in a run has a count from 1 to 64. A run can have any number of entries.
- A trace only runs on configs of the architecture it was captured on.
- The app launches up to 8 sims at a time. Larger runs launch in waves.
- A subscriber to the event stream can fall up to 2000 events behind before the app disconnects it.
- `log_tail` holds the last 20 lines. The full log is always available as a download.
- A sim's output files can be up to 64 MiB each and 256 MiB in all. The app has no way to delete them.

## Size and speed

- Traces grow with the layers, tokens and forward passes you trace. A small layer is tens of MB. Whole models with long prompts can reach hundreds of GB.
- The app needs about 2.2 times the uploaded size in disk, for the raw and processed trace.
- Simulation is far slower than real hardware. Start with one layer and a few tokens.
- The profiler's Modal function times out after 4 hours.