Cascadedocs

Guides

Bring your own simulator

Cascade runs whatever command is in CASCADE_SIM_CMD inside each sim. The app ships one, cascade worker, which runs Accel-Sim. Any other program that follows this contract can be the simulator: a patched build, or something else entirely.

The contract#

  1. Read your identity from the environment.
    variablevalue
    CASCADE_SIM_IDThis sim's id, such as sim-6b4c91e71437.
    CASCADE_MAIN_URLThe app's URL, from CASCADE_PUBLIC_URL.
    CASCADE_TOKENThe bearer token for every request.
    CASCADE_GPU_CONFIGThe config to simulate, such as SM90_H100.
    CASCADE_TRACE_DIRThe trace's directory. The processed trace is at $CASCADE_TRACE_DIR/traces/kernelslist.g.
    CASCADE_ACCELSIM_DIRThe app's CASCADE_ACCELSIM_DIR, where Accel-Sim and its configs live.
  2. Send POST /api/sims/{id}/heartbeat with {"state": "running"} right away, and again every few seconds. If the reply has "cancel": true, stop and exit.
  3. Send your output as POST /api/sims/{id}/logs with {"seq": n, "lines": [...]}, counting seq up from 0. Send Accel-Sim's stdout as it is. The app parses the per-kernel stats out of it.
  4. Optionally, upload the files your simulator wrote with PUT /api/sims/{id}/files/{name}, one file per call, with the raw bytes as the body. They show up under the run's Outputs.
  5. Finish with one heartbeat of succeeded or failed, with your exit_code, and an error if you failed. Send every log chunk and file before it.

Simulator endpoints has the full request and response for each call.

Retries are safe#

Every call is safe to repeat, so retry any 5xx or network error with backoff. The app ignores a log chunk whose seq it already has, a heartbeat that repeats the current state only refreshes the heartbeat time, and a file put again replaces the earlier one. Don't retry a 4xx. It means the token, the body or the file name is wrong, the file is too big, or the sim doesn't exist.

Keep heartbeats going while you wait on retries. After 60 seconds of silence the sim shows as lost, and after 600 seconds it fails and the app stops its sandbox.

The built-in worker#

cascade worker follows this contract for Accel-Sim. To use it, deploy with --sim-cmd "/app/.venv/bin/cascade worker".

Output files#

Before it starts the simulator, cascade worker notes every file in its run directory, which is the copied configs. After the simulator exits, it uploads each regular file in that directory that the simulator created or changed, in name order, and then its own log as accelsim.log.gz. It skips symlinks and subdirectories. A cancelled sim uploads nothing.

A custom simulator uploads with the same call. Upload an output file has the name rules and errors.

Step timing#

A trace can carry two more files next to metadata.json. When both are there, the trace's has_step_timeline is true. The app checks on every scan, so they can arrive after the trace is ready.

After a sim succeeds on such a trace, cascade worker runs the step-timing model in cascade/sim/ on its own log. It imports each mapped kernel's cycles with the config's core clock, rebuilds every step with cascade simulate's defaults, and compares the result with the corrected timeline using cascade compare. Then it sends both results to Send step timing, before its final heartbeat. If either file is missing, it writes one log line and skips this. If the model fails, for example because the mapping names a kernel the log doesn't have, the sim still succeeds, nothing is sent, and the error goes to the log.

A complete client#

To run a simulator other than Accel-Sim, write your own client. This one runs any command, forwards its stdout as log chunks, sends a heartbeat every 5 seconds, stops the command if the run is cancelled, and reports the exit code. It uses only the Python standard library.

cascade_sim.py
"""Run a simulator command as a Cascade sim: send heartbeats and forward its stdout.

CASCADE_SIM_CMD="python3 cascade_sim.py accel-sim.out -trace ... -config ..."
"""

import json
import os
import subprocess
import sys
import threading
import time
import urllib.error
import urllib.request

BASE = f"{os.environ['CASCADE_MAIN_URL']}/api/sims/{os.environ['CASCADE_SIM_ID']}"
TOKEN = os.environ["CASCADE_TOKEN"]
cancelled = threading.Event()


def post(path, body):
    data = json.dumps(body).encode()
    headers = {"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"}
    for attempt in range(6):
        req = urllib.request.Request(f"{BASE}/{path}", data, headers, method="POST")
        try:
            with urllib.request.urlopen(req, timeout=10) as resp:
                raw = resp.read()
                return json.loads(raw) if raw else None
        except urllib.error.HTTPError as e:
            if e.code < 500:
                raise
        except (urllib.error.URLError, TimeoutError):
            pass
        time.sleep(2**attempt)
    raise RuntimeError(f"POST {path} kept failing")


def heartbeat(state, exit_code=None, error=None):
    ack = post("heartbeat", {"state": state, "exit_code": exit_code, "error": error})
    if ack["cancel"]:
        cancelled.set()


def keep_beating(proc):
    while proc.poll() is None:
        try:
            heartbeat("running")
        except Exception as e:
            print(f"heartbeat failed: {e}", file=sys.stderr)
        if cancelled.is_set():
            proc.terminate()
        time.sleep(5)


def main():
    heartbeat("running")
    if cancelled.is_set():
        return 0
    proc = subprocess.Popen(
        sys.argv[1:], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, errors="replace"
    )
    threading.Thread(target=keep_beating, args=(proc,), daemon=True).start()
    seq, batch, sent_at = 0, [], time.monotonic()
    for line in proc.stdout:
        batch.append(line.rstrip("\n"))
        if len(batch) >= 200 or time.monotonic() - sent_at > 1:
            post("logs", {"seq": seq, "lines": batch})
            seq, batch, sent_at = seq + 1, [], time.monotonic()
    if batch:
        post("logs", {"seq": seq, "lines": batch})
    code = proc.wait()
    if cancelled.is_set():
        return 0
    if code == 0:
        heartbeat("succeeded", exit_code=0)
    else:
        heartbeat("failed", exit_code=code, error=f"simulator exited with {code}")
    return code


if __name__ == "__main__":
    sys.exit(main())

Put it in your simulator image and point CASCADE_SIM_CMD at it, followed by your command. On Sail, the app runs the command through a shell inside the fork, so $VARIABLES such as $CASCADE_GPU_CONFIG expand there. Quote them with single quotes so your own shell leaves them alone. The local launcher splits the command into arguments with shlex and runs it without a shell, so they don't expand locally.

What the image needs#

The app is layered on top of your image with the Dockerfile's BASE_IMAGE argument, so the app and every sim run the same image. It needs these things.

whatwhere the app looks
Accel-Sim's GPU configs$CASCADE_ACCELSIM_DIR/gpu-simulator/gpgpu-sim/configs/tested-cfgs/*/gpgpusim.config. Each directory becomes a GPU config.
The trace post-processorCASCADE_POSTPROCESS_CMD, by default $CASCADE_ACCELSIM_DIR/util/tracer_nvbit/tracer_tool/traces-processing/post-traces-processing. The app runs it on the main sandbox, not in sims.
Accel-Sim$CASCADE_ACCELSIM_DIR/gpu-simulator/bin/release/accel-sim.out, and each config's trace.config under $CASCADE_ACCELSIM_DIR/gpu-simulator/configs/tested-cfgs/. cascade worker runs it.
Your simulator commandCASCADE_SIM_CMD, a shell command, such as /app/.venv/bin/cascade worker. Sims run it with the environment above.
A shell and apt-getThe Dockerfile installs rclone and uv with apt-get, so the base must be Debian or Ubuntu.

Pin the simulator to the version the profiler traced with. The profiler at b348d7d builds the tracer from Accel-Sim commit 7ba54ded. The app's notes name Accel-Sim tag v2.0.1 for the simulator.

Where output goes#

Anything your command prints to stdout or stderr is also saved to $CASCADE_DATA_DIR/work/sims/<sim id>.out on the sim's disk. Only what you send to /logs and /files reaches the app, and only the log gets parsed. On Sail, the sim's disk is gone when its fork ends, so upload any file you want to keep.

esc