Guides

Traces

The app watches its bucket directory. A trace appears on the Traces screen as soon as its first file lands, and it's ready once the upload is complete and post-processed.

Follow a trace#

Open Traces in the UI, or poll List traces. Each trace moves through these states.

value	meaning
`uploading`	Files are arriving. `kernel_count` and `bytes` grow on each scan, every 2 seconds.
`processing`	The upload is complete. The app is running the trace post-processor, unless the upload already has `traces/kernelslist.g`.
`ready`	Post-processing produced `traces/kernelslist.g`. Runs can launch.
`failed`	`metadata.json` has `status: "failed"`, or post-processing failed. `error` says which.

Open a trace to see what it came from: the model, the layers, the prompts, the source GPU, the suggested config, and every traced kernel with its grid and block size. The kernel list fills in when the trace becomes ready.

When an upload counts as complete#

The app scans $CASCADE_DATA_DIR/s3/<bucket>/<prefix>/ every 2 seconds. It skips dotfiles, which covers rclone's in-flight .rclone_temp_put_* objects. The upload is complete when metadata.json exists and one of these holds.

metadata.json has a files list, and every listed file is on disk with the listed size in bytes. scripts/upload_trace.py writes this list.
There is no files list, every traces/kernelslist_ctx_* has a matching traces/stats_ctx_*, and the file tree hasn't changed since the previous scan. This is the case for the streaming profiler, which uploads metadata.json before its traces/ files.

If metadata.json has status: "failed", the trace goes to failed straight away, without processing.

Post-processing#

The profiler uploads raw traces. Accel-Sim reads processed ones, traces/kernelslist.g and one .tracez per kernel. So the app runs the post-processor once per trace.

bash

$CASCADE_POSTPROCESS_CMD <trace dir>/traces -j <cpu count>

Its output goes to $CASCADE_DATA_DIR/work/postprocess/<trace id>.log. If it exits non-zero, the trace fails with the exit code and the last 5 lines of that log. If it succeeds but leaves no kernelslist.g, the trace fails too.

An upload that already contains traces/kernelslist.g skips post-processing. The quickstart's stand-in trace works this way.

If the app restarts during post-processing, the trace stays processing and the post-processor runs again on the next start.

Where the trace's fields come from#

field	source
`id`	The directory name under the prefix, which is the profiler's `--run-name`.
`model`, `layers`	`metadata.json`.
`source_gpu`	`metadata.json` `gpu`.
`trace_gpu_arch`	`metadata.json` `trace_gpu_arch`, or `sm_90` if it's missing.
`suggested_sim_config`	`metadata.json` `suggested_sim_config`. The profiler doesn't set it, so the New run screen preselects the first compatible config instead.
`kernel_count`	While uploading, the raw `kernel-*.trace` and `.trace.xz` files seen. Once ready, the kernel lines in `kernelslist.g`.
`bytes`	Every file under the trace's directory.
`kernels`	The rows of `traces/stats_ctx_*`, or `stats.csv` from older tracers.
`has_real_timing`	Whether `real_timing.json`, or the file `metadata.json` names in `real_timing`, holds a list of rows with integer `kernel_id`s.
`has_step_timeline`	Whether the trace directory holds both `corrected_timeline.json` and `kernel_mapping.json`. The app rechecks on every scan, so it can turn true after the trace is `ready`. See Step timing.
`sample`	True only for the sample trace `cascade serve` seeds from `samples/` (see Introduction). It is `ready` but has no instruction traces, so it can't start a run.

Disk space#

Traces grow with the layers, tokens and forward passes you trace. A single small layer is tens of MB. Whole models with long prompts reach hundreds of GB. Processed files sit next to the raw ones, so plan for about 2.2 times the uploaded size on the app's disk, and pass --disk-gib when you deploy.