# Test runs and reports

Run a flow on a phone, watch the plan fill in live, read a report with a picture per step, run again from the step that failed, and get JUnit and JSON files for your pipeline.

A test run is one flow on one or more phones, from start to a verdict. While it runs the app shows a plan with each step's state and how long is left. When it ends you get a report with the verdict, the moment it failed, a timeline and a picture for every step. The same report comes out as HTML, JUnit XML and JSON for anything that reads test results.

## What it does

A run ends with one of five verdicts: passed, failed, cancelled, could not run, or interrupted when Silan stopped before the end. The live plan shows each step as waiting, running, done or failed and estimates the time left from the newest 5 runs of that flow.

The report has the verdict, the failure moment, a timeline, every step with its picture, the values the run used with secrets shown as `•••`, the model calls an agent step made, and run facts. A step the agent took over because the exact step no longer matched is marked healed. Steps after a failed one are marked not run.

| Button | What it does |
| --- | --- |
| **Run again** | The same flow on the same phones. |
| **Run again from step N** | Skips the steps before N. The new run's report shows them as skipped. |
| **Share report…** | Hands the HTML report to the Mac's share sheet. |
| **Download report…** | Saves one HTML file with the pictures inside it. |
| **Keep this run** | Keeps it past the retention window. |
| **Copy run id** | For the API or a bug report. |

**Busy phones.** A run started while its phone is in use waits in line. The app asks how long to wait, 30 minutes by default and 4 hours at most. **Stop Everything** holds a run where it is, and **Resume** lets it go on.

**Retention.** **Settings › Privacy & Data › Finished runs** keeps runs 7, 30, 90 or 365 days or with no limit. Each flow keeps at most its newest 200 runs. A kept run is never dropped. Deleting a running run is refused.

**Files.** Each run folder holds `index.html`, `junit.xml`, `report.json`, `timeline.json` and a `media` folder. The run's ⋯ menu has **Download report…**, which saves the HTML report, the JUnit XML and the JSON report where you choose, and **Copy run id**. The run folders sit under `~/Library/Application Support/Silan/runs/<run id>/`.

The `silan run` command is meant to run a flow from Terminal and write those files where you ask. On the current release it answers "the scenario engine is not available in this engine" (seen 4 Nov 2026), so run flows from the app for now.

## Step by step

1. Open a flow under **Flows** and click **Run**. Pick a phone for each role.
2. Watch the plan. Each step turns green or red as it goes. The time left updates.
3. When it ends, open the report. Click a step to see its picture.
4. If a step failed, fix the flow or the app, then click **Run again from step N**.
5. Click **Download report…** to keep the HTML, or **Share report…** to send it.

The spec has `run_scenario`, `get_run` and `get_report` over MCP and the same under [runs](/reference/rest/runs/) over REST ([run-scenario](/reference/mcp/run-scenario/), [get-report](/reference/mcp/get-report/)). On the current release the engine answers them with `the scenario engine is not available in this engine`, and `silan mcp` does not list them. Start runs from the app, and read the files from the run folder (below).

## Limits

- A run needs a phone with control. An iPad in the current release is refused for runs.
- The judged Done state with a real model, the words on a failed wait, an expired take-over during a run and the **Share report…** menu were proved in unit tests and offscreen pictures. Not yet proved on a real device.
- Silan writes its own HTML, JUnit and JSON only.

## Proof

:::note[Proof]
Proved on an iPhone 11 (iOS 27) on a cable, 14 Oct 2026, by a Silan worker: Download report gave `report.html` at 146,555 bytes with 8 pictures for run `run_5hvm668trq`, run `20261014-230235-22762`. Run again from step 2 after `run_16lucagv486` failed there gave `run_lwovdq6ztr`, which passed steps 2 to 8, run `20261014-173933-44324`. Two runs for one busy phone, A `run_rseosysrfw` and B `run_ysnnowujjc`, B waited then ran, 15 Oct 2026, run `20261015-045213-86795`. Retention and Keep this run, run `20261015-013237-31876`. A healed step, `run_nt65ypquid`, run `20261015-071851-76457`. Not-run steps after a failure, `run_2dw1tp84db`, 25 Oct 2026, run `20261025-235120-61245`. A run held by Stop Everything, run `20261014-163527-25223`. Source: verify-silan feature map, test-runs and app-lists.
:::

<figure class="shot"><img src="/media/run-plan-live.webp" alt="A live run: steps 1 to 3 done, step 4 running, about 20 s left, and the timeline filling under the phone." width="1440" height="900" loading="lazy" /><figcaption>A live run: steps 1 to 3 done, step 4 running, about 20 s left, and the timeline filling under the phone.</figcaption></figure>
<figure class="shot"><img src="/media/run-report-failed.webp" alt="A failed run's report: the picture from the moment step 2 failed, the timeline, the steps, and Run again from step 2 at the top." width="1440" height="900" loading="lazy" /><figcaption>A failed run's report: the picture from the moment step 2 failed, the timeline, the steps, and Run again from step 2 at the top.</figcaption></figure>