# Operator tools for the Mariano's Part A probe on Azure

These run outside the reviewed package. They start the VM, put the reviewed package on it, run it once through
`infra/run_probe.ps1`, export the evidence, and deallocate the VM. None of them contains a credential.

| File | What it does |
|---|---|
| `vm_probe_session.tpl.ps1` | The script the VM runs (Azure action Run Command, as SYSTEM). The orchestrator fills its placeholders: version, package SHA-256, launcher SHA-256, expected test count, jobs cap, executions already done. |
| `probe_orchestrator.py` | One process on the operator's machine, with a bounded overall deadline (v2.2.16, below). It starts the VM, uploads the package and reads it back by SHA-256, runs the session script, downloads the evidence and checks the manifest, reads Log Analytics, and in `finally` deallocates the VM with retries and confirms the power state. |
| `test_orchestrator.py` | Offline tests of the orchestrator: virtual clock, fake Azure. A session that never ends, a failed start, a deallocation that never completes, an unreadable power state. |
| `azlite.py` | A minimal Azure REST client (standard library only, device-code sign-in). Its token file stays on the operator's machine and is never shipped. Read-only exports: `blob-get`, `la-export` (complete query result as JSON + CSV), `activity-log`. |

v2.2.16 removed the encrypted credential hand-over (`cred_send.ps1` and its code in both scripts). The Oxylabs API user
comes only from Key Vault through the VM's managed identity.

## What the session script does on the VM

1. It reads `oxylabs-username` / `oxylabs-password` from Key Vault `shif1021-kv-hfpzua5n` with the VM's managed identity, into memory only. If the read fails or a secret is empty, it stops with exit 5, writes the Key Vault error (HTTP status and message) to the session log and as Application event 1202, and sends nothing. There is no other route.
2. **Guard against a second execution:** it counts every `C:\marianos\probe-*\scraper\evidence\oxylabs-probe-*` folder on the VM and stops if more exist than the executions already accounted for.
3. It downloads `deploy/marianos-scraper-<version>.zip` from Blob, checks its SHA-256 against the reviewed hash, and unpacks it into a new folder `C:\marianos\probe-<version>-<UTC stamp>`. It stops if that folder already exists, so no earlier folder is reused or removed. It checks the SHA-256 of `run_probe.ps1` against the reviewed launcher hash.
4. It creates an isolated venv with `C:\Python313\python.exe`, installs `requirements-dev.txt` (pinned), and runs `pytest`. The probe starts only if the exact expected number of tests passed.
5. **Cumulative allowance:** it passes the ledger blob to the launcher as `-Ledger blob:<url of probe/cumulative-usage.json>`. The probe itself leases the blob, reconciles unfinished reservations, reserves its cap with If-Match and verifies it by read-back before any request, and records the final count the same way (see `docs/TRANSPORT_OXYLABS.md` in the package). The script only reads the ledger for its log, before and after; it never downloads, keeps or uploads a copy.
6. It runs `infra\run_probe.ps1` once: Vernon Hills, the five UPCs and roles, `-MaxRequests <cap>`, `-AzureImds`. The 1,200 s deadline, 1,250 s in-process limit and 1,320 s outer kill are unchanged.
7. It uploads the evidence with `infra\upload_probe_evidence.ps1` (Blob read-back by SHA-256 against the manifest), writes Application-log events 1200 (start) and 1201 (end), with 1100 written by the upload script, and uploads the session log.

## The two session-script bugs of Oct 30 (attempts 1 and 2: stopped before the probe, 0 Oxylabs jobs)

**1. PowerShell variable collision (attempt 1).** PowerShell variable names are case-insensitive, so `$Tests` and `$tests` were the same variable:

```
$Tests = "72"                                   # expected count (template)
...
$tests = (pytest output, last line)            # overwrote $Tests with "72 passed in 9.60s"
if ($tests -notmatch "^=*\s*$Tests passed") ... # pattern became "...72 passed in 9.60s passed": never matched -> STOP
```

Fixed: the expected count is `$ExpectedTests`; the result stays `$tests`.

**2. Unset timestamp (attempt 2).** The folder name used `$Stamp` one line before `$Stamp` was assigned:

```
$Root = "C:\marianos\probe-$Version-$Stamp"     # $Stamp still empty -> "C:\marianos\probe-2.2.14-"
$Vault = "..."
$Stamp = (Get-Date).ToUniversalTime().ToString("yyyyMMddTHHmmssZ")
```

Attempt 1 had already created `probe-2.2.14-`, so `Expand-Archive` refused to overwrite it and the script stopped. Fixed: `$Stamp` is assigned first. The script also now stops if the folder already exists, instead of relying on `Expand-Archive`.

Before attempt 3, the filled session script was dry-run end to end on a Windows machine. Azure, Key Vault, Blob and the launcher were replaced by local stand-ins; hashes, unpacking, the venv, the pinned install, the real test run and its check, and the ledger round trip ran for real. The same dry run passed for v2.2.15: 86 tests, launcher received `-MaxRequests 14` and `-Ledger`, ledger read-back OK.

## The three-consecutive-failure rule (package, v2.2.15)

A provider failure is any of these: no answer from Oxylabs; an outer Oxylabs 429, 5xx or 524; a nested provider code (15001, 15002, 15003, 612, 613, or any non-HTTP code). Each one increments `consecutive_failures`. Only a retailer answer, of any status, resets it to 0. At 3 the run stops with exit 2.

Each failure also uses one attempt of that request's retry budget, one job of the request cap, and one job of the cumulative ledger. A failed session (15001, 15002, 15003, 613 or an unknown code) is replaced and never sent again. 612 is retried on the same session, because Oxylabs documents it as a free retry of a faulted job. The following stop at once without counting toward the three: Oxylabs 401/403 (access), Oxylabs 400/422 (payload, including "session is failed"), a retailer 401/403 or challenge, and a store or fulfilment mismatch.

## Time limits of the orchestrator (v2.2.16)

| Phase | Limit | When it is reached |
|---|---|---|
| Whole session before deallocation | `--session-deadline`, default 3,600 s, monotonic | stop waiting, go straight to deallocation, exit 2 |
| VM start | 900 s, and never past the session deadline | session failed, deallocation |
| Run Command polling | the session deadline | the probe still running on the VM is ended by the deallocation; its ledger reservation stays counted in full |
| Each HTTP call | 120 s timeout, at most 4 tries | the error is reported; the deadline still applies |
| Deallocation | 3 attempts × (600 s operation poll + 300 s power-state confirmation) | exit 3, `VM NOT CONFIRMED DEALLOCATED` with the manual command |

Exit codes: 0 = session finished and the VM confirmed deallocated; 2 = the session failed or reached its deadline, VM
confirmed deallocated; 3 = the deallocation could not be confirmed. An unreadable power state never counts as
deallocated. Worst case from start to exit: the session deadline plus about 45 minutes.