# Shared-FM 20k / 40k: exact evaluation inputs

This is a frozen 12-case evaluation, not a task-success benchmark. The same selected cases, source pixels, episode times, prompts, states and seeds were used for both checkpoints. It is a different experiment from the later T5 + hierarchy-decoder training.

## Quick start

Install numpy, download `load_inputs.py`, then:

```sh
python load_inputs.py --case case15 --chunk 0 --output case15-first.npz
python load_inputs.py --case case15 --chunk 7 --output case15-gt-chunk8.npz
python load_inputs.py --case case02 --history current_repeat --variant full_stop --seed 314
```

Use `--base /path/to/site` for offline data, or a custom deployed base URL. All binary inputs are checked against SHA256 records. Each `reproduction/<case>/case.json` contains its own asset index; `cases.json` and `cases.csv` summarize all12 cases. Frame assets are split into files smaller than25MiB, with no lossy recoding. Arrays are NumPy `.npz`, no pickle. `initial-input.npz` alone contains the nine initial RGB frames, current state and its normalization for quick first-chunk inference. The full loader additionally downloads all frame packs for GT-conditioned later chunks and evaluation targets.

## Inputs, timeline, outputs

- Episode tick is a **25Hz processed timeline**, not a native source-video frame index. Tick407 means16.28 seconds after episode start. Packed camera videos can start that episode thousands of seconds into the file; each camera has its own offset and native FPS in the case JSON.
- Each call gets9 observations: `[episode frame0, max(0,t-175), max(0,t-150), ..., max(0,t-25), t]`. Repeated frame0 slots for early starts are intentional. The first item is a fixed episode anchor, not the start of a continuous7-second clip.
- Pixels are exact inference composites: uint8 RGB `[9,240,416,3]`. The model uses float32 `[3,9,240,416]` scaled as `RGB/127.5-1`. Use supplied pixels for exact comparison; MP4s are display-only. The primary view occupies the left640x480 area of an880x480 canvas. Center-cropped left/right wrists occupy upper/lower240x240 cells on the right. The complete canvas is bilinear-resized to416x240.
- Per call, predict8 future RGB frames at `t+4,t+8,...,t+32` and32 future EE16 states at `t+1,...,t+32`. That is1.28 seconds. The displayed1/3/8 chunks cover1.28/3.84/10.24 seconds.
- EE16 is `[left xyz, left qw qx qy qz, left gripper, right xyz, right qw qx qy qz, right gripper]`. XYZ is metres; gripper retains source-native units. Use `valid`: some pose/gripper channels are missing or unverified and must not be scored. This evaluation targets **future observed state**, not raw actuator commands. Each robot's source coordinate convention is retained; no shared global coordinate frame is asserted.
- Current proprio is `where(valid, (state-mean)/std, 0)` using supplied per-tree training-split statistics. Predictions are de-normalized using `pred*std+mean` for scoring. Return normalized or physical outputs explicitly when comparing models.
- Exact prompts are stored per chunk. These checkpoints used frozen UMT5 128-token context, full L3/L2/L1 prompts, Video10 steps -> clean latent cache -> Action10 steps, inference shift3. Other models should use the same raw text with their own encoder and disclose any truncation or conditioning differences.
- Rollout seed is `42 + selected_case_ordinal*100 + chunk_index`, **not** the number in `case15`. Each JSON lists the actual seed. Diagnostic seeds are42 and314. Equal seeds across architectures are a reproducibility convention, not identical sampled noise.

## GT feedback versus generated feedback

GT feedback uses recorded real history and current state at every chunk. `load_case(...,chunk=k)` provides those exact real inputs.

Generated feedback shares only the initial observations/state. For every later requested history tick after the initial current tick, use the tested model's latest generated RGB at or before that tick. Before any generated sample exists, hold the initial current RGB. Keep real frame0 and real pre-start history. `generated_feedback_history` implements this rule. Proprio is the previous chunk's last predicted normalized EE16, masked with the initial valid mask. Prompts still follow the recorded annotation schedule. Do not feed GT future pixels/states from the download into generated-feedback inference. This is an offline autoregressive sequence, not a real robot/environment rollout.

## Interventions and fair scoring

Diagnostic variants use only the first chunk and the exact prompt in `case.json:diagnostics`. `current_repeat` keeps the episode anchor and repeats the current image in the other8 slots. `reverse_past` keeps anchor/current and reverses the seven past slots. State is held fixed. The page's checkpoint-specific views compare original and changed conditions; the main page compares20k and40k under the same condition.

Score video PSNR from uncompressed RGB, not browser MP4s. Score action XYZ RMSE in millimetres over valid dimensions `[0,1,2,8,9,10]`. The existing page reports pooled per-case predictions and then the original protocol summaries. Prompt edits have no executed counterfactual GT; motion sensitivity is not evidence of semantic task success. Do not tune new models on these cases and call them an untouched generalization benchmark.

All requested source-camera timestamps/native frame indices, episode source IDs, eight prompts, initial states, normalization arrays and binary SHA256s are included. Downloaded ground truth is provided for scoring and real-observation feedback only.