ISP → CVML Bench

RESEARCH / DEMONSTRATION — SYNTHETIC DATA. The aggregate half of the Camera ISP lab: instead of one frame at one setting, this runs the pipeline dozens of times and asks the four questions a deployment actually turns on. WHAT IS EACH STAGE BUYING? Ablate switches every ISP stage off in turn and reports the cost in image quality and in detection side by side, and names any stage whose removal makes the picture measurably worse and detection measurably better — or the reverse. WHERE IS THE OPTIMUM, AND WHOSE? Sweep walks one numeric parameter and plots SSIM against detection F1 on a shared axis, with the argmax of each metric reported separately plus how much F1 it costs to tune for image quality instead. On the denoise radius with an unconditional blur, SSIM, PSNR and two of the three detectors all peak at different values — that chart is the whole point of the exercise. DOES IT SURVIVE THE FIELD? Robustness holds the pipeline fixed and varies the imaging condition, as a condition-by-detector grid, and names the weakest cell rather than reporting a mean, because a mean over conditions is exactly the number that hides the one that fails. WHAT DOES IT COST? Time takes a median per stage over a warmed loop, and Budget models the same pipeline against a stated GPU roofline from each stage's declared operations and memory traffic per pixel — and the architecture diagram is built from that same table, so it cannot drift from the code. LIMITS, STATED RATHER THAN DISCOVERED: EVERYTHING THE LAB PLATE SAYS ABOUT ITS FIXTURE APPLIES HERE UNCHANGED. The scene is procedural and contains nothing from any real imaging system; the detector is a SURROGATE and no convolutional network runs; ground truth places a proposal and never enters its score. BROWSER JAVASCRIPT MILLISECONDS ARE NOT CUDA MILLISECONDS. Time reports wall-clock single-threaded JavaScript on one CPU core. It is useful for ranking stages against each other — and the measured ranking does agree with the modelled one about which stage dominates, which is a real cross-check — but it is not an estimate of what a GPU would do. THE GPU FIGURE IS A MODEL AND SAYS SO IN ITS OWN RESULT. It multiplies each enabled stage's declared operations and bytes per pixel by the pixel count and divides by a stated device ceiling, taking whichever of compute and memory is worse. It ignores occupancy, kernel launch overhead, cache behaviour, and every fusion a real implementation would perform; the neighbourhood reads it charges a bilateral filter for are mostly cache hits on real hardware, so it is deliberately pessimistic there. A fused implementation beats this model comfortably and a naive pass-per-stage one does not. The CUDA listing is ILLUSTRATIVE SOURCE showing what the tiled form of one stage looks like — it is not the code that produced any timing on this plate, it is not compiled, and nothing here runs on a GPU. THE SWEEPS RUN ON THE MAIN THREAD. A forty-run sweep is a few seconds of arithmetic and the tab will stutter; it yields between runs so a readout can paint, and it runs at whatever resolution the lab is set to, so use Preview for exploring and Measure for a figure you intend to quote. A worker would fix the stutter and would also make every one of these actions unreachable from the headless action API, which is what makes them verifiable, so it is deliberately not used. AP AND F1 CAN MOVE IN OPPOSITE DIRECTIONS along a sweep and the result says so when they do: AP measures ranking over every proposal while F1 measures one fixed operating point, so a setting that degrades objects and background together keeps the ranking intact and pushes every absolute confidence below the cut. That is precisely why a deployed detector with a frozen threshold fails long before its published AP suggests it should. Comparing MTF50 between resolutions is meaningless — it is in cycles per pixel. No training, no real model, no real RAW.