In-Browser Object Detection with RF-DETR on AWS Amplify

The Problem

I built a “magic mirror” web app for my kids: they hold objects up to the camera and the app labels them out loud in French. Two constraints made the obvious architecture wrong. A GPU inference endpoint would idle 99% of the time for a family toy — and more importantly, I refused to ship camera frames of my children over the network. I needed real-time-ish object detection where no image ever leaves the device, hosted for pennies.

The Solution

Run the model in the browser. RF-DETR Nano (Roboflow’s detection transformer, Apache 2.0, COCO 80 classes) exported to ONNX, quantized to int8, executed by ONNX Runtime Web on WASM — served as static files from AWS Amplify Hosting.

The solid path is the whole product: the browser downloads the app and the 36 MB model once, then everything runs locally at ~390 ms per frame — 1–2 detections per second, indistinguishable from instant for a kid holding up a teddy bear, especially with an EMA-smoothed overlay and a 600 ms label hold. Inference cost: $0. Privacy: by construction, not by policy.

From PyTorch to the browser

Three steps, one contract:

1. Export to ONNX (opset 17) and validate against a reference image in Python before touching any UI. Keep that image — it becomes your QA oracle.
2. Quantize to int8: 115 MB → 36 MB, and roughly 2× faster on CPU, with no visible quality loss on my living-room test set.
3. Preprocess to the pixel: squash resize to 384×384 (bilinear, no antialias — canvas.drawImage happens to match), ImageNet normalization, NCHW. Outputs are normalized cxcywh boxes plus raw logits you sigmoid yourself, indexed by COCO’s sparse ids. Any mismatch produces the same symptom: plausible-looking garbage.

Two Traps Worth the Whole Article

ort.env.wasm.wasmPaths must be an absolute URL. ONNX Runtime Web loads its WASM backend through a dynamic import() of an .mjs file. A bare relative path ("vendor/") is not a valid module specifier, so the import fails — and the failure cascades: WebGPU dies first, then the WASM fallback dies on previous call to initWasm() failed, and both execution providers look broken. One line fixes it:

ort.env.wasm.wasmPaths = new URL("vendor/", location.href).href;

An execution provider can initialize perfectly and detect nothing. WebGPU (JSEP) initialized without a warning on my int8 model, ran at a healthy pace… and returned zero detections, every frame — on images where WASM found a dog at 0.84. Something in the GridSample/QDQ path fails silently; the numbers are just wrong. The generalizable rule:

Never trust an execution provider because it initialized. Assert actual detections against a reference.

The app ships a ?qa=1 hook that skips the camera, runs inference on the bundled reference image, and exposes window.__DETECTO_QA for Playwright to assert (dog ≈ 0.84, person ≈ 0.77, per forced provider via ?ep=). That harness caught the WebGPU issue in minutes. WASM stays pinned until WebGPU passes the reference check.

The Amplify Details That Matter

COOP/COEP custom headers doubled the frame rate. Threaded WASM requires crossOriginIsolated, which requires two headers:

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp

Measured on the same laptop: ~390 ms/frame with them, ~820 ms without. Four lines of Amplify custom-header configuration for a 2× speedup. (COEP constrains cross-origin embeds — audit what your page loads; Google Fonts passes.)

Content types. .wasm must arrive as application/wasm, and .onnx/.mjs files must not be swallowed by an SPA rewrite rule returning index.html for unknown extensions. The failure mode — an HTTP 200 serving text/html where a binary should be — is easy to miss and baffling to debug. Manual zip deployments serve everything correctly; if you use SPA redirects, exempt onnx|wasm|mjs.

Optional: One Snapshot to Bedrock, Behind a Parental Gate

The one feature that sends an image anywhere is a “What’s that?” button: a single snapshot goes to Claude Haiku 4.5 (vision) on Amazon Bedrock, which answers like a friendly mirror talking to a five-year-old. The path is serverless and never public: Cognito identity pool (guest) → AppSync with IAM auth (SigV4) → Lambda → Bedrock Converse. The parental code gating it lives in SSM Parameter Store as a SecureString and is compared server-side, constant-time, before any model call — a kid hammering the button costs zero invocations. If the backend config isn’t deployed, the button never appears and the app stays 100% local.

Key Takeaways

• In-browser inference inverts the cost and privacy model: $0 per frame, nothing to secure server-side, offline-tolerant after first load. The price is throughput — know whether 1–2 detections/s is enough for your UX before committing.
• Validate the preprocessing contract in Python first, then treat the browser port as a faithful reimplementation of that contract.
• EP init success means nothing — build a reference-detection assertion into your QA before you trust any execution provider, and re-run it per provider on every runtime upgrade.
• COOP/COEP on Amplify is the cheapest 2× you’ll ever get for WASM workloads.
• Add server-side AI only where it earns its keep, behind IAM-authorized paths — never a public endpoint.

The pattern generalizes to kiosks, in-store demos, accessibility tools, and field apps on flaky connectivity — anywhere “good-enough FPS, zero infrastructure, private by construction” beats “maximum FPS, GPU fleet”.