From Hackathon to Cloud Pipeline: Inspecting Physical Parts with a Humanoid and AgentCore

I have written before about teaching a robot to pick up a cube in simulation and about the cloud plumbing behind Physical AI. Those posts were, respectively, about a simulated arm on my laptop and about the general architecture of a robotics workload. This one is different. This one had a real humanoid, on a real floor, for two real days.

We spent a hackathon building an autonomous inspection agent on a full-size humanoid robot. The use case was deliberately concrete: walk a route, look at a physical assembly, and check whether what is actually there matches what the bill of materials says should be there. Missing parts, extra parts, wrong parts. That is the whole job, and it is more interesting than it sounds.

This is the build log. What worked, what we had to fake, and the one architectural decision that I think actually generalizes.

The problem is not classification

Let me kill the obvious misconception first, because it shaped everything.

The hard part of this use case is not “identify what is in the photo.” Modern vision models are good at that. The hard part is comparing what the camera sees against a structured expectation, item by item, and producing a verdict a human can act on. The assembly in front of the robot is supposed to contain a specific set of components in a specific configuration. The question is not “what is this?” It is “what is missing, what is extra, and what is in the wrong place?”

That reframing matters because it tells you where the intelligence has to live. A pure image classifier does not have the bill of materials. It cannot reason about absence. You need something that holds the expected state, looks at the observed state, and does the diff. That is an agentic task, not a classification task, and it is why the design ended up centered on an agent rather than a model endpoint.

The split: cloud handles judgment, the robot handles movement

The single decision that organized the whole build was where each model runs.

Robotics models — navigation, spatial reasoning, anything that has to react to the physical world in real time — belong on the robot. Network round-trips are not acceptable when the thing deciding whether to take a step is on the other end of a WiFi link that might drop. Those models get trained in the cloud and deployed on-device.

Judgment models — the defect detection, the bill-of-materials conformity check, the “is this assembly correct” verdict — belong in the cloud. They are not latency-critical in the same way. A verdict that arrives two seconds after the photo is fine. And keeping them in the cloud means you can update the inspection logic without touching a single robot.

The mantra we kept repeating: the cloud handles judgment, the robot handles movement. It sounds glib, but it is a genuinely useful partition. Every time we were unsure where a piece of logic belonged, that sentence answered it.

The cloud pipeline

Here is what the inspection loop actually looked like once a photo was captured:

   Humanoid (on-device)
        | capture photo
        v
   +----------------+
   |  Amazon S3     |  raw inspection image lands in a bucket
   +--------+-------+
            | object-created event
            v
   +----------------+
   |  EventBridge   |  routes the event
   +--------+-------+
            v
   +----------------+
   |  Lambda        |  reads the bill of materials for this station
   +--------+-------+
            | invoke agent with image + expected BoM
            v
   +--------------------------------+
   |  AgentCore (Strands + vision)  |  compares observed vs expected
   |  returns a structured verdict  |
   +--------+-----------------------+
            | verdict JSON
            v
   +----------------+
   |  DynamoDB      |  stores the result per station
   +--------+-------+
            v
   +----------------+
   |  Admin console |  operator reviews flagged stations
   +----------------+

Nothing here is exotic, and that is the point. Photo lands in S3. S3 emits an object-created event. EventBridge routes it. A Lambda function looks up the bill of materials for the station being inspected and invokes an agent, handing it both the image and the expected component list. The agent — a Strands agent running on Bedrock AgentCore with a vision-capable model — does the comparison and returns a structured verdict: conforming, or a list of discrepancies. That verdict goes into DynamoDB, keyed by station, and a small React admin console lets an operator review anything that got flagged.

The entire thing was serverless, defined in AWS CDK, and stood up inside two days. That last fact is worth sitting with. The cloud side of a physical inspection system is now a weekend of work. The robot is the hard part.

Local tools: moving versus perceiving

On the robot side, the cleanest thing we did was separate the tools into two categories, and I would do it again on any embodied agent.

Tools that move the robot change the physical world and carry risk. There were three: a preset-motion tool for arm gestures, a walk-a-bounded-distance tool that moved the robot forward in a closed loop with a hard cap per call and a LiDAR-based stop, and a turn-in-place tool that pivoted the robot by a fixed increment. Every one of these is a mutation. Every one has a safety envelope baked in — the walk tool will not go more than a few meters per call, and it stops if the LiDAR sees something.

Tools that perceive are read-only. Capture a photo from the head camera. Perceive-front, which returns a photo plus a chest LiDAR point cloud projected and colored by distance. Perceive-ground, which returns head RGB-D data for the floor a few meters ahead. And a capture-and-upload tool that pushed an image to S3 to kick off the cloud pipeline above.

The reason this split is worth enforcing: it maps exactly to the blast radius. A read-only perception tool can be retried freely, called speculatively, and composed without much thought. A movement tool needs confirmation semantics, rate limits, and a stop condition. When you hand an agent a flat list of tools that mixes the two, you lose that distinction and you are one hallucinated tool call away from a robot walking into a wall. Treating “move” and “look” as different classes of capability is the embodied-agent equivalent of separating reads from writes.

The perception gap we had to code around

Here is the honest part. The robot’s native perception module was not available to us during the hackathon. That sounds like a disaster and it nearly was, but it forced a decision that turned out to be interesting.

Instead of a dedicated perception stack producing clean object detections, we let the agent reason directly over raw sensor data — the camera image plus the LiDAR point cloud — to locate the assembly and estimate distances. The vision model became the perception layer. It is not as precise as a purpose-built detector, and I would not ship it that way. But it worked well enough to prove the loop, and it made something clear: for the coarse “where is the thing and roughly how far” question, a capable vision model reasoning over fused sensor data is a viable stopgap. For fine navigation, that is exactly where a proper Vision-Language-Action model would slot in.

Where a VLA fits, and which one

That navigation gap is where the current wave of Vision-Language-Action models becomes relevant, and it is worth being precise about the trade-offs, because they are not all the same.

The multi-frame approach we looked at reasons over a short history of frames rather than a single snapshot, which helps it stay oriented as the robot moves. It outperformed the single-frame alternative in our reading, and if you are navigating a cluttered floor, temporal context is not a luxury.

The more interesting recent entrant is a single-camera VLA that predicts the pixel coordinates of where it should go — navigation by pointing — rather than depending on depth or a multi-camera rig. Trained entirely in simulation, it generalizes across wheeled, legged, and flying platforms because the pointing formulation is robust to camera intrinsics and scale. For a build like ours, that is attractive: one RGB camera, no LiDAR dependency for the navigation model, and a formulation that does not care much what body it is driving.

The architectural point is that navigation is a pluggable module. Whether you use a multi-frame depth-aware model or a single-camera pointing model, it slots into the same place: a tool the agent calls to get from one inspection station to the next. The inspection logic in the cloud does not change. That separation — navigation as a swappable module behind a stable interface — is the thing I would carry into any production version.

The agent-on-the-robot idea

The narrative we landed on by the end was not “a robot that inspects.” It was “an agent that happens to have a body.”

Concretely: a Strands agent running on the robot itself as an edge component, reasoning with a cloud model, deployable from the cloud to a fleet the same way you push any other edge workload. The robot’s capabilities — move, turn, capture, perceive — become composable tools the agent orchestrates. Onboard a robot’s capabilities once, and any inspection behavior you can describe becomes a matter of composing those tools, not rewiring the system.

I want to be careful not to oversell this, because at hackathon scale it is a sketch, not a product. But the direction feels right. Today, most robot-plus-AI solutions are hand-wired: a specific model bolted to a specific robot for a specific task, rebuilt from scratch when any of those three changes. An integration layer that turns robot capabilities into agent tools would decouple them. That is a real problem worth solving, and the hackathon convinced me the agent-as-orchestrator framing is a plausible way in.

An honest recommendation

A two-day hackathon proves a loop, not a product. Be clear-eyed about the difference.

What we proved: the cloud side of a physical inspection system — capture, route, reason, verdict, review — is genuinely easy now. Serverless primitives plus an agent on Bedrock got us a working bill-of-materials conformity check in two days. If you are evaluating whether the cloud architecture for a Physical AI use case is the risky part, it is not. That part is solved.

What we faked, and what you cannot fake in production: reliable perception and reliable navigation. We reasoned over raw sensors with a vision model because the real perception stack was not available, and we kept the walking distances short and bounded because fine autonomous navigation was out of scope. In production, those are the hard problems, and they are on the robot, not in the cloud. A vision model reasoning over a point cloud is a fine demo and a poor foundation for a robot moving unsupervised around people.

So if you are considering a build like this, my advice is to invert the usual instinct. Do not spend your time on the cloud pipeline — that is a known quantity you can assemble quickly. Spend it on the two things a hackathon lets you skip: how the robot knows where it is, and how it moves safely when the network drops and no one is watching. Get the movement-versus-perception tool split right from day one, treat navigation as a swappable module behind a stable interface, and keep judgment in the cloud where you can fix it without recalling a fleet. The demo is the easy 80 percent. The last 20 percent is the entire reason robots are hard.