Blog 4 of 4: Three Days in Orlando
The Setup
The booth setup at Orlando Maker Faire was simple by design. The droid stood at the front, facing the crowd. Behind it, mounted up high and angled outward, was an iPad streaming the live camera feed from the web interface, the same view the droid was using to see the world, with bounding boxes, tracking indicators, and gesture overlays visible in real time. That turned out to be one of the best decisions of the whole event. People would walk past, notice the screen showing a live annotated view of themselves, realize the droid was watching them, and stop. It was a better attention-getter than the droid alone.
The admin page I'd built to control the droid remotely didn't cooperate the way I'd planned, so I ended up SSH-ing into the Nano to start the Python script directly. Not elegant, but it worked, and once the script was running I didn't need to touch it again. The droid ran for most of three days.
What the Crowd Actually Did
Kids treated it like it was alive. That's the only way to put it. Anyone who has ever puppeted R2 at a public event knows exactly what I mean: adults understand on some level that they're looking at a prop, but kids just interact with it. They'd walk up, wave, watch the droid's head follow them, wave again. Some of them would spend several minutes just testing what it would respond to. The LED effects when a gesture was recognized got an immediate reaction every time: the rainbow pulse on a wave, the green flash on a thumbs up. Those moments landed exactly the way I'd hoped they would.
Adults were impressed in a different way. A lot of the conversations were about how it worked, what hardware it was running on, whether it was cloud-connected, how the gesture detection worked. Having the camera feed visible on the iPad made those conversations easier because people could literally see the detection happening. Showing someone a bounding box tracking their head in real time is a faster explanation of computer vision than anything I could say.
What Didn't Work
MediaPipe running on the CPU in a crowded environment was the biggest problem on the day. Gesture detection failed roughly 80% of the time. In a quiet room with one person in frame it works well enough, but at a busy convention with multiple people, movement everywhere, and the system already carrying the load of detection and tracking, the CPU simply couldn't keep up. When gesture detection lagged, tracking suffered too, and occasionally the web interface would freeze. The droid would be watching someone wave at it and just not respond. That's the opposite of the experience I was building toward.
The engagement logic had its own issues. The intent was to have the droid prioritize people who were actively trying to interact: approaching the camera, gesturing, making eye contact. What actually happened in a dense crowd was that the largest bounding box won, which often meant a parent standing behind their kid got locked as the tracking target while the child right in front of the droid was ignored. The logic that works in a controlled test doesn't always survive contact with a real convention floor.
The admin interface not working as intended was a minor frustration rather than a real problem; SSH got the job done. But it's on the list.
What the List Looks Like Now
Coming out of Maker Faire with a list of things to fix is the best possible outcome for a first public run. It means the thing ran long enough and got enough real interaction to surface actual problems. Here's where the focus goes next:
The MediaPipe performance problem has a real solution: migrating gesture detection from CPU-based MediaPipe to a GPU-accelerated PyTorch model. That keeps everything on the GPU pipeline alongside the human detection and should get frame rates back above 20 consistently, even in crowded environments. The architecture was designed with this migration in mind, and it's the right fix.
The engagement logic needs to get smarter about who is actually trying to interact. Bounding box size is a crude proxy for engagement. What I want is something closer to intentionality: is this person facing the droid, are they in the foreground, did they gesture? The convention floor exposed exactly how shallow the current approach is, which is useful information.
The admin interface and startup process will get cleaned up before the next event. Starting a droid via SSH is fine for a developer; it's not a good long-term solution.
Where It Goes From Here
The plan is to take the droid to other local conventions and back to Maker Faire. Each event will generate a new version of this same list, and that's fine; that's how these builds evolve. The R2 I took to my first convention years ago is not the R2 I take to events now. This Pit Droid will go through the same process and evolution.
The development work is moving into a new phase using Cowork, Anthropic's tool that brings the natural language interface of Claude together with Claude Code, a direct code editing environment. The first phase of this project was developed entirely in chat, which worked well for planning, reasoning through problems, and drafting, but had a real friction point: code had to be copied and pasted back and forth between the conversation and the actual files on the Jetson. Changes could get out of sync, context could get lost across long sessions, and getting updated code onto the droid meant a manual transfer process every time.
With Cowork, the code gets edited directly and pushed to GitHub, where the Jetson pulls it down. The development environment has evolved in the same direction the droid has, from something that required a lot of manual handling to something more integrated and capable. There's something fitting about that parallel. The project that started as an experiment in bringing AI into a droid build is now being developed with AI tooling that didn't exist when we started.
There will be follow-up posts as the project develops and as the droid makes its way to future conventions. There's plenty of story left.
Thanks for following along.
