More work · Human–robot interaction
A robot that draws in sand while you steer it with your hand — the design changes during making, not before it.
Year
2023 · CMU
Type
Academic · team project
Course
48-555 Introduction to Architectural Robotics
Stack
OpenCV · Grasshopper · OpenABB · RobotStudio · ABB IRB 120
01 · Description
Most manufacturing runs in a straight line: design, then make. We wanted the designer inside the loop.
A linear workflow leaves no room to change your mind once the machine starts. This project explores smooth collaboration between a person and a robot: the designer can change the design during production, and the robot responds in real time.
The final drawing is never fully planned. It emerges from continuous back-and-forth between hand and machine.

02 · Human–robot workflow
Four pieces of software talk to each other in real time. OpenCV tracks the hand from a webcam; Grasshopper turns the hand into a drawing path and solves the robot’s joint positions; OpenABB passes the commands to RobotStudio, which drives the arm.
Everything is linked over UDP with three sockets, so a change of hand position reaches the robot while it is still drawing.
01 · See
OpenCV hand-tracking
02 · Map
Grasshopper path
03 · Send
OpenABB · UDP
04 · Draw
ABB IRB 120
03 · ABB IRB 120
A small six-axis arm fitted with a custom sand-drawing tool, so a path on screen becomes a groove in the sand.

04 · Interaction
Your hand is the controller. The pattern is a Maurer rose.
The distance between fingertips sets the scale: open your palm and the pattern grows, close it and it shrinks. The tracked points are projected onto a flat plane in Rhino rather than into 3D space.
The angle of the hand sets the number of petals — fewer when the hand lies flat, more as the palm turns upright.

The same rule the robot follows: each slider stands in for one hand measurement.
05 · Demo
Thirty seconds of a live session. Left: the robot over smooth sand. Right: the webcam tracking my hand. As the palm opens and turns, the pattern the robot draws changes with it.
06 · Results
Each tray is one session — the same rule, steered differently by hand.
Next project
Bamboo Weaving →