Duality Arm
Natural-language control for a UR3 robotic arm in simulation — “pick up the blue block” becomes a motion plan.
Duality Arm sits between plain language and a robot. A parser turns an instruction into a typed goal, a grounding step resolves it against the scene, and a planner produces a collision-free trajectory the simulated UR3 executes.
The challenge
Language is ambiguous; robots are not. “The block near the cup” has to resolve to exact coordinates, and any plan that clips the table is a failure, not a suggestion.
I wanted the language layer to fail loudly and safely — ask again rather than guess when the scene is unclear.
Approach
An LLM emits a structured goal (verb, target, constraints). A grounding step matches the target to a detected object; if confidence is low it asks a clarifying question instead of acting.
MoveIt plans the trajectory; PyBullet executes and reports success, so the loop can retry with feedback.
What I built
Language → typed goal
Instructions parse into a structured, checkable goal object.
Scene grounding
Targets resolve against detected objects; low confidence asks, never guesses.
Collision-free plans
MoveIt guarantees the trajectory clears the workspace before motion.
Closed loop
PyBullet reports outcome so failed grasps retry with feedback.
Under the hood
a slice of the core logic1goal = parse(instruction) # {verb:"pick", target:"blue block"}2match = ground(goal.target, scene.objects)3 4if match.confidence < 0.6:5 return ask(f"I see {len(scene.objects)} objects — which one?")6 7plan = moveit.plan(reach=match.pose, avoid=scene.obstacles)8sim.execute(plan) # UR3 in PyBullet, reports success