Teaching a Franka Research 3 to Pick and Place

The project in one sentence

This is an end-to-end pipeline for teaching a physical Franka Research 3 arm a task from demonstrations: record the task being done, turn those recordings into training data, fine-tune a pi0.5 vision-language-action policy on them, and then let that policy control the real robot.

The task itself is intentionally modest. The arm picks a cube up from a cell on a table and drops it into a basket. What makes the project interesting isn't the task, it's everything that has to go right around it before a learned model can be trusted near real hardware.

Why the model is the easy part

In simulation you can reset the world perfectly, record every value you like, and crash the robot as often as you want. None of that is true on a real arm. The cameras have to agree with the robot about where things are. Every demonstration has to begin from the same starting conditions, or the model learns the noise. The recorded images, joint states, and commands have to line up in time. And the policy only makes a decision fifteen times a second, while the arm's controller expects a new command every millisecond.

Most of the work in this project lives in those gaps. The 1 kHz control loop isn't an achievement of mine; it's a hard constraint of the hardware. The engineering problem was making slower, learned decisions safe to execute inside it.

How the pipeline fits together

Data collection comes first. I built Apple Vision Pro teleoperation to capture synchronized joint states, actions, camera views, and gripper commands, but the released dataset uses a scripted collection routine instead of a human operator. Before each recording, the robot resets the scene itself: it takes the cube out of the basket, places it on the chosen cell, and returns to the same start pose. Only then does it record itself picking the cube back up. Two RealSense cameras, the robot's state, and the commands it sent are all saved together, and failed attempts are kept apart so they never end up in training.

Those recordings are checked for completeness, audited so evaluation cells don't leak into training, and converted into a LeRobot dataset in the action format pi0.5 expects. Training happens on a GPU cluster. The resulting policy runs on a separate GPU machine, and the robot's workstation asks it for actions over the network, then smooths and bounds them before they reach the arm.

Nothing moves until a chain of read-only checks has passed: the environment, the connection to the policy, the policy's behaviour on live camera frames without executing anything, and its agreement with recorded demonstrations when replayed offline. Only after that does a guarded live trial run, with a height limit above the table and the emergency stop within reach.

What works well

• Consistency. Letting the robot reset its own scene means every demonstration starts the same way, which takes a whole class of data problems off the table before training even begins.
• Treating raw recordings as the source of truth. The training dataset is derived from them and can always be rebuilt, so a fix to the conversion step never means re-recording anything.
• Separating tools by whether they move the robot. It makes the safe checks the natural first step, instead of something you have to remember to do.
• Releasing the artifacts. The dataset and checkpoint are public on Hugging Face, and the evaluations are on video, so the work can be checked rather than taken on trust.

Where it falls short

It is still one task in one workcell. The calibration is specific to this table, camera placement, and robot, and the documentation is explicit that it can't simply be reused on another setup. A policy trained here has learned this corner of the lab, not pick-and-place in general.

Scripted demonstrations are a trade-off too. They buy consistency, but they are less varied than a person would be, and variety is part of what helps a policy recover when something unexpected happens.

And the live runs are guarded rather than fully autonomous. The policy decides how to move the arm and when to grip or release, but the trial runner still executes and verifies the physical gripper commands and handles the turnover after a confirmed release. That's the right call for safety at this stage, but it's worth being clear that the learned policy isn't running the entire loop unassisted.

Where I want to take it

The evaluation page already includes a bracket assembly run alongside pick-and-place, and harder, multi-step tasks like that are where the pipeline has to prove itself next. The other direction is loosening the dependence on a single calibrated workcell, so the collection and validation process can move to a new setup without starting from scratch.