Skip to content

Lab notebook

Dated reports of experiments, design decisions, observations, and lessons learned.

The blog is intentionally chronological. For stable procedures and reference material, use the documentation sections in the navigation.

Experiment v8: wide camera randomisation and an explicit camera description (negative result)

Idea

The first test of the v7 policy on the real SO-101 showed that the real camera stood outside the narrow range of camera positions used in training (closer, and looking down more steeply). Requiring the real camera to match the simulator is impractical, and it becomes impossible once the camera is moved by the robot itself (planned active-perception experiments). Experiment v8 therefore tried to make the policy independent of the exact camera placement, in two ways:

  1. Wide randomisation. Each training episode uses a different stereo camera position, orientation and field of view, so that the policy must cope with a broad range of viewpoints.
  2. An explicit camera description. Model B additionally receives ten numbers describing the camera (position, viewing direction, image-up direction, field of view), as it would obtain them on the real setup from a calibration. Model A receives no such information and must infer the viewpoint from the images and joint angles. Comparing A and B was meant to show whether explicit calibration is worth building.

The experiment also added a model of the real servos' behaviour, measured on the 7.4 V arm.

Result in brief

v8 is a clear failure. Both models grasp successfully in only 13–20 % of scenes from their own training range of viewpoints and 9–11 % on held-out viewpoints, against 79 % for the narrowly trained v7 policy. Follow-up diagnostics show that the broad camera randomisation is the cause: the v8 policy lost the precise positioning that v7 had learned, even for viewpoints like those of v7. Training length and the servo model are not the cause. Because both models fail, the experiment cannot answer whether the explicit camera description helps.

Setup

All v8 results are from simulation; the v8 policies were not tested on the real arm. The simulated setup is the v7 scene, modelled on the real one:

  • Arm and workspace: SO-101 on a 50×49 cm plywood board (pan axis 7 cm from the back edge), on a desk with background clutter; textures made from photos of the real board, desk and room.
  • Camera: a stereo pair modelled on the real side-by-side USB stereo camera (6.2 cm baseline, rectified, 640×360 per eye), the only camera; no wrist camera. The real camera, with its original 3.6 mm lenses, sees about 44° horizontally after rectification; its measured pose is 52.5 cm in front of the pan axis, 31 cm above the board, looking down 38.7°.
  • Servos: a model of the 7.4 V follower arm. Measurements on the real arm showed that shoulder_lift and elbow_flex stop 2–4.5° short of their target when moving in one direction and arrive within about 0.5° in the other (a dead band that grows with the gravity load). The simulator reproduces this (about 8–10° per N·m of gravity torque, scaled ×0.5–1.5 per episode), adds a command delay of 0–2 control steps, and uses ideal servos in 15 % of episodes.
  • Task: one cube (22–35 mm) or ball (24–40 mm), 12 colours, 12–24 cm from the pan axis within ±30°, instruction such as "pick up the red cube".

Data

One set of about 2,000 generated episodes was exported twice, as two datasets with identical episodes: so101-sim-grasp-v8a and so101-sim-grasp-v8b. Only successful demonstrations were kept (roughly 1,740), of which 87 were held out for evaluation.

  • Camera randomisation per episode: azimuth −45…+30° around the front of the robot, 0.45–0.85 m from the pan axis, 0.15–0.60 m high, 15–55° looking down, roll ±4°, horizontal field of view 42–68° (covering the current lenses and the 2.8 mm and 2.1 mm lenses ordered for the real camera). A camera was accepted only if the robot base, the gripper's ready pose and the object area were visible in both eyes; objects were placed only where both eyes see them. Viewpoints with azimuth beyond +30° or farther than 0.85 m were excluded from training and kept for testing.
  • Demonstrations: scripted expert as in v6/v7 (lift as soon as the grip is secure, sideways perturbations in half of the episodes). Under the servo model, the expert needed an explicit anticipation of the dead band; without it, its own success rate fell from 97 % to 87 %. With it, the expert succeeds in 97 % of v8 scenes, with or without the servo model.
  • State input: v8a: 6 joint values; v8b: 6 joint values plus the 10-number camera description.

Policies

Two SmolVLA models were fine-tuned from lerobot/smolvla_base with identical settings (40,000 steps, batch size 64, about 16 hours each on an RTX 3090): smolvla-so101-sim-v8a (model A, images and joint angles) and smolvla-so101-sim-v8b (model B, images, joint angles and camera description). Both receive the left and right images as two camera inputs and re-plan every 10 steps.

Evaluation

Each run uses the same 187 scenes (87 held out from the dataset, 100 new); success means the object is held between both jaws and lifted at least 3 cm. "Training range" uses cameras drawn from the training distribution, "held out" uses only the excluded viewpoints. In the last run model B was given a camera description that was wrong by 2 cm in position and 3° in orientation.

Model Viewpoints Success Median distance gripper–object at closing
A training range 25/187 (13 %) 4.3 cm
B training range 37/187 (20 %) 3.7 cm
A held out 17/187 (9 %) 5.2 cm
B held out 20/187 (11 %) 4.9 cm
B, description 2 cm / 3° wrong held out 19/187 (10 %) 5.0 cm
v7 (narrow camera range, for reference) v7 scenes 79 % 1.9 cm

Almost all failures are positioning errors: the arm moves towards the right object but closes 3–5 cm away from it, while grasps generally need less than 2 cm. Failures are spread evenly over camera distance, height, azimuth, pitch and field of view, and over episodes with and without the servo model. Model B is somewhat better than A within the training range (28 scenes gained, 16 lost, p ≈ 0.1), which is suggestive but not conclusive; on held-out viewpoints there is no difference, and the wrong camera description changes nothing.

Three diagnostics on 50 shared scenes separated the possible causes:

Diagnostic Result Interpretation
Model A at 20k / 30k / 40k training steps 7 / 7 / 5 successes, median distance 3.6–4.1 cm no improvement after 20k steps: longer training will not help
Model A on v7 scenes (narrow cameras, ideal servos) vs the v7 policy 5 vs 40 successes (35 scenes won by v7, none by v8a), 5.0 vs 1.1 cm the broad training destroyed precision even under v7's own conditions
Model A with the servo model switched off 2 vs 5 successes, 5.1 vs 4.1 cm the servo model is not the cause

Conclusions

With about 1,650 training episodes and a policy that sees a single frame at a time, spreading the training over this range of camera positions and fields of view left too few examples per viewpoint to learn precise positioning anywhere. The early plateau suggests that the limit is the data and the breadth of the task rather than the training time. Whether a much larger dataset would recover the precision remains untested. The question that motivated the experiment, whether an explicit camera description helps, remains open: it needs a policy that positions precisely in the first place.

The next experiment (v9) narrows the camera range to a band around the measured real camera pose and the lens actually in use, keeps the servo model, and repeats the A/B comparison under conditions in which the policies can succeed.

Research artifacts

Datasets (Hugging Face, LeRobot format):

Models (Hugging Face):

Code (GitHub): Koziev/so101-lab/experiments/exp8: simulator, data generation and LeRobot export, training and evaluation scripts.

SO-101 grasping with SmolVLA trained on simulated data: experiments v1–v7

This entry summarises the first seven experiments of a project that trains a small vision-language-action model to grasp objects with an SO-101 arm, using only synthetic trajectories from a simulator. The results below are from simulation unless stated otherwise. Each experiment changes one thing (or one group of related things) relative to the previous one.

Common setup

Simulator. PyBullet with the official SO-101 model (so101_new_calib.urdf, joint zero = middle of each joint's range). The jaw collision meshes were replaced by convex decompositions; without that, PyBullet treats each jaw part as a single convex hull and physical grasping is impossible. Grasps are fully physical: gravity, position-controlled joints with torque limits, friction between jaws and object.

Task. One object on the table: a cube (22–35 mm) or a ball (24–40 mm diameter) in one of 12 colours, placed 12–24 cm from the arm's base. The instruction is a short template such as "pick up the red cube".

Demonstrations. A scripted expert with access to the object's true position plans the grasp with inverse kinematics: approach above the object, straight descent, close, lift, hold. The grasp point is computed from the measured gap between the jaws (the URDF's tool frame turned out to lie inside the fixed finger).

Model. lerobot/smolvla_base (about 450 M parameters), fine-tuned with LeRobot on an RTX 3090, batch size 64.

Evaluation. Closed-loop runs in the simulator on 129 scenes never used for training (182 in v7). Success means the object is held between both jaws and lifted at least 3 cm. Most comparisons are done scene by scene with the same random seeds; with about 130 scenes, a single success rate has an uncertainty of roughly ±8 percentage points. SmolVLA predicts chunks of 50 actions; we evaluate executing all 50 ("chunk 50") and re-planning every 10 steps ("re-plan 10").

Results at a glance

Exp. Change Episodes generated Success, chunk 50 Success, re-plan 10
v1 baseline: front + wrist camera, 320×240 600 40 % 47 %
v2 perturbed demonstrations 600 63 % 74 %
v3 v2 + overhead camera 600 54 % 74 %
v4 v2 + 512×512 images, new camera framing 600 57 %* 67 %*
v5 v2 recipe with 2000 episodes 2000 68 % 81 %
v6 v5 + lift as soon as the gripper is blocked 2000 – 89–91 %
v7 new scene modelled on the real setup, stereo camera only 2000 – 79 % (86 %)**

Only successful demonstrations are used for training (about 86–97 % of the generated ones, depending on the recipe). * on the 126 scenes shared with v2 (v2 on the same scenes: 62 % / 73 %). ** 79 % on table/background textures held out from training, 86 % on training textures.

v1: baseline

Front camera and wrist camera, 600 generated episodes, 20 k training steps. Success was 40–47 %. Analysis of the failures showed that almost all of them were positioning errors: grasps succeeded almost every time when the gripper closed within 2 cm of the intended grasp point, and almost never beyond 3 cm. The median distance at closing was 2.5 cm. Inference took about 220 ms per action chunk (32-bit precision, no optimisation).

v2: perturbed demonstrations

In half of the episodes the executed arm motion was pushed sideways by 1–3 cm during approach and descent, while the recorded actions remained the expert's clean plan. The data then contains off-target states paired with corrective actions, which the clean demonstrations of v1 never showed. Success rose to 63 % / 74 %, the median closing distance fell to 1.5 cm (re-plan 10), and the difference to v1 is far beyond chance. Re-planning every 10 steps now helped significantly, which it had not in v1. Training curves showed a plateau after about 15 k steps.

v3 and v4: more visual input (negative results)

v3 added a third, overhead camera. With re-planning the result was identical to v2 (95 vs 95 successes); with 50-step chunks it was worse (54 % vs 63 %). v4 rendered 512×512 images instead of upscaled 320×240, aimed the front camera at the object area and moved the wrist camera. No measurable change (67 % vs 73 % on the same scenes, not significant). Neither more views nor sharper images improved positioning, which suggested that the limit lay elsewhere (data quantity or content) rather than in image detail.

v5: more data

The v2 recipe with 2000 episodes and 40 k training steps. Success with re-planning rose to 81 %; the gain was concentrated where data had been thin (objects close to the base: 8/23 → 15/23). A new failure type appeared: the gripper closed correctly on the object but the arm never lifted. The cause was in the data: the expert waited motionless for 0.8 s while closing, so a nearly identical image was labelled "wait" in some frames and "lift" in others. A policy without memory that re-plans every 10 steps kept choosing "wait". Longer chunks (25 steps) removed the stall but made positioning worse, with no net gain.

v6: lift as soon as the grip is secure

The expert now monitors the measured gripper position and starts lifting once the jaws stop moving against the object, which reduces the ambiguous motionless frames from up to 14 to about 5. Success with re-planning reached 89–91 %, the stall almost disappeared, and the median closing distance improved to about 1.0 cm. Remaining failures were mostly large objects close to the base.

v7: a scene modelled on the real setup

The scene was rebuilt after the real setup: a 50×49 cm plywood board on a desk, textures made from photos of the actual board, desk and room, background clutter, a mostly white arm, and lighting from behind the robot. The front and wrist cameras were replaced by a single stereo camera modelled after the real one (a side-by-side USB stereo camera, 6.2 cm baseline, about 45° horizontal field of view after rectification), placed 55–65 cm in front of the base. Objects were restricted to the ±30° sector this camera can see. Episodes start from a low ready pose so that the gripper is visible from the first frame, and rendered images pass through a noise model fitted to real frames (white balance, contrast, blur, sensor noise, JPEG).

Two rendering problems had to be solved: PyBullet's CPU renderer drew no shadows for small objects (a 1 cm near clipping plane left too little depth precision), and its dithered shadows were washed out by the camera noise model. Object shadows are now computed geometrically.

Result: 79 % on held-out textures and 86 % on training textures. Positioning was less precise than in v6 (median closing distance 1.9 cm), objects near the base remained the weakest zone (60 %), and balls were harder than cubes (74 % vs 84 %). The texture split contained only a few held-out textures, so the 7-point gap should be read as an indication, not a measurement.

First tests on the real arm

The v7 policy was run on a real SO-101 with 7.4 V servos and the real stereo camera. After a configuration error was found (a wrongly recorded joint direction for the base rotation), the arm moved towards the correct object but closed about 2 cm away from it. Offline tests on the logged frames showed that the policy located the object on the correct side, but reacted less strongly to real images than to simulated ones. Three systematic differences from the simulation were measured:

  • the shoulder and elbow servos stop 2–4.5° short when moving in one direction (a direction-dependent dead band), enough to shift the gripper by about 1 cm;
  • the control loop initially ran at about 5 commands per second instead of 30 (about 17 after restructuring; inference still pauses the loop for about 240 ms per chunk);
  • the real camera was closer (52.5 cm) and looked down more steeply (38.7°) than any camera in the training data.

Summary

  • Showing the policy how to recover from small offsets (v2) gave the largest single improvement.
  • More cameras and higher image resolution (v3, v4) did not help in this setup.
  • More data helped mainly where data had been sparse (v5).
  • Ambiguous waiting frames in demonstrations can make a memoryless policy stall (v5, fixed in v6).
  • A simulated success rate around 80–90 % did not transfer directly to the real arm; servo behaviour, control rate and camera placement all differ from the simulation.

All results come from a single training run per configuration; differences of a few percentage points between configurations are within the run-to-run and evaluation noise. The next entry covers v8, which addresses the real-arm findings: widely randomised camera positions, an explicit camera description as model input, and the measured servo behaviour in simulation.

Starting SO-101 Lab

This repository is the home for my experiments with the SO-101 robotic arm. I want to document the entire process: printing and assembling the hardware, configuring the servos, writing small control utilities, building simulations, and eventually exploring more autonomous manipulation.

Why keep a lab notebook?

Robotics experiments involve many small details that are easy to lose: which servo variant was used, what voltage was tested, which calibration file belongs to which build, what changed between software versions, and why a particular approach was abandoned.

The goal is to capture those details alongside the code, CAD exports, diagrams, and photographs. Successful results matter, but so do failed experiments and the measurements that explain them.

Initial platform

The current physical platform is an SO-101 follower arm with six STS3215 serial bus servos and an FE-URT-2 controller. The software work uses LeRobot, with PyBullet for selected simulation experiments.

These are notes about my particular build; hardware variants and software versions should always be checked before reproducing a procedure.

What comes next

  • Organize the printing and assembly notes.
  • Document the controller, servo configuration, and calibration procedure.
  • Publish the small utilities used for teleoperation and teach-and-repeat experiments.
  • Add simulation models and compare their kinematics with the physical arm.

This is the starting point. I'll update the documentation as experiments are repeated and verified.