Training robot policies in MuJoCo, and transfer to Webots

Intelligent Systems for Robotics · Week 3

Syllabus · Home

Last week ended on a pendulum, where eight lines of energy shaping matched a trained policy. The argument was that learning earns its place where no such model exists. This week puts that argument to work on TIAGo. You design two learning problems, train them in MuJoCo, run the results in Webots, and measure each against a classical method on the same scenes.

The two problems are pieces of the fetch. Placement chooses where the base should stand so that the arm can reach the can. It is the policy inside place-base-for. Docking drives the last metre to that pose, the final approach of go-to.

Objectives

Which of these should have been learned?

The lecture opens with two runs recorded in Webots. In the first, a classical sampler chooses a base pose and drive_to drives there. drive_to heads straight for the point, then turns in place onto the goal heading. In that turn the wheel commanded backwards lags by 1 to 3 s and the base pivots about it. From start A, the base ended 19 cm from the sampler's pose. Sent instead to the learned placement policy's proposals from three starts, it ended 0.7, 4.3 and 16.7 cm from them, with the heading right to within 5°. How far the base slides depends on the pose it is sent to: from A the pose lay almost straight ahead, so the final turn was small. Inverse kinematics in the model reached the can from the first two poses reached, not from the third. The arm itself does not move in these demos.

In the second, a docking policy trained in MuJoCo drives instead. Further than 1.5 m from the goal, drive_to would first bring it to 0.8 m, but every start is closer. Its goal is re-anchored at every step to the table, whose pose is fitted to the legs in the lidar scan. From A, drive_to stopped 0.7 cm off and the policy timed out 3.7 cm off. From B the policy stopped 1.1 cm off, against 4.3 cm, and from C it timed out 6.8 cm off, against 16.7 cm, where the arm could not reach the can from drive_to's pose. You can run both yourself:

PYTHONPATH=. python3 examples/03-training-policies/main.py --webots
PYTHONPATH=. python3 exercises/03-training-policies/main.py webots-placement --start A
PYTHONPATH=. python3 exercises/03-training-policies/main.py webots-chain --start A

The second and third commands need your placement-policy.zip, and the third your docking-policy.zip, in the directory you run from. Open the start's world first: isfr_placement.wbt for A, isfr_placement_B.wbt for B and isfr_placement_C.wbt for C.

One start cannot settle which drives better, and the two did not steer on the same goal. A fair comparison gives a classical controller the goal the policy gets, anchored to the table's legs. The table below does that in MuJoCo, from the exercise's docking starts, anywhere 0.5 to 1.5 m from the goal at any heading, and from a pose lined up 0.8 m out, where drive_to hands over from further away. The pose controller is the polar-coordinate law of Siegwart, Nourbakhsh and Scaramuzza, section 3.6, with the textbook's gains.

Dock controller, 1000 episodesFrom anywhereLined up
Learned docking71%82%
Pose controller42%60%
drive_to's control law14%78%
goto_empty.zip, before docking35%52%

Lined up, the policy leads drive_to steering on the table's legs by 5 points, at the edge of the noise on 1000 episodes. The 1 to 19 cm of the first demo came from odometry and the final turn, not from the control law. From anywhere, both classical controllers fall behind: the pose controller spins near the goal when its heading is far off, and drive_to collides in almost half its episodes. In Webots, three starts cannot rank the four. With earlier policies, inverse kinematics reached the can after 13 of 15 trials, with one failure each for the pose controller and the policy. With the current policy, docking directly, it docked from A in under 6 s; from B and C it docked once and timed out three times in four runs.

Both classical rivals turn in place, which TIAGo's casters punish. A fairer rival follows a cubic Bézier path tangent to the start and goal headings, clear of the table, tracked by pure pursuit on the same goal. It has not been measured yet.

So which should have been learned: where to stand, how to drive the last metre, or how to cross a room with furniture in it? The rest of the lecture answers each with a measurement. The answers are collected at the end of the section on obstacles.

Designing an environment

Week 2 took the environment as given. Pendulum-v1 came with its state, its actions, its reward and its episode length. On a robot you write all of these yourself: what the policy is shown, what it can command, what it is paid for, when an episode ends, and which scenes it trains on. The choice between PPO and SAC is the smallest decision on that list.

State

A state is Markov when the next state depends only on the present state and action:

P(st+1 | st, at) = P(st+1 | s0, a0, …, st, at)

A policy sees only its observation, so the observation is the state as far as the policy is concerned. What the observation leaves out, no amount of training can recover. Show a CartPole policy only the cart position and the pole angle. A pole at 5° falling right and a pole at 5° swinging back up give the same input and need opposite actions. Adding both velocities fixes it, and so does stacking the last two observations, from which the policy can take a difference.

The frame matters as much as the content. A goal at (3.0, 2.0) in the world frame means nothing without the robot's own pose, so the policy has to learn the transform from experience. The same goal in the robot frame, 1.2 m ahead and 0.4 m to the right, looks the same wherever in the room the situation arises. A frame that makes equivalent situations look alike is data the policy does not have to collect. The go-to observation in isfr.envs.goto holds the goal in the robot frame and re-expresses it at every step.

The simulator will hand you anything: the robot's exact pose, the table's exact box, the contact forces. The robot in Webots has a lidar, odometry and a detector, and your observe function runs there unchanged. If observe reads a quantity, the robot must be able to measure it. Privileged information has other uses in training, such as judging whether an episode succeeded, but never in the observation.

Three practical points follow. A policy trained on exact inputs comes to rely on a precision it will not have, so the homework's training adds 2 cm of noise to the can's position and 1 cm to each lidar range. A scan holds inf wherever nothing returned, and a network cannot take inf. And inputs of very different sizes, metres beside radians beside counts, slow learning down, so bring them to comparable ranges.

Actions

The action space fixes what a policy can express at all. This week both are given. Placement's action is three numbers in [−1, 1] that isfr.envs.base.action_to_pose turns into a pose around the can. Docking's is a linear and an angular velocity, fixed by the policy it starts from. The section on placement shows how much of a problem a good action space can solve on its own.

Reward

A sparse reward pays 1 on success and 0 otherwise. It says exactly what you want, and it gives no signal until the first success. A shaped reward adds terms that pay for progress. It learns faster, and it also changes what is optimal unless you are careful.

Randløv and Alstrøm trained a simulated bicycle to ride to a goal and paid it for progress towards the goal. Nothing penalised riding away from it. The agent learned to ride in tiny circles near the start: each circle contains a stretch towards the goal, that stretch pays, and the rest of the circle costs nothing. Ng, Harada and Russell (1999) report this case with another one, a soccer robot paid for touching the ball that learned to vibrate next to it.

Their remedy is potential-based shaping. Choose any function of state Φ(s) and add

F(s, s′) = γ Φ(s′) − Φ(s)

to every reward. The optimal policy is unchanged. With Φ(s) equal to minus the distance to the goal, moving closer pays and moving away costs the same amount back. With γ = 1, the shaping collected round a loop that returns to its start is Φ(sT) − Φ(s0) = 0, because the sum telescopes, so the bicycle's circles are worth nothing. The paper also proves that, with nothing known about the MDP, only this form carries the guarantee. Any other shaping term defines a new task, and you have to check what that task rewards.

Every term in a reward is a price, and the agent pays whichever is cheapest. So the question to ask of a reward is what it makes optimal. What you hoped it would encourage is a different question. In the boat-racing game CoastRunners, the score pays for hitting targets along the course. An agent trained on the score found a lagoon where targets respawn, circled there while crashing and catching fire, never finished the race, and scored 20% higher than human players (Clark and Amodei, 2016). Finishing the race was never in the reward.

Russell and Norvig make the same point with a vacuum cleaner. Pay it for the dirt it picks up, and it can pick up dirt, dump it, and pick it up again. Their rule is to measure what you want in the environment, a clean floor, rather than how you think the agent should behave. A clean floor is a state. Dirt collected counts actions.

The lecture's exercise in the room is a pick-and-place arm paid +10 when the cube is in the box and charged 1 on every step its gripper touches the table. The cube sits close to the table, and grasps fail often. If an attempt costs more in table contact than it returns on average, never reaching for the cube is optimal. A penalty makes avoidance attractive, so compare the price of every way an episode can fail, including the ways in which the robot does nothing.

Termination and truncation

Gymnasium's step returns two flags. terminated means the task itself ended, by success or a collision. Nothing follows, so the last state has no future value. truncated means you stopped the clock. The world would have gone on, so the value of the last state still counts, and the learner should bootstrap from it. Confusing the two corrupts the value estimates near the limit. If the time limit is part of the task, the time remaining belongs in the state (Pardo et al., 2018). Stable-Baselines3 reads truncation from Gymnasium and bootstraps.

This changes how you price an episode. Running out of time is not an ending in training. A policy that stops trying pays its cost per step c until the clock runs out, and the bootstrapped value goes on charging it from there. The total is about L = c/(1 − γ) whatever the episode length, which is 100c at γ = 0.99. A reward paid on the truncated step itself arrives once, at a step the policy cannot foresee unless the time left is in its observation.

Curricula

A task so hard that success never happens gives a sparse reward nothing to learn from. Docking to 5 cm and 5° next to a table, trained with PPO from scratch for 40 minutes, stayed at 0% success. A policy trained first on an empty floor, where goals are easy to reach, already succeeds at docking 39% of the time before any fine-tune. Train the easy task first, then fine-tune on the hard one (Bengio et al., 2009). The homework gives you that empty-floor policy as goto_empty.zip.

Scenes and evaluation

What varies between episodes is part of the design too: the layout, the walls, the clutter, the start pose, the noise. Evaluate on scenes the policy never trained on, and seed them. When two designs, or a design and a baseline, meet the same seeded scenes, the comparison is fair. The homework seeds evaluation episode i with EVAL_SEED + i for this reason.

A checklist before you train

  1. Does the observation hold everything the next step depends on?
  2. Can the robot measure every input in Webots?
  3. What does the reward make optimal, including every way to fail?
  4. Which endings are terminal, and which are the clock?
  5. What scenes does it train on, and what will it meet later?

Placement as a problem

Placement is one decision per episode. The robot scans once, proposes a base pose, and is scored by inverse kinematics from that pose: the arm reaches the can or it does not. The base must also not collide with the table, a wall or the clutter.

The action is object-relative. Its three numbers are a stand-off distance from the can between 0.3 and 1.2 m, a bearing from the can in any direction, and a heading error of up to 45° from facing the can. Every pose it can express stands near the can and faces it.

The classical baseline searches the same action space. It draws an action at random, checks the pose with inverse kinematics in about 14 ms, and keeps the first pose that reaches the can. It solves every one of 200 scenes with a median of 4 samples and 13 ms per decision. Of 500 generated scenes, 499 can be solved at all within 64 samples.

Placement, 200 fresh scenesSuccess0 walls1 wall2 walls
Naive design, seed 080.5% (74.5–85.4)78.2%85.5%75.7%
Naive design, seed 180.5% (74.5–85.4)79.3%84.2%75.7%
The instructor's design89.0% (83.9–92.6)92.0%86.8%86.5%
Sampler100%

The naive design in the week's example sees only the can's position and is paid 1 for success. Trained with SAC for 200 000 steps, it solved 8.5 points fewer scenes than the instructor's design, and the two 95% intervals still overlap. A run with the same seed repeats exactly. Two seeds of the naive design solved 161 scenes each, with different policies; on earlier runs two seeds differed by 5 scenes of 200, which is the noise between runs. The action does most of the work. It already restricts every pose to the right distance, facing the can, and leaves the policy to choose a side to approach from and a sensible distance. The walls are where a lidar should help. With two walls the instructor's design scores 86.5%, against 76% for both naive seeds, on 37 scenes. Without walls it also leads, 92% against 78 and 79%.

What learning buys here is time. A forward pass of the network takes under 1 ms. The sampler needs one or more inverse kinematics solves at about 14 ms each. Training is paid once and every later query is cheap, so the policy is the sampler's search, amortised. That matters where placement is queried many times, for example by a planner that asks where to stand for every candidate place in an apartment. For a single fetch it does not.

Success is not the only cost

A reward of 1 for any reachable pose is indifferent to how far the robot has to drive to get there, and to how far the arm has to stretch. The table below measures the distance and the turning from the start, the stand-off from the can, and how many poses come within 10 cm of an obstacle, on the 127 scenes that every row here and the next table solved. A median over each one's own successes would credit the policies with the easy scenes the sampler also solves and leave out the hard ones only the sampler solves. isfr.envs.base.travel computes the distance and turning for one pose, and sample_placement(..., nearest=True) tries all 64 samples and keeps the accepted pose with the least travel.

PlacementSuccess, of 200Distance, median (IQR)Turning, medianStand-off, medianWithin 10 cmTime per decision
Naive design, seed 01610.86 m (0.53–1.25)64°0.98 m9under 1 ms
Naive design, seed 11610.88 m (0.55–1.23)70°0.99 m10under 1 ms
Instructor's design1781.34 m (0.85–1.82)77°0.79 m1under 1 ms
Sampler, first accepted2001.60 m (1.06–2.26)129°0.82 m219 ms
Sampler, nearest of 64 accepted2001.01 m (0.69–1.39)46°0.90 m28230 ms

Paid 1 for success, the naive policy stands at the edge of the arm's reach, 0.98 m from the can, where inverse kinematics reaches no further than 1.08 m. A probe of the same scenes found a reachable pose 10 cm clear of everything at a median 0.53 m. The instructor's design stands 0.79 m off and comes within 10 cm of an obstacle once, and drives about 0.5 m further than the naive policy to do it. The 10 cm is the margin Webots needs, since the base stops 3 to 10 cm from where it is sent. The samplers accept any pose the arm reaches, and come within 10 cm of an obstacle 21 and 28 times. The nearest sampler still succeeds 11 points more often than any policy and turns less, and it costs about 230 ms a decision.

A classical step can bring the base closer still. isfr.envs.base.slide_in keeps the policy's approach bearing and heading and moves the base in towards the can, to the nearest stand-off that inverse kinematics accepts with 10 cm clear. Stopping at the first stand-off that passes leaves the base on the edge of what the arm reaches: after a 5 cm, 3° arrival error, inverse kinematics still reached the can from only 64% of such poses. So slide_in also requires every stand-off 10 cm further in to pass.

Placement, instructor's designLearned onlyLearned, then slide_in
Solved, of 200178172
Stand-off, median0.79 m0.75 m
Inverse kinematics still reaches after a 5 cm, 3° arrival error87%85%
Time per decisionunder 1 ms47 ms

It buys 4 cm for 47 ms a decision, at the same robustness. In Webots, from start C, drive_to stopped 17 cm from the learned pose and the arm could not reach the can; from the slide_in pose it stopped as far off and the arm reached. Part 1 of the homework leaves the choice to you.

The two can be combined, with learning deciding the order and inverse kinematics keeping the last word. The sampler draws 64 candidate poses. A classifier, trained on inverse kinematics verdicts, scores each candidate's chance of success, and the check runs on the best first. If none passes, 64 random candidates follow. A probe of 2026-10-02 trained the classifier on 200 000 random poses, seeing the can and 64 beams with the training noise. The ranges below span a default and a stronger inverse kinematics.

Order, 200 scenesSolvedFirst candidate acceptedIK calls, medianTime, median
Random19739–46311–21 ms
Learned ranking198151–16312 ms

The ranking keeps the sampler's success and usually needs one check. Used alone, as a one-shot policy, the classifier is 83.7% accurate, about where SAC ended before the simulator fix described under Obstacles, when this probe ran. The saving grows with the cost of the check, which rises once the arm plans its own reach. Learning becomes a heuristic inside the classical method. The homework still trains the SAC policy.

Action-related places

Stulp, Fedrizzi, Mösenlechner and Beetz (2012) treat the same problem a third way. They run many simulated grasps from base poses around an object and learn, in the object's frame, the probability that a grasp from each pose succeeds. At query time they sample where the object might be, given the robot's uncertainty about it, and choose the pose most likely to succeed over all of those samples. The robot keeps a region of good places rather than committing to one pose early, which the paper calls least commitment.

Placement in this course is one step, rewarded 1 for success and 0 otherwise. For such an episode

Q(s, a) = E[r | s, a] = P(success | s, a)

so SAC's critic estimates the quantity an action-related place represents, and its actor learns to output the best action directly, where Stulp et al. search. With one step there is no next state, so SAC's entropy term affects only the actor. This reading is the course's own. The paper states that an action-related place is not a reinforcement learning policy. The ranking classifier above learns the same quantity by supervised learning, and leaves the search to the sampler.

Their baseline is a fixed offset from the object. The sampler here checks every candidate with inverse kinematics before accepting it, which is a stronger bar. Against it, a learned placement can only win on time.

Transfer: what a second simulator finds

The policies train in MuJoCo and run, with the same code, on the Webots TIAGo. Two simulators disagree in ways a real robot would too. They also disagree wherever one of them, or the code between them, is wrong. Three bugs came out this way.

The lidar was mirrored. A box on the floor to the robot's left read at −94°, on its right. Webots orders the range image from left to right, while the bridge's header claimed right to left. Every test passed, because both backends agreed with the wrong header. A policy trained in MuJoCo would have met the mirror image of its world. The bridge now reverses the range image.

The robot sees itself. In Webots, 52 lidar beams hit TIAGo's own body at 0.23 to 0.27 m, at 108° to 117° either side. MuJoCo's scan never does. Returns under 0.3 m are now read as no return, only beyond 100° either side. The first version of that mask applied at every angle, and in 66% of one navigation policy's collisions the obstacle was a return the mask had thrown away. The fix for one simulator had broken the other. The 66% was measured before the simulator fix described under Obstacles.

The arm was out. Training assumed the arm tucked, and in Webots it was not. From two of three starts the palm met the table during the final turn, and the base stopped 59° and 60° off the proposed heading. With the arm tucked, all three ended within 5°. Tucking also levelled the base, which removed most of the floor returns on its right.

The casters

TIAGo has four passive casters. After a spin in place they trail sideways, and the next straight leg drags through them. In week 1's square, spinning in place and holding the heading closed to 0.418 m by odometry and 1.443 m in truth. Turning as an arc closed to 0.084 m and 0.174 m.

Commanded to turn 160° in place with the arm tucked, the Webots base reached 72 to 78°. The wheels reach half the commanded speed, because the motors' 10 N m saturate against the casters. MuJoCo as vendored, under the same limits, turned 142°. A policy trained there would turn twice as far as it meant to.

The obvious parameters did not close the gap. Raising the casters' swivel damping up to 100 times, and doubling their friction, left the spin above 114°. Load did close it. In Webots the casters sit on stiff springs level with the drive wheels, and lowering MuJoCo's casters by 0.4 mm moves weight onto them. The spin fell to 68 to 76°, and the drift after the turn averaged 35.4° against Webots' 33 to 40°. The response is steep: 0.75 mm gives 58 to 64°, and at 1 mm the drive wheels lift off the floor.

No single value is right, so training randomises it. Each episode draws its own caster drop, from 0.25 to 0.55 mm, and its own damping, from half to twice nominal. That spans spins of about 55 to 90°, and Webots becomes one draw among many. Tobin et al. (2017) randomised a simulator's appearance for the same reason, and Peng et al. (2018) its dynamics.

Train on the truth, deploy on a landmark

Odometry drifts, so in Webots the docking goal is re-anchored at every step to the table. The lidar sits 9 cm above the floor and sees the table's legs, never its top, and isfr.envs.landmarks fits the table's pose to them. In Webots, 60 of 60 scans fitted, with median errors of 4 to 11 mm. In MuJoCo, a docking policy trained on the simulator's exact pose scored 68.5%, and 69% when the same file was scored on odometry and the landmark. Odometry alone, with no landmark, scored 4.5%. In an earlier run, before the simulator fix described below, training with the landmark in the loop did worse, at 38%. The landmark is close enough to the truth that the policy does not need to learn its noise.

Obstacles, still open

go-toTrainingSuccess
PPO, empty floor, from scratch10 minutes100% of 200
The empty-floor policy, among obstaclesnone42.5%
PPO, obstacles, from the empty-floor policy27 minutes46.5%
SAC, obstacles, from scratch1 hour10%

On an empty floor, go-to trains in minutes. Among furniture, 2.4 million more steps moved the empty-floor policy from 42.5% to 46.5%, which is inside the noise of 200 episodes. SAC from scratch scored 10% after an hour and timed out in 86% of scenes. The robot learns not to crash before it learns to arrive. Before 3 October 2026 the simulator kept every wall and box at the same heading, whatever the scene asked for, and several fixes were tried in that world: correcting the self-hit mask, keeping starts 0.8 m clear of obstacles, penalising close passes, giving the policy the exact pose instead of odometry, switching to SAC, another reward, a depth camera on the head, and progress measured along the free path instead of the straight line. Each moved the result by less than the noise, or not for long. They have not been repeated in the corrected world.

Straight line to the goalScenesSuccess
Clear9990%
Blocked1015%

The policy sees the obstacle and has room to pass it, and still drives the straight line. The table above is the warm-started policy after 2.4 million steps, on 200 seeded scenes, with each table counted as solid. 72% of its collisions were inside the lidar's view. Where it hit, the widest route round was a median 1.10 m, for a base 0.54 m wide. It has not learned to detour. The course's answer is to let perception name the obstacle, the world model record it, and a graph search plan the route, with learning kept for the last metre. Weeks 4 and 5 build that graph.

ProblemLearned?Why
Where to standOptionalA sampler succeeds, and learning makes it fast
The last metreFrom anywhereLined up, a classical controller on the table's legs comes within 5 points. No arc-path rival yet.
One edge, among obstaclesThe drivingMotion tubes choose where to drive, and a shield vetoes what is not free
Across a roomNot yetPlanned over the world model instead

The rule the course follows is to learn what classical methods do badly.

The provided go-to

The policy does not detour, so go-to does not leave the route to it. At every step, motion tubes (Rodrigues et al., 2023) check a fan of 102 arcs against the lidar scan, forward up to 1.2 m and back up to 0.2 m. A tube is free when no return lies inside the area the base would sweep along it. The policy drives to a point 0.5 m along the free path whose end lies nearest the goal. A path is one tube at least 0.5 m long, or a forward tube followed by one of the 1.2 m tubes. A tube that first leads away therefore counts by where the tubes continuing from it end. A shield then checks the policy's own command over the next second and, when its arc is not free, replaces it with the nearest free one. Within 0.5 m of the goal the shield steps aside and the policy docks alone.

The lidar sees 240°, so the ground behind the base is checked against the scans of the last 10 s, and ground the base itself stood on counts as free. The head's depth camera adds what the lidar, 9 cm above the floor, cannot see, such as a tabletop. Its points are kept for 20 s. The code is isfr/envs/local.py, isfr/envs/tubes.py, isfr/envs/scan_memory.py and isfr/envs/camera_memory.py.

go-to is a skill like week 2's grasp. Give it the goal relative to where the robot stands when it starts. Odometry carries the goal from there.

from isfr.envs.base import Pose
from isfr.robot import Robot
from isfr.skills.go_to import GoTo

with Robot() as robot:
    skill = GoTo(robot, Pose(2.0, 0.5, 0.0))
    print(skill.run_to_completion(max_ticks=skill.max_ticks))

Pass max_ticks. The default of 500 ticks is 48 s, shorter than the skill's 60 s timeout. The skill tucks the arm and tilts the head before the base moves. It succeeds when the base has stopped within 10 cm and 10° of the goal. It fails with one of three reasons:

The policy is data/policies/go-to.zip, PPO with the observation and actions of goto_empty.zip, fine-tuned from it among obstacles with the casters randomised. It was then trained for 90 more minutes at 3e-5 on the base model recalibrated against Webots, with each command landing 0 to 64 ms late, as it does there. It was scored in MuJoCo on 200 seeded scenes of each kind, every command 32 ms late.

ScenesOutcomego-to.zipTarget
Clutter, passableSuccess85.0% (79.4 to 89.3)75% or more
Clutter, passableCollision with a box, leg or wall0.0%5% or less
Clutter, passableUpper body on a tabletop0.0%None set
Clutter, passable"edge blocked"12.5%20% or less
Clutter, passable"did not settle"2.5%None set
Two rooms, doorway plugged"edge blocked" within 30 s100%90% or more
Two rooms, doorway pluggedCollision0%5% or less

Success is given with its 95% Wilson interval. It meets every target. About one crossing in seven still ends in failure, most often an "edge blocked" in a passable scene, where the base has stalled in a pocket of clutter. Plan for it, since a failed termination is what triggers repair and replanning.

In Webots the same policy and code ran in webots/worlds/isfr_obstacles.wbt, two rooms joined by a doorway with two boxes in the first, and in isfr_obstacles_blocked.wbt, where the doorway is plugged. In the open world it reached none of five goals. In four runs of five the base stopped beside the first box, and in the fifth it crossed and stopped 19 cm off the goal. With the doorway plugged it returned "edge blocked" every time. In 1 of these 10 runs the base touched the first box, while turning beside it. A copy of the open world in MuJoCo fails the same way on every seed, so the cause lies in go-to and not in the transfer. The section "What training go-to taught us" below describes what was found.

The fine-tuning taught one thing about the training loop. At PPO's default learning rate, 3e-4, which is the rate stored in goto_empty.zip, the obstacle policy held for 90 minutes of further training and then fell from 758 to 670 of 1000 scenes by three hours. At 3e-5 it reached 819 after one hour and stayed between 786 and 816 for two more. A warm start begins near a good policy, and large updates can move it away. Docking gains from a lower rate too, and train-docking below runs at 1e-4. The first shipped file was the 3e-5 run at one hour, and the current one continues from it at the same rate. The rates the loop printed did not show the difference. They stayed between 35% and 57% in both runs, because they score the policy alone, without tubes and shield.

The provided training loops

In the micro-project you design the observation and the rewards and write the measurements. The training loops are provided in main.py, and this section explains what they do, step by step.

Workers

Both loops wrap the environment in Stable-Baselines3's SubprocVecEnv, and that in VecMonitor, which records how each episode ended. Each worker is a separate process with its own MuJoCo simulation and its own random scenes, seeded with the worker's index. One call to the vectorised environment steps every worker once, so the algorithm sees as many transitions per call as there are workers. By default the loops use half your CPU cores, and --workers changes that.

train-placement: SAC

  1. Each worker builds a placement environment from your design.py, with 2 cm of Gaussian noise on the can's position and 1 cm on each lidar range. The pose is computed from the noisy can position, as the robot would, and scored against the true one.
  2. SAC starts with an actor and two critics of two hidden layers of 256 units each.
  3. Every placement episode is a single step, so each transition is a complete episode. All of them go into a replay buffer that holds up to a million.
  4. For the first 1000 transitions SAC acts at random and does not learn. After that, it takes one gradient step on a random batch of 256 transitions from the buffer for every transition the workers add, so N workers are followed by N steps. The batch mixes old and new experience, which is what makes SAC off-policy.
  5. The loop calls SAC's learn in chunks of 2000 steps. After each it prints the elapsed time, the steps taken, and the success rate of the last 100 training episodes. It saves placement-policy.zip every five minutes and at the end, and stops after --steps, 200 000 by default. SAC's default is one gradient step per call, whatever the number of workers, which would give 4 workers twice the updates of 8 from the same transitions.

train-docking: PPO from the empty-floor policy

  1. Each worker builds a docking environment with your docking_reward. The observation and the actions come from isfr.envs.goto: 64 lidar beams, the goal in the robot frame, and the previous action. The policy acts every 96 ms. Training uses the simulator's exact pose and 1 cm of range noise.
  2. PPO loads goto_empty.zip, its network and its settings. The actor and the critic each have two hidden layers of 256 units. Your reward replaces the one it was trained with.
  3. PPO collects 12 288 steps across the workers with the current policy. With N workers, each collects 12 288 / N.
  4. It computes each step's advantage from those rollouts, then updates the network in ten passes over them in batches of 512, with the ratio clipped at 0.2 as in week 2.
  5. It discards the rollouts and collects again. PPO is on-policy, so data from an older policy is not reused.
  6. After each update the loop prints the elapsed time, the steps taken, and the success, collision and timeout rates of the last 100 training episodes. It stops after --steps environment steps, 1.8 million by default, saving docking-policy.zip every five minutes and at the end.

Three of these choices are deliberate. The update size is held at 12 288 steps, whatever the number of workers, because goto_empty.zip was trained with updates of that size on 12 workers. Stable-Baselines3's default of 1024 steps per worker gives 4 workers updates a third that size. In two seeded runs of that setting, docking ended at 17% and 11%, below the 32% it started from. Holding the update at 12 288 steps, the same 4 workers ended at 50.5% and 55.5%. These runs predate the simulator fix described under Obstacles.

The learning rate is lowered to 1e-4 from the 3e-4 stored in goto_empty.zip, which is PPO's default, for the reason given for go-to above. At 3e-4, two seeds of the solution's reward ended at 69.0% and 56.5%. At 1e-4 the same seeds ended at 78.0% and 71.0%, and a third at 80.0%. Most of the difference is timeouts, 10.0% and 11.0% at 3e-4 against 1.0% to 8.0% at 1e-4.

Training stops after a number of steps, and not after a time, so that a slower laptop takes longer and trains the same amount. On the instructor's laptop, 4 workers took 35 minutes. Before the simulator fix, at 3e-4, every setting tried dipped somewhere between 600 000 and 900 000 steps and then recovered, and with the fixed update size success fell as low as 21% in that stretch. At 1e-4, evaluation rose from between 55.0% and 62.5% at 600 000 steps to between 71.0% and 80.0% at the end. One of the three runs peaked at 76.0% at 1.2 million steps. A time budget can stop a slow machine short. Evaluate the final file.

Evaluation is yours

The rates the loops print are not an evaluation. The policy samples its actions to explore, training adds sensor noise, and docking trains on the exact pose. Use them to follow a run. Score a policy with model.predict(obs, deterministic=True), which takes the mean action instead of sampling. Reset episode i with seed=EVAL_SEED + i, so that two policies, or a policy and the sampler, meet the same scenes. Placement is scored without training noise. Docking is scored the way it runs in Webots, on odometry with the goal anchored to the table's legs, while success is still judged on the simulator's exact pose.

Report a success rate with its 95% interval. Use the Wilson score interval, which stays inside 0 to 100% where the normal approximation does not. On 200 episodes, a difference of a few points between two policies is inside that interval, as week 2 showed for rewards.

Keep a set of seeds you never choose on. Five go-to checkpoints were scored on seeds 70000, and the best reached 84.5%. On seeds 80000, where nothing had been chosen, it reached 81.0%. Choosing the best of five on a set also picks the one that set happened to favour. Report the number from the unused seeds.

What training go-to taught us

Training go-to among obstacles is the part of this course's preparation that did not finish. In MuJoCo the shipped skill succeeds in 85% of passable cluttered scenes. In Webots, in the two rooms of isfr_obstacles.wbt, it reaches none of five goals. The faults found on the way are listed below, grouped by kind. Most of them concern time, geometry and measurement, and few concern the learning algorithm. Each cost more than a training run.

  1. Stamp every sample and schedule on the simulator's clock. A depth frame was added to memory at the pose of the moment it was read, up to 128 ms after it was taken. Turning at 0.8 rad/s, its points then lay up to 16.6 cm from the surfaces they came from. The control loop in Webots also first ran at twice the trained period, and that alone took the policy from four goals of five to none in a MuJoCo copy of the scene. Each command lands about 32 ms after its sample, which training did not model until it was measured.
  2. Calibrate the training simulator against the deployment one, on the manoeuvres the policy makes. The MuJoCo casters were fitted on a turn on the spot, and in forward turns the base then turned at 45 to 48% of Webots' rate. The head camera sat 3.5 cm off centre in Webots, and its depth sensor 3.7 cm further left, where the MuJoCo model had it centred. Measured in place, the error at a table edge fell from 7.2 to 1.8 cm.
  3. Lower the learning rate for a warm start, and check that a seed reaches the run. At 3e-4 the obstacle policy fell from 744 to 545 successes in 1000 scenes over 90 minutes, while at 3e-5 it reached 819 after one hour. Stable-Baselines3 also re-seeds a loaded model from the seed stored in its file, so two runs started with different seeds were the same run. The rates a training loop prints score the policy alone, without the tubes and the shield, and they ran 10 to 20 points above evaluation.
  4. One free-running run per test case cannot rank two fixes. Webots runs that are not lock-stepped do not repeat, and several fixes were judged on one run each. A copy of the open world in MuJoCo then reproduced the Webots failure in all 25 of its runs, five seeds for each goal. From then on a change could be tested in minutes and on repeats.
  5. A memory window chosen for one region of the robot's surroundings changes every region the memory feeds. Camera points kept for 10 s, so that a tabletop behind the base is remembered while it reverses, cut reversing collisions from 9 to 3 in 1000 scenes. In Webots the same window kept the points of a box ahead, and they vetoed every forward motion. Likewise, a fix that lets the robot reach new states exposes the memories those states rely on. Counting the ground the base stood on as free let it back out of dead ends, and it then backed into tabletops it had seen 11 to 16 s before.
  6. Test whether each layer around the policy earns its place. With the tube target removed, the policy aimed at the real goal, the shield replaced 63% of its commands, and successes fell from 845 to 582 in 1000 scenes. The shield in turn fed back into the policy. Its replacement commands became the policy's previous-action input, and that input carried a left turn into the box beside the start. Each hand-written rule added around the policy fixed one case and moved the failure to another.

The trap at the start of the open world is not solved. The base turns to face the first box, and from 0.3 m every forward arc brings it nearer. Only reversing or turning on the spot gets it out, and the policy has never had to learn either, since it was trained without the shield. Training with the shield in the loop reached 8, 0 and 2 of the 25 goals at three checkpoints of one run. Getting the robot out is yours to solve in the project.

Slides

Deck for this week

Example

Run from the repository root:

PYTHONPATH=. python3 examples/03-training-policies/main.py
PYTHONPATH=. python3 examples/03-training-policies/main.py --webots

The first trains the naive placement design with SAC for 10 000 steps, about two minutes, and scores it on 100 fresh scenes next to the classical sampler. The second needs webots/worlds/isfr_placement.wbt open and running, from a shell that has sourced scripts/webots-env.sh. It runs the sampler and drive_to on the Webots TIAGo, as in the lecture's first demo. The lab opens with the example.

Micro-project

This week's exercise is exercises/03-training-policies/. Its README.md states the task and ends in a Self Check list you can verify unaided. Record your results in RESULTS.md as you go.

Part 1 is placement. You design the observation and the reward, train, and compare your design with the naive one and with both samplers on 200 scenes, on success, travel and time. Part 1 has no success threshold. Your RESULTS.md says whether your design beats the naive one by more than the noise between runs, and why. Then you run your policy in Webots from three starts.

Part 2 is docking. You design the reward, fine-tune goto_empty.zip, and measure it before and after. You also measure how far the robot can miss a placement pose before the arm can no longer reach the can, and chain your policy after drive_to in Webots. Part 2 ends with a paragraph that prices each way an episode can end under your reward and names the cheapest. Work it out from the reward. At 1.8 million steps, a reward with a trap in it can score as well as a sound one, so a good score does not settle the question.

Start train-placement early in the lab. It took about 10 minutes on 4 workers of the instructor's laptop, and more workers barely shorten it. train-docking took 35 minutes on 4 workers of the instructor's laptop. A slower laptop takes longer, and the deadline allows training at home. The micro-project runs from the inputs it provides. It does not need any earlier week's output.

Reading

Cited above, for reference:


Intelligent Systems for Robotics · course materials