Intelligent Systems for Robotics · Week 3
Last week ended on a pendulum, where eight lines of energy shaping matched a trained policy. The argument was that learning earns its place where no such model exists. This week puts that argument to work on TIAGo. You design two learning problems, train them in MuJoCo, run the results in Webots, and measure each against a classical method on the same scenes.
The two problems are pieces of the fetch. Placement chooses where the base should stand
so that the arm can reach the can. It is the policy inside place-base-for.
Docking drives the last metre to that pose, the final approach of go-to.
The lecture opens with two runs recorded in Webots. In the first, a classical sampler chooses a
base pose and drive_to drives there. drive_to heads straight
for the point, then turns in place onto the goal heading. In that turn the wheel
commanded backwards lags by 1 to 3 s and the base pivots about it. From start A,
the base ended 19 cm from the sampler's pose. Sent instead to the learned placement
policy's proposals from three starts, it ended 0.7, 4.3 and 16.7 cm from them, with
the heading right to within 5°. How far the base slides depends on the pose it is sent
to: from A the pose lay almost straight ahead, so the final turn was small. Inverse
kinematics in the model reached the can from the first two poses reached, not from the
third. The arm itself does not move in these demos.
In the second, a docking policy trained in MuJoCo drives instead. Further than
1.5 m from the goal, drive_to would first bring it to 0.8 m, but every
start is closer. Its goal is re-anchored at every step to the table, whose pose is fitted
to the legs in the lidar scan. From A, drive_to stopped 0.7 cm off and the
policy timed out 3.7 cm off. From B the policy stopped 1.1 cm off, against
4.3 cm, and from C it timed out 6.8 cm off, against 16.7 cm, where the arm could
not reach the can from drive_to's pose. You can run both yourself:
PYTHONPATH=. python3 examples/03-training-policies/main.py --webots
PYTHONPATH=. python3 exercises/03-training-policies/main.py webots-placement --start A
PYTHONPATH=. python3 exercises/03-training-policies/main.py webots-chain --start A
The second and third commands need your placement-policy.zip, and the third
your docking-policy.zip, in the directory you run from. Open the start's world
first: isfr_placement.wbt for A, isfr_placement_B.wbt for B and
isfr_placement_C.wbt for C.
One start cannot settle which drives better, and the two did not steer on the same goal.
A fair comparison gives a classical controller the goal the policy gets, anchored to the
table's legs. The table below does that in
MuJoCo, from the exercise's docking starts, anywhere 0.5 to 1.5 m from the goal at any
heading, and from a pose lined up 0.8 m out, where drive_to hands over from
further away. The
pose controller is the polar-coordinate law of Siegwart, Nourbakhsh and Scaramuzza,
section 3.6, with the textbook's gains.
| Dock controller, 1000 episodes | From anywhere | Lined up |
|---|---|---|
| Learned docking | 71% | 82% |
| Pose controller | 42% | 60% |
drive_to's control law | 14% | 78% |
goto_empty.zip, before docking | 35% | 52% |
Lined up, the policy leads drive_to steering on the table's legs by 5 points,
at the edge of the noise on 1000 episodes. The 1 to 19 cm of the first demo came from odometry and the final turn,
not from the control law. From anywhere, both classical controllers fall behind: the pose
controller spins near the goal when its heading is far off, and drive_to
collides in almost half its episodes. In Webots, three starts cannot rank the four. With earlier policies, inverse kinematics
reached the can after 13 of 15 trials, with one failure each for the pose controller and
the policy. With the current policy, docking directly, it docked from A in under 6 s;
from B and C it docked once and timed out three times in four runs.
Both classical rivals turn in place, which TIAGo's casters punish. A fairer rival follows a cubic Bézier path tangent to the start and goal headings, clear of the table, tracked by pure pursuit on the same goal. It has not been measured yet.
So which should have been learned: where to stand, how to drive the last metre, or how to cross a room with furniture in it? The rest of the lecture answers each with a measurement. The answers are collected at the end of the section on obstacles.
Week 2 took the environment as given. Pendulum-v1 came with its state, its
actions, its reward and its episode length. On a robot you write all of these yourself:
what the policy is shown, what it can command, what it is paid for, when an episode ends,
and which scenes it trains on. The choice between PPO and SAC is the smallest decision on
that list.
A state is Markov when the next state depends only on the present state and action:
P(st+1 | st, at) = P(st+1 | s0, a0, …, st, at)
A policy sees only its observation, so the observation is the state as far as the policy is concerned. What the observation leaves out, no amount of training can recover. Show a CartPole policy only the cart position and the pole angle. A pole at 5° falling right and a pole at 5° swinging back up give the same input and need opposite actions. Adding both velocities fixes it, and so does stacking the last two observations, from which the policy can take a difference.
The frame matters as much as the content. A goal at (3.0, 2.0) in the world frame
means nothing without the robot's own pose, so the policy has to learn the transform
from experience. The same goal in the robot frame, 1.2 m ahead and 0.4 m to
the right, looks the same wherever in the room the situation arises. A frame that makes
equivalent situations look alike is data the policy does not have to collect. The
go-to observation in isfr.envs.goto holds the goal in the robot
frame and re-expresses it at every step.
The simulator will hand you anything: the robot's exact pose, the table's exact box, the
contact forces. The robot in Webots has a lidar, odometry and a detector, and your
observe function runs there unchanged. If observe reads a
quantity, the robot must be able to measure it. Privileged information has other uses in
training, such as judging whether an episode succeeded, but never in the observation.
Three practical points follow. A policy trained on exact inputs comes to rely on a
precision it will not have, so the homework's training adds 2 cm of noise to the
can's position and 1 cm to each lidar range. A scan holds inf wherever
nothing returned, and a network cannot take inf. And inputs of very
different sizes, metres beside radians beside counts, slow learning down, so bring them to
comparable ranges.
The action space fixes what a policy can express at all. This week both are given.
Placement's action is three numbers in [−1, 1] that
isfr.envs.base.action_to_pose turns into a pose around the can. Docking's
is a linear and an angular velocity, fixed by the policy it starts from. The section on
placement shows how much of a problem a good action space can solve on its own.
A sparse reward pays 1 on success and 0 otherwise. It says exactly what you want, and it gives no signal until the first success. A shaped reward adds terms that pay for progress. It learns faster, and it also changes what is optimal unless you are careful.
Randløv and Alstrøm trained a simulated bicycle to ride to a goal and paid it for progress towards the goal. Nothing penalised riding away from it. The agent learned to ride in tiny circles near the start: each circle contains a stretch towards the goal, that stretch pays, and the rest of the circle costs nothing. Ng, Harada and Russell (1999) report this case with another one, a soccer robot paid for touching the ball that learned to vibrate next to it.
Their remedy is potential-based shaping. Choose any function of state Φ(s) and add
F(s, s′) = γ Φ(s′) − Φ(s)
to every reward. The optimal policy is unchanged. With Φ(s) equal to minus the distance to the goal, moving closer pays and moving away costs the same amount back. With γ = 1, the shaping collected round a loop that returns to its start is Φ(sT) − Φ(s0) = 0, because the sum telescopes, so the bicycle's circles are worth nothing. The paper also proves that, with nothing known about the MDP, only this form carries the guarantee. Any other shaping term defines a new task, and you have to check what that task rewards.
Every term in a reward is a price, and the agent pays whichever is cheapest. So the question to ask of a reward is what it makes optimal. What you hoped it would encourage is a different question. In the boat-racing game CoastRunners, the score pays for hitting targets along the course. An agent trained on the score found a lagoon where targets respawn, circled there while crashing and catching fire, never finished the race, and scored 20% higher than human players (Clark and Amodei, 2016). Finishing the race was never in the reward.
Russell and Norvig make the same point with a vacuum cleaner. Pay it for the dirt it picks up, and it can pick up dirt, dump it, and pick it up again. Their rule is to measure what you want in the environment, a clean floor, rather than how you think the agent should behave. A clean floor is a state. Dirt collected counts actions.
The lecture's exercise in the room is a pick-and-place arm paid +10 when the cube is in the box and charged 1 on every step its gripper touches the table. The cube sits close to the table, and grasps fail often. If an attempt costs more in table contact than it returns on average, never reaching for the cube is optimal. A penalty makes avoidance attractive, so compare the price of every way an episode can fail, including the ways in which the robot does nothing.
Gymnasium's step returns two flags. terminated means the task
itself ended, by success or a collision. Nothing follows, so the last state has no future
value. truncated means you stopped the clock. The world would have gone on, so
the value of the last state still counts, and the learner should bootstrap from it.
Confusing the two corrupts the value estimates near the limit. If the time limit is part
of the task, the time remaining belongs in the state (Pardo et al., 2018).
Stable-Baselines3 reads truncation from Gymnasium and bootstraps.
This changes how you price an episode. Running out of time is not an ending in training. A policy that stops trying pays its cost per step c until the clock runs out, and the bootstrapped value goes on charging it from there. The total is about L = c/(1 − γ) whatever the episode length, which is 100c at γ = 0.99. A reward paid on the truncated step itself arrives once, at a step the policy cannot foresee unless the time left is in its observation.
A task so hard that success never happens gives a sparse reward nothing to learn from.
Docking to 5 cm and 5° next to a table, trained with PPO from scratch for 40
minutes, stayed at 0% success. A policy trained first on an empty floor, where goals are
easy to reach, already succeeds at docking 39% of the time before any fine-tune. Train the
easy task first, then fine-tune on the hard one (Bengio et al., 2009). The homework gives
you that empty-floor policy as goto_empty.zip.
What varies between episodes is part of the design too: the layout, the walls, the
clutter, the start pose, the noise. Evaluate on scenes the policy never trained on, and
seed them. When two designs, or a design and a baseline, meet the same seeded scenes, the
comparison is fair. The homework seeds evaluation episode i with
EVAL_SEED + i for this reason.
Placement is one decision per episode. The robot scans once, proposes a base pose, and is scored by inverse kinematics from that pose: the arm reaches the can or it does not. The base must also not collide with the table, a wall or the clutter.
The action is object-relative. Its three numbers are a stand-off distance from the can between 0.3 and 1.2 m, a bearing from the can in any direction, and a heading error of up to 45° from facing the can. Every pose it can express stands near the can and faces it.
The classical baseline searches the same action space. It draws an action at random, checks the pose with inverse kinematics in about 14 ms, and keeps the first pose that reaches the can. It solves every one of 200 scenes with a median of 4 samples and 13 ms per decision. Of 500 generated scenes, 499 can be solved at all within 64 samples.
| Placement, 200 fresh scenes | Success | 0 walls | 1 wall | 2 walls |
|---|---|---|---|---|
| Naive design, seed 0 | 80.5% (74.5–85.4) | 78.2% | 85.5% | 75.7% |
| Naive design, seed 1 | 80.5% (74.5–85.4) | 79.3% | 84.2% | 75.7% |
| The instructor's design | 89.0% (83.9–92.6) | 92.0% | 86.8% | 86.5% |
| Sampler | 100% |
The naive design in the week's example sees only the can's position and is paid 1 for success. Trained with SAC for 200 000 steps, it solved 8.5 points fewer scenes than the instructor's design, and the two 95% intervals still overlap. A run with the same seed repeats exactly. Two seeds of the naive design solved 161 scenes each, with different policies; on earlier runs two seeds differed by 5 scenes of 200, which is the noise between runs. The action does most of the work. It already restricts every pose to the right distance, facing the can, and leaves the policy to choose a side to approach from and a sensible distance. The walls are where a lidar should help. With two walls the instructor's design scores 86.5%, against 76% for both naive seeds, on 37 scenes. Without walls it also leads, 92% against 78 and 79%.
What learning buys here is time. A forward pass of the network takes under 1 ms. The sampler needs one or more inverse kinematics solves at about 14 ms each. Training is paid once and every later query is cheap, so the policy is the sampler's search, amortised. That matters where placement is queried many times, for example by a planner that asks where to stand for every candidate place in an apartment. For a single fetch it does not.
A reward of 1 for any reachable pose is indifferent to how far the robot has to drive to
get there, and to how far the arm has to stretch. The table below measures the distance
and the turning from the start, the stand-off from the can, and how many poses come within
10 cm of an obstacle, on the 127 scenes that every row here and the next table solved.
A median over each one's own successes would credit the policies with the easy scenes the
sampler also solves and leave out the hard ones only the sampler solves.
isfr.envs.base.travel computes the distance and turning for one pose, and
sample_placement(..., nearest=True) tries all 64 samples and keeps the
accepted pose with the least travel.
| Placement | Success, of 200 | Distance, median (IQR) | Turning, median | Stand-off, median | Within 10 cm | Time per decision |
|---|---|---|---|---|---|---|
| Naive design, seed 0 | 161 | 0.86 m (0.53–1.25) | 64° | 0.98 m | 9 | under 1 ms |
| Naive design, seed 1 | 161 | 0.88 m (0.55–1.23) | 70° | 0.99 m | 10 | under 1 ms |
| Instructor's design | 178 | 1.34 m (0.85–1.82) | 77° | 0.79 m | 1 | under 1 ms |
| Sampler, first accepted | 200 | 1.60 m (1.06–2.26) | 129° | 0.82 m | 21 | 9 ms |
| Sampler, nearest of 64 accepted | 200 | 1.01 m (0.69–1.39) | 46° | 0.90 m | 28 | 230 ms |
Paid 1 for success, the naive policy stands at the edge of the arm's reach, 0.98 m from the can, where inverse kinematics reaches no further than 1.08 m. A probe of the same scenes found a reachable pose 10 cm clear of everything at a median 0.53 m. The instructor's design stands 0.79 m off and comes within 10 cm of an obstacle once, and drives about 0.5 m further than the naive policy to do it. The 10 cm is the margin Webots needs, since the base stops 3 to 10 cm from where it is sent. The samplers accept any pose the arm reaches, and come within 10 cm of an obstacle 21 and 28 times. The nearest sampler still succeeds 11 points more often than any policy and turns less, and it costs about 230 ms a decision.
A classical step can bring the base closer still. isfr.envs.base.slide_in keeps
the policy's approach bearing and heading and moves the base in towards the can, to the
nearest stand-off that inverse kinematics accepts with 10 cm clear. Stopping at the
first stand-off that passes leaves the base on the edge of what the arm reaches: after a
5 cm, 3° arrival error, inverse kinematics still reached the can from only 64% of
such poses. So slide_in also requires every stand-off 10 cm further in to
pass.
| Placement, instructor's design | Learned only | Learned, then slide_in |
|---|---|---|
| Solved, of 200 | 178 | 172 |
| Stand-off, median | 0.79 m | 0.75 m |
| Inverse kinematics still reaches after a 5 cm, 3° arrival error | 87% | 85% |
| Time per decision | under 1 ms | 47 ms |
It buys 4 cm for 47 ms a decision, at the same robustness. In Webots, from start C,
drive_to stopped 17 cm from the learned pose and the arm could not reach the
can; from the slide_in pose it stopped as far off and the arm reached. Part 1 of
the homework leaves the choice to you.
The two can be combined, with learning deciding the order and inverse kinematics keeping the last word. The sampler draws 64 candidate poses. A classifier, trained on inverse kinematics verdicts, scores each candidate's chance of success, and the check runs on the best first. If none passes, 64 random candidates follow. A probe of 2026-10-02 trained the classifier on 200 000 random poses, seeing the can and 64 beams with the training noise. The ranges below span a default and a stronger inverse kinematics.
| Order, 200 scenes | Solved | First candidate accepted | IK calls, median | Time, median |
|---|---|---|---|---|
| Random | 197 | 39–46 | 3 | 11–21 ms |
| Learned ranking | 198 | 151–163 | 1 | 2 ms |
The ranking keeps the sampler's success and usually needs one check. Used alone, as a one-shot policy, the classifier is 83.7% accurate, about where SAC ended before the simulator fix described under Obstacles, when this probe ran. The saving grows with the cost of the check, which rises once the arm plans its own reach. Learning becomes a heuristic inside the classical method. The homework still trains the SAC policy.
Stulp, Fedrizzi, Mösenlechner and Beetz (2012) treat the same problem a third way. They run many simulated grasps from base poses around an object and learn, in the object's frame, the probability that a grasp from each pose succeeds. At query time they sample where the object might be, given the robot's uncertainty about it, and choose the pose most likely to succeed over all of those samples. The robot keeps a region of good places rather than committing to one pose early, which the paper calls least commitment.
Placement in this course is one step, rewarded 1 for success and 0 otherwise. For such an episode
Q(s, a) = E[r | s, a] = P(success | s, a)
so SAC's critic estimates the quantity an action-related place represents, and its actor learns to output the best action directly, where Stulp et al. search. With one step there is no next state, so SAC's entropy term affects only the actor. This reading is the course's own. The paper states that an action-related place is not a reinforcement learning policy. The ranking classifier above learns the same quantity by supervised learning, and leaves the search to the sampler.
Their baseline is a fixed offset from the object. The sampler here checks every candidate with inverse kinematics before accepting it, which is a stronger bar. Against it, a learned placement can only win on time.
The policies train in MuJoCo and run, with the same code, on the Webots TIAGo. Two simulators disagree in ways a real robot would too. They also disagree wherever one of them, or the code between them, is wrong. Three bugs came out this way.
The lidar was mirrored. A box on the floor to the robot's left read at −94°, on its right. Webots orders the range image from left to right, while the bridge's header claimed right to left. Every test passed, because both backends agreed with the wrong header. A policy trained in MuJoCo would have met the mirror image of its world. The bridge now reverses the range image.
The robot sees itself. In Webots, 52 lidar beams hit TIAGo's own body at 0.23 to 0.27 m, at 108° to 117° either side. MuJoCo's scan never does. Returns under 0.3 m are now read as no return, only beyond 100° either side. The first version of that mask applied at every angle, and in 66% of one navigation policy's collisions the obstacle was a return the mask had thrown away. The fix for one simulator had broken the other. The 66% was measured before the simulator fix described under Obstacles.
The arm was out. Training assumed the arm tucked, and in Webots it was not. From two of three starts the palm met the table during the final turn, and the base stopped 59° and 60° off the proposed heading. With the arm tucked, all three ended within 5°. Tucking also levelled the base, which removed most of the floor returns on its right.
TIAGo has four passive casters. After a spin in place they trail sideways, and the next straight leg drags through them. In week 1's square, spinning in place and holding the heading closed to 0.418 m by odometry and 1.443 m in truth. Turning as an arc closed to 0.084 m and 0.174 m.
Commanded to turn 160° in place with the arm tucked, the Webots base reached 72 to 78°. The wheels reach half the commanded speed, because the motors' 10 N m saturate against the casters. MuJoCo as vendored, under the same limits, turned 142°. A policy trained there would turn twice as far as it meant to.
The obvious parameters did not close the gap. Raising the casters' swivel damping up to 100 times, and doubling their friction, left the spin above 114°. Load did close it. In Webots the casters sit on stiff springs level with the drive wheels, and lowering MuJoCo's casters by 0.4 mm moves weight onto them. The spin fell to 68 to 76°, and the drift after the turn averaged 35.4° against Webots' 33 to 40°. The response is steep: 0.75 mm gives 58 to 64°, and at 1 mm the drive wheels lift off the floor.
No single value is right, so training randomises it. Each episode draws its own caster drop, from 0.25 to 0.55 mm, and its own damping, from half to twice nominal. That spans spins of about 55 to 90°, and Webots becomes one draw among many. Tobin et al. (2017) randomised a simulator's appearance for the same reason, and Peng et al. (2018) its dynamics.
Odometry drifts, so in Webots the docking goal is re-anchored at every step to the table.
The lidar sits 9 cm above the floor and sees the table's legs, never its top, and
isfr.envs.landmarks fits the table's pose to them. In Webots, 60 of 60 scans
fitted, with median errors of 4 to 11 mm. In MuJoCo, a docking policy trained on the
simulator's exact pose scored 68.5%, and 69% when the same file was scored on odometry and
the landmark. Odometry alone, with no landmark, scored 4.5%. In an earlier run, before the
simulator fix described below, training with the landmark in the loop did worse, at 38%. The landmark is close enough
to the truth that the policy does not need to learn its noise.
go-to | Training | Success |
|---|---|---|
| PPO, empty floor, from scratch | 10 minutes | 100% of 200 |
| The empty-floor policy, among obstacles | none | 42.5% |
| PPO, obstacles, from the empty-floor policy | 27 minutes | 46.5% |
| SAC, obstacles, from scratch | 1 hour | 10% |
On an empty floor, go-to trains in minutes. Among furniture, 2.4 million
more steps moved the empty-floor policy from 42.5% to 46.5%, which is inside the noise of
200 episodes. SAC from scratch scored 10% after an hour and timed out in 86% of scenes.
The robot learns not to crash before it learns to arrive. Before 3 October 2026 the
simulator kept every wall and box at the same heading, whatever the scene asked for, and
several fixes were tried in that world: correcting the self-hit mask, keeping starts 0.8 m clear of obstacles,
penalising close passes, giving the policy the exact pose instead of odometry, switching
to SAC, another reward, a depth camera on the head, and progress measured along the free
path instead of the straight line. Each moved the result by less than the noise, or not
for long. They have not been repeated in the corrected world.
| Straight line to the goal | Scenes | Success |
|---|---|---|
| Clear | 99 | 90% |
| Blocked | 101 | 5% |
The policy sees the obstacle and has room to pass it, and still drives the straight line. The table above is the warm-started policy after 2.4 million steps, on 200 seeded scenes, with each table counted as solid. 72% of its collisions were inside the lidar's view. Where it hit, the widest route round was a median 1.10 m, for a base 0.54 m wide. It has not learned to detour. The course's answer is to let perception name the obstacle, the world model record it, and a graph search plan the route, with learning kept for the last metre. Weeks 4 and 5 build that graph.
| Problem | Learned? | Why |
|---|---|---|
| Where to stand | Optional | A sampler succeeds, and learning makes it fast |
| The last metre | From anywhere | Lined up, a classical controller on the table's legs comes within 5 points. No arc-path rival yet. |
| One edge, among obstacles | The driving | Motion tubes choose where to drive, and a shield vetoes what is not free |
| Across a room | Not yet | Planned over the world model instead |
The rule the course follows is to learn what classical methods do badly.
go-to
The policy does not detour, so go-to does not leave the route to it. At every
step, motion tubes (Rodrigues et al., 2023) check a fan of 102 arcs against the lidar
scan, forward up to 1.2 m and back up to 0.2 m. A tube is free when no return
lies inside the area the base would sweep along it. The policy drives to a point
0.5 m along the free path whose end lies nearest the goal. A path is one tube at
least 0.5 m long, or a forward tube followed by one of the 1.2 m tubes. A tube
that first leads away therefore counts by where the tubes continuing from it end. A
shield then checks the policy's own command over the next second and, when its arc is
not free, replaces it with the nearest free one. Within 0.5 m of the goal the shield
steps aside and the policy docks alone.
The lidar sees 240°, so the ground behind the base is checked against the scans of
the last 10 s, and ground the base itself stood on counts as free. The head's depth
camera adds what the lidar, 9 cm above the floor, cannot see, such as a tabletop.
Its points are kept for 20 s. The code is isfr/envs/local.py,
isfr/envs/tubes.py, isfr/envs/scan_memory.py and
isfr/envs/camera_memory.py.
go-to is a skill like week 2's grasp. Give it the goal relative
to where the robot stands when it starts. Odometry carries the goal from there.
from isfr.envs.base import Pose
from isfr.robot import Robot
from isfr.skills.go_to import GoTo
with Robot() as robot:
skill = GoTo(robot, Pose(2.0, 0.5, 0.0))
print(skill.run_to_completion(max_ticks=skill.max_ticks))
Pass max_ticks. The default of 500 ticks is 48 s, shorter than the
skill's 60 s timeout. The skill tucks the arm and tilts the head before the base
moves. It succeeds when the base has stopped within 10 cm and 10° of the goal.
It fails with one of three reasons:
The policy is data/policies/go-to.zip, PPO with the observation and actions
of goto_empty.zip, fine-tuned from it among obstacles with the casters
randomised. It was then trained for 90 more minutes at 3e-5 on the base model
recalibrated against Webots, with each command landing 0 to 64 ms late, as it does
there. It was scored in MuJoCo on 200 seeded scenes of each kind, every command 32 ms
late.
| Scenes | Outcome | go-to.zip | Target |
|---|---|---|---|
| Clutter, passable | Success | 85.0% (79.4 to 89.3) | 75% or more |
| Clutter, passable | Collision with a box, leg or wall | 0.0% | 5% or less |
| Clutter, passable | Upper body on a tabletop | 0.0% | None set |
| Clutter, passable | "edge blocked" | 12.5% | 20% or less |
| Clutter, passable | "did not settle" | 2.5% | None set |
| Two rooms, doorway plugged | "edge blocked" within 30 s | 100% | 90% or more |
| Two rooms, doorway plugged | Collision | 0% | 5% or less |
Success is given with its 95% Wilson interval. It meets every target. About one crossing in seven still ends in failure, most often an "edge blocked" in a passable scene, where the base has stalled in a pocket of clutter. Plan for it, since a failed termination is what triggers repair and replanning.
In Webots the same policy and code ran in webots/worlds/isfr_obstacles.wbt,
two rooms joined by a doorway with two boxes in the first, and in
isfr_obstacles_blocked.wbt, where the doorway is plugged. In the open world
it reached none of five goals. In four runs of five the base stopped beside the first box,
and in the fifth it crossed and stopped 19 cm off the goal. With the doorway plugged
it returned "edge blocked" every time. In 1 of these 10 runs the base touched the first box,
while turning beside it. A copy
of the open world in MuJoCo fails the same way on every seed, so the cause lies in
go-to and not in the transfer. The section "What training
go-to taught us" below describes what was found.
The fine-tuning taught one thing about the training loop. At PPO's default learning rate,
3e-4, which is the rate stored in goto_empty.zip, the obstacle policy held
for 90 minutes of further training and then fell from 758 to 670 of 1000 scenes by three
hours. At 3e-5 it reached 819 after one hour and stayed between 786 and 816 for two more.
A warm start begins near a good policy, and large updates can move it away. Docking gains
from a lower rate too, and train-docking below runs at 1e-4. The first
shipped file was the 3e-5 run at one hour, and the current one continues from it at the
same rate. The rates the loop printed did not show the difference. They
stayed between 35% and 57% in both runs, because they score the policy alone, without
tubes and shield.
In the micro-project you design the observation and the rewards and write the
measurements. The training loops are provided in main.py, and this section
explains what they do, step by step.
Both loops wrap the environment in Stable-Baselines3's SubprocVecEnv, and
that in VecMonitor, which records how each episode ended. Each
worker is a separate process with its own MuJoCo simulation and its own random scenes,
seeded with the worker's index. One call to the vectorised environment steps every worker
once, so the algorithm sees as many transitions per call as there are workers. By default
the loops use half your CPU cores, and --workers changes that.
train-placement: SACdesign.py, with
2 cm of Gaussian noise on the can's position and 1 cm on each lidar range.
The pose is computed from the noisy can position, as the robot would, and scored against
the true one.
N workers are followed by N
steps. The batch mixes old and new experience, which is what makes SAC off-policy.
learn in chunks of 2000 steps. After each it prints
the elapsed time, the steps taken, and the success rate of the last 100 training
episodes. It saves placement-policy.zip every
five minutes and at the end, and stops after --steps, 200 000 by
default. SAC's default is one gradient step per call, whatever the number of workers,
which would give 4 workers twice the updates of 8 from the same transitions.
train-docking: PPO from the empty-floor policydocking_reward. The
observation and the actions come from isfr.envs.goto: 64 lidar beams, the
goal in the robot frame, and the previous action. The policy acts every 96 ms.
Training uses the simulator's exact pose and 1 cm of range noise.
goto_empty.zip, its network and its settings. The actor and the
critic each have two hidden layers of 256 units. Your reward replaces the one it was
trained with.
N workers, each collects 12 288 / N.
--steps environment steps, 1.8 million by default, saving
docking-policy.zip every five minutes and at the end.
Three of these choices are deliberate. The update size is held at 12 288 steps,
whatever the number of workers, because goto_empty.zip was trained with
updates of that size on 12 workers. Stable-Baselines3's default of 1024 steps per worker
gives 4 workers updates a third that size. In two seeded runs of that setting, docking
ended at 17% and 11%, below the 32% it started from. Holding the update at
12 288 steps, the same 4 workers ended at 50.5% and 55.5%. These runs predate the
simulator fix described under Obstacles.
The learning rate is lowered to 1e-4 from the 3e-4 stored in goto_empty.zip,
which is PPO's default, for the reason given for go-to above. At 3e-4, two
seeds of the solution's reward ended at 69.0% and 56.5%. At 1e-4 the same seeds ended at
78.0% and 71.0%, and a third at 80.0%. Most of the difference is timeouts, 10.0% and 11.0%
at 3e-4 against 1.0% to 8.0% at 1e-4.
Training stops after a number of steps, and not after a time, so that a slower laptop takes longer and trains the same amount. On the instructor's laptop, 4 workers took 35 minutes. Before the simulator fix, at 3e-4, every setting tried dipped somewhere between 600 000 and 900 000 steps and then recovered, and with the fixed update size success fell as low as 21% in that stretch. At 1e-4, evaluation rose from between 55.0% and 62.5% at 600 000 steps to between 71.0% and 80.0% at the end. One of the three runs peaked at 76.0% at 1.2 million steps. A time budget can stop a slow machine short. Evaluate the final file.
The rates the loops print are not an evaluation. The policy samples its actions to
explore, training adds sensor noise, and docking trains on the exact pose. Use them to
follow a run. Score a policy with model.predict(obs, deterministic=True), which takes the
mean action instead of sampling. Reset episode i with
seed=EVAL_SEED + i, so that two policies, or a policy and the sampler, meet
the same scenes. Placement is scored without training noise. Docking is scored the way it
runs in Webots, on odometry with the goal anchored to the table's legs, while success is
still judged on the simulator's exact pose.
Report a success rate with its 95% interval. Use the Wilson score interval, which stays inside 0 to 100% where the normal approximation does not. On 200 episodes, a difference of a few points between two policies is inside that interval, as week 2 showed for rewards.
Keep a set of seeds you never choose on. Five go-to checkpoints were scored
on seeds 70000, and the best reached 84.5%. On seeds 80000, where nothing had been
chosen, it reached 81.0%. Choosing the best of five on a set also picks the one that
set happened to favour. Report the number from the unused seeds.
go-to taught us
Training go-to among obstacles is the part of this course's preparation
that did not finish. In MuJoCo the shipped skill succeeds in 85% of passable cluttered
scenes. In Webots, in the two rooms of isfr_obstacles.wbt, it reaches none of
five goals. The faults found on the way are listed below, grouped by kind. Most of them
concern time, geometry and measurement, and few concern the learning algorithm. Each cost
more than a training run.
The trap at the start of the open world is not solved. The base turns to face the first box, and from 0.3 m every forward arc brings it nearer. Only reversing or turning on the spot gets it out, and the policy has never had to learn either, since it was trained without the shield. Training with the shield in the loop reached 8, 0 and 2 of the 25 goals at three checkpoints of one run. Getting the robot out is yours to solve in the project.
Run from the repository root:
PYTHONPATH=. python3 examples/03-training-policies/main.py
PYTHONPATH=. python3 examples/03-training-policies/main.py --webots
The first trains the naive placement design with SAC for 10 000 steps, about two
minutes, and scores it on 100 fresh scenes next to the classical sampler. The second needs
webots/worlds/isfr_placement.wbt open and running, from a shell that has
sourced scripts/webots-env.sh. It runs the sampler and drive_to
on the Webots TIAGo, as in the lecture's first demo. The lab opens with the example.
This week's exercise is exercises/03-training-policies/. Its
README.md states the task and ends in a Self Check list you
can verify unaided. Record your results in RESULTS.md as you go.
Part 1 is placement. You design the observation and the reward, train, and compare your
design with the naive one and with both samplers on 200 scenes, on success, travel and
time. Part 1 has no success threshold. Your RESULTS.md says whether your
design beats the naive one by more than the noise between runs, and why. Then you run your
policy in Webots from three starts.
Part 2 is docking. You design the reward, fine-tune goto_empty.zip, and
measure it before and after. You also measure how far the robot can miss a placement pose
before the arm can no longer reach the can, and chain your policy after
drive_to in Webots. Part 2 ends with a paragraph that prices each way an
episode can end under your reward and names the cheapest. Work it out from the reward.
At 1.8 million steps, a reward with a trap in it can score as well as a sound one, so a
good score does not settle the question.
Start train-placement early in the lab. It took about 10 minutes on 4
workers of the instructor's laptop, and more workers barely shorten it.
train-docking took 35 minutes on 4 workers of the instructor's laptop. A
slower laptop takes longer, and the deadline allows training at home. The micro-project runs from the inputs it provides. It does not
need any earlier week's output.
Cited above, for reference:
Intelligent Systems for Robotics · course materials