Intelligent Systems for Robotics · Week 2
Every layer above talks to skills, and to nothing else.
None of that can be designed until a skill has a fixed shape.
The first answer is the outside of a skill. The second is what goes inside it. The outside stays the same either way.
PYTHONPATH=. python3 examples/02-grasp-skill/main.py
grasp closes on nothing and terminates in failure.
release then terminates in success.
gripper closed on nothingPart 1
What exactly is a skill?
\[ \langle \mathcal{I}, \pi, \beta \rangle \]
Sutton, Precup & Singh (1999), Artificial Intelligence 112(1–2), 181–211.
A function call
Runs, returns, and the caller trusts the result.
An option
Runs one tick at a time, is judged after every tick, and stops with a verdict.
A robot skill takes seconds. The world moves while it runs, and it can end without having done its job.
| Plan in | Steps to fetch a drink |
|---|---|
| Motor commands, one every 32 ms | Thousands |
| Options | About a dozen |
This was Sutton, Precup and Singh's reason for options. The verdict on failure is this course's addition.
can_start
def can_start(self, state: dict) -> bool: ...
When may this skill be started at all?
grasp answers: the finger must be near its open limit. Whether
an object sits in the jaws is a world fact, and the planner checks that one.
step
def step(self, obs: dict) -> None: ...
One tick. It issues commands and returns nothing.
The skill never advances the simulator. It waits for the next sample, and the world moves on its own.
is_done
def is_done(self, state: dict) -> Termination: ...
It returns a Termination. A bool could say only that the skill stopped.
Success and failure are different facts about the world. Week 8 replans on one of them and not the other.
Termination, in code
Termination.succeeded("holding the object")
Termination.failed("gripper closed on nothing")
Termination.RUNNING
def __bool__(self) -> bool:
return self.outcome is not Outcome.RUNNING
Truthy once the skill has terminated either way, so
if skill.is_done(state): reads as it should. The reason string
is what an execution layer logs and replans on.
class Skill(abc.ABC):
name: str
def can_start(self, state: dict) -> bool: ...
def step(self, obs: dict) -> None: ...
def is_done(self, state: dict) -> Termination: ...
run_to_completion drives them: reset, check the initiation set
once, then sense, judge, step and wait until a verdict comes back.
grasp's verdict, in codeClose the finger. Once it stops moving, where it stopped is the verdict.
def _settled(self, pos: float) -> Termination:
if self._at(pos, self._closed):
return Termination.failed("gripper closed on nothing")
if self._at(pos, self._open):
return Termination.failed("gripper did not close")
return Termination.succeeded("holding the object")
Until the finger stops, is_done returns RUNNING. A
finger that never stops within the tick budget is a failure too.
Atom("holding", ("?obj",)) # (holding ?obj)
Atom("gripper-empty", negated=True) # (not (gripper-empty))
These are in PDDL's shape on purpose. Week 6 hands them to a planner without translating them.
can_start or a precondition?can_start
What the skill can sense, now.
grasp: the finger is near its open limit.
Preconditions
What the planner knows about the world.
grasp: (gripper-at ?obj) and (gripper-empty).
The skill cannot see whether an object is between the jaws. The planner cannot see where the finger is. Each checks only what it can know.
SkillSpec(
name="go-to",
parameters=("from", "to"),
preconditions=(Atom("at", ("?from",)), Atom("connected", ("?from", "?to"))),
effects=(Atom("at", ("?to",)), Atom("at", ("?from",), negated=True)),
learned=True,
implementation=None,
summary="drive the base along one topological edge, handling obstacles the map does not know",
)
In the registry, as data. The class holds none of it.
go-to has
implementation=None, and nobody has trained it yet
SkillSpec(name="grasp", ..., implementation="isfr.skills.grasp:Grasp")
registry.spec("grasp") # preconditions and effects; nothing imported
registry.load("grasp") # imports isfr.skills.grasp, returns the Grasp class
A plan step names a skill as a string. load() turns the string
into a class, and only then is the module imported.
| Skill | Policy | What it does |
|---|---|---|
go-to | learned | Drive the base along one edge of the map, around obstacles the map does not know |
navigate-to | provided | Follow a route through the map, one go-to per edge |
place-base-for | learned | Move the base to a pose from which the object is reachable |
reach-to | learned | Collision-free arm motion to a goal configuration from IK |
ik-solve | provided | Classical inverse kinematics |
grasp | provided | Close the gripper until the fingers stall on the object |
release | provided | Open the gripper until the fingers reach the open stop |
hand-over | provided | Offer the held object to a person, then release it |
detect | provided | Ground object symbols in camera frames, with YOLO |
Three learned, six provided. Learn only what classical methods do badly.
Part 2
How do you get a policy you cannot write by hand?
grasp.step() is a few lines you could write yourself.
Now write go-to's.
A rule you write handles one arrangement of furniture and breaks on the next. A room has more arrangements than you can list.
The classical stack rebuilds the cost map from every scan. It works, and week 3 measures against it. The learned policy has no map and no path in between.
That is enough. The rest of this part is how.
Pendulum-v1The motor can hold the pendulum only within 24° of upright. From anywhere lower, it has to swing.
\[ (\mathcal{S}, \mathcal{A}, p, r, \gamma) \]
The dynamics are never written down. The simulator has them.
\[ r = -\left( \theta^2 + 0.1\,\dot\theta^2 + 0.001\,u^2 \right) \]
The terms charge for the angle from upright, the spin, and the effort, the torque \( u \).
One number per tick. Hanging straight down costs about 10 a tick. Standing still at the top costs nothing.
The reward never says how to swing. It says only where the pendulum should end up.
\[ G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1} \]
What the whole attempt was worth, seen from tick \( t \).
Reward is per tick. Return is per trajectory.
\[ \gamma \in [0, 1) \]
\[ \pi_\theta(a \mid s) \]
\( \theta \) is a neural network's weights. Learning is adjusting \( \theta \), and nothing more.
\[ J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\!\left[ G_0 \right] \]
Maximise the return you expect, averaged over the trajectories your own policy produces.
\[ V^\pi(s) \]
What this state is worth, if I keep behaving like this.
\[ Q^\pi(s, a) \]
What this action in this state is worth, same condition.
Both are predictions of return. No Bellman derivation here, because nothing in this course needs it.
There are no right answers to collect and fit. So try, and learn from the score.
Push up the log-probability of actions that led to high return. Push down the others.
\[ \nabla_\theta J(\theta) = \mathbb{E}\!\left[ \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, G_t \right] \]
\[ A^\pi(s_t, a_t) = Q^\pi(s_t, a_t) - V^\pi(s_t) \]
Was this action better than what I usually do here?
Subtracting \( V \) removes the part of the return that was about the state rather than the action.
PPO and SAC are both actor-critic methods.
\[ r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_\text{old}}(a_t \mid s_t)} \]
How much more likely the new policy makes an action than the policy that collected the data did.
At 1, nothing has changed. At 1.5, the new policy makes the action 50% more likely.
\[ L(\theta) = \mathbb{E}\!\left[ \min\left( r_t(\theta) A_t,\; \mathrm{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon) A_t \right) \right] \]
Once an update has moved an action's probability by more than \( \epsilon \), 0.2 by default, the clip throws the extra improvement away.
Each update stays small, so training rarely collapses and the default settings work on tasks like these.
Left: once a good action is 20% more likely, pushing further earns nothing. Right: once a bad action is 20% less likely, the same.
| Knob | What it does | Wrong value |
|---|---|---|
n_steps | Experience gathered per update | Too small and the advantage estimate is noise |
| batch size | Samples per gradient step | Too large and each epoch sees too few updates |
| learning rate | How far each step moves \( \theta \) | Too high and it diverges, too low and nothing happens in the time you have |
| \( \gamma \) | How far ahead it cares | Too low and the policy never plans past the next tick |
| clip range | How much one update may change the policy | Too wide and the stability PPO exists for is gone |
CartPole-v1The demo trains it with PPO because PPO goes from nothing to a perfect score in under ten seconds. Your homework is the pendulum with SAC, because its action is a continuous torque.
model = PPO("MlpPolicy", "CartPole-v1", seed=0)
model.learn(total_timesteps=25_000)
mean, std = evaluate_policy(model, env, n_eval_episodes=20, deterministic=False)
model.save("cartpole-policy")
reloaded = PPO.load("cartpole-policy")
This is examples/02-rl-fundamentals/main.py, which you run in
the lab.
On-policy
Throws its data away after each update. PPO works this way.
Off-policy
Keeps a replay buffer and reuses every sample many times. SAC works this way.
The consequence is sample efficiency, and on a real robot samples are the expensive thing.
Your micro-project uses it for those three reasons. PPO, tuned, needs 100 000 steps on the same task and still scores −217 ± 128.
Haarnoja, Zhou, Abbeel & Levine (2018), Proceedings of the 35th ICML, PMLR 80, 1861–1870.
\[ J(\pi) = \sum_t \mathbb{E}\!\left[ r(s_t, a_t) + \alpha\, \mathcal{H}\big(\pi(\cdot \mid s_t)\big) \right] \]
ent_coef="auto"Part 3
Up to here, every claim was about what an algorithm is supposed to do. From here on, every claim is a measurement.
PYTHONPATH=. python3 examples/02-rl-fundamentals/main.py
steps mean reward std
0 25.1 17.2
5000 47.2 27.3
10000 191.3 40.9
25000 500.0 0.0
25000 steps in 7.7s
500 is the maximum. The episode truncates there, and it pays 1.0 per tick.
At 5 000 steps the policy averages 47 with a spread of 27, so it balances well sometimes and fails early often. At 25 000 every episode scores 500 and the spread is zero.
Now the pendulum. One saved SAC policy, scored on fresh episodes.
| Episodes per reading | How far the mean moves between readings |
|---|---|
| 10 | ± 26 |
| 20 | ± 19 |
| 100 | ± 8 |
Two 20-episode means closer than \( 1.96 \sqrt{2} \cdot 19 \approx 50 \) are a tie.
Most of the noise is the start angle. From hanging down the policy averages −264, from near upright −57.
Part 4
This is a pendulum. Control engineers solved it before most of you were born. Why train anything?
def energy_shaping(obs):
cos_theta, sin_theta, rate = obs
theta = np.arctan2(sin_theta, cos_theta)
if abs(theta) < 0.8:
return float(np.clip(-8.0 * theta - 2.0 * rate, -2.0, 2.0))
energy = 0.5 * rate**2 + 10.0 * cos_theta
return float(np.clip(rate * (10.0 - energy), -2.0, 2.0))
No training, no network, no data. Pump energy in, then hold near the top.
Åström & Furuta (2000), Automatica 36(2), 287–295.
| Policy | Reached upright | Cost |
|---|---|---|
| SAC, 10 000 steps | 20/20 | 49 s of training |
| Energy shaping, 8 lines | 20/20 | none |
| Random | 0/20 | none |
Parity. Fifty seconds of training and a policy network bought nothing here. Reward cannot separate them either, and by slide 54 it could not have.
Because you knew \( m \), \( g \) and \( l \).
go-to in a furnished apartment has no such model, no closed
form, and no eight lines. That is where learning earns its place, and it is
what week 3 is about.
Part 5
Pendulum-v1, 10 000 steps, about 50 sSwingUpSkill, the contract around the trained policySkillSpec with Atoms, registered as swing-upexercises/02-skill-model/. No Webots, no world.
What --watch shows. Top: --run-untrained, random torques.
Bottom: --run, SAC after 10 000 steps, from the same four starts.
| Method | For swing-up |
|---|---|
can_start | When may a swing-up start at all? |
step | Ask the trained policy for a torque, and apply it |
is_done | Yours to design. The next slide |
spec | What a swing-up needs, and what it achieves |
step is the mechanical one. The questions in the other rows are
yours to answer.
is_done is the design workGymnasium never terminates Pendulum-v1. It truncates at 200 ticks, whatever happened.
Pendulum-v1 starts at a random angle, so some starts are already upright.
If can_start rejects them, about one episode in sixteen fails
with initiation set not satisfied. Your policy will look broken
when it is not.
Decide what to do about it in reset(). There is more than one defensible answer.
--run reaches upright in at least 18 of 20 episodes--run-untrained fails every episode, with a reason that names your tick budgetregistry.load("swing-up") returns your classThe README's Self Check has the full list. It gates on successes, never on reward.