The skill model, and learning basic skills

Intelligent Systems for Robotics · Week 2

Where we are

Execution and monitoring Planning World model Skills Robot this week weeks 6 to 8 weeks 4 and 5

Why the bottom layer comes first

Every layer above talks to skills, and to nothing else.

  • Week 4: the world model records what skills change
  • Week 6: the planner chains skills into a plan
  • Week 8: the executor watches skills fail

None of that can be designed until a skill has a fixed shape.

Two questions

  • What exactly is a skill?
  • How do you get a policy you cannot write by hand?

The first answer is the outside of a skill. The second is what goes inside it. The outside stays the same either way.

Live: a skill running


PYTHONPATH=. python3 examples/02-grasp-skill/main.py
					

grasp closes on nothing and terminates in failure. release then terminates in success.

What you just watched

  • No exception, no crash, no traceback
  • A verdict, with a reason: gripper closed on nothing
  • A failure that is not an error

Today

  1. The contract: what a skill is, and the interface every skill here implements
  2. Learning a policy: from the MDP up to PPO and SAC
  3. Evidence: training something live, and what one result is worth
  4. The classical baseline, and when learning earns its place
  5. Your micro-project

Part 1

The contract

What exactly is a skill?

A skill is an option

\[ \langle \mathcal{I}, \pi, \beta \rangle \]

  • \( \mathcal{I} \): the initiation set. Where the skill may start
  • \( \pi \): the policy. What it does, one tick at a time
  • \( \beta \): the termination. When it stops, and how it went

Sutton, Precup & Singh (1999), Artificial Intelligence 112(1–2), 181–211.

Why an option, and not a function call?

A function call

Runs, returns, and the caller trusts the result.

An option

Runs one tick at a time, is judged after every tick, and stops with a verdict.

A robot skill takes seconds. The world moves while it runs, and it can end without having done its job.

Options also set the planner's step size

Plan inSteps to fetch a drink
Motor commands, one every 32 msThousands
OptionsAbout a dozen

This was Sutton, Precup and Singh's reason for options. The verdict on failure is this course's addition.

\( \mathcal{I} \) becomes can_start


def can_start(self, state: dict) -> bool: ...
					

When may this skill be started at all?

grasp answers: the finger must be near its open limit. Whether an object sits in the jaws is a world fact, and the planner checks that one.

\( \pi \) becomes step


def step(self, obs: dict) -> None: ...
					

One tick. It issues commands and returns nothing.

The skill never advances the simulator. It waits for the next sample, and the world moves on its own.

\( \beta \) becomes is_done


def is_done(self, state: dict) -> Termination: ...
					

It returns a Termination. A bool could say only that the skill stopped.

Success and failure are different facts about the world. Week 8 replans on one of them and not the other.

Termination, in code


Termination.succeeded("holding the object")
Termination.failed("gripper closed on nothing")
Termination.RUNNING

def __bool__(self) -> bool:
    return self.outcome is not Outcome.RUNNING
					

Truthy once the skill has terminated either way, so if skill.is_done(state): reads as it should. The reason string is what an execution layer logs and replans on.

The contract, in code


class Skill(abc.ABC):
    name: str

    def can_start(self, state: dict) -> bool: ...
    def step(self, obs: dict) -> None: ...
    def is_done(self, state: dict) -> Termination: ...
					

run_to_completion drives them: reset, check the initiation set once, then sense, judge, step and wait until a verdict comes back.

grasp's verdict, in code

Close the finger. Once it stops moving, where it stopped is the verdict.


def _settled(self, pos: float) -> Termination:
    if self._at(pos, self._closed):
        return Termination.failed("gripper closed on nothing")
    if self._at(pos, self._open):
        return Termination.failed("gripper did not close")
    return Termination.succeeded("holding the object")
					

Until the finger stops, is_done returns RUNNING. A finger that never stops within the tick budget is a failure too.

Preconditions and effects


Atom("holding", ("?obj",))          # (holding ?obj)
Atom("gripper-empty", negated=True) # (not (gripper-empty))
					

These are in PDDL's shape on purpose. Week 6 hands them to a planner without translating them.

can_start or a precondition?

can_start

What the skill can sense, now.

grasp: the finger is near its open limit.

Preconditions

What the planner knows about the world.

grasp: (gripper-at ?obj) and (gripper-empty).

The skill cannot see whether an object is between the jaws. The planner cannot see where the finger is. Each checks only what it can know.

Where they live


SkillSpec(
    name="go-to",
    parameters=("from", "to"),
    preconditions=(Atom("at", ("?from",)), Atom("connected", ("?from", "?to"))),
    effects=(Atom("at", ("?to",)), Atom("at", ("?from",), negated=True)),
    learned=True,
    implementation=None,
    summary="drive the base along one topological edge, handling obstacles the map does not know",
)
					

In the registry, as data. The class holds none of it.

Why they are not on the class

  • A planner reads what a skill needs and achieves without importing its module
  • So no policy weights are loaded, no torch is imported, and no robot is constructed
  • A skill can be described before it exists. go-to has implementation=None, and nobody has trained it yet

Resolving a skill by name


SkillSpec(name="grasp", ..., implementation="isfr.skills.grasp:Grasp")
registry.spec("grasp")   # preconditions and effects; nothing imported
registry.load("grasp")   # imports isfr.skills.grasp, returns the Grasp class
					

A plan step names a skill as a string. load() turns the string into a class, and only then is the module imported.

The catalogue: every skill the fetch needs

SkillPolicyWhat it does
go-tolearnedDrive the base along one edge of the map, around obstacles the map does not know
navigate-toprovidedFollow a route through the map, one go-to per edge
place-base-forlearnedMove the base to a pose from which the object is reachable
reach-tolearnedCollision-free arm motion to a goal configuration from IK
ik-solveprovidedClassical inverse kinematics
graspprovidedClose the gripper until the fingers stall on the object
releaseprovidedOpen the gripper until the fingers reach the open stop
hand-overprovidedOffer the held object to a person, then release it
detectprovidedGround object symbols in camera frames, with YOLO

Three learned, six provided. Learn only what classical methods do badly.

Part 2

Learning a policy

How do you get a policy you cannot write by hand?

The hinge

grasp.step() is a few lines you could write yourself.

Now write go-to's.

  • It knows the room only as an edge in a graph
  • Somebody moved a chair this morning
  • A full lidar scan arrives every tick

A rule you write handles one arrangement of furniture and breaks on the next. A room has more arrangements than you can list.

Two ways to drive past the chair

Classical Learned scan cost map planner tracker wheels scan + goal direction policy wheels

The classical stack rebuilds the cost map from every scan. It works, and week 3 measures against it. The learned policy has no map and no path in between.

What you have instead

  • A simulator you can reset, as often as you like
  • A number that says how well an attempt went
  • Time

That is enough. The rest of this part is how.

The running example: Pendulum-v1

θ θ = 0 is upright
  • Observation: \( (\cos\theta, \sin\theta, \dot\theta) \)
  • Action: one torque \( u \in [-2, 2] \)
  • An episode: 200 ticks of 0.05 s, 10 s in all, from a random angle

The motor can hold the pendulum only within 24° of upright. From anywhere lower, it has to swing.

The MDP

\[ (\mathcal{S}, \mathcal{A}, p, r, \gamma) \]

  • \( \mathcal{S} \): states. For the pendulum, \( (\cos\theta, \sin\theta, \dot\theta) \)
  • \( \mathcal{A} \): actions. For the pendulum, one torque
  • \( p(s' \mid s, a) \): the dynamics. Where the world goes next

The dynamics are never written down. The simulator has them.

Reward

\[ r = -\left( \theta^2 + 0.1\,\dot\theta^2 + 0.001\,u^2 \right) \]

The terms charge for the angle from upright, the spin, and the effort, the torque \( u \).

One number per tick. Hanging straight down costs about 10 a tick. Standing still at the top costs nothing.

The reward never says how to swing. It says only where the pendulum should end up.

Return

\[ G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1} \]

What the whole attempt was worth, seen from tick \( t \).

Reward is per tick. Return is per trajectory.

Discounting

\[ \gamma \in [0, 1) \]

  • Keeps the sum finite over an unbounded horizon
  • Says how far ahead the agent cares. \( \gamma = 0.99 \) looks about a hundred ticks out
  • Too low and the policy is short-sighted, too high and the return is noisy to estimate

Policies

\[ \pi_\theta(a \mid s) \]

  • Deterministic: one action per state
  • Stochastic: a distribution over actions, which is what lets it explore

\( \theta \) is a neural network's weights. Learning is adjusting \( \theta \), and nothing more.

The objective

\[ J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\!\left[ G_0 \right] \]

Maximise the return you expect, averaged over the trajectories your own policy produces.

Value functions

\[ V^\pi(s) \]

What this state is worth, if I keep behaving like this.

\[ Q^\pi(s, a) \]

What this action in this state is worth, same condition.

Both are predictions of return. No Bellman derivation here, because nothing in this course needs it.

Why this is not supervised learning

  • No labels. One scalar, and it arrives late
  • Credit assignment: which of two hundred ticks earned the reward?
  • The data distribution shifts, because your own policy generates it

From an objective to an algorithm

There are no right answers to collect and fit. So try, and learn from the score.

act with the policy score the attempt make what scored well more likely

Policy gradient, the intuition

Push up the log-probability of actions that led to high return. Push down the others.

\[ \nabla_\theta J(\theta) = \mathbb{E}\!\left[ \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, G_t \right] \]

Why the raw estimator does not work

  • \( G_t \) is a sum over a whole noisy trajectory
  • Every action in a good episode is rewarded, including the bad ones
  • The variance is high enough that updates fight each other

Baselines and the advantage

\[ A^\pi(s_t, a_t) = Q^\pi(s_t, a_t) - V^\pi(s_t) \]

Was this action better than what I usually do here?

Subtracting \( V \) removes the part of the return that was about the state rather than the action.

Actor and critic

  • The actor learns \( \pi_\theta \): what to do
  • The critic learns \( V \): what a state is worth
  • The critic turns a noisy return into an advantage, so the actor's updates stop fighting

PPO and SAC are both actor-critic methods.

PPO, step one: the ratio

\[ r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_\text{old}}(a_t \mid s_t)} \]

How much more likely the new policy makes an action than the policy that collected the data did.

At 1, nothing has changed. At 1.5, the new policy makes the action 50% more likely.

PPO, step two: the clip

\[ L(\theta) = \mathbb{E}\!\left[ \min\left( r_t(\theta) A_t,\; \mathrm{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon) A_t \right) \right] \]

Once an update has moved an action's probability by more than \( \epsilon \), 0.2 by default, the clip throws the extra improvement away.

Each update stays small, so training rarely collapses and the default settings work on tasks like these.

The clip, drawn

A > 0: a good action A < 0: a bad action r r 1 1 + ε 1 1 − ε

Left: once a good action is 20% more likely, pushing further earns nothing. Right: once a bad action is 20% less likely, the same.

The knobs that matter

KnobWhat it doesWrong value
n_stepsExperience gathered per updateToo small and the advantage estimate is noise
batch sizeSamples per gradient stepToo large and each epoch sees too few updates
learning rateHow far each step moves \( \theta \)Too high and it diverges, too low and nothing happens in the time you have
\( \gamma \)How far ahead it caresToo low and the policy never plans past the next tick
clip rangeHow much one update may change the policyToo wide and the stability PPO exists for is gone

The live demo: CartPole-v1

  • A pole balanced on a cart. The action is push left or push right
  • +1 for every tick the pole stays up
  • The episode fails if the pole tips past 12° or the cart leaves the track, and stops at 500

The demo trains it with PPO because PPO goes from nothing to a perfect score in under ten seconds. Your homework is the pendulum with SAC, because its action is a continuous torque.

All of that, in five lines


model = PPO("MlpPolicy", "CartPole-v1", seed=0)
model.learn(total_timesteps=25_000)
mean, std = evaluate_policy(model, env, n_eval_episodes=20, deterministic=False)
model.save("cartpole-policy")
reloaded = PPO.load("cartpole-policy")
					

This is examples/02-rl-fundamentals/main.py, which you run in the lab.

On-policy and off-policy

On-policy

Throws its data away after each update. PPO works this way.

Off-policy

Keeps a replay buffer and reuses every sample many times. SAC works this way.

The consequence is sample efficiency, and on a real robot samples are the expensive thing.

SAC

  • Off-policy, so it reuses experience
  • Built for continuous actions, which a torque and a wheel speed are
  • Reaches a working pendulum swing-up in 10 000 steps, about 50 seconds

Your micro-project uses it for those three reasons. PPO, tuned, needs 100 000 steps on the same task and still scores −217 ± 128.

Haarnoja, Zhou, Abbeel & Levine (2018), Proceedings of the 35th ICML, PMLR 80, 1861–1870.

What SAC adds: entropy

\[ J(\pi) = \sum_t \mathbb{E}\!\left[ r(s_t, a_t) + \alpha\, \mathcal{H}\big(\pi(\cdot \mid s_t)\big) \right] \]

  • The policy is paid for reward, and also for staying uncertain
  • So it keeps exploring after it has found something that works
  • \( \alpha \) sets the price. stable-baselines3 tunes it during training, with ent_coef="auto"

Part 3

What a result is worth

Up to here, every claim was about what an algorithm is supposed to do. From here on, every claim is a measurement.

Live: training something


PYTHONPATH=. python3 examples/02-rl-fundamentals/main.py
					

   steps   mean reward     std
       0          25.1    17.2
    5000          47.2    27.3
   10000         191.3    40.9
   25000         500.0     0.0

25000 steps in 7.7s
					

500 is the maximum. The episode truncates there, and it pays 1.0 per tick.

Reading the curve

  • The first row is an untrained network. It scores 25, close to the 19 that coin flips score
  • The climb is not smooth, and a dip between checkpoints is normal
  • The standard deviation matters as much as the mean

At 5 000 steps the policy averages 47 with a spread of 27, so it balances well sometimes and fails early often. At 25 000 every episode scores 500 and the spread is zero.

Four carts, one per checkpoint, from the same start. The pole falls at step 29 untrained, 51 after 5 000 steps and 221 after 10 000. After 25 000 it balances all 500 steps.

What one result is worth

Now the pendulum. One saved SAC policy, scored on fresh episodes.

Episodes per readingHow far the mean moves between readings
10± 26
20± 19
100± 8

Two 20-episode means closer than \( 1.96 \sqrt{2} \cdot 19 \approx 50 \) are a tie.

Most of the noise is the start angle. From hanging down the policy averages −264, from near upright −57.

Part 4

The obvious objection

This is a pendulum. Control engineers solved it before most of you were born. Why train anything?

Energy shaping, eight lines


def energy_shaping(obs):
    cos_theta, sin_theta, rate = obs
    theta = np.arctan2(sin_theta, cos_theta)
    if abs(theta) < 0.8:
        return float(np.clip(-8.0 * theta - 2.0 * rate, -2.0, 2.0))
    energy = 0.5 * rate**2 + 10.0 * cos_theta
    return float(np.clip(rate * (10.0 - energy), -2.0, 2.0))
					

No training, no network, no data. Pump energy in, then hold near the top.

Åström & Furuta (2000), Automatica 36(2), 287–295.

Side by side

PolicyReached uprightCost
SAC, 10 000 steps20/2049 s of training
Energy shaping, 8 lines20/20none
Random0/20none

Parity. Fifty seconds of training and a policy network bought nothing here. Reward cannot separate them either, and by slide 54 it could not have.

So why learn anything?

Because you knew \( m \), \( g \) and \( l \).

go-to in a furnished apartment has no such model, no closed form, and no eight lines. That is where learning earns its place, and it is what week 3 is about.

Part 5

Your micro-project

  1. Train: SAC on Pendulum-v1, 10 000 steps, about 50 s
  2. Wrap: SwingUpSkill, the contract around the trained policy
  3. Annotate: a SkillSpec with Atoms, registered as swing-up

exercises/02-skill-model/. No Webots, no world.

What you are aiming for

Eight pendulums. Top row, random torques: none swings up and holds, and all four fail at tick 200. Bottom row, the same four starts under the trained SAC policy: each swings up and holds, and succeeds between tick 45 and tick 61.

What --watch shows. Top: --run-untrained, random torques. Bottom: --run, SAC after 10 000 steps, from the same four starts.

The contract on a pendulum

MethodFor swing-up
can_startWhen may a swing-up start at all?
stepAsk the trained policy for a torque, and apply it
is_doneYours to design. The next slide
specWhat a swing-up needs, and what it achieves

step is the mechanical one. The questions in the other rows are yours to answer.

is_done is the design work

Gymnasium never terminates Pendulum-v1. It truncates at 200 ticks, whatever happened.

  • How far from vertical still counts as upright?
  • Must it be balanced, or may it pass through the top at speed?
  • How long must it stay upright before you call it success?
  • What is failure, when nothing fails on its own?

A trap worth knowing about

Pendulum-v1 starts at a random angle, so some starts are already upright.

If can_start rejects them, about one episode in sixteen fails with initiation set not satisfied. Your policy will look broken when it is not.

Decide what to do about it in reset(). There is more than one defensible answer.

Done when

  • --run reaches upright in at least 18 of 20 episodes
  • --run-untrained fails every episode, with a reason that names your tick budget
  • registry.load("swing-up") returns your class
  • You can say in one sentence what your threshold means physically

The README's Self Check has the full list. It gates on successes, never on reward.

Reading

  • Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., MIT Press. Chapters 3 and 13
  • Sutton, Precup & Singh (1999), Artificial Intelligence 112(1–2), 181–211
  • Schulman et al. (2017), Proximal Policy Optimization Algorithms, arXiv:1707.06347
  • Haarnoja et al. (2018), Soft Actor-Critic, PMLR 80, 1861–1870
  • Poole & Mackworth (2023), Artificial Intelligence, 3rd ed., Cambridge University Press. Section 13.6
  • Åström & Furuta (2000), Automatica 36(2), 287–295