The skill model, and learning basic skills

Intelligent Systems for Robotics · Week 2

Syllabus · Home

Last week named four layers and left the bottom one undefined. This week defines it, because every layer above talks to skills and to nothing else. The world model in week 4 records what they change, the planner in week 6 chains them, and the executor in week 8 watches them fail. None of that can be designed until a skill has a fixed shape.

The week asks two questions. What exactly is a skill, and how do you get a policy you cannot write by hand? The first has a short answer: three methods and a list of preconditions and effects. The second is about what goes inside those methods. The outside stays the same whether the policy was written or trained, which is what lets a learned skill sit where a hand-written one did.

Objectives

The contract

A skill is an option: an initiation set, a policy, and a termination. The formalism is from Sutton, Precup and Singh (1999), and this course uses it with one addition. The termination says whether the skill succeeded or failed, not merely that it stopped.

Why an option, and why not a function the planner calls? A function runs, returns, and the caller trusts the result. A robot skill takes seconds, the world moves while it runs, and it can end without having done its job. An option runs one tick at a time, is judged after every tick, and stops with a verdict. It also sets the planner's step size. Planning in motor commands, fetching a drink takes thousands of steps. Planning in options, it takes about a dozen. That was Sutton, Precup and Singh's reason for options, and it is this course's too.

class Skill(abc.ABC):
    name: str

    def can_start(self, state: dict) -> bool: ...
    def step(self, obs: dict) -> None: ...
    def is_done(self, state: dict) -> Termination: ...

can_start is the initiation set. step is the policy, driven one tick at a time, issuing commands and returning nothing. is_done is the termination, and it returns a Termination rather than a bool. A boolean could say only that the skill stopped. Success and failure are different facts about the world, and week 8 replans on one of them.

run_to_completion in isfr/skills/base.py drives the three. It resets the skill, checks the initiation set once, then senses, judges, steps and waits until a verdict comes back.

grasp shows all three outcomes. It closes the finger, waits until the finger stops moving, and reads where it stopped. This is Grasp._settled in isfr/skills/grasp.py:

def _settled(self, pos: float) -> Termination:
    if self._at(pos, self._closed):
        return Termination.failed("gripper closed on nothing")
    if self._at(pos, self._open):
        return Termination.failed("gripper did not close")
    return Termination.succeeded("holding the object")

Until the finger stops, is_done returns RUNNING, and a finger that never stops within the tick budget is a failure too. The live demo ended on the first branch. The is_done you write in the micro-project has the same shape. It decides what the state means and says why in the reason string.

Preconditions and effects live in the registry

A skill's preconditions and effects are not attributes of its class. They are fields of a SkillSpec, which is pure data held in the registry, and the class is named there by a "module:attr" string.

SkillSpec(
    name="go-to",
    parameters=("from", "to"),
    preconditions=(Atom("at", ("?from",)), Atom("connected", ("?from", "?to"))),
    effects=(Atom("at", ("?to",)), Atom("at", ("?from",), negated=True)),
    learned=True,
    implementation=None,
    summary="drive the base along one topological edge, handling obstacles the map does not know",
)

The separation buys two things. A planner can read what a skill needs and achieves without importing the skill's module, so no policy weights are loaded and no robot is constructed. And a skill can be described before it exists: go-to above has implementation=None, and nobody has written its policy yet.

A plan step names a skill as a string. load() turns the string into a class, and only then is the module imported:

registry = default_registry()
registry.spec("grasp")   # preconditions and effects; nothing imported
registry.load("grasp")   # imports isfr.skills.grasp, returns the Grasp class

The planner in week 6 does exactly this. The micro-project asks you to do it with your own skill.

What the skill checks, and what the planner checks

can_start and a skill's preconditions both say when the skill may start. They answer from different places. can_start reads what the skill can sense at that moment. For grasp, that is the finger's position, which must be near its open limit. Preconditions are symbolic facts the planner holds about the world. For grasp, they are (gripper-at ?obj) and (gripper-empty).

The skill cannot see whether an object sits between the jaws, and the planner cannot see where the finger is, so each checks only what it can know. Mixing them up produces one of two mistakes: a can_start that checks a world fact the robot cannot sense, or an Atom that restates a sensor threshold.

The skills the fetch needs

isfr/skills/catalogue.py registers nine skills:

SkillPolicyWhat it does
go-tolearnedDrive the base along one edge of the map, around obstacles the map does not know
navigate-toprovidedFollow a route through the map, one go-to per edge
place-base-forlearnedMove the base to a pose from which the object is reachable
reach-tolearnedCollision-free arm motion to a goal configuration from IK
ik-solveprovidedClassical inverse kinematics
graspprovidedClose the gripper until the fingers stall on the object
releaseprovidedOpen the gripper until the fingers reach the open stop
hand-overprovidedOffer the held object to a person, then release it
detectprovidedGround object symbols in camera frames, with YOLO

Three are learned and six are provided. The course learns only what classical methods do badly: driving around furniture the map does not know, placing the base, and reaching without collision. Inverse kinematics and closing a gripper have exact classical answers, and detection uses an off-the-shelf YOLO model. Today only grasp and release have an implementation, and week 3 trains the learned three. The predicate names may still change in weeks 4 to 6.

Reinforcement learning

grasp.step() is a few lines you could write yourself. The policy for go-to is not, and that is not a failure of effort.

Try to write it. go-to has to cross a room it knows only as an edge in a graph, past a chair somebody moved this morning, reading a full lidar scan every tick. A rule you write handles one arrangement of furniture and breaks on the next, and a room has more arrangements than you can list. The classical answer replaces the rules with a cost map, rebuilt from the scan every tick, a planner that searches it for a path, and a controller that tracks the path. That works, and week 3 measures against it. A learned policy reads the scan and the direction of the goal and outputs wheel speeds, with no map and no path in between.

What you have for learning one is a simulator you can reset, a number that says how well an attempt went, and time.

The running example

Every definition below is shown on Pendulum-v1, the task you train in the micro-project. A rod hangs from a motorised pivot. The observation is (cos θ, sin θ, ω), where θ is the angle from upright and ω = dθ/dt its angular velocity. The slides write ω as θ with a dot over it. The action is one torque between −2 and 2. An episode lasts 200 ticks of 0.05 s and starts at a random angle.

The motor can hold the rod only within 24° of upright. Gravity contributes up to 15 sin θ to the angular acceleration and the motor at most 6, so from anywhere lower the pendulum has to swing up.

The MDP

A task is a tuple of states, actions, transition dynamics p(s′ | s, a), a reward and a discount factor. For the pendulum, the state is the observation above and the action is the torque. The dynamics are never written down. The simulator has them, and no method here needs them in closed form. That is what separates this from the planning in week 6, which does use a model.

The reward is one number per tick. The pendulum pays

r = −(θ2 + 0.1 ω2 + 0.001 u2)

where u is the torque. Hanging straight down costs about 10 a tick, and standing still at the top costs nothing. The reward never says how to swing. It says only where the pendulum should end up.

The return is what a whole attempt was worth from tick t onwards, Gt = Σk γk Rt+k+1. The discount factor γ, between 0 and 1, keeps that sum finite and says how far ahead the agent cares. A value of 0.99 looks roughly a hundred ticks out.

A policy πθ(a | s) maps a state to an action or to a distribution over actions, and θ is a neural network's weights. This θ has nothing to do with the pendulum's angle. Training maximises J(θ), the return you expect over the trajectories your own policy produces. Value functions V(s) and Q(s, a) predict that return from a state, or from a state and an action. This course does not derive the Bellman equations. Sutton and Barto's chapter 3 does.

Three things make this harder than supervised learning. There are no labels, only a scalar that arrives late. Credit has to be assigned among the two hundred ticks of an episode. And the data distribution shifts while you train, because the policy being optimised is the thing generating the data.

Policy gradients and PPO

Those three difficulties rule out collecting the right answers and fitting them, because there are no right answers to collect. What you can do is try, score the attempt, and make what you just did a little more likely if it scored well. The policy gradient makes that precise. It pushes up the log-probability of actions that led to a high return and pushes down the rest.

In raw form the estimator is unusably noisy, because the return covers a whole trajectory and every action in a lucky episode gets rewarded. A baseline fixes most of it: the advantage A(s, a) = Q(s, a) − V(s) asks whether an action beat what the policy usually does in that state, which removes the part of the return that was about the state rather than the action.

The policy being trained is called the actor, and the network that learns V is the critic. The critic turns a noisy return into an advantage, so the actor's updates stop fighting each other. PPO and SAC are both actor-critic methods.

PPO collects a batch with the current policy, then takes several gradient steps on it. The ratio r = πθ(a | s) / πθ old(a | s) measures how much more likely the updated policy makes each recorded action than the policy that collected it. PPO clips that ratio to between 1 − ε and 1 + ε, with ε = 0.2 by default. Once an update has moved an action's probability by more than 20%, the extra improvement is thrown away.

Each update stays small, so training rarely collapses and the default settings work on tasks like CartPole. The lecture's live demo trains it from nothing to a perfect score in under ten seconds.

Why the micro-project uses SAC, not PPO

PPO is on-policy. It discards its experience after each update. SAC is off-policy: it keeps a replay buffer and reuses every sample many times, which makes it far more sample efficient. On a real robot, samples are the expensive thing, so that property is worth more than it looks on a laptop.

SAC is also built for continuous actions, and a pendulum torque is continuous. In practice this is what decides the micro-project. SAC reaches a working swing-up on Pendulum-v1 in 10 000 steps, about 50 seconds. PPO, tuned, needs 100 000 steps on the same task and still scores −217 ± 128, which is too unreliable to wrap in a skill whose Self Check asks for 18 successes out of 20.

SAC adds one idea to actor-critic. Its objective pays the policy for reward and also for entropy, which measures how spread out its action distribution is. A policy paid to stay uncertain keeps exploring after it has found something that works. A coefficient α sets the price of that uncertainty, and stable-baselines3 tunes it during training by default.

The lecture builds up to PPO because the policy-gradient story is the one worth teaching from scratch. The homework uses SAC because it works on this task. Both facts are worth holding at once.

What one result is worth

Up to here, every claim has been about what an algorithm is supposed to do. From here on, every claim is a measurement, and a measurement is worth as much as its spread. Watch the standard deviation as closely as the mean.

One SAC policy, trained for 10 000 steps and then frozen, was scored on 2 000 fresh episodes of Pendulum-v1. Grouping those episodes into readings shows how far the mean of one reading moves from the next:

Episodes per readingStandard deviation of the mean
1026
2019
1008

A single episode varies by 87 around the mean of −146, and the mean of n episodes varies by 87 divided by the square root of n. Four times the episodes halves the spread. Two means of 20 episodes each differ by chance by up to 1.96 × √2 × 19, about 50. The √2 is there because both means carry the noise, and 1.96 makes it a 95% bound. A claim that one policy beats another by less than that is a statement about the episodes drawn. You will want to make such a claim in week 3 and in the project.

Most of that noise is the start angle. Starts within 45° of hanging down average −264, and starts within 45° of upright average −57. The policy is no worse on those episodes. A swing-up from the bottom simply costs more reward than a small correction near the top.

This is why the micro-project's Self Check gates on a success rate, 18 of 20 episodes reaching upright, and never on a reward number. Reaching upright does not depend on where the pendulum started.

The classical baseline

This is a pendulum, and control engineers solved it before most of you were born. Training a network on it looks like a wasted afternoon, and the objection is fair. It deserves a measurement, so here is the classical controller, run on the same task and scored the same way.

Energy shaping solves the same task with no training, no network and no data. It pumps energy into the swing until it matches the energy of standing upright, then switches to a PD hold near the top. The method is from Åström and Furuta (2000), and this is the whole controller:

def energy_shaping(obs):
    cos_theta, sin_theta, rate = obs
    theta = np.arctan2(sin_theta, cos_theta)
    if abs(theta) < 0.8:
        return float(np.clip(-8.0 * theta - 2.0 * rate, -2.0, 2.0))
    energy = 0.5 * rate**2 + 10.0 * cos_theta
    return float(np.clip(rate * (10.0 - energy), -2.0, 2.0))

Run the comparison yourself. The solution's --baseline flag scores this controller, and --run scores the trained policy.

PolicyReached uprightCost
SAC, 10 000 steps20/2049 s of training
Energy shaping, 8 lines20/20none
Random0/20none

That is parity, and it is the honest result. Fifty seconds of training and a policy network bought nothing here. Reward cannot separate the two either: energy shaping read −190.3 ± 89.2 and −151.6 ± 81.7 over 20 episodes, which sits inside the noise measured above.

So why learn anything? Because every constant in those eight lines is a statement about a model you happen to possess: the pendulum's mass, its length, and gravity. A furnished apartment comes with no such model, no closed form and no eight lines. That is where learning earns its place, and it is what week 3 is about.

Slides

Deck for this week

Examples

Two run alongside this lecture. Run both from the repository root:

PYTHONPATH=. python3 examples/02-rl-fundamentals/main.py
PYTHONPATH=. python3 examples/02-grasp-skill/main.py

Run them exactly that way. python3 examples/02-rl-fundamentals/main.py puts the script's own directory on sys.path rather than the current one, so the bare form fails with ModuleNotFoundError: No module named 'isfr' no matter which directory you start from.

examples/02-rl-fundamentals/ is the script the lecture trains live. It needs no simulator. examples/02-grasp-skill/ runs the skill contract on TIAGo and needs Webots open on isfr_devices.wbt.

Micro-project

This week's exercise is exercises/02-skill-model/. Its README.md states the task and ends in a Self Check list you can verify unaided. A solution/ sits alongside the starter code. Do not open it until you are done.

Follow the lab in the session or at home. Run both examples first, then start the micro-project: train SAC on Pendulum-v1, wrap it in a Skill, and run the Self Check. Training takes under a minute, so most of your time goes on the wrapper. The wrapper is the part that carries into every later week.

The micro-project runs from the inputs it provides. It does not need any earlier week's output.

Reading

Further reading carries the background this course does not teach, including a survey of reinforcement learning on real robots.


Intelligent Systems for Robotics · course materials