LQR control explained

What a linear quadratic regulator is: price the error and the effort, and the gains fall out of the arithmetic instead of out of hand tuning. Drive the same turn with hand-tuned gains and with gains from a cost, chart both errors, and sweep the price of effort to see the response change. Every demo runs in the browser.

Guidefree, runs in your browser

LQR stands for linear quadratic regulator. It is a way of choosing the gains of a controller by writing down what you are paying for, rather than by turning knobs until the robot looks right. You say what one degree of error costs and what one percent of motor push costs, and a short piece of arithmetic hands you the gains that make the total bill as small as it can be. It is the standard method on drones, rockets, balancing robots and camera gimbals, and it is what people reach for when tuning a PID by hand stops being practical.

On this page a small robot turns 90 degrees on the spot, the same turn as in the PID guide. Each demo below is a real program you can change and run.

The bill

A controller is trying to do two things at once that pull against each other: get the error to zero quickly, and not use much effort doing it. LQR makes you price them:

bill = the sum, every step, of
           Q × (error in degrees)²
         + R × (push in percent)²

• Q is what one degree of error costs. Raise it and the controller is told the error matters more, so it pushes harder.
• R is what one percent of push costs. Raise it and the controller is told effort matters more, so it eases off.
• Both are squared, which is the quadratic in the name. Squaring means a big error costs much more than two small ones, and it is also what makes the answer come out as tidy arithmetic instead of a search.

Only the ratio of Q to R matters. Q = 1, R = 0.05 and Q = 20, R = 1 give exactly the same gains, so in practice you fix Q at 1 and choose R.

The model

LQR needs to know how the robot answers the motors. That is the linear in the name: the model has to be a straight-line one, written as "what the state is next step, from the state now and the push now". For this turn the state is two numbers: how far past the target the robot is, and how fast it is turning.

Measure the robot and you get both numbers it needs. Ask for 50 percent push and the turn rate settles at about 64 degrees per second, so one percent of push is worth 1.28 degrees per second. The rate gets about a third of the way to its new value in each 0.1 second step, because the drive lags.

next angle = angle + 0.1 × rate
next rate  = 0.67 × rate + 0.33 × 1.28 × push

Written as a 2 by 2 and a column, which is how the arithmetic below wants them:

A = [ 1    0.1  ]        B = [ 0        ]
    [ 0    0.67 ]            [ 0.4224   ]

A says what the robot would do next step if you did nothing, and B says what one percent of push adds. There is nothing else to them: the first row is "the angle changes by the rate times the time", the second is "the rate creeps towards what the push asks for".

Gains tuned by hand, and what they cost

Here is the turn with the gains the PID guide arrived at by hand, Kp = 4 and Kd = 0.8, written in the form LQR uses: push in proportion to the error, minus a brake in proportion to how fast the robot is already turning.

from bugbot import *
connect()

# change these two and press Run
K1 = 4.0         # push per degree of error
K2 = 0.8         # push taken off per degree per second of turn rate

H = 0.1          # seconds per step
TARGET = 90.0    # degrees to turn

last = heading()
errors = 0.0
pushes = 0.0
for tick in range(40):
    h = heading()
    error = (TARGET - h + 180) % 360 - 180
    rate = ((h - last + 180) % 360 - 180) / H
    last = h
    push = max(-100, min(100, K1 * error - K2 * rate))
    errors = errors + error * error
    pushes = pushes + push * push
    plot("error", error)
    plot("push", push)
    drive(0, 0, push)
    wait(H)
stop()
print("degrees of error, squared and added up:", round(errors))
print("percent of push, squared and added up:", round(pushes))
print("the bill at 0.05 per percent:", round(errors + 0.05 * pushes))
print("still to turn:", round(error, 1), "degrees")

That is a good controller, and it took someone several runs to find. The two numbers at the bottom are the point of this page: whatever you were doing while you tuned, you were trading those two bills against each other. Try K1 = 8: the error bill falls by 2 percent and the push bill goes up by nearly a third. Try K1 = 1: the push bill falls by 60 percent and the error bill nearly doubles.

The robot stops 1.8 degrees short because below about 15 percent the motors do not move it at all. Keep that in mind: nothing in the arithmetic below knows about it.

Where the gains come from

The gains come out of a loop that works backwards. Think of it as asking: if I am in some state with one step left, what is the cheapest I can get away with? Then: with two steps left? With three? Each answer is worked out from the one before, and after a few dozen steps the answer stops changing. That settled answer is the cost of being in a state and playing well from there, and the gains that go with it are the ones you want.

The cost of being in a state is itself a quadratic, so it is held in three numbers, p11, p12 and p22, exactly like the uncertainty in a Kalman filter:

cost from here = p11 × angle² + 2 × p12 × angle × rate + p22 × rate²

Written out by hand, with c = 0.67 and b = 0.4224 from the model, one turn of the loop is:

s  = R + b² × p22                     how much a push costs, all told
k1 = b × p12 / s                      the gain on the error
k2 = b × (p12 × 0.1 + p22 × c) / s    the gain on the rate
p11 <- Q + p11 - s × k1²
p12 <- p11 × 0.1 + p12 × c - s × k1 × k2
p22 <- p11 × 0.1² + 2 × p12 × 0.1 × c + p22 × c² - s × k2²

That is the Riccati equation for this system, with the matrices multiplied out by hand. The three lines at the bottom are "the cost of a state is what it costs you now, plus what the state you land in costs, minus what the best push saves you". All three use the old p11, p12 and p22, not the new ones, which is why the program sets all three in a single line. Run it a few hundred times and k1 and k2 stop moving.

The same turn, by hand and from the cost

Now both, one after the other, in one program. The robot turns 90 degrees with the hand-tuned pair, then another 90 with the pair that falls out of the cost. The chart shows both errors, one after the other.

from bugbot import *
connect()

# the model of the turn, measured from this robot
H = 0.1          # seconds per step
ALPHA = 0.33     # of the way to the asked-for turn rate in one step
G = 1.28         # degrees per second of turn rate, per percent of push

# change these two and press Run: what you are paying for
Q = 1.0          # per degree of error, squared
R = 0.05         # per percent of push, squared

# work backwards until the gains stop changing
c, b = 1 - ALPHA, ALPHA * G
p11, p12, p22 = Q, 0.0, 0.0
for i in range(200):
    s = R + b * b * p22
    k1 = b * p12 / s
    k2 = b * (p12 * H + p22 * c) / s
    p11, p12, p22 = (Q + p11 - s * k1 * k1,
                     p11 * H + p12 * c - s * k1 * k2,
                     p11 * H * H + 2 * p12 * H * c + p22 * c * c - s * k2 * k2)
print("from the cost: K1 =", round(k1, 2), " K2 =", round(k2, 2))


def one_turn(gain1, gain2, name):
    target = heading() + 90
    last = heading()
    errors, pushes = 0.0, 0.0
    for tick in range(40):
        h = heading()
        error = (target - h + 180) % 360 - 180
        rate = ((h - last + 180) % 360 - 180) / H
        last = h
        push = max(-100, min(100, gain1 * error - gain2 * rate))
        errors = errors + error * error
        pushes = pushes + push * push
        plot(name, error)
        drive(0, 0, push)
        wait(H)
    stop()
    wait(0.5)
    print(name, " error", round(Q * errors), " push", round(R * pushes),
          " bill", round(Q * errors + R * pushes), " left", round(error, 1))


one_turn(4.0, 0.8, "by hand")
one_turn(k1, k2, "from the cost")

The two lines on the chart sit almost on top of each other, one after the other in time. That is the result worth taking away: a cost of R = 0.05 per percent of push, chosen in one line, lands within a few percent of gains that took a person several tries to find. Nobody tuned anything. The model and the price list were enough.

The hand-tuned pair comes out slightly cheaper on the bill it is being judged by, 40,888 against 41,648, which is worth being honest about. The gains from the cost are the best possible for the model, and the model is a straight line. The real robot has a dead band, so the slightly gentler cost gains give up in the last few degrees and stop 4 degrees short instead of 1.8. On the part of the turn the model does describe, the two are the same controller.

What R does

Raise R and you are telling the controller that motor effort is expensive. This demo runs the turn five times: once with the hand-tuned pair, then four times with the gains that come out of four different effort prices. Each run prints its own bill, and what the hand-tuned pair would have cost under that same price.

from bugbot import *
connect()

# the model of the turn, measured from this robot
H = 0.1          # seconds per step
ALPHA = 0.33     # of the way to the asked-for turn rate in one step
G = 1.28         # degrees per second of turn rate, per percent of push

# change this list and press Run: what one percent of push costs
EFFORT = [0.02, 0.2, 2.0, 20.0]
Q = 1.0          # what one degree of error costs
HAND = (4.0, 0.8)


def gains_from(r):
    c, b = 1 - ALPHA, ALPHA * G
    p11, p12, p22 = Q, 0.0, 0.0
    for i in range(200):
        s = r + b * b * p22
        k1 = b * p12 / s
        k2 = b * (p12 * H + p22 * c) / s
        p11, p12, p22 = (Q + p11 - s * k1 * k1,
                         p11 * H + p12 * c - s * k1 * k2,
                         p11 * H * H + 2 * p12 * H * c + p22 * c * c - s * k2 * k2)
    return k1, k2


def one_turn(k1, k2, name):
    target = heading() + 90
    last = heading()
    errors, pushes, peak = 0.0, 0.0, 0.0
    for tick in range(40):
        h = heading()
        error = (target - h + 180) % 360 - 180
        rate = ((h - last + 180) % 360 - 180) / H
        last = h
        push = max(-100, min(100, k1 * error - k2 * rate))
        errors = errors + error * error
        pushes = pushes + push * push
        peak = max(peak, abs(push))
        plot(name, error)
        drive(0, 0, push)
        wait(H)
    stop()
    wait(0.5)
    return errors, pushes, peak, error


hand_errors, hand_pushes, hand_peak, hand_left = one_turn(HAND[0], HAND[1], "by hand")
print("by hand: K1", HAND[0], "K2", HAND[1], "biggest push", round(hand_peak),
      "left", round(hand_left, 1))
for r in EFFORT:
    k1, k2 = gains_from(r)
    errors, pushes, peak, left = one_turn(k1, k2, "R=" + str(r))
    print("R", r, "K1", round(k1, 2), "K2", round(k2, 2),
          "biggest push", round(peak), "left", round(left, 1),
          "bill", round(errors + r * pushes),
          "same bill by hand", round(hand_errors + r * hand_pushes))

Read the last two numbers on each line. They are the same run priced two ways: what the cost's own gains paid, and what the hand-tuned pair would have paid under that same price list.

When push is cheap the two are within two percent of each other, and at R = 0.2 the hand pair is actually the cheaper, because the model does not know about the dead band. When push is expensive the hand pair is hopeless: at R = 20 it spends three and a half times what it needs to, because it was tuned by someone who only cared about the error. That is the case for writing the price list down. Once "effort matters" is a number instead of a feeling, the gains change on their own.

Look at the last row on the chart as well. At R = 20 the gains are so gentle that the push falls into the dead band at about 67 degrees of error and the robot stops dead. The arithmetic thinks it is still creeping in. This is the same warning as before: the answer is the best one for the model, and the model is not the robot.

Where LQR goes wrong

• The model is the weak point. The gains are exactly right for A and B, and only as good as those are. Measure them on the machine you are going to run on, as Measuring your robot does. Getting G wrong by a factor of two here moves K1 from 3.68 to 3.23, so it is forgiving, but a model with the wrong shape is not.
• Nothing in it knows about limits. Dead bands, motors that top out at 100 percent, and anything that is not a straight line are all invisible to the arithmetic. Everything on this page that surprised us came from that.
• It needs the whole state. The controller multiplies the state, so you need every part of it, every step. Here the turn rate came from two headings and a subtraction. When the state is something you cannot measure, you put a Kalman filter in front of it, which is so common it has a name: LQG.
• There is no I term. LQR alone leaves the same steady state error a PD controller leaves, which is why the turns above stop a degree or two short. The usual fix is to add the running total of the error to the state and let LQR work out a gain for it too.
• The cost is still a choice. LQR does not tell you what you want, only what follows from what you said you want. Choosing R is the new tuning, but it is one number with a meaning, instead of three with none.

Where this is taught

• The feedback loop, Proportional control and The derivative term build the controller whose gains this page chooses.
• Tuning is the hand method LQR replaces, and Saturation and windup is the limit the arithmetic cannot see.
• Measuring your robot is where A and B come from.
• Feedforward and feedback is the other half of using a model in a controller.
• Fitting a model to data fits the numbers this page measures by hand.
• Two loops at once and Project: precision parking are the same job at GCSE level.
• The PID guide is the hand-tuned controller these gains are compared with.

Questions

What is LQR in simple terms?

It is a way of working out controller gains from a price list instead of by trial and error. You write down what an error costs and what effort costs, both squared, and the arithmetic gives you the gains that make the total smallest for your model of the machine. The controller that comes out is a set of numbers multiplied by the state, which for a turn looks exactly like a PD controller.

What do Q and R mean in LQR?

Q prices the state: how much one unit of error costs, per step. R prices the control: how much one unit of push costs. Both are squared in the sum. Only their ratio matters, so it is usual to fix Q and move R. A small R means effort is cheap, and gives high gains and a fast, hard-pushing response. A large R means effort is expensive, and gives low gains and a slow, gentle one.

How do you choose Q and R?

Start from units that mean something. A common rule is to make one unit of each cost the same as the largest amount of it you are happy with, so if 5 degrees of error and 20 percent of push are both acceptable, set Q = 1/5² and R = 1/20². Then move R up or down and watch the response, as the sweep demo on this page does. With several states, Q is usually diagonal, one price per state.

Is LQR better than PID?

For a system you can model and whose state you can measure, LQR gives you the gains directly and handles several states and several motors at once, which hand tuning cannot. PID needs no model, includes the integral term that removes steady state error, and is what nearly all industrial control is. They are not rivals: on a turn like this one, LQR is a way of choosing a PD controller's two gains.

What is the Riccati equation?

The equation that the cost of being in a state settles to when you work backwards from the end. In the form used here it is P = Q + A'PA - A'PB(R + B'PB)⁻¹B'PA, and the demos solve it by starting from P = Q and applying that line a few hundred times until P stops changing. The gains are then K = (R + B'PB)⁻¹B'PA, and the controller is push = -K × state.

What does linear quadratic regulator mean?

Linear because the model of the machine must be a straight-line one, next state = A × state + B × push. Quadratic because the bill squares the error and the push. Regulator because the job is to hold the state at zero, which is what "turn to face 90 degrees" becomes once the error is the state.

What is LQG?

LQR with a Kalman filter in front of it. LQR needs the whole state and the filter estimates it from noisy readings. The two are designed separately and still work together, which is a result called the separation principle, and it is one of the reasons this pair is used as much as it is.

Where is LQR used?

Anywhere there is a decent model and more than one thing to balance at once: aircraft and spacecraft attitude control, drone stabilisation, camera gimbals, balancing robots, and the low-level joint control of some robot arms. Walking robots use its relative, model predictive control, which solves a similar cost but over a short window and can be told about limits.

Learn it step by step

These lessons build the same ideas one at a time, each with tasks the simulator marks.

1. U5.1 The feedback loop Feedback control, University
2. U5.2 Proportional control Feedback control, University
3. U5.3 The derivative term Feedback control, University
4. U5.5 Tuning Feedback control, University
5. U5.6 Saturation and windup Feedback control, University
6. U1.6 Measuring your robot The robot as a system, University
7. U10.5 Feedforward and feedback Following a trajectory, University
8. U12.2 Fitting a model to data Learning, and the capstone, University
9. 3.5 Two loops at once Control, Robot club
10. 3.6 Project: precision parking Control, Robot club