Artificial intelligence → Module 4 → micro:bit recognizes gestures

micro:bit recognizes gestures

The micro:bit has a built-in accelerometer that measures acceleration along three axes. In this lesson the micro:bit learns by itself to tell gestures apart – still, waving and shaking. Everything happens on the board itself: learning, remembering examples and recognizing. We use the nearest neighbor algorithm from module 3, written in MicroPython.

What we need

• a micro:bit (V1 or V2) and a USB cable; optionally batteries, so that the micro:bit is easier to move;
• the editor python.microbit.org (it works in the browser) or Thonny/Mu;
• MicroPython basics: introduction (SR), buttons (SR) and sensors (SR).

The idea: features of a gesture

One accelerometer measurement (x, y, z) doesn't say much – what matters is how the acceleration changes during the gesture. That is why the micro:bit measures 25 times in half a second and calculates the spread (standard deviation) for each axis – how much the values differed from the average:

• still: the values hardly change – all three numbers are small;
• waving: a big change along one axis (left–right), small changes along the others;
• shaking: big changes along all axes.

The three spread numbers are the features of the gesture, and the class number (0, 1, 2) is the label.

The program

# micro:bit learns gestures: the nearest neighbor algorithm (k-NN) in MicroPython
# Classes: 0 = still, 1 = waving, 2 = shaking
# LEARNING:  B – next class,  A – record an example of the current class (move the micro:bit for 0.5 s)
#          A and B together – end of learning, recognition begins
from microbit import *
import math

NUM_CLASSES = 3
examples = []              # a list of pairs (features, class)
label = 0
learning = True


def spread(values):
    """How much the values differ from the average (standard deviation)."""
    s = sum(values) / len(values)
    return math.sqrt(sum((v - s) ** 2 for v in values) / len(values))


def features():
    """Measures for 0.5 s and returns three numbers: how much the acceleration changed along the axes x, y and z."""
    xs, ys, zs = [], [], []
    for i in range(25):
        x, y, z = accelerometer.get_values()
        xs.append(x)
        ys.append(y)
        zs.append(z)
        sleep(20)
    return [spread(xs), spread(ys), spread(zs)]


def knn(o, k=3):
    """The class that the majority of the k nearest recorded examples have."""
    ordered = sorted(examples, key=lambda p: sum((a - b) ** 2 for a, b in zip(o, p[0])))
    votes = [0] * NUM_CLASSES
    for _, c in ordered[:k]:
        votes[c] += 1
    return votes.index(max(votes))


display.show(label)
while True:
    if learning:
        if button_a.is_pressed() and button_b.is_pressed():
            learning = False
            button_a.was_pressed()          # we clear the remembered presses
            button_b.was_pressed()
            display.scroll("GO")
        elif button_b.was_pressed():
            label = (label + 1) % NUM_CLASSES
            display.show(label)
        elif button_a.was_pressed():
            display.show(Image.DIAMOND_SMALL)   # a sign that we are recording
            examples.append((features(), label))
            display.show(label)
    else:
        if len(examples) < 3:
            display.scroll("TOO FEW EXAMPLES")
            learning = True
            display.show(label)
        else:
            display.show(knn(features()))
    sleep(50)

How to use it

1. The screen shows 0 (still). Hold the micro:bit still and press A – a small diamond appears while recording (0.5 s). Record 4–5 examples.
2. Press B – the screen shows 1 (waving). Press A and immediately wave left and right. Repeat 4–5 times.
3. Press B – 2 (shaking). Record 4–5 examples of shaking.
4. Press A and B together – it shows “GO” and the micro:bit starts recognizing: the screen shows the number of the gesture you are making.

We checked the program with a simulation on a computer (with made-up accelerometer values): after 12 recorded examples it recognized all three gestures. On a real micro:bit, try how many examples you need.

What happens “under the hood”

• The list examples is the whole “model”: k-NN only remembers examples. When the micro:bit is turned off, the examples are deleted – every start is new learning.
• Every half second the micro:bit measures a new gesture, calculates three numbers and compares them with all the recorded examples.
• The micro:bit V1 has only 16 KB of working memory, but for a few dozen examples of three numbers each, that is quite enough.

Common misconceptions

• “The micro:bit is too small for artificial intelligence” – for simple tasks (gestures, sound, tilt) small models also work on very small chips.
• “The model recognizes any gesture” – it recognizes only the gestures someone taught it; it puts everything else into the most similar class.

Practice tasks

Try it yourself first, and only then open the solution.

1. Add a fourth gesture – “jump”. What needs to be changed in the program?
2. Instead of numbers, show pictures: still – Image.ASLEEP, waving – Image.HAPPY, shaking – Image.SURPRISED.
3. Why do we use the spread of the values and not the values x, y and z themselves?

Check your knowledge