Artificial intelligence → Module 3 → Rules or learning

Rules or learning: the same task in two ways

To really see the difference between an ordinary program and machine learning, we will solve the same task twice: once with a rule we come up with ourselves, and once so that the program learns the rule by itself from examples. The task: based on its mass, say whether a fruit is an apple or an orange.

The idea: a threshold

Oranges are mostly heavier than apples. So the rule can be: if the fruit is heavier than some threshold – it's an orange, otherwise it's an apple. The only question is: how big a threshold?

• A person will say it “by eye”, for example 170 g.
• The program will try all thresholds from 100 to 300 g and choose the one with the fewest errors on the training examples. This is the simplest form of machine learning.

The program

# The same task in two ways: apple or orange – by the mass of the fruit
# 1) a rule a person comes up with, 2) a rule (threshold) the program “learns” by itself from examples

# training examples: (mass in grams, correct label)
train = [(152, "apple"), (168, "apple"), (140, "apple"), (185, "apple"), (175, "apple"),
          (160, "apple"), (195, "apple"), (210, "orange"), (230, "orange"),
          (195, "orange"), (245, "orange"), (220, "orange"), (205, "orange")]
# test examples – the program doesn't see them while learning
test = [(148, "apple"), (172, "apple"), (158, "apple"), (238, "orange"),
           (198, "orange"), (215, "orange"), (250, "orange")]


def guess(mass, threshold):
    """Rule: heavier than the threshold is an orange, otherwise an apple."""
    return "orange" if mass > threshold else "apple"


def count_errors(examples, threshold):
    errors = 0
    for mass, correct in examples:
        if guess(mass, threshold) != correct:
            errors += 1
    return errors


# 1) the rule “by eye”: a person thinks everything over 170 g is an orange
person_threshold = 170
print("The person's rule (threshold 170 g): errors in training =", count_errors(train, person_threshold))

# 2) learning: we try all thresholds from 100 to 300 g and remember those with the fewest errors
fewest = min(count_errors(train, threshold) for threshold in range(100, 301))
good = [threshold for threshold in range(100, 301) if count_errors(train, threshold) == fewest]
# of all equally good thresholds we choose the one in the middle – the farthest from the examples of both types
best_threshold = good[len(good) // 2]
print("Equally good thresholds:", good[0], "to", good[-1], "g")
print("Learned threshold:", best_threshold, "g, errors in training =", fewest)

# testing on new examples
for name, threshold in [("person", person_threshold), ("learned", best_threshold)]:
    e = count_errors(test, threshold)
    accuracy = (len(test) - e) / len(test) * 100
    print(f"Test ({name}): {len(test) - e} of {len(test)} correct, accuracy {accuracy:.0f} %")

Program output:

The person's rule (threshold 170 g): errors in training = 3
Equally good thresholds: 185 to 204 g
Learned threshold: 195 g, errors in training = 1
Test (person): 6 of 7 correct, accuracy 86 %
Test (learned): 7 of 7 correct, accuracy 100 %

What we saw

• The rule “by eye” (170 g) makes 3 mistakes already on the training examples.
• The program found that all thresholds from 185 to 204 g are equally good (only 1 error). It chose the one in the middle (195 g) – the farthest from the examples of both types, so there is the smallest chance that a new fruit falls “on the wrong side”.
• No threshold can avoid one error: the data contains both an apple and an orange of exactly 195 g. By mass alone it is impossible to tell them apart – we will need another feature (length) in the next lesson.
• On the test (examples the program hasn't seen), the learned threshold got all 7 right, and the person's rule 6.

What is “learning” here?

• The model is the rule “heavier than the threshold → orange”, and the model's parameter is the threshold.
• Learning is searching for the parameter value that gives the fewest errors on the examples.
• Real models have thousands or billions of parameters, so you can't “try them all” – that's why there are smarter ways of learning (we will see them with the perceptron). But the idea is the same.

Common misconceptions

• “Machine learning is something magical” – here it is just a loop that tries thresholds and counts errors.
• “A learned rule is always perfect” – if the examples overlap (two fruits of 195 g), no rule can be without errors.

Practice tasks

Try it yourself first, and only then open the solution.

1. Change the person's threshold from 170 to 190 g. How many errors are there now in training and in the test?
2. Write a program that learns a threshold based on the length of the fruit to tell bananas and apples apart.
3. Why can't any threshold based on mass be without errors for the apples and oranges from this table?

# Task: the program learns a threshold based on the LENGTH of the fruit by itself – banana or apple
examples = [(7.5, "apple"), (8.0, "apple"), (7.2, "apple"), (8.6, "apple"), (8.9, "apple"),
           (19.0, "banana"), (20.5, "banana"), (17.5, "banana"), (18.2, "banana"), (17.0, "banana")]


def count_errors(threshold):
    return sum(1 for d, v in examples if ("banana" if d > threshold else "apple") != v)


thresholds = [p / 10 for p in range(50, 250)]           # from 5.0 to 24.9 cm
fewest = min(count_errors(p) for p in thresholds)
good = [p for p in thresholds if count_errors(p) == fewest]
print("Errors:", fewest)
print("Good thresholds from", good[0], "to", good[-1], "cm; we choose the middle:", good[len(good) // 2], "cm")

Errors: 0
Good thresholds from 8.9 to 16.9 cm; we choose the middle: 12.9 cm

Check your knowledge