Artificial intelligence → Module 3 → Project: a recommendation system

Mini project: a recommendation system

YouTube, shops and music apps constantly suggest what to watch, buy or listen to next. How do they know what we would like? One of the basic ideas is: recommend to me what people with similar taste liked. In this project we build a system that recommends books from the reading list to students.

Data: book ratings

Six students rated books with grades from 1 to 5. If a book isn't in someone's list, that student hasn't read it. Such data is nicely stored in Python as a dictionary of dictionaries.

The idea

1. Similarity of two students: we look at the books both of them rated and calculate the average difference of their ratings. Difference 0 → similarity 1 (the same taste), difference 4 → similarity 0.
2. Recommendation: for each book the student hasn't read, we look at the ratings of similar students (similarity greater than 0.5). The expected rating is the average of their ratings, where a more similar student “counts more”.
3. We recommend the books with the highest expected rating.

The program

# A recommendation system: which books a student should read, based on the ratings of similar students
import math

# ratings from 1 to 5; a book that isn't in the dictionary hasn't been read by the student
ratings = {
    "Ana":    {"The Little Prince": 5, "Heidi": 4, "Pinocchio": 2, "Robinson Crusoe": 1},
    "Marko":  {"Robinson Crusoe": 5, "Eagles Fly Early": 5, "Pinocchio": 2, "Heidi": 1},
    "Jelena": {"The Little Prince": 5, "Heidi": 5, "Alice in Wonderland": 4, "Eagles Fly Early": 2},
    "Luka":   {"Eagles Fly Early": 4, "Robinson Crusoe": 4, "Alice in Wonderland": 2, "The Little Prince": 2},
    "Sara":   {"The Little Prince": 4, "Alice in Wonderland": 5, "Heidi": 4, "Pinocchio": 3},
    "Nikola": {"Robinson Crusoe": 5, "Pinocchio": 4, "Eagles Fly Early": 4},
}


def similarity(a, b):
    """How similar two students are (from 0 to 1), based only on the books both of them rated."""
    common = [k for k in ratings[a] if k in ratings[b]]
    if not common:
        return 0
    # average difference in ratings: 0 = they rate the same, 4 = completely different
    difference = sum(abs(ratings[a][k] - ratings[b][k]) for k in common) / len(common)
    return 1 - difference / 4


def recommend(student):
    """For each unread book, calculates the expected rating from the ratings of similar students."""
    total = {}          # book -> sum of (similarity · rating)
    weight = {}        # book -> sum of similarities
    for other in ratings:
        if other == student:
            continue
        s = similarity(student, other)
        if s <= 0.5:                       # we look only at students who are similar enough
            continue
        for book, rating in ratings[other].items():
            if book not in ratings[student]:
                total[book] = total.get(book, 0) + s * rating
                weight[book] = weight.get(book, 0) + s
    estimate = {k: total[k] / weight[k] for k in total}
    return sorted(estimate.items(), key=lambda p: -p[1])


for other in ratings:
    if other != "Ana":
        print(f"similarity Ana – {other}: {similarity('Ana', other):.2f}")

print("Recommendations for Ana:")
for book, rating in recommend("Ana"):
    print(f"  {book}: expected rating {rating:.1f}")

print("Recommendations for Marko:")
for book, rating in recommend("Marko"):
    print(f"  {book}: expected rating {rating:.1f}")

Program output:

similarity Ana – Marko: 0.42
similarity Ana – Jelena: 0.88
similarity Ana – Luka: 0.25
similarity Ana – Sara: 0.83
similarity Ana – Nikola: 0.25
Recommendations for Ana:
  Alice in Wonderland: expected rating 4.5
  Eagles Fly Early: expected rating 2.0
Recommendations for Marko:
  Alice in Wonderland: expected rating 2.0
  The Little Prince: expected rating 2.0

• Ana is most similar to Jelena and Sara (they love The Little Prince and Heidi). Both rated Alice in Wonderland highly – so that is the first recommendation for Ana.
• Marko loves adventures (Robinson Crusoe, Eagles Fly Early); the students similar to him didn't rate Alice and The Little Prince highly, so the expected ratings are low – the system actually doesn't recommend anything special to him.
• This is a simple version of a method called collaborative filtering. Real systems have millions of users and also use other data (how long you watched, what you skipped).

Things to think about

• A new user has no ratings, so we don't know whom they are similar to – this is called the “cold start” problem (task 3).
• The “bubble”: if you are always recommended only things similar to what you already like, you will rarely discover something completely new. Good systems deliberately add some different recommendations.
• Privacy: ratings and habits say a lot about a person. With real systems, users should know which data is used.

Common misconceptions

• “Recommendations are made by people who know you” – they are made by programs based on the behavior of many users.
• “If something is recommended to you, it is the best for you” – a recommendation is only an estimate based on similar users – and it can close you in a “bubble”.

Practice tasks

Try it yourself first, and only then open the solution.

1. Add yourself to the dictionary ratings with ratings of three books and print the recommendations for yourself.
2. Change the similarity limit from 0.5 to 0.3. What changes in the recommendations for Ana?
3. Write a program that recommends books to a new student (without ratings) based on the average rating of all students.

# Task: a recommendation for a new student who hasn't rated any book yet
# (we don't know whom they are similar to, so we recommend the books with the best average rating)
ratings = {
    "Ana":    {"The Little Prince": 5, "Heidi": 4, "Pinocchio": 2, "Robinson Crusoe": 1},
    "Marko":  {"Robinson Crusoe": 5, "Eagles Fly Early": 5, "Pinocchio": 2, "Heidi": 1},
    "Jelena": {"The Little Prince": 5, "Heidi": 5, "Alice in Wonderland": 4, "Eagles Fly Early": 2},
    "Luka":   {"Eagles Fly Early": 4, "Robinson Crusoe": 4, "Alice in Wonderland": 2, "The Little Prince": 2},
    "Sara":   {"The Little Prince": 4, "Alice in Wonderland": 5, "Heidi": 4, "Pinocchio": 3},
    "Nikola": {"Robinson Crusoe": 5, "Pinocchio": 4, "Eagles Fly Early": 4},
}

all_ratings = {}                                   # book -> list of all ratings
for student in ratings.values():
    for book, rating in student.items():
        all_ratings.setdefault(book, []).append(rating)

average = {k: sum(o) / len(o) for k, o in all_ratings.items()}
for book, p in sorted(average.items(), key=lambda x: -x[1]):
    print(f"{book:22} average {p:.1f} ({len(all_ratings[book])} ratings)")

The Little Prince      average 4.0 (4 ratings)
Robinson Crusoe        average 3.8 (4 ratings)
Eagles Fly Early       average 3.8 (4 ratings)
Alice in Wonderland    average 3.7 (3 ratings)
Heidi                  average 3.5 (4 ratings)
Pinocchio              average 2.8 (4 ratings)

Check your knowledge