Data structures and data shapes

Programs rarely work with just one number or string. A typical data analysis or dashboard works with a collection of records. Each record might describe a book, person, place or observation, and each record may contain several named values.

The most important skill is not memorising every method available for every Python data type. It is being able to recognise the shape of the data: what kind of structure it is, what is nested inside what, and which path leads to the value you need.

This section provides an initial survey of common Python data shapes and the minimum syntax needed to read structured records and return to the Part 0 change. Extensive practice with nested tables, dictionary schemas, reference handling and in-memory data pipelines will follow throughout Part 5.

Choosing a data structure

The following four structures cover most of the data you will encounter at this stage:

A list is usually the right choice when the order matters or you want to process every item. A dictionary is useful when values have names or when you want to find a value using a key. A set is useful when each value should occur only once. A tuple, introduced in the previous section, is a convenient way to keep or return a small, fixed number of values together when their positions are clear. As the number of fields grows, a dictionary with descriptive keys often becomes easier to read.

Dictionaries

A dictionary maps keys to values. This dictionary describes one work:

work = {
    "title": "Frankenstein",
    "year": 1818,
    "language": "English"
}

Values are accessed using their keys:

print(work["title"])
print(work["year"])

Keys are also used when adding or changing a value:

work["author"] = "Mary Shelley"
work["year"] = 1831

The in operator checks whether a dictionary contains a key:

if "author" in work:
    print(work["author"])

When a loop iterates over a dictionary directly, it goes through the keys:

for key in work:
    print(key, work[key])

The method items provides the key and value together:

for key, value in work.items():
    print(key, value)

Similarly, work.keys() provides all keys, and work.values() provides all values (useful for calculations such as sum(totals.values())).

Building a lookup dictionary

A dictionary can also act as an index into a larger collection. Suppose records arrive as a list, but the program repeatedly needs to find a particular work by its identifier. With only a list, finding a work means checking records one at a time:

works = [
    {"id": "hamlet", "title": "Hamlet", "year": 1603},
    {"id": "frankenstein", "title": "Frankenstein", "year": 1818},
    {"id": "odyssey", "title": "The Odyssey", "year": -700}
]

def find_in_list(works, wanted_id):
    for work in works:
        print("checking", work["id"])
        if work["id"] == wanted_id:
            return work

    return None

found = find_in_list(works, "odyssey")
print(found["title"])

The print statement makes the search visible. Finding the first record requires one pass through the loop body, while finding the last requires three. A missing identifier would require checking every record, and the function then returns None — Python's way of saying "no value", which lets the caller tell a missing result apart from a found one.

If identifiers are used repeatedly, we can first construct a dictionary which maps each identifier to its complete record:

works_by_id = {}

for work in works:
    works_by_id[work["id"]] = work

wanted_id = "odyssey"
print("looking up", wanted_id)

if wanted_id in works_by_id:
    print(works_by_id[wanted_id]["title"])

The construction loop visits every record once. After that, each search uses wanted_id in works_by_id and works_by_id[wanted_id] instead of running our list-search loop again. The dictionary is therefore useful when the program will perform many lookups by the same field.

Sets

A set contains each value at most once. Repeated values disappear when a list is converted into a set:

languages = ["English", "Finnish", "English", "Swedish", "Finnish"]
unique_languages = set(languages)
print(unique_languages)

The order in which a set is printed is not fixed. Sets are especially useful for membership checks:

selected_languages = {"Finnish", "Swedish"}

if "Finnish" in selected_languages:
    print("Finnish is selected")

If the result must be in a predictable order, it can be converted into a sorted list:

ordered_languages = sorted(unique_languages)

Nested data

Lists and dictionaries can contain other lists and dictionaries.

Tables as lists of lists

A small table can be represented as a list of rows, where each row is itself a list:

table = [
    ["Hamlet", 1603],
    ["Frankenstein", 1818],
    ["Beloved", 1987]
]

print(table[1])       # the second row
print(table[1][0])    # the first field in that row

for row in table:
    print(row[0], row[1])

The first index chooses a row and the second chooses a field within that row. Code which processes every cell uses a loop over rows containing another loop over the fields in each row.

Tables are compact, but the program must know what each column position means. If records have several fields, named dictionary keys are often clearer.

Lists of records

A list of dictionaries is a common way to represent records with named fields:

works = [
    {
        "title": "Hamlet",
        "year": 1603,
        "authors": ["William Shakespeare"],
        "metrics": {"characters": 132, "scenes": 20}
    },
    {
        "title": "Frankenstein",
        "year": 1818,
        "authors": ["Mary Shelley"],
        "metrics": {"characters": 18, "chapters": 24}
    }
]

Read a nested expression from left to right, one step at a time:

works[0]                         # the first record
works[0]["title"]                # the title in the first record
works[1]["authors"]              # the list of authors in the second record
works[1]["authors"][0]           # the first author in that list
works[0]["metrics"]["scenes"]   # a value in a nested dictionary

The same path can be split into intermediate variables when that is easier to follow:

first_work = works[0]
metrics = first_work["metrics"]
number_of_scenes = metrics["scenes"]

Three common transformations

Many analytical workflows repeatedly use the same three operations: filtering records, selecting fields, and aggregating values.

Filtering records

Filtering keeps only records which meet a condition:

recent_works = []

for work in works:
    if work["year"] >= 1800:
        recent_works.append(work)

Selecting a field

Selecting creates a collection containing only the values needed later:

titles = []

for work in works:
    titles.append(work["title"])

Aggregating by category

Aggregation combines several records into a summary. A dictionary is useful for keeping one running total for each category:

records = [
    {"language": "English", "count": 3},
    {"language": "Finnish", "count": 2},
    {"language": "English", "count": 4}
]

totals = {}

for record in records:
    language = record["language"]

    if language not in totals:
        totals[language] = 0

    totals[language] += record["count"]

The result is {"English": 7, "Finnish": 2}.

When the data shape changes

Code often has to be adapted because data from a file, API or earlier processing step changes shape. Suppose a program originally receives these records:

{"title": "Hamlet", "views": 1200}

A new version of the data nests the value inside a dictionary:

{"title": "Hamlet", "metrics": {"views": 1200}}

The filtering logic does not need to be redesigned. Only the path to the value changes from record["views"] to record["metrics"]["views"].

Sanity checks during exploration

Data analysts often use notebooks: interactive documents divided into cells, where code can be run a section at a time and its output appears alongside it. The exercises on this course use ordinary Python files, but the same checking practice is useful in both settings.

An output can look plausible even when an aggregation or data path is wrong. A few small checks are often enough to catch mistakes early. The Python command assert checks that a Boolean expression is true. If it is false, execution stops and points to the failed check.

records = [
    {"language": "English", "count": 3},
    {"language": "Finnish", "count": 2},
    {"language": "English", "count": 4}
]

result = total_by_language(records)

assert result["English"] == 7       # a known small example
assert set(result) == {"English", "Finnish"}  # the expected structure
assert sum(result.values()) == 9    # a total which should be preserved

These are not a full test suite. They are sanity checks: quick, explicit statements about results you already know should hold. When an AI writes or changes an analytical workflow, checks like these are a useful way to verify the logic independently of how convincing the final chart looks.

Return to the opening modification

At the opening of the course, you couldn't read this change because you had not yet learned to read its Python structures:

 active_records = filter_collection(records, selected_collection)
+visible_records = filter_minimum_year(active_records, minimum_year)

-metrics = make_metrics(active_records)
-table_rows = make_table(active_records)
+metrics = make_metrics(visible_records)
+table_rows = make_table(visible_records)
 chart_series = make_chart(active_records)

You now know enough to read the primary implementation evidence yourself:

1. Which variable refers to the records after both filters?
2. Which three function calls consume record lists?
3. Which call still receives the list before the minimum-year filter?
4. What is the smallest change which makes all three outputs consume the same records?

You can check your current points from the blue blob in the bottom-right corner of the page.