Creative Python Programming with Strype Volume 3: Structuring data

Books in this series build on the knowledge from the previous volume, developing deeper skills and understanding as you work along. Each book will take you through a carefully thought-out path of activities and exercises to develop your programming knowledge. If you are entirely new to programming, you should start with Volume 1. If, however, you have some prior experience, you may well choose to jump in with a later volume. The summary below gives you an idea of the level of understanding expected for each book.

The first volume starts at the very beginning. It does not assume that you know anything about Python, Strype, or indeed programming at all. It takes you from a fresh start to writing quite sophisticated programs.

The emphasis in this book is on creativity: We use a variety of graphical examples, such as an interactive game, to go through various open-ended projects that encourage you to bring in your own ideas and invent your own goals.

Through a series of carefully constructed activities, you learn many general and fundamental programming concepts and constructs: functions, variables, assignment, conditional statements and loops, data types, random behaviour and more. At the end of this book, you have worked with graphics and sound, and understand program structure, modularisation and good programming practice.

For the second volume, we assume that you are somewhat comfortable with the constructs listed above in the description of the first volume, and build up your skills from there. One important aspect that is central to the material in this book is one of the most powerful aspects of computers: the ability to work with large amounts of data.

In studying this volume, you will learn to work with data sets to process and analyse them. These can be text (such as books), or sets of numbers (such as temperature data or sound samples). You will learn a lot about using lists in Python, which is one of the central constructs to work with data.

Towards the end of this volume, we also look at graphics and sound again, and we see how the representation of data can help us to build more interesting games.

The third volume – the one you are looking at right now – builds on the data processing of volume 2 and progresses to even more interesting projects. We learn how to use a greater variety of data structures, such as tuples and dictionaries in addition to lists, and how to combine these to model more interesting data. We do not only analyse data sources, but we also take a look at generating text, in a way a text prediction algorithm might do. In the process, we will gain a basic understanding of the inner workings of large language models (LLMs) when we write a simple version of an algorithm that is at the core of these systems.

The most important thing to realise as you work through this book to improve your programming skills – or to find out what programming can do for you – is to play along. You should concentrate on the practical exercises, get your hands on the keyboard, and write some code. You might first like to follow closely the activities we present to you, but we very much encourage you, at each step, to think about additional extensions that you may find interesting. Add your own ideas, try things out, experiment, play.

You will only become a good programmer by doing lots of programming. No amount of reading about programming will turn you into a confident programmer. Only doing it will make a difference.

This book will give you lots of information and ideas about programming projects; it is down to you to put it into practice.

We make use of the Strype environment (available at strype.org) – an online, frame-based Python development environment – for all practical exercises in this book. Strype is very tightly integrated into this text, and some projects are difficult to achieve without it.

For this volume, we assume you are familiar with editing in the Strype editor. If you are not, we recommend that you read Appendix A and B of the first volume before starting your work.

We wish to thank Pierre Weill-Tessier for his work on Strype. Pierre is a core member of the Strype development team – jointly with the authors – and has contributed key elements to the implementation of Strype. We also wish to thank Pete Dring, who has read and provided feedback on an early draft of this book. His comments helped us greatly to improve the content and structure of this work.

As mentioned above, our plan to do a thematic analysis rests on the idea of finding out which words are used unusually often in a given text.

This does not mean that a word is used more often than another word in a given text – that is entirely expected. Some words just occur more frequently than others: words like "the" or "go" are very frequent. To give us some useful information, we need to find out which words are used more often than we would expect them in an average English language text. For example, if the word "crime" is used more often than expected in a book, that tells us it is probably a detective novel. In that way, we can find out what the document is likely talking about.

This means we can achieve our content analysis in these three steps:

We will look at these three steps one at a time. We start at the beginning: counting how often each word is used in a text. This is also known as the frequency of the word.

To start this task, we have prepared a starter project named thematic-analysis-v1, which you can find in the book projects. This project includes only two helper functions which we will use later, but no code to help us with the initial task.

Whenever we aim to implement a solution for a sub-task, it is useful to use functions in our programs to provide the sub-task functionality. So this time, we will start by writing a function to count the frequency of a given word in a text.

If you have reached this stage in the project, you will have seen that we can now count the frequency of a given word in a text. You may have noticed, however, that some problems remain that can result in incorrect counting. If not, you should be able to see the problem with the following exercises.

Why does the test work correctly for the word "say", but not for the word "go"? You have probably worked out that this is because of the uppercase character in one of the "Go" occurrences in the text.

For our purposes, we do not care about the upper- or lowercase nature of the words – we would like to count them in any case. We can easily achieve this by first converting our entire text to lowercase before starting our counting:

The lower() method is a built-in method of the string class that returns a new string with all characters converted to lowercase. Note that it does not modify the original string. Thus, we have to assign the result back to our string if we want to keep it.

With these exercises, you will have discovered two things: First, there are many useful string methods available which can save us a lot of work of writing code ourselves. Knowing what methods are available is important to becoming a good programmer. You do not need to memorise all methods, but you should have a link to the documentation page readily available when you work, so that you can look them up whenever you need them.

And secondly, we can see that a count method is available that is somewhat similar to the count_word function we are currently writing. The reason we are writing our own is two-fold: The built-in count method does not find words, but substrings. Thus, "car", for example, would also be found if the actual word is "scarlet". Also important is our goal to modify and extend this function to count not one word, but all words (and there is no ready-made method for this). So our counting function is a stepping stone for more powerful functionality.

Again, you may have noticed that your function does not count correctly. This time, it is the punctuation characters that are throwing a spanner in the works. When we split the text, we separate the original string at space characters, but all other characters are left in the words. Thus, our list of words contains the word "say," (including the comma), instead of the word "say".

To fix this, we will need to remove the punctuation characters from each word.

What we have done in these exercises is to "clean the data". Cleaning the data for processing means to get all data into a known standard format so that it is properly recognised by our program. (In other data sets, it might also mean to remove invalid or duplicate data, but this is not a problem in our case.)

Our program deals with text, so the data points are words. But the words can appear in different formats and contexts. Consider, for example, the following strings, which may all be produced as a "word" by the string split() function:

Without cleaning, these would all be treated as different words, because they are different strings. We want, however, to recognise all of them as the word "hello", so we "clean" the string to produce the canonical (standard) representation. In our case, we do this by converting to lower case and removing punctuation, so that we are left with the string "hello" in each case. Then we are ready to count the words.

Counting words in single sentences is not very interesting – we could do that by hand. But counting words in long documents is not so easily done; this is where a computer can help. Let us try that out.

Strype includes various text files in its built-in file system. You can see the file system, and the files available within it, by using the "Files" tab in the bottom right of the Strype interface (Figure 1.1).

By browsing the built-in files, you can see that there is, for example, a file named "winnie-the-pooh.txt". This is the full text of A. A. Milne’s children’s book "Winnie The Pooh". In this book is a character called "Piglet". Let us find out how often Piglet is mentioned in the book.

Your project includes a read_file function that can be used to read in the contents of a text file. When you call this function, and you place the text cursor inside a string parameter (between the quotes), you can use auto-completion (Ctrl-Space) to pop up a list and select from all the available files (Figure 1.2).

You can now use this read_file function to read in the text of the Winnie The Pooh book, and then pass the text to your count_word function to count the number of times "piglet" is mentioned in the text.

A project with the functionality discussed so far is available in the book projects as thematic-analysis-v2. If you are unsure about any of your own code, you can compare your project to this one.

Let us pause for a moment and take stock of what we have achieved so far. We can now count the number of occurrences of a given word in any given text. We do this ignoring the case of the letters, and we can ignore punctuation characters. That is a good start. But a major problem remains: We would like to count all the words in a text, not only one given word. We know from previous chapters that we can hold a count in a variable, but the challenge here is that we need a separate count for each different word in the text.

It is not feasible to create a separate variable for each word. There may be thousands of different words, and we do not know in advance how many variables we would need. A list is also not directly suitable because we need to hold two pieces of information, the word and its count, together.

The solution to this problem lies in a new data structure: a dictionary. In the next chapter, we discuss what a dictionary is, how we use it, and how it can help us to solve this problem.

A dictionary (or dict, as Python calls this type internally) is a data structure that stores key/value pairs.

Every entry in the dictionary associates a key with a value, and we can look up the value by providing the key. An example is a list of phone numbers for your friends (Figure 2.1).

We see that we can think of a dictionary as a table with two columns. The first column holds the keys (here: the names), and the second column holds the values (the phone numbers).

We can then look up the value by providing the key. Let us look at an example how this works. Assume we have the dictionary shown in Figure 2.1 stored in a variable called phonebook. We can then write

This will print Adam’s phone number. We use the key like an index in the dictionary, and we receive the associated value. This makes looking up a particular entry in the dictionary very easy.

We can also add more entries to our dictionary. For example

will add a new entry in the dictionary (a new row in the table) with the phone number for Zola. Again, we see that we use the intended key like an index in square brackets, and we just assign the value to create the entry.

Dictionaries are useful in many situations, and we are familiar with the concept from our daily lives. One important aspect is that they do not only work with strings. Both the key and the value may be any type we choose.

For example, your contacts app on your phone holds a dictionary that uses the name of a person as a key and all their details — email address, phone number, home address — as the value. (Here we can see that the value can be more than a single string: it can itself be a complex data type with several types of information). You use it by looking up the name, and it shows you the associated details (the value).

Other examples are a password file on a computer (associating usernames with passwords) or a shopping basket on a website that associates a product name with a count of the number in the basket. Perhaps the most familiar example is an English language dictionary (from which the Python data structure gets its name): The key here is a word, and the value is its definition. We can find a word in the dictionary to look up its definition.

The last thing we need to know is how to create a dictionary. Python dictionaries are written with curly brackets: { }. Writing the curly brackets on their own creates an empty dictionary:

We can also create a dictionary pre-filled with some data using a colon to separate keys and values, and a comma to separate the entries:

Before we go on to use a dictionary in our word counting function, let us do some simple exercises to become more familiar with dictionaries.

Let us now get back to the word counting problem in our thematic-analysis project from the previous chapter. We were at the point where we could count the frequency of a single word, and we need to figure out how to count the frequencies of all words. As we have stated above, a dictionary can help with this.

The idea is that we use a dictionary – let’s call it counts – with an entry for each word. We can use the word itself as a key, and use the value for the count (the number of times we have found the word).

For example, a word count of the words in the sentence

would result in the dictionary shown in Figure 2.2.

In other words: we will create a dictionary with one entry for every unique word in our text, where the value holds the number of times each word was used.

It is useful to remind ourselves that this dictionary – if we do a word count for a whole book – will hold thousands of entries. This, however, is not a problem: computers are good at dealing with large amounts of data.

The algorithm we will use can be informally described as follows:

You can modify your existing program to implement the algorithm described above by working step by step through the following exercises.

A version of the project that implements these changes is available for you to study in the project folder as thematic-analysis-v3.

We have seen that printing out the dictionary using a simple print call works, but the output is not very readable. The list of word counts is very long, and the formatting is not very readable. Let us improve this by implementing a custom printing function for our word count table.

A first attempt at printing out the entries in our dictionary might involve printing each entry on a separate line:

With the above exercise, we can see that this version prints out the words in the dictionary, but not the counts. In other words: it prints the keys, but not the values of the dictionary.

How does this happen? Let us step back and take another look at the methods from the dictionary class that allow us to access its entries. There are three methods that give us lists of the dictionary contents:

The first two of these show us that we can get a list containing just the keys or the values. This means that we could print out either the keys or the values, line by line, using these two loops:

Try this out. What we observed with the last exercise above is the following: If we just iterate over dict, without stating whether we want to get the keys or the values, we will get the keys by default.

Let us look at the third method: items(). This method states that it returns a tuple with the key and value. A tuple is another of Python’s collection types. It can hold multiple arbitrary data items to group them together. We will discuss tuples in more detail in the next chapter; for now it is sufficient to know that it consists of multiple data items and is written using round brackets:

When we think about our dictionary, we can view a dictionary as a list of tuples. The first two entries of the dictionary shown in Figure 2.2, for example, can be regarded as two tuples:

Let us now try to use the items() method in our loop:

Try this out. You will see that this loop prints each entry of the dictionary as a tuple.

This is, so far, the most useful method for us: it allows us to get the full data (key and value), and print them out line by line. But we can get even more control using an additional feature of the for-loop.

If we use a Python for-loop to iterate over a collection, and the item type of the collection is a tuple, then the loop can contain multiple loop variables:

The rules for this type of for-loop are quite straightforward:

Using this variant of the for-loop allows us to get access to each key and value individually, and we are free to format the output ourselves. In the code example above, we have done this using an f-string.

Our output is now more nicely formatted. It is, however, still very long, and identifying frequently used words is still tricky. Change this by doing the following exercises.

The list is now much shorter, but it still contains a lot of very common words, such as "a", "to" and "the" – words that do not tell us much about what the text is concerned with. Let us do a first attempt at identifying interesting words by dropping all short words.

We can see that we are getting a bit closer. If you run this version of your program on one of the book texts provided, you can see that a list of words slowly emerges that may characterise the book. But the rules we have used to select the words (filtering by a word count that is more than a fixed value, and by length of the word) are not really good enough.

What we need to do instead is not look at the absolute frequency of a word (the number of times it appeared), but look for words that appear more often than they would in a typical English text. The word "the", for example, may appear very often, but it does so in all English writing, so this does not make it an especially interesting word.

This will be our goal in the next section: We will use existing statistics about expected word frequencies in the English language to identify "over-used" words in our text: words that are used more often in this text than they are used in average English texts.

In our project, we have provided a helper function called get_expected_frequencies(). This function returns a dictionary that lists the expected frequencies for the most common words in an average English language text. (It uses a dictionary of about 64,000 words.)

The dictionary has a word as its key, and the expected frequency as its value (Figure 2.3). The frequency here is the ratio of occurrences of the word relative to the length of the text. For example, as a percentage, the word "the" is around 6.9% of all words in a typical text. The ratio can be computed by dividing the word count by the total number of words in a text:

Using this formula, we can easily calculate the frequency of each word in the text we wish to analyse and then compare it to the expected frequency. Those words that are used far more often than expected are the "interesting words" – the words that might give us a clue to what the text is about.

Let us go ahead and implement this idea. We will identify the frequent words, which we define as words with a frequency that is at least 10 times the expected frequency and which were used at least 10 times.

With the code so far, we can print out the dictionary of frequent words. We have, however, no direct control over the exact number of words we will receive. This will depend on the length of the actual text and the words within it.

Let us make one last improvement to specify exactly how many words we wish to see. To do this, we will sort the frequent words by their word count, and then select the top words from this list.

We can do this by making good use of Python’s built-in sorted function:

We can add this statement to our get_overused_words function to create a list of the frequent words, sorted by their word count.

This is quite an advanced use of this function, so let us investigate that line in some detail.

After doing this, we can use a slice to take the first few elements. For example

will give us the 12 most frequent words. We can then return this list from our function.

In making these changes, it is likely that you ran into a number of errors. This is to be expected. We have made many changes, and we will always make errors along the way. It is not the goal to write our code error-free at the first attempt, but to get to know Python well enough that we can identify and fix any errors that occur. So do not worry if it takes you many iterations of writing code, running, getting error messages and fixing errors to get to the eventual result.

Once you manage to complete these tasks, you will see that you can now get a vague impression what a book is about. You can see that War and Peace is about a prince, a princess, an emperor and Napoleon, or that Three Men in a Boat is about a boat, a picture, locks and rowing.

A version of this project that includes all the changes discussed here is available as thematic-analysis-v4; you may like to compare your own version to this one.

In this chapter, we have learned how to use dictionaries to achieve more advanced functionality.

Dictionaries are perhaps a little more unusual than lists, but they are equally useful. To become a good programmer, it is important to become familiar with dictionaries, and fluent in using them. They should be one tool in your standard toolbox, so that you can use them whenever they can help to solve a problem. You will see that they come in handy more often than you initially expect.

Python has four main built-in collection classes that every programmer should know: list, set, dict and tuple. (There are other collection classes, but these four are the most important ones.) All of these serve to collect several items of data into a single grouping, which we can then store in a single variable. They differ in their characteristics and their uses.

All of these classes have methods that allow us to perform useful operations on them.

Let us have a quick look at each of these four classes before we describe them in more detail.

A list is an ordered collection of flexible length. Elements are accessed by index.

A dictionary (also known as dict) holds entries of key/value pairs. We can look up the value by providing the key.

A set holds an unordered collection of elements. Each element can occur only once.

A tuple is a fixed, unchangeable grouping of several items of data into a single item.

Below, we describe each of these classes and their characteristics and uses in more detail.

We assume that you have used the list class previously, and you hopefully have quite a good understanding already of what a list is and how it works.

We can represent a list graphically as a sequence of elements:

In Python, lists are written using square brackets:

An empty list is created by using the square brackets on their own:

We can access individual list elements using their index (where the first item is index 0, not 1):

Indices can also be negative; in that case, they are counted from the end of the list. For example, -1 is the last element in the list, -2 is the second last, and so on:

List elements can be replaced. The following code will replace the second sighting with a different one:

Copies of parts of a list can be created by slicing. The start index is included in the slice, the end index is excluded. The following creates a new list with the middle thrtee elements from the sightings list:

Slices can include a step width as a third value. For example, the following will select every third element:

Index values for the start and end can be left out. If they are not specified, the beginning or the end of the list is used, respectively. Index values can be nagative to count from the end of the list. All the following statement are valid:

A list has the following characteristics:

To find details of the following methods, look them up online. Each of these methods may have required and optional parameters.

Assume we are programming a bird watching app, and we would like to record sightings of birds on a given day. The diagram above shows how each individual sighting can be appended to a list. With each new sighting, we could append a new entry at the end of the list (using its append() method), and the list would keep a record of all the sightings of the day.

Lists can contain duplicates. So if we see two robins in a row, we can add two list elements for "robin".

Since the length of the list is flexible, we can record any number of sightings, and the order of the sightings would be maintained (we could later retrieve the information in which order we saw each bird).

The list class also allows us to insert elements at arbitrary positions in the middle of the list, delete individual elements, or access elements by index.

We can iterate over the list to receive the elements one by one, in the order in which they are stored.

While lists allow elements of different types to be mixed – we could, for example, have a list where some elements are strings and some are integers – this is rarely useful. Most of the time, we use lists where all elements are of the same type.

A dictionary is a mapping from a key to a value. We can think of it as a table with two columns. Each entry in the dictionary is a new row. We can then look up entries by providing the key and receiving the associated value.

Internally, Python calls the dictionary class "dict". Dictionaries are written using curly brackets and colon-separated key: value pairs.

The statement above creates a dictionary with four entries, where each entry is a key:value pair.

An empty dictionary is created by using the curly brackets on their own:

To access elements, we can use the get method with the key as a parameter, and the dictionary will return the associated value. For example:

This will print the number 8 (assuming we have a dictionary as shown above.)

An alternative syntax allows us to use square brackets instead of the get method:

This achieves a similar result. In general, however, the get method should be preferred: If the key used in the get call is not present in the dictionary, it will return None. We can then check the result to see whether we did receive a value. If we do the same with the square bracket syntax (look up a non-existent key), this causes an immediate error, and we do not get the opportunity to check the result. Therefore, the square bracket syntax can only be used if we are certain that the entry exists.

A dictionary has the following characteristics:

The most important dictionary methods are listed here. There are more methods available; you can find details online.

In our bird watching app, we might not be interested in the exact order in which we sighted each bird, but only in the number of each type of bird we have seen. In this case, a dictionary is an ideal collection to use. The example above illustrates how we can create a dictionary that uses the name of the bird species as a key and uses the value to store the number of sightings for that kind of bird.

When we later wish to add a sighting to our data, we need to carefully distinguish the case where we already have an entry for the bird species from the one where the bird does not yet exist. If the species is not in the dictionary, we will add a new entry. If it exists already, we need to read the current value, increment it by one, and write it back:

Note that using the square brackets to access the value is safe here, because we have checked that the entry exists.

We can safely omit the check for the existence of the entry by using the setdefault method:

This statement is worth studying carefully, since it can be confusing at first. The reason for the confusion is the puzzling name of the method. The method is more correctly described with a name such as get_value_or_set_default: What it does is this: If the first parameter (species) exists as a key, then it returns its value. If it does not exist, then it will create it, and use the second parameter (0) as its value.

Think this through with an existing and with a non-existent key: if the key exists, its value will be incremented. If it does not exist, it will be created and set to 1. Thus, this statement works in every case, without explicitly checking whether the entry existed.

We can also iterate over a dictionary in a very similar manner as we do with a list:

Be aware, though, that this pattern – where we use only a single loop variable, will just give us the keys of the dictionary. To retrieve both the keys and the values, we can write a loop with two loop variables iterating over the dictionary’s items:

If we compare this version to the first example using a list, we can see the difference: In the dictionary version, we do not store the order in which we saw the birds. In return, the storage is much more compact: if we see 300 birds of five different species, then the list version will have 300 entries. The dictionary version, on the other hand, has only five entries with a key and value each. Thus, we save a lot of space.

Looking up the number of sightings for each species is also much faster and easier in the dictionary version – in fact, dictionaries are exceptionally fast in looking up values.

Every time we choose a collection class, the choice will be determined by these questions: What information do I need to store? How much space will this need? How quick and easy is it to look up the information I need?

In our case, if the order of sightings is important, we use a list. If the count is all we are interested in, we use a dictionary.

A set is a collection we have not encountered in this book before. It is, however, quite easy to understand: a set is an unordered collection that stores each element at most once. We can represent it graphically like this:

Sets in Python are written using curly brackets:

Note that with this example, the sightings set would hold only three elements. We inserted a "robin" twice – this does not cause an error, but since the robin was already included before, the second adding of a robin has no effect.

Note also that the set uses the same symbol (curly brackets) as the dictionary. When writing literals, we can distinguish them by their elements: if the elements are key:value pairs, we are creating a dictionary, if the elements are single values, then we create a set.

We cannot, however, create an empty set by just writing a pair of curly brackets ({ }), because this creates a dictionary. To create an empty set, we use the set() function instead:

We cannot access the elements individually (there is no index provided). When we iterate over a set, we will receive all the elements, but they will be provided in an unspecified order. The order in which they are inserted is not maintained, and it may change at any time.

A set has the following characteristics:

The most important set methods are listed here. There are more methods available; you can find details online.

Thinking about our bird watching app again, we might use a set if we are only interested in the kinds of birds we spotted that day. If we do not care about the number of birds we saw, or in which order, then a set provides a much simpler, smaller and faster data structure to store this information.

We would still add an element each time we see a bird, and the set would automatically ignore duplicate entries. The size of the set is flexible as well, so an arbitrary number of species can be stored.

Elements in a set cannot be replaced. We can add and remove elements, but we cannot directly replace an element with another, as we can in a list.

As with lists, sets containing elements of different types are possible, but rarely useful. When mixing different types of elements, we also have to be aware of details of equality. For example, the values False and 0 are considered equal in Python, so in a set they would be considered duplicates. A set that contains the value False will therefore ignore the addition of a 0. This can be confusing, and sets with different element types are best avoided.

Use a set when duplicates cannot occur, or they do not matter to you. Also be aware that the elements do not maintain a specific order. If these conditions are true, then a set is quicker, easier and often consumes less space than other collections.

A tuple is a grouping of a number of different data items into a single, fixed group. The following example shows two tuples which we may use for our bird watching app. In these tuples, we combine three pieces of data: the common name of the bird, the scientific name, and the maximum size (in cm).

Tuples are used to combine items of data that logically belong together and should be handled (created, stored, processed) as a single unit. The individual data items may be of different types. Once a tuple has been created, it is unchangeable: neither can the values in it be modified, nor can it be extended. It will remain as it is.

The name tuple may seem odd. In English for referring to groupings of two, three and more elements we use the words pairs, triples, quadruples, quintuples, sextuples, septuples and octuples. Programmers wanted a word to refer to this concept generally so they use "tuple" as it’s the last part of most of the words.

Tuples are written using round brackets (parentheses):

In theory, we can create an empty tuple by writing just a pair of parentheses: ( ), but in practice this is not really useful. Since tuples cannot change later, this particular tuple does not hold any interesting information.

To access elements from the tuple, we can either use indices (in the same way as we do in a list):

or we can iterate over the tuple using a for-loop.

Tuples can be sliced in the same manner we have seen for lists. For example

selects and returns the first two elements of a tuple (as another tuple), and the following statement

selects the last two elements. The statement

creates a tuple containing every second element from the original tuple.

A tuple has the following characteristics:

The tuple class has two built-in methods.

Examples of tuples may include the representation of playing cards: ("diamonds", "7"), information about a person: ("Sam", "Smith", 21, 3, 2011, "f"), or information about a font style used in a text document: ("Courier", "Bold", 12, RED). In any situation in which different pieces of information belong together, they can be grouped in a tuple.

In the context of our bird watching app, let us assume that birds are actually captured, measured, recorded, and then released. Then we might want to record each sighting individually, including the recorded measurements. For this purpose, we might use a tuple to store the details:

When we store the sightings, we might then use a set or a list of these tuples, instead of storing just a list of strings.

Sometimes we want to store two data items that belong together – this is often called a pair. Examples might be coordinates in a Cartesian coordinate system (x, y), or the description of a musical note (frequency, amplitude).

While we may talk about a "pair" as a logical data structure, in Python it is not a separate type. A pair is simply a shortcut name for a tuple with two elements.

We use a tuple when we store a fixed set of data items, where the number, kind and value of data is known in advance and does not change after being created.

The last – and most powerful – aspect of collections is this: collections can be combined.

Collection classes themselves are types of data, and collections can hold elements of any type, so a collection can hold other collections.

In the last section above, we have already mentioned the possibility of a list of tuples. Equally, we could have a list of lists, or a dictionary where each value is a set. The depth of these nestings can be even deeper: There is nothing stopping us from creating a list of sets of dictionaries, in which the values are tuples.

This may sound complicated at first, but is in fact quite common, and not that hard to understand. Think, for example, of a contacts app on your phone or computer. At the top level, it might hold a dictionary with the name of a person as the key. The value – the information stored under the key – however, is not a simple type, but a tuple holding various pieces of data (phone number, address details, email address, picture, etc.). If we look at this even more carefully, we might realise that a person may have multiple email addresses, or multiple phone numbers. We do not know how many there are, so this should probably be a list. So we now have a dictionary holding tuples, in which some items are lists.

Constructs like these are common in computing, and we will see an example of this in action in the following chapters.

Python has four main collection types: list, set, dictionary and tuple.

It is important for a good programmer to become familiar with these types. Each has different characteristics – some can grow, some cannot, some are ordered, others are not, etc. – and we need to be able to choose an appropriate collection for a task.

Collections are often combined – collections can hold instances of other collections – and this is useful for storing structured data.

So how can we predict what the next word in a sentence may be? The obvious idea is to look at the preceding words, and then try to figure out what next word might fit to continue a sentence.

At its most basic, we can look at only the one word immediately before the next one. This will not be terribly good at producing sensible text, but it’s a start. So let us try to do that first.

How can we know what word might work to follow another word? Again, the basic idea here is simple: We look at a good amount of existing text, and look what words follow other words there.

Let us investigate an example by looking at an English sentence:

In this sentence, we can see that the word "I" can be followed by the word "read", the word "read" can be followed by the word "the", and so on.

We can now write down all the pairs of words that we can see may follow each other:

These pairs of consecutive words from a text are called bigrams. The list of pairs above are all the bigrams from our sentence.

Now, doing this with just a short sentence is not very useful, but doing the same with a very large body of text – say, a whole library – gives us very interesting information. It will tell us all the words that may follow a given other word, and it will also hold information about the likelihood of one word appearing after another. Pairs of words that appear frequently together will appear more often in our list of bigrams.

To experiment with this, let us write a program that can produce a list of bigrams from a text.

Python has a standard function called zip, which can combine elements from two lists into a list of pairs. It will pair the first elements from each list, and the second elements, and the third elements, and so on. Let us look at this with an example.

Figure 4.1 shows two lists of strings. We can take these two lists and merge them with the zip function:

The zip function will pair up the elements of the two lists as shown in Figure 4.2. A pair is simply another name for a tuple of length 2, so what the zip function actually creates is a sequence of tuples with two elements each.

The zip function does not return a collection itself, but a special object called an iterator. This object represents a sequence of elements that can then be turned into a collection of our choice. We can turn it into a list by writing:

We could also have chosen to turn it into another type of collection. The resulting list in this case is shown in Figure 4.3.

So the question now is: How can we use the zip function to create the bigrams from our text?

The solution lies in the idea to take the list of words in the text, and then to match this with a list that is shifted by one word. We create this shift by just leaving out the first word in the second copy of the list. (Remember you can create a copy of a list without the first element by using the slice list[1:].) This is illustrated in Figure 4.4.

If we zip these two lists as we did previously, we will get our list of bigrams.

The text that we use to create our bigrams is the training text. We can use different texts to train our system, leading to different output. If we train the text, for example, on Jane Austen books, our system will use the language of Jane Austen. If, on the other hand, we use scientific texts to train the system, it will use that vocabulary.

Now that we can generate a list of bigrams for any given text, we can use this list to predict the next word. The idea is now straightforward: First, we look at the previous word. From the list of bigrams, we select just those where the first word of the bigram matches our previous word. For example, if our previous word was "It", then the list of matching bigrams might start with

From this, we can make a list with all the second words, and then randomly select a word from that list. Since words that more frequently follow our previous word occur more often in that list, they will have a higher chance of being selected as our next word. In other words: the frequency of a word appearing as the second word in the training text determines its probability to be selected. (For predictive text, we would select the most frequent word. For creative writing, we will randomly select one of the list.)

We have now reached a stage where we can give a word and a list of bigrams to our function, and it will return a plausible following word. It is plausible in the sense that there is at least one occasion in previous writing where this sequence of words appeared. While producing a single next word does not seem like much of an achievement, generalising this to produce longer text is now very straightforward.

The only additional thing that is needed to go from generating a word to generating longer text is to do so repeatedly. If we put our call to generate the next word into a loop, and at each iteration take the previous word that was produced as the input word, we can generate sequences of text of arbitrary length.

Let us try that with the following exercises.

A copy of the project with all this functionality implemented is in the book projects as text-generation-v2. Feel free to compare your own code to the one in that version.

So what have we achieved so far?

We can now read in a training text that generates a list of bigrams. The bigram-list is our "model" – this is what is used to generate text later on. We "train our model" by reading in text and generating the bigrams. The kind of text documents we train our model on determines the kind of language our system produces. If you train your model using a different text, it will produce different language. (Try it out!)

Of course, we could read in not just a single book, but a whole stack of books. The more text we use to train our model, the closer our system will follow normal English speech patterns.

When we look at the output (for example, in Figure 4.5), we can see that the result so far is somewhat entertaining, but not very impressive. Very superficially, it looks like English, but there is no recognisable grammar, and this output is not fooling anyone. We will improve this in the next chapter.

Another observation is worth pointing out: This time, we have not stripped out punctuation when we split the text into words. As a result, punctuation is still included in the bigrams, and thus in the output. The words "it" and "it," are just considered different words, and handled separately, each with their own following words. This makes the resulting text look more natural.

What we have called our "model" is a simple list of bigrams: a list of words that can follow another given word. We have used this to produce our sequence of words. This looks a little bit like English, but the word sequence is frequently odd and a bit too random. What, for example, should be the next word if the previous word was "your"? Could you predict a reasonable next word?

It is obvious that the possibilities are very wide, and you are unlikely to come up with a word that makes much sense in any specific context.

But how about predicting the next word if the previous words were "How is your"? Now the options for a reasonable next word are much narrower, and you can probably come up with a few choices that could make sense ("How is your mother?", "How is your headache?", etc.)

The idea we get from this observation is quite straightforward: Maybe we can improve things if we look at longer sequences of words.

Let us start investigating this idea in a simple form at first: instead of looking at bigrams – pairs of words – we look at trigrams: sequences of three words. Consider, for example, the text from Figure 5.1:

The trigrams from this text are all the three-word sequences that occur in the text. We can write them down in a list; they are:

Once we have a list of trigrams, we can produce new text with a similar idea as before: we predict the next word. This time, however, we do not only look at the preceding word – we look at the two preceding words. Given the two previous words in a text, we then find the trigrams where the first two words match the two preceding words, and we select the third word from one of the matching trigrams as the next word in our generated text.

In Python, each of the trigrams is stored as a tuple of length 3, and all the trigrams – as before – in a list. Generating the trigrams is quite straightforward, since the built-in zip function can also merge three lists together (or, in fact, any number of lists).

As before, we have to generate lists with the words shifted by one word, and we do so by slicing off the first word for the first copy, and the first two words for the second copy (Figure 5.2).

In our program, generating the trigrams then looks like this:

Change your entire program to work with trigrams now, using the following list of exercises.

If you find these exercises difficult, you can compare your attempt to our implementation in text-generation-v3.

If you have done the exercises so far – how is your output now?

You will see that this simple improvement of our "model" – going from bigrams to trigrams – significantly improves the quality of the text it produces. Now the output looks almost like grammatically correct English! That is quite astonishing for such a simple algorithm.

Of course, the sentences are still gibberish – they don’t mean anything, and you cannot read any sense into them. We discuss below how Large Language Models (LLMs) manage to do better than this.

We can now, of course, go even further with the same idea: instead of using three-word sequences, how about four-word sequences? Or five-word sequences? This idea takes us from trigrams to n-grams, and we will discuss it below. But first, let us look at another idea to improve our system: improving our training data.

There is a very simple change that we can make to improve the quality of our system: use more data.

In real LLMs, the models are trained on millions or billions of documents. Many of them are fed any text you can access on the Internet, and much more besides.

We will not use that amount of text, but we have two files available that contain just a bit more than a single book. The Strype file system folder includes a file called /books/fairy-tales.txt, which holds the full text of 65 fairy tales by the Brothers Grimm, and another file called /books/sherlock-holmes.txt, which has the text of 27 Sherlock Holmes stories (three books and 24 short stories). Try your program with these texts as the training data.

A version of the project with these changes is in the book projects as text-generation-v3.

If you did these experiments yourself, you will have noticed two things: Firstly, using longer texts to "train our model" improves the quality of the output. The more text we feed in, the more closely the output resembles actual language. We have not, in fact, used very much data. Even the "sherlock-holmes" file contains only about 330,000 words. This is minuscule in the context of large language models, but it already starts to deliver quite surprising results.

And secondly, the content of the text we feed into the model determines the language and style of the text that is produced.

These two observations lead to two very significant issues: Firstly, with the aim to be able to produce any kind of output, large AI companies have been scanning all writing they could get access to. This includes, for example, millions of published books. For the performance of the system, this is great: you can tell your AI system to write a short story in the style of Stephen King, and it will very competently do this. It has read in all existing Stephen King stories, and so it can copy the style amazingly well.

The problem is that these stories are under copyright, the AI companies have not legally licensed them, and Stephen King has not explicitly given his permission.

So, is this legal?

Currently, this is an undecided question. The AI companies argue that it falls under the "Fair use" clause in copyright law, since the source works are not reproduced directly. Many authors argue that this is copyright theft, since their own original hard work is being used – and being commercially exploited for profit – by other companies without their permission.

The second problem is the sheer amount of data involved, and the amount of processing power and electricity required to process it. Reading and transforming the "sherlock-holmes" file is trivial, and a standard laptop can easily do it. But processing the amount of data involved in real LLMs – more text than the content of all the libraries in the world combined – requires so much processing power that the environmental impact is immense. Power is not only needed to train the model (this is only done infrequently), but for every single question that an AI system is asked to answer. Even the most trivial query requires huge processing power and consumes large amounts of energy.

To cope with this, AI companies are building huge data centres in many areas of the world, full of processors which only serve AI queries from users. A single data centre can consume as much energy as a city of a million people. For the environment, this is a disaster; it creates significant new problems for the goal of reducing the environmental damage which we cause.

One other interesting observation is this: Our algorithm needs absolutely no understanding of the words it produces. If you feed in a text in Spanish, it will happily produce Spanish output. Or any other language written in our alphabet. (Try it!) The outcome is quite impressive, but it shows that the system has no understanding of the text at all – it just shuffles words around.

We already mentioned the next obvious improvement above: using even longer sequences of words to match before selecting the next word. But instead of making a version now for sets of four or five words, let us generalise our algorithm: we can write a function that can produce n-grams (groups of n words) for any given n.

We are getting into quite advanced territory now. Let us start, this time, by analysing and studying a solution to this problem.

Let us go through the program and investigate the interesting aspects more closely.

We can see that the make_ngrams function now has two parameters: the source text (text) and the n to specify the length of our n-grams:

To create our n-grams, we now need n lists, each shifted by one word. We do this by creating a list of lists, and adding the individual lists to it:

Think through this code carefully: You will be able to see that it creates n separate lists, each with a different number of words cut off at the start by slicing, and then adds these lists to another list of lists (word_lists).

The next line contains a construct we have not seen before:

Here, we have a list, preceded by a single asterisk (*). The * operator, when written as a prefix before a list, is the list unpacking operator: It takes all the elements of the given list and presents them as separate parameters. For example:

is the same as writing

The elements of the list are unpacked into separate parameters, and then passed to the print function. In the same manner, the statement

is the same as writing

if the length of the list is 4. The power of this operator is that it works for any length of a list. We could not write the fully written out alternative without knowing in advance how long the list is.

In the generate_next_word function, the most interesting statement is as follows:

Here, we make use of the fact that we can use a negative index both for slicing, and for accessing elements in a tuple. If we have a tuple

and then we create a slice t[:-1], we are slicing from the beginning of the tuple to one position before the end (-1). The resulting tuple will be

Similarly, we make use of the fact that we can write t[-1] to refer to the last element in a tuple. Taken together, the if-statement above checks whether the n-1 first words of the n-gram match our previous words, and if they do, append the last word of the ngram to our results list.

Study this code until you are comfortable with it. Don’t be surprised if this looks difficult or confusing at first – this is getting into more advanced areas than we have seen before, and it is not surprising if understanding this is not easy at first.

Another few observations are worth making about this version of our project:

You will see that the produced text now looks pretty good, even though we are still using quite a simple algorithm, which we can write in a few dozen lines of code. Before we wrap up this chapter, we will look at two more internal improvements we can make to our program.

In this section, we will not make any improvements to the functionality of the program. Instead, we will just write the same functionality using different Python constructs.

If you like, you can safely skip this section. If you are mainly interested in learning about the principles of programming, then the solution we have written so far is perfectly good; we have used lists, loops and if-statements to build and process our data structures, and this is how we could do the same in many different programming languages. It gives us a good understanding how this algorithm works.

If, however, you are interested in the specific details of Python, then read on. Python has some constructs that allow us to write the same code more concisely. This version makes use of list comprehensions, which is an elegant way in Python to achieve the kind of processing we have used here.

The statements most likely to be new to you have a form we have not seen before. For example:

This is called a list comprehension, and it is a construct that can create a new list by filtering and mapping elements from a source list. Let us assume we have a list of numbers:

We can then create a new list that holds all elements from the original list which are greater than 10 by writing the following:

This code is equivalent to writing the following without list comprehensions:

Using list comprehensions has a number of advantages:

We will not discuss this variant in detail here. If you aim at becoming an expert Python programmer, then you should study this construct more closely. You can do this by reading Appendix C: Python list processing, and then studying this version of the project.

One last change that we will briefly present here is another improvement that is not essential, but may be interesting to study. Again, this is an advanced topic, and you can safely skip it if you like.

This version of the project is provided in the program text-generation-v6. You can open and study this version.

The main difference in this version is that we are now using a dictionary, instead of a list, to hold our n-grams.

Previously, we had a list of n-grams, and every time we wanted to find a next word, we would provide a prefix tuple with the previous words, and then search our list for n-grams where the first n-1 elements match our prefix tuple (Figure 5.3). As a result, each new word involved searching our entire n-gram list again.

In our new version, we instead create a dictionary. The keys in the dictionary are the n-1 words of each n-gram, and the value for each key is a list with all the possible following words (Figure 5.4).

Now, finding each next word consists of one dictionary lookup, and then a random selection of the following word from the list of possible words. Since a single dictionary lookup is much faster than searching a whole list, this version performs much better.

Discussing this solution in detail is beyond the scope of the book, and we leave it to the interested reader to study this version. It is included mainly for especially keen or advanced readers – you can safely progress without studying it.

We have seen that our model does not come close to the abilities of current AI systems, and there are many reasons for this.

Firstly, there is room to improve our current system. One idea is again simple: use longer chains of words. We have used n-grams with a small n – no more than 4. If we had much more source text (such as a whole library), we could work with longer n-grams.

Using more contextual words does indeed improve the performance of these systems. And the project that we investigated in this chapter does indeed give a glimpse into the history of text processing.

Until the mid-2010s, n-gram-based systems were considered the state of the art in text processing. They were successfully used in predictive text systems and speech recognition.

After that, text processing systems evolved significantly, and modern LLMs have a number of important differences that make them more effective:

The resulting systems are known as generative pre-trained transformers (GPT). The full details of these are more complex than we can discuss here. But ultimately, they are generalisations of the simple system we have built here. Or in other words: An n-gram model is a special, extremely limited case of what a GPT transformer can represent.

N-gram systems, however, are not obsolete. They continue to be useful in predictive text, spell checking, speech recognition, and in document retrieval systems.

While our project here does not allow us to understand fully how LLMs work, it does give an insight into the core of the idea that underlies LLM-based AI systems. Perhaps most importantly: It is still the case that LLMs have no real understanding of the content and meaning of the text they produce, and are ultimately based on clever manipulation and prediction of language. These systems may state an entirely invented "fact" with the same confidence as an obvious truth.

AI companies are working hard to improve the reliability of LLMs. This tension, however, between our desire of trust and these systems, which ultimately just manipulate language, will remain at the core of the issue of reliability in AI systems for some time.

The implementation of the improved n-gram algorithm in this chapter has given us more practice with tuples and lists. We have seen how we can use these data structures very flexibly to solve a number of very different problems. While doing this, we have also gained an understanding of how text prediction systems can work, such as those used for predictive text in messaging apps.

We have, however, only scratched the surface of the possible uses of lists and other data structures. In the next chapters, we will look at the use of these classes in an entirely different context.

The game we will work on in this chapter is a text-based adventure game named Haunted Mansion (Figure 6.1). Games like these have a long tradition in computing. Using purely text-based output looks quite old-school, but you will see how you can build really fun games with this idea. And we will look at adding graphics and sound towards the end of this chapter.

To start this project, we will use a partially developed version of this game. You can find it in the Chapter 6 folder as haunted-mansion. Once you open this project, you can see that there is already a good amount of code there, so our first task will be to analyse and understand the existing code. This is a valuable skill: When you work with software development, it will happen much more often that you make extensions and improvements to an existing software system than starting your own new project from scratch. Code reading is almost always at the start of a new project.

So let us get stuck in and find out what this project can do so far.

The exercise above has shown you that the game, at this stage, does not do very much, and is certainly not interesting to play. It will be our task in this project to change that.

Before we jump in and make improvements, however, let us try to understand how the code works that is already there. You can make a start on this by working through the next set of exercises.

From this initial exploration, we can see that there is some useful code already in the project. It gives us a good starting point for a game of this type, but it does not implement a game as such. Nothing interesting is happening.

Throughout this chapter, we will work on improving this game and make it into something that is actually fun to play. However, as a first task, you should invent your own game idea. You do not need to make yours a game set in a haunted mansion, you can invent your own story, your own setting, and your own challenges.

For the rest of this chapter, we will continue with our haunted mansion theme, but you should instead work on your own game version.

From the exercises above, you will have gained an initial understanding of the structure of the code. There are, however, a number of details which will probably not yet be clear (that is normal), and it is worth investigating a bit deeper.

One way to become more familiar with existing code is to try to make small changes, and see whether your initial assumptions and guesses are correct. The existing code is actually quite well written: There are well chosen function names, functions all have comments, and the variable names give us good clues what they are used for. All this is a great help. Imagine the same code with no comments and bad naming – it would be a nightmare to untangle! (This shows you the importance of good naming and good documentation: when you need to read other people’s code, you appreciate when this is done well.)

Because of the clear naming, we can probably make some changes quite easily without understanding the whole structure. Let us try to make some changes of this kind.

These exercises should have shown you that some changes are quite straightforward. So far, so good.

The next step is to make our own map: add some more rooms and connect them, so that you can move in and out of each location. Looking at our code, we can see that creating rooms is easy. There are statements at the beginning of the setup function that create each room, and it is not hard to copy those and make more rooms. These statements, however, use a mechanism we have not seen before: a named tuple. We need to look at this more closely in a moment.

The other challenge is setting the exits. Linking the rooms together to move from one to the other looks a little more mysterious. It is worth investigating this part of the code more closely as well. Let us look at these two issues next.

To store the information for each room, we would like to store two strings: a short description (a single line) and a longer description, where we can provide a more extensive text.

We have previously seen that we can use a tuple to store two bits of related data together in one variable. We could now make a tuple with these two strings to store the room information:

This works well to store the data we need.

When we later want to use this data (let’s say, we want to print out the long description), then we can access the elements in the tuple using an index. For example

would print the long description. This works, but it is not nice. When we see the expression outside[1], it is not clear what gets printed. We would have to look up (or memorise) what is stored at index 1 to know what is going on. The code is correct, but not very readable.

To improve this situation, Python has an additional construct: named tuples. Named tuples are tuples like any other, but in addition to storing the data, we can give a name to each component, and to the tuple itself.

The original tuple stores only the data:

We can see that it automatically assigns an index, which allows us to access each component.

A named tuple stores the same data, but it allows us to attach a name to the tuple, and a name to each component:

If we do this, then we can use the name (instead of the index) to access a component:

We can immediately see that this code is more readable. Without needing to memorise the details of the tuple, the code tells us exactly what is printed out here.

To use a named tuple instead of a standard tuple for our rooms, we have to do three things. First, we have to add an import statement to make the named-tuple functionality available in our program:

Secondly, we create a description of our named tuple:

As we can see, this is an assignment. On the left hand side of the assignment is the type name of the named tuple we are creating. On the right hand side is a call to the namedtuple function (which creates the named tuple) with two parameters: The first parameter is a readable name for the tuple overall, and the second parameter is a list of strings with a name for each component. This creates the names which we can later use to access the components.

The third and last part is the creation of the tuple itself. It looks similar to the original creation of the tuple, but we now write the type name of the named tuple in front of the parentheses:

This turns our standard tuple into a named tuple. It does not store any more or less data, but it makes the resulting code much more readable.

The other interesting aspect to investigate more closely in this example is how the exits are stored. Figure 6.2 shows the map we wish to represent for our game. So, how can we store this information?

In our setup function, we see these lines:

We should now be able to recognise that this is a dictionary: It is written with curly brackets, and the colons separate the keys from the values.

Looking carefully at the exits dictionary, we can see that the first entry is outside : {"north" : entrance}. This is the key:value pair that makes up each entry in a dictionary. Let us look carefully at the types and values used here.

On the left of the colon is the key. It is outside, which is a variable holding a room. So the type of the key is Room. On the right side of the first colon is the expression { "north" : entrance }. This is itself another dictionary! So the value of the entries in our dictionary is another dictionary.

To fully understand this structure, it helps to draw a diagram. Figure 6.3 shows the data structure we are constructing here to store the information. It is worth spending some time studying this, and comparing it to the map in Figure 6.2, until you understand how this stores all the information about the exits.

Some observations are worth noting here:

If you have read the code in this project carefully, you will have seen that it creates a number of items, which are placed into the rooms. When we walk through the rooms, however, the items are currently not shown. Let us fix this now.

We have seen here that items are created and correctly placed in the rooms. What is missing is telling the player about these items as they walk around. Let us do that now.

These last exercises are getting quite tricky. Do not be discouraged if you find them difficult at first. It may help to discuss them with others if you have people nearby who you can work together with. If you can solve those tasks, and fully understand how they work, then you have reached a good level of understanding of data structures.

If you like to check your code against ours, you can find a version of the project with the 'look' command implemented in the book projects as haunted-mansion-v2.

From here on, it is really up to you. We will not discuss much additional functionality in detail, but leave it to you to invent and implement more ideas to make your game more interesting and more fun to play. The constructs we have discussed so far enable you to achieve many varied things.

Here are some ideas what you might add to your game. But they are just this: ideas. You are free to add them to your own game, vary them, or do something different entirely.

Of course, there are many other possibilities here. Invent some functionality of your own and implement it.

One last thing that we will suggest and briefly discuss here is addition of some images and sound to our game. The most straightforward way to do this is to leave the game with text-based input, but show an image – and possibly play a sound – when you enter a room (or when something else happens in the game).

The project version haunted-mansion-v3 shows a demonstration how this might be done. Open this project and run it to see what it looks like. To do something similar in your own game, you can do the following:

That is all that this version of the project does. You could, of course, also add images for items, or attach sounds to other events. It is all up to you. We leave you to finish the game on your own from here. You now know enough to be able to turn this into something exciting.

In this chapter, we have made a start of creating our own game. This involved a good amount of code reading: In many cases, the work of a programmer starts by reading existing code, before we make modifications and extensions. This is an important skill that we should practice.

We have introduced one new construct (the named tuple), and we have combined the data structures we already knew in some more interesting ways. We have seen that we can put collections into other collections, and that we can build useful data structures this way.

Programming will increasingly be like this from here onwards: You will encounter new constructs less often, and you will increasingly work by putting what you know together in more interesting ways. There are still some constructs to discover – we have not shown you all of Python yet – but not many; you now know all the important ones.

The subsequent final chapter just lists some more project ideas, without further discussion. We leave you with those half-baked projects and hope that they spark some ideas, that you choose to continue to write programs, and that it brings you joy.

This project does not implement any stand-alone functionality, but a visual effect: an explosion.

It starts off with some rocks. Press a key on the keyboard to set off an explosion, and sit back and watch.

The explosion is implemented by replacing a rock on screen with 40 smaller parts of rock fragments (of random size), and giving each of these fragments a random initial speed and direction of movement. The inherent movement speed is stored as a dx/dy value pair – the distance which the fragment will travel in the horizontal and vertical direction at each step.

We can then easily apply gravity: At each animation step, we change the vertical movement speed (dy) by subtracting a gravity constant. This will make dy a larger and larger negative number, increasing the speed downwards at each step.

This project implements flocking behaviour.

Flocking is a natural animal behaviour exhibited by various groups of animals. The name flocking is taken from flocks of birds (such as starlings) which are famous for this phenomenon. With schools of fish, it is called "shoaling". Swarms of insects or herds of land animals can also show this behaviour.

In a flock, each individual’s motion is influenced by several factors: the desire to stay near to others, the intent to travel in the same direction as everyone else, the goal not to bump into each other, and so on.

If you are interested, do a web search to find out more.

This is an implementation of an old game, first written for the Greenfoot environment in Java, and here translated to Python.

It is very similar to the Yellow-Fish project, which we discussed in the first volume of this book series. The main difference (apart from the use of different images) is that the project here was implemented using classes and objects.

It serves here as an example of object-oriented programming. In a way, using objects results in cleaner, more structured code, that can be easier to understand and maintain. If you would like to study object-orientation in Python, have a close look here.

"Cheese" is a fully implemented, playable game. You fly your wedge of cheese through the game, shooting blobs of ketchup at meatballs.

This is quite a sophisticated program, implemented using classes and objects, including a reusable Vector class. Various constants allow you to make the game easier or harder, such as the number of meatballs, the speed of the cheese, or the rate of firing.

The game is based on the classic arcade game "asteroids" (do a web search!), and there is also an asteroid-themed version of the game available here. (It is essentially the same code, with different graphics.)

If you would like to investigate data analysis (and data science in general) a bit more, then you will eventually move on from writing your own loops to sift through the data, and instead use custom-made libraries that make this functionality easier and more convenient to access. The main libraries that Python provides for this kind of project are pandas, numpy and matplotlib (which includes pyplot).

You may have seen a first example of pyplot in the temperature-plot project in Volume 2 of this book series, and an example of the use of pandas to read and analyse csv files at the end of Chapter 2 of this book (in the top-10-songs project).

Here is a new project that combines use of pandas, numpy and pyplot to do a bit more sophisticated analysis. It reads in a csv file with detailed data about renewable energy creation (an example of a dataset provided on kaggle.com) and produces several graphs of different types. It can also save images of those graphs as PNG image files.

The project is called renewables. If you get it to write PNG files, they will be stored in Strype’s local file store (accessible in the "Files" tab) if you do not use a cloud file system. If your project is stored in a cloud file system, then the generated files will be stored there.

See Appendix A (Python operators) in Volume 2 for a description of operators that work on lists, and slicing.

Lists have methods that can be called on a list object, using the syntax list_name.method_name(). We summarise some of the most important methods below:

There are more list methods, which you can find in the online documentation: https://docs.python.org/3/tutorial/datastructures.html#more-on-lists

The Strype file system is shown in a tab next to the graphics world and console tabs (Figure B.1). It shows two sections: Local files and Built-in files.

The local file area will initially be empty. Files can be uploaded to this area from your computer’s file system to make them available to your program. They are stored for the current session only, and will be discarded when the browser window is closed (unless they have been downloaded or pinned – see below.)

The built-in file area shows a number of files for use in Strype that are provided by the environment. They can be read by your program. They are read-only; they cannot be changed or deleted.

Files from the built-in file area can be accessed using absolute paths (starting with the / character). For example, to read a data file, you might use

You can also access image or sound files directly for use in your graphics projects:

You can click on a file name in the file system view to copy its path for easy pasting into your code.

If you use a filename without an absolute path, you are accessing the current working directory. By default, this is the directory holding the local files shown at the top in Figure B.1. The following code reads a file from the local file area:

To access your own data files, use the "Upload" button in the local file area to upload your file, and then it is available for reading in your program.

Similarly, when you write to a file without an absolute path, it will be written into the local file area. You can then either view its contents or download it to your local computer (see below).

If your project has been stored to an online file system, such as Google Drive or OneDrive, then your current working directory is the online folder where your project is stored.

Thus, writing to a new file named "myfile.txt" will create this file in the online directory where your project is saved. Similarly, reading from "myfile.txt" will look for the file in that directory. Writing to "data/myfile.txt" will write to a file in a folder called "data" in your project’s online folder.

If your file is saved in an online file system, but you still wish to access Strype’s local file area, you can do so by prefixing the filename with the path "/local/". For instance, "/local/air-pollution-liverpool-2026.txt" will access this file in the local area independent of the storage location of the project.

Files can be "pinned" to the project by using the small, pin-shaped control next to the filename. When a file is pinned, the pin symbol is permanently displayed next to the filename (Figure B.2).

Pinned files are saved together with the Strype project. If the project is later opened again, the pinned file will be loaded with the project.

When the mouse pointer hovers over a filename, a set of controls is displayed to the right of the filename for operations on the file (Figure B.3)

The available controls are, from left to right: Delete, Pin, View, Download.

Clicking on the name of the file copies the filename (including its path for built-in files) for easy pasting into the program code.

The Delete operation deletes the file from the local file area. This operation cannot be undone. Built-in files cannot be deleted.

Using the View control shows the contents of the file. This can also be used with image or sound files to inspect those files.

The Download control downloads a copy of the file to the computer’s local file system.

Iteration accesses every element in a list in turn.

The standard for-loop processes every element in a list.

Output:

The enumerate function delivers an index and an element:

Output:

A standard pattern to check for an empty list – in any language – uses a check for its length:

In Python, programmers often make use of a concept of truthyness: certain values in Python are considered "truthy", while others are "falsey".

"Truthy" values, when used in an if-statement, are treated as True, while "falsey" values are treated as False.

Falsey values include 0, None, the zero-length string ("") and the empty list ([ ]). This means that the list itself can be used in the condition.

The Python idiom makes use of the falseyness of the empty list to check for an empty list:

Output:

We are hesitant to make too much use of this feature because its behaviour can be more difficult to understand. In other programming languages truthyness can behave differently: for example in JavaScript the empty list is truthy not falsey.

A list comprehension creates a new list as a mapping from an existing list. They are a replacement for a general pattern that would be written without comprehensions as:

The pattern above can be written using a list comprehension as a one-line statement, without a loop. It takes the form

Comprehensions can be used for map and filter operations, and combinations of these two. Examples are shown below.

A map operation creates a new list from a given list with the same length. Every element in the new list is derived from the equivalent element in the original list. The format is:

The expression is evaluated for every element in the original list, and its result defines the new list elements.

Output:

Output:

Output:

A filter operation creates a new list by selecting specific elements from an original list. The filtered list is always shorter (or the same length) as the original.

Using an if-clause at the end of a comprehension creates the filter condition.

Output:

Output:

Output:

A mapping expression and a filter condition can be used at the same time in a comprehension.

Output:

Output:

The all function checks whether all elements in a list are True:

Output:

Output:

This can be combined with a mapping operation to check a condition for all elements in a list. This is done by using a boolean expression for the mapping to create a list of boolean values, and then applying the all function.

Output:

Output:

A similar test can be done with the function any, which returns True if any elements in its parameter list are True.

Output:

The mapping comprehension described above can also be extended to map to two different expressions depending on a condition. This is achieved by writing an if-else expression before the for-expression:

Output:

For the example above, we can also make use of the "truthyness" of a string, which would detect both None and the empty string as missing data:

Output:

Comprehensions can also be used to create lists, if the list elements follow a fixed pattern.

Output: