Creative Python Programming with Strype Volume 2: Working with data

Chapter 3. Introduction to lists

Topics:

Working with lists

Constructs:

type, type conversion
list operations: index, append, len, min, max, slicing, copy, sort

In the previous chapter, we have seen that we can write programs that handle quite large amounts of data. In the example we have programmed, the data was text, and the individual data items were characters, which were held in a string. Almost all programs, in fact, work with collections of data, often very large collections, and the ability to process large collections of data is one of the core characteristics of computers.

In this chapter, we will investigate how we can work with other types of data – data that is not text. Once you start thinking of applications on a computer, you will notice that almost all of them have at their core the processing of collections of data: word processors handle collections of words and sentences; search engines process collections of webpages; social media apps handle collections of posts; games animate collections of game items; weather forecast programs process collections of environmental readings, and so on.

Think about the programs you use most often on your phone or your computer: there are very few programs that are not based, in some form, on the processing of collections of data.

In previous chapters, we have seen variables, which allow us to store data. We have also seen that variables can store data of different types – such as int, float, or str. However, each variable stores only one value. What if we want to work with hundreds, or thousands, of data items?

It would not be practical to use a thousand different variables. Instead, Python offers collection types – types of data that can store a whole collection of data items in a single object.

The most common of the collection types in Python are lists, dictionaries, tuples and sets. In a later volume of this book series we will look at all of them. Here, we will start by concentrating on the most commonly used type of collection of the four: lists.

3.1. A list example: The temperature dataset

We will investigate and practice working with a list using the temperature project from the book projects. This program provides a dataset of the average daily temperature (in degrees Celsius) in London, UK, for each day in 2025.

Let us start by looking at this data.

Exercise 3.1 Open the temperature project from Chapter 3 of the book projects. You will see that it reads a file called "/data/london-temperature-2025.txt" into a variable called data. Print out the data variable. What do you see?

With the previous exercise, you will have seen that data contains a long list of numbers, which are printed separated by commas, and enclosed in square brackets. This indicates that data is a list: Lists in Python are written in square brackets, and their elements are separated by commas. Each value in this list is the average temperature of one day, starting at 1^st January.

Exercise 3.2 Add the following lines to your program:

numbers = [1,2,3]
print(numbers)

What do you see? Try this with values other than 1, 2, 3 – also use strings and floating point numbers in your experiments. Can you tell from the output what the type of each element is?

In Python, the square brackets [ ] create a list, and we can write the list elements between the brackets. So the expression [1, 2, 3] created a list with three elements in it; printing out the list prints the elements to the console.

Concept Python provides a list type, which can store a list of items. Lists are defined using square brackets and values separated by commas: [1, 2, 3].

If we are unsure about the type of one of our variables, we can inspect it in our code. The type() function tells us the type of data held in a variable. For example,

print(type(numbers))

will print out the type of the variable numbers.

Exercise 3.3 Print out the type of the numbers variable. What is it?

Exercise 3.4 What is the type of 5, the type of 5.0 and the type of "5.0"?

Concept We can find out the type of a data item using the built-in type() function.

3.2. Accessing elements

Once we have a list, we can easily access individual list elements using square brackets and an index after the list name. For example

data[1]

accesses the element number 1 from the list. Let’s try this out.

Exercise 3.5 Remove your numbers list from your program again if it is still there, and go back to working with the temperature data. Print out element number 1 from that list using the following statement:

print(data[1])

Compare this to the elements you see when you printed the whole list. Can you see the individual value you printed in the list? Which element did you print?

Exercise 3.6 Print out the fourth element of the list. Then print out the first element.

Concept We can access individual elements in a list by providing an index in square brackets, in the form list[index].

3.3. Index numbering

When you did the previous exercises, you will have see that printing data[1] does not print the first element of the list, but rather the second. This is because element numbering in lists starts with 0. We call the position of a list element its index, and Figure 3.1 shows a list with the assigned index numbers. In this diagram, we can see that we need to print out the element at index zero if we want to see the first element.

Figure 3.1: Index numbering in a list

Exercise 3.7 In a list with 32 elements, what is the index number of the last element?

Exercise 3.8 In a list of length n, what is the index number of the last element?

Concept Index numbers in lists start with index 0.

3.4. Element types

We can now start writing code to do some analysis of the temperature data. We might, for example, like to know what the lowest temperature of the year was, or the average temperature in July.

Once we start doing this, we will notice a small problem. To illustrate this problem, let us start by doing a very simple exercise.

Exercise 3.9 Add the first and the second element of the list together and print out the result. What do you see? Is this what you expected?

If you have done this exercise, and if you have written data[0] + data[1] to add the two values, you will have noticed that the output is not what we would expect.

The two first values in the list are 9.6 and 3.1, so we would expect to see a result of 12.7. This is not what we saw. So what is going on here?

The reason for the unexpected result is that the elements in the list are not numbers, but strings. So the first element is not the number 9.6, but the string "9.6". We get a clue to this fact from the output: if you pay close attention, you notice that the values in the printed-out list are all enclosed in quotes: '9.6' instead of 9.6. Remember that strings can be written with double quotes or single quotes – so this tells us that the elements are strings.

The + operator, which is performing addition when the operands are numbers, actually performs string concatenation if the operands are strings. String concatenation just glues two strings together. For example, the result of

"Cat" + "Fish"

"CatFish"

(Note that Python does not automatically add a space when concatenating strings.)

The reason that our data is represented in strings is that it was stored in a textfile. The content of a textfile, when we read it, is always delivered to us as strings when we read a file. In our case, we know that the file content is a set of numbers, but to get it into a number format in our program, we first have to convert the strings back into numbers.

Concept String concatenation joins two strings together into a single string. It is written with a + sign: string1 + string2.

Exercise 3.10 The type() function which we have seen above can also be used to check the type of the individual list elements. Try this by printing the type of the first element in the list. What is it?

3.5. Type conversion

When we see the values, 5, 5.0 or "5", we tend to read them all as "five". In Python, these values represent an int, a float and a string, respectively, and they are internally stored in completely different ways.

We have also seen that performing operations on these different types can do entirely different things. For example, 5+5 is 10, but "5"+"5" is "55". On the other hand, "5"-"5" will produce an error (since the minus operator is not defined for strings), but this is a perfectly valid thing to do with ints or floats.

In short: the fact is that the number 5 is not the same as the string "5".

We can, however, convert values of one type to another type. We do this by using functions that have the same name as the type we wish to convert to. For example:

number_string = "13.7"
number = float(number_string)

In this example, the str type is converted to a float, and the type of number will be float. Equally, we can use the functions int(n) and str(n) to convert values to the int or str types. We have to take care, though: if we try to convert a string to a number, and the string does not actually contain a number, we will get an error.

Concept Values can be converted from one type to another by using built-in type conversion functions. The most commonly used conversion functions are int(n), float(n) and str(n).

Exercise 3.11 Write some code to convert the string "001.00300" to a float and print the result. What is the value?

Exercise 3.12 What happens when you convert 3 to a float and print it? What happens when you convert 3.8 to an int? What are the resulting values?

3.6. Building a new list

The last two sections serve to illustrate just one important insight: to use our list of temperature data properly, we need to convert our list of strings to a list of floats.

We do this by building a new list. We iterate (loop) over the original list, read every element in turn, convert it to a float, and add it to our new list of floats. The code to do this looks like this:

temp = []
for elem in data:
    temp.append(float(elem))

In this code snipped we do several things:

We create a new, empty list by using the square brackets: [ ]. This list will initially have no elements in it, but it is still a valid list. We assign it to the variable temp (short for temperature).
We iterate through the data list. This takes each list element in turn, converts it to a float, and appends it to the temp list.

We can see that list objects have a method called append(n). This method adds its parameter to the end of the list it is called on.

At the end of this, we expect the temp list to have the same number of elements as the data list, but holding floats instead of strings. Check this by checking the length of each list. Remember, we have seen in the previous chapter that we can use the len(list) function to get the length of a list.

Exercise 3.13 Implement the conversion of the list of strings to a list of floats, as shown above. To test whether this was successful, print out the temp list. Can you tell whether the elements are now numbers?

Exercise 3.14 Print out the length of both lists. Are they the same? How many elements are in each list?

Exercise 3.15 If you prefer to work with degrees Fahrenheit, rather than Celsius, you can do the following: Convert the temperature list to Fahrenheit. Do this by building a new list, where each element is calculated from the existing list. The formula to calculate degrees in Fahrenheit (F) from Celsius (C) is F = (C × 9/5) + 32.

Concept We can create an empty list by writing just a pair of square brackets: [ ].

Concept The list method append(n) adds an element to the end of the list it is called on. It is called in the format list.append(elem).

Now that we have a list of numbers, rather than a list of strings, we can start looking at values in a more meaningful way.

3.7. Finding elements: Hottest temperature of the year

The first thing we might ask about the London temperatures is: How hot was the hottest day of the year? How cold was the coldest?

We can find the answer to both of these questions quite easily now: Python has built-in min(list) and max(list) functions, which find the minimum and maximum values in a list.

Exercise 3.16 Print out the temperature of the coldest day of the year. Print out the temperature of the hottest day of the year. Make sure you print some text as well to state what the values are.

Note that the data is the average temperature for each day, with the first data point the average for the 1^st of January, and the last the average for 31^st of December. So the minimum we see here is not the coldest temperature recorded in the year, but the coldest daily average.

Concept The min(list) and max(list) functions can be used to find the minimum or maximum value in a list.

3.8. Finding the index of an element: When was the hottest day?

We can now see what the minimum and maximum temperatures were – but on which day did they happen?

To answer this question, we need to know where in our list the maximum value was found. We can find this out using the index method. For example

temp.index(13.6)

will return the index of the value 13.6 in the list temp. (We have to be careful with this: If the value we are searching for is not found in the list at all, then this will be an error. So we should call the index method only if we know the element exists.)

We can now find the index of the hottest day by writing

max_index = temp.index(max(temp))

This statement first finds the maximum value, and then finds the index of that value.

Exercise 3.17 Write code to find the index number of the hottest and coldest days. Print out the results. Make sure to print your output with information about what the number means.

Currently, we only get the index of the day in our list, rather than a properly formatted date. The index identifies the day (we know that index 0 is the 1^st of January, and index 364 is the 31^st of December, and we could count out the days inbetween), but it is not a convenient format.

For now, we will leave it as it is – just printing out the day number is a decent start. We will look a little later, further down in this chapter, at how to convert this to a date.

Another limitation worth mentioning is that the index method only finds the first occurrence of a value in a list. So if the maximum temperature was reached twice, at exactly the same value, we will find only the first of these days.

Concept The list.index(n) method can be used to find the index of the first occurrence of element n in the list. The element must exist in the list when this is called.

Note Functions versus methods: Python makes use of both functions and methods in the functionality it provides for lists. This can be a bit confusing.

Functions are called in the format

function_name(list)

while methods are called in the format

list.method_name(param)

For list functions, we pass the list into the function as a parameter, while for list methods, the list is named first, in front of the dot.

The reason is partially historic, and partially pragmatic. Functions are typically used when the operation can also be applied to types other than lists. len(n), for example, can be applied equally to a list and to a string. Methods, on the other hand, belong to the list type.

The most important list functions are:

len(list), min(list), max(list), sum(list), sorted(list)

The most important methods are:

list.append(), list.clear(), list.copy(), list.count(), list.extend(), list.index(), list.pop(), list.remove(), list.reverse(), list.sort()

You can find a description of these functions and methods in Appendix B: Built-in functions.

3.9. Slicing: Looking at individual months

The next thing we may want to do is to analyse specific months.

Taking a specific sub-list out of an original list is called slicing: We are taking a slice out of a list. Slices are written in square brackets, with the beginning and end index specified. For example

jan = data[0:31]

creates a new list with the first 31 values copied from data and assigns it to the variable jan. If we want a slice from the start or to the end of a list, we can just leave off the lower or upper bound. For example

jan = data[:31]

does the same thing: it creates a slice from the start of the list to index 31. If we leave off the end index, then we get a list with all values from the start index to the end of the list:

rest_of_year = data[31:]

Exercise 3.18 Add code to your project to create a temperature list for January and print it out.

Exercise 3.19 Print out the warmest day in January.

It is worth looking very carefully at the indices used in the slicing operation. In the slice above, we specified 0 as the lower boundary, and 31 as the upper boundary. When creating slices, the element at the lower boundary is included, while the element at the upper boundary is excluded. In other words: the new list will hold a copy of the elements 0 to 30 from the original list. These are 31 elements. In any slice [n:m], the value of m-n will tell us the number of elements we will receive.

Exercise 3.20 Print out the warmest day in February. Then the warmest day in March.

Concept We can create a slice of a list by providing two indices separated by a colon: list[start:end]. This will create a new list holding the elements from start (inclusive) to end (exclusive). We can leave out the start index (list[:end]) to create a slice starting from the beginning of the list, or leave out the end index (list[start:]) to create a slice that reaches to the end of the list.

When we want to slice the data for later months in the year, determining the beginning and end of the slice becomes quite tedious. Counting or calculating the numbers by hand quickly gets annoying. We will come back to this in the next chapter, where we will implement a solution to deal with this problem.

First, however, let us look at one more list method: sorting.

3.10. Sorting: Finding the 10 coldest days

In the next step, we would like to find the 10 coldest temperatures of the year. There is no built-in function to find the 10 most minimal values, so we will have to think of something else.

We could, of course, write a loop and iterate over the list, look at each element, and try to identify the 10 smallest numbers. There is, however, an easier solution.

Part of the solution is the built-in sort() method of the list type. The statement

temp.sort()

will sort our list, with the smallest value first and the largest at the end. We can also sort the list in descending order by providing a parameter:

temp.sort(reverse=True)

With the knowledge of sorting and slicing, we can now solve our problem: To print out the 10 coldest days, we can sort the list (coldest days first) and then take a slice of the first 10 elements.

Exercise 3.21 Print out the temperatures of the 10 coldest days of the year.

Exercise 3.22 Print out the temperatures of the 10 hottest days of the year.

Did you manage to print the 10 coldest days? If you did – very good. There is, however, a potential problem. To discover what it is, do the following exercise.

Exercise 3.23 Run your program again. Make a note of the day number of the hottest day of the year and/or the hottest temperature in January, which your program should still print out.

Then move the code to print the 10 coldest days up in your program, so that this information is printed first, before the other lines of information. Run your program. Inspect the data for the hottest day and the hottest January day and compare them with the values you had noted before. What do you observe?

With the exercise above, you should have noticed the problem: all the computations after printing the 10 coldest days are wrong. This is because the sort function has sorted the list, and we have now lost the order of the original list. We can no longer use it for our calculations, because it is no longer in its original order.

The solution to this is to take a copy of the temperature list before sorting it, so that we leave the original list intact:

sorted_temp = temp.copy()
sorted_temp.sort()

We can then use the sorted list for the 10 coldest days, while still being able to do further investigations with the original list.

Exercise 3.24 Change the calculations of the 10 coldest days to work with a copy of the original list. Test your program: are the other values shown correct again? Once they are, you can move the printing of the 10 coldest values down in your program again.

Concept The list.sort() method sorts a given list. The list.copy() method creates a copy of a list.

A version of the project with the implementation discussed thus far is in the book projects as temperature-v2.

3.11. Summary

In this chapter we had a good look at lists and some of their operations. We worked mostly with a list of numbers – floats, to be precise – and we have seen that we can find specific elements, find an index of a value, and sort or slice the list.

Many convenient list functions are provided in Python by default, and it will be very useful to become familiar with them.

Concept summary

Python provides a list type, which can store a list of items. Lists are defined using square brackets and values separated by commas: [ 1, 2, 3].
We can find out the type of a data item using the built-in type() function.
We can access individual elements in a list by providing an index in square brackets, in the form list[index].
Index numbers in lists start with index 0.
String concatenation joins two strings together into a single string. It is written with a + sign: string1 + string2.
Values can be converted from one type to another by using built-in type conversion functions. The most commonly used conversion functions are int(n), float(n) and str(n).
We can create an empty list by writing just a pair of square brackets: [ ].
The list method append(n) adds an element to the end of the list it is called on. It is called in the format list.append(elem).
The min(list) and max(list) functions can be used to find the minimum or maximum value in a list.
The list.index(n) method can be used to find the index of the first occurrence of element n in the list. The element must exist in the list when this is called.
We can create a slice of a list by providing two indices separated by a colon: list[start:end]. This will create a new list holding the elements from start (inclusive) to end (exclusive). We can leave out the start index (list[:end]) to create a slice starting from the beginning of the list, or leave out the end index (list[start:]) to create a slice that reaches to the end of the list.
The list.sort() method sorts a given list. The list.copy() method creates a copy of a list.

Project Ideas

Kaggle

In this chapter, we have started to analyse some data, and we have used a London temperature dataset for this. Of course, you can analyse all sorts of other data.

A good source to find interesting data files is Kaggle. It is a website where people share all sorts of interesting data, and you can usually find something that interests you. The available data is very wide-ranging, from most-streamed songs on Spotify, via mental health data, sports statistics, transport data, environmental data to crime data, and much more inbetween.

Have a look, pick something that interests you, and start to analyse.