Creative Python Programming with Strype Volume 3: Structuring data

As mentioned above, our plan to do a thematic analysis rests on the idea of finding out which words are used unusually often in a given text.

This does not mean that a word is used more often than another word in a given text – that is entirely expected. Some words just occur more frequently than others: words like "the" or "go" are very frequent. To give us some useful information, we need to find out which words are used more often than we would expect them in an average English language text. For example, if the word "crime" is used more often than expected in a book, that tells us it is probably a detective novel. In that way, we can find out what the document is likely talking about.

This means we can achieve our content analysis in these three steps:

We will look at these three steps one at a time. We start at the beginning: counting how often each word is used in a text. This is also known as the frequency of the word.

To start this task, we have prepared a starter project named thematic-analysis-v1, which you can find in the book projects. This project includes only two helper functions which we will use later, but no code to help us with the initial task.

Whenever we aim to implement a solution for a sub-task, it is useful to use functions in our programs to provide the sub-task functionality. So this time, we will start by writing a function to count the frequency of a given word in a text.

If you have reached this stage in the project, you will have seen that we can now count the frequency of a given word in a text. You may have noticed, however, that some problems remain that can result in incorrect counting. If not, you should be able to see the problem with the following exercises.

Why does the test work correctly for the word "say", but not for the word "go"? You have probably worked out that this is because of the uppercase character in one of the "Go" occurrences in the text.

For our purposes, we do not care about the upper- or lowercase nature of the words – we would like to count them in any case. We can easily achieve this by first converting our entire text to lowercase before starting our counting:

The lower() method is a built-in method of the string class that returns a new string with all characters converted to lowercase. Note that it does not modify the original string. Thus, we have to assign the result back to our string if we want to keep it.

With these exercises, you will have discovered two things: First, there are many useful string methods available which can save us a lot of work of writing code ourselves. Knowing what methods are available is important to becoming a good programmer. You do not need to memorise all methods, but you should have a link to the documentation page readily available when you work, so that you can look them up whenever you need them.

And secondly, we can see that a count method is available that is somewhat similar to the count_word function we are currently writing. The reason we are writing our own is two-fold: The built-in count method does not find words, but substrings. Thus, "car", for example, would also be found if the actual word is "scarlet". Also important is our goal to modify and extend this function to count not one word, but all words (and there is no ready-made method for this). So our counting function is a stepping stone for more powerful functionality.

Again, you may have noticed that your function does not count correctly. This time, it is the punctuation characters that are throwing a spanner in the works. When we split the text, we separate the original string at space characters, but all other characters are left in the words. Thus, our list of words contains the word "say," (including the comma), instead of the word "say".

To fix this, we will need to remove the punctuation characters from each word.

What we have done in these exercises is to "clean the data". Cleaning the data for processing means to get all data into a known standard format so that it is properly recognised by our program. (In other data sets, it might also mean to remove invalid or duplicate data, but this is not a problem in our case.)

Our program deals with text, so the data points are words. But the words can appear in different formats and contexts. Consider, for example, the following strings, which may all be produced as a "word" by the string split() function:

Without cleaning, these would all be treated as different words, because they are different strings. We want, however, to recognise all of them as the word "hello", so we "clean" the string to produce the canonical (standard) representation. In our case, we do this by converting to lower case and removing punctuation, so that we are left with the string "hello" in each case. Then we are ready to count the words.

Counting words in single sentences is not very interesting – we could do that by hand. But counting words in long documents is not so easily done; this is where a computer can help. Let us try that out.

Strype includes various text files in its built-in file system. You can see the file system, and the files available within it, by using the "Files" tab in the bottom right of the Strype interface (Figure 1.1).

By browsing the built-in files, you can see that there is, for example, a file named "winnie-the-pooh.txt". This is the full text of A. A. Milne’s children’s book "Winnie The Pooh". In this book is a character called "Piglet". Let us find out how often Piglet is mentioned in the book.

Your project includes a read_file function that can be used to read in the contents of a text file. When you call this function, and you place the text cursor inside a string parameter (between the quotes), you can use auto-completion (Ctrl-Space) to pop up a list and select from all the available files (Figure 1.2).

You can now use this read_file function to read in the text of the Winnie The Pooh book, and then pass the text to your count_word function to count the number of times "piglet" is mentioned in the text.

A project with the functionality discussed so far is available in the book projects as thematic-analysis-v2. If you are unsure about any of your own code, you can compare your project to this one.

Let us pause for a moment and take stock of what we have achieved so far. We can now count the number of occurrences of a given word in any given text. We do this ignoring the case of the letters, and we can ignore punctuation characters. That is a good start. But a major problem remains: We would like to count all the words in a text, not only one given word. We know from previous chapters that we can hold a count in a variable, but the challenge here is that we need a separate count for each different word in the text.

It is not feasible to create a separate variable for each word. There may be thousands of different words, and we do not know in advance how many variables we would need. A list is also not directly suitable because we need to hold two pieces of information, the word and its count, together.

The solution to this problem lies in a new data structure: a dictionary. In the next chapter, we discuss what a dictionary is, how we use it, and how it can help us to solve this problem.