Creative Python Programming with Strype Volume 2: Working with data

The incredible speed of computers can be used to achieve many very different things. In the first volume of this book series, we have seen how it can be used to create graphical games: the computer can draw an image consisting of many, many pixels, then draw it again slightly differently, and then again, so fast that it appears to us as smooth motion. Animation is just an optical illusion: it is a series of still images drawn so quickly that we cannot recognise the individual images.

You will likely have seen some professional computer games, possibly including three-dimensional representations of imaginary worlds and complex interactions of game characters. Those examples show you even more clearly how fast computers are: they compute complicated models of a three-dimensional scene and display these on screen so quickly that we see it as smooth animation.

Ultimately, this effect is based just on the fact that the computer can hold vast amounts of data (thousands of game objects, millions of pixels) and can process them very quickly.

The same is true for any other of the many things that computers can do. You see examples of the amount of data and the computer’s speed when a search engine searches hundreds of millions of webpages in fractions of a second, or when an online map can show you details of any place in the world almost instantaneously.

One specific area which makes use of the memory size and speed of computers is data science. We will now have a closer look at this topic, and especially at a specific sub-field: text analysis.

Data science is a field that is concerned with gaining insights from large amounts of data. It usually combines techniques from statistics, computer science and visualisation to extract meaning out of data sets that are too big to analyse without computers.

The actual data can be many different things: data science can be used to analyse healthcare data to predict disease risks, or it can be used to analyse traffic and transport data to improve planning of public transport systems. Equally, it could be used to analyse financial data to detect online fraud, or environmental data to make climate or weather predictions.

A sub-field of data science is text analysis. This is one of the earliest examples of data science techniques, and it uses text documents as the data to be analysed. This might be used, for example, by historians to classify large amounts of ancient scanned documents, or it could be used by political scientists to analyse public opinion by scanning online comments.

In this chapter, we will start looking at some examples of simple text analysis to get a feel for how this works. Doing this, we will practice some important Python programming concepts and gain a first insight into data science.

The Python string type (usually abbreviated to str in the documentation) stores text strings. Text strings are just sequences of characters of any length. Thus, you can store your name in a string, or much longer pieces of text. In one extreme, a string might be empty (a text string of length zero), or it may hold the entire text of a Harry Potter book.

To get started with text analysis, we essentially have to get proficient with working with strings.

One of the simplest things we can do with the string is to count the number of characters in it. We will do this – and a number of other exercises – using a simple starter project called text-analysis.

We can very easily count the number of characters in a string using Python’s built-in len() function. Calling this function with a string as a parameter is straightforward:

In this statement, we call the len function with a string parameter, and the function returns a number of type int to us. Here, we store that number into a variable.

If we want to format text output in a specific way, we have several options that give us various levels of control over the exact appearance. Assume we want to embed the number of characters in the middle of a sentence. The intended output of our program might be:

We might write:

Here we can see two things: The first is that spaces are also characters. The string "Wolfgang Amadeus Mozart" does indeed consist of 23 characters, not 21, because each of the two spaces is also counted. For a computer, a space is just a character like any other.

The second thing to note is that the print statement accepts multiple parameters, separated by commas, and it will print out each parameter, on the same line.

From this exercise we can see that the parameters to the print functions can be literals (strings or numbers), or they can be the names of variables. We could also write a function call directly into the print function. Let us consider these two lines from the previous code snippet:

If we do not need the number of characters again later, we can achieve the same task without the variable:

In this version, the len() function is called as part of evaluating the parameter list of the print() function, and the value it returns is passed straight to the print() function without assigning it to a variable. This is a perfectly valid alternative achieving the same goal.

One limitation when using this version of the print() function is the automatic insertion of spaces. If you paid close attention, you will have noticed that the print() function automatically added a space between each of its parameter outputs (before and after the number, in our example). What if we wanted our output in a slightly different format:

If it is important to us to have the period after the number, then we now have a problem: Simply writing

does not do the trick, because it inserts a space between the number and the period. We might ignore the problem, because it does not seem terribly important. However, in some cases we care about the exact format of the output, and we want to have full control. For those cases, we can use an alternative mechanism: formatted strings (sometimes also called f-strings by Python programmers). We can write the following:

Note the character f before the string. Adding this prefix, the string becomes a formatted string, and it can now include placeholders in parentheses: { }. Between these parentheses, we can write the name of a variable, and the placeholder will be replaced by the value of the variable. It is also allowed to insert a call to a function (or any other expression) directly in the placeholder, and the resulting value will be inserted:

A formatted string can include any number of placeholders, and the programmer is in full control of the formatting and spacing of the resulting string.

The next thing we might want to do is to access the individual characters in our string one character at a time. This, too, is fairly straightforward. We can use a for-loop:

In this example, we assume that the variable text holds our string. In the for-loop, we declare a variable called char to be used as our loop variable. The for-loop will assign each character of the string to the char variable in turn, and execute the loop body once for each character. Here, we just print out the character, but we could of course add more code in the loop body for more sophisticated processing.

The project with the changes discussed thus far is available in the book projects as text-analysis-v2.

Now that we know how to access and process characters in a string, we can start with some simple analysis. As a first step, we would like to print out information in the following format:

You can make a start towards this with the following exercises.

In the last exercise, you may have encountered a small issue: We would like to print out double-quotes around the actual text. Since we have used double-quotes to identify the string literal itself, we cannot simply write a double quote inside the string. If we tried to use the following string:

our program would fail, because the quotes intended to be within the string end the string itself, and we end up with invalid Python code. There are two possible solutions to this. The first one is to use a different string character. Since Python allows us to enclose strings with either single or double quotes, we can use single quotes to delimit the string if we wish to have double quotes within the string:

The second solution involves using a special character, the backslash: \. The \ character allows us to insert special characters into a string. Writing \" inserts a double-quote character into a string without ending the string:

Using this second technique is useful if our string includes both single and double quotes.

Let us now work on producing the third line: counting the characters that are not spaces.

Counting all characters was very easy, since Python has a function built-in that does the job for us. We only had to call len(text) and we were done. If we want to count the non-space characters, life is not quite so easy: Python does not have a ready-made function for this, so we have to do the counting ourselves.

We can do this by looping through all the characters in the string (we have done this before), looking at each character, and counting all the non-space characters. For the counting, we need to use a variable that we can increment for each non-space character:

In this code, we first initialise a counter variable to 0. We do this so that we can increment this variable for each non-space character that we find, thus counting the characters. Note that we have named the counter variable non_space – it is always good practice to name the variable as precisely as possible to express what we are using it for. This name indicates quite clearly what we are counting with this variable.

We then write a for-loop to go through all the characters of our string, as before. But this time we have an if-statement in the body of the loop: We check whether the character is "not equal to" a space, and if this is true, we increment our variable.

Two things are worth noting here:

We can write similar code to count the punctuation characters. Again, we use a for-loop that iterates through the characters in our text, looking at each character in turn. The main difference is that we now need to check this character against a whole group of other characters.

We can do this by first creating a string that includes all punctuation characters we wish to recognise:

In the condition of the if-statement (inside the loop), we now check whether the current character is contained in our punctuation string:

The in operator checks whether the character is included in the string, and returns True if it is.

One other small improvement we can make is to use a more complete set of punctuation characters. Luckily, we do not need to find out what they are and type them all in ourselves – Python already has a string variable with all punctuation characters in it. This variable is called punctuation and is defined in a separate module called string. We can use it by importing it from the string module:

When you add this import to your program, you should remove your own definition of the punctuation string. Your program will now use the complete set of punctuation as defined by Python.

If we want our program to be able to easily do this analysis for different texts, we can now move our code into a function which we can call repeatedly, with different text as a parameter.

An example of the project after these changes is in the book projects as text-analysis-v3. Compare your own version with this one. You can see that an advantage of moving your analysis code into a function is that it can now be called multiple times without much effort.

We have said above that an advantage of using computers is that they can do tasks very quickly, and with amounts of data that we would not want to process by hand. So let’s try this out: Before we go on to count words and sentences, let us first see whether our program can cope with larger text sizes.

Strype has a read-only file system that provides some data files to use with our projects, including the text of some books. One of them is Pride and Prejudice by Jane Austen. The file is a plain text file, accessible at /books/pride-and-prejudice.txt. (This is the full text of the book.)

The project you have been working with includes a helper function called read_file(filename), which you can use to read a file from the file system. This function returns the content of the file as a string. So we can now easily read in this whole book from our file system, and then pass it to our analyse_text function:

In this chapter, we have concentrated on two areas. Firstly, we have worked with strings. Strings represent text, and when we can work with strings, we can analyse and manipulate text. Text analysis is a first example of data science, of which we will see a bit more in a later chapter.

In working with strings, we have seen how we can access individual characters and how we can format strings to create output that looks exactly as we would like it to look.

Secondly, we have again seen examples of improving the structure of our code. We have used functions to create a program that is modular and readable. This is important to create code of good quality, as it makes it easier for others reading our code to understand it.

At the same time, we have also seen how to use variables to store values, count and make calculations. This will be of good use in many of our programs in future.