Creative Python Programming with Strype Volume 3: Structuring data

What we have called our "model" is a simple list of bigrams: a list of words that can follow another given word. We have used this to produce our sequence of words. This looks a little bit like English, but the word sequence is frequently odd and a bit too random. What, for example, should be the next word if the previous word was "your"? Could you predict a reasonable next word?

It is obvious that the possibilities are very wide, and you are unlikely to come up with a word that makes much sense in any specific context.

But how about predicting the next word if the previous words were "How is your"? Now the options for a reasonable next word are much narrower, and you can probably come up with a few choices that could make sense ("How is your mother?", "How is your headache?", etc.)

The idea we get from this observation is quite straightforward: Maybe we can improve things if we look at longer sequences of words.

Let us start investigating this idea in a simple form at first: instead of looking at bigrams – pairs of words – we look at trigrams: sequences of three words. Consider, for example, the text from Figure 5.11:

The trigrams from this text are all the three-word sequences that occur in the text. We can write them down in a list; they are:

Once we have a list of trigrams, we can produce new text with a similar idea as before: we predict the next word. This time, however, we do not only look at the preceding word – we look at the two preceding words. Given the two previous words in a text, we then find the trigrams where the first two words match the two preceding words, and we select the third word from one of the matching trigrams as the next word in our generated text.

In Python, each of the trigrams is stored as a tuple of length 3, and all the trigrams – as before – in a list. Generating the trigrams is quite straightforward, since the built-in zip function can also merge three lists together (or, in fact, any number of lists).

As before, we have to generate lists with the words shifted by one word, and we do so by slicing off the first word for the first copy, and the first two words for the second copy (Figure 5.12).

In our program, generating the trigrams then looks like this:

Change your entire program to work with trigrams now, using the following list of exercises.

If you find these exercises difficult, you can compare your attempt to our implementation in text-generation-v3.

If you have done the exercises so far – how is your output now?

You will see that this simple improvement of our "model" – going from bigrams to trigrams – significantly improves the quality of the text it produces. Now the output looks almost like grammatically correct English! That is quite astonishing for such a simple algorithm.

Of course, the sentences are still gibberish – they don’t mean anything, and you cannot read any sense into them. We discuss below how Large Language Models (LLMs) manage to do better than this.

We can now, of course, go even further with the same idea: instead of using three-word sequences, how about four-word sequences? Or five-word sequences? This idea takes us from trigrams to n-grams, and we will discuss it below. But first, let us look at another idea to improve our system: improving our training data.

There is a very simple change that we can make to improve the quality of our system: use more data.

In real LLMs, the models are trained on millions or billions of documents. Many of them are fed any text you can access on the Internet, and much more besides.

We will not use that amount of text, but we have two files available that contain just a bit more than a single book. The Strype file system folder includes a file called /books/fairy-tales.txt, which holds the full text of 65 fairy tales by the Brothers Grimm, and another file called /books/sherlock-holmes.txt, which has the text of 27 Sherlock Holmes stories (three books and 24 short stories). Try your program with these texts as the training data.

A version of the project with these changes is in the book projects as text-generation-v3.

If you did these experiments yourself, you will have noticed two things: Firstly, using longer texts to "train our model" improves the quality of the output. The more text we feed in, the more closely the output resembles actual language. We have not, in fact, used very much data. Even the "sherlock-holmes" file contains only about 330,000 words. This is minuscule in the context of large language models, but it already starts to deliver quite surprising results.

And secondly, the content of the text we feed into the model determines the language and style of the text that is produced.

These two observations lead to two very significant issues: Firstly, with the aim to be able to produce any kind of output, large AI companies have been scanning all writing they could get access to. This includes, for example, millions of published books. For the performance of the system, this is great: you can tell your AI system to write a short story in the style of Stephen King, and it will very competently do this. It has read in all existing Stephen King stories, and so it can copy the style amazingly well.

The problem is that these stories are under copyright, the AI companies have not legally licensed them, and Stephen King has not explicitly given his permission.

So, is this legal?

Currently, this is an undecided question. The AI companies argue that it falls under the "Fair use" clause in copyright law, since the source works are not reproduced directly. Many authors argue that this is copyright theft, since their own original hard work is being used – and being commercially exploited for profit – by other companies without their permission.

The second problem is the sheer amount of data involved, and the amount of processing power and electricity required to process it. Reading and transforming the "sherlock-holmes" file is trivial, and a standard laptop can easily do it. But processing the amount of data involved in real LLMs – more text than the content of all the libraries in the world combined – requires so much processing power that the environmental impact is immense. Power is not only needed to train the model (this is only done infrequently), but for every single question that an AI system is asked to answer. Even the most trivial query requires huge processing power and consumes large amounts of energy.

To cope with this, AI companies are building huge data centres in many areas of the world, full of processors which only serve AI queries from users. A single data centre can consume as much energy as a city of a million people. For the environment, this is a disaster; it creates significant new problems for the goal of reducing the environmental damage which we cause.

One other interesting observation is this: Our algorithm needs absolutely no understanding of the words it produces. If you feed in a text in Spanish, it will happily produce Spanish output. Or any other language written in our alphabet. (Try it!) The outcome is quite impressive, but it shows that the system has no understanding of the text at all – it just shuffles words around.

We already mentioned the next obvious improvement above: using even longer sequences of words to match before selecting the next word. But instead of making a version now for sets of four or five words, let us generalise our algorithm: we can write a function that can produce n-grams (groups of n words) for any given n.

We are getting into quite advanced territory now. Let us start, this time, by analysing and studying a solution to this problem.

Let us go through the program and investigate the interesting aspects more closely.

We can see that the make_ngrams function now has two parameters: the source text (text) and the n to specify the length of our n-grams:

To create our n-grams, we now need n lists, each shifted by one word. We do this by creating a list of lists, and adding the individual lists to it:

Think through this code carefully: You will be able to see that it creates n separate lists, each with a different number of words cut off at the start by slicing, and then adds these lists to another list of lists (word_lists).

The next line contains a construct we have not seen before:

Here, we have a list, preceded by a single asterisk (*). The * operator, when written as a prefix before a list, is the list unpacking operator: It takes all the elements of the given list and presents them as separate parameters. For example:

is the same as writing

The elements of the list are unpacked into separate parameters, and then passed to the print function. In the same manner, the statement

is the same as writing

if the length of the list is 4. The power of this operator is that it works for any length of a list. We could not write the fully written out alternative without knowing in advance how long the list is.

In the generate_next_word function, the most interesting statement is as follows:

Here, we make use of the fact that we can use a negative index both for slicing, and for accessing elements in a tuple. If we have a tuple

and then we create a slice t[:-1], we are slicing from the beginning of the tuple to one position before the end (-1). The resulting tuple will be

Similarly, we make use of the fact that we can write t[-1] to refer to the last element in a tuple. Taken together, the if-statement above checks whether the n-1 first words of the n-gram match our previous words, and if they do, append the last word of the ngram to our results list.

Study this code until you are comfortable with it. Don’t be surprised if this looks difficult or confusing at first – this is getting into more advanced areas than we have seen before, and it is not surprising if understanding this is not easy at first.

Another few observations are worth making about this version of our project:

You will see that the produced text now looks pretty good, even though we are still using quite a simple algorithm, which we can write in a few dozen lines of code. Before we wrap up this chapter, we will look at two more internal improvements we can make to our program.

In this section, we will not make any improvements to the functionality of the program. Instead, we will just write the same functionality using different Python constructs.

If you like, you can safely skip this section. If you are mainly interested in learning about the principles of programming, then the solution we have written so far is perfectly good; we have used lists, loops and if-statements to build and process our data structures, and this is how we could do the same in many different programming languages. It gives us a good understanding how this algorithm works.

If, however, you are interested in the specific details of Python, then read on. Python has some constructs that allow us to write the same code more concisely. This version makes use of list comprehensions, which is an elegant way in Python to achieve the kind of processing we have used here.

The statements most likely to be new to you have a form we have not seen before. For example:

This is called a list comprehension, and it is a construct that can create a new list by filtering and mapping elements from a source list. Let us assume we have a list of numbers:

We can then create a new list that holds all elements from the original list which are greater than 10 by writing the following:

This code is equivalent to writing the following without list comprehensions:

Using list comprehensions has a number of advantages:

We will not discuss this variant in detail here. If you aim at becoming an expert Python programmer, then you should study this construct more closely. You can do this by reading Appendix C: Python list processing, and then studying this version of the project.

One last change that we will briefly present here is another improvement that is not essential, but may be interesting to study. Again, this is an advanced topic, and you can safely skip it if you like.

This version of the project is provided in the program text-generation-v6. You can open and study this version.

The main difference in this version is that we are now using a dictionary, instead of a list, to hold our n-grams.

Previously, we had a list of n-grams, and every time we wanted to find a next word, we would provide a prefix tuple with the previous words, and then search our list for n-grams where the first n-1 elements match our prefix tuple (Figure 5.13). As a result, each new word involved searching our entire n-gram list again.

In our new version, we instead create a dictionary. The keys in the dictionary are the n-1 words of each n-gram, and the value for each key is a list with all the possible following words (Figure 5.14).

Now, finding each next word consists of one dictionary lookup, and then a random selection of the following word from the list of possible words. Since a single dictionary lookup is much faster than searching a whole list, this version performs much better.

Discussing this solution in detail is beyond the scope of the book, and we leave it to the interested reader to study this version. It is included mainly for especially keen or advanced readers – you can safely progress without studying it.

We have seen that our model does not come close to the abilities of current AI systems, and there are many reasons for this.

Firstly, there is room to improve our current system. One idea is again simple: use longer chains of words. We have used n-grams with a small n – no more than 4. If we had much more source text (such as a whole library), we could work with longer n-grams.

Using more contextual words does indeed improve the performance of these systems. And the project that we investigated in this chapter does indeed give a glimpse into the history of text processing.

Until the mid-2010s, n-gram-based systems were considered the state of the art in text processing. They were successfully used in predictive text systems and speech recognition.

After that, text processing systems evolved significantly, and modern LLMs have a number of important differences that make them more effective:

The resulting systems are known as generative pre-trained transformers (GPT). The full details of these are more complex than we can discuss here. But ultimately, they are generalisations of the simple system we have built here. Or in other words: An n-gram model is a special, extremely limited case of what a GPT transformer can represent.

N-gram systems, however, are not obsolete. They continue to be useful in predictive text, spell checking, speech recognition, and in document retrieval systems.

While our project here does not allow us to understand fully how LLMs work, it does give an insight into the core of the idea that underlies LLM-based AI systems. Perhaps most importantly: It is still the case that LLMs have no real understanding of the content and meaning of the text they produce, and are ultimately based on clever manipulation and prediction of language. These systems may state an entirely invented "fact" with the same confidence as an obvious truth.

AI companies are working hard to improve the reliability of LLMs. This tension, however, between our desire of trust and these systems, which ultimately just manipulate language, will remain at the core of the issue of reliability in AI systems for some time.

The implementation of the improved n-gram algorithm in this chapter has given us more practice with tuples and lists. We have seen how we can use these data structures very flexibly to solve a number of very different problems. While doing this, we have also gained an understanding of how text prediction systems can work, such as those used for predictive text in messaging apps.

We have, however, only scratched the surface of the possible uses of lists and other data structures. In the next chapters, we will look at the use of these classes in an entirely different context.