A Literary Guide to Natural Language Processing

An exercise in interdisciplinary explication, this section is designed to explain the concepts and processes that underpin natural language processing (NLP) to humanities scholars.

In Youth, J. M. Coetzee’s third fictional autobiography, the narrator comments:

Let’s take one of these supersubtle conversations as the basis of our analysis. Conducting NLP on literary texts is of course notoriously difficult. My purpose here is to set out the general process and logic at play in this activity – both as a starting point for supporting literary critique of NLP more broadly, but also as a means to think through the challenges that literary text poses for NLP and the insights that studying such examples might bring.

As a starting point, I note that I am emphatically not an expert in computational linguistics or NLP, what follows is the result of a couple of years of software coaching, self-learning (drawing on some wonderful online resources and communities), and significant help from tolerant colleagues. What I hope to do here is introduce some of the process and conceptual points at stake for readers unfamiliar with NLP. For those readers with a good working knowledge of this material, I would recommend skipping to How to Read a Chatbot.

I have chosen a passage from the end of James’s novel The Wings of the Dove (1909). While the plot of this novel does turn on financial motivations, it is generally regarded as evincing precisely the supersubtle conversations that Coetzee’s narrator attempts to emulate. The novel as a whole is representative of James’s so-called late style (complex sentences, nuanced dialogue, and famously dictated ²).³ The conclusion to the novel turns on the reader’s interpretation of the dialogue. It is therefore a semantically significant (if not necessarily representative) passage.

This is the extract as it appears in the 1909 Charles Scribner edition, as presented in Project Gutenberg’s HTML file:

In order to conduct NLP on this extract, we need to go through a number of steps. These steps should not necessarily be considered as temporally discrete, but rather as entailing certain types of activity.

Before we begin, I want to flag that we have already made a number of interpretative decisions. Above, I gave you a rationale for my choice of passage. I have selected what I think to be an example of James’s “supersubtle conversations.” I have also selected the edition and the length of the passage, selections that you may or may not agree with. These decisions will affect the analysis that follows.

This stage involves preparing the text, or raw data, so that it is machine-readable. Otherwise known as preprocessing, the activities can include:

• Removing “noise”: Depending on your interpretation, noise might include whitespace, line breaks, and blank lines; text file headers and footers (which I have already removed in the above passage); markup and metadata, etc. We might even use an expansive definition of noise and remove words which we regard as low value in this context, for example prepositions.
  
• Normalizing text: This is again, quite task and goal specific and might entail changing all letters to the same case (in this example from upper to lowercase) in order to avoid the program thinking that, for example, T and t are separate characters. It can also involve the removal of: nonalphanumeric characters and diacritics; italics (which appear frequently in the above) and bold; or contractions (of which James is very fond in our passage). We might even want to remove “stop words,” common vocabulary that could be regarded as carrying less meaning within the data set—in the case of English: prepositions, pronouns, etc.
  
• Tokenization: “the task of cutting a string into identifiable linguistic units that constitute a piece of language data.”⁵ Tokenization produces a list of words and punctuation. This can be done at a word or sentence level. For some texts, tokenization is tricky – for example, raw text data from spoken language might not demarcate word boundaries, or written text might include punctuation that confuses the tokenizer, e.g., the full stops in U.K.
  
• Punctuation removal: Depending upon our analytical aims, at some point we might decide that having tokenized our string into words, we no longer need punctuation.
  
• Stemming or lemmatization: The former involves stripping affixes from words, the latter involves that stemming activity which produces dictionary-recognizable words, or lemmas. These processes are not regularized, and there are numerous different programs for doing them which produce various results.

At the end of step one, the extract might look something like this:

As you can see, the preprocessing stage entailed making numerous decisions and assumptions on the basis of the research question we are trying to answer. It also entailed the loss of a number of elements that the literary scholar might consider vitally important. James’s use of italics to denote inflection, his precise punctuation, tenses, and his frequent use of contractions will likely affect a scholar’s interpretation of the meaning of passage as they conduct close reading. However, it might also suggest new ways of analyzing the text data.

At this stage, we might decide that we want to categorize our tokens in order that we can better manipulate them. In the above example the removal of stop words has resulted in the loss of pronouns, which in the case of this conversation-based passage, has significantly reduced the visibility of the subjects. We have also lost sentence structure and stems. As we move to step two, I will therefore reinsert the stop words and punctuation, reverse the lemmatization, and tokenize at the level of sentence, rather than word, in order to explore different processing possibilities.

In this step we might use:

• Part-of-speech (POS) tagging: This process enables us to process a sequence of words according to their syntactical status; it sets us up to conduct analysis based upon the syntactical features of the text. For example, the first few words will print like this:

('He', 'PRP' [personal pronoun]), ('hesitated—as', 'VBZ' [incorrectly marked as verb, third person singular present thanks to punctuation reading error]), ('if', 'IN' [preposition/subordinating conjunction]), ('there', 'EX' [existential there])

• Chunking/chinking: The former segments and labels multitoken sequences. We can do this in a number of ways: for example, we could chunk the text into noun phrases, proper nouns, or via other regular expressions. Chinking involves the removal of multitoken sequences from a chunk.
  
• Bag-of-words model: Creating such a model involves the casting aside of word order in order to produce a count of all the word tokens in the text, for example:

he 6

hesitated 1

as 6

...

stupendous 2

love 5

she 15

This step opens up new ways of approaching the text. In applying the first two processes, it is now much easier to examine certain aspects of James’s lexical choice and phrase formation. We can also conduct statistical comparison of syntactical features of this passage against other texts – whether James’s earlier novels; modernist texts often characterized for their supposedly innovative and spare prose; that other conversational novelist, Edith Wharton; or indeed Coetzee’s writing in Youth.

If we are using a bag-of-words model that disregards word order entirely, we can use the table to examine, for example, word frequency. The table enables us to render word frequency numerically and then easily compare this data against other linguistic corpora. We might note in the above, the frequency of the female pronoun in comparison to the male – indicative of the syntactically confusing, but thematically significant, conflation of Kate and Milly in this passage. Alternatively, we might choose to compare Henry James’s representation of conversation with the spoken dialogue contained within the British National Corpus (BNC), or another similar database. Such a comparison could potentially enable us to develop a quantitative definition of the “supersubtle” features that mark James’s conversation.

The bag-of-words model is also commonly used to train machine learning algorithms and forms the basis of Baysian email spam filters which compare the frequency of various words in the email (e.g., viagra) with a comparison corpora (incidentally, this is why nineteenth-century novel data sets – freely available and rarely containing spam filter key word frequencies – were often included in “litspam” emails in the 1990s by spammers trying to circumvent this issue⁶). Henry James as spam.

To (I hope) state the obvious: although we have now prepared the text, we still need to analyze the data and interpret findings (as I began to do with the pronouns above). The decisions entailed in stages one and two will likely be determined by the data, the aims of the researcher, and the kind of question she is attempting to answer. The number of these decisions, and – for literary scholars – the degree to which they can seem to disregard or lose aspects of the text we consider fundamental to analysis, should indicate that NLP is itself an interpretative process. The rendering of text into data (numerical or otherwise) entails translation and modeling.

Crucially, our analysis here is further complicated by our interest in conversation. This passage has more than one speaker; it contains within it a representation of dialogue. We have not, in the above, captured that. We need therefore to incorporate a model of conversation into our text processing. We might want to distinguish between dialogic utterances and surrounding descriptive comment (“Densher said,” “With which Kate slowly rose.”). We could tag lines of dialogue according to their speakers, conducting sentiment analysis on each character’s speech separately. We might, alternatively, want to analyze each line of dialogue as a response to a specific input (turn-taking). So much of the power of this passage comes from the verbal echoes and the shifting semantics created through repetitions: “As we were?” / “As we were.” Our processing might therefore want to flag this modeling when analyzing the passage.

Notably, in this passage, there is a significant lack of speech tags, and in the second half of the extract we don’t even have names, only pronouns. The reader must therefore follow the order of the passage closely so as to keep track of who is speaking. The effect of this is to hugely increase the mental burden on the reader, adding to the intensity of the passage. Capturing these features in our NLP analysis requires modeling of state change or context. In a chatbot, for example, we need to incorporate both short-term memory and context switching functions in order that our chatbot can respond to implicit contexts (e.g., the temporal references to “now” in the passage above) and also reset as the implicit contexts change. The mental burden on the reader in the passage is the result of having to maintain and repeatedly update their short-term memory across numerous lines of dialogue. Easy for a computer, slightly harder for a human reader.

Modeling conversation as part of our NLP of this passage, although posing lots of questions, is relatively straightforward. This is thanks to a history of intellectual cross-fertilization between NLP and linguistic analysis of conversation. Since the 1980s in particular, human-computer interaction (HCI), which studies the design of computer technologies and the ways in which humans interact with them, has drawn heavily on the work of scholars working in conversation analysis and discourse analysis (and vice versa). Such work on interactive communication has been crucial for the development of todays’ chatbots and NLP in general.

By contrast, in literary studies, conversation has often taken a backseat in scholars’ analysis (Bakhtin’s dialogic imagination notwithstanding). While a written representation of conversation, rather than its oral (or computer-mediated) embodiment, literary texts offer models of dialogue that can be instructive when examining our cultural assumptions around what literature should do in the world. They deserve their own formal analysis, and NLP can offer important insights.

Much of the pushback against literary computing in recent years has focused on the shortcomings of such models and, indeed, the logic that an activity that entails lengthy preprocessing might usefully supplant close reading.⁷ I would instead simply argue, rather than being put off by an illusion of objectivity, neutrality, or quantitation, literary and cultural scholars might embrace the critique and interpretation that such a process should properly entail. We need not abandon traditional methods of reading within the discipline, but in a world in which NLP is ever expanding, our expertise is valuable and sorely needed.

Where to next?

How to read a chatbot

Natural Language Processing

Chatbots

Literature, Talk & Computing

Talk Interfaces