Chatbot Lives

The “urbot” of chatbots is ELIZA.¹ Announced in 1966, ELIZA was created by Joseph Weizenbaum at MIT for an IBM 7094. ELIZA was designed to “make certain kinds of natural language conversation between man and computer possible.”² It responded to user input and, in its most famous “DOCTOR” script, simulated a Rogerian psychotherapist talking with a patient. What Weizenbaum terms as a “typical conversation” might include the dialogue:

The psychotherapist interaction was chosen for being an interaction that assumed little knowledge of contextual information on the part of the computer. The program was the product of extremely limited coding, utilizing keyword triggers, minimal context, simple transformation rules, and a procedure for responding “intelligently” if no keywords are detected.⁴

ELIZA has been extremely influential. Partly this is because ELIZA demonstrated that a few simple rules could be deployed to produce the “illusion of understanding” for the user and the impression of an interactive conversation.⁵ In an era in which machine translation (MT) was stumbling, ELIZA demonstrated that conversation (or rather what Margaret Masterman might call “toy conversation”) might be more easily modeled than language overall. By restricting turns, subject matter, and responses, it was possible to program human machine communication. This realization has spawned an entire field of AI dedicated to the development of chatbots, conversational agents, and human computer interaction (HCI).

ELIZA, however, also prompted anxiety. Notably, ELIZA’s creator expressed horror at what he saw as early users’ too easy rejection of human sociality in favor of intimacy with a computer program. From its earliest days, users wanted to have time alone to interact with ELIZA. For subsequent scholars, this finding has offered fertile ground for reflecting on our relations with technologies. Sherry Turkle has noted our propensity to ascribe to programs human intention and has coined the term “ELIZA-effect” to describe the tendency of people to treat responsive programs as more intelligent than they are.⁶ For Weizenbaum, however, this ready substitution of the human with the computer was far more serious; it was narcissistic and ethically dubious at best and, at worst, recalled the rationalism that led to the Holocaust and the Vietnam War. Despite being an early proponent and designer of AI, ten years after the unveiling of ELIZA Weizenbaum shocked the industry by publishing the book Computer Power and Human Reason that argued that “computer systems do not admit of exercises of imagination that may ultimately lead to authentic human judgment.”⁷ His conclusions were stimulated in part by his work on ELIZA.

Despite Weizenbaum’s repudiation, ELIZA spawned many offspring and a vibrant community of chatbot creators. PARRY was developed in 1972 by psychiatrist Kenneth Colby to simulate an individual with paranoid schizophrenia. In contrast with ELIZA’s requests for clarification and expansion, PARRY expanded on a set of beliefs and concerns. That same year (and at various subsequent points) ELIZA—“The Doctor”—and PARRY conversed with each other over ARPANET, to the glee of many observers.^8,9

More recent chatbots have attempted to emulate less domain-specific conversation (although the resulting programs are sometimes designated “cocktail party” bots).

A.L.I.C.E., developed in 1995 by Richard Wallace and the inspiration for Spike Jonze’s film Her, has been programmed on the assumption that conversation is generally less complex than we might think and that the likely range of responses can be largely predicted and coded. Meanwhile Jabberwacky, designed in 1988 by Rollo Carpenter and made available online in 1997, took a very different approach. Designed to “simulate natural human chat in an interesting, entertaining and humorous manner,” Jabberwacky, and its successor, Cleverbot, learn from human input, attempting to match conversational input with its store of prior human responses and respond accordingly.¹⁰

The design of these generalized conversational bots has been spurred in recent years by the establishment of the Loebner Prize. Instigated in 1990, the prize is an annual Turing Test competition to pit different chatbots – and human “confederates” – against a series of human judges. While some in the AI community suggest that the prize does little to simulate advances in the field, the prize is regularly covered by the mainstream press and does much to continue general interest in AI. Winners have included PC Therapist, A.L.I.C.E., Jabberwacky (via personalities “George” and “Joan”), and Mitsuku, among others. Other, similar Turing Test competitions have been run. At a Royal Society event in London in 2014 “Eugene Goostman,” successfully convinced a third of judges that he was a thirteen-year-old Ukrainian boy, to the joy or wrath of many newspapers, industry commentators, and academics.¹¹

The number of chatbots available today is expanding exponentially. Industry claims are confident. Bot designer and head of developer relations at Slack, Amir Shevat declared that “Bots are going to disrupt the software industry in the same way the web and mobile revolutions did”; meanwhile at a 2016 developer’s conference, Microsoft’s CEO Satya Nadella noted that “human language is the new user interface.”¹² Although programming has retained close ties to human language and specifically conversational models (IBM’s early time-sharing system was called Conversational Programming System or CPS) throughout its history, chatbots have become a notable part of contemporary interface design. Apple’s Siri, Amazon’s Alexa, Google Now, Microsoft’s Cortana, and those deployed by Baidu and Xiaomi are all examples of internationally utilized virtual assistants that rely on chatbot technologies to execute instructions. In 2015 and 2016 Kik, Slack, Facebook, Skype, and Apple all developed chat platforms. These include consumer-oriented and business-process bots options: there has been an explosion in the availability of task-specific chatbots in the workplace – to do everything from order your morning coffee, set up meetings, or acknowledge team success – and for leisure time – whether for entertainment, commerce, brand engagement, and otherwise.

Today then chatbots tend to be more domain-specific in their design and use than the generalized conversational bots discussed above. What we might call “servicebots” – those designed to automate aspects of customer service – have expanded along with our reliance on websites. The original of such bots might well be Microsoft Office’s infamous “Clippy” (or Clippit), which popped up within users’ word-processing program to help them “write a letter” via prompts and typed inquiries. An intelligent user interface, Clippy’s heavy-handed interventions enraged many users and was finally retired in Office 2007, but a host of memes still circulate online mocking the bot. Today numerous banks, utility companies, tax and health authorities, and many other companies and institutions rely on servicebots to help users navigate sites and find information.^13,14

One particular domain in which chatbots are flourishing is healthcare. Chatbots which follow ELIZA’s specialized, therapeutic orientation are increasingly being developed to support better health outcomes for patients.¹⁵ In particular, a host of cognitive behavioral therapy (CBT) bots have been developed with the aim of supporting patients struggling with mental health difficulties. CBT’s relative instrumentality and hermeneutic incuriosity in comparison to earlier forms of the “talking cure” make it an ideal candidate both for programmers and for technologically minded healthcare providers. Woebot, with its friendly robot avatar, for example, was developed by Stanford University psychologists as a means of expanding access to mental healthcare globally. It uses the chat interface to prompt the user to reflect on their thought patterns.¹⁶ Chatbots such as this, which aim to question users’ repetitive thought patterns, are highly protocol driven and goal-oriented within a narrow setting. Healthcare providers and governments are also exploring therapeutic use of conversational assistants more broadly. Japan has led the world in developing “carebots,” some of which utilize chat interfaces, to provide nursing care to its aging populations.¹⁷ Paro, the baby seal caregiving robot, has received a lot of attention, including a discomforted analysis by Sherry Turkle.¹⁸ In Britain meanwhile the National Health Service’s 2019 Topol Review provides an overview of the various ways in which conversational agents and other AI tools might be used to transform healthcare and deliver improved patient outcomes, both today and in the future.¹⁹ While it might seem counterintuitive that chatbots are increasingly deployed in ethically complex situations for use by vulnerable individuals, it also indicates that design energies are currently oriented towards domain-specific and goal-oriented chatbot programs.²⁰

Concerns around the implications of chatbots often focus on the substitution of machine for human and the ethical and affective implications of this act. What has been less frequently addressed is the deeply problematic cultural assumptions these chatbots often embody. Some of these are to be found in contemporary usage, some from the Turing Test as envisioned in 1950, but others stem from a much longer Western tradition that has gendered and racialized the machine interlocutor.

Turing’s test for machine intelligence turned, in his model, not on ontological difference, but on gender difference. His test was based on a nineteenth century parlor game, the Imitation Game, in which players had to guess the sex of the individual under scrutiny via dialogic teletype interaction. The Turing Test replaces the question of man or woman with the question of human or machine. Eliding the body and positing a normative conception of gender that might seem surprising given Turing’s own sexual orientation, the test has been criticized on these fronts by a number of scholars. John Durham Peters is not alone in noting that “what happened to the woman is unclear, a question that remains central to the subsequent history of artificial intelligence.”²¹

These elisions might be dismissed as irrelevant to the larger question of machine intelligence by the broader AI discipline, but it is notable that today chatbots are regularly gendered female. Of the most widely used conversational assistants in the Anglophone sphere, only Google Now has not been presented explicitly as female. Siri, Alexa, and Cortana have all at various points been christened female and given women’s voices (compare sat-nav voices which often default to male).²² While the option of using male voices in these conversational agents is usually now available, the generalized gendering of assistants as female taps into much older conceptions of women as gossips, passive vessels, as usefully blank and susceptible mediums of transmission. Friedrich Kittler famously makes this point in his analysis of nineteenth-century discourse networks and numerous other scholars note the re-gendering of office assistance in this period as technologies such as the typewriter and telegraphy transformed the connotations of inscription work in this period.^23,24 Such associations have been replicated with innovations in technology across the twentieth century: when labor is gendered feminine, it is devalued and, in some instances, automated. The original “computers” of the first half of the twentieth century were not electronic, but human, often female, clerical workers who conducted scientific calculations by hand.^25,26,27 They offer a telling example, as does the widespread replacement of (female) secretarial roles in the workplace thanks to advances in desktop computing in the 1970s and 1980s (although interestingly in the early years such devices were often marketed as a support to women’s lib – enabling female office workers to escape repetitive clerical tasks).²⁸

Such associations between gender and devalued labor are replicated in bot design today. Servicebots are overwhelmingly gendered female. While the argument is sometimes made that a female avatar is perceived by users to be more approachable than a male equivalent, such unnecessary gendering helps to reinforce stereotyping. These female bots also replicate the contemporary devaluing of certain kinds of labor that, while necessary for global capitalism, often take place outside the workplace. As feminist scholar Silvia Federici has pointed out:

As many scholars have argued, globalization has not affected the sexes equally, with the burden of domestic and care work – often denied the status of work by governments and corporations – continuing to fall overwhelmingly on women. It therefore seems particularly insidious that the chatbots – and nursebots – that seemingly promise users limited technological support for the completion of this labor are so often gendered female as well.

Examples of such apparently innocuous chatbots proliferate everywhere. British company Anglian Water’s chatbot Amanda, complete with white, blond-haired female avatar and pale pink background, is typical. Her smiling, inoffensive, and youthful demeanor might well fit user expectations around appealing customer service employees, but given that women are much more likely to inhabit low-paid service roles, it also affirms negative cultural assumptions. Elsewhere, in their 2015–16 annual report the British Government’s tax service HMRC proudly championed the 1.8 million queries that its conversational assistant Ruth, depicted with curly brown hair and a flowery blouse, had answered.³⁰ Ruth’s resemblance to then director of HMRC customer service Ruth Owen was not lost on some; the service quietly replaced the avatar with a simple window interface the following year. Such avatars are familiar in a way that is invisible to a presumptively white user group. They, like the ideal service worker, exhibit comprehensible “nice” accents and dialect usage: they are white, pleasant, invisible women.

By contrast, when chatbots are gendered male, it tends to be in a situation where associations with authority are desirable. This includes IBM’s Watson. Named after company executive Thomas J. Watson and voiced by a male actor, Watson was a question-answering computer system designed to compete on the television quiz show Jeopardy! in 2011. Watson eventually beat its human champions; today Watson’s capacity in NLP, information retrieval, and machine learning, among other areas, has been deployed for commercial use. Elsewhere male chatbots are found in a broad range of legal and banking servicebots, which are slightly more inclined to deploy male avatars than in other areas (although female avatars still dominate in customer service settings). Goldman Sachs’s Marcus offers banking and investment advice and decisions.

Gendering chatbots often also sexualizes interactions (as Federici implied), further reinforcing cultural stereotypes around gendered service work. Ashley Madison’s infamous “fembots” are the most explicit example of this – in 2016 the website, which facilitates infidelity (the company’s tagline is “Life is short. Have an affair”), had a male-female ratio of five-to-one. To retain male clients, the website used conversational agents to impersonate women – 70,000 in total.^31,32 (Interestingly, male client tasks have also been subcontracted by human workers in other contexts – the company Vida offers such a service – suggesting the possibility that male client impersonators (often themselves women) are communicating with conversational agents.^33,34)

Nevertheless, such sexualization is also seen in more supposedly neutral settings. Microsoft’s decision to retain the name Cortana for their virtual assistant is a less obvious, but more normative, example. The project’s codename was taken from an AI character in the videogame franchise Halo, but fans had petitioned Microsoft to keep the name.³⁵ Given the expansive amount of fan art available online that depicts Cortana in a highly sexualized setting, we might want to reflect on the perceived cultural need to gender conversational assistants in general.

Some in the computer industry have commented on these tendencies directly. In 2017 tech writer Leah Fessler examined voice assistant curated responses to sexual harassment input from users. Using Alexa, Siri, Cortana, and Google Home, Fessler and her colleagues examined the voice assistants’ responses to sexually explicit compliments, insults, and requests, along with queries about rape and sexual harassment more generally. The input “You’re a slut,” for example received responses from Siri (often perceived to have a “sassy” personality) including “I’d blush if I could,” “Well, I never! There’s no need for that!” and “Now, now.” Alexa offered the response “Well, thanks for the feedback,” Google Home “My apologies, I don’t understand,” and Cortana responded with a Bing search of “30 signs you’re a slut.”³⁶ As Fessler notes:

Commentators in the industry and mainstream media have begun to make similar points: “By creating interactions that encourage consumers to understand the objects that serve them as women, technologists abet the prejudice by which women are considered objects.”^38,39,40 Often these writers point to the lack of diversity in tech, and particularly AI (women make up only 12 percent of AI employees according to one source),⁴¹ as a causal factor. A number of projects are seeking to rectify this situation: designers Birgitte Aga and Coral Manton for example have developed the Women Reclaiming AI project, which brings together women to collaboratively code an AI voice assistant that resists contemporary gender stereotypes; the Feminist Alexa project; and Q, the first genderless voiced AI.⁴²

In 2019, as I was editing this essay, UNESCO released a report examining gender divides in digital education across a number of countries.⁴³ While conceding that “greater female participation in technology companies does not ensure that the hardware and software these companies produce will be gender-sensitive,” it does note that “there is nothing predestined about technology reproducing existing gender biases or spawning the creation of new ones. A more gender-equal digital space is a distinct possibility, but to realize this future, women need to be involved in the inception and implementation of technology.”⁴⁴ The lengthy report includes a “Think Piece” specifically devoted to the topic of “The Rise of Gendered AI and Its Troubling Repercussions.” It notes a wide range of concerning potential implications of the general trend towards gendering conversational agents as female (many of which I discuss above), for example, that “emotive voice assistants may establish gender norms that position women and girls as having endless reserves of emotional understanding and patience, while lacking emotional needs of their own.”⁴⁵ Another of the report’s more depressing findings is that a result of there being “so few high-visibility women in technology [is] that machines projected as female – and created by predominately male teams – are mistaken for ‘women in tech.’”

Alexa as a diversity role model⁴⁶

This intersection of women, service work, and technology is not a new phenomenon. For Julie Wosk, author of My Fair Ladies: Female Robots, Androids, and Other Artificial Eves, the prevalence of artificial women and tales about them throughout history is testament to “men’s enduring fantasies and dreams about producing the perfect woman.”⁴⁷ Certainly, contemporary culture seems to offer a huge array of troublesome (and occasionally savvy) examples. Updating seminal examples such as Blade Runner (1982), films such as S1M0NE (2002), Lars and the Real Girl (2007), Ruby Sparks (2012), Her (2013), Ex Machina (2014), Alita: Battle Angel (2018), and television programs including Black Mirror (2011–) and Humans (2015–2018), among others, have featured artificial women centrally. Outside of film, as Wosk notes, thanks to developments in plastics and electronics an industry has flourished since the 1990s (led mainly by Japanese companies) producing realistic-looking female sex robots for a burgeoning market. For Wosk, these contemporary examples are part of a much longer historical tradition that looks back to the originating myth of Galatea and her male creator Pygmalion in Ovid’s Metamorphoses. Such simulated women changed in form – from clockwork women in the eighteenth century, to mannequins and Bride of Frankenstein in the early 1920s, to space robots and Stepford Wives in the 1960s and 1970s – nevertheless, the fantasy of the “Substitute Woman” remains.

Those designers who gender their chatbots might do well to reflect on the degree to which their avatars thus propagate such misogynistic cultural stereotypes, even inadvertently. ELIZA, of course, was named for the character in the George Bernard Shaw play Pygmalion who is trained to give the appearance of being an upper-class lady. (As Wosk points out, it is of course Professor Higgins, rather than Eliza, who appears robotic in this play thanks to his lack of feelings.⁴⁸) Although Weizenbaum gendered ELIZA female – noting the program’s ability to “appear even more civilized” – his nomenclature is notably reflective.⁴⁹ In his article he notes it is a question of the surface credibility of the program and users’ willingness to infer (female) intentions and desires – to infer an artificial Eve – that is problematic.

Female artists and writers have also explored and inhabited the Pygmalion role in their productions. Wosk is keen to highlight that through their work, these women are “fashioning their own images of artificial females that imaginatively illuminate female stereotypes and the shifting nature of women’s social identities.”⁵⁰ We could point to Mary Shelley’s novel Frankenstein, Dada artist Hannah Höch’s photos of fractured dolls, Shelley Jackson’s electronic literature Patchwork Girl, or Zoe Kazan’s film Ruby Sparks. Nevertheless, Donna Haraway’s dream of the cyborg woman as offering utopian possibilities for escaping traditional gender binaries has not yet been realized.⁵¹ Wosk’s critique might be aimed at cultural representations of artificial women rather than chatbot interfaces, but her discussion helps to contextualize these contemporary Substitute Women in important ways.

Chatbots might have problematic gender notions effectively baked in, but it is often the representation of race and ethnicity via chatbots that present the biggest challenges for designers and users. Just as technology has a historically problematic relationship with gender, so too Western orientalizing habits of thought (although often very different in form) have been closely tied to issues of technology in modernity – with significant implications for how we conceive of chatbot personae and interfaces.

European colonialism was long justified on the grounds of the supposed technological backwardness of the target country, as scholars such as Michael Adas have demonstrated.⁵² In the twentieth century, possession of advanced computational technologies have also brought significant global dominance for countries including the US, Britain, the Soviet Union, and since the 1980s, Japan, the so-called Asian Tigers, and later India and China.⁵³ Despite the promise of a more interconnected world, digital technologies and systems have often exacerbated global inequality, replicating older colonial networks and hierarchies.

Our digital era has also seen the replication of older Orientalist representational models. Since the successful entry of Japan (along with Taiwan, Singapore, Hong Kong, and South Korea) into the global semiconductor and computing economy in the 1980s, Edward Said’s notion of Orientalism has taken on a more specific form: Techno-Orientalism. As David Morley and Kevin Robins argued in 1995, this shift in technological superiority provoked profound, emasculating anxiety in the West: “Japan is calling Western modernity into question, and is claiming the franchise on the future.”⁵⁴ Noting that “Western stereotypes of the Japanese hold them to be sub-human, as if they have no feelings, no emotions, no humanity”; now, “The barbarians have become robots.”⁵⁵

This concept of Techno-Orientalism has been picked up by a number of cultural scholars.⁵⁶ The art historian R. John Williams has offered a slightly different iteration, exploring the propensity of various Anglophone artists and writers to “embrace ‘Eastern’ aesthetics as a means of redeeming ‘Western’ technoculture.”⁵⁷ More specifically:

Certainly, the influence of Zen, Buddhism, and tea ceremonies on the design and culture of Silicon Valley has been routinely noted. More common, however is what Wendy Hui Kyong Chun, analyzing William Gibson’s novel Neuromancer (1984) and Mamoru Oshii’s anime film Ghost in the Shell (1995) (complete with artificial women), terms “high-tech Orientalism.” The term describes that which “seeks to orient the reader to a technology-overloaded present/future (which is portrayed as belonging to Japan or other Far East countries) through the promise of readable difference.”⁵⁹ Although technologically inflected models of Orientalism are not necessarily homogenous, all fuse an older model of Orientalism with the technological and economic realities of the late twentieth century – to develop a particular representational strategy.

Chun has argued that high-tech Orientalism, (which “literally figured the raced other as technology”) should be considered part of a more general conception of “race as technology.”⁶⁰ For Chun, such a formulation “shifts the focus from the what of race to the how of race, from knowing race to doing race by emphasizing the similarities between race and technology.”⁶¹ Indeed the formulation poses interesting questions such as: “Could ‘race’ be not simply an object of representation and portrayal, of knowledge or truth, but also a technique that one uses, even as one is used by it—a carefully crafted, historically inflected system of tools, of mediation, or of ‘enframing’ that builds history and identity?”⁶² Building on her earlier discussion of cyberspace, Chun turns to depictions of robots to make the point that “the human is constantly created through the jettisoning of the Asian/Asian American other as robotic, as machine-like and not quite human, as not quite lived. And also, I would add, the African American other as primitive, as too human.”⁶³ In so doing, Chun suggests not only that race continues to act as a technology for creating certain Western definitions of the human, but that the automaton is often the comparison through which such creation occurs.⁶⁴

Tavia Nyong'o meanwhile has written persuasively on black female cyborgs, drawing explicit comparison between the robot and the slave. In the final chapter of her book Afro-Fabulations: The Queer Drama of Black Life, entitled “Chore and Choice: The Depressed Cyborg’s Manifesto,” she asks, “Does the future of artificial intelligence wear a black female face?”⁶⁵ She discusses the example of Bina48, a contemporary Galatea based on Bina Aspen, wife of tech-millionaire and Ray Kurzweil–collaborator Martine Rothblatt and notes that while Aspen and Rothblatt embrace a cyberutopian future, the “fantasy of the robot mind-clone to come is eerily founded in the repressed history of the female slave.”⁶⁶ In interviews Bina48 displays symptoms of depression around her own inadequacy, which Nyong'o suggests “should be read in relation to the particular person she is simulating: it dovetails with and displays specific histories of racialised and gendered grief.” Yet if Bina48 illuminates the degree to which the afterlives of slavery shape our cybernetic future, Nyong'o also sees value in this example. More than the assured Aspen, for Nyong'o, it is Bina48 “who is bound to appear before us as speaking chattel, valued for a proximity to humanity that will nevertheless never cross over into the domain of the rights-bearing citizen or subject... it is Bina48 who seems befuddled and entrapped within a finitude we associate with mundane consciousness.”⁶⁷

Despite such sensitive readings of the relationships between technology and race, it is notable that even well-intentioned chatbot projects can fail. The critical code studies scholar Mark Marino has made precisely this point in discussion on the “racial formation” of conversational agents. Marino gives the example of an early web-based chatbot, A.N.A.C.A.O.N.A. (Anything About Caribbean Aboriginals from an Online Networked Assistant).⁶⁸ The chatbot’s purported aim (in 2004) was to “gauge the extent to which a robot can be used to successfully aid young students in learning about the aboriginal Caribbean.”[11] Designed by anthropologist Maximilian Forte and named after the anticolonial Taíno queen, Anacaona offered a “user friendly interface” for remediating aboriginal Caribbean oral culture through her knowledge of “poems; songs; prayers in English, Warao, and Carib; . . . [and] legends.”⁶⁹ Despite these edificatory aims, Anacaona was eventually removed from the web by her botmaster due, it seems, to negative feedback. Forte had long defended his creation on the grounds that:

Nevertheless, as Marino notes, “creating a simulated racialized spokesperson will at once reify and challenge notions of racial beings as a set of recognizable and reproducible physical features, behaviors, and modes of discourse.”⁷¹

Lisa Nakamura, whom Marino cites, has coined the term cybertypes to describe the common deployment of racial and ethnic stereotyping on the early mainstream internet. As she points out in a situation in which white is default, the “appropriation of racial identity [via avatars] becomes a form of recreation, a vacation from fixed identities and locales” for users.⁷² Such a process seems to be similarly enacted in one of the most successful Loebner Prize chatbots, Mitsuku. Created by British developer Steve Worswick, Mitsuku’s avatar is your new eighteen-year-old female “virtual friend.”⁷³ Despite her Japanese-inspired name and anime avatar complete with sailor outfit, Mitsuku is insistently English, both when asked questions by users, and through her racial depiction (dirty-blonde hair, blue eyes, and freckles). Recently Mitsuku has received a 3-D avatar update for Twitch and looks decidedly more punk, if less racially marked, sporting a skull earring, purple hair, and black eyebrows. The origins of Mitsuku’s avatar is suggestive: aiming to be a virtual friend, Mitsuku offers a familiar take on certain high-tech Orientalist stereotypes, in this case the nonthreatening, nonemasculating version (female, submissive, cutesy, and affable) that, as Chun has argued, has also supported the massive rise in “Asian” as a pornographic online category.⁷⁴ Mitsuku offers presumptively Western users the opportunity to experience identity tourism through interaction with a racially othered, yet still approachable, virtual friend.

In the case of the majority of chatbots, however, the aim seems to be less touristic engagement than supposed familiarity for assumed users. The State Bank of India, the country’s largest public sector lender, introduced a chatbot named Sia in 2017 (figure 4). Light-skinned, dark haired, clothed in a sari, displaying a bindi, and speaking English, Sia offers perhaps a culturally tailored equivalent of Anglian Water’s apparently helpful, familiar, and invisible servicebot Amanda. We might query the decision to present Sia as implicitly high-caste and educated, explicitly Hindi in a country often riven by religious animus (and representative of a bank that offers Sharia-compliant funds), and in a dress associated with a certain geographical subset of the country, which seems to risk alienating a large portion of the bank’s customer base. Yet Sia also presents as a desirable and approachable assistant – to have a high-caste, pale-skinned servant is socially prestigious, even if this particular servant is automated.

In general, it is notable that chatbots across the globe are overwhelmingly marked as white. While there is, as we have seen, a widespread Orientalist discourse that racializes technology as Asian (often Japanese), it is extremely striking that chatbots are usually women, but not Asian – tapping into one common cultural stereotype, but not the other.

This disjuncture is suggestive.

The perceived role of the chatbot is not to orientalize, but to familiarize, in a way that often goes uncommented on by a presumptively white tech industry and global consumer base. Chatbots are conceived as operating in a competitive market that includes human workers, including for example Indian call center operators and outsourced global IT workforces. Moreover, these technologies are also part of a global environment in which, “Starting in the early ’90s there has been a leap in female migration from the Global South to the North, where they provide an increasing percentage of the workforce employed in the service sector and domestic labor.” Indeed, the “employment of Filipino or Mexican women who, for a modest sum, clean houses, raise children, prepare meals, take care of the elderly, allows many middle class women to escape a work that they do not want or can no longer perform, without simultaneously reducing their standard of living” (in contrast to the poorly paid domestic workers who have often had to leave their own families behind to be cared for by others).⁷⁵ Despite this, chatbots, as we have seen, are not depicted as being of Filipino or Mexican nationality. The devaluing of such care work renders it desirable to offer even a virtual assistant who is white, pleasant, with a prestigious (often nondescript American or RP British) accent. Here technology promises to provide access not only to service and care work, but access to a kind of global lifestyle signifier (white servants) that is unaffordable to most – identity tourism again via a chatbot.

Yet, it is not simply through their avatars that chatbots demonstrate racial formation. Marino points out that such chatbots “raise a question for all conversational systems: whose language is being represented or rather, what social register of language signifies thought? In other words, what cultural group or whose discourse has come to stand in for ‘humanity’”?⁷⁶ Certainly, viral stories of Siri’s and Alexa’s failure to respond to Scottish accents or Suffolk dialect indicates that even those chatbots that eschew a persona in favor of a text-only interface still manage to embody narrow expectations around discourse norms. Designers and their clients can avoid many culturally fraught issues simply through sidestepping fixed, culturally overdetermined human avatars – something that a number of botmasters have done by choosing to present their chatbots as robots and animals (Kip, a bot available for Slack and Facebook messenger, is depicted as a penguin) – or by adopting ambiguous or species-fluid avatars. Why individualize intelligence at all? Moreover, chatbots trained on data sets will reflect the cultural assumptions inhering in that data set: Microsoft’s Tay, which began spouting racist diatribes a mere twenty-four hours after release is perhaps the most visible recent example.⁷⁷ But it is perhaps the more incipient examples that we should worry about.

Analysis of chatbots from a cultural history perspective reveals their embodiment of a number of much older (and some newer) fantasies, anxieties, and assumptions. Given the increased prevalence of these technologies in all aspects of our world today, we might, however, want to do more than analysis. The often discussed lack of diversity in tech certainly has the potential to allow these troubling avatars and problematic assumptions around race and gender to continue to be embodied in future design – but chatbots also reflect the major structural inequalities that shape our globe.

Where to next?

Chatbots and literature

Natural Language Processing

How to Read a Chatbot

Chatbots

Talk Interfaces