Exploring a UX-Centered AI Design Process for Creating Successful Human and Machine Dialog Interactions


August 30, 2019

By Ellen Kolstø, M.A. and Jessica Raedler, M.A.

Dialog interface systems (e.g. chatbots, virtual agents) represent the potential to create a smooth and low-effort interaction between machines and humans. When a digital product is well designed, it leaves a positive impression and creates engagement stickiness. The challenge comes from having to design a system that must seize the largest benefits of machine learning—namely automating prediction of large-volume, high-frequency usage patterns for specific subject matter—while creatively engineering solutions proactively designed to accommodate human language and online behaviors. Further, while a dialog interface ultimately involves deliberate user interface (UX) design choices, it should also be subject to fundamental UX principles such as setting expectations, always giving the user an “out,” reducing cognitive load, etc. This indicates a need to fundamentally change how artificial intelligence (AI) teams go about solving problems. Further, it requires different ways of thinking that bring about simplicity in design without dismissing the complexity of the problem.

The AI Design Process

In this paper, we take a closer look at one enterprise’s process for changing the way they work in order to build a thoughtful, user-centered, AI-driven dialog interface system. Here is their process:

1. Creating Alignment Around the Goal

At the start, the business purpose of the dialog interface system had to be clearly set and communicated, along with corresponding requirements. All disciplines involved needed to understand the big picture, be aligned and know how to reach the goal. Design thinking workshops that created focus and consensus were crucial at this time. Teams from upper management, AI engineering, line-of-business, data science, design, content, and marketing worked through current and ideal scenarios and narrowed to a key goal and a roadmap to get there. But who, exactly, was on this team?

2. Building a Team

Enterprise leadership had to facilitate cooperative work between traditional (e.g. developers) and non-traditional (e.g. UX designers, business analysts) roles in the engineering practice that needed to work across the disciplines. As the project progressed, it became clear that:

  • the complexity of the system required that each player needed to be rigorous in their own discipline, but with a multidisciplinary mindset; and
  • traditional job descriptions had to change as domain experts started to practice UX design and even some coding.
Illustration of a data scientist, domain expert, project manager, and developer

Creating enterprise-level dialog interface systems required building a team that had to learn to work continuously together despite their varied backgrounds. This team consisted of:

  • developers, who created early proofs of concept and connected backend systems;
  • domain experts with less technical backgrounds (e.g. customer service, linguistics, UX or market research), who were tasked with designing the dialog to reflect industry verticals and specific subject matter;
  • data scientists, who trained and measured the performance of the model; and
  • product managers, who packaged AI for external users (customers).

In the past, these roles had not worked together as a tightly knit team doing day-to-day tasks.

At the heart of this development team was AI model training. With it came the unlikely paring of data scientists and domain experts designing dialog. This odd-couple combination of data-driven and human-driven roles ensured machine accuracy within the context of human interaction. Let’s take a closer look at these roles:

Illustration of equation Data Scientist plus Domain Expert
Data Scientist + Domain Expert

Data Scientists: Machine learning (ML) and natural language processing (NLP) models built on extensive training data have allowed dialog interfaces to achieve fairly high accuracy in responding to humans without the cost and complexity of rules-based models. Data scientists play a big role in assessing these models, and approach optimization by focusing on the accuracy of the entire system (e.g. evaluating the data set as a whole). They often look at how well the model ”generalizes” by determining how accuracy can be improved across all user intents. (Intents are the purposes or goals expressed in a chatbot user's input, such as the desire to get a question answered or process a bill payment).

Domain Experts (Dialog Designers): Those who design and train the dialog, on the other hand, are concerned with the interface at a more granular level. Much of dialog design is meant to serve multi-turn (e.g. back-and-forth) conversations, giving dialog designers more of a UX role with the task of emulating nuanced human interactions. This dialog designer’s process, therefore, often follows a much more intuitive approach involving observational research. They will perform deep analysis on customer log data of real conversations in order to extract interaction patterns (ranging from issues of misclassification to diverging dialog flows) that aid in creating a better user experience. For example, a dialog designer found through observation that while users did complete a task successfully in a dialog flow, her users also typed the word “log-in” multiple times during the flow, expecting to be able to log in earlier in the conversation. This created frustration even though the interaction ended successfully. The dialog designer redesigned the dialog flow to allow for an earlier login within the interaction to create a better user experience.

3. Working Together

It was crucial for all participants in the design and development of the dialog interface to be open to continuous exchange. Through interdisciplinary discussions, design trade-offs became apparent, leading to collective understanding of the compromises required to achieve the optimal product. A series of regular meetings and touchpoints had to be adhered to in order to keep the dialog interface consistent and up to date based on user needs:

  • weekly meetings with content owners and the dialog designer to ensure that the chatbot reflected current language and information;
  • meetings between the data scientist and dialog designer to assess the accuracy and precision of the virtual agent (where is it getting things wrong and at what rate?);
  • daily scrums with development and the dialog designer to examine the success of the virtual agent’s use of backend databases; and
  • regular check-ins with upper management, marketing and product teams to ensure that the virtual agent reflects any current promotions or changes.
Illustration of a book

Notes from the Field

At first look, building a conversational engagement would seem to be a creative, free-flow type of exercise. To a great extent, there is room to be creative with the wording, the UI look and feel, etc. Ultimately, however, an enterprise-level application has to serve pragmatic business and customer needs. That requires crafting a design based on a clear end-to-end understanding of customer needs, the related business processes to provide the answer, and the information available to facilitate the answer. The most effective dialog flows represent nothing more than well thought thought-through business processes, but they must be worded and broken out in such way that the flow is natural and easy to understand. This requires approaching the design from an information architecture perspective wherein the terminology of the domain is well organized and mapped against the ways in which users may express their needs.

Be selective with the use cases that are chosen for conversational engagement in your dialog interface systems. The ability to recognize user input has become a lot more powerful thanks to advances in NLP. Yet, it cannot recognize everything. Machine learning models are usually good for predicting very specific things. Be clear on the scope of the ML classification, leverage UI elements to set expectations and give the user an “out” when the model cannot understand a given input. Further, not everything requires an online engagement involving back and forth.

Agile or non-agile? Ideally, the dialog design is set up in a way that allows for the ability to quickly capture, analyze, understand and iterate on the design. From that perspective, the agile approach is good in that it accommodates the need for quick iteration, as the type of questions coming in through a given UI may be different based on the audience visiting that site.. However, since the dialog interface is most likely just one touchpoint of a larger online enterprise web ecosystem (FAQ sites, product info, support, etc.), there are many intersystem and contextual dependencies that have to be accounted for.  Worst-case scenario, dialog interfaces may unintentionally introduce more confusion and frustration. It is therefore important to invest enough time and focus in the early problem framing. The agile process enables quick development and deliverables. The initial problem framing, however, should not be done in a rush, as this can increase the risk that people are operating under a different or incomplete understanding of the problem.

Design for failures: An additional benefit of initial problem framing through design thinking is proper understanding of the trade-offs between options. For instance, it is important to choose between designing full dialog (every interaction) versus dialog plus UI elements such as buttons and sliders to solve the problem. The ability to isolate and size the problem allows for proper experimentation and quick understanding of the problem’s root cause. This allows for a design that is experimental, designed to detect failures and learn about their impact so that the proper adjustments can be made to mitigate them.

Ellen Kolstø is a Design Principal working in quantum computing with IBM Design and a lecturer in the School of Design and Creative Technologies.

Jessica Raedler is Senior Conversation Engineering Manager at Autodesk.

Illustrations by Russell Huffman, IBM Design

Read More Stories