How AI Textual content Technology Fashions Are Reshaping Buyer Assist at Airbnb | by Gavin Li | The Airbnb Tech Weblog | Nov, 2022

Leveraging textual content era fashions to construct more practical, scalable buyer assist merchandise.

Gavin Li, Mia Zhao and Zhenyu Zhao

One of many fastest-growing areas in fashionable Synthetic Intelligence (AI) is AI text generation models. Because the title suggests, these fashions generate pure language. Beforehand, most industrial pure language processing (NLP) fashions had been classifiers, or what could be known as discriminative fashions in machine studying (ML) literature. Nonetheless, lately, generative fashions primarily based on large-scale language fashions are quickly gaining traction and essentially altering how ML issues are formulated. Generative fashions can now get hold of some area information by large-scale pre-training after which produce high-quality textual content — as an example answering questions or paraphrasing a bit of content material.

At Airbnb, we’ve closely invested in AI textual content era fashions in our neighborhood assist (CS) merchandise, which has enabled many new capabilities and use instances. This text will focus on three of those use instances intimately. Nonetheless, first let’s discuss among the helpful traits of textual content era fashions that make it a very good match for our merchandise.

Making use of AI fashions in large-scale industrial functions like Airbnb buyer assist just isn’t a simple problem. Actual-life functions have many long-tail nook instances, may be laborious to scale, and infrequently change into pricey to label the coaching information. There are a number of traits of textual content era fashions that handle these challenges and make this feature notably worthwhile.

The primary engaging trait is the aptitude to encode area information into the language fashions. As illustrated by Petroni et al. (2019), we will encode area information by large-scale pre-training and switch studying. In conventional ML paradigms, enter issues rather a lot. The mannequin is only a transformation perform from the enter to the output. The mannequin coaching focuses primarily on making ready enter, function engineering, and coaching labels. Whereas for generative fashions, the secret’s the information encoding. How properly we will design the pre-training and coaching to encode high-quality information into the mannequin — and the way properly we design prompts to induce this data — is much extra important. This essentially modifications how we resolve conventional issues like classifications, rankings, candidate generations, and so on.

Over the previous a number of years, we have now collected huge quantities of data of our human brokers providing assist to our company and hosts at Airbnb. We’ve then used this information to design large-scale pre-training and coaching to encode information about fixing customers’ journey issues. At inference time, we’ve designed immediate enter to generate solutions primarily based instantly on the encoded human information. This method produced considerably higher outcomes in comparison with conventional classification paradigms. A/B testing confirmed vital enterprise metric enchancment in addition to considerably higher consumer expertise.

The second trait of the textual content era mannequin we’ve discovered engaging is its “unsupervised” nature. Massive-scale industrial use instances like Airbnb typically have massive quantities of consumer information. Tips on how to mine useful info and information to coach fashions turns into a problem. First, labeling massive quantities of information by human effort could be very pricey, considerably limiting the coaching information scale we might use. Second, designing good labeling pointers and a complete label taxonomy of consumer points and intents is difficult as a result of real-life issues typically have long-tail distribution and many nuanced nook instances. It doesn’t scale to depend on human effort to exhaust all of the doable consumer intent definitions.

The unsupervised nature of the textual content era mannequin permits us to coach fashions with out largely labeling the information. Within the pre-training, with the intention to learn to predict the goal labels, the mannequin is compelled to first acquire a sure understanding about the issue taxonomy. Primarily the mannequin is performing some information labeling design for us internally and implicitly. This solves the scalability points on the subject of intent taxonomy design and value of labeling, and subsequently opens up many new alternatives. We’ll see some examples of this once we dive into use instances later on this submit.

Lastly, textual content era fashions transcend the normal boundaries of ML downside formulations Over the previous few years, researchers have realized that the additional dense layers in autoencoding fashions could also be unnatural, counterproductive, and restrictive. In reality, all the typical machine studying duties and downside formulations may be considered as completely different manifestations of the one, unifying downside of language modeling. A classification may be formatted as a kind of language mannequin the place the output textual content is the literal string illustration of the lessons.

In an effort to make the language mannequin unification efficient, a brand new however important position is launched: the immediate. A immediate is a brief piece of textual instruction that informs the mannequin of the duty at hand and units the expectation for what the format and content material of the output needs to be. Together with the immediate, further pure language annotations, or hints, are additionally extremely helpful in additional contextualizing the ML downside as a language era activity. The incorporation of prompts has been demonstrated to considerably enhance the standard of language fashions on a wide range of duties. The determine beneath illustrates the anatomy of a high-quality enter textual content for common generative modeling.

Now, let’s dive into a number of ways in which textual content era fashions have been utilized inside Airbnb’s Neighborhood Assist merchandise. We’ll discover three use instances — content material suggestion, real-time agent help, and chatbot paraphrasing.

Our content material suggestion workflow, powering each Airbnb’s Assist Middle search and the assist content material suggestion in our Helpbot, makes use of pointwise rating to find out the order of the paperwork customers obtain, as proven in Determine 2.1. This pointwise ranker takes the textual illustration of two items of enter — the present consumer’s difficulty description and the candidate doc, within the type of its title, abstract, and key phrases. It then computes a relevance rating between the outline and the doc, which is used for rating. Previous to 2022, this pointwise ranker had been carried out utilizing the XLMRoBERTa, nonetheless we’ll see shortly why we’ve switched to the MT5 mannequin.

Following the design choice to introduce prompts, we remodeled the basic binary classification downside right into a prompt-based language era downside. The enter remains to be derived from each the difficulty description and the candidate doc’s textual illustration. Nonetheless, we contextualize the enter by prepending a immediate to the outline that informs the mannequin that we anticipate a binary reply, both “Sure” or “No”, of whether or not the doc could be useful in resolving the difficulty. We additionally added annotations to supply further hints to the supposed roles of the assorted elements of the enter textual content, as illustrated within the determine beneath. To allow personalization, we expanded the difficulty description enter with textual representations of the consumer and their reservation info.

We fine-tuned the MT5 mannequin on the duty described above. In an effort to consider the standard of the generative classifier, we used manufacturing site visitors information sampled from the identical distribution because the coaching information. The generative mannequin demonstrated vital enhancements in the important thing efficiency metric for assist doc rating, as illustrated within the desk beneath.

As well as, we additionally examined the generative mannequin in an internet A/B experiment, integrating the mannequin into Airbnb’s Assist Middle, which has thousands and thousands of energetic customers. The profitable experimentation outcomes led to the identical conclusion — the generative mannequin recommends paperwork with considerably larger relevance as compared with the classification-based baseline mannequin.

Equipping brokers with the fitting contextual information and highly effective instruments results in higher experiences for our prospects. So we offer our brokers with just-in-time steering, which directs them to the proper solutions persistently and helps them resolve consumer points effectively.

For instance, by agent-user conversations, recommended templates are displayed to help brokers in downside fixing. To ensure our recommendations are enforced inside CS coverage, suggestion templates are gated by a mix of API checks and mannequin intent checks. This mannequin must reply inquiries to seize consumer intents akin to:

• Is that this message a few cancellation?
• What cancellation motive did this consumer point out?
• Is that this consumer canceling because of a COVID illness?
• Did this consumer by chance e book a reservation?

In an effort to assist many granular intent checks, we developed a mastermind Query-Answering (QA) mannequin, aiming to assist reply all associated questions. This QA mannequin was developed utilizing the generative mannequin structure talked about above. We concatenate a number of rounds of user-agent conversations to leverage chat historical past as enter textual content after which ask the immediate we care about on the cut-off date of serving.

Prompts are naturally aligned with the identical questions we ask people to annotate. Barely completely different prompts would lead to completely different solutions as proven beneath. Primarily based on the mannequin’s reply, related templates are then really helpful to brokers.

We leveraged spine fashions akin to t5-base and Narrativa and did experimentations on numerous coaching dataset compositions together with annotation-based information and logging-based information with further post-processing. Annotation datasets normally have larger precision, decrease protection, and extra constant noise, whereas logging datasets have decrease precision, larger case protection, and extra random noises. We discovered that combining these two datasets collectively yielded the perfect efficiency.

As a result of massive measurement of the parameters, we leverage a library, known as DeepSpeed, to coach the generative mannequin utilizing multi GPU cores. DeepSpeed helps to hurry up the coaching course of from weeks to days. That being mentioned, it sometimes requires longer for hyperparameter tunings. Due to this fact, experiments are required with smaller datasets to get a greater course on parameter settings. In manufacturing, on-line testing with actual CS ambassadors confirmed a big engagement price enchancment.

Correct intent detection, slot filling, and efficient options are usually not ample for constructing a profitable AI chatbot. Customers typically select to not interact with the chatbot, regardless of how good the ML mannequin is. Customers need to resolve issues shortly, so they’re consistently attempting to evaluate if the bot is knowing their downside and if it would resolve the difficulty quicker than a human agent. Constructing a paraphrase mannequin, which first rephrases the issue a consumer describes, may give customers some confidence and make sure that the bot’s understanding is appropriate. This has considerably improved our bot’s engagement price. Beneath is an instance of our chatbot mechanically paraphrasing the consumer’s description.

This technique of paraphrasing a consumer’s downside is used typically by human buyer assist brokers. The most typical sample of that is “I perceive that you simply…”. For instance, if the consumer asks if they will cancel the reservation at no cost, the agent will reply with, “I perceive that you simply need to cancel and want to know if we will refund the cost in full.” We constructed a easy template to extract all of the conversations the place an agent’s reply begins with that key phrase. As a result of we have now a few years of agent-user communication information, this easy heuristic provides us thousands and thousands of coaching labels at no cost.

We examined common sequence-to-sequence transformer mannequin backbones like BART, PEGASUS, T5, and so on, and autoregressive fashions like GPT2, and so on. For our use case, the T5 mannequin produced the perfect efficiency.

As discovered by Huang et al. (2020), one of the crucial widespread problems with the textual content era mannequin is that it tends to generate bland, generic, uninformative replies. This was additionally the key problem we confronted.

For instance, the mannequin outputs the identical reply for a lot of completely different inputs: “I perceive that you’ve got some points along with your reservation.” Although appropriate, that is too generic to be helpful.

We tried a number of completely different options. First, we tried to construct a backward mannequin to foretell P(Supply|goal), as launched by Zhang et al. (2020), and use it as a reranking mannequin to filter out outcomes that had been too generic. Second, we tried to make use of some rule-based or model-based filters.

Ultimately, we discovered the perfect resolution was to tune the coaching information. To do that, we ran textual content clustering on the coaching goal information primarily based on pre-trained similarity fashions from Sentence-Transformers. As seen within the desk beneath, the coaching information contained too many generic meaningless replies, which brought about the mannequin to do the identical in its output.

We labeled all clusters which might be too generic and used Sentence-Transformers to filter them out from the coaching information. This method labored considerably higher and gave us a high-quality mannequin to place into manufacturing.

With the quick progress of large-scale pre-training-based transformer fashions, the textual content era fashions can now encode area information. This not solely permits them to make the most of the applying information higher, however permits us to coach fashions in an unsupervised approach that helps scale information labeling. This permits many modern methods to deal with widespread challenges in constructing AI merchandise. As demonstrated within the three use instances detailed on this submit — content material rating, real-time agent help, and chatbot paraphrasing — the textual content era fashions enhance our consumer experiences successfully in buyer assist situations. We imagine that textual content era fashions are a vital new course within the NLP area. They assist Airbnb’s company and hosts resolve their points extra swiftly and help Assist Ambassadors in reaching higher effectivity and a better decision of the problems at hand. We look ahead to persevering with to speculate actively on this space.

Thanks Weiping Pen, Xin Liu, Mukund Narasimhan, Joy Zhang, Tina Su, Andy Yasutake for reviewing and sharpening the weblog submit content material and all the nice recommendations. Thanks Joy Zhang, Tina Su, Andy Yasutake for his or her management assist! Thanks Elaine Liu for constructing the paraphrase end-to-end product, operating the experiments, and launching. Thanks to our shut PM companions, Cassie Cao and Jerry Hong, for his or her PM experience. This work couldn’t have occurred with out their efforts.

Enthusiastic about working at Airbnb? Try these open roles.