What is Google LaMDA? Here's what you need to know

If you’ve read anything about state-of-the-art AI chatbots like ChatGPT and Google Bard, you’ve probably come across the term large language models (LLMs). OpenAI’s GPT family of LLMs power ChatGPT, while Google uses LaMDA for its Bard chatbot. Under the hood, these are powerful machine learning models that can generate natural-sounding text. However, as is usually the case with new technologies, not all large language models are equal.

So in this article, let’s take a closer look at LaMDA — the large language model that powers Google’s Bard chatbot.

What is Google LaMDA?

LaMDA is a conversational language model developed entirely in-house at Google. You can think of it as a direct rival to GPT-4 — OpenAI’s cutting-edge language model. The term LaMDA stands for Language Model for Dialogue Applications. As you may have guessed, that signals the model has been specifically designed to mimic human dialogue.

When Google first unveiled its large language model in 2020, it wasn’t named LaMDA. At the time, we knew it as Meena — a conversational AI trained on some 40 billion words. An early demo showed the model as capable of telling jokes entirely on its own, without referencing a database or pre-programmed list.

Google would go on to introduce its language model as LaMDA to a broader audience at its annual I/O keynote in 2021. The company said that LaMDA had been trained on human conversations and stories. This allowed it to sound more natural and even take on various personas — for example, LaMDA could pretend to speak on behalf of Pluto or even a paper airplane.

Besides generating human-like dialogue, LaMDA differed from existing chatbots as it could prioritize sensible and interesting replies. For example, it avoids generic responses like “Okay” or “I’m not sure”. Instead, LaMDA prioritizes helpful suggestions and witty retorts.

According to a Google blog post on LaMDA, factual accuracy was a big concern as existing chatbots would generate contradicting or outright fictional text when asked about a new subject. So to prevent its language model from sprouting misinformation, the company allowed it to source facts from third-party information sources. This so-called second-generation LaMDA could search the Internet for information just like a human.

How was LaMDA trained?

Before we talk about LaMDA specifically, it’s worth talking about how modern language models work in general. LaMDA and OpenAI’s GPT models both rely on Google’s transformer deep learning architecture from 2017. Transformers essentially enable the model to “read” multiple words at once and analyze how they relate to each other. Armed with this knowledge, a trained model can make predictions to combine words and form brand-new sentences.

As for LaMDA specifically, its training took place in two stages:

1. Pre-training: In the first stage, LaMDA was trained on a dataset of 1.56 trillion words, sourced from “public dialog data and web text”. According to Google, LaMDA used a dataset 40 times larger than the company’s previous language models.
2. Fine-tuning: It’s tempting to think that language models like LaMDA will perform better if you simply feed it with more data. However, that’s not necessarily the case. According to Google researchers, fine-tuning was much more effective at improving the model’s safety and factual accuracy. Safety measures how often the model generates potentially harmful text, including slurs and polarizing opinions.

For the fine-tuning stage, Google recruited humans to have conversations with LaMDA and evaluate its performance. If it replied in a potentially harmful way, the human worker would annotate the conversation and rate the response. Eventually, this fine-tuning improved LaMDA’s response quality far beyond its initial pre-trained state.

You can see how fine-tuning improved Google’s language model in the screenshot above. The middle column shows how the basic model would respond, while the right is indicative of modern LaMDA after fine-tuning.

LaMDA vs GPT-3 and ChatGPT: Is Google’s language model better?

On paper, LaMDA competes with OpenAI’s GPT-3 and GPT-4 language models. However, Google hasn’t given us a way to access LaMDA directly — you can only use it through Bard, which is primarily a search companion and not a general-purpose text generator. On the other hand, anyone can access GPT-3 via OpenAI’s API.

Likewise, ChatGPT isn’t the same thing as GPT-3 or OpenAI’s newer models. ChatGPT is indeed based on GPT-3.5, but it was further fine-tuned to mimic human conversations. It also came along several years after GPT-3’s initial developer-only debut.

So how does LaMDA compare vs. GPT-3? Here’s a quick rundown of the key differences:

1. Knowledge and accuracy: LaMDA can access the internet for the latest information, while both GPT-3 and even GPT-4 have knowledge cut-off dates of September 2021. If asked about more up-to-date events, these models could generate fictional responses.
2. Training data: LaMDA’s training dataset comprised primarily of dialog, while GPT-3 used everything from Wikipedia entries to traditional books. That makes GPT-3 more general-purpose and adaptable for applications like ChatGPT.
3. Human training: In the previous section, we talked about how Google hired human workers to fine-tune its model for safety and quality. By contrast, OpenAI’s GPT-3 didn’t receive any human oversight or fine-tuning. That task is left up to developers or creators of apps like ChatGPT and Bing Chat.

Can I talk to LaMDA?

At this point in time, you cannot talk to LaMDA directly. Unlike GPT-3 and GPT-4, Google doesn’t offer an API that you can use to interact with its language model. As a workaround, you can talk to Bard — Google’s AI chatbot built on top of LaMDA.

There’s a catch, however. You cannot see everything LaMDA has to offer through Bard. It has been sanitized and further fine-tuned to serve solely as a search companion. For example, while Google’s own research paper showed that the model could respond in several languages, Bard only supports English at the moment. This limitation is likely because Google hired US-based, English-speaking “crowdworkers” to fine-tune LaMDA for safety.

Once the company gets around to fine-tuning its language model in other languages, we’ll likely see the English-only restriction dropped. Likewise, as Google becomes more confident in the technology, we’ll see LaMDA show up in Gmail, Drive, Search, and other apps.

FAQs