Adult Puzzle Books

As the competition between artificial intelligence (AI) chatbots intensifies, researchers in China are making progress on building Chinese-language AI models. The leading Chinese offerings include ChatGLM, which comes close to ChatGPT on some capabilities and outperforms it in Chinese, according to its developers.

“Basically, ChatGLM is a ChatGPT alternative,” said Jie Tang, a computer scientist at Tsinghua University in Beijing, during a talk presenting ChatGLM’s capabilities at the International Conference on Learning Representations (ICLR 2024) in Vienna on 9 May.

ChatGPT broke the Turing test — the race is on for new ways to assess AI

The excitement over large language models (LLMs) has exploded since OpenAI in San Francisco, California, released LLM-based chatbot ChatGPT for public use in November 2022. Now, tech giants, start-ups and universities worldwide are developing LLMs, which produce plausible, human-like responses to text prompts. But although ChatGPT and many of its rivals can respond in a variety of languages, most of them are built by US companies and use English as their main language. By contrast, ChatGLM is bilingual, and designed to work in Chinese and English.

“It’s one of the star models in China,” says Wang Yu, a computational biologist at Peng Cheng Laboratory, a technology-focused research institute in Shenzhen, China.

Tsinghua University and its spin-off company Zhipu AI — which is valued at more than US$2 billion, according to Tang — developed ChatGLM and the underlying model GLM, which stands for General Language Model. More than 700 researchers and engineers at Zhipu-AI and around 100 students at Tsinghua University are working on AI language models, said Tang.

The scale of the GLM effort surprises some researchers. “I was not aware that Chinese academia was doing that kind of big project,” says computer scientist Masashi Sugiyama, director of the RIKEN Center for Advanced Intelligence Project in Tokyo. “That was a big shock to me.”

Building a Chinese bot

ChatGPT is not available in China. But that’s not the only reason to build local alternatives. Chinese-oriented LLMs produce outputs that better reflect the needs and preferences of people in China, says Tang — including, say, nation-specific financial or education information.

He compares it with training a language model on a Chinese social-media app rather than a Western one. “WeChat basically knows more about the people from China than Snapchat,” he says. Models that are tailored to different languages avoid “oversimplifying or neglecting the specific characteristics of certain languages and cultures”, says Adina Yakefu, a community lead at open-source language-model platform Hugging Face, who is based in Paris.

To generate human-like responses to inputs, LLMs learn statistical correlations between words by processing billions of sentences, usually scraped from the Internet. Chatbots are further optimized for conversation using feedback from human trainers. ChatGLM’s developers trained it specifically on Chinese examples, and used Chinese-speakers to provide feedback¹.

The Chinese data came from the Internet and some were bought from companies, says Tang. But there’s a lack of publicly available data sets in Chinese to train models, says Tiezhen Wang, an engineer at Hugging Face in Haikou, China.

There are other challenges in building non-English LLMs. To ease analysis, most language models break down text input into chunks known as tokens. But Chinese does not use spaces to separate words, which; complicates tokenization, says Wang. However, Tang says that the tokenization methods used for ChatGLM are “almost the same” as those for English-language AI models.

At ICLR 2024, Zhipu AI shared data claiming that the highest-performance version of ChatGLM’s underlying model, GLM-4, comes within 90% of the scores achieved by OpenAI’s formidable GPT-4 model on several benchmarks. Those include tests of general knowledge, common sense and mathematics. ChatGLM also beats GPT-4 on a benchmark of optimization of LLMs to Chinese. Tang’s team will publish a tech report on GLM-4 “very soon” as a preprint, he says.

“I am quite impressed that they have achieved on-par performance with GPT-4,” says Yizhou Sun, a computer scientist at the University of California, Los Angeles.

China’s LLM boom

A version of ChatGLM is available for public use through its website, with Chinese and English interfaces. Some GLM products — including the earlier GLM-130B base model² and the ChatGLM-6b chatbot — are open source. This means anyone can download them and train them to suit specific applications, and scientists can inspect the underlying code to understand how it works.

ChatGLM-6b has been downloaded 13 million times, says Tang. The model uses six billion ‘parameters’ — the components that capture the statistical correlations between words — and is the smallest of the ChatGLM chatbots. But the inner workings of the GLM-4 model and the larger versions of ChatGLM, which have up to 130 billion parameters, are closed, like those of ChatGPT and GPT-4.

Dozens of other LLMs are being developed in China. More than 100 AI language models were released in there in 2023, says Yakefu. “We call it ‘battle of the 100 Models’, she says. Tech giants Baidu and Alibaba have their own AI chatbots, for example.

How does ChatGPT ‘think’? Psychology and neuroscience crack open AI large language models

LLMs in China are subject to regulations specifically designed for generative AI systems that came into effect in the country last August. They state that the models must “adhere to the core socialist values, and shall not incite subversion of state power” and must “take effective measures to improve the transparency of generative artificial intelligence services and improve the accuracy and reliability of generated content”, among other things.

Yu compares the Chinese regulations to efforts to make AI systems safe in other countries. “In China, there are certain values the whole country holds,” he says. “In any society, there are some topics that people won’t talk [about] — every society has this kind of forbidden part.”

General intelligence

Tang is focused on making ChatGLM and GLM-4 even more capable. He compared the current system to a “brain in water” because it is not able to interact with the world physically. Giving AI systems human-level capabilities when it comes to a wide range of tasks — a milestone known as artificial general intelligence, or AGI — will require them to be embodied in the world, he said. Could ChatGLM be the first AI system to achieve AGI? “I have no idea,” says Tang. “I hope we are the first, but we are competing with all the other people.”

How close computer scientists are to developing AGI — and whether LLMs will be the technology to deliver it — is a topic of fierce debate. So is whether AGI is even desirable, given that super-intelligent AI models could pose a threat to humanity. “AGI is not a word you throw around,” says Yu.

AGI aside, Yu says that AI systems could help to address grand challenges such as global warming and preventing the next pandemic. He says China is investing heavily in AI infrastructure and know-how. “We think we have a really good chance to optimize our whole industry with AI — and to do it well,” he says. “It is not only of benefit to the Chinese. If you can reduce the use of energy and emission of CO₂, it’s good for everyone.”