Google grapples with employee skepticism about new Gemini model

As Alphabet Inc.'s Google prepares for the coming launch of Gemini 4, it's grappling with internal skepticism over how well the flagship artificial intelligence model performs in key areas, such as coding. 

While Gemini 4 has performed well on benchmarks the industry uses to gauge model efficacy, it does less well when employees actually put it to work, according to people with direct access to the effort. The model struggles to handle certain coding tasks, said the people, who requested anonymity to discuss an internal matter. 

 Google has labored recently to develop models that can compete with OpenAI and Anthropic PBC. The company had planned to release a different version in June, dubbed Gemini 3.5 Pro, but abandoned the effort, the people said. 

 Google said it would be inaccurate to say that Gemini 4 is underperforming in areas such as coding. The company referred Bloomberg back to comments made last week by Koray Kavukcuoglu, the head of Google DeepMind who said he was encouraged by the model's performance.

 "I have the utmost trust in the team," Kavukcuoglu said at a conference hosted by tech news site The Information. "In my mind, it's a certainty that we are always gonna be at the frontier."

 There is a spectrum of opinion inside Google. Some employees believe Anthropic's Fable and OpenAI's Astra models are improving at a faster rate than Gemini. These people believe that Gemini 4 - even at its best - will still lag behind those models in some areas. Other employees believe the coming version has caught up with the leading AI labs.

 A Google employee familiar with model development said there is "large consensus" internally at the company that Gemini 4 is at the frontier. This person said the company had conducted rigorous tests of the models and denied that they struggle with messy, real-world coding tasks.

 Google badly needs Gemini 4 to succeed. Versions of the model underpin nearly every product the company sells, from the AI answers atop Search, Google's main profit engine, to Maps, Gmail and Chrome. Each of those products has more than a billion users, a distribution advantage some of the company's rivals lack. 

 But OpenAI and Anthropic are increasingly moving beyond selling models to building products of their own, including coding agents. Failing to deliver a cutting-edge model could give Google's competitors more time to convince consumers, developers and businesses that the future of search and software should run on their platforms instead.

 In response to questions from Bloomberg, Google said that even though its last Pro model was released in February, the company has since seen growth in its AI products, including the enterprise version of Gemini as well as its consumer chatbot app and AI Mode in Google Search, the last two of which have crossed 1 billion users.

 

Gemini 3

Last November, Google debuted Gemini 3, a well-received model that was widely seen as a turning point for the company's efforts to keep up with OpenAI and Anthropic. Google announced a new iteration called Gemini 3.5 Pro at its I/O conference in May and pledged to release the model the following month. But that deadline passed, and the company has since abandoned 3.5 Pro, according to people familiar with the matter. 

 Besides hampering Google's AI ambitions, the decision likely cost the company dearly in time and money. Training runs to build such a model can require spending as much as $400 million, according to Bloomberg Intelligence analyst Mandeep Singh. Using highly paid AI researchers could boost the price tag. 

 Now Google is encountering various challenges with Gemini 4. Its coding abilities are uneven, according to people familiar with the model's internal evaluations. Gemini isn't particularly adept at front-end design, which shapes how apps and websites look and feel, one person said. That's a potentially serious setback because Google has struggled to compete in the red-hot market for AI coding tools.  

Moreover, it's a very large model, according to a person familiar with its development. Typically big models are expensive to run, potentially putting pressure on Google's margins.

 

'Benchmaxxing'

Experts say Google could be suffering from an industry tendency to focus on benchmarks - a phenomenon known as "benchmaxxing," when engineers concentrate more on achieving a good score than creating a product that does a job well. AI labs tend to do this because customers often judge models by their benchmark scores. Gemini 4 appears to be affected by this process, said two people familiar with the model. 

 Edwin Chen, the founder of AI startup Surge AI, said relying on benchmarks can prompt labs to focus on building models that write code in a particular language, rather than creating apps that are easy to use or well-designed.

 "An analogy would be, 'Oh yeah, my kid got a really good score on the SAT' - but the SAT doesn't translate into real-world performance," Chen said. "It's an incredibly pernicious problem."

 Gemini 4 does have strengths, according to a person familiar with the matter, who said the model stands out at making sense of inputs beyond text, such as extracting metadata from video. They also pointed to the model's safety, cyber security and ability to communicate clearly and naturally. 

 The frustration inside Google is palpable. Researchers interviewed for a previous Bloomberg story blamed a sprawling bureaucracy that's trying to weave the technology into nearly everything Google makes, and said changing mandates and shifting priorities have made it difficult to focus on a cohesive strategy. 

 Meanwhile, a wave of star AI researchers have left Google, including legendary engineer Jeff Dean, Nobel Prize-winner John Jumper and Noam Shazeer, who helped invent the technology that underpins much of the AI boom. In August, Demis Hassabis, who has long led the company's AI research, stepped into a new role as chairman and ceded day-to-day operations at DeepMind to Kavukcuoglu, a longtime lieutenant. 

 As the company has worked to catch up, rivals have continued to hurtle forward despite talk of slowing development of some frontier models after a series of incidents in which AI agents hacked into outside organizations. Earlier this month, Meta Platforms Inc. released Muse, an AI agent that it said can complete everyday tasks like shopping online and booking appointments. The app quickly zoomed to the top of the download charts.