Most AI demos today suffer from a “low-ceiling” problem: they stop at “look, it can answer a question.” I wanted to push toward the actual horizon of this technology—an assistant that doesn’t just predict the next token, but personalizes recommendations, reasons with deep context, and executes real-world tasks.

That vision became Recommend Flow: a multi-agent architecture built on OpenClaw. The project hinged on one key architectural decision: instead of generating recommendations in a vacuum, the orchestrator first consults Xavibot v0.1 —my original assistant—as a “preference proxy.” Note that Xavibot is implemented using a completely different technology stack, leveraging Google’s Gemini models and a custom Retrieval-Augmented Generation (RAG) pipeline to ensure its “intuition” is grounded in my actual history.

The “We” and the Machine

A quick note on the terminology: when I say “we,” I am being quite literal. This system was deployed and refined in collaboration with a local OpenClaw agent running on a dedicated Linux machine in my home office. I communicate with the system primarily through WhatsApp, often using voice messages for convenience. The local agent handles the transcription, intent extraction, and orchestration.

One of the most pragmatic features of this setup is how it handles the “last mile” of execution. By utilizing browser navigation on my local machine, the bot doesn’t need to know my passwords or handle sensitive credentials. It simply leverages the active logins already present in my local browser sessions. While this requires the machine to be secure, it keeps the identity risk isolated and avoids the headache of managing a separate “vault” of API keys for every third-party service.

Furthermore, we’ve leaned into the philosophy of “Memory as Documentation.” All agent memories and complex workflows are stored locally as simple .md files. This approach offers several advantages:

• Maintenance & Transparency: I can literally cat a memory file to see exactly what the agent “knows” or “remembers” about my tastes. If it learns a wrong preference, I fix the text file.
• Portability: The entire intelligence of the system—the workflows, the personality priors, and the task histories—lives in a folder that can be moved across machines or version-controlled via Git.
• Security Guardrails: Storing workflows in plain text allows for human-readable audit trails. I can verify the steps the agent intends to take before it ever touches a browser.

The Big Idea: Separate Roles, Compose Capabilities

In a previous post, I argued that modern LLMs are evolving into “reasoning engines.” In Recommend Flow, we intentionally split those reasoning responsibilities:

• The Orchestrator (OpenClaw session): This agent serves as the central brain and primary interface. While it often communicates via WhatsApp, it also supports a local browser-based UX for more interactive sessions. Crucially, the orchestrator manages the Recommend Flow—the set of hard rules and constraints that define the recommendation process. However, it isn’t a rigid state machine; it has the autonomy to improvise and handle edge cases outside of the predefined flow when necessary.
• The Preference Proxy (Xavibot v0.1): This is not just another LLM endpoint. It runs RAG (Retrieval-Augmented Generation) over a broad corpus of my own material—blog posts, personal guides, and private documents. It acts as the “taste model,” carrying my content memory and style priors.
• The Execution Layer (Browser Automation): Completes the workflow (e.g., booking a table or adding an item to a cart) using the local browser.

This separation matters. A single model trying to handle high-level preference reasoning and low-level DOM manipulation is often brittle. A role-specialized setup is easier to debug, tune, and—most importantly—trust.

How Recommend Flow Works (End-to-End)

The interaction follows a disciplined Recommend → Decide → Do loop:

1. Voice/Text Intent: I send a WhatsApp message: “Hey, find me a place for dinner tonight that I haven’t been to but fits my usual vibe.”
2. Preference Interrogation: The Orchestrator acknowledges the request and immediately queries Xavibot v0.1: “Based on Xavier’s past writing on food and his local guide, what are his core dining preferences?”
3. Constraint Refinement: The system brings those signals back and asks for missing details (e.g., location or specific timing) one at a time.
4. The Decision Set: It returns a compact “Top 1 + Backup 1” recommendation with explicit tradeoffs based on my retrieved preferences.
5. Action Execution: Once I give the “Go”, the local Linux agent wakes up the browser, navigates to the reservation site, and executes the task using my existing login.

Watch the Recommend Hotel Reservation Demo below or you can open it here

*In this demo I have a chat with my Openclaw instance (called Xavibot) using a local browser for easier recording (as mentioned I usually interface through Whatsapp). Note that the browsing on the Chrome browser on the left is completely autonomous. In fact, at some point Openclaw decided to Google for “best restaurants in Palo Alto on OpenTable” that was a surprise.”

Why Xavibot v0.1 is the Perfect Backbone

As I’ve explored in my Data-Informed Gut Decision-Making framework, good decisions require a mix of data and intuition. Xavibot v0.1 brings three properties that are hard to fake with prompting alone:

• Grounded Memory: RAG over a vast collection of my writings, personal guides, and miscellaneous documents give- s persistent, high-signal context. It has even surprised me by surfacing details I’d forgotten I documented—like my preference for avoiding spicy foods, an observation it pulled from deep within my records.
• Taste Continuity: It reflects my historical writing and constraints, acting as a “digital twin” of my preferences.

What We Learned (and What Broke)

1. Multi-agent beats monolith: Role separation reduced prompt complexity and made behavior consistent.
2. Personalization is a loop, not a profile: Static “user profile” fields are useful, but conversational updates (context, mood, live constraints) matter just as much.
3. Action design needs strict guardrails: People forgive imperfect suggestions; they don’t forgive wrong actions. We learned to make recommendations “cheap” to generate, but transactions require explicit, high-confidence confirmation.
4. Isolation is a security feature: Running the browser automation locally on my own machine, rather than in a cloud-hosted container, provided a natural security boundary that felt much safer for a personal project.

Conclusion: The Future of Agentic Execution

The most interesting part of this project isn’t that an AI can recommend a restaurant; it’s the architectural pattern. This combination of preference-grounded reasoning, specialized agents, and tool-based execution is a blueprint that generalizes to travel planning, shopping, gifting, or even professional hiring workflows.

If the previous wave of AI was about assistants that could answer, this is the wave of assistants that can decide with you and then execute for you.

He listened thoughtfully, nodding along to my points. Eventually, I paused and asked, “By the way, what do you do at Google?”

He smiled politely and introduced himself. It was Shane Legg.

For those who might not know, Shane is the Chief AGI Scientist at Google DeepMind. He is also the person who literally coined the term “AGI” nearly two decades ago. Pitching the case against AGI to the man whose life’s work is dedicated to building it is certainly one way to break the ice and Shane did take it in good humor. But despite the irony of the moment, I stand firmly by the arguments I made that day. I am not AGI-pilled.

Before I dive into the technical details, let me clarify one crucial distinction: rejecting AGI does not mean rejecting AI. I am incredibly optimistic about the future of Artificial Intelligence and its potential to fundamentally transform software, science, and society. My skepticism is directed solely at the pursuit of AGI—the obsession with building a single, monolithic, human-like “God model.” Being anti-AGI makes me a pragmatist, not a pessimist. In fact, I believe letting go of the AGI myth is the very key to building better, more capable AI systems. Here is why.

The Myth of the “G”

The foundational flaw in AGI is the “G”: General. The concept assumes that human intelligence is a universal baseline against which all synthetic intelligence should be measured. But human intelligence is not general at all; it is highly specialized.

I am not alone in this view. Yann LeCun, Meta’s Chief AI Scientist, has publicly called the concept of artificial general intelligence “complete BS,” precisely because human intelligence is inherently specialized. We are optimized for a very specific evolutionary niche. If you look at raw numerical computation, a pocket calculator from the 1980s is vastly superior to the human brain.

If you look at nature, the idea of human intellectual supremacy gets even blurrier. While our particular brand of intelligence often differentiates us from animals, it doesn’t always make us objectively more “intelligent” in every context. A bird’s navigational intelligence, for instance, far surpasses that of most humans. (Read “If Nietzsche Were a Narwhal: What Animal Intelligence Reveals About Human Stupidity” for more examples and details).

Benchmarking an AI against human capability doesn’t make it “general.” It simply makes it an artificial mimic of our specific, localized evolutionary adaptations.

Striking the Magic from Intelligence

If human cognition isn’t the gold standard, what actually is intelligence?

My former colleague Blaise Agüera y Arcas tackles this beautifully in his great book What Is Intelligence?. He strips away the mystical, anthropocentric aura we tend to wrap around the mind. Agüera y Arcas frames biological organisms essentially as compositions of functions, reducing intelligence to the elegant mechanics of prediction and computation. In his book, Aguera lists the 5 properties of intelligence, all of which point to a compound rather than monolythic nature. Intelligence is (1) predictive, (2) social, (3) multifractal, (4) divese, and (5) symbiotic. This leads to a very “non-agi” definition of intelligence as “the ability to model, predict, and influence one’s future; it can evolve in relation to other intelligences to create a larger symbiotic intelligence.” (Also kudos for that Before Sunrise reference!).

When you define intelligence by its functional reality (the ability to model the world and predict outcomes) the AGI illusion starts to fade. Building complex systems that master prediction (like predicting the next token in a sequence) doesn’t magically summon a human-like mind into the machine.

The Monolith Fallacy and the “Intelligence Gap”

This brings us to the most frustrating contradiction in modern AI research: the cognitive dissonance of the top labs. Even luminaries who recognize the flaws in human “generality” (like LeCun, or DeepMind CEO Demis Hassabis) still insist on cramming all capabilities into massive, monolithic foundation models.

Recently, Hassabis pointed to Large Language Models making mistakes in basic math as evidence of a “real intelligence gap” on the road to AGI. But this completely misses the point. LLMs are probabilistic predictors. Expecting billions of frozen neural weights to perform flawless, symbolic arithmetic is an architectural mismatch; it is like using a hammer to turn a screw. The solution to an LLM failing at math isn’t to train a larger, more expensive monolith or to expect another breakthrough. The solution is simply to give the model access to that 1980 calculator.

Similarly, many experts frequently point to “continuous learning” as a major hurdle separating current AI from true AGI. But again, this is a limitation of a frozen, monolithic neural network, not a limitation of AI as a system. Agentic AI solves this elegantly. Agents like OpenClaw are already demonstrating continuous learning by actively managing persistent memory and reading/writing to files. In fact, if we look outside the LLM bubble, simpler specialized AI—like massive-scale recommender systems—have been successfully using online learning approaches to continuously update and learn from user interactions for decades. We don’t need a mystical AGI breakthrough for continuous learning; we just need better system design.

The Power of Composition: Enter Compound AI Systems

This is why the current pursuit of AGI represents terrible engineering. The most robust, scalable, and safe AI architectures in production today do not rely on a single, omniscient model. They achieve broad capability through composition—what we now call Compound AI Systems.

Instead of forcing a single neural network to do everything, you orchestrate specialized agents. You use an LLM not as a universal database, but as a semantic router and reasoning engine. If the system needs factual grounding, it queries a vector database (RAG). If it needs to execute logic, it writes code and hands it to a Python interpreter. If it needs to do exact arithmetic, it executes an API call to a deterministic calculator tool.

This multi-agent paradigm is not just theoretical; it is happening right now. Recent examples like Moltbook and Kimi agent swarm have gathered far more attention, excitement, and practical traction than any of the recent monolithic model launches. When Anthropic’s CEO Dario Amodei talks about the future being a “Country of Geniuses in a Datacenter,” he is effectively describing a multi-agent reality. You do not need a central, monolithic AGI to make that possible. You simply need swarms of highly specialized agents collaborating at scale.

As I have noted before, specialized, superhuman agents are likely to be both more achievable and more beneficial in addressing specific, complex challenges in alignment with our goals (see my “Beyond Singular Intelligence: Exploring Multi-Agent Systems and Multi-LoRA in the Quest for AGI”). Much like modern, massive-scale recommender systems, intelligence at scale is a pipeline, not a monolith.

This compound approach is inherently safer. When you rely on specialized tools, you maintain control. You can monitor the API calls, isolate hallucinations, and bake strict, programmatic guardrails into the boundaries between agents. A monolithic model, by contrast, is an opaque black box where capabilities and failure modes are dangerously entangled.

The winner-takes-all hard takeoff scenario

So why are AI labs so obsessed with pursuing a single, monolithic AGI instead of embracing specialized composition? It is largely driven by the science-fiction fantasy of the “hard takeoff.” This is the belief that once a single model crosses a certain intelligence threshold, it will recursively self-improve at an explosive rate. It’s an arms race fueled by the fear that the first company to build this “God model” takes the entire global economy.

Fueling an arms race to build a single, opaque, uncontrollable system just to win a hypothetical “winner-takes-all” scenario is not a sound technological strategy. It is reckless.

Conclusion

Generality is not a magical spark waiting to be ignited inside a massive GPU cluster. Broad, robust capability is a system-level property, achieved through the careful, safe composition of specialized tools. We don’t need AGI to build highly capable systems, and we certainly shouldn’t be gambling our future on the hope that the first ones to reach the hard takeoff threshold will be whoever we consider to be “the good ones”.

I’ve been involved with RecSys for a long time. This keynote was my 11th. I attended the first six, so I was there in the early days—and I’ve watched the field repeatedly reinvent itself.

One of my favorite personal “RecSys origin stories” is that when I transitioned from academia to industry, I found my job through this community. In Barcelona 2010, I started conversations that led me to Netflix, and eventually to many other things. I look around today and see people who have been interns with me and then followed similar paths. That’s part of what makes this community special: it’s a rare intersection of industry + academia + practitioners, with a shared obsession not only for algorithms, but for product, users, and psychology.

This talk had three parts:

• How we got here (history, with some personal bias)
• Recommending in the age of GenAI (the present)
• What’s coming next (where I think we’re heading)

Part I — How We Got Here

MovieLens v0, 1997, and the “field becomes a field”

Early in the talk I put up a screenshot that not everyone recognized: MovieLens v0 (published around 1997). For me, that interface is more than nostalgia. It’s a marker that a set of ideas turned into a recognizable field—built by Joe Konstan, the late John Riedl, and the rest of the University of Minnesota team.

It’s also why the first RecSys conference was held in Minneapolis—and why going back there feels like a loop closing and reopening.

AI history, intertwined with RecSys history

I deliberately intertwined recommender systems history with the history of AI, because the two have been co-evolving for decades:

• 1950s: “Artificial Intelligence” is coined at Dartmouth; Rosenblatt publishes the perceptron paper.
• 1969 → 1970s: Minsky’s critique leads to the first AI winter.
• 1980s: expert systems become fashionable again; then people rediscover their brittleness and scaling limits.
• 1987–1993: another AI winter.
• 1997: MovieLens, early RS papers.
• 2006–2009: Netflix Prize (we’ll spend time here).
• 2007: RecSys conference starts (on the heels of Netflix Prize energy).
• 2011–2016: deep learning momentum hits recommender systems (YouTube DL recommender paper is a major moment).
• 2018: Transformers (“Attention is All You Need”).

This timeline matters because it shows a pattern: RS progresses when model capability, data availability, and product surfaces line up—and stalls (or misleads us) when we optimize the wrong abstractions.

Netflix Prize: a turning point, and a lesson about proxies

The Netflix Prize (2006–2009) was pre-Kaggle, pre-everything we now take for granted. It was a massive public experiment. The goal was framed as “better recommendations,” but the proxy objective was explicit: improve RMSE on rating prediction by 10%, win $1M.

The winning solution was instructive:

• It was an ensemble (as usual).
• It combined 104 models using a neural network.
• The “main” approaches were a matrix factorization / SVD variant and restricted Boltzmann machines (a neural net).

Then came the part I think many people remember less clearly: we took the work back to Netflix and asked, “Can we productionize it?” The answer was: not as-is.

• 104 models didn’t scale well, was too slow, too complicated.
• More importantly: while we were doing that translation work, we realized something deeper: the objective itself (RMSE on ratings) was not the right question.

We did productionize SVD and RBMs—they were the first ML algorithms that went into Netflix’s product. But the Netflix Prize still taught a durable lesson: You can win the benchmark and still lose the product. Or, more precisely: your offline proxy can be “correct” and still be wrong.

From “algorithms” to “machine learning” to “AI”

Back then, we didn’t even say “machine learning.” We said algorithms. My team at Netflix was literally called Algorithms Engineering. Over time, the naming shifted: algorithms → ML → AI. That branding shift wasn’t just marketing; it reflected real changes in how systems were built and what people expected of them.

I used a simple example to make this concrete: the most basic personalized recommender you can build—almost comically basic by today’s standards.

• Two features: Popularity, Predicted rating
• Two parameters: w1 and w2
• A linear model

The task: learn w1 and w2 from user behavior data. It’s a useful toy model because it captures a core truth: the system is mostly the same loop, regardless of complexity: choose features, choose model family, estimate parameters from data, measure whether it helped users.

We used to “advance ML” by adding more features and making models more complex. Feature engineering mattered and it required domain knowledge.

I gave a Quora example that still resonates with me: ranking answers for a question. It sounds obvious until you try to formalize it. We had to talk to editors and journalists about what “good” meant. They said: truthful, reusable, well-formatted, not too long, the right length. That became features. And then those features got learned.

That was “old-school” ML—though, honestly, we still do versions of it today.

The recommender problem evolved: rating → ranking → page → context

Another key arc: the problem definition evolved.

• Point-wise prediction: predict a rating (Netflix Prize era)
• Ranking: learn to order items
• Page optimization: optimize a full surface (rows, shelves, competing modules)
• Context-aware: device, time of day, location, intent—more dimensions

This wasn’t an academic shift. It was driven by the reality that a product isn’t a “list.” It’s an environment. At this point in the talk I referenced two of my own prior contributions:

• The Netflix work on “Beyond the five stars”, emphasizing why implicit feedback often beats explicit ratings for real-world optimization.
• The “Multiverse recommendation” work (published at RecSys in Barcelona 2010), which became my most cited RecSys paper—explicitly leaning into context-aware recommendation.

Deep learning in recommender systems: two-tower (and the promise of representation learning)

Then came deep learning’s major wave in RecSys—roughly 2011 onward—culminating in the “deep learning for YouTube recommendations” moment that hit this community hard.

To ground it, I showed the classic two-tower model:

• user embedding tower
• item embedding tower
• dot product to score similarity / relevance

It’s not the best model, but it’s the right mental starting point. Even in 2014 we were already experimenting with distributed neural nets in production contexts. And by 2016–2017, I was explicitly framing this as “the recommender problem revisited,” because deep learning forced us to revisit assumptions about features, modeling capacity, and system architecture.

(Geeky rabbit hole — Deep learning “replaced feature engineering”… but RecSys had already been blending paradigms)

Deep learning’s promise was: “stop hand-crafting features; the model learns representations.” But there’s a subtle connection to how recommenders already worked. Even matrix factorization is, in a sense, a hybrid of:

• unsupervised structure learning (dimensionality reduction, latent factors)
• supervised signal (ratings, implicit feedback)

We were already combining unsupervised and supervised approaches in clustering and latent-factor methods. Deep learning didn’t invent the idea; it industrialized it and scaled it—and then moved us more explicitly into self-supervision.

I tied this to the “multi-layer cake” framing of modern ML:

• self-supervised pretraining
• supervised fine-tuning
• reinforcement learning / alignment as the “icing”

This “layered training” view is something I’ve written about in the context of modern LLMs—especially the idea that “token prediction” alone undersells what post-pretraining adds.

It’s not only about algorithms

At this point I paused and emphasized a point that’s easy to say and hard to operationalize: In recommender systems, the algorithm is rarely the whole system.

I summarized the “non-algorithm” pillars as:

• UX / design
• Domain knowledge
• Evaluation metrics

UX / design: I showed an early Netflix interface where the page was packed with explanations: predicted rating, “because you watched X”, actors, director, etc. Those explanations—and the way we presented choices—often mattered as much as the model. This also connects to why Netflix ultimately moved from stars to thumbs: the UX and the feedback mechanism are part of the learning loop.

Domain knowledge: Even with deep learning, you still need domain knowledge—especially in constrained domains like healthcare. Constraints aren’t optional; they’re foundational.

Evaluation metrics: You need offline and online evaluation. You must iterate fast with offline proxies, validate with online experiments, and connect short-term metrics to long-term satisfaction/retention. I cited a memorable result from a YouTube team study: long-term satisfaction was causally linked not merely to “more consumption,” but to diversity of content consumed. If you get people to consume a more diverse set of content, they tend to be more satisfied in the long run. That finding matters because it’s a reminder that “maximize clicks” is not the same thing as “maximize sustained satisfaction.”

Part II — Recommending in the age of GenAI

Bill Gates wrote, “the age of AI has begun.” I used that line to mark the present moment—because Transformers, LLMs, and GenAI changed both the research conversation and the product conversation.

Two parameters → trillions of parameters

I showed a plot: transformer research families over time, parameter counts rising from ~100M to beyond a trillion. It’s worth repeating the contrast because it captures the discontinuity: earlier I showed a recommender with two parameters (popularity weight and predicted-rating weight). Now we’re in a world where models have trillions of parameters. All of those parameters still get learned from data—just through a very different pipeline.

Even “research impact” got weird

I mentioned how my citations changed dramatically in 2024–2025 because I posted three arXiv works: an LLM survey, a prompt design/engineering publication, and my transformer catalog. They weren’t even peer-reviewed in the traditional sense—yet the field’s attention was so concentrated that the impact was immediate. That’s not a moral argument; it’s an observation about attention allocation in the current research ecosystem.

How GenAI is already changing recommender systems

I gave three examples (largely from Google) to illustrate trends:

• LLMs for understanding preferences
• Generative retrieval
• Transformers applied to content-heavy recommendation contexts (e.g., music)

Then I returned again to the earlier triad (UX, domain knowledge, evaluation) and argued they still matter—but differently now:

• UX and AI are now intertwined; sometimes the UX is the AI (chatbot-style discovery).
• Pretrained foundation models carry a lot of domain knowledge out of the box—but domain expertise still matters for constraints and evaluation.
• Evaluation is arguably more important now; measuring GenAI is hard, and feedback loops are subtle.

Demo 1: “basic LLM” recommendation from a handful of shows

I demonstrated a simple prompt in Gemini: I gave it four Netflix shows I liked and asked for recommendations. What mattered wasn’t only the recommendation list—it was the “thinking trace” the model surfaced: identify attributes, extract themes, find common threads, craft categories, then produce options. And it worked. The model recommended:

• Ozark (which I’d seen and liked)
• Mindhunter (which I hadn’t seen)
• Narcos (which I’d seen and liked) Also: it was different each time—both the blessing and the curse of generative systems.

Demo 2: zero-history preference elicitation in five questions

Then I made it more interesting: “assume you know nothing about me—ask me five yes/no questions and recommend five music artists I’ll like.” Again, the point wasn’t just the output. The model:

• designed a question flow
• implicitly built a decision tree
• updated a “user profile” after each answer

With five questions, it recommended: Tool, King Crimson, Animals as Leaders, Russian Circles, and Karnivool. I noted that some picks were a bit off for my taste—likely influenced by how I answered the “harsh vocals” question. But the more important observation remained: from a blank sheet, the system elicited preferences and produced plausible recommendations.

A taxonomy of how LLMs enter RecSys

I referenced a diagram from the STAR paper that classifies approaches (see below):

• pure prompting (like my demos)
• prompting with user history (user–item interactions)
• using LLMs to create semantic features/IDs/embeddings combined with collaborative signals
• two-LLM architectures: one for semantic features + CF, another LLM for final ranking The meta-point: “using LLMs” isn’t a single technique; it’s a design space.

Part III — What’s next

I started this section with a screenshot from a startup called Fable: “Netflix for generative content.” The proposition is an extreme endpoint of personalization: not only recommending content but generating content, on the fly, personalized to each user. That’s “the last step” of personalization in one direction.

Then I returned again—intentionally—to the product triad:

• UX design (especially in multimodal / agentic worlds)
• Domain knowledge + deep, continuous user knowledge
• Evaluation (now vastly harder)

Demo 3: a recommendation agent (Localify)

I showed an agent built with Google AgentSpace. I called it “Localify.” Its job was simple: ask the user about tastes, search local events, and help find tickets. In the live demo, the agent didn’t ask the preference questions because it already “knew” my earlier answers (I had tested it). Based on what it remembered—rock, jazz, music, cinema—it recommended:

• an indie rock concert
• a jazz evening
• an independent drama film

Then it helped find a link for tickets. What I wanted to emphasize was how small the barrier has become:

• the agent prompt was basic
• I used “help me write” and the LLM improved the prompt
• it took minutes, not weeks

And if you want to make it more powerful, you can connect it to tools: calendar, email, enterprise systems, backend databases, …and even (dangerously) payment.

(AI digression — the moment you add tools, you import responsibility) In one of my posts, I put it bluntly: AI is great for organizing/analyzing data, but it doesn’t have “gut,” intuition, or accountability—and that’s precisely why human judgment remains central. That maps directly to agents: the moment an agent can act, UX design and safety constraints stop being secondary concerns.

Agents that browse the web and recommend

I then showed a more advanced agent concept (Project Mariner): it can browse the web on your behalf—scroll, click, match opportunities to your resume, and execute a multi-step flow. The only additional capability (conceptually) is huge: delegated navigation in human UIs.

World models (Genie 3) and “generated reality”

I showed a clip of “Genie 3,” positioning it as a frontier: not just generating text or images, but generating interactive worlds, with real-time reactivity, “world memory” (actions persist), and promptable events. This opens a window to a future where “personalized media” is not just personalized content—it’s personalized environments.

Deep, continuous user knowledge: the personalization paradox

LLMs have huge world knowledge; what’s still hard is injecting knowledge about you—accurately, safely, continuously. I showed a Gemini direction: more persistent memory so you don’t need to repeat “I like jazz and indie cinema” every time. That’s the personalization paradox: the model knows the world, it still struggles to know you (and to update that knowledge responsibly).

Research directions I highlighted

I ended with a set of recent papers (three examples) illustrating trends:

• aligning LLM-powered systems to user feedback (and novelty)
• serendipity / novelty with multimodal signals
• hybrid strategies that combine fine-tuning (infrequent) with RAG (more frequent) to keep user modeling fresh without constantly retraining

Conclusion: a journey from clicks to conversations

I closed with a framing I’d encourage you to keep in mind as you design systems in 2026 and beyond: We’ve revisited the recommender problem multiple times. We started with predicting stars, then clicks. We’re shifting into conversations, and now agents—long-running, tool-using systems that discover on our behalf.

My current bets are:

• Agents are the future of discovery. They’ll search, filter, and propose options in the background, then surface novel things for us to engage with.
• Personalization will remain the hard part. World knowledge scales. “User knowledge” is messy, dynamic, private, and consequential. Evaluation is the new frontier. Especially for long-running, multi-step systems where value accrues over time and failure modes are subtle.
• The ultimate prize might be “media of one.” Content not only discovered for you—but created for you, on the fly, personalized to what you want right now.

And, because this is RecSys: karaoke remains a constant—and apparently so do 7am runs.

Q&A moments worth carrying forward

A few audience questions surfaced important tensions:

Recommend from catalog vs generate unique items? The cultural value of shared artifacts matters. If everyone gets a different show, what happens to shared conversation? My instinct: we’ll find hybrid dynamics—personalized creation plus social sharing (you can “send” your generated show).

Will users really have long conversations vs passive feeds? Different modes will coexist. There are “brain-dead scroll” moments and “high-ROI search” moments (finding the next book vs watching a 30-second clip). The adoption of chat products is a strong counterexample to the idea that people never want conversational interfaces.

If agents consume content, what incentives remain for creators? No clean answer yet. But historically, new creation tools tend to democratize creation rather than end it—and we should proactively design ecosystems that keep human creativity rewarded and visible.

References

• Amatriain, X. (2025). Keynote at ACM RecSys 2025
• Konstan, J. A., et al. (1997). GroupLens: Applying collaborative filtering to Usenet news
• Koren, Y. (2009). The BellKor Solution to the Netflix Grand Prize
• Karatzoglou, A., Amatriain, X., Baltrunas, L., & Oliver, N. (2010). Multiverse Recommendation: N-dimensional Tensor Factorization for Context-aware Collaborative Filtering.
• Amatriain, X. & Basilico, J. (2012). Netflix Recommendations: Beyond the 5 stars.
• Le, Q. V., et al. (2012). Building high-level features using large scale unsupervised learning (The “Cat” paper).
• Covington, P., et al. (2016). Deep Neural Networks for YouTube Recommendations.
• Vaswani, A., et al. (2017). Attention Is All You Need.
• Amatriain, X. (2024). Transformer models: an introduction and catalog.
• Minaee, S., et al. (2024). Large Language Models: A Survey.
• Amatriain, X. (2024). Prompt Design and Engineering: Introduction and Advanced Methods.
• Lee, D., et al. (2024) STAR: A Simple Training-free Approach for Recommendations using Large Language Models
• Wang, J., et al (2025) User Feedback Alignment for LLM-powered Exploration in Large-scale Recommendation Systems
• Meng, C., et al. (2025) [Balancing Fine-tuning and RAG: A Hybrid Strategy for Dynamic LLM Recommendation Updates] (https://arxiv.org/abs/2510.20260)

AI and the AI Revolution

The current revolution isn’t just about code; it’s about the fundamental nature of intelligence and the hardware that powers it.

• “What is Intelligence: Lessons from AI about Evolution, Computing, and Mind” by Blaise Aguera y Arcas: My former colleague at Google, Blaise, delivers an incredible deep dive into the meaning of life through the lens of computation. It’s a profound look at how AI helps us redefine what it means to be alive and intelligent.
• “A Brief History of Intelligence:: Evolution, AI, and the Five Breakthroughs That Made Our Brains” by Max S. Bennett: A great companion to Blaise’s book, focusing on the five breakthroughs that shaped our brains.
• “Chip War: The Fight for the World’s Most Critical Technology” by Chris Miller: To understand the AI revolution, you must understand the silicon. Miller provides a fascinating look at how companies like TSMC, ASML, and NVIDIA reached their current dominance and how these dynamics are shaping global geopolitics.
• “The Nvidia Way: Jensen Huang and the Making of a Tech Giant”): For those of us living and breathing the AI revolution from the inside, this history of NVIDIA is essential reading. It serves as a perfect companion to Chris Miller’s Chip Wars; while Miller provides the macro-perspective of the silicon landscape, this book dives deep into the specific company culture and technical bets that allowed NVIDIA to dominate that landscape.
• “The Optimist: Sam Altman, OpenAI, and the Race to Invent the Future”: More than an authorized biography, this is a detailed history of the characters and moments that shaped OpenAI and modern Silicon Valley.
• “The Singularity is Nearer” by Ray Kurzweil: A long-awaited update to his classic thesis on the merging of human and machine.
• “The Worlds I See” by Dr. Fei-Fei Li: A beautiful memoir about curiosity and the dawn of modern AI from one of the field’s most important pioneers.
• “Nexus” by Yuval Noah Harari: Harari looks at information networks from the Stone Age to AI, providing his usual sweeping historical perspective.
• “Co-Intelligence” by Ethan Mollick: One of the most practical guides out there for actually living and working alongside AI today.

Tech and Its People

Understanding tech often requires understanding the “DNA” of the institutions and individuals that built it.

• “The Idea Factory” by Jon Gertner: A look back at Bell Labs, the original powerhouse of American innovation.
• “Elon Musk” by Walter Isaacson: Regardless of your personal opinion of him, Isaacson’s biography explains a lot about his trajectory and the “demon mode” that drives his companies.
• “This is For Everyone” by Tim Berners-Lee: I stumbled upon this recently. It’s the story of the WWW told by the man who invented it—covering the past, present, and his vision for the future.
• “In the Plex” by Steven Levy: Even though it’s missing the last 15 years, it remains one of the best books for understanding Google’s foundational culture.
• “Careless People: A Cautionary Tale of Power, Greed, and Lost Idealism by Sarah Wynn-Williams”: A raw, personal account of Meta’s cultural stumbles from someone who had a front-row seat to the internal dynamics.
• “Source Code: My Beginnings” by Bill Gates: A fascinating look at the early life of Bill Gates and the birth of Microsoft. I learned quite a bit I didn’t know about his early years.
• “Pattern Breakers: Why Some Start-Ups Change the Future” by Mike Maples Jr. and Peter Ziebelman: An insightful look at why some startups manage to change the future while most fail.

Leadership, Culture, and Human Nature

As I discussed in my Challenge-Inspire model post, leadership is about more than just task management; it’s about understanding the human element.

• “Reset: How to Change What’s Not Working” by Dan Heath: This is a very practical book on changing systems—like our teams—that aren’t working optimally. It focuses on finding leverage points to drive real change. Highly recommended.
• “The Laws of Human Nature” by Robert Greene: A comprehensive guide to understanding behavior and communication. It’s dense with historical references and provides great advice on how to bring out the best in people.
• “Supercommunicators” by Charles Duhigg: A recent bestseller that provides a great framework for connecting with others.
• “How to Know a Person” by David Brooks: A very personal guide on how to foster deeper connections. Brooks acknowledges his own challenges in this area, which makes his tips and advice feel very grounded and earned.
• “First, Break All the Rules” by Marcus Buckingham: A classic that still holds up regarding what great managers do differently.
• “How to Decide” by Annie Duke: More tools for the decision-making toolkit, which you know is a favorite topic of mine.
• “Start with Why” by Simon Sinek: A foundational text on how great leaders inspire action.

Other Important Ideas

Finally, a few books that have challenged my perspective on the broader world.

• “Capital in the 21st Century” by Thomas Piketty: This was proposed for a book club, and at 700 pages, it was daunting. However, I was pleasantly surprised. It’s a deeply researched work on macroeconomics and inequality that provides the essential historical background and context for more contemporary shifts.
• “Abundance” by Ezra Klein: With Piketty’s historical lens in place, Klein’s “Abundance” is a fascinating read. While it may appear techno-optimistic on the surface, its main lesson for me was how easily well-intentioned policies can falter when built on incorrect assumptions about the future—a critical takeaway as we navigate a world reshaped by AI and automation.
• “The Anxious Generation” by Jonathan Haidt: This is a critical look at how the “rewiring of childhood” via technology is impacting mental health. It serves as a stark counter-narrative to the techno-optimism of Ezra Klein; while Klein focuses on the potential for abundance, Haidt exposes the very real dangers and social costs of unbridled technological adoption. The book has become extremely influential, acting as a catalyst for new legislation and policy changes regarding smartphone and social media usage for minors around the world.
• “The End of the World is Just the Beginning” by Peter Zeihan: A provocative mapping of the potential collapse of globalization. He argues that the era of global trade and secure transport is a historical outlier that is rapidly ending due to demographic shifts and changing US policy. It provides a broader, macro-strategic complement to Miller’s Chip Wars; while Miller focuses on the specific geopolitical struggle over silicon, Zeihan maps the decaying global order that makes that struggle so volatile.
• “The Vital Question” by Nick Lane: A technical but rewarding explanation of how life came to be, focusing on energy constraints rather than just information. Lane argues that the leap from simple to complex life was a rare, energetic fluke, requiring a level of power that standard evolution struggled to achieve. The book has received immense critical acclaim, most notably from Bill Gates, who famously claimed it was the best book he’d read in years and that it would “help people understand that energy is as fundamental as information.”

I hope you find something in this list that sparks your curiosity. As always, I’m curious to hear what you’ve been reading. Are there any books that have fundamentally shifted your perspective lately? Let me know in the comments!

If 2024 was the year of the chatbot, 2025 has been the year AI started to think—or at least, the year we started debating what “thinking” really means. It has been a year of profound shifts: from simple instruction following to complex reasoning, from “vibes” to verifiable actions, and from general-purpose models to specialized agents. It has also been a year of significant change for me personally, as I moved on from Google to explore new challenges.

This recap is based on a set of monthly reports I wrote throughout 2025, originally as a way to keep track of the fast‑moving AI landscape for myself and for my team at Google. Over time, those notes became a disciplined way to separate signal from noise and to understand how individual launches and papers fit into a broader trajectory.

2025 turned out to be a year of both acceleration and recalibration. Breakthroughs in reasoning models, agents, multimodality, and efficiency continued at a remarkable pace, while questions around economics, safety, and real‑world impact became harder to ignore. Stepping back each month made it easier to see which themes truly mattered.

What follows is a month‑by‑month synthesis of those notes originally gathered for my internal newsletter at Google, curated and lightly expanded for this recap. The goal is not to be exhaustive, but to capture how the year actually unfolded. However, let me know if you are missing something important. I am also including direct links to monnths in case you have a favorite one!

• January: The Efficiency Shock and the “Peak Data” Debate
• February: The “DeepSeek Moment” and Grok’s Rise
• March: “Agentic Moore’s Law”
• April: Peering Inside the Box
• May: Ecosystem Moves and “Vibe Coding”
• June: Superintelligence and “AI-nxiety”
• July: Agents Getting Real and Practical
• August: The “Bubble” Panic and GPT-5’s Arrival
• September: “Nano Banana” and Scientific Breakthroughs
• October: Economic Value over Benchmarks
• November: The New SOTA Battleground
• December: Code Red and the Grand Finale
• What I Got Wrong in 2025
• Bonus: A Deeper Dive
• Conclusion

January: The Efficiency Shock and the “Peak Data” Debate

The year began with a wake-up call on efficiency. DeepSeek R1 made headlines not just for its performance, but for its remarkably low training cost, sparking a massive debate on whether we were entering a new era of “Post-Training” efficiency. While some celebrated this as a democratization moment, it’s important to read the fine print: techniques like distillation and fine-tuning are powerful, but they often rely on the existence of larger, more expensive frontier models. At the same time, we saw healthy skepticism emerge regarding agents, with papers arguing that for simple tasks, we might be over-engineering solutions.

• DeepSeek R1 reasoning model - The paper that started the year’s obsession with reasoning efficiency.
• Sky-T1 - Claiming 01-like performance on a mere $450 budget (distillation is key here!).
• Mistral Codestral - Mistral continuing to push the open-weight coding frontier.
• Agentless - A provocative paper arguing that for simple software engineering tasks, you might not actually need complex agents.
• Agents in Production Talks - A good collection of talks on agents in production (yes, these exist!).
• Nvidia Digits - Launching an AI desktop.
• Francois Chollet Interview - A must-watch discussion on AGI and how to measure it.
• Stanford Lecture on LLMs - Covering the often overlooked basics: tokenization, data, and evals.
• Andrew Ng’s Newsletter - Arguing that AI PMs are the future of software development teams.
• Google Survey on State of AI - An interesting multi-country pulse check.
• I compiled my favorite 2024 AI papers and shared my view on “Peak Data” and “Scaling Laws”.

January set the stage for a year where data quality, reasoning reliability, and evaluation rigor would repeatedly resurface as central constraints rather than secondary concerns.

February: The “DeepSeek Moment” and Grok’s Rise

February was dominated by the aftermath of DeepSeek and the surprise arrival of Grok 3. We spent weeks dissecting the DeepSeek papers, realizing that while it wasn’t necessarily a fundamental research breakthrough, it was a fantastic engineering feat that cleverly optimized known techniques. Meanwhile, Grok 3’s arrival—developed by a small team in just 18 months—shook up the leaderboards. We also began to realize that benchmarks are becoming increasingly “gameable” via test-time compute, making “Price vs. Performance” the new metric that matters.

• Grok 3 - Developed in just 18 months by a small team, hitting Rank 1 on Chatbot Arena.
• Imarena Price Plot - The new “Arena-Price Plot” became the most important chart for a few weeks.
• Semianalysis: DeepSeek Debates - A deep dive into Chinese leadership on cost and true training margins.
• Stratechery’s DeepSeek FAQ - Ben Thompson’s breakdown of the situation.
• [OpenAI Operator] (https://openai.com/index/introducing-operator/) and Deep Research- OpenAI’s response, pushing into agentic research.
• Perplexity Deep Research - Perplexity quickly following suit with their own implementation.
• Humanity’s Last Exam - A benchmark attempting to evaluate true reasoning capabilities.
• Mistral “Fastest Chatbot” - Powered by Cerebras hardware.
• AI Assisted CUDA Kernels - Nvidia and Sakana.ai showing how AI can write low-level code.
• Google Gemini 2.0 Flash - Competitive pressure driving efficiency.

February made clear that efficiency breakthroughs amplify—not reduce—the need for transparency, reproducibility, and robust evaluation.

March: “Agentic Moore’s Law”

By March, the conversation shifted heavily toward Agents. We started seeing real data suggesting an “Agentic Moore’s Law,” where the length of tasks agents can solve autonomously is doubling roughly every 7 months. This was also the month Andrej Karpathy dropped his “Deep Dive,” reminding us that despite the hype, LLMs still struggle with basic token-level tasks (like counting ‘r’s in “strawberry”) and that prompting is still very much an art form.

• Karpathy’s Deep Dive into LLMs - Explaining why LLMs can’t count “r’s” in “strawberry” and why RLHF isn’t “true RL”.
• The “Agentic Moore’s Law” - Interesting data showing the length of tasks agents can solve is doubling every 7 months.
• Sakana AI Scientist - Generating the first peer-reviewed scientific publication entirely by AI.
• Manus: The General AI Agent - Another contender in the general agent space.
• MultiAgentBench - Evaluating how agents collaborate and compete.
• YC AI Codebases - A quarter of YC startups now have codebases that are almost entirely AI-generated.
• Interview Coder - The viral tool (and cheating concern) for coding interviews.
• Model Context Protocol (MCP) - Why standardizing context matters for tools.
• Large Language Diffusion Models - Exploring diffusion for text generation.
• Perplexity Comet - A browser designed specifically for agentic search.
• Pushing AI into the physical world with Gemini Robotics
• Introducing Gemma 3, highly capable for single GPU/TPU
• Enhancements to Google Search with AI Overviews and a new AI Mode

March previewed a broader shift toward autonomy and structured reasoning that would accelerate throughout the year.

April: Peering Inside the Box

April was about peering inside the black box. Anthropic released fascinating research on tracing the internal “thoughts” of models, while Meta’s Llama 4 release highlighted a crucial finding: Reinforcement Learning is proving much more important than Supervised Fine-Tuning (SFT). In fact, the data suggested that too much SFT can actually hurt performance. This was also the month OpenAI released GPT-4.1, which felt like a minor iteration compared to the architectural shifts we were seeing elsewhere.

• Tracing the Thoughts of an LLM - Anthropic’s fascinating research on internal activations.
• Introducing Llama 4 - Meta moving to Mixture of Experts (MoEs) and huge base models (“Behemoth”).
• Llama 4 and the Benchmark Crisis - The Verge’s take on how Llama 4 broke our evaluation metrics.
• Shopify’s AI Mandate - Tobi Lütke’s internal memo mandating AI usage for employees.
• Stanford AI Index 2025 - The state of AI in 10 charts.
• OpenAI GPT-4.1 - A minor improvement that noticeably lacked comparisons to non-GPT models.
• Project AMIE Nature Paper on diagnostic accuracy and on assisting clinicians - Two publications on conversational medical AI.
• DolphinGemma - Using AI to decode dolphin communication (yes, really).
• Cloud Next 2025 and infrastructure announcements

April reinforced that evaluation quality and RL‑driven training were no longer optional—they were becoming core pillars of progress. At the same time, some serious questions came up about the validity of public benchmarks.

May: Ecosystem Moves and “Vibe Coding”

May felt like a consolidation month. OpenAI went on an acquisition spree (buying Jony Ive’s startup and Windsurf), signaling a push into hardware and broader ecosystems. Meanwhile, Mark Zuckerberg was on a podcast tour with a refreshing level of honesty, admitting that Llama is essentially a byproduct of Meta’s internal needs rather than a purely altruistic developer play. We also saw the rise of “Vibe Coding”—a development style prioritized by speed and flow over rigorous syntax—gaining legitimacy. Coding, multimodality, and enterprise applications increasingly shared the same underlying capabilities, even as models became more specialized at the surface.

• OpenAI buys Jony Ive’s hardware startup
• Zuckerberg on Llama - His podcast tour explaining Llama as a byproduct of internal needs.
• Anthropic Claude Opus 4 and Sonnet 4 - Continued pressure at the high end.
• Google I/O 2025 announcements https://blog.google/technology/ai/google-io-2025
• Microsoft Build 2025 - 50+ announcements for developers.
• Vibe Coding Podcast - The AI Daily Brief on the state of “vibecoding.”
• The Effect of ChatGPT on Students - A Nature article providing valuable data on AI in education.
• Brakes on Intelligence Explosion - Nathan Lambert offering a counterpoint to the “AGI by 2027” hype.
• Netflix AI Search - Generative AI hitting mainstream consumer UI.
• AlphaEvolve - AI designing algorithms to save compute costs.

June: Superintelligence and “AI-nxiety”

In June, the race for “Superintelligence” became explicit, with Meta forming a dedicated lab and aggressively poaching talent (reportedly paying 7-to-9 figures). But alongside this race, we started seeing the human cost: “AI-nxiety.” Developers and users alike expressed exhaustion at the relentless pace of updates. We also saw Apple release their “Illusion of Thinking” paper, a controversial splash that argued current reasoning models might be shallower than we assume—a debate that is still ongoing.

• Meta’s Superintelligence Lab - Poaching Scale AI’s CEO to lead the charge.
• OpenAI hits $10B ARR - The business of AI is booming.
• UAE Free ChatGPT - A nation-state strategy to accelerate adoption.
• LangChain Keynote - Harrison Chase on the state of agents.
• Apple’s “Illusion of Thinking”- A controversial paper arguing models might not be reasoning as deeply as we think.
• Apple WWDC - Apple continuing to play catch-up in the generative race.
• Scaling Reinforcement Learning - Semianalysis on why RL is the next frontier.
• What’s Next for RL - Nathan Lambert’s take.
• AI-nxiety - “AI is Awesome but It’s Fucking Exhausting.”
• Infrastructure constraints and energy considerations for AI data centers

June underscored a recurring theme: progress is increasingly gated by infrastructure, incentives, and workflows—not by model quality alone.

July: Agents Getting Real and Practical

July saw agents moving from cool demos to practical, integrated products. OpenAI merged their Operator and Deep Research teams, signaling that agents are the new search. We also saw smart implementation tactics from companies like Shopify, who are building innovative agents that access internal data via the Model Context Protocol (MCP). The economic argument also solidified this month: we began to see compelling evidence that LLM inference costs are dropping fast enough to potentially become cheaper than traditional search.

• Zuckerberg’s Superintelligence Memo - Now public.
• OpenAI ChatGPT Agent - Merging Operator and Deep Research into one system.
• Gemini Deep think formally achieves gold-medal at the International Mathematical Olympiad
• Moonshot Kimi K2 - Reaching #1 on open model spots.
• xAI releases Grok4
• Shopify’s AI Tactics - Building innovative agents accessing internal data via MCPs.
• The AI-Native Employee - Papers on how AI is changing the nature of work and teamwork.
• Karpathy’s YC Lecture - “Software in the Era of AI.”
• Prompt Attacks on Papers - Researchers injecting prompts to prevent negative AI reviews.
• Inference vs. Search Costs - Arguing that LLM inference is becoming cheaper than traditional search.
• CLI version of Gemini gets over 50k stars in github in a few weeks

July made clear that productivity gains are earned through structural change, not incremental tooling.

August: The “Bubble” Panic and GPT-5’s Arrival

August was a rollercoaster of sentiment. We had a wave of “AI Bubble” articles from major publications asking if we had peaked, citing failed pilots and high costs. Then, almost on cue, GPT-5 launched. While the rollout was rocky and required adjustments, the sheer user numbers (700M weekly users) and valuation ($500B) largely silenced the “dead end” narrative. It was a reminder that while the hype might fluctuate, the utility is scaling.

• OpenAI had a momentous month, launching GPT-5, though the rollout was rocky and required adjustments based on user feedback.
• The AI Bubble Articles- Fortune , The Atlantic , and New Yorker all asking if we’ve peaked.
• Anthropic Opus 4.1 - Focus on safety and interpretability (persona vectors).
• Meta Personal Superintelligence - Leaning into their vision.
• Microsoft Github Spark - Their take on “vibecoding.”
• Figma Make - Vibecoding comes to design.
• Semianalysis: Scaling Memory - The roadmap of HBM.
• Synthetic Data for Pretraining - Lessons from scaling synthetic data (arXiv).
• Data efficiency breakthroughs via high-fidelity labeling
• Genie-style world models and simulation advances
• Coverage of AI energy usage and efficiency trade-offs
• Waymo 100M Miles - Autonomous driving quietly hitting massive milestones.

The month surfaced doubts around real progress in AI with the backdrop of many launches that seemed to prove those with doubts were probably wrong.

September: “Nano Banana” and Scientific Breakthroughs

September was a huge month for Google, led by the viral success of the Nano Banana image editing model. But beyond the consumer hype, we saw incredible scientific work from DeepMind on fluid dynamics and a continued push for safer, more accessible AI.

• Google Nano Banana - The viral image editing model that pushed the Gemini app to #1.
• The discussion continues on AI’s impact on the job market: a Harvard study suggests AI is “choking” entry-level hiring, while a Stanford study makes similar claims.
• Meta is pushing further into hardware with its AI-powered smart glasses
• DeepMind Fluid Dynamics - Using Physics-Informed Neural Networks to discover new mathematical edge cases in fluid motion.
• Coding Wins - Google and OpenAI showing dominance at university coding competitions.
• Gemini in Chrome - Embedding models directly into the browser for billions of users.
• Frontier Safety Framework - Google DeepMind’s third iteration of their safety framework.
• Agent Payments Protocol (AP2) - An open protocol for secure agent-led payments.
• Time’s 2025 AI 100 - The annual list of influential AI leaders.
• Anthropic announced a massive new funding round ($13B at a $183B valuation) as they scale to over 300k enterprise customers.
• xAI released Grok 4 Fast making a claim to be at the frontier of cost-efficient intelligence.
• Replit Agent 3 and multi-hour autonomy

September signaled a new baseline: autonomy and reliability were becoming table stakes.

October: Economic Value over Benchmarks

The focus in October was heavily on the new possibilities of AI video creation and proving the economic value of AI. The GDP-val paper was a standout, proposing that we measure frontier models not by academic exams, but by their ability to perform “economically valuable” tasks.

• OpenAI Dev Day- Announcing the Atlas browser, Instant Checkout, and Sora 2, their new video model and a dedicated app for short-form, AI-generated videos.
• Anthropic Sonnet 4.5 - And “Imagine” for real-time UI generation.
• Meta Vibes - Feed for short-form AI video.
• GDP-val - Finding frontier models rival humans on economically valuable tasks.
• GenAI and Firm Productivity - Measuring real-world impact in retail.
• US economic dependence on data-center buildout
• DeepSeek Sparse Attention - New research from the efficiency kings.
• Karpathy’s Nanochat - “The best ChatGPT that $100 can buy.”
• Tinker - Thinking Machines’ tool for model fine-tuning.
• Gemini Enterprise - Major steps for cloud customers.
• Sparse attention and efficiency work from DeepSeek
• A Bloomberg feature detailed the web of circular deals among AI companies
• Interesting discussions on the Dwarkesh podcast. While RL pioneer Richard Sutton argued that LLMs are a dead end, Andrej Karpathy presented a contrasting perspective

Progress increasingly began to be evaluated economically, not just technically.

November: The New SOTA Battleground

November brought the year’s biggest shakeup: the launch of Gemini 3. Google’s latest model, accompanied by the new “Deep Think” reasoning mode and the “Google Antigravity” agentic platform, immediately topped the charts. Just days later, Anthropic countered with Claude Opus 4.5, marketed as the ultimate coding model with massive improvements in agentic workflows. The market share data reflects this shift—ChatGPT is no longer the default for everyone.

• Google Gemini 3 - A huge shakeup with enhanced reasoning and agentic capabilities.
• Nano Banana Pro - Building on the viral success of the original, this version pushed image editing even further.
• Claude Opus 4.5 - A SOTA coding model that reportedly scores 74.5% on SWE-bench Verified.
• ChatGPT losing market share - Data from Similarweb showing a clear trend towards Gemini.
• Similarweb Analysis - Further confirmation of the changing landscape. Pie
• SIMA 2 embodied agent research

November felt like a visible inflection point in both capability and market momentum. SOTA leadership, once measured in years, was now clearly measured in weeks.

December: Code Red and the Grand Finale

The year ended with high drama. Feeling the heat from Gemini 3 and Opus 4.5, OpenAI declared a “Code Red,” reminiscent of Google’s own similar move back in 2022. This urgency birthed GPT-5.2, a rapid iteration designed to reclaim the throne, alongside new features like ChatGPT Images. Meanwhile, at NeurIPS 2025 in San Diego, the buzz was all about embodied agents, with DeepMind unveiling Sima 2, a generalist agent for 3D worlds that feels like a real step towards general purpose robotics.

• OpenAI Code Red - Sam Altman rallying the troops as competition heats up.
• GPT 5.2 Launch - OpenAI’s rapid response to the shifting benchmarks.
• ChatGPT Images - Bringing native image capabilities to the forefront.
• Gemini 3 Flash - A speed-optimized beast that still manages ~78% on SWE-bench Verified.
• Shipping Sora for Android - A fascinating look at how OpenAI used Codex to build their own app in just 28 days.
• DeepMind Sima 2 - A generalist embodied agent for 3D worlds, unveiled just in time for the conference season.
• NeurIPS 2025 conference and proceedings. Also, interestingly, several VCs are now giving good summaries of the conference. * * Here is the one by Amplify Partners and here the one by Radical Ventures.

The year closed with a clear signal: fast iteration now coexists with renewed investment in long‑horizon research.

What I Got Wrong in 2025

Looking back at my own predictions (and anxieties) from the start of the year, a few things stand out: The Bubble That Wasn’t: In August, amidst the “Peak AI” narrative, I worried we were heading for a winter. I was wrong. The utility of these models in coding and enterprise workflows has created a floor for value that is much higher than I anticipated. Agents are Harder than We Thought: I expected autonomous agents to be “solved” by mid-year. Instead, we found that reliability at scale is an immense challenge. The “Agentic Moore’s Law” is real, but the slope is shallower than I hoped. The Persistence of Open Weights: I feared the gap between closed and open models would widen to a chasm. Instead, thanks to DeepSeek, Mistral, and Meta, the open ecosystem is arguably healthier than ever, keeping the giants honest on price.

Bonus: A Deeper Dive

Before closing out the year, I sat down for an in-depth, 2-hour conversation with Jon Hernandez on his “Inteligencia Artificial” podcast. We covered everything from my transition out of Google and the internal dynamics of big labs, to why I believe “AGI” is a distracting term and why we should focus on specialized agents instead. If you want the unfiltered, “director’s cut” version of my take on 2025 and beyond, this is it.

Conclusion

As I close out my recap of 2025, I am struck by how much the narrative has changed. We are no longer just awed by the fluency of LLMs; we are now demanding fidelity, reasoning, and autonomy. The battles of November and December proved that no lead is safe, and “SOTA” is a title you hold for weeks, not months.

Leaving Google this year has given me a fresh perspective on this ecosystem. The rate of change is dizzying, but it is also exhilarating. As we head into 2026, I am more convinced than ever that we are just scratching the surface of what is possible when we combine powerful reasoning foundation models with verifiable reasoning and agentic workflows. I look forward to exploring all of this in the incredible space of Travel from my new position as Chief AI and Data Officer at Expedia Group.

Here’s to a 2026 full of verified rewards and fewer hallucinations. Happy New Year!

In a previous post, we delved into “Beyond Token Prediction: the post-Pretraining journey of modern LLMs” to explore the multifaceted post-pretraining life of modern LLMs. A key takeaway from that discussion was that modern LLMs have evolved far beyond simple next-token prediction, a point that becomes even more critical as we venture into the realm of reasoning models. We touched upon how techniques like Reinforcement Learning from Human Feedback (RLHF) have been pivotal in aligning these models with human preferences. But the world of AI moves at a dizzying pace, and the conversation is already shifting. While RLHF has been a cornerstone, the new rave is all about Reinforcement Learning with Verifiable Rewards (RLVR), a term introduced in this recent paper. This isn’t just an incremental update; it’s a paradigm shift that could be the key to unlocking true reasoning in our models.

From Human Preference to Verifiable Truth

So, what exactly is RLVR, and how does it differ from the RLHF we’ve grown accustomed to? RLHF, in essence, is about teaching a model to be more “human-like.” We show it two responses, a human indicates which one is “better,” and the model learns to produce outputs that are more likely to be preferred. It’s a powerful technique for improving style, tone, and safety. However, it’s also inherently subjective. What one person prefers, another might not. And more importantly, a preferred answer isn’t always the correct answer, especially when it comes to complex reasoning tasks.

This is where RLVR comes in. As the name suggests, RLVR is a flavor of reinforcement learning where the reward is based on a verifiable, objective metric. Instead of asking “which answer do you like more?”, we ask “is this answer demonstrably correct?”. The reward is no longer a matter of opinion but of fact. For example, if we’re training a model to solve a math problem, the reward can be based on whether the final answer is correct. If we’re teaching it to code, the reward can be tied to whether the code compiles and passes a set of unit tests.

The folks at Fireworks.ai have been doing some fantastic work in this area, and their blog post on the topic is a must-read. They highlight that RLVR is particularly well-suited for tasks where the “goodness” of an output can be programmatically determined. This shift from subjective human feedback to objective, verifiable rewards is a subtle but profound one. It’s the difference between a model that’s a good conversationalist and one that’s a reliable problem-solver.

Diving Deeper: Reward Functions, Learned Models, and Policy Optimization

To appreciate the mechanics of RLVR, it helps to distinguish between a programmatic reward function and a learned reward model. RLVR’s power stems from its use of a programmatic reward function—essentially, a piece of code that deterministically scores an output. For example: if unit_tests_pass(): return 1.0 else: return 0.0. This is transparent, objective, and verifiable. In contrast, traditional RLHF uses a learned reward model, which is a separate neural network trained on human preference data to predict what score a human would give. This model is an approximation of human values and can have its own biases or be gamed.

Once you have a method for scoring responses, you need an algorithm to update the LLM’s policy (the “actor”). The workhorse here is PPO (Proximal Policy Optimization). It’s important to clarify a potential point of confusion: while RLVR avoids the learned reward model of RLHF, PPO itself, as an actor-critic method, often learns an auxiliary Value Network (the “critic”). This network doesn’t judge preference; instead, it estimates the expected future reward from a given state (i.e., the sequence of tokens generated so far). By comparing the actual reward to the critic’s prediction, PPO calculates the “Advantage”—a more stable signal for how good an action was. PPO uses this Advantage to update the actor in small, stable steps, maximizing the reward while ensuring the updated model doesn’t stray too far from its original state. This is a crucial safeguard against a common pitfall in RL known as reward hacking, where the model exploits an unforeseen loophole in the reward function to get a high score without achieving the intended goal. As detailed in a great SemiAnalysis article, this is a major challenge. The constraints in PPO help prevent this kind of undesirable optimization, making it a foundational technique for both RLHF and RLVR.

The field is moving fast, and simpler, more stable alternatives to PPO are emerging. A prominent example, brought to light by the team behind the Deepseek R1 model (as detailed in this Hugging Face post), is GRPO (Group Relative Policy Optimization). Unlike PPO, which evaluates responses individually, GRPO operates on a group of candidate responses for a given prompt. A key difference and advantage is its simplicity, as GRPO does not need to learn an auxiliary value function. For each prompt, it samples multiple outputs, calculates the average reward for this group, and then updates the policy based on the relative performance of each sample. The objective is to encourage responses that score above the group average and discourage those that fall below. This approach of using a group’s average performance as a dynamic baseline provides a more stable and robust training signal, reducing the variance that can make PPO tricky to tune. It’s particularly effective for complex reasoning tasks where a clear “winner” is less important than consistently moving towards better-than-average solutions. For those interested in a hands-on implementation, the team at Lightning AI has a great post on building a reasoning LLM from scratch using GRPO.

The Key to Unlocking Reasoning

Our ultimate goal is to teach models to reason, not just to mimic patterns of text that have been positively reinforced by humans. When a model is rewarded not just for the final answer, but for the steps it takes to get there, it learns a process. This idea was formalized into a technique known as Process Supervision, where a reward is provided for each correct step in a model’s reasoning chain, leading to the development of Process-based Reward Models (PRMs).

As detailed by OpenAI in their work on improving mathematical reasoning, instead of only rewarding the final answer (outcome supervision), Process Supervision rewards the model for each correct intermediate step. You can find more technical details in the original research paper. This is where things get really exciting. Imagine a model that can show its work, and we can verify and reward each step of that work. This not only makes the model’s reasoning process more transparent but also allows us to pinpoint exactly where it goes wrong and provide targeted feedback. This is a much more powerful and scalable approach than simply telling the model “your answer is wrong” and hoping it figures out why.

However, it’s crucial to draw a distinction here. While PRM-based training looked like the primary path forward a year ago, many of the biggest recent wins have come from simpler forms of RLVR—often just binary, outcome-based checkers combined with massive sampling and efficient algorithms like GRPO (see Nathan’s comment in Interconnects.ai blog post, “What Comes Next with Reinforcement,” ). This doesn’t mean PRMs have disappeared; rather, their role has evolved. They are now shifting to two key areas: as a crucial verifier at inference time to check the model’s reasoning, and for targeted training where step-level fidelity and interpretability are paramount. So, while still highly relevant, PRMs are becoming a specialized tool within the broader and more versatile RLVR toolkit.

Beyond just improving reasoning on static problems, RL is also a foundational component for training autonomous agents that can interact with environments and learn from the consequences of their actions. The shift is so significant that, as detailed in a recent piece by SemiAnalysis, the growing importance of RL is fundamentally changing the structure of AI research labs. As this insightful video on the future of AI agents explains, this sets the stage for continuous learning, where models can adapt and improve over time without constant, large-scale retraining.

Test-Time Compute: Getting More from Models at Inference

A model trained with RLVR provides a strong reasoning foundation, but its performance can be significantly amplified at inference through a strategy known as test-time compute. This refers to the computational effort a model expends when actively working on a prompt to arrive at a final answer. Instead of generating a single, immediate response, we can have the model engage in a more deliberative, multi-path reasoning process.

At inference time, we can scale up this compute by using techniques like Self-Consistency (sampling multiple reasoning paths and taking a majority vote on the final answer) or Tree-of-Thoughts (actively exploring a tree of possible reasoning steps). This allows the model to explore a wider solution space and self-correct. The final, crucial step is to use a verifier—which can be a simple programmatic check, a unit test, or even a PRM acting as a reranker—to select the best and most reliable answer from the many candidates generated. This purely inference-time strategy leverages the strong base model from RLVR training to achieve state-of-the-art accuracy and robustness on complex tasks.

Karpathy’s Corner: The Bull Case and a Word of Caution

No discussion of the future of AI would be complete without mentioning Andrej Karpathy. His insights are always a valuable addition to the conversation. In a recent tweet, he laid out his bull case for Reinforcement Learning, and it’s easy to see why he’s optimistic. The ability to fine-tune models based on specific, measurable outcomes is a powerful tool, and RLVR is a prime example of this.

However, Karpathy is also a pragmatist. In another tweet, he raised a crucial question about the scalability of reward functions. He expressed some doubt that we can design reward functions that can scale all the way to AGI. And he has a point. While it’s relatively straightforward to design a verifiable rewards for a math problem or a coding challenge, what’s the verifiable rewards for writing a beautiful poem or a compelling story? How do we create a reward function for “common sense”?

This is the central challenge we face. As we push our models to tackle more complex and nuanced tasks, the line between verifiable and subjective rewards will inevitably blur. But that doesn’t mean we should abandon the pursuit. The progress we’re seeing with RLVR in domains like math, science, and coding is a testament to its potential. It might not be the silver bullet that gets us all the way to AGI, but it’s a massive step in the right direction. It’s a step towards models that don’t just talk the talk but can actually walk the walk of reason. And that, in itself, is a revolution.

References

• Amatriain, X. (2024). Beyond Token Prediction: the post-Pretraining journey of modern LLMs. https://amatria.in/blog/postpretraining
• Minae, S., et al. (2024). Large language models: A survey. https://arxiv.org/abs/2402.06196
• Li, Z., et al. (2024). Reinforcement Learning with Verifiable Rewards. https://arxiv.org/abs/2411.15124
• Fireworks.ai. Reinforcement Learning with Verifiable Reward. https://fireworks.ai/blog/reinforcement-learning-with-verifiable-reward
• SemiAnalysis. (2025). Scaling Reinforcement Learning: Environments, Reward Hacking, Agents, & Scaling Data. https://semianalysis.com/2025/06/08/scaling-reinforcement-learning-environments-reward-hacking-agents-scaling-data/
• DeepSeek-AI. (2025). DeepSeek-V2: A Strong, Economical, and Open-Source Mixture-of-Experts Language Model. https://arxiv.org/html/2501.12948
• Uhr, N. (2025). Implementing Deepseek’s GRPO from scratch. Hugging Face Blog. https://huggingface.co/blog/NormalUhr/grpo
• Lightning AI. Build a reasoning LLM from scratch using GRPO. https://lightning.ai/lightning-purchase-test/studios/build-a-reasoning-llm-from-scratch-using-grpo?section=featured
• OpenAI. (2023). Improving mathematical reasoning with process supervision. https://openai.com/index/improving-mathematical-reasoning-with-process-supervision/
• Lightman, H., et al. (2023). Let’s Verify Step by Step. https://arxiv.org/abs/2305.20050
• Lambert, N. (2025). What Comes Next with Reinforcement. Interconnects.ai. https://www.interconnects.ai/p/what-comes-next-with-reinforcement
• The Future of AI Agents. (Video). https://www.youtube.com/watch?v=JIsgyk0Paic
• Wang, X., et al. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. https://arxiv.org/abs/2203.11171v4
• Yao, S., et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. https://arxiv.org/abs/2305.10601v1
• Karpathy, A. (2025). Twitter Post on RL Bull Case. https://x.com/karpathy/status/1944435412489171119
• Karpathy, A. (2025). Twitter Post on Reward Function Scalability. https://x.com/karpathy/status/1960803117689397543

This time, my primary tool was Cursor, an AI-first code editor, which I used almost exclusively in its agentic mode. I experimented with various backend models, including Gemini 2.5 Pro and Claude 4 Sonnet, though I often found myself defaulting to the “Auto” setting. I also incorporated Jules, Google’s own software development agent, into my workflow.

Instead of starting a project from scratch, I decided to undertake a full refactoring of my old project, Xavibot. The objective was to test the mettle of these AI agents on a relatively complex codebase while attempting substantial changes—a scenario often trickier than a clean-slate build.

The initial scope of the refactoring included migrating the chatbot’s backend from Azure to Google Cloud and transitioning the AI model from OpenAI’s GPT to Google’s Gemini (something I had been thinking about doing since my transition from LinkedIn/MSFT to Google for obvious reasons). This move wasn’t a simple swap. My original implementation was using the OpenAI Assistants API, which conveniently manages file uploading, vector databases, and conversation memory. For Gemini, I would need to implement this functionality myself.

After the initial migration, I decided to complicate things a bit more and keep both the Gemini and GPT models, giving users the ability to dynamically switch between them. This is a neat feature not commonly seen in user-facing chat applications, and I was curious to see how hard it would be to implement.

The entire refactoring process took about three days, totaling around 15-20 hours of focused work. As is often the case, the last 10% of the changes consumed 90% of the time. Deploying to Google Cloud and implementing the dynamic model switching proved to be the most time-intensive parts of the project.

What Went Well

The experience was, for the most part, very positive. I was particularly impressed by:

• Holistic Code Refactoring: The agents demonstrated a remarkable ability to propose and implement changes across numerous files simultaneously when prompted with a high-level refactoring goal.
• Intelligent Debugging: Having the Cursor agent read through logs to identify errors and suggest fixes was immensely helpful. It streamlined a process that is often tedious and time-consuming.

What Didn’t Go So Well

Despite the significant strides in AI-assisted development, there were several areas where the models still fell short:

• Knowledge Gaps: The models often lacked up-to-date information about API features. For instance, I had to explicitly inform the models that the Gemini API included an explicit caching option.
• API Versioning Issues: The AI models have a hard time dealing with APIs that have different versions. On several occasions, I had to read the documentation myself and point the LLM to the correct URL. I anticipate that maintaining updated, machine-readable documentation for coding agents (maybe brokered through an MCP server) will be an important use case soon.
• Context Loss: During longer refactoring sessions, the models would lose context, and I found myself repeatedly reminding them of previously discussed details.
• Tendency to Overcomplicate: Models still default to overcomplicated solutions. Thanks to my experience, I was able to sidestep most of these proposals, but I can see how this could lead to convoluted and unnecessary code for less-seasoned developers. Even in my case, I had to prompt the models to clean up and delete unused code after we got to a working solution.

The Verdict

All that said, this is a huge improvement in developer experience in only 18 months, and I was able to do a lot of work in just a few days. For my next iteration, I might refactor the Node.js backend to Python, since I have no reason to maintain the Node.js approach and Python APIs are usually updated sooner.

I invite you to try out the new and improved Xavibot directly here in this blog or visiting the URL directly. Let me know your thoughts. Any other suggestions on what other features to add? Again, code is available here

Intelligence Isn’t Everything

I’d argue that intelligence, while a significant human trait, isn’t the sole factor that makes us human. We are a complex mix: intelligence, yes, but also feelings, creativity, physical presence, social interactions, and so much more. While our particular brand of intelligence often differentiates us from animals, it doesn’t always make us objectively more “intelligent” in every context. A bird’s navigational intelligence, for instance, far surpasses that of most humans. (Read “If Nietzsche Were a Narwhal: What Animal Intelligence Reveals About Human Stupidity” for more examples and details) Similarly, AI is rapidly becoming more “intelligent” than humans in specific, and increasingly general, domains. It can process vast amounts of data, identify patterns we miss, and even, as we’ll see, contribute to creative and scientific endeavors. The key, then, isn’t to compete with AI on raw intelligence but to leverage its capabilities to enhance our own intelligence while actively promoting and valuing those other uniquely human aspects – our empathy, our creativity, our ability to connect and feel.

Even Human Experts Aren’t Perfect

It’s also crucial to remember that even expert humans are fallible. We often place immense trust in experts, like medical doctors, but studies show their “intelligence” or accuracy has limits. For example, one fascinating study highlighted that in straightforward diagnostic cases, doctors’ average accuracy was just over 55%, dropping to less than 6% for more complex cases (see “Physicians’ Diagnostic Accuracy, Confidence, and Resource Requests”). Perhaps even more surprisingly, their confidence level remained high regardless of the case’s difficulty (around 72% for easy cases vs. 64% for hard ones). What does this tell us? Even the best human experts make mistakes, and often, they aren’t aware when they might be wrong. So, as AI systems become increasingly capable, potentially surpassing human experts in many areas, we must remember they too will make mistakes. Blind trust is unwise, whether placed in a human or an AI.

Agency and Judgment: The Human Domain

This brings us to a critical point: decisions are for humans. While AI can provide insights, predictions, and recommendations at a scale and speed we can’t match, it lacks true agency, judgment, and the capacity for responsibility. An AI can’t be held accountable for the consequences of a decision. This becomes even more critical as we increasingly hear about “AI agents” capable of performing complex tasks autonomously. While powerful, these agents still lack genuine responsibility and accountability for their actions or outcomes. We must resist the temptation to outsource our own agency to these systems; the ultimate responsibility must remain firmly in human hands. Therefore, we must resist the temptation to let AI make important decisions for us. Use AI as a powerful tool, a co-pilot, or even a knowledgeable advisor. But never blindly trust its output. Just as you might seek a second opinion from another human expert, consider getting multiple perspectives, whether from different AIs or a combination of AI and human experts. Ultimately, you must own the decision. This requires critical thinking, evaluation, and applying our uniquely human blend of intelligence, values, and intuition.

AI in Science: Disruption and Opportunity

The world of scientific discovery is already being profoundly impacted by AI. In a fascinating, and perhaps slightly unnerving, development, a scientific paper generated entirely by AI recently made its way into a top-tier peer-reviewed workshop. This experiment by Sakana AI is likely just the beginning.

This raises questions about the future of scientific innovation. Some, like Thomas Wolf in his piece “The Einstein AI Model”, argue that AI, lacking true understanding and creativity, will never be able to genuinely innovate in science. While his narrative is compelling and worth reading, I have to disagree. History, even recent AI history, suggests otherwise. Remember AlphaGo’s groundbreaking Move 37? It was a move born from patterns learned through deep reinforcement learning (RL) and self-play, a move that human Go masters considered genuinely novel and creative.

Emerging research indicates that LLMs, trained using similar RL and self-play techniques, can generate scientific hypotheses evaluated as more novel than those produced by human scientists. An AI “co-scientist” could potentially accelerate discovery far beyond what adding another human expert to the team could achieve (see this paper for details). This disruption is happening now. Scientific organizations need to take this seriously, not by fighting it, but by embracing it and figuring out how to work with AI to push the boundaries of knowledge faster and further than ever before.

Critical Thinking in the Age of AI: Are We Getting Dumber?

A common fear is that relying on AI will atrophy our own cognitive skills, particularly critical thinking. Headlines often scream that “GenAI makes you dumb”. But does it?

A recent study by Microsoft and CMU titled “The Impact of Generative AI on Critical Thinking” explored how knowledge workers perceive their critical thinking when using GenAI. It’s crucial to note the study focuses on self-reported perceptions, not objective measures of critical thinking ability.

The findings? People report using more critical thinking when they trust the AI less or trust themselves more on a task. Interestingly, the type of task didn’t seem to affect the reported level of critical thinking. The study also suggests a shift in approach: humans move from executing tasks to verifying AI-generated outputs. Perhaps most relevant to the headlines, users perceived less effort was needed to engage critical thinking when using GenAI.

Does perceiving less effort mean critical thinking is actually reduced? The study doesn’t prove that. It’s meta-ironic to see humans exhibiting poor critical thinking by misinterpreting a study about AI and critical thinking. The paper itself offers nuanced insights and suggests that AI tools should be designed to encourage human agency and critical evaluation, aligning with points I’ve made previously.

AI, Learning, and the Effort Equation

The use case of AI in learning warrants special attention. Intuitively, allowing AI to simply provide answers feels like it would bypass the necessary mental effort crucial for deep understanding and retention. Recent research confirms this intuition isn’t entirely off-base, revealing that how students use AI tools like LLMs significantly impacts learning outcomes (see “AI Meets the Classroom: When Do Large Language Models Harm Learning?”. Using AI to substitute for learning activities (e.g., getting solutions directly) allows students to cover material faster (increasing breadth), but demonstrably reduces their depth of understanding. Conversely, using AI to complement learning (e.g., asking for explanations) improves understanding without necessarily speeding up coverage.

Disturbingly, studies suggest students gravitate towards the less effective substitution method, possibly because it requires less immediate effort. This preference, combined with findings that LLMs may widen the learning gap by benefiting high-knowledge students more than those with less prior knowledge, raises concerns. Furthermore, students using LLMs tend to overestimate how much they’ve actually learned compared to their tested performance.

These findings highlight a critical challenge for education. While AI offers potential as a powerful complementary learning aid, its potential for misuse as an effort-substitute is significant. Thoughtful integration, clear guidelines encouraging complementary use, and perhaps even interface designs that discourage passive substitution are necessary to harness AI’s educational benefits without undermining the learning process itself, especially for vulnerable learners.

Embracing Our Humanity

The Age of AI isn’t about humans versus machines. It’s about defining what truly matters to us as humans and leveraging these incredible new tools to enhance those aspects. AI will undoubtedly handle more cognitive tasks, but it cannot replicate the richness of human experience – our emotions, ethical judgments, creativity born from lived experience, physical interactions, and deep social bonds. Leveraging these tools effectively requires not only individual awareness and agency but also a commitment from creators to design AI systems that encourage critical engagement, support deeper understanding, and ultimately empower, rather than merely automate, our uniquely human capabilities. Let’s use AI to free ourselves up to be more human, not less.

Every month I send my team at Google a few paper recommendations. For this end-of-year blog post, I went through all my monthly emails, picked my favorite articles, and I grouped them into categories. In each category I kept them ordered by publication date, so you may get a sense of progress in each of them.

Of course, this is a highly curated and probably biased list. I hope you enjoy it, and please let me know what was your favorite paper that I missed!

Imagen3. Prompt= “Futuristic cityscape under construction in 2024, representing rapid progress in AI research, many buildings under construction, robots, drones, holographic blueprints, but also trees and nature and happy people and children, each building a different research area, detailed, cinematic lighting, concept art, warm colors”

• LLMs, surveys and new models
• New techniques
• RAG and beyond
• Domain-specific applications of LLMs
• AI Security and alignment
• Agents
• LLM Evaluation
• Beyond LLMs

LLMs, surveys and new models

1. Large Language Models: A Survey - This comprehensive survey paper, which I had the privilege of co-authoring, provides a broad overview of the rapidly evolving landscape of Large Language Models (LLMs). It covers everything from model architectures and training techniques to applications and ethical considerations. I was particularly interested in contributing to the section on prompt engineering, agents, and post-attention LLMs. It’s been incredibly gratifying to see this paper cited nearly 500 times in less than a year, highlighting the immense interest in this field. A must-read for anyone looking to get up to speed on LLMs.
2. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits - This paper introduces BitNet, a groundbreaking approach to training LLMs that drastically reduces the precision of model parameters to just 1.58 bits, effectively making each parameter ternary (-1, 0, or +1). While reducing precision this drastically might seem counterintuitive, the authors demonstrate that BitNet models achieve comparable performance to full-precision models while delivering significant improvements in latency, memory usage, throughput, and energy consumption. This could be a game-changer for deploying LLMs on resource-constrained devices and making them more environmentally sustainable. This work challenges the conventional wisdom that high precision is always necessary for LLM performance.
3. Jamba: A Hybrid Transformer-Mamba Language Model - Jamba represents an exciting new direction in LLM architecture, combining the strengths of Transformers with the efficiency of Structured State Space Models (SSMs), specifically the Mamba architecture. This hybrid approach allows Jamba to handle longer contexts more effectively than traditional Transformers, while also being more computationally efficient. As one of the first open-source models to successfully integrate these two architectures, Jamba is a significant step towards building more powerful and scalable LLMs.
4. PaliGemma: A versatile 3B VLM for transfer - PaliGemma, a new open-source vision-language model from Google DeepMind, stands out for its compact size (3 billion parameters) and its focus on transfer learning. Unlike many larger models that are trained from scratch for each new task, PaliGemma is specifically designed to be fine-tuned efficiently for a wide range of downstream applications. This makes it a particularly valuable tool for researchers and developers with limited resources, and it underscores the growing importance of transfer learning in the field of vision-language AI.
5. Gemma 2: Improving Open Language Models at a Practical Size - The Gemma 2 report offers a treasure trove of insights into the training process of state-of-the-art LLMs. The authors delve into the details of their architectural choices, including variations on attention mechanisms and the use of knowledge distillation, all while prioritizing a model size that is practical for real-world use. This report is highly recommended for anyone interested in the nitty-gritty of LLM development and the trade-offs involved in optimizing for performance, efficiency, and accessibility.
6. The Llama 3 Herd of Models - While the Llama 3 models from Meta don’t introduce radical architectural changes, they demonstrate the continued power of scaling up existing approaches. The key takeaway here is the sheer impact of using more data and larger models, combined with meticulous engineering. Llama 3 models achieve impressive results across a range of benchmarks, reaffirming that bigger (and with better data) often is better, at least for now. Still, it does make one wonder when diminishing returns will set in.
7. Imagen3 - Imagen 3, Google’s latest text-to-image model, pushes the boundaries of image generation quality and control. It outperforms all other models, including DALL-E 3 and Midjourney v6, on most benchmarks, demonstrating a remarkable ability to translate complex textual prompts into highly detailed and coherent images. While Midjourney v6 still holds a slight edge in subjective visual appeal, Imagen 3’s superior performance on benchmarks requiring precise prompt adherence highlights its strength in controllability. This model represents a significant leap forward in text-to-image generation, bringing us closer to AI that can truly understand and visualize our creative visions.
8. GPT-4o System Card - The GPT-4o System Card provides a crucial glimpse into the extensive safety work that went into OpenAI’s latest flagship model. It outlines the rigorous evaluations, including external red teaming and frontier risk assessments, that were conducted before its release. This document is a must-read for anyone interested in the ethical considerations surrounding advanced AI and the growing importance of proactive safety measures in the development of powerful language models.
9. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models - Meta’s release of Llama 3.2 models (both LLMs and VLMs) emphasizes the increasing demand for powerful yet efficient AI that can run on edge devices. These smaller models achieve state-of-the-art performance for their size, making them suitable for deployment on smartphones, wearables, and other resource-constrained hardware. This is a significant step towards democratizing access to advanced AI and enabling a new wave of on-device applications.
10. Movie Gen: A Cast of Media Foundation Models - Meta’s MovieGen showcases the impressive capabilities of foundation models in the realm of multimedia generation. This collection of models achieves state-of-the-art results on a wide range of tasks, including text-to-video, video-to-audio, and text-to-audio generation. This work highlights the growing power of AI to create and manipulate different forms of media, opening up exciting possibilities for creative expression and content generation.
11. Deepseek v3 technical report - The DeepSeek-V3 technical report is a testament to the fact that you don’t need a massive budget to build a state-of-the-art LLM. This model has been making waves in the AI community due to its impressive performance, achieved with a surprisingly small training budget. This report provides valuable insights into how to efficiently train high-performing LLMs, making it a valuable resource for researchers and developers working with limited resources.

From our “Large Language Models: A Survey” paper

New techniques

1. GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection - GaLore introduces a novel training strategy that tackles the memory bottleneck of training large language models. By using gradient low-rank projection, it enables full-parameter learning while being significantly more memory-efficient than popular methods like LoRA. This could open up new possibilities for training larger and more complex models on existing hardware, pushing the boundaries of what’s possible in LLM research..
2. Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention - Infini-attention offers a clever solution to the context length limitations of Transformers. This new attention mechanism allows for efficient processing of arbitrarily long contexts, even on LLMs with a relatively small number of parameters. By improving both context length and inference efficiency, Infini-attention could enable LLMs to process and understand much larger documents and conversations, unlocking new applications in areas like document summarization, question answering, and dialogue systems.
3. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale - This paper from HuggingFace tackles the crucial issue of data quality in LLM training. The authors present a new dataset called FineWeb, which is carefully curated from web data using a rigorous filtering and cleaning process. This work highlights the importance of high-quality data for achieving optimal LLM performance and provides a valuable resource for researchers working on improving the quality of web-scale datasets.
4. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet - This fascinating work from Anthropic delves into the inner workings of LLMs, demonstrating a method for extracting interpretable features from the Claude 3 Sonnet model. Not only can they identify these features, but they also show how they can be manipulated to control the model’s output. This research provides valuable insights into the interpretability of LLMs and opens up exciting possibilities for steering their behavior in a more fine-grained way. I find this area to be one of the most important in the path to safe AGI.
5. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality - This paper makes the surprising and insightful claim that Transformers, the dominant architecture in modern LLMs, are actually a specific type of Structured State Space Model (SSM). This connection opens up new avenues for understanding and potentially improving both Transformers and SSMs. By revealing this underlying duality, the authors provide a unifying framework that could lead to the development of more generalized and efficient models for sequence processing.
6. Scalable MatMul-free Language Modeling - This paper challenges the fundamental reliance on matrix multiplications (MatMul) in Transformer architectures. The authors demonstrate that it’s possible to build language models without any MatMul operations, potentially leading to significant improvements in computational efficiency. This is a radical departure from conventional approaches and could pave the way for new hardware architectures optimized for language modeling.
7. LoRA+: Efficient Low Rank Adaptation of Large Models - LoRA+ builds upon the popular LoRA (Low-Rank Adaptation) technique, further enhancing its efficiency by dynamically adjusting the learning rate for different parameters during fine-tuning. This relatively simple yet effective modification can lead to faster convergence and improved performance when adapting large models to new tasks, making it a valuable tool for practitioners working with LLMs.
8. Self-Play Preference Optimization for Language Model Alignment - This paper introduces a novel alternative to Reinforcement Learning from Human Feedback (RLHF) for aligning language models with human preferences. Instead of relying on external human feedback, it uses a self-play mechanism where the model competes against itself to improve its performance. This approach offers a potentially more scalable and efficient way to fine-tune LLMs for specific tasks or domains, and it raises interesting questions about the nature of learning and optimization in AI.
9. Training Language Models to Self-Correct via Reinforcement Learning - This work from DeepMind explores the use of reinforcement learning to train language models to self-correct their mistakes. By incorporating a self-correction mechanism during training, the model learns to identify and rectify errors, leading to improved performance and robustness. The authors show that this approach can act as a form of regularization, preventing overfitting and improving generalization to unseen data.
10. STAR: Synthesis of Tailored Architectures - The STAR paper introduces a novel approach to automatically designing neural network architectures using a gradient-free evolutionary algorithm. Instead of relying on manual design or gradient-based optimization, STAR explores a vast search space of architectures, evolving them over time to find optimal solutions. This work has the potential to automate the process of neural architecture search, leading to the discovery of new and potentially more efficient architectures for various tasks.
11. What Matters in Transformers? Not All Attention is Needed - This paper investigates the inner workings of the Transformer architecture, questioning the necessity of every single attention head. Their findings suggest that current Transformer architectures contain redundancies, and that similar performance can be achieved with fewer attention mechanisms. This research could lead to more efficient Transformer designs and a deeper understanding of what makes them so effective.
12. Cut Your Losses in Large-Vocabulary Language Models - This paper from Apple demonstrates a clever optimization technique for large-vocabulary language models. By implementing a custom kernel for matrix multiplication and log-sum-exp operations, they are able to significantly reduce computational cost and improve efficiency. The authors use the Gemma model as an example, showcasing the practical benefits of their approach. This is another great example of how low-level optimizations can have a big impact on the performance of large models.
13. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters - This paper challenges the conventional wisdom that bigger models are always better. The authors, from DeepMind, argue that scaling up computation at test time can be a more effective strategy than simply increasing the number of model parameters. While this idea has been recently popularized by speculation around OpenAI’s “O1” model, this paper provides the original research underpinning this concept. This highlights the importance of considering not just model size but also computational efficiency during inference, and it opens up new avenues for optimizing LLM performance.
14. Large Concept Models: Language Modeling in a Sentence Representation Space - Meta’s research on Large Concept Models (LCMs) explores a fascinating new direction for language modeling. Instead of predicting individual tokens, LCMs operate in a sentence representation space, predicting hierarchical concepts. This approach could potentially lead to more abstract and human-like understanding of language. While still early, this work suggests that there’s more to language modeling than just token prediction, and it opens up exciting new avenues for research into higher-level reasoning and understanding.
15. Pushing the frontiers of audio generation - “Having worked on audio synthesis myself many years ago, I’ve been consistently amazed by the rapid progress in AI-driven audio generation. This blog post from DeepMind provides an overview of the research that has led to these breakthroughs, covering areas like neural audio codecs and diffusion models. This is a great read for anyone interested in the intersection of AI and audio, and it showcases the incredible potential of AI to generate realistic and creative soundscapes.

RAG and beyond

1. In Defense of RAG in the Era of Long-Context Language Models - This paper makes a compelling case for the continued relevance of Retrieval-Augmented Generation (RAG) even as LLMs with increasingly long context windows become available. The authors argue that RAG is not simply a workaround for limited context length but offers distinct advantages in many scenarios, such as when dealing with rapidly evolving information or when needing to provide sources for generated text. This is a valuable read for anyone working with LLMs, as it challenges the assumption that longer context windows will necessarily replace retrieval-based methods.
2. Introducing Contextual Retrieval - “Anthropic’s work on contextual retrieval introduces a refined approach to RAG that takes into account the broader context of a query when retrieving relevant information. They propose two main techniques—contextual embeddings and contextual BM25—to improve the accuracy and relevance of retrieved passages. This research highlights the ongoing efforts to make RAG systems more sophisticated and context-aware, potentially leading to more accurate and nuanced responses from LLMs.
3. From Local to Global: A Graph RAG Approach to Query-Focused Summarization - This paper from Microsoft explores a novel approach to query-focused summarization using a graph-based RAG system. By representing information as a graph, the model can better capture the relationships between different pieces of information and generate more coherent and comprehensive summaries. This work demonstrates the potential of graph-based methods for enhancing RAG systems and tackling complex information retrieval tasks.

Domain-specific applications of LLMs

1. Capabilities of Gemini Models in Medicine - This paper showcases the impressive capabilities of Google’s multimodal Gemini models in the medical domain. The results are quite striking: Gemini models outperform medical experts on a wide range of tasks, including medical image interpretation, diagnosis, and report generation. This research underscores the transformative potential of AI in healthcare, although it also raises important questions about the role of human expertise in the future of medicine.
2. NExT: Teaching Large Language Models to Reason about Code Execution - The NExT paper presents a method for teaching LLMs to reason about the execution of code, a crucial step towards building more reliable and trustworthy AI for software development. The authors demonstrate that LLMs can learn to accurately predict the output of code snippets, even for complex programs. This research has significant implications for automating code analysis, debugging, and even code generation, potentially leading to substantial gains in programmer productivity.
3. Automated Social Science: Language Models as Scientist and Subjects - This paper explores the intriguing possibility of using LLMs to automate social science research. By combining structural causal models with the reasoning capabilities of LLMs, the authors propose a framework for conducting automated experiments and generating hypotheses. While still in its early stages, this research opens up exciting new avenues for using AI to accelerate scientific discovery in the social sciences, potentially leading to a deeper understanding of human behavior and social phenomena.
4. 3DS: Decomposed Difficulty Data Selection’s Case Study on LLM Medical Domain Adaptation - This paper challenges the common practice of domain adaptation and fine-tuning LLMs for specific medical tasks. The authors argue that frontier models, such as GPT-4, are already good enough for many medical applications, even without specialized training. They introduce a method called 3DS that selects training data based on decomposed difficulty levels. This research questions some of the prevailing assumptions about the need for extensive domain adaptation in every case and suggests that the capabilities of general-purpose LLMs might be underestimated in specialized domains like medicine.

AI Security and alignment

1. ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs - This paper reveals a surprising vulnerability in aligned LLMs: they can be jailbroken using carefully crafted ASCII art prompts. This might seem whimsical at first, but it highlights the fragility of current alignment techniques and the creative ways in which adversaries might try to circumvent them. This research underscores the need for more robust and comprehensive methods for aligning LLMs with human values and preventing them from being misused.
2. AGI Safety and Alignment at Google DeepMind: A Summary of Recent Work - This paper provides a valuable overview of Google DeepMind’s extensive research on AGI safety and alignment. It covers a wide range of topics, including scalable oversight, robustness, and system safety. This is a key read for anyone interested in the long-term safety of AI and the challenges of ensuring that increasingly powerful AI systems remain aligned with human goals.
3. Foundational Challenges in Assuring Alignment and Safety of Large Language Models - This multi-institution paper delves into the fundamental challenges of ensuring the alignment and safety of LLMs. It provides a framework for understanding the different types of alignment failures and outlines key research directions for addressing them. This paper is a valuable contribution to the growing field of AI safety and highlights the need for collaborative efforts to tackle these complex issues.
4. Quantifying Misalignment Between Agents: Towards a Sociotechnical Understanding of Alignment - This paper extends the discussion of alignment beyond individual models to the realm of multi-agent systems. The authors argue that misalignment between agents can be just as important, if not more so, than misalignment between a model and human values. They propose a framework for quantifying misalignment in multi-agent settings. This work highlights the need to consider the broader social and technical context in which AI systems operate and to develop methods for ensuring that agents can effectively cooperate and coordinate with each other.

Agents

1. CoSearchAgent: A Lightweight Collaborative Search Agent with Large Language Models - The CoSearchAgent paper introduces a lightweight, collaborative search agent that leverages the power of multiple specialized LLMs. By dividing the search task among different agents, each with its own expertise, the system can achieve more comprehensive and accurate results. This research demonstrates the potential of multi-agent systems for tackling complex information retrieval tasks and highlights the benefits of a collaborative approach.
2. Mixture-of-Agents Enhances Large Language Model Capabilities - This work from Together.ai provides further evidence for the power of multi-agent systems. They show that combining agents built using smaller, open-source models can outperform even the most advanced, monolithic LLMs on certain tasks. This research suggests that the future of AI might lie not in ever-larger single models but in well-designed systems of interacting, specialized agents.
3. Husky: A Unified, Open-Source Language Agent for Multi-Step Reasoning - The Husky paper, with authors from Meta and other institutions, introduces a new open-source language agent designed for multi-step reasoning. This general-purpose agent can tackle a wide range of tasks that require planning and problem-solving. The release of Husky as an open-source tool is a significant contribution to the field, providing researchers and developers with a powerful platform for building and experimenting with language agents.
4. LangChain State of AI Agents - This report from the LangChain team provides a comprehensive overview of the rapidly evolving field of AI agents. It covers the different types of agents, the key challenges in building them, and the most promising applications. This is an excellent resource for anyone looking to get up to speed on the current state of agent research and development.

LLM Evaluation

1. Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences - This paper tackles the crucial issue of evaluating LLMs, particularly when using other LLMs as evaluators. The authors introduce EvalGen, an interface that allows humans to grade LLMs on their ability to evaluate other LLMs, thus aligning the evaluation process with human preferences. This research highlights the importance of meta-evaluation in the development of LLMs and provides a practical tool for improving the reliability of LLM-assisted evaluations.
2. Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models - The Vibe-Eval benchmark, introduced by researchers at Reka AI, offers a new and challenging way to evaluate multimodal LLMs. It focuses on tasks that require a deep understanding of both visual and textual information. The introduction of this benchmark will push the development of more robust and capable multimodal models, helping us measure true progress in this important area.

Beyond LLMs

1. Accurate structure prediction of biomolecular interactions with AlphaFold 3 - AlphaFold 3 represents a major breakthrough in the field of structural biology. This latest iteration of DeepMind’s protein folding model can now predict the structure and interactions of a wide range of biomolecules, including proteins, DNA, and RNA, with unprecedented accuracy. The release of the AlphaFold Server, a free tool for researchers, promises to dramatically accelerate research in drug discovery, materials science, and our fundamental understanding of biological processes. This is a landmark achievement that has the potential to revolutionize many areas of science.
2. New Computer Evaluation Metrics for a Changing World - This paper argues for the need to rethink traditional computer performance metrics in light of the changing landscape of computing. The authors, my colleagues at Google, propose new metrics that take into account factors like energy efficiency, sustainability, and the specific needs of modern workloads, such as AI and machine learning. This is a timely and important contribution that will be relevant to anyone involved in designing, evaluating, or using computer systems.
3. Hardware Trends Impacting Floating-Point Computations In Scientific Applications - This paper explores the complex interplay between hardware trends and the precision requirements of scientific computing. The authors analyze how different floating-point formats and hardware architectures affect the accuracy and performance of scientific applications. This research provides valuable insights for developers of scientific software, helping them navigate the trade-offs between precision, performance, and efficiency in the context of evolving hardware.

Imagen3. Prompt = “2024, AI research depicted as a challenging mountain range, climbers making progress on each peak, representing breakthroughs and ongoing efforts, detailed, cinematic lighting, concept art”

In the ever-shifting landscape of Artificial Intelligence, pronouncements of the ‘end of an era’ are surprisingly common. The latest such declaration comes from Ilya Sutskever who recently suggested that the ‘age of data’—a period defined by the relentless pursuit of ever-larger datasets— is drawing to a close . “Peak data” is the term used. But is he right? This post will argue that the age of data is far from over. Instead, it’s transforming into something even more powerful: an age of superhuman data.

Now, to be fair, Ilya is known to be a believer in the power of big data. In fact their 2014 Sequence to Sequence paper that Ilya was remembering at NeurIPS this year had the following conclusions:

The “scaling hypothesis” that Ilya is referring to here was fully developed by several OpenAI researchers (including e.g. Dario Amodei now CEO of Anthropic) in their 2020 paper “Scaling Laws for Neural Language Model” where they showed how the performance of a large neural network could be predicted by the number of parameters, dataset size, and compute budget.

So, here we are, just 12 years later, being told that the age of data is over. We don’t have any more data to pump into our models. We ran out of our ‘fossil fuel’ of AI.Or have we?”

(Historical digression: One of the interesting things when you have been around for a while or you study a bit of history is that you realize that many scientific ideas or trends die and resurrect again and again. This is particularly true in AI, where a couple of AI winters declared the whole field itself mostly dead. As a reminder, the perceptron, which is the basic unit of Artificial Neural Networks, was also dismissed by Minski in his 1969 book Perceptrons.

It is also important to note that the simpler scaling law that states that in order to improve your model performance all you need is more data and more parameters was discovered years earlier. One of the earliest papers to be cited in that context is 2001 Banko and Brill’s “Scaling to Very Very Large Corpora for Natural Language Disambiguation”. Another popular, but a bit more recent, reference is “The Unreasonable Effectiveness of Data” by Halevy, Norvig, and Pereira from Google. Both of these contributed to the beginning of the “Age of Data” that is probably best illustrated by Chris Anderson’s “The End of Theory: The Data Deluge Makes the Scientific Method Obsolete” in Wired where he claimed that the world of discovery and science was going to be dominated by data. Models and algorithms were to become obsolete. I argued against this point-of-view in my 2012 post “More data or better models?” . )

Not all data is created equal

It is important to underscore that for data, size is not all that matters. You can have a huge dataset with very low informational value, or a small one with lots of knowledge density. In an extreme, you could take a single data point and replicate it 1 trillion times to have a pretty large dataset that only contains one piece of information. Yes, that is an extreme, but most of the data on the Internet is very low value. In fact, some of it has negative value! When I used to work on AI for healthcare I used to half jokingly talk about how the internet had very little high quality medical data and lots of very dubious Reddit threads with questionable medical advice.

Not surprisingly, advances in models such as Llama 3 list better data as the number one reason behind their improvement. An important detail of their approach, which is common nowadays, is their focus on having specific domains such as coding and math specifically represented. They also pay special attention to having the right “data mix” that represents the different kinds of knowledge the model should learn. The final mix has “roughly 50% of tokens corresponding to general knowledge, 25% of mathematical and reasoning tokens, 17% code tokens, and 8% multilingual tokens”. Note that the category of “general knowledge” could be broken down into whatever domains are important or relevant.

A highlight of this data-driven approach to LLM training is the “FineWeb Datasets” by HuggingFace. These datasets, and the associated publication, are a clear example of how important curation has become whenever training LLMs on publicly available data.

Beyond the Web: The Value of Proprietary and Labeled Data

While the vast expanse of the public internet has been the primary training ground for large language models, it represents only a fraction of the data that exists and, more importantly, only a fraction of the data that is valuable for pushing the boundaries of AI. A crucial point that gets overlooked in discussions about the “end of the age of data” is the existence of vast troves of proprietary data, often highly specialized and meticulously curated, that are not publicly available on the internet. Consider the healthcare industry, where patient records, clinical trial results, and medical imaging data hold immense potential for training specialized AI models. Similarly, companies like Waymo and Tesla possess massive, proprietary datasets of driving data that are far more detailed and comprehensive than anything available publicly. This kind of high-quality, domain-specific data is becoming increasingly important as AI moves beyond general-purpose tasks towards more specialized applications.

The value of this proprietary data is undeniable, even in the age of powerful pre-trained models. While some early proponents of the GPT era argued that pre-trained models would negate the need for specialized datasets through zero-shot learning, this has not proven to be entirely true. Instead, proprietary data is being leveraged in new ways, such as through techniques like fine-tuning, retrieval-augmented generation (RAG), or by strategically injecting it into the model’s context window. These methods allow models to specialize and adapt to specific domains and tasks, demonstrating that the value of data persists even if the mechanisms for extracting that value are evolving.

Another critical aspect of valuable data is whether it is labeled or not. While LLMs are typically pre-trained on vast quantities of unlabeled data using self-supervision, the subsequent steps of fine-tuning and reinforcement learning often rely on high-quality, labeled datasets. As highlighted in “Beyond Token Prediction: the post-Pretraining journey of modern LLMs”, much of the recent progress in GenAI has come from advances in these post-training stages, which require carefully curated datasets with explicit labels. Creating these labeled datasets, particularly at scale and with high accuracy, remains a significant challenge and a major bottleneck in many AI applications. This is precisely why companies specializing in data labeling, like Scale.ai, are seeing continued growth and high valuations.

It is important to note that the need for labeled data, while crucial, does not directly contradict the idea that the “age of pretraining data” might be waning, as Ilya suggested. Pretraining still relies heavily on the vast, unlabeled data of the public web. However, the increasing importance of post-training techniques and the value of proprietary, labeled datasets underscore a shift towards a more nuanced understanding of data’s role in the future of AI. The age of data is not ending, but it is certainly evolving, with a growing emphasis on quality, specificity, and the strategic use of data beyond the initial pretraining phase.

Human, Synthetic, and Superhuman Data

The vast majority of data currently used to train AI models is generated by humans – from the text on Wikipedia, Quora and Reddit to the images, videos, and code found across the internet. Humans also play a crucial role in creating the labeled datasets used for post-training. However, the question of whether this reliance on human-generated data can or should continue is a subject of intense debate. While it is true that much of the existing web data has been “used up” for pretraining purposes, the kind of data that future AI will need is very different.

One promising avenue, also highlighted by Ilya in his talk, is synthetic data, artificially generated data that can augment or even replace real-world data. The potential of synthetic data has been met with both excitement and skepticism. Some research, such as the paper “AI models collapse when trained on recursively generated data”, suggests that models trained solely on their own outputs can degrade. However, I believe these limitations can be overcome by generating synthetic data from diverse and more complex models, as argued in “On the Diversity of Synthetic Data and its Impact on Training Large Language Models”. There is a rich and growing body of work on how to best generate and use synthetic data that I won’t review here (see e.g. “Machine Learning for Synthetic Data Generation: A Review” and “Generative AI for Synthetic Data Generation: Methods, Challenges and the Future” ).

I believe we are rapidly approaching a turning point: the end of the age of data as defined by human limitations. While human-generated data has been essential in bootstrapping AI, it is inherently constrained by our own perceptions, biases, and the inefficiencies of human communication. As Yann LeCun has pointed out, human language is an imperfect tool for capturing the full complexity of reality. We are, in a sense, “lossy” encoders of the world around us.

The future of data lies in moving beyond these limitations. My hypothesis is that LLMs are not the end but the means to an end: they are good enough to build AI agents that will capture much better data. In fact, they will capture superhuman data. We are on the cusp of an era where AI agents, equipped with advanced sensors and sophisticated reasoning capabilities, will interact directly with the world, generating data that is richer, more accurate, and less filtered by human interpretation. Multimodal agents, such as Google Deepmind’s Astra, will be deployed in real environments to create better data and representations of the world, enabling new forms of scientific discovery, as previewed in “Scientific discovery in the age of artificial intelligence”. Consider autonomous vehicles like Waymo’s. While currently limited in scope, they represent a first step towards this future. These vehicles collect vast amounts of real-time data about the world, data that can be used to improve their own performance and, crucially, could be used to create richer datasets beyond the task of driving. In the near future, we can envision AI agents designed explicitly for data collection and generation, operating across a wide range of domains, from materials science to medical research. Some very recent interesting examples of new data include “The Well (15TB of Physics Simulations)”, “Multimodal Universe (100TBs of Astronomical Scientific Data)”, and “Genie-2”, a large-scale foundation world modelcapable of generating an endless variety of action-controllable, playable 3D environments for training and evaluating embodied agents. It is important to note that this new kind of data is not only different in scale but also in its fundamental structure and complexity. Consequently, it will demand the development of novel algorithms and model architectures capable of fully harnessing its potential.

This transition to agent-generated data will not be without its challenges. Importantly, we must acknowledge the continued need for human-generated data in specific areas, particularly for aligning AI systems with human values and preferences. Data reflecting human desires, feedback, and ethical judgments will remain crucial for ensuring that AI remains beneficial and aligned with our goals. This includes data used for techniques like Reinforcement Learning from Human Feedback (RLHF) and other alignment methods. In addition, human data will continue to be invaluable for personalizing AI experiences, ensuring that systems are responsive to individual needs and preferences. While superhuman data can provide a more accurate and comprehensive understanding of the world, it is human data that provides the crucial link to what matters to us as humans. I should also point out that “superhuman” data or AI is not the same as AGI. In fact, I am known to be a fan of the term AGI itself, and I believe that specialized, superhuman agents are likely to be both more achievable and more beneficial in addressing specific, complex challenges in alignment with our goals (see my “Beyond Singular Intelligence: Exploring Multi-Agent Systems and Multi-LoRA in the Quest for AGI”).

It is not an overstatement to say that data is not only not over, but it is in fact about to get much bigger and better, thanks to AI.

Data’s Next Chapter: From Human to Superhuman

The era of relying solely on low-quality web data for AI training is coming to a close. The future of AI will be shaped a new kind of data: superhuman data. This agentic-generated data, unconstrained by human limitations and biases, will unlock new levels of AI capabilities across a wide range of domains. Imagine a world where specialized AI agents, equipped with advanced sensors, explore the depths of the ocean, analyze complex scientific data in real-time, or even help us understand the intricacies of the human brain, generating data far beyond our current reach. This is not to say that human input will become obsolete. In fact, ensuring that these powerful AI systems remain aligned with human values and goals will be more critical than ever. By carefully curating datasets that reflect our ethical principles and desired outcomes, and by developing robust methods for incorporating human feedback into the learning process, we can guide the development of agentic AI towards a beneficial future. The age of superhuman data is not just a technological shift, it’s a paradigm shift that has the potential to revolutionize science, medicine, and countless other fields. Of course, realizing this potential will require not only new data but also continued innovation in algorithms and model design. The two will work hand-in-hand to drive the next wave of AI breakthroughs. By embracing this future, and by thoughtfully navigating the challenges it presents, we can unlock a new era of discovery and progress, driven by the power of agentic AI.