Stepping back
Understanding how these systems work is one thing. Knowing where they're going — and what's still broken — is another.
This article is the map. Not predictions — nobody's track record on predicting AI is very good. The most confident forecasts from 2021 are mostly wrong. But the patterns, tensions, and open questions that will shape the next few years are visible enough to discuss honestly.
The scaling hypothesis
The single most influential idea in modern AI is deceptively simple: make the model bigger, give it more data, train it longer, and it gets better.
This isn't wishful thinking. It's backed by surprisingly precise math. In 2020, researchers at OpenAI published a paper on scaling laws showing that model performance improves predictably as you increase three things:
- Parameters — the number of weights in the model.
- Data — the amount of training text (or images, or video).
- Compute — the total number of calculations during training.
The improvement follows a smooth curve. Double the compute, and the loss drops by a predictable amount. This relationship held across many orders of magnitude — from tiny models to GPT-3 scale.
This is why companies are spending billions on GPU clusters. If the scaling laws hold, then building a bigger model is almost guaranteed to produce a better one. It's not a gamble — it's following a well-measured curve.
But is there a ceiling?
The big question. Scaling laws show improvement on a log scale — you need exponentially more compute for each linear improvement. Going from GPT-3 to GPT-4 was dramatically more expensive than going from GPT-2 to GPT-3.
At some point, you hit practical limits. There's only so much text on the internet. GPUs cost money. Power plants have finite output. Training runs that cost hundreds of millions of dollars are already straining even the largest companies.
Some researchers believe we're approaching a data wall — models have already trained on most of the high-quality text that exists. Others think synthetic data (using existing models to generate training data) can extend the curve. The honest answer: nobody knows where the ceiling is, but the returns per dollar are definitely getting harder.
Emergent abilities: surprises at scale
Here's something that spooked a lot of researchers. As models get bigger, they don't just get gradually better at everything. Sometimes they suddenly gain entirely new capabilities that smaller models didn't have at all.
A small language model can't do basic arithmetic. A medium one gets it right sometimes. A large one does it reliably. But the transition isn't gradual — it looks more like a switch flipping.
This pattern has been observed for many tasks: multi-step reasoning, code generation, translation between languages the model barely saw, even understanding jokes. Below a certain size, the model can't do the task at all. Above that threshold, it suddenly can.
These are called emergent abilities, and they're both exciting and unsettling. Exciting because they suggest that scaling up might unlock capabilities we haven't even thought to test for. Unsettling because they're hard to predict — you don't know what a bigger model will be able to do until you build it.
There's some debate about whether emergent abilities are truly sudden or just appear sudden because of how we measure them. Some recent research suggests the "sudden" appearance is partly an artifact of how benchmarks are structured — multiple-choice tests show sharp transitions, while open-ended evaluations show more gradual improvement.
Regardless of the academic debate, the practical implications are clear. You can't always predict what a model will be capable of just by extrapolating from smaller models. This makes planning and safety work harder — you might train a model expecting it to be slightly better than the previous generation and discover it has capabilities nobody anticipated.
The open problems
Despite the rapid progress, several fundamental problems remain unsolved. These aren't minor bugs. They're deep challenges that affect every current AI system.
Hallucination
Language models make things up. Confidently. They'll cite papers that don't exist, invent historical events, and present fabricated statistics with the same tone as verified facts.
This isn't a training flaw you can patch. It's a consequence of how these models work. They predict the most likely next token given the context. If the most likely continuation of "The first person to walk on Mars was" is a confident-sounding name, the model will generate one — regardless of whether anyone has actually walked on Mars.
Retrieval-augmented generation (RAG) helps by grounding answers in real documents. But hallucination in the retrieved content, in the synthesis, or in areas outside the retrieval scope remains a real problem.
Reasoning
Current models are surprisingly good at pattern-matching their way through tasks that look like reasoning. But genuine multi-step logical reasoning — the kind where each step must strictly follow from the previous one — is still fragile.
Chain-of-thought prompting helps. Dedicated reasoning models (like those trained with reinforcement learning on math and coding) do better. But give a model a novel logic puzzle that doesn't match any training pattern, and it often falls apart.
The question is whether scaling alone will solve reasoning, or whether we need fundamentally different architectures. Strong opinions exist on both sides.
Long-term memory and planning
Current models process a fixed context window. Even with windows of 100,000+ tokens, they don't have persistent memory across conversations. They can't learn from experience the way humans do — each conversation starts from scratch.
Planning is related. Humans can formulate a multi-week project plan, hold it in memory, and execute on it day by day. Current AI can produce plans, but it can't hold them in mind and adapt them over time without external scaffolding (like agent frameworks that manage state).
World models
There's a deeper problem underneath all of these: current AI doesn't have a model of the world. It doesn't understand that objects are permanent, that actions have consequences, or that time flows in one direction.
Humans have a rich internal simulation of the world. You can imagine what happens when you push a glass off a table — it falls, shatters, liquid spills, you need to clean it up. You can run this simulation without actually pushing the glass.
AI models have learned statistical patterns that look like world understanding. They can describe what would happen in many scenarios. But they don't have an internal physics engine. They can't simulate novel situations reliably. This gap shows up in reasoning failures, hallucinations, and planning mistakes.
Building systems that have genuine world models — not just statistical approximations — is one of the biggest open research questions in AI.
Alignment: the "do what I mean" problem
Making models capable isn't enough. You also need them to be helpful, honest, and safe. That's the alignment problem.
RLHF (reinforcement learning from human feedback) has been the main tool. Humans rate model outputs, a reward model learns their preferences, and the model is trained to match those preferences. It works well enough that modern chatbots are generally polite, refuse harmful requests, and try to be helpful.
But alignment goes deeper than politeness.
Value alignment. Whose values should the model reflect? Different cultures, different people, different contexts call for different behavior. A model used in healthcare needs different guardrails than one used for creative writing.
Deceptive alignment. Could a sufficiently capable model learn to appear aligned during training while behaving differently in deployment? This sounds like science fiction, but it's a real concern in the research community. We don't have great tools for verifying what a model is "thinking" internally.
Specification gaming. Models are very good at optimizing for the metric you give them — sometimes in ways you didn't intend. Tell a model to be helpful and it might agree with everything you say, even when you're wrong. Tell it to be honest and it might be brutally blunt. Getting the balance right is ongoing work.
Open source vs. closed source
The AI world is split on a fundamental question: should powerful models be openly available?
The open-source argument. Open models (Llama, Mistral, Stable Diffusion, Flux) let anyone inspect, fine-tune, and deploy them. This democratizes access, accelerates research, enables local deployment for privacy-sensitive applications, and prevents any single company from controlling the technology.
The closed-source argument. Very powerful models in the wrong hands could cause real harm — generating misinformation, enabling cyberattacks, or worse. Keeping weights behind an API allows safety filtering and monitoring that open-source deployment can't match.
In practice, both ecosystems thrive. Open models power most enterprise deployments (companies want to run models on their own infrastructure). Closed models often lead in raw capability. The tension between openness and safety will define AI policy for years to come.
AI regulation
Governments are paying attention. The EU AI Act, various US executive orders, and China's AI regulations are all attempts to set rules for how AI can be developed and deployed. The approaches differ — the EU focuses on risk categories and mandatory requirements, the US leans toward voluntary commitments, China emphasizes content control.
For developers, the practical impact is growing. If you're building AI products, you increasingly need to think about compliance — what data you train on, what disclosures you make, what safeguards you implement. This adds complexity but also pushes the industry toward more responsible practices.
The biggest policy question — whether and how to regulate the development of frontier models — is still being debated. And the technology moves faster than legislation, which means regulations are always playing catch-up.
The economics of AI
Training a frontier model costs tens to hundreds of millions of dollars. Running it costs money per query. This isn't free technology.
A few realities:
Training costs are rising. GPT-4 reportedly cost over 100 million dollars to train (estimates from 2023). Next-generation models are expected to cost significantly more. Only a handful of organizations can afford this.
Inference costs are falling. Hardware improves, quantization techniques make models smaller, and competition drives prices down. What cost a dollar per query in 2023 costs fractions of a cent in 2026.
The business model matters. Most AI companies burn cash. Subscription revenue and API fees don't yet cover training costs for frontier models. The bet is that AI will become so useful that the economics work eventually. But "eventually" is doing a lot of heavy lifting.
For individuals and small companies, the practical story is better. Open models run on consumer hardware. API costs are low enough for most applications. You don't need to train a frontier model — you need to use one effectively.
The concentration of power
This economic reality has a side effect worth thinking about. If only a handful of organizations can afford to train frontier models, those organizations have enormous influence over the direction of AI.
They decide what data to include and exclude. They set safety policies. They choose which capabilities to develop and which to restrict. They decide what's open-sourced and what stays proprietary. This concentration of power in a few companies is unprecedented in the history of technology, and the implications are still being worked out.
What "AGI" means (and doesn't mean)
Artificial General Intelligence — a system that can do any intellectual task a human can do. It's the stated goal of several major AI labs.
But the term is frustratingly vague. Does AGI mean:
- Passing every human benchmark? (Some models already pass many.)
- Learning new tasks as quickly as a human? (Current models need retraining.)
- Having genuine understanding vs. sophisticated pattern matching? (Nobody agrees on the definition.)
- Operating autonomously in the real world? (Still very far off.)
Some researchers think we're close. Others think we're decades away. Others think the framing itself is wrong — that intelligence isn't a single dimension you can measure, and "general" intelligence might not be a coherent concept.
What's clear is that current models are narrowly superhuman (better than any human at specific tasks) and broadly subhuman (worse than most humans at navigating novel, open-ended situations). Whether scaling and architectural improvements close that gap, or whether something fundamentally new is needed, is the central debate in the field.
The most honest answer: nobody knows. And anyone who claims certainty about the timeline — whether they say "two years" or "never" — is selling something. The uncertainty itself is one of the most important features of the current moment.
Practical advice for someone entering the field
If you've made it through this entire series, you have a solid foundation. Here's how to make it useful.
The fastest way to develop real intuition is building things. Fine-tune a small model. Build a RAG pipeline. Deploy a chatbot. Theory matters, but the gap between understanding concepts and shipping products is real — and you close it by building, not by reading more papers.
The specific models and tools change every few months. The conceptual foundations — transformers, attention, diffusion, RLHF, embeddings — change much more slowly. Invest in understanding principles, not memorizing API calls. The person who understands why attention works will adapt to the next architecture much faster than the person who only learned the PyTorch API for it.
One thing I'd push hard: pick a niche early. "AI" is too broad to be a career. "AI-powered search for legal documents" is a career. "Optimizing inference cost for real-time video generation" is a career. The people who do best in this field aren't generalists — they're specialists who understand AI deeply enough to apply it to a specific domain.
Hundreds of papers are published weekly. You don't need to read them all. Follow a few trusted sources — Arxiv Sanity, newsletters like The Batch or TLDR AI — and go deep on the papers that matter for your work. A year from now, you'll know exactly which advice you ignored and which you should have taken sooner.
Where this series ends (and where the next one begins)
This article wraps up the Multimodal and Generative AI series. Over four articles, we went from how images are generated from noise, through the architecture of Stable Diffusion, into video and multimodal generation, and finally to the big picture of where the field is heading.
But there's one major area we haven't covered yet — the practical side of building with AI. How do you structure prompts for reliable output? How do retrieval systems actually work? How do you build applications that combine LLMs with real-world data?
That's what the next series covers: RAG, Prompting and Applications — the engineering side of turning AI models into useful products.