Next Action Is All You Need 🪜

TL;DR — In the agentic era, the unit that matters is the next action, not the next token: something the model does to its workspace, like reading a file or running code. Yet we still prepare data and evaluate models as if they were chatbots. The question is not whether models can act. The question is whether our tasks let them.

But the models have changed.

They have learned to do things. We are still chatting with them.

The signal: in the past few months, chatbots got a computer

Over the past three months, several frontier labs did almost the same thing: they merged “chat” and “work”, and gave the chatbot a computer.

• OpenAI, July 9: launched ChatGPT Work, “an agent that can take action across your apps and files, stay with a project for hours if needed, and turn a goal into finished work.” The desktop app now puts Chat, Work and Codex in one place, on one task runtime.
• SpaceXAI, August 11: launched Grok Bot, “your team of always-on agents. They have their own computer, work inside tools and apps like you do, and keep working 24/7.”
• Meta, September 8: launched Muse. Every user gets a dedicated cloud machine running a full Ubuntu image, shared by the user and the agent.
• Anthropic, September 16: Claude Cowork and chat became one Claude. “You don’t have to choose where a task goes.” A quick question gets a quick answer; a bigger task keeps running in the cloud after you close your laptop.

The models are changing too. According to public technical write-ups, Alibaba’s Qwen3.5 series was trained with reinforcement learning in large-scale agent environments and supports tool calling natively. Models used to be trained to hold a good conversation. Now they are trained to act in an environment.

The products have changed. The models have changed. The way many domain tasks are defined has not.

That gap shapes how we define tasks, prepare data and evaluate models. Let me show it with a task from my own field.

Starting from a domain-specific problem

First, my research area. My PhD is in multimodal medical AI: in short, getting models to read medical images and turn what they see into text a clinician can use. One classic task is radiology report generation (RRG): the input is a chest X-ray, the output is a radiology report. To evaluate it, we compare the generated text with the report the radiologist wrote and score its quality along several dimensions, such as clinical entities, factual errors and semantic similarity.

While discussing this task, I noticed a problem: neither our method nor our evaluation took the workspace into account. In its plainest sense, a workspace is about one question: can the agent operate on files?

Our first, vanilla “agent” worked like this: a few models extract evidence from the image, the evidence is pasted into a prompt, and a large model writes the report. Every piece of evidence was decided by the designer; the only thing the model could do was write the next token. Call it a workflow or an agent, it was still a chatbot.

We had missed something: modern models can already work inside a workspace.

One request, two modes

The difference is easiest to see in a single, illustrative turn. In both modes, the request is the same: write the report for this chest X-ray.

On the left is the chatbot mode. The image arrives with the user message and becomes image tokens at the very first step; the model then generates the text in one pass, starting from the first token.

On the right is the workspace mode. The system message adds two tools: bash runs commands and read_file reads files. In a hospital, a chest X-ray is a DICOM file, and its header records what kind of exam it was, how it was taken and how large each pixel is. A few lines of code through bash read that header and convert the pixels into an image. When read_file opens an image, it hands the picture to the model’s own vision. The agent first works out what it has, then looks at the image, and finally writes its conclusion to a file together with the evidence.

It does not need to see better. It can look things up, then look, then check.

The file itself is information

Why does the chatbot mode not get this information? Because we threw it away before the model ever saw the data.

This is the default pipeline in multimodal research: convert the raw data into a format the model can take, compress the image into a JPG, turn it into image tokens in latent space, and let the model start from those tokens. Medical imaging is no exception. Widely used datasets convert the original DICOM files into JPGs: the header does not travel with the image, a few fields are moved into a CSV that few people read, and the physical scale and the original bit depth are gone. A large share of published work starts from those JPGs.

If you are studying the model itself, this is the right call: one format, no private information, and the only variable left is the model’s perception.

But if you are studying a task in the agentic era, multimodal understanding should not mean only image tokens in latent space. The format, the metadata and the structure of a file are information too.

Why not paste the metadata into the prompt? You can, but then the dataset author still decides once, on everyone’s behalf. In a workspace, which field to read and whether to check it against the pixels is the model’s own action.

Multimodality has not become less important. What has changed is where the image comes in: no longer as the input pushed in at step one, but as an action the agent takes halfway through its analysis, calling its own vision through read_file.

DeepSeek’s Liang Wenfeng made a closely related point at an investor meeting in May (according to a circulated transcript, full English translation): “Multimodality, for intelligence, is a component. We don’t treat it as intelligence itself.” He also said: “With the current generation of AI technology, if you can describe a problem very clearly, give it complete context and instructions, it already exceeds humans. But there’s a definition, a premise: you give it complete context, you give it complete instructions.”

Complete context is exactly what we removed, by hand, in the preprocessing step.

And this does not happen only with images. Logs get truncated, tables get serialized into text, PDFs get reduced to plain text: we keep preparing data in whatever shape fits into a chatbot’s input box.

Preprocessing decides what the model can know before the experiment even starts.

Next token, reasoning token, next action

At this point, three things that are easy to conflate need to be pulled apart.

An action is also a sequence of tokens; a tool call is generated by the model too. But it is reasoning made external: reasoning tokens change only the model’s own context, while an action changes the outside world and brings an observation back.

Reasoning tokens are thinking. Actions are doing. Thinking stays in the context; doing stays in the workspace.

The distinction is not new. In 2022, ReAct already had models interleave reasoning and acting. Two things are new: models are now trained specifically to act, and the results of their actions stay in a persistent workspace.

One more point. Chain-of-thought (CoT) is not always faithful: Turpin et al. showed at NeurIPS 2023 that the reasoning a model writes down can systematically misrepresent the real reason for its prediction. An action, by contrast, can be checked from the outside: which files it read, what code it ran and what it wrote down are all there.

From multimodal processing to long-horizon tasks

Multimodal processing is only the starting point. One image becomes twenty images; one report becomes a whole research project. Auto research is exactly such a chain: idea generation, literature review, experiments, writing, each a long-horizon task in its own right. In these tasks, whether the model writes a beautiful next token matters far less than what it does next.

In a workspace, the scripts, checklists and notes the agent writes for the first image stay there; by the twentieth image it can reuse them, fix them and build on them. This is the second meaning of a workspace: state accumulates across tasks.

The evaluation gap

If an agent’s ability lives in its actions but evaluation reads only the final text, there are at least three blind spots.

First, the score mixes in variables nobody reports. By harness I mean the layer wrapped around the model: the prompts, the tools, how the loop runs and how results flow back. Epoch AI found that on SWE-bench Verified, switching only the harness changes GPT-5’s score by up to 11% and Kimi K2 Thinking’s by up to 15%. ASI-Bench ran the same model under different harnesses on 60 research tasks: Kimi K2.7 scores 19.72 in its own Kimi Code and 27.34 in Claude Code; yet in the same paper, Kimi K3 barely moves (36.22 vs 37.09). METR likewise found that Claude Code and Codex did not beat METR’s own default harnesses (ReAct and Triframe) on time horizon, and a study on a private task suite measured only ±1.25 points between vendor and neutral harnesses. A harness does not always raise the score, but it is always part of what is being measured. As Anthropic puts it: “When we evaluate ‘an agent,’ we’re evaluating the harness and the model working together.”

Second, the process is invisible. An agent that reads the header, looks at the image and checks one against the other, and a chatbot that glances once and answers, get about the same score as long as their final text is about the same. A final score cannot tell “right” apart from “right by accident, for the wrong reasons”. AutoResearchEval analyzed 800 trajectories from 8 harness–model combinations on 100 real frontier research tasks: 77.5% contain at least one conclusion that the agent’s own method does not support, and the evidence exposing the gap sits in the run directory the agent itself created; it simply never compared them. The authors put it plainly: “an evaluation that reads only the final report cannot see any of this, because the report is one half of the disagreement and the other half is on disk. That is the case for scoring the artifacts a run leaves behind rather than the answer it ends with.”

Third, accumulation is erased. LifelongAgentBench has “lifelong” in its name, yet the paper states that “containers are destroyed and re-instantiated after each task.” Anthropic’s guide to agent evals recommends starting every trial from a clean environment, because “unnecessary shared state between runs (leftover files, cached data, resource exhaustion) can cause correlated failures due to infrastructure flakiness rather than agent performance.” Statistically, that is entirely sound. The price is that accumulation disappears from evaluation.

Time to rethink

My view is this: over the next few years, what a system can do will depend less and less on how well it writes the next token, and more and more on what it does next, and on whether we give it a place to do things. Multimodality, data and evaluation are all worth rethinking in that unit.

Two questions seem most worth our time:

• How do we evaluate an action, not just a piece of text? Can the process itself be scored: what it looked up, what it checked, what it left behind?
• How do we bring out a model’s agentic ability in a specific domain? In radiology reports, in literature review, in any long-horizon task, that is a research question in its own right.

Back to the chest X-ray from the beginning. In the hospital, it was always a DICOM file. We are the ones who turned it into tokens.

Citation

If you found this useful, please cite it as:

Zhang, X. "Next Action Is All You Need" (October 2026). https://xizhang.ai/blog/next-action/

Or use the BibTeX citation:

@article{zhang2026nextaction,
  title = {Next Action Is All You Need},
  author = {Xi Zhang},
  year = {2026},
  month = {October},
  journal = {xizhang.ai},
  url = {https://xizhang.ai/blog/next-action/}
}