TokenMix Research Lab · 2026-07-15

Choosing GPT-5.6 Agent-Coding Models: Compare Effort, Not Just Model Names

A model chart on X triggered a useful discussion. The durable lesson is not that one model is always number one. For agentic coding, the model tier and the reasoning-effort setting should be evaluated together.

Summary

When people configure coding agents, they often jump straight to the most expensive model with the biggest name. Sebastian Raschka’s recent discussion of GPT-5.6, Terra, Luna, Sol, and different effort settings suggests a more practical approach: when a task does not require the very highest capability tier, start with a cheaper model and give it a larger reasoning budget.

That idea is useful, but it has to be read together with the benchmark scope and a correction to the chart. Raschka later explained that the original plot had a label shift: the highest-tier label was incorrect, and the Luna labels were also shifted by one level. The chart was modified from an OpenAI chart and annotated by matching six data points to effort levels. It should not be treated as an official, complete OpenAI configuration table.

This article turns the post and its public discussion into a practical workflow: understand what the chart measures, run a small A/B test on your own repository, and then choose a default model.

1. What the original X post says

Raschka’s recommendation can be summarized in four points:

1. Unless you genuinely need Terra Max-level performance, try Luna with a higher effort setting first.
2. If the target is roughly the Sol High performance range, try higher-effort Luna before moving up to Sol.
3. If you are considering a higher effort setting, compare it with Terra Max.
4. The additional cost of Sol Max over the next-lower setting may not be worth it.

The important change is the order of decisions:

Start with a cheaper baseline that can plausibly finish the task.
Increase effort before increasing the model tier.
Upgrade the model only when the baseline is not enough.

This is different from using the strongest model for everything. A task that only requires reading a module, changing one configuration, or adding a test may not benefit from the highest tier. It may only add waiting time and cost.

2. Read the corrected plot before copying its labels

After the original post, the author published a correction. He said the original plot contained a label shift: the highest tier should be read as Max, and the Luna labels were shifted by one level. He considered the relative comparisons between models to remain valid, but the exact effort names should come from the corrected plot.

He also suggested that the highest tier may run Max with multiple sub-agents, mainly to improve speed. That wording is qualified and belongs to the author’s interpretation; it should not be presented as a confirmed implementation detail.

Both images are preserved with this article. Use the corrected version when publishing:

Figure 1. The original image attached to the X post. Source: Sebastian Raschka.

Figure 2. The author’s follow-up correction. Use this version for publication.

3. What the benchmark actually measures

The author says the values come from the Artificial Analysis Coding Agent Index v1.1. Artificial Analysis defines the index as a composite score across three benchmarks:

• DeepSWE: software-engineering tasks, 113 tasks.
• Terminal-Bench v2: terminal use and multi-step command-line workflows, 84 tasks.
• SWE-Atlas-QnA: technical question answering and repository understanding, 124 tasks.

The page says the index uses average pass@1 across three runs of each benchmark. It is useful for quick relative comparisons, but it is not a substitute for the individual evaluations. Two systems with similar headline scores can still behave very differently on repository questions, terminal workflows, and patch implementation.

That explains one of the disagreements in the replies. A commenter argued that the chart’s placement of Fable 5 and GPT-5.5 looked wrong. Raschka answered that the result depends on benchmark composition: this chart uses a composite average, not one universal coding test.

The safer interpretation is:

This is a relative performance view for a particular coding-agent benchmark mix.
It is not a permanent ranking for every model, task, and product surface.

4. Why the model and effort setting belong on the same chart

A single score for a model hides many decisions that matter in practice. The same model can show different success rates, token usage, wall time, and cost at different reasoning-effort settings.

Think of the two variables this way:

That is why Luna with higher effort can be an attractive option for some workloads. It does not magically turn a small model into a larger one. It allocates more budget to the model’s reasoning process. For code reading, test generation, and medium-complexity changes, that may be enough.

For the selector used in this article, the effort levels are Light, Medium, High, Extra High, and Max. The highest-tier label in the original plot was incorrect, so this article does not reuse that label.

There are clear limits:

• Higher effort can increase latency and token usage.
• More effort does not improve every task; simple tasks may just become slower.
• Model knowledge, context handling, and tool-use ability cannot always be recovered by spending more reasoning tokens.
• If failure is expensive, a higher-priced model can still be cheaper than human rework.

Effort is therefore not a “higher is always better” switch. It should be chosen together with task value.

5. Turning the post into a practical routing strategy

5.1 Code reading and small changes: start with Luna

Good starting tasks include:

• explaining a module and its call graph,
• adding tests to an existing function,
• changing a configuration or small interface,
• organizing files or writing a small script.

The first goal is completion, not the highest benchmark score. Start with Luna and adjust effort after observing failures and rework.

5.2 Routine feature work: use higher-effort Luna as a baseline

For a multi-file task with a clear acceptance criterion, higher-effort Luna is a reasonable baseline. Check whether it can:

1. understand the request,
2. locate the right files,
3. produce a patch that passes the tests,
4. diagnose and recover from a failed attempt.

If it repeatedly fails near the end, compare it with Sol Medium or a nearby tier. Do not compare only the first response’s apparent intelligence.

5.3 Multi-file refactors and long debugging chains: compare Terra Max with Sol Max

When a task spans modules, changes interfaces, runs tests, and requires repeated repairs, the model ceiling matters more. Put Terra Max and Sol Max on the same task and compare:

• completion rate,
• number of human interventions,
• recovery after failure,
• total wall time,
• actual cost.

Terra Max may be the better production default if it finishes the same work at a materially lower cost. A cheap model is not automatically a good choice if it repeatedly loses the thread of the architecture.

5.4 High-value tasks that are genuinely stuck: consider Sol Max

Sol Max is better treated as an escalation tier than as the default for every task. It is worth testing when:

• failure has a high human or business cost,
• complex reasoning is the main bottleneck,
• avoiding one manual intervention saves substantial time,
• the task value justifies the additional spend.

If Max already finishes the task reliably, the extra cost of the top setting needs to be justified by fewer failures or less rework, not by its name.

6. Four questions that matter in the replies

Question 1: Is this a model test or an agent test?

Both are involved. Raschka said GPT-5.6 was tested with Codex and the Claude models with Claude Code. The evaluation therefore includes a specific coding-agent harness, not just a raw model completion.

Artificial Analysis also has a harness comparison showing that one underlying model can perform differently across Claude Code, Cursor CLI, and OpenCode. The same model name does not imply the same tool loop, context assembly, terminal interaction, or completion path.

When applying the chart to Codex, ask:

Am I using the same harness?
Is the model and effort level actually available in my entry point?
Are permissions, context limits, and default instructions comparable?

Question 2: Why can Luna not simply scale to more tokens?

A commenter asked whether Luna’s curve could continue upward and whether context length was the limiting factor. Raschka gave a cautious answer: the slope suggests that more improvement may be possible, but it is not necessarily a hard context-length limit. It could also relate to the token budgets used during training or to the way inference scaling is implemented.

This is a useful boundary. A curve can show how performance changes with effort; it cannot, by itself, reveal training details, parameter count, or internal inference architecture.

Question 3: Why do Fable 5, GPT-5.5, and GPT-5.6 provoke disagreement?

Because a composite score averages different task types. One system may be better at repository Q&A, another at terminal operations, and another at reliable patching. Compressing those capabilities into one point hides the differences.

Artificial Analysis recommends reading the per-benchmark breakdowns alongside the headline index. For a real project, choose the evaluation dimension that resembles your workload instead of treating one score as a universal answer.

Question 4: Is the chart an official complete OpenAI release?

That is not the safest description. Raschka explained that he modified an official OpenAI chart and added labels by matching six data points to effort levels. OpenAI’s original post provides an official comparison involving GPT-5.6 Sol, but the chart with Sol, Terra, Luna, and the effort annotations is a reworked version.

That is why the article should preserve all four links:

• the original X post,
• the correction,
• the OpenAI source post,
• the Artificial Analysis benchmark page.

Clear provenance helps readers distinguish original measurements from interpretation.

7. A reproducible A/B test for your own repository

If you want to apply this strategy to your codebase, use a small controlled test.

Step 1: Fix the tasks

Choose three to five real tasks rather than inventing an impressive-looking prompt. Include a small bug fix, a cross-file feature, a test-completion task, a repository-understanding question, and a terminal workflow if relevant.

Step 2: Fix the harness and input

Use the same Codex, Claude Code, or other agent entry point for the comparison. Fix the repository commit, initial files, prompt, tool permissions, and acceptance criteria.

If you change the harness, treat it as a new experimental variable. Do not attribute the entire difference to the model.

Step 3: Run a cheaper baseline first

Start with Luna or the currently available mid-tier model at a reasonable effort level. Record the result before changing every setting after the first failure.

Step 4: Record four categories of metrics

Step 5: Change one variable at a time

If Luna fails, raise effort first. If it still fails, compare Terra. Only then compare Sol Max. This makes it possible to tell whether the gain came from a larger model or from a larger reasoning budget.

8. Benchmark score and real cost are different things

The Artificial Analysis page presents performance, token usage, cost per task, and execution time, but these metrics should be read separately. Its cost-per-task number is primarily pay-per-token API cost. It is not consumer subscription pricing and does not include engineering, infrastructure, supervision, or rework.

Real cost is closer to:

input tokens
+ cache behavior and cache charges
+ output tokens
+ retries
+ agent runtime
+ human review and rework

A more expensive model can have a lower cost per completed task if it avoids several retries. A cheaper model can become expensive when its patches require repeated manual repair.

If you use an OpenAI-compatible multi-model gateway such as TokenMix, you can keep the same workflow and compare models in the current catalog while recording request logs, usage, and per-call cost. The useful question is not which model is permanently best; it is which model, effort setting, harness, and budget produce the most reliable completed task. Availability follows the current TokenMix model catalog and pricing page.

Conclusion: choose a default configuration, not a permanent champion

The most useful idea in the X discussion is to turn model selection into a multi-variable decision:

model tier × effort × agent harness

The four rules in the original post are a good starting heuristic, but the correction and the replies define its limits. The chart had a label shift, the benchmark is a composite of several evaluations, harness choice affects results, and Luna’s future scaling behavior cannot be inferred from a single curve.

My practical rule is simple:

Run a real task with the cheaper model first, then use data to justify an upgrade.

The best default is not necessarily the model at the top of a chart. It is the configuration that reliably delivers in your repository, through your agent entry point, within your budget.

Sources and further reading

1. Sebastian Raschka: original X post
2. Sebastian Raschka: corrected plot and label note
3. Sebastian Raschka: chart source and effort-label explanation
4. OpenAI: GPT-5.6 Sol post
5. Artificial Analysis Coding Agent Benchmarks

This article summarizes public posts, public replies, and the benchmark page. Model names, effort levels, prices, and product availability can change; verify them against the current product surface and a small real-world test before publishing results.