How I Evaluate AI Models for Coding Work

Every week there is a new model ranking. One model wins coding. Another wins reasoning. Another is cheaper. Another is faster. The numbers are useful, but they are not enough for real coding work.

When I choose a model for coding, I care about how it behaves inside a real repo.

Benchmarks are the starting point

I still look at benchmarks. They help filter the obvious choices.

The ones I usually care about are:

• SWE-bench style issue fixing
• Aider Polyglot style edit tasks
• Terminal task benchmarks
• Long-context retrieval tests
• Human preference leaderboards
• Price and latency comparisons

But a leaderboard cannot fully tell me if the model will work well in my codebase.

My real test

I test models with tasks that look like my actual work:

• Fix a failing test
• Add a small feature in an existing style
• Refactor a repeated pattern without changing behavior
• Read a repo and explain the architecture
• Update content without breaking routes
• Make a frontend change and inspect the result
• Handle a build failure without guessing

This shows the model’s habits.

Some models are strong at solving isolated problems but weak at following repo constraints. Some write beautiful explanations but forget to run tests. Some are fast but careless. Some are expensive but save time because they need less correction.

The best model depends on the job.

I care about context discipline

A good coding model reads before editing. It should find nearby patterns, understand naming, and respect the codebase.

I do not like when a model immediately creates a new abstraction because it looks clean in isolation. In real software, the local pattern matters more than the model’s preferred style.

Good behavior looks like:

• Read package scripts before running commands
• Read existing components before adding a new one
• Check imports before inventing helpers
• Update tests near the behavior
• Keep unrelated files untouched

This is more important than benchmark score for day-to-day coding.

Cost is not just token price

A cheap model that takes five retries can be expensive in real time.

An expensive model that solves the task cleanly can be cheaper overall.

So I think about cost like this:

real cost = token cost + latency + correction time + risk

For small tasks, fast and cheap models are great. For risky production changes, I would rather use a model that is slower but more careful.

I want models that can say no

This is underrated. A coding model should push back when the request is unclear, risky, or inconsistent with the codebase.

If it cannot verify something, it should say so. If tests did not run, it should say so. If a change touches unrelated files, it should explain why.

False confidence is one of the worst traits in a coding model.

What I compare

When I compare models, I usually track:

• Did it find the right files?
• Did it preserve existing style?
• Did it make the smallest useful change?
• Did it run the right checks?
• Did it fix errors from its own edit?
• Did it avoid unrelated refactors?
• Did the final explanation match the actual diff?

This is simple, but it gives a clear picture.

My current conclusion

There is no single best model for everything.

For code, I want a model that can work inside a harness: read files, run commands, inspect failures, and correct itself. For architecture, I want judgment. For content, I want voice control. For UI, I want visual inspection.

Benchmarks tell me where to start. Real repo behavior tells me what to trust.