Can AI write code? (honest limits)

A plain-language guide to code-generating models as next-token engines that can draft and explain: with human review: using Codex/HumanEval as the door into measured benchmarks and why fluency is not production-ready ownership.

Primary source: Chen et al., Evaluating Large Language Models Trained on Code (Codex / HumanEval, 2021): https://doi.org/10.48550/arXiv.2107.03374

Can AI write code?

Sometimes, drafts, boilerplate, explainers, a first stab at a function, with a human who reviews, runs, and owns the result. It is not a senior engineer who understands your system, signs off on security, or stays on call when the deploy breaks.

Nothing here is conscious. Coding assistants are one product family on language models. They are not the whole of AI. For the chat shape many people meet first, see What ChatGPT is (and isn’t).

The short version

1. Under the hood it is still next-token prediction, often specialized on code, same family as chat, different data and eval.
2. Useful for scaffolding, translating snippets, explaining unfamiliar APIs, and rubber-ducking, not for unreviewed merges.
3. HumanEval-style benchmarks measure whether generated functions pass unit tests on short problems. That is not your production codebase.
4. Limits: invented APIs, insecure patterns, code that “looks right” and fails your edge cases.
5. Fluency ≠ correctness, What hallucination means. Eval design matters, Why evaluation matters.

Same engine, code-shaped practice

A coding model is usually a language model that saw a lot of source (and docs) in training or finetuning. Given a comment, a docstring, or a partial file, it predicts likely next tokens, which happen to be syntax more often than English prose.

That is the pretrain-then-specialize story again: general patterns first, then more weight on code-like jobs. Door: Pretrain then finetune. Chat packaging and instruction tuning can make the same family feel like a pair programmer, still prediction plus product rules, not a colleague with judgment.

What “can write code” means in a paper

The primary source for this piece is Chen et al., Evaluating Large Language Models Trained on Code (2021), the Codex paper that also introduces HumanEval.

They finetuned GPT-style models on public GitHub Python and asked a sharper question than “does the code look pretty?”: does a generated function pass unit tests? HumanEval is a hand-written set of 164 programming problems (docstring in, function body out) built so solutions are less likely to be memorized copies from the web.

From their abstract, with a single sample, their Codex model solved 28.8% of HumanEval problems (vs 0% for GPT-3 and 11.4% for GPT-J on the same setup). With 100 samples per problem and picking ones that pass tests, they report solving 70.2%. Those numbers are a 2021 benchmark snapshot, not a promise about today’s tools, and not a claim about your repo.

Gloss: pass@k means “among k tries, did any sample pass the tests?” Useful for research. Easy to overread as “the model knows how to program.”

Where helpers actually help

Used as a drafting tool, coding models can:

• Sketch boilerplate and glue code you would write anyway
• Explain a dense function in plainer words
• Suggest tests or refactors you still verify
• Translate “roughly this algorithm” into a starting implementation

The win is speed on work you can check. The loss is when the check is skipped.

Honest limits (read these twice)

• Hallucinated APIs. Smooth calls to functions, flags, or packages that do not exist, or existed in another version. Fluency sells the fiction; see What hallucination means.
• Security bugs. Models trained on public code can reproduce insecure patterns (weak crypto habits, injection-friendly snippets, over-broad permissions). Review like you would a stranger’s PR.
• Wrong for your codebase. Benchmarks are short, self-contained tasks. Your app has legacy modules, internal libraries, latency budgets, and unspoken conventions the prompt never stated.
• HumanEval ≠ production. Passing docstring puzzles is not owning migrations, incident response, or design tradeoffs. Chen et al. themselves discuss limits: long chains of operations, binding the right steps to the right variables, and broader safety/security impacts of deployable codegen.
• Over-trust. The paper’s hazard analysis flags over-reliance, code that looks right and isn’t. Novices are especially exposed; experts can still automate-bias past a subtle bug.

Eval is the difference between demo and job

A viral clip of an assistant writing a snake game is a demo. A serious claim needs a test: unit tests, typecheckers, linters, security review, and staging traffic that matches reality.

Why evaluation matters is the Easy door: training fits the homework; evaluation asks whether the test is the real job. For code, “compiles” and “looks idiomatic” are weak exams. Prefer functional checks, and remember that weak tests can bless wrong programs.

Easy → technical doors

The honest caveat

AI can draft code the way it drafts prose: sometimes useful, often fluent, never automatically correct. Treat assistants as fast juniors who need review, tests, and ownership that stays with you. HumanEval-style scores show progress on short functional puzzles; they do not hand anyone a senior engineer. On this site, Easy reading aims for clear claims, clear limits, and a path to the paper when you want to verify.