LLM 101 — 02: Why Few-Shot Fails
An example is not a rule shown by instance but a complete observation — an extremely dense conditioning signal that can narrow not just the form of the answer but the model's distribution over hypotheses.
The previous note argued that prompting is conditioning: whatever enters the context changes the distribution over what comes next.
That leaves an interesting problem for few-shot prompting.
An instruction conditions the model. An example conditions the model too. In that sense, they are made of the same thing: context.
But they do not behave like the same kind of signal.
A few examples can make a task dramatically more specific — sometimes usefully, sometimes in ways we never intended.
Why?
Suppose the instruction is:
Given a problem and the available evidence, identify the most plausible causes.
A broad instruction, and deliberately so: plausible causes can live anywhere.
Now, to make the task clearer, we attach three demonstrations — all real, all correct:
Sign-up conversion dropped → new friction in the onboarding flow.
Retention fell in recent cohorts → weaker early product engagement.
Checkout completion declined → an extra step added to the payment funnel.
On the surface, the demonstrations show how to do root-cause analysis: read the evidence, name the cause, be specific.
But an example is not a rule shown by instance. It is a complete observation — an output that already sits at one specific location in semantic space. Count what these three actually say. Each has a shape: symptom, arrow, mechanism. A length: one line. A tone: confident, no hedging. And — much quieter than any of that — each one locates its cause in the same place. All three problems turned out to live inside the product. All three answers name a proximal product mechanism. All three explain a metric movement by decomposing a funnel.
Nobody decided that. It is an accident of which three cases got picked. But a demonstration cannot exist without taking a position on where causes live — to show a worked case at all, the case had to have its cause somewhere. And to the model, all of it is context. All of it conditions.
An example is an extremely dense conditioning signal.
This changes what few-shot prompting actually does.
The usual picture is that examples teach the model the method: show it a few worked cases, and it generalizes. The conditioning picture is different. The demonstrations pull the model toward the local region of semantic space around those points — not toward “root-cause analysis” in general, but toward this kind of cause, found in this kind of place, explained in this kind of way.
Now a new case arrives:
Revenue suddenly fell. Product metrics are unchanged. A major competitor cut prices by 40% last month.
The model has an enormously wide reasoning space available for this. Competition. Pricing. Seasonality. Distribution. Customer mix. Measurement error. It knows all of these; nothing about them is missing from its capability.
But the three demonstrations have already told it what a plausible explanation looks like here: a metric problem is explained by an internal product mechanism. So it goes looking for one — a subtle funnel change, a hidden engagement issue — while the 40% price cut sits in the prompt, unused.
The failure is not that the model cannot consider competition. It is that the examples changed what kind of explanation it treats as plausible. They did not just condition the form of the answer.
They conditioned the distribution over hypotheses.
The model has not lost any knowledge or skill. The probability mass has been pulled toward the region suggested by those three examples.
Few-shot fails not by teaching the model too little, but by conditioning it on too much.
And here is the tension that makes this more than a cautionary tale.
To choose those three demonstrations, you had to know what a good root-cause analysis looks like. You knew the answers should be evidence-driven, specific, mechanism-level. You knew — presumably — that causes can also live outside the product. Choosing the examples was an act of applying that knowledge.
But if you know all of that, it can be written down directly:
Consider internal and external causes — product changes, competition, pricing, seasonality, measurement. Name the specific mechanism, and weigh each candidate against the evidence.
So the conclusion is not that instructions are better than examples. It is stranger than that:
When the target space can be described explicitly, representing the entire space by a few concrete points is a very odd choice of encoding — an extremely high-bandwidth signal, most of whose bandwidth carries things you never meant to send.
This is why “just add a few examples,” as a default prompting move, deserves more suspicion than it usually gets.
But everything above leans on one phrase: when the target space can be described explicitly.
There is a kind of ambiguity this does not cover: the instruction can be perfectly clear about what the model should do, and the output can still be underdetermined.
What does an example do then?