Wax On, Weights Off: Knowledge Distillation Explained

Jan Daniel Semrau (MFin, CAIO) Jul 20, 2026 3 1 Share No such thing as bad student, only bad teacher.” — Mr. Miyagi, The Karate Kid (1984)

Supervised learning from a more experienced teacher is a credible strategy since the dawn of time. If it were fathers teaching how to hunt, the Platonic Academy teaching mathematics, or Ronald Fisher designing Linear discriminant analysis in 1936.

In the same vein, Daniel LaRusso in the 1984 movie did not rediscover karate from first principles. He painted the fence, waxed the car, and absorbed Miyagi’s decades of hard-won motion over a single intense summer.

The apprentice does not repeat the master’s mistakes. Copy the expert, and you compress a lifetime of trial and error into a few months.

But now you have millions of students harvesting the teacher’s intelligence on an industrial scale. Pretty much all relevant Chinese Frontier Labs have been accused of mind-raping Claude to speed up their training and especially Anthropic stopped having it. In February 2026, Anthropic announced it had caught three Chinese labs, DeepSeek, Moonshot, and MiniMax, running industrial-scale campaigns to extract Claude’s capabilities. As a result, we have seen Anthropic and the US Department of Commerce taking action and restricting (and later re-establishing, probably due to better guardrails and revenue concerns) access to their most powerful models.

In this post, I will be explaining what knowledge distillation is, how it helps you in post-training, and what the implications on the tech stack of frontier labs is.

Competitive Advantage or Capability Trap?

In the original Hinton (2015) sense, distillation is training a smaller student model to reproduce a larger teacher model’s behavior, instead of learning the underlying task from scratch.

Starting from the trained baseline full network, the specialists train extremely fast (a few days instead of many weeks for JFT). Also, all the specialists are trained completely independently .

Naturally, in an ultra-competitive world where weeks ahead mean billions of dollars in revenue, speed matters. If a frontier model is reachable through an API, it can be distilled by querying it at scale and training on the transcripts. That single fact turned distillation from a lab technique into a competitive weapon.

This is why DeepSeek-R1 became such a flashpoint in early 2025.

Before we go further, it helps to know that “distillation” is not one technique and has expanded from Hinton’s initial concept.

Response-based distillation is the simplest and most commonly used distillation method today. The student trains on the teacher’s outputs. In the strong form, it matches the teacher’s full probability distribution over tokens, the soft labels Hinton originally described, which carry far more information than a single right answer because they encode how confident the teacher was and what it almost said instead. In the weak form, when all you have is an API that returns text, the student trains on the generated completions themselves.

Feature-based distillation goes deeper. Instead of matching outputs, the student matches the teacher’s internal representations, the activations in its hidden layers. While this transfers richer structure, it requires access to the teacher’s internals. You cannot do it through an API. And therefore is much more expensive and risky. It is a technique for labs distilling their own models, not for harvesting someone else’s.

Relation-based distillation is subtler still. It teaches the student to reproduce the relationships between the teacher’s representations, how examples sit relative to each other, rather than the representations themselves. It is more of a research direction than a workhorse, but it matters because it points at what distillation is really transferring: not answers, but structure.

The US/China capability war lives almost entirely in the first category. Response-based distillation is the only flavor you can run against a closed model you do not own, which is exactly why it works. And it has worked for China for almost 3 decades. While China established the socialist market economy already in 1978, the true opening to the world only happened in 2001 when China joined the World Trade Organization. If you wanted to invest in the growing Chinese market, foreign firms had to enter joint ventures with Chinese partners, often holding at least 51% to keep control. This facilitated the transfer of technology and know-how to domestic companies, to the effect that many products can not be built in the West anymore.

Of course, Chinese companies, due to their own ingenuity and innovative spirit, have by now surpassed most Western companies in quality and price.

That should not excuse executing a distributed attack to gain an edge in the frontier wars.

Dorothy, we are not training narrow logistic regression models for Credit Scoring anymore. Distillation against a closed model takes the shape of a distributed attack and not a single exploit because there is no choke point. Thousands of API accounts, each issuing traffic that looks unremarkable, maybe even successful for the AI provider, on its own, collectively harvest the model’s behavior across the task distribution. This form of IP-Theft is much easier. There is no insider access. No smuggled hard drive. No post-resigning access to company resources. Just API keys and a prompt strategy. The product surface is the attack surface. API abuse is remarkably easy to track and might also suffer from stylistic fingerprinting or watermarking . Therefore, a serious attacker needs to spread data collection across accounts, IPs, and time. Then paraphrases the outputs to defeat. A team running 50,000 API calls to build an eval harness might look identical to a team running 50,000 calls to build enterprise software.

To answer this question, we have to go one level deeper in understanding how these attacks work.

But let’s start with what makes them not work.

You do not randomly ask a handful of manually selected questions. You use self-instruct loops or Evol-Instruct-style prompt mutation to sample the teacher broadly, covering the task distribution instead of the easy cases.

When executing the Self-Instruct loop, you seed a small set of instructions, prompt a model to generate new ones in the same style, filter duplicates/low-quality outputs, generate answers for the survivors, fold them back into the seed pool, repeat. Optimizes for diversity of tasks.

seed_pool = load_seed_instructions() # ~100-200 human-written for iteration in range(N): sampled = sample(seed_pool, k=8) new_instructions = model.generate( prompt=f"Given these examples: {sampled}, generate 5 new diverse instructions" ) new_instructions = dedupe(new_instructions, seed_pool, threshold=0.7) new_instructions = filter_quality(new_instructions) for instr in new_instructions: output = model.generate(instr) seed_pool.append((instr, output)) save(seed_pool) Then Evol-Instruct mutation is taking an existing instruction and rewrites it via fixed mutation templates. Then you add a constraint, deepen reasoning, concretize, broaden to a new domain, then filter and feed survivors into the next generation. Optimizes for difficulty progression on existing tasks.

mutation_prompts = { "deepen": "Add constraints or reasoning steps to make this harder: {instr}", "concretize": "Replace general concepts with more specific ones: {instr}", "broaden": "Create a new instruction in a different domain, similar difficulty: {instr}", "increase_reasoning": "Rewrite to require multi-step reasoning: {instr}", } pool = load_seed_instructions() for generation in range(N): next_gen = [] for instr in pool: mutation = random.choice(list(mutation_prompts)) evolved = model.generate(mutation_prompts[mutation].format(instr=instr)) if passes_quality_check(evolved) and not is_too_similar(evolved, instr): next_gen.append(evolved) else: next_gen.append(instr) pool = next_gen save(pool) Together, this is called Query Diversification .

If you only train on prompt and response pairs, you are teaching the student only what the answer is, not how to arrive at that answer. Therefore, you need to enrich the captured signal with full reasoning traces .

While you need the full reasoning traces, these raw scrapes are extremely noisy. To improve this, you need to apply data engineering methods to filter and deduplicate your data points . Two useful methods are embedding-based clustering for killing near-duplicates and a heuristic approach to strip out the low-value samples. Sometimes LLM-as-a-judge is used, but I am not a fan of this method.

Then you can start training the student model . Here, you perform supervised fine-tuning on the harvested pairs, often followed by preference optimization like DPO or KTO if you also captured comparative judgments. Because the student is fitting an already-compressed function instead of learning from raw reality, this converges far faster than pretraining ever could.

In theory, that’s fine. Let’s build a training pipeline.

If you draw our pipeline as a diagram, most people put the training run in the center. That is the wrong center.

By 2026, the training run is the easy part. A night on rented H100 GPUs with an off-the-shelf recipe does the trick for most builders. The real system, as most of you have already expected, is a data-engineering beast. Almost every design decision that matters is a decision about data.

At the front is the prompt generation layer . This is the engine that decides what to ask the teacher, and it is where the whole attack lives or dies, because your student can only be as good as the slice of the teacher’s behavior you managed to sample. The naive approach would be to ask whatever questions the AI engineer happens to come up with and end up with a student who is sharp on trivia and useless everywhere else. The better approach is to treat coverage as an optimization target. You seed with a taxonomy of the capabilities you actually want, then run a self-instruct loop (see above) where an LLM generates thousands of task variants from a small set of hand-written exemplars, then layer Evol-Instruct on top to mutate each prompt along two axes, deeper (add constraints, edge cases, multi-step reasoning) and wider (rephrase into new domains). The output is not a prompt list. It is a synthetic distribution engineered to tile the region of capability space you are trying to steal.

Next is the collection layer , and this is where you fight the defenses. As mentioned, you want the richest signal the API is willing to surrender. Full outputs at minimum, chain-of-thought traces if the teacher exposes them, and log-probs or top-k token distributions if you are lucky enough to get them, because soft targets carry orders of magnitude more information per sample than a hard label does. This layer is also where the “distributed” in distributed attack becomes concrete: request volume is spread across many accounts, IP ranges, and time windows to stay under per-key rate limits and below the anomaly-detection threshold, with retry logic, temperature sweeps to capture output diversity, and sometimes a paraphrase pass to launder stylistic fingerprints out of the harvested text before it ever hits storage.

Then the curation layer , which is the part amateurs skip and professionals obsess over. Raw harvest is 60 to 80 percent waste, and training on the waste actively poisons the student. So you build a filtering cascade. Exact and near-duplicate removal through embedding-based clustering, so the model does not overfit to whatever the prompt engine happened to over-sample. Quality scoring, increasingly by an LLM judge rather than heuristics, to drop incoherent, refused, or low-value completions. Correctness filtering wherever you have ground truth, so on math and code, you keep only the traces that actually verify, which is the single highest-leverage move in the entire