Reasoning Trace Quality in Chain-of-Thought Training Data

OpenAI keeps the raw chain-of-thought hidden in its frontier models rather than exposing it to users. Anthropic exposes more of Claude's thinking by default. DeepSeek made R1's full traces public from the start, which is part of why so much of the distillation research in this piece leans on R1 as a source model. That's the whole debate in three sentences: how much of the reasoning a company shows is a business decision dressed up as a safety one, and it deserves to be argued that way rather than waved off as a UX detail.

Chain-of-thought used to be a prompting trick: ask a model to "think step by step" and watch accuracy climb. That's no longer what it is. In current frontier systems, CoT is the training target itself, the thing a model is directly supervised to reproduce, and that shift changes what quality means. A long reasoning trace doesn't just carry an answer to a benchmark question. It carries a trajectory the model is meant to imitate, structural flaws included. GPT-5's reasoning modes, Claude Opus 4.7's adaptive thinking, Gemini 3 Pro's deep think, and DeepSeek R1 all build extended reasoning into the model rather than bolting it on at prompt time, and the field's attention has moved with them: earlier work asked whether CoT helps, current work asks what makes one trace worth learning from and another worth throwing out.

What answer correctness tells you about a trace

The default filter most practitioners reach for is simple: does the trace land on the right answer? If yes, keep it. If no, throw it out. That gate is cheap to run and feels principled, but it only checks the last token of a thousand-token document, and that's the whole problem with it.

Two traces can both arrive at the correct answer and still teach a fine-tuned model completely different habits. One might reach the answer through a clean, load-bearing chain of steps. Another might stumble into it after a detour of dead-end hypotheses and irrelevant computation, landing right for the wrong reason. Correctness at the finish line says nothing about what happened on the way there, and prior work on trajectory style already shows that the phrasing and structure of source traces shapes what a fine-tuned model generalizes to. Treating "correct" as the only axis that matters is the mistake. It belongs alongside trace compatibility (does the reasoning style match what the base model can actually use), the soundness of the reasoning pattern itself, and learnability, whether the steps form a repeatable procedure or just a one-off coincidence that happened to work. The answer label carries none of that information.

The harmful continuation problem: what happens after the answer in a long trace

He et al. isolate a specific failure mode inside this gap: a trace can stay answer-correct while continuing to reason well past the point where the answer was already settled. The model solves the problem, and then it keeps going, generating steps that add nothing but still sit inside the supervised target.

The method is direct. A delete-only editor performs answer-preserving suffix removals, cutting the trace off once the answer is sufficiently supported, then SFT runs on both the original traces and the trimmed versions. Removing the post-conclusion continuation improves SFT outcomes, and that result overturns a comfortable assumption. Extra reasoning after the answer is active instruction, shaping what the model learns to do next. It's active instruction, shaping what the model learns to do next, and it's actively harmful, because the trace teaches the model, by demonstration, that reasoning ought to keep running past the point of resolution. Nothing in that continuation is factually wrong. It's just misplaced, and misplaced content in a supervised trace still gets reproduced.

Faithfulness: whether a trace reflects the reasoning that produced the answer

Faithfulness asks a harder question than correctness or structure: does the trace actually describe the reasoning the model used to get there? That's a demanding standard. It requires the written steps to correspond, completely and accurately, to whatever computation actually drove the output, not just to a plausible-sounding narrative wrapped around it after the fact.

Work formalizing instance-level unfaithful CoT detection across four domains and four LLMs, using a multi-stage human annotation pipeline, finds this failure to be a recurring rather than edge-case phenomenon. CoT explanations routinely fail to represent the underlying reasoning process faithfully. Part of the mechanism is post-hoc rationalization: once the model has already settled on an answer internally, the "reasoning" that follows is decoration rather than derivation, and since that decoration never changes the answer, nothing pressures the model to keep it honest. Faithfulness isn't a fixed trait of a model, either. Research shows LLMs lean on CoT heavily for some tasks and barely touch it for others, so any claim about how faithful a model's reasoning is has to be scoped to the task, never stated as a blanket property of the system.

Confirmation bias and pattern matching: how unfaithful traces teach the wrong generalizations

Unfaithful CoT is prevalent across tasks and models. It's prevalent across tasks and models, and it correlates with model size, with how much question-relevant information appears in the CoT, and with how that information interacts with the final answer. Turpin et al. document models generating reasoning that reads as plausible, internally coherent, and completely disconnected from the actual cause of the answer: the trace rationalizes a biased judgment while never once naming the bias driving it.

A parallel line of work pushes on the structural version of the same concern, asking whether pattern matching over training distributions produces the appearance of reasoning, fluent step-by-step text with nothing resembling genuine derivation producing it. Combining the two findings compounds the risk. A model trained on traces that already show confirmation bias and pattern-matched structure doesn't just fail to correct the problem. It learns to reproduce fluent, confident-sounding reasoning that rationalizes conclusions instead of reaching them. At scale, that's the training signal telling the model that sounding right is the job, full stop, regardless of whether the reasoning producing it is doing any actual work.

Ground-truth verifiability as a quality dimension: what can be checked

Verifiability draws a line between two very different kinds of check. Outcome verification asks only whether the final answer matches a known correct value. Process verification asks whether each intermediate step is grounded in something checkable independent of the final answer, a much harder bar to clear and a far more informative one once it's cleared.

Code is where this distinction stops being abstract. IBM Research points out that most synthetic CoT data for code consists of explanations that sound plausible but were never checked against what the program actually does when it runs, and models trained on that kind of data pick up logically flawed reasoning patterns even when the code itself compiles fine and passes a linter. The fix is mechanical: instrument the code to capture its dynamic behavior as it executes, narrate that execution trace into natural language, then verify each rationale against what the trace actually shows happened. The resulting dataset, 54,000 rationales, is execution-verified and bi-directional, covering forward reasoning (given this input, what output follows) and backward reasoning (given this output, what input produced it). That's process verification at scale, and it's a workable model for what verifiability looks like anywhere there's an independently checkable ground truth to narrate a trace against, not just in code.

Dataset curation strategies that operationalize these quality dimensions in practice

Curation turns this taxonomy into line-item decisions with measurable downstream effects. A handful of prominent projects show just how differently teams have chosen to handle structure, faithfulness, and verifiability, and the differences aren't cosmetic.

Distillation set the baseline pattern. Once models like QwQ and DeepSeek-R1 made their long reasoning traces visible, smaller models could train directly on that output. Datasets like OpenR1, OpenMathReasoning, and OpenCodeReasoning pull source questions from CodeForces, AoPS, and StackOverflow, while Natural Reasoning takes a different route with its own approach to source material.

OpenThoughts3-1.2M generates long-form traces across math, science, code, and puzzle prompts, verifying correctness for math, code, and puzzles before assembling the final set, but not for science. OpenThoughts3-1.2M goes further and drops answer verification entirely, a decision reflecting findings about scale-dependent behavior of verification interventions. That's an acknowledgment that a quality intervention is not universally good. It's scale-contingent, and applying it blindly at the wrong model size backfires. The 1.2M version pairs 850,000 math questions with 250,000 code questions and 100,000 science questions, generating its reasoning traces with QwQ-32B rather than R1. One finding here cuts against the grain of what most people assume about reasoning length: the relationship between trace length and correctness is not straightforward, and more reasoning is not the same thing as better reasoning. More reasoning is not the same thing as better reasoning. There's also a safety cost nobody was specifically hunting for: SFT for reasoning capability can affect a model's other behaviors in ways that have nothing to do with math or code accuracy.

S1K takes the opposite bet on volume, and it's the more defensible one for teams without frontier-scale compute. Built from just 1,000 questions, chosen for quality, diversity, and difficulty, paired with reasoning traces distilled from a teacher model, s1-32B is on the sample-efficiency frontier after fine-tuning on that set. The claim S1K makes, and the evidence backs it up, is that a model's ability to actually absorb the reasoning behavior being demonstrated depends on the quality and difficulty of the underlying prompt, not the sheer number of traces stacked on top of it. Scale for scale's sake is the weaker bet.

Where to inject high-quality reasoning traces across the training pipeline

Timing matters as much as content, maybe more. Akter et al., in the NVIDIA "Front-Loading Reasoning" paper, run the most thorough empirical test yet of where in the training pipeline reasoning data actually pays off, and the answer cuts against standard practice.

The headline finding: putting reasoning data into pretraining, rather than saving it for supervised fine-tuning later, produces an average gain of 19% and builds foundational capability that later-stage SFT cannot fully replicate, even when that SFT stage gets more data to work with. Waiting until SFT to introduce good reasoning traces is, on this evidence, too late. The paper also finds an asymmetry in what each stage responds to: pretraining rewards broad diversity in reasoning patterns, worth an 11% average gain, while SFT is far more sensitive to the quality of the data it sees, worth a 15% average gain when that data is high-quality. The least intuitive result: high-quality pretraining data has effects that stay latent until SFT activates them. The benefit isn't visible checking the model right after pretraining. Fine-tuning activates it later, once it has something to compound against.

Measuring step-level quality: process reward models versus outcome reward models

Once quality has been curated into a trace, the next question is how it gets rewarded during training, and the choice between outcome reward models and process reward models is where that plays out. ORMs score only the final answer. PRMs score each intermediate step in the trajectory, which sounds like the more rigorous approach and, in one important sense, is, though it brings its own failure modes along with it.

ORMs are anchored to ground-truth verification of the final answer, leaving less room for a model to game the signal without actually getting the answer right. PRMs, by contrast, are more prone to length hacking and verbosity bias, since step-wise optimization introduces high variance and a model can learn to pad steps in ways that look thorough without adding real derivation. But ORMs leave a gap of their own: a reward tied only to the final answer can still shape the reasoning trace in unintended directions, since nothing in the signal cares what the steps looked like as long as the ending is right. One approach to address that gap combines answer accuracy and trace validity into a single metric, checking whether sampled reasoning chains reach the correct answer while also maintaining a valid reasoning trajectory, rather than trading one off against the other.

A newer approach, generative PRMs, has the reward model write out a justification, essentially its own short chain of thought, before assigning a correctness label to a step. That approach leans on the reward model's own reasoning capacity to do the labeling work, which is intended to make process-level filtering more practical to run at scale.

Reasoning trace exposure as a secondary quality concern: what hidden traces reveal about value

A second, quieter dimension of trace quality has nothing to do with correctness, faithfulness, or verification: whether the reasoning trace gets shown to the user at all, or hidden behind the final answer. OpenAI keeps GPT-5's raw chain-of-thought hidden rather than exposing it to users. Anthropic exposes more of Claude's thinking by default. DeepSeek made R1's traces widely available, which is part of why so much of the distillation research covered above leans on R1 as a source model.

Whether a trace stays visible or gets suppressed reflects how a system's builders weigh transparency against the commercial and safety incentives to keep raw reasoning out of view, and that tension drives a good share of the product decisions across current reasoning models. It operates on a different axis from everything discussed above it: not "is this reasoning good," but who gets to see it, and what that choice says about what the trace is actually for. Treating exposure as a footnote to the correctness-and-verification conversation undersells it. It's a decision with its own stakes, made for its own reasons, and it deserves to be argued on those terms rather than folded quietly into a UX afterthought.