Essay Data Infrastructure
Distributed Data Preprocessing Pipelines for Petabyte Corpora
Exact and fuzzy deduplication bottlenecks the entire pipeline at petabyte scale.
Annotation Methodology · Data Infrastructure · Model Capability and Data
A blog and newsletter on AI training data quality and research methodology and Teaching models to perform complex, real-world work.
Essay Model Capability and Data
Reasoning Trace Quality in Chain-of-Thought Training Data
Correct answers alone don't guarantee models learn the right reasoning patterns.
Long read Annotation Methodology
Annotator Disagreement as a Signal in RLHF Pipelines
Disagreement between annotators reveals genuine value differences, not errors to eliminate.
That assumption is wrong now, and clinging to it produces Meta's number.