SFT vs RLHF: What Data Each Needs
SFT learns from demonstrations: a prompt and the answer an expert would write. RLHF learns from preferences: which of two answers is better. What each dataset looks like, how much you need, and what to check before you buy.

The short answer
SFT (supervised fine-tuning) needs demonstrations: a prompt plus the answer you want the model to give, written or approved by someone who knows the task. RLHF (reinforcement learning from human feedback) needs preferences: a prompt, two or more model answers, and a human judgment of which one is better. SFT teaches the model what a good answer looks like. RLHF teaches it to tell good from better. Most production pipelines use both, in that order.
If you are fine-tuning a model or buying training data for one, the choice between SFT and RLHF is mostly a choice about data. The algorithms are well documented. What usually decides the result is who produced the examples, how consistently they were judged, and whether they cover the cases your users actually bring.
This guide covers what SFT data and RLHF data look like, how much the best-known papers used, where newer methods like DPO fit, and what to check before you pay for either.
What is the difference between SFT and RLHF?
SFT trains on correct answers. RLHF trains on comparisons between answers. That one difference changes almost everything about the dataset:
- Unit of data: SFT uses a prompt and a target response. RLHF uses a prompt, several candidate responses and a ranking or pairwise choice.
- Who writes the answer: in SFT, a person (or a vetted source) writes the target. In RLHF, the model writes the answers and the person only judges them.
- What it teaches: SFT teaches format, task behavior and domain knowledge. RLHF teaches finer qualities that are easier to recognize than to write: helpfulness, tone, safety, when to refuse, which of two correct answers is clearer.
- Training mechanics: SFT is ordinary supervised learning on the target text. Classic RLHF first trains a reward model on the comparisons, then optimizes the language model against that reward model with reinforcement learning (PPO in the InstructGPT paper).
- Typical failure: weak SFT data teaches the model to copy mistakes and inconsistent formats. Weak RLHF data teaches it to please the reward model rather than the user (often called reward hacking), for example by getting longer or more agreeable instead of more correct.
What does SFT data look like?
An SFT record is a prompt and the response you would be happy to ship. For chat models it is often a short multi-turn conversation where only the assistant turns are trained on. "SFT data" in practice means instruction-response pairs, worked solutions, annotated walkthroughs and domain demonstrations.
Here is an illustrative record (an example of the shape, not data from a real project):
- Prompt: "Review this RTL module for a 4-bit counter and list any bugs."
- Target response: a hardware engineer's review that names the reset bug, explains why it fails in simulation, and gives the corrected line.
- Metadata: domain (digital hardware), author expertise, difficulty, rubric version, license.
What makes SFT data good:
- Correctness: every target answer is something an expert would sign off on. The model will imitate errors as faithfully as it imitates good work.
- Consistency: one format, one level of detail, one voice per task type. Mixed styles produce a model that picks one at random.
- Coverage: examples span the real distribution of requests, including the awkward ones, not only the easy cases that are cheap to write.
- Reasoning shown where it matters: for math, code, law or engineering, a worked solution teaches more than a bare final answer.
Quantity matters less than people expect. In the LIMA paper, Meta researchers fine-tuned a 65B LLaMa model on only 1,000 carefully curated prompt-response pairs, with no reinforcement learning, and reported strong results. In InstructGPT, OpenAI's SFT set had about 13k training prompts with demonstrations written by labelers. Both numbers are specific to those studies, but they point the same way: a smaller set of excellent demonstrations beats a large set of average ones.
What does RLHF data look like?
An RLHF record is a prompt, two or more responses from the model being trained, and a human judgment between them. Annotators either pick the better of two responses or rank several. Good datasets also capture why: a short rationale or rubric scores for accuracy, helpfulness and safety.
An illustrative record:
- Prompt: "Summarize this lease clause for a tenant."
- Response A: accurate but full of legal jargon.
- Response B: plain-language, accurate, and flags the renewal deadline.
- Judgment: B preferred; rationale "both correct, B is clearer and surfaces the deadline the tenant needs."
In classic RLHF, those comparisons train a reward model, and the reward model then scores the language model's outputs during reinforcement learning. InstructGPT illustrates the split: about 33k prompts were used for reward-model training, with labelers ranking between 4 and 9 responses per prompt, and about 31k prompts were used for the PPO stage, which needs prompts but no new human labels. Anthropic's helpful and harmless assistant paper collected its preference data in phases, starting with about 44k helpfulness comparisons and 42k harmlessness (red-teaming) comparisons.
What makes preference data good:
- On-policy responses: the candidate answers should come from the model you are training, or one close to it. Preferences between answers your model would never write teach it little.
- Clear guidelines: annotators need a written rubric for what "better" means, especially when both answers are acceptable.
- Measured agreement: track how often two annotators make the same call. Low agreement usually means the rubric is vague, not that the annotators are careless.
- Judges who can spot errors: a generalist can rank tone. Only a domain expert can tell that the confident, well-written answer cites the wrong statute.
Do DPO and newer methods change the data you need?
They change the training, not the data shape. Direct Preference Optimization (DPO) learns straight from preference pairs with a simple classification-style objective, without training a separate reward model or running a reinforcement learning loop. It is cheaper and more stable to run, but it still needs the same prompt, chosen response and rejected response records that RLHF uses.
Reinforcement learning with verifiable rewards, used with methods like GRPO (introduced in the DeepSeekMath paper), replaces human preference labels with automatic checks where the answer can be verified: unit tests for code, exact answers for math. The data requirement shifts again, to well-specified tasks with reliable checkers, which still take experts to design.
When should you use SFT, RLHF, or both?
- Use SFT alone when the task has clear right answers and a stable format: extraction, classification with explanations, structured reports, a house style. It is cheaper, faster to iterate on and easier to debug.
- Add RLHF or DPO when "correct" is not enough and quality is a judgment call: helpfulness, tone, safety behavior, choosing between several valid answers.
- Use both, in order, for an assistant that users talk to directly. SFT gets the model into the right behavior; preference training polishes it. This is the sequence InstructGPT and most later chat models followed.
How much SFT and RLHF data do you need?
There is no universal number. The published figures are useful as a sense of scale, not as targets:
- LIMA: 1,000 curated SFT examples.
- InstructGPT: about 13k SFT prompts, 33k reward-model prompts and 31k PPO prompts.
- Anthropic HH: about 86k comparisons in the first collection phase, more in later phases.
A practical approach is to start with a pilot of a few hundred high-quality examples for one task, train, evaluate on a held-out set, and only then scale the parts of the data that move your metric. Buying volume before you know which slices matter is the most common way to waste a data budget.
Why does expert data matter more as models improve?
Because the easy gains are gone. Modern base models already write fluent, plausible text, so generic demonstrations and generalist preference labels add little. The remaining errors sit in specialist territory: a CAD design review, a verification bug in RTL, a citation in a legal memo, a subtle flaw in a financial model. Writing a good demonstration or catching the wrong answer in a comparison there requires someone who does that work for a living.

That is the gap Mesh Flow Lab's Expert Data service is built for. Lab buys existing specialist work and commissions demonstrations, annotations and evaluations from experts in areas such as mechanical engineering, digital hardware design, machine learning research, biology, healthcare and legal research. Demonstrations map to SFT data; evaluations and rankings map to preference data.
What should you check before buying SFT or preference data?
- Rights: written permission to train on the data, keep derived models and use them commercially.
- Who produced it: the expertise of the people who wrote demonstrations or made judgments, and how they were vetted.
- The rubric: the written guidelines annotators followed, and the version used for each record.
- Agreement and QA: how disagreements were resolved and what share of records were reviewed twice.
- Model provenance: for preference data, which model generated the candidate responses.
- Contamination: whether any records overlap with public benchmarks or your own evaluation sets.
- Format: a schema you can load directly, such as JSONL with prompt, response, chosen and rejected fields.
For a fuller buyer's checklist, see how to evaluate AI training datasets.
Frequently Asked Questions
Is RLHF better than SFT?
Neither is better in general; they do different jobs. SFT teaches the model to perform a task correctly. RLHF tunes how it chooses between acceptable answers. Most strong chat models use SFT first, then preference training.
Can you do RLHF without SFT?
You can, but it rarely works well. Preference training improves a model that already behaves roughly right; starting from a raw base model means the candidate responses are too poor for comparisons to teach much.
What is SFT data?
SFT data is a set of prompts paired with high-quality target responses, usually written or reviewed by people who know the task. It includes instruction-response pairs, worked solutions and annotated demonstrations.
Is DPO a type of RLHF?
DPO is a preference-learning method that reaches the same goal as RLHF without a separate reward model or a reinforcement learning loop. It uses the same kind of human preference data, so it is often grouped with RLHF.
How many examples do you need for fine-tuning?
It depends on the task and the base model. The LIMA study used 1,000 curated examples; InstructGPT used about 13k SFT prompts. Start with a few hundred excellent examples, evaluate, then scale what works.
The bottom line
- SFT needs demonstrations: prompts paired with answers an expert would ship.
- RLHF and DPO need preferences: prompts, candidate answers from your model, and expert judgments between them.
- Quality beats volume for both; the published datasets behind well-known models are smaller than most people assume.
- As models improve, the useful data moves to specialist domains where only practitioners can write or judge the answer.
If you need expert demonstrations, annotations or evaluations for a specific domain, tell Mesh Flow Lab what you are training. If you are an expert with work to license, start on the Expert Data page.
Sources
- Ouyang et al., Training language models to follow instructions with human feedback (InstructGPT), 2022
- Zhou et al., LIMA: Less Is More for Alignment, 2023
- Bai et al., Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, 2022
- Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model, 2023
- Shao et al., DeepSeekMath (introduces GRPO), 2024


