LLM Evaluation Dataset Generation in 2027: Why Test Sets Fail

Why This Is Harder Than It Looks

Most teams building LLM features run into the same situation. The demo looks good, then leadership asks how well the system actually works, and no one has a number to answer with. The usual shortcut is to have an engineer spend an afternoon typing a few questions by hand. The resulting test set looks reasonable but reveals very little. LLM evaluation dataset generation means turning an enterprise's real documents, tickets, and business processes into questions with known reference answers, so that every change to the model, prompt, or retrieval pipeline can be measured against the same yardstick. By 2027, enterprises that have moved past the pilot stage can no longer treat this as optional, yet most teams are still improvising.

A useful evaluation set has to reflect the questions people actually ask, including the messy ones: vague phrasing, questions that require several reasoning steps, and questions the system should refuse to answer. Writing correct reference answers takes domain knowledge, and that knowledge sits with experts who are busy doing their own jobs. Source documents also keep changing. A policy updated last quarter can quietly invalidate half of last month's reference answers, and nobody notices until much later.

Synthetic generation can ease the shortage of test cases, but it introduces quality problems. Questions generated from a single passage tend to be too easy and often amount to paraphrases of a sentence in the source. A model can score well on such a set and still fail when a user asks a question that spans three documents or uses the sales team's vocabulary instead of the wording in the policy manual. The score goes up, but the signal it carries gets weaker.

Common Mistakes

The first mistake is relying only on questions the team wrote itself. Engineers tend to write questions that reflect how they understand the system, not how customer service agents or procurement analysts actually phrase their needs.

The second mistake is generating both the question and the answer from the same chunk of text. The answer then sits inside that chunk by design, so the test never checks whether retrieval can find the right material across documents. It only measures reading comprehension over a single passage.

The third mistake is leaving out negative examples. If the test set contains no questions the knowledge base cannot answer, no metric shows whether the system can admit uncertainty. In practice, many production incidents start here.

The fourth mistake is trusting a single overall score, since one number hides which categories are failing. The fifth mistake is freezing the test set after launch. Once real traffic arrives, production logs are the best source of new test cases. If the set never absorbs them, it gradually drifts away from reality.

Start From Real Tasks and Generate With LLMs Under Control

A usable dataset should draw on three kinds of sources: query logs from existing systems or manual workflows, support tickets or internal requests, and short interviews with the people who do the work. Cluster these inputs by intent. The goal is coverage across task types, not sheer volume. Forty carefully chosen questions that span the real intent categories are worth more than four hundred near-identical ones.

For example, a team building an assistant for an after-sales support group might first sort real tickets by product line, then break them down by question type: installation, billing, warranty, and policy exceptions. Each category should contain several items. Policy exceptions deserve extra attention, because a wrong answer there causes the most damage.

If the underlying knowledge base is still unstable, fix that first. Many accuracy problems surfaced by evaluation trace back to chunking, metadata, and document version control. Our guide on why enterprise RAG knowledge bases still give inaccurate answers in 2027 covers these upstream causes in more detail.

LLMs can help scale up question writing, but the process has to be explicit. A workable sequence is: sample document passages, weighted by how often each topic appears in real traffic; generate questions of specified types, such as fact lookup, comparison, procedural steps, or unanswerable cases; generate candidate answers that cite the supporting passage; then filter. Filtering should remove near-duplicate questions, questions whose answers do not actually require the cited passage, and items that are too easy for a baseline model.

Generation and judging should use different prompts, and ideally different models. Models tend to favor their own phrasing, which inflates scores. Every item should carry metadata: the source document, its version, the question type, a difficulty label, and the reviewer who approved it. Without this, it becomes impossible to explain later why a score changed.

Verify and Annotate With a Scoring Rubric

Generated items need human review, at least in the early phase of a project. During a pilot, review every item. Once the process is stable, switch to random sampling. The rubric should check four things: whether the reference answer is correct, whether it has a cited basis in the source text, whether it is complete, and whether refusing to answer is appropriate when the source contains no answer.

Measure agreement between reviewers. If two experts often disagree on the same item, the cause is usually a vague rubric rather than the people involved. In that case, tighten the definitions, add examples for edge cases, and re-review the disputed items. A dataset built on fuzzy criteria produces scores that look precise but mean very little.

Once you have an annotated dataset and a clear rubric, the cycle can repeat: sample, generate, review, score, and feed failures back into the dataset. Duaer is one platform where teams can build this kind of workflow on their own knowledge bases and rerun the same questions against each version without rebuilding from scratch. Its value still depends on the quality of the rubric and sampling design, so the human steps described above remain the priority.

How to Tell Whether the Dataset Itself Is Sound

The dataset itself also needs evaluation. Four checks are worth running. The first is coverage: compare the distribution of question types against real production traffic and flag categories that are over- or under-represented. The second is discrimination: run a deliberately weak baseline alongside a stronger configuration. If both score above 90%, the test set is too easy to guide decisions. The third is failure analysis: break errors down by category and source document. Failures concentrated in one policy area are far more actionable than a drop of a few points in the overall number. The fourth is stability: rerun the same system several times and observe the variance. If scores swing widely with no change to the configuration, the test set or the scoring method needs work.

If you also need to know how often the system fabricates facts, pair this dataset with a dedicated check. Our article on how to verify LLM hallucination rates before trusting vendor data explains how to design such a test so that it does not simply reward systems that sound confident.

What You Can Do This Week

Pick one workflow that has a clearly named owner. Extract fifty real questions from the logs or tickets of the past month. Ask a domain expert to write a reference answer for each one, noting its source, and to mark at least ten questions that the existing documents cannot answer. Run the current system against this set and record results by category. Do not fix anything yet. The value of a first baseline is that it shows where the system really stands and gives you a fixed reference point for every change that follows.

If you want to run this loop without writing your own scripts, you can sign up for a trial at https://www.duaer.com and test the process on a small part of your own documentation first. Start small, check results against the rubric, and expand the scope only once the numbers agree with expert judgment.

A good evaluation dataset is not a pile of generated questions. It is a small, precise, annotated, and representative set that reflects the work employees actually do, and it stays current as that work changes.