Understand why to evaluate in Duaer

What evaluations are

Evaluation is how you check that an AI digital organization in Duaer is reliable. It is often what turns a flaky demo into something you can ship. You need it while building and after you go live.

The foundation is running a test dataset through the digital organization. The dataset has many cases. Each case has a sample input, and often the expected output too.

• Try a range of inputs so you see edge-case behavior
• Change prompts or models with more confidence, without breaking something else
• Compare performance across models or prompts

Why evaluation is needed

Models are not like ordinary code you can reason about. They are black boxes. You measure them by running real inputs and reading the outputs.

You only gain confidence after enough cases that match the edge cases production will see.

Two types of evaluation

Light evaluation (before go-live): a handful of hand-written examples is often enough to reach a releasable or proof-of-concept state. Compare outputs side by side without formal metrics yet. Steps are in light evaluations.

Metric-based evaluation (after go-live): grow the set from production runs. When you find a bug, add that input. After a fix, rerun the whole set as a regression. When there are too many rows to read one by one, score quality with metrics and track them on the evaluations view. See use metrics to measure quality.

How the two types compare

• Gain per iteration: large in light evaluation; smaller once you use metrics
• Dataset size: small for light; large for metrics
• Dataset sources: mostly hand-written (or lightly generated) for light; often production executions for metrics, plus optional generated rows
• Actual outputs: required for both
• Expected outputs: optional for light; usually required for metrics
• Evaluation metric: optional for light; required for metrics

When wiring fails or scores stay empty, start with common issues. The Evaluations tab “More info” link also points here.