LangSmith Online Eval Engineering

Build online evaluators iteratively:

inspect production behavior -> choose one quality signal -> build and test
-> attach a scoped rule -> inspect scores and logs -> calibrate

Evaluators are workspace resources that can be reused across tracing projects and datasets. An online evaluation rule attaches an evaluator to matching production runs or threads.

Read LangSmith API before creating or modifying evaluators.

1. Inspect representative behavior