llmp.components.evaluation.engine.EvaluationEngine

class llmp.components.evaluation.engine.EvaluationEngine(job: JobRecord, num_runs: int = 5)

The EvaluationEngine is responsible for evaluating the generated examples and updating the job accordingly.

Initialize the EvaluationEngine with a job and job settings.

Parameters

• job – JobRecord

• num_runs – int

Methods

__init__(job[, num_runs])	Initialize the EvaluationEngine with a job and job settings.
compute_metrics(sample_metrics)	Compute metrics for a single generation sample from multiple runs.
compute_sample_metrics(outputs_metrics, ...)	Compute metrics for a single generation sample from multiple runs.
evaluate(records[, job_settings])	Evaluate the generated examples for a specific job.

__init__(job: JobRecord, num_runs: int = 5)

Initialize the EvaluationEngine with a job and job settings.

Parameters

• job – JobRecord

• num_runs – int

compute_metrics(sample_metrics: list[dict]) → dict

Compute metrics for a single generation sample from multiple runs.

compute_sample_metrics(outputs_metrics: list[Tuple[dict, dict]], ideal_output: dict, sample_input: dict) → dict

Compute metrics for a single generation sample from multiple runs.

evaluate(records: list[llmp.data_model.example_record.ExampleRecord], job_settings: dict = None)

Evaluate the generated examples for a specific job.