llmp.components.base.BaseEvaluationEngine

class llmp.components.base.BaseEvaluationEngine(job: JobRecord)

Abstract base class for the EvaluationEngine. Provides an interface for evaluating generation results and computing metrics.

Methods

__init__(job)
compute_metrics(*args, **kwargs)	Compute and aggregate metrics based on the run metrics and the evaluation results.
evaluate(*args, **kwargs)	Evaluate the generation result against the desired output and compute metrics.

__init__(job: JobRecord)

abstract compute_metrics(*args, **kwargs) → Dict[str, Any]

Compute and aggregate metrics based on the run metrics and the evaluation results.

Returns

A dictionary containing the aggregated metrics.

Return type

Dict[str, Any]

abstract evaluate(*args, **kwargs) → Dict[str, Any]

Evaluate the generation result against the desired output and compute metrics.

Returns

A dictionary containing the computed metrics.

Return type

Dict[str, Any]