llmp.components.generator.examples.ExampleGenerator

class llmp.components.generator.examples.ExampleGenerator(job: JobRecord, job_settings: dict = None, num_votes: int = 5, mode: VerificationType = VerificationType.MAJORITY_VOTE, **kwargs)[source]

A Generator dedicated to generating examples for a given job.

This generator is used to generate examples for a given job with a predefined Prompt template. Under the hood, it uses a MajorVoteGenerator to generate the examples.

Initialize the generator with a job and job settings.

Parameters

  • job (JobRecord) – the job to be executed

  • job_settings (dict) – the job settings to be used

  • num_votes (int) – number of votes to be collected

  • mode (VerificationType) – the verification type to be used (majority vote, majority grade, human verified)

  • **kwargs – any - passed to MajorVoteGenerator

Attributes

TEMPLATE

verification_type

Return the validation type for the generator.

Methods

__init__(job[, job_settings, num_votes, mode])

Initialize the generator with a job and job settings.

generate(num_items, **kwargs)

Generate examples for a given job.

log_generation(input_object, ...)

Log the generation result.
__init__(job: JobRecord, job_settings: dict = None, num_votes: int = 5, mode: VerificationType = VerificationType.MAJORITY_VOTE, **kwargs)[source]

Initialize the generator with a job and job settings.

Parameters

  • job (JobRecord) – the job to be executed

  • job_settings (dict) – the job settings to be used

  • num_votes (int) – number of votes to be collected

  • mode (VerificationType) – the verification type to be used (majority vote, majority grade, human verified)

  • **kwargs – any - passed to MajorVoteGenerator

generate(num_items, **kwargs) → list[Union[llmp.data_model.example_record.ExampleRecord, dict]][source]

Generate examples for a given job.

  1. Generate inputs with a template

  2. Generate outputs with a MajorVoteGenerator

Parameters

  • num_items (int) – number of examples to be generated

  • **kwargs – any - passed to MajorVoteGenerator.generate() method

Returns

list of generated examples

Return type

list[Union[ExampleRecord, dict]]

log_generation(input_object: dict, generated_object: dict, run_metrics: dict, **kwargs) → Optional[Event]

Log the generation result.

Parameters

  • input_object (dict) – the input object

  • generated_object (dict) – the generated object

  • run_metrics (dict) – the run metrics

  • **kwargs – any - passed to JobRecord.log_generation