llmp API Reference

llmp.components

Classes

components.base.BaseEvaluationEngine(job)	Abstract base class for the EvaluationEngine.
components.base.BaseExampleManager()
components.base.BaseGenerator(job[, ...])	Base class for the Generator component.
components.base.BaseHumanVerification()
components.base.BaseInstructionHandler(job)
components.base.BaseOptimizer(job[, debug, ...])
components.evaluation.engine.EvaluationEngine(job)	The EvaluationEngine is responsible for evaluating the generated examples and updating the job accordingly.
components.example_manager.ExampleManager(job)	Initialize the ExampleManager with a job.
components.generator.concurrent.AsyncGenerator(job)	Generates a job multiple times asynchronous.
components.generator.concurrent.MultiThreadingAsyncGenerator(...)	Run SequentialAsyncGenerator in multiple threads.
components.generator.concurrent.SequentialAsyncGenerator(job)	Execute a generation job within one thread multiple times for multiple Inputs in Sequence.
components.generator.concurrent.SequentialAsyncGenerator2(job)	A faster version of SequentialAsyncGenerator.
components.generator.consensus.MajorVoteGenerator(job)	Execute a job with a specific input multiple times and return the majority vote output.
components.generator.examples.ExampleGenerator(job)	A Generator dedicated to generating examples for a given job.
components.generator.simple.Generator(job[, ...])	Basic generator that runs a job once.
components.human_verification.HumanVerification()
components.instruction.generation.InstructionGenerator(job)
components.instruction.mutation.InstructionMutation(job)	Generate Mutated Instructions.
components.job_factory.ExampleJobCreator()	Class for creating example jobs.
components.job_factory.JobCreator()	Abstract class for creating jobs.
components.job_factory.ModelJobCreator()	Class for creating zero-shot jobs.
components.job_factory.TemplateJobCreator()	Class for creating template jobs.
components.optimizer.examples.ExampleOptimizer(job)	Example Optimizer for a given job.
components.optimizer.instructions.InstructionOptimizer(job)	Instruction Optimizer for a given job.
components.optimizer.optimizer.Optimizer(job)
components.settings.global_settings.GlobalSettings(...)	Configuration class to store the state of bools for different scripts access.
components.settings.program_settings.ProgramSettings	Create a new model by parsing and validating input data from keyword arguments.
components.settings.program_settings.PromptType(value)

Functions

components.evaluation.metrics.avg_accuracy(...)	Compute the average accuracy of the runs.
components.evaluation.metrics.avg_efficiency(...)	Compute the average efficiency of the runs.
components.evaluation.metrics.avg_failure_rate(...)	Compute the average failure rate of the runs.
components.evaluation.metrics.avg_num_runs(outputs)	Compute the average number of runs.
components.evaluation.metrics.avg_token_usage(outputs)	Compute the average token usage of the outputs.
components.evaluation.metrics.explicit_accuracy(...)	Compute the accuracy of the outputs.
components.evaluation.metrics.implicit_accuracy(...)	Compute the accuracy of the outputs.
components.generator.verification.get_best_output(...)
components.generator.verification.get_human_vote(outputs)	Returns the human verified output from a list of outputs.
components.generator.verification.get_majority_grade(...)	Returns the grading output from a list of outputs.
components.generator.verification.get_majority_vote(outputs)	Returns the consensus and share of votes output from a list of outputs.
components.generator.verification.get_majority_vote_by_key(outputs)	Returns the consensus for each key from a list of outputs.
components.generator.verification.remove_reasoning(data)
components.instruction.utils.extend_instruction_by_example(...)	Extend an instruction with context of input and output model.
components.instruction.utils.extend_instruction_by_model(...)	Extend an instruction with context of input and output model.
components.instruction.utils.generate_instruction_from_model_and_template(...)	Generate an instruction for a specific job.
components.instruction.utils.generate_instruction_from_models(...)	Generate an instruction for a specific job.
components.instruction.utils.instruction_from_working_out(...)	Generate an instruction from the working out.
components.instruction.utils.mutate_instruction(...)	Mutate an instruction by randomly changing one character.
components.job_factory.job_factory(job_name)	param job_name The job_name name or id of the job.

llmp.components.evaluation

Classes

components.evaluation.engine.EvaluationEngine(job)

The EvaluationEngine is responsible for evaluating the generated examples and updating the job accordingly.

Functions

components.evaluation.metrics.avg_accuracy(...)	Compute the average accuracy of the runs.
components.evaluation.metrics.avg_efficiency(...)	Compute the average efficiency of the runs.
components.evaluation.metrics.avg_failure_rate(...)	Compute the average failure rate of the runs.
components.evaluation.metrics.avg_num_runs(outputs)	Compute the average number of runs.
components.evaluation.metrics.avg_token_usage(outputs)	Compute the average token usage of the outputs.
components.evaluation.metrics.explicit_accuracy(...)	Compute the accuracy of the outputs.
components.evaluation.metrics.implicit_accuracy(...)	Compute the accuracy of the outputs.

llmp.components.example_manager

Manage and store examples for a given job.

Classes

components.example_manager.ExampleManager(job)

Initialize the ExampleManager with a job.

llmp.components.generator

Generator module Used to perform generation tasks for a given job and input object.

Classes

components.generator.concurrent.AsyncGenerator(job)	Generates a job multiple times asynchronous.
components.generator.concurrent.MultiThreadingAsyncGenerator(...)	Run SequentialAsyncGenerator in multiple threads.
components.generator.concurrent.SequentialAsyncGenerator(job)	Execute a generation job within one thread multiple times for multiple Inputs in Sequence.
components.generator.concurrent.SequentialAsyncGenerator2(job)	A faster version of SequentialAsyncGenerator.
components.generator.consensus.MajorVoteGenerator(job)	Execute a job with a specific input multiple times and return the majority vote output.
components.generator.examples.ExampleGenerator(job)	A Generator dedicated to generating examples for a given job.
components.generator.simple.Generator(job[, ...])	Basic generator that runs a job once.

Functions

components.generator.verification.get_best_output(...)
components.generator.verification.get_human_vote(outputs)	Returns the human verified output from a list of outputs.
components.generator.verification.get_majority_grade(...)	Returns the grading output from a list of outputs.
components.generator.verification.get_majority_vote(outputs)	Returns the consensus and share of votes output from a list of outputs.
components.generator.verification.get_majority_vote_by_key(outputs)	Returns the consensus for each key from a list of outputs.
components.generator.verification.remove_reasoning(data)

llmp.components.optimizer

Classes

components.optimizer.examples.ExampleOptimizer(job)	Example Optimizer for a given job.
components.optimizer.instructions.InstructionOptimizer(job)	Instruction Optimizer for a given job.
components.optimizer.optimizer.Optimizer(job)

llmp.data_model

Classes

data_model.events.Event	Create a new model by parsing and validating input data from keyword arguments.
data_model.example_record.ExampleRecord	Create a new model by parsing and validating input data from keyword arguments.
data_model.job_record.JobRecord	A representation of a job in the LLMP system.

Functions

data_model.job_record.get_record_by_input(...)	Find an example record by its input.
data_model.job_record.get_template_from_job(job)	Get a template from a job.
data_model.job_record.input_in_records(job, ...)	Check if an input example is already in the example records.
data_model.job_record.load_engine_from_job(job)	Load a Engine engine from a job.

llmp.integration

Classes

integration.example_selector.ExampleManagerSelector

Create a new model by parsing and validating input data from keyword arguments.

llmp.services

Classes

services.job_manager.JobManager(config)
services.job_storage.JobStorage([base_path])
services.program.Program(signature, ...)

Functions

services.job_manager.load_generator_cls(...)

Load a generator class by type.

llmp.types

Classes

types.EventType(value[, names, module, ...])
types.MajorVoteType(value[, names, module, ...])
types.TestSetMode(value[, names, module, ...])
types.VerificationType(value[, names, ...])	Validation rank for ExampleRecords

llmp.utils

Classes

utils.encoder.JSONEncoder(*[, skipkeys, ...])	Constructor for JSONEncoder, with sensible defaults.
utils.helper.dotdict
utils.singleton.Singleton(name, bases, ...)	Singleton metaclass for ensuring only one instance of a class.

Functions

utils.encoder.dumps_encoder(obj)
utils.eval.f1_score(prediction, answers)
utils.eval.fuzzy_match(s1, s2)
utils.eval.get_consensus(answers)
utils.eval.normalize(s)	Lower text and remove punctuation, articles and extra whitespace.
utils.helper.flatten(ar)
utils.helper.get_timestamp()	Return a timestamp.
utils.helper.int_or_float(val)
utils.helper.safe_getter(key, *args[, default])	performs get operatation in descending order and returns the first value found
utils.helper.update_by_kwargs(d, **kwargs)	Update a dictionary with kwargs.
utils.signature.is_valid_uuid(val)	Check if val is a valid UUID.
utils.signature.safe_job_name(planned_name, ...)