TestRun

Test_Run utilities for data models and type definitions used throughout the maxim sdk.

View module source on GitHub

YieldedOutputTokenUsage

@dataclass
class YieldedOutputTokenUsage()

This class represents the token usage of a yielded output. Users can pass custom token usage to the yieldsOutput function.

YieldedOutputCost

@dataclass
class YieldedOutputCost()

This class represents the cost of a yielded output. Users can pass custom cost to the yieldsOutput function.

YieldedOutputMeta

@dataclass
class YieldedOutputMeta()

This class represents the meta of a yielded output. Users can pass custom meta to the yieldsOutput function.

YieldedOutput

@dataclass
class YieldedOutput()

Yielded output represents the output of yieldsOutput function.

EvaluatorArgs

@dataclass
class EvaluatorArgs()

This class represents the arguments of an evaluator.

EvaluatorConfig

@dataclass
class EvaluatorConfig()

This class represents the config of an evaluator.

TestRun

@dataclass
class TestRun()

This class represents a test run.

TestRunEntry

@dataclass
class TestRunEntry()

This class represents an entry of a test run.

TestRunWithDatasetEntry

@dataclass
class TestRunWithDatasetEntry(TestRun)

This class represents a test run with a dataset entry.

TestRunStatus

@dataclass
class TestRunStatus()

This class represents the status of a test run.

EvaluatorMeanScore

@dataclass
class EvaluatorMeanScore()

This class represents the mean score of an evaluator. This helps users to specify the score of an custom evaluator.

TestRunTokenUsage

@dataclass
class TestRunTokenUsage()

This class represents the token usage of a test run.

TestRunCost

@dataclass
class TestRunCost()

This class represents the cost of a test run.

TestRunLatency

@dataclass
class TestRunLatency()

This class represents the latency of a test run.

TestRunResultObj

@dataclass
class TestRunResultObj()

Object representing a result of a test run.

TestRunResult

@dataclass
class TestRunResult()

This class represents the result of a test run.

RunResult

@dataclass
class RunResult()

This class represents the result of a comparison test run.

TestRunLogger

class TestRunLogger(ABC)

info

@abstractmethod
def info(message: str) -> None

Log an informational message.

Arguments:

Name	Type	Description
message	str	The message to be logged.

error

@abstractmethod
def error(message: str, e: Optional[Exception] = None) -> None

Log an error message.

Arguments:

Name	Type	Description
message	str	The error message to be logged.

TestRunConfig

@dataclass
class TestRunConfig(Generic[T])

Configuration for a test run.

Attributes:

Name	Type	Description
base_url	str	The base URL for the API.
api_key	str	The API key for authentication.
in_workspace_id	str	The ID of the workspace.
workflow_id	Optional[str]	The ID of the workflow.
prompt_version_id	Optional[str]	The ID of the prompt version.
prompt_chain_version_id	Optional[str]	The ID of the prompt chain version.
name	str	The name of the test run.
data_structure	Optional[T]	The structure of the test data.
data	Optional[Union[str, DataValue[T], Callable[[int], Optional[DataValue[T]]]]]	The test data or a function to retrieve it.
test_config_id	Optional[str]	The ID of the test configuration.
platform_evaluators	List[PlatformEvaluatorType[T]]	List of platform evaluators to use.