whylogs_container.whylabs.llm_validation package

Submodules

whylogs_container.whylabs.llm_validation.actions module

class whylogs_container.whylabs.llm_validation.actions.BlockAction(*, block_message: str, action_type: Literal['block'] = 'block', is_action_block: bool)

Bases: BaseModel

action_type: Literal['block']

block_message: str

is_action_block: bool

model_computed_fields: ClassVar[dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[dict[str, FieldInfo]] = {'action_type': FieldInfo(annotation=Literal['block'], required=False, default='block'), 'block_message': FieldInfo(annotation=str, required=True), 'is_action_block': FieldInfo(annotation=bool, required=True)}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo].

This replaces Model.__fields__ from Pydantic V1.

class whylogs_container.whylabs.llm_validation.actions.PassAction(*, action_type: Literal['pass'] = 'pass', is_action_pass: bool)

Bases: BaseModel

action_type: Literal['pass']

is_action_pass: bool

model_computed_fields: ClassVar[dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[dict[str, FieldInfo]] = {'action_type': FieldInfo(annotation=Literal['pass'], required=False, default='pass'), 'is_action_pass': FieldInfo(annotation=bool, required=True)}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo].

This replaces Model.__fields__ from Pydantic V1.

whylogs_container.whylabs.llm_validation.actions.get_action(result: WorkflowResult, config: ConfigInstance, dataset_id: str) → PassAction | BlockAction

whylogs_container.whylabs.llm_validation.metadata module

class whylogs_container.whylabs.llm_validation.metadata.ConfigurationMetadata(config_error: bool, timestamp: int, message: str)

Bases: object

config_error: bool

message: str

timestamp: int

class whylogs_container.whylabs.llm_validation.metadata.ModuleDefinition(*, module: Literal['toxicity', 'sentiment', 'themes', 'textstat', 'regexes', 'input_output'], upper_threshold: float | int | None = None, lower_threshold: float | int | None = None, include_upper: bool | None = None, config_path: str | None = None, metric: str | None = None)

Bases: BaseModel

config_path: str | None

include_upper: bool | None

lower_threshold: float | int | None

metric: str | None

model_computed_fields: ClassVar[dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[dict[str, FieldInfo]] = {'config_path': FieldInfo(annotation=Union[str, NoneType], required=False), 'include_upper': FieldInfo(annotation=Union[bool, NoneType], required=False), 'lower_threshold': FieldInfo(annotation=Union[float, int, NoneType], required=False), 'metric': FieldInfo(annotation=Union[str, NoneType], required=False), 'module': FieldInfo(annotation=Literal['toxicity', 'sentiment', 'themes', 'textstat', 'regexes', 'input_output'], required=True), 'upper_threshold': FieldInfo(annotation=Union[float, int, NoneType], required=False)}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo].

This replaces Model.__fields__ from Pydantic V1.

module: Literal['toxicity', 'sentiment', 'themes', 'textstat', 'regexes', 'input_output']

upper_threshold: float | int | None

class whylogs_container.whylabs.llm_validation.metadata.ValidationMetadata(validator_name: str | None = None, failed_metric: str | None = None, value: str | int | float | NoneType = None, timestamp: int | None = None, is_valid: bool | None = False)

Bases: object

failed_metric: str | None = None

is_valid: bool | None = False

timestamp: int | None = None

validator_name: str | None = None

value: str | int | float | None = None

class whylogs_container.whylabs.llm_validation.metadata.ValidationReport(failures: List[whylogs_container.whylabs.llm_validation.metadata.ValidationMetadata])

Bases: object

failures: List[ValidationMetadata]

class whylogs_container.whylabs.llm_validation.metadata.YamlConfig(*, whylabs_dataset_id: str, policy: str, id: str, policy_version: str, schema_version: str, rules: Dict[Literal['prompt', 'response'], List[ModuleDefinition]] = {}, profiling: List[Literal['toxicity', 'sentiment', 'themes', 'textstat', 'regexes', 'input_output', 'all']] = [])

Bases: BaseModel

id: str

model_computed_fields: ClassVar[dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[dict[str, FieldInfo]] = {'id': FieldInfo(annotation=str, required=True), 'policy': FieldInfo(annotation=str, required=True), 'policy_version': FieldInfo(annotation=str, required=True), 'profiling': FieldInfo(annotation=List[Literal['toxicity', 'sentiment', 'themes', 'textstat', 'regexes', 'input_output', 'all']], required=False, default=[]), 'rules': FieldInfo(annotation=Dict[Literal['prompt', 'response'], List[ModuleDefinition]], required=False, default={}), 'schema_version': FieldInfo(annotation=str, required=True), 'whylabs_dataset_id': FieldInfo(annotation=str, required=True)}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo].

This replaces Model.__fields__ from Pydantic V1.

policy: str

policy_version: str

profiling: List[Literal['toxicity', 'sentiment', 'themes', 'textstat', 'regexes', 'input_output', 'all']]

rules: Dict[Literal['prompt', 'response'], List[ModuleDefinition]]

schema_version: str

whylabs_dataset_id: str

whylogs_container.whylabs.llm_validation.validator module

class whylogs_container.whylabs.llm_validation.validator.AsyncValidationMessage(request: bytes, timestamp: int, log: bool = True, trace: bool = True, id: str = <factory>, carrier: Optional[Dict[str, Any]] = None)

Bases: ValidationMessage

class whylogs_container.whylabs.llm_validation.validator.ReturnType(result: langkit.core.workflow.WorkflowResult, score_result: langkit.core.workflow.WorkflowResult | None)

Bases: object

result: WorkflowResult

score_result: WorkflowResult | None

class whylogs_container.whylabs.llm_validation.validator.SynchronousValidator

Bases: ABC

This class is a convenience wrapper that has all of the logic required for synchronously calling the ValidationActor. There shouldn’t be any real logic in here that does validation, just calls to actor.send and synchronous utilities.

abstract close_validator() → None

evaluate(request: LLMValidateRequest, log: bool = True, trace: bool = True) → ReturnType

Perform validation on the given LLM message and wait for the result.

abstract start_validator() → None

warm_caches() → None

class whylogs_container.whylabs.llm_validation.validator.ValidationActor(config_instance: ConfigInstance, rolling_logger: ContainerProcessRollingLogger)

Bases: ProcessActor[Union[ValidationMessage, AsyncValidationMessage], Optional[ReturnType]], SynchronousValidator

close_validator() → None

process_async_validation_message(messages: List[AsyncValidationMessage]) → None

process_batch(batch, batch_type) → None

process_validation_message(messages: List[ValidationMessage]) → None

process_validation_messages(messages: List[AsyncValidationMessage] | List[ValidationMessage], return_results: bool) → None

run() → None

Method to be run in sub-process; can be overridden in sub-class

start() → None

The process version of the actor apparently has to be manually started after it’s created, unlike the thread version which can just be automatically started from within its init. There must be some post-init setup that needs to be done.

start_validator() → None

class whylogs_container.whylabs.llm_validation.validator.ValidationMessage(request: bytes, timestamp: int, log: bool = True, trace: bool = True, id: str = <factory>, carrier: Optional[Dict[str, Any]] = None)

Bases: object

carrier: Dict[str, Any] | None = None

id: str

log: bool = True

request: bytes

timestamp: int

to_validation_request() → LLMValidateRequest

trace: bool = True

Module contents