kiara.data.values

A module that contains value-related classes for Kiara.

A value in Kiara-speak is a pointer to actual data (aka 'bytes'). It contains metadata about that data (like whether it's valid/set, what type/schema it has, when it was last modified, ...), but it does not contain the data itself. The reason for that is that such data can be fairly large, and in a lot of cases it is not necessary for the code involved to have access to it, access to the metadata is enough.

Each Value has a unique id, which can be used to retrieve the data (whole, or parts of it) from a [DataRegistry][kiara.data.registry.DataRegistry]. In addition, that id can be used to subscribe to change events for a value (published whenever the data that is associated with a value was changed).

Value pydantic-model

The underlying base class for all values.

The important subclasses here are the ones inheriting from 'PipelineValue', as those are registered in the data registry.

creation_date: datetime pydantic-field

The time this value was created value happened.

hashes: List[kiara.data.values.__init__.ValueHash] pydantic-field

Available hashes relating to the actual value data. This attribute is not populated by default, use the 'get_hashes()' method to request one or several hash items, afterwards those hashes will be reflected in this attribute.

id: str pydantic-field required

A unique id for this value.

is_none: bool pydantic-field

Whether the value is 'None'.

is_set: bool pydantic-field

Whether the value was set (in some way: user input, default, constant...).

metadata: Dict[str, Dict[str, Any]] pydantic-field

Available metadata relating to the actual value data (size, no. of rows, etc. -- depending on data type). This attribute is not populated by default, use the 'get_metadata()' method to request one or several metadata items, afterwards those metadata items will be reflected in this attribute.

type_obj property readonly

Return the object that contains all the type information for this value.

value_schema: ValueSchema pydantic-field required

The schema of this value.

__eq__(self, other) special

Return self==value.

Source code in kiara/data/values/__init__.py

def __eq__(self, other):

    # TODO: compare all attributes if id is equal, just to make sure...

    if not isinstance(other, Value):
        return False
    return self.id == other.id

__hash__(self) special

Return hash(self).

Source code in kiara/data/values/__init__.py

def __hash__(self):
    return hash(self.id)

__repr__(self) special

Return repr(self).

Source code in kiara/data/values/__init__.py

def __repr__(self):

    return f"Value(id={self.id}, type={self.type_name}, is_set={self.is_set}, is_none={self.is_none}"

__str__(self) special

Return str(self).

Source code in kiara/data/values/__init__.py

def __str__(self):

    return self.__repr__()

get_hash(self, hash_type)

Calculate the hash of a specified type for this value. Hashes are cached.

Source code in kiara/data/values/__init__.py

def get_hash(self, hash_type: str) -> ValueHash:
    """Calculate the hash of a specified type for this value. Hashes are cached."""

    hashes = self.get_hashes(hash_type)
    return list(hashes)[0]

get_metadata(self, *metadata_keys, *, also_return_schema=False)

Retrieve (if necessary) and return metadata for the specified keys.

By default, the metadata is returned as a map (in case of a single key: single-item map) with metadata key as dict key, and the metadata as value. If 'also_return_schema' was set to True, the value will be a two-key map with the metadata under the metadata_item subkey, and the value schema under metadata_schema.

If no metadata key is specified, all available metadata for this value type will be returned.

Parameters:

Name	Type	Description	Default
metadata_keys	str	the metadata keys to retrieve metadata (empty for all available metadata)	()
also_return_schema	bool	whether to also return the schema for each metadata item	False

Source code in kiara/data/values/__init__.py

def get_metadata(
    self, *metadata_keys: str, also_return_schema: bool = False
) -> typing.Mapping[str, typing.Mapping[str, typing.Any]]:
    """Retrieve (if necessary) and return metadata for the specified keys.

    By default, the metadata is returned as a map (in case of a single key: single-item map) with metadata key as
    dict key, and the metadata as value. If 'also_return_schema' was set to `True`, the value will be a two-key
    map with the metadata under the ``metadata_item`` subkey, and the value schema under ``metadata_schema``.

    If no metadata key is specified, all available metadata for this value type will be returned.

    Arguments:
        metadata_keys: the metadata keys to retrieve metadata (empty for all available metadata)
        also_return_schema: whether to also return the schema for each metadata item
    """

    if not metadata_keys:
        _metadata_keys = set(
            self._kiara.metadata_mgmt.get_metadata_keys_for_type(self.type_name)
        )
        for key in self.metadata.keys():
            _metadata_keys.add(key)
    else:
        _metadata_keys = set(metadata_keys)

    result = {}

    missing = set()
    for metadata_key in sorted(_metadata_keys):
        if metadata_key in self.metadata.keys():
            if also_return_schema:
                result[metadata_key] = self.metadata[metadata_key]
            else:
                result[metadata_key] = self.metadata[metadata_key]["metadata_item"]
        else:
            missing.add(metadata_key)

    if not missing:
        return result

    _md = self._kiara.metadata_mgmt.get_value_metadata(
        self, *missing, also_return_schema=True
    )
    for k, v in _md.items():
        self.metadata[k] = v
        if also_return_schema:
            result[k] = v
        else:
            result[k] = v["metadata_item"]
    return {k: result[k] for k in sorted(result.keys())}

item_is_valid(self)

Check whether the current value is valid

Source code in kiara/data/values/__init__.py

def item_is_valid(self) -> bool:
    """Check whether the current value is valid"""

    if self.value_schema.optional:
        return True
    else:
        return self.is_set and not self.is_none

item_status(self)

Print a human readable short description of this values status.

Source code in kiara/data/values/__init__.py

def item_status(self) -> str:
    """Print a human readable short description of this values status."""

    if self.value_schema.optional:
        if self.is_set:
            if self.is_none:
                return "no value (not required)"
            else:
                if self.value_schema.default and self.type_name in [
                    "string",
                    "integer",
                    "boolean",
                ]:
                    if self.get_value_data() == self.value_schema.default:
                        return "set (default)"
                return "set"
        else:
            return "not set (not required)"
    else:
        if self.is_set:
            if self.is_none:
                return "no value"
            else:
                if self.value_schema.default and self.type_name in [
                    "string",
                    "integer",
                    "boolean",
                ]:
                    if self.get_value_data() == self.value_schema.default:
                        return "set (default)"
                return "set"
        else:
            return "not set"

save(self, aliases=None, register_missing_aliases=True)

Save this value, under the specified alias(es).

Source code in kiara/data/values/__init__.py

def save(
    self,
    aliases: typing.Optional[typing.Iterable[str]] = None,
    register_missing_aliases: bool = True,
) -> "Value":
    """Save this value, under the specified alias(es)."""

    # if self.get_value_lineage():
    #     for field_name, value in self.get_value_lineage().inputs.items():
    #         value_obj = self._registry.get_value_obj(value)
    #         try:
    #             value_obj.save()
    #         except Exception as e:
    #             print(e)

    value = self._kiara.data_store.register_data(self)
    if aliases:
        self._kiara.data_store.link_aliases(
            value, *aliases, register_missing_aliases=register_missing_aliases
        )

    return value

ValueAlias pydantic-model

repo_name: str pydantic-field

The name of the data repo the value lives in.

tag: str pydantic-field

The tag for the alias.

version: int pydantic-field

The version of this alias.

ValueHash pydantic-model

hash: str pydantic-field required

The value hash.

hash_type: str pydantic-field required

The value hash method.

ValueInfo pydantic-model

deserialize_config: DeserializeConfig pydantic-field

The module config (incl. inputs) to deserialize the value.

hashes: List[kiara.data.values.__init__.ValueHash] pydantic-field

All available hashes for this value.

is_valid: bool pydantic-field required

Whether the item is valid (in the context of its schema).

lineage: ValueLineage pydantic-field

Information about how the value was created.

metadata: Dict[str, Dict[str, Any]] pydantic-field required

The metadata associated with this value.

value_id: str pydantic-field required

The value id.

value_schema: ValueSchema pydantic-field required

The value schema.

ValueLineage pydantic-model

Model containing the lineage of a value.

inputs: Dict[str, ValueInfo] pydantic-field required

The inputs that were used to create the value this refers to.

output_name: str pydantic-field

The result field name for the value this refers to.

value_index: Dict[str, ValueInfo] pydantic-field

Index of all values that are associated with this value lineage.

create_renderable(self, **config)

Create a renderable for this module configuration.

Source code in kiara/data/values/__init__.py

def create_renderable(self, **config: typing.Any) -> RenderableType:

    all_ids = sorted(find_all_ids_in_lineage(self))
    id_color_map = {}
    for idx, v_id in enumerate(all_ids):
        id_color_map[v_id] = COLOR_LIST[idx % len(COLOR_LIST)]

    show_ids = config.get("include_ids", False)

    tree = fill_lineage_tree(self, include_ids=show_ids)
    return tree

ValueSchema pydantic-model

The schema of a value.

The schema contains the [ValueType][kiara.data.values.ValueType] of a value, as well as an optional default that will be used if no user input was given (yet) for a value.

For more complex container types like array, tables, unions etc, types can also be configured with values from the type_config field.

default: Any pydantic-field

A default value.

desc property readonly

The first line of the 'doc' value.

doc: str pydantic-field

A description for the value of this input field.

is_constant: bool pydantic-field

Whether the value is a constant.

optional: bool pydantic-field

Whether this value is required (True), or whether 'None' value is allowed (False).

type: str pydantic-field required

The type of the value.

type_config: Dict[str, Any] pydantic-field

Configuration for the type, in case it's complex.

__eq__(self, other) special

Return self==value.

Source code in kiara/data/values/__init__.py

def __eq__(self, other):

    if not isinstance(other, ValueSchema):
        return False

    return (self.type, self.default) == (other.type, other.default)

__hash__(self) special

Return hash(self).

Source code in kiara/data/values/__init__.py

def __hash__(self):

    return hash((self.type, self.default))

ValueSlot pydantic-model

id: str pydantic-field required

The id for this slot.

tags: Dict[str, int] pydantic-field

The tags for this value slot (tag name as key, linked version as value.

value_schema: ValueSchema pydantic-field required

The schema for the values of this slot.

values: Dict[int, kiara.data.values.__init__.Value] pydantic-field

The values of this slot, with versions as key.

__eq__(self, other) special

Return self==value.

Source code in kiara/data/values/__init__.py

def __eq__(self, other):

    if not isinstance(other, ValueSlot):
        return False

    return self.id == other.id

__hash__(self) special

Return hash(self).

Source code in kiara/data/values/__init__.py

def __hash__(self):

    return hash(self.id)

add_value(self, value, trigger_callbacks=True, tags=None)

Add a value to this slot.

Source code in kiara/data/values/__init__.py

def add_value(
    self,
    value: Value,
    trigger_callbacks: bool = True,
    tags: typing.Optional[typing.Iterable[str]] = None,
) -> int:
    """Add a value to this slot."""

    if self.latest_version_nr != 0 and value.id == self.get_latest_value().id:
        return self.latest_version_nr

    # TODO: check value data equality
    if self.value_schema.is_constant and self.values:
        if is_debug():
            import traceback

            traceback.print_stack()

        raise Exception("Can't add value: value slot marked as 'constant'.")

    version = self.latest_version_nr + 1
    assert version not in self.values.keys()
    self.values[version] = value
    if tags:
        for tag in tags:
            self.tags[tag] = version

    if trigger_callbacks:
        for cb in self._callbacks.values():
            cb.values_updated(self)

    return version