kiara.data.types¶
This is the base module that contains everything data type-related in kiara.
I'm still not 100% sure how to best implement the kiara type system, there are several ways it could be done, for example based on Python type-hints, using JSON-schema, Avro (which is my 2nd favourite option), as well as by implementing a custom type-class hierarchy. Which is what I have choosen to try first. For now, it looks like it'll work out, but there is a chance requirements I haven't forseen will crop up that could make this become ugly.
Anyway, the way it works (for now) is that kiara comes with a set of often used types (the standard set of: scalars, list, dict, table & array, etc.) which each come with 2 functions that can serialize and deserialize values of that type in a persistant fashion -- which could be storing as a file on disk, or as a cell/row in a database. Those functions will most likley be kiara modules themselves, with even more restricted input/output type options.
In addition, packages that contain modules can implement their own, custom types, if suitable ones are not available in core-kiara. Those can either be 'serialized/deserialized' into kiara-native types (which in turn will serialize them using their own serializing functions), or will have to implement custom serializing functionality (which will probably be discouraged, since this might not be trivial and there are quite a few things to consider).
ValueType
¶
Base class that all kiara types must inherit from.
kiara types have 3 main responsibilities:
- serialize into / deserialize from persistent state
- data validation
- metadata extraction
Serializing being the arguably most important of those, because without most of the data management features of kiara would be impossible. Validation should not require any explanation. Metadata extraction is important, because that metadata will be available to other components of kiara (or frontends for it), without them having to request the actual data. That will hopefully make kiara very efficient in terms of memory management, as well as data transfer and I/O. Ideally, the actual data (bytes) will only be requested at the last possible moment. For example when a module needs the input data to do processing on it -- and even then it might be that it only requests a part of the data, say a single column of a table. Or when a frontend needs to display/visualize the data.
calculate_value_hash(value, hash_type)
classmethod
¶
Calculate the hash of this value.
If a hash can't be calculated, or the calculation of a type is not implemented (yet), this will return None.
Source code in kiara/data/types/__init__.py
@classmethod
def calculate_value_hash(cls, value: typing.Any, hash_type: str) -> str:
"""Calculate the hash of this value.
If a hash can't be calculated, or the calculation of a type is not implemented (yet), this will return None.
"""
raise Exception(f"Value type '{cls._value_type_name}' does not support hash calculation.") # type: ignore
check_data(data)
classmethod
¶
Check whether the provided input matches this value type.
If it does, return a ValueType object (with the appropriate type configuration).
Source code in kiara/data/types/__init__.py
@classmethod
def check_data(cls, data: typing.Any) -> typing.Optional["ValueType"]:
"""Check whether the provided input matches this value type.
If it does, return a ValueType object (with the appropriate type configuration).
"""
return None
get_type_hint(self, context='python')
¶
Return a type hint for this value type object.
This can be used by kiara interfaces to document/validate user input. For now, only 'python' type hints are expected to be implemented, but in theory this could also return type hints for other contexts.
Source code in kiara/data/types/__init__.py
def get_type_hint(self, context: str = "python") -> typing.Optional[typing.Type]:
"""Return a type hint for this value type object.
This can be used by kiara interfaces to document/validate user input. For now, only 'python' type hints are
expected to be implemented, but in theory this could also return type hints for other contexts.
"""
return None
parse_value(self, value)
¶
Parse a value into a supported python type.
This exists to make it easier to do trivial conversions (e.g. from a date string to a datetime object). If you choose to overwrite this method, make 100% sure that you don't change the meaning of the value, and try to avoid adding or removing information from the data (e.g. by changing the resolution of a date).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
v |
|
the value |
required |
Returns:
| Type | Description |
|---|---|
Any |
'None', if no parsing was done and the original value should be used, otherwise return the parsed Python object |
Source code in kiara/data/types/__init__.py
def parse_value(self, value: typing.Any) -> typing.Any:
"""Parse a value into a supported python type.
This exists to make it easier to do trivial conversions (e.g. from a date string to a datetime object).
If you choose to overwrite this method, make 100% sure that you don't change the meaning of the value, and try to
avoid adding or removing information from the data (e.g. by changing the resolution of a date).
Arguments:
v: the value
Returns:
'None', if no parsing was done and the original value should be used, otherwise return the parsed Python object
"""
return None
validate(cls, value)
¶
Validate the value. This expects an instance of the defined Python class (from 'backing_python_type).
Source code in kiara/data/types/__init__.py
def validate(cls, value: typing.Any) -> None:
"""Validate the value. This expects an instance of the defined Python class (from 'backing_python_type)."""
ValueTypeConfigMetadata
pydantic-model
¶
ValueTypeConfigSchema
pydantic-model
¶
Base class that describes the configuration a [ValueType][kiara.data.types.ValueType] class accepts.
This is stored in the _config_cls class attribute in each ValueType class. By default,
a ValueType is not configurable, unless the _config_cls class attribute points to a sub-class of this class.
__eq__(self, other)
special
¶
Return self==value.
Source code in kiara/data/types/__init__.py
def __eq__(self, other):
if self.__class__ != other.__class__:
return False
return self.dict() == other.dict()
__hash__(self)
special
¶
Return hash(self).
Source code in kiara/data/types/__init__.py
def __hash__(self):
return hash(self.config_hash)
get(self, key)
¶
Get the value for the specified configuation key.
Source code in kiara/data/types/__init__.py
def get(self, key: str) -> typing.Any:
"""Get the value for the specified configuation key."""
if key not in self.__fields__:
raise Exception(
f"No config value '{key}' in module config class '{self.__class__.__name__}'."
)
return getattr(self, key)
requires_config()
classmethod
¶
Return whether this class can be used as-is, or requires configuration before an instance can be created.
Source code in kiara/data/types/__init__.py
@classmethod
def requires_config(cls) -> bool:
"""Return whether this class can be used as-is, or requires configuration before an instance can be created."""
for field_name, field in cls.__fields__.items():
if field.required and field.default is None:
return True
return False
get_type_name(obj)
¶
Utility function to get a pretty string from the class of an object.
Source code in kiara/data/types/__init__.py
def get_type_name(obj: typing.Any):
"""Utility function to get a pretty string from the class of an object."""
if obj.__class__.__module__ == "builtins":
return obj.__class__.__name__
else:
return f"{obj.__class__.__module__}.{obj.__class__.__name__}"