module_types
deserialize.file
Documentation
Deserialize data to a 'file' value instance.
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo: https://github.com/DHARPA-Project/kiara
documentation: https://dharpa.org/kiara_documentation/
Module config schema
Field Type Description Required Default
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
constants object Value constants for this module. no
defaults object Value defaults for this module. no
serialization_profile string The name of the serialization profile used to serialize yes
the source value.
target_profile string The profile name of the de-serialization result data. yes
value_type string The value type of the actual (unserialized) value. yes
Python class
python_class_name DeserializeFileModule
python_module_name kiara.modules.included_core_modules.filesystem
full_name kiara.modules.included_core_modules.filesystem.DeserializeFileModule
Processing source code ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: ValueMap) -> None:
value_type = self.get_config_value("value_type")
serialized_value = inputs.get_value_obj(value_type)
config = inputs.get_value_obj("deserialization_config")
target_profile = self.get_config_value("target_profile")
func_name = f"to__{target_profile}"
func = getattr(self, func_name)
if config.is_set:
_config = config.data
else:
_config = {}
result: Any = func(data=serialized_value.serialized_data, **_config)
outputs.set_value("python_object", result)
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
deserialize.file_bundle
Documentation
Deserialize data to a 'file' value instance.
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo: https://github.com/DHARPA-Project/kiara
documentation: https://dharpa.org/kiara_documentation/
Module config schema
Field Type Description Required Default
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
constants object Value constants for this module. no
defaults object Value defaults for this module. no
serialization_profile string The name of the serialization profile used to serialize yes
the source value.
target_profile string The profile name of the de-serialization result data. yes
value_type string The value type of the actual (unserialized) value. yes
Python class
python_class_name DeserializeFileBundleModule
python_module_name kiara.modules.included_core_modules.filesystem
full_name kiara.modules.included_core_modules.filesystem.DeserializeFileBundleModule
Processing source code ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: ValueMap) -> None:
value_type = self.get_config_value("value_type")
serialized_value = inputs.get_value_obj(value_type)
config = inputs.get_value_obj("deserialization_config")
target_profile = self.get_config_value("target_profile")
func_name = f"to__{target_profile}"
func = getattr(self, func_name)
if config.is_set:
_config = config.data
else:
_config = {}
result: Any = func(data=serialized_value.serialized_data, **_config)
outputs.set_value("python_object", result)
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
deserialize.from_json
Documentation
-- n/a --
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo: https://github.com/DHARPA-Project/kiara
documentation: https://dharpa.org/kiara_documentation/
Module config schema
Field Type Description Required Default
──────────────────────────────────────────────────────────────────────────────────────────
constants object Value constants for this module. no
defaults object Value defaults for this module. no
result_path string The path of the result dictionary to return. no "data"
Python class
python_class_name DeserializeFromJsonModule
python_module_name kiara.modules.included_core_modules.serialization
full_name kiara.modules.included_core_modules.serialization.DeserializeFromJsonModule
Processing source code ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: ValueMap):
value: Value = inputs.get_value_obj("value")
serialized: SerializedData = value.serialized_data
chunks = serialized.get_serialized_data(self.get_config_value("result_path"))
assert chunks.get_number_of_chunks() == 1
_data = list(chunks.get_chunks(as_files=False))
assert len(_data) == 1
_chunk: bytes = _data[0] # type: ignore
deserialized = orjson.loads(_chunk)
outputs.set_value("python_object", deserialized)
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
import.file
Documentation
Import a file from the local filesystem.
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo: https://github.com/DHARPA-Project/kiara
documentation: https://dharpa.org/kiara_documentation/
Module config schema
Field Type Description Required Default
────────────────────────────────────────────────────────────────────────────
constants object Value constants for this module. no
defaults object Value defaults for this module. no
Python class
python_class_name ImportFileModule
python_module_name kiara.modules.included_core_modules.filesystem
full_name kiara.modules.included_core_modules.filesystem.ImportFileModule
Processing source code ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: ValueMap):
path = inputs.get_value_data("path")
file = FileModel.load_file(source=path)
outputs.set_value("file", file)
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
import.file_bundle
Documentation
Import a folder (file_bundle) from the local filesystem.
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo: https://github.com/DHARPA-Project/kiara
documentation: https://dharpa.org/kiara_documentation/
Module config schema
Field Type Description Required Default
────────────────────────────────────────────────────────────────────────────
constants object Value constants for this module. no
defaults object Value defaults for this module. no
Python class
python_class_name ImportFileBundleModule
python_module_name kiara.modules.included_core_modules.filesystem
full_name kiara.modules.included_core_modules.filesystem.ImportFileBundleModule
Processing source code ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: ValueMap):
path = inputs.get_value_data("path")
file_bundle = FileBundle.import_folder(source=path)
outputs.set_value("file_bundle", file_bundle)
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
load.bytes
Documentation
-- n/a --
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo: https://github.com/DHARPA-Project/kiara
documentation: https://dharpa.org/kiara_documentation/
Module config schema
Field Type Description Required Default
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
constants object Value constants for this module. no
defaults object Value defaults for this module. no
serialization_profile string The name of the serialization profile used to serialize yes
the source value.
target_profile string The profile name of the de-serialization result data. yes
value_type string The value type of the actual (unserialized) value. yes
Python class
python_class_name LoadBytesModule
python_module_name kiara.modules.included_core_modules.serialization
full_name kiara.modules.included_core_modules.serialization.LoadBytesModule
Processing source code ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: ValueMap) -> None:
value_type = self.get_config_value("value_type")
serialized_value = inputs.get_value_obj(value_type)
config = inputs.get_value_obj("deserialization_config")
target_profile = self.get_config_value("target_profile")
func_name = f"to__{target_profile}"
func = getattr(self, func_name)
if config.is_set:
_config = config.data
else:
_config = {}
result: Any = func(data=serialized_value.serialized_data, **_config)
outputs.set_value("python_object", result)
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
load.internal_model
Documentation
-- n/a --
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo: https://github.com/DHARPA-Project/kiara
documentation: https://dharpa.org/kiara_documentation/
Module config schema
Field Type Description Required Default
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
constants object Value constants for this module. no
defaults object Value defaults for this module. no
serialization_profile string The name of the serialization profile used to serialize yes
the source value.
target_profile string The profile name of the de-serialization result data. yes
value_type string The value type of the actual (unserialized) value. yes
Python class
python_class_name LoadInternalModel
python_module_name kiara.modules.included_core_modules.serialization
full_name kiara.modules.included_core_modules.serialization.LoadInternalModel
Processing source code ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: ValueMap) -> None:
value_type = self.get_config_value("value_type")
serialized_value = inputs.get_value_obj(value_type)
config = inputs.get_value_obj("deserialization_config")
target_profile = self.get_config_value("target_profile")
func_name = f"to__{target_profile}"
func = getattr(self, func_name)
if config.is_set:
_config = config.data
else:
_config = {}
result: Any = func(data=serialized_value.serialized_data, **_config)
outputs.set_value("python_object", result)
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
load.string
Documentation
-- n/a --
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo: https://github.com/DHARPA-Project/kiara
documentation: https://dharpa.org/kiara_documentation/
Module config schema
Field Type Description Required Default
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
constants object Value constants for this module. no
defaults object Value defaults for this module. no
serialization_profile string The name of the serialization profile used to serialize yes
the source value.
target_profile string The profile name of the de-serialization result data. yes
value_type string The value type of the actual (unserialized) value. yes
Python class
python_class_name LoadStringModule
python_module_name kiara.modules.included_core_modules.serialization
full_name kiara.modules.included_core_modules.serialization.LoadStringModule
Processing source code ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: ValueMap) -> None:
value_type = self.get_config_value("value_type")
serialized_value = inputs.get_value_obj(value_type)
config = inputs.get_value_obj("deserialization_config")
target_profile = self.get_config_value("target_profile")
func_name = f"to__{target_profile}"
func = getattr(self, func_name)
if config.is_set:
_config = config.data
else:
_config = {}
result: Any = func(data=serialized_value.serialized_data, **_config)
outputs.set_value("python_object", result)
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
pipeline
Documentation
-- n/a --
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo: https://github.com/DHARPA-Project/kiara
documentation: https://dharpa.org/kiara_documentation/
Module config schema
Field Type Description Required Default
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
constants object Value constants for this module. no
context object Metadata for this workflow. no
defaults object Value defaults for this module. no
doc -- check source -- Documentation about what the pipeline does. no "-- n/a --"
input_aliases object A map of input aliases, with the calculated yes
(<step_id>__<input_name> -- double underscore!)
name as key, and a string (the resulting
workflow input alias) as value. Check the
documentation for the config class for which
marker strings can be used to automatically
create this map if possible.
output_aliases object A map of output aliases, with the calculated yes
(<step_id>__<output_name> -- double underscore!)
name as key, and a string (the resulting
workflow output alias) as value. Check the
documentation for the config class for which
marker strings can be used to automatically
create this map if possible.
pipeline_name string The name of this pipeline. yes
steps array A list of steps/modules of this pipeline, and yes
their connections.
Python class
python_class_name PipelineModule
python_module_name kiara.modules.included_core_modules.pipeline
full_name kiara.modules.included_core_modules.pipeline.PipelineModule
Processing source code ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: ValueMapWritable, job_log: JobLog):
pipeline_structure: PipelineStructure = self.config.structure
pipeline = Pipeline(structure=pipeline_structure, kiara=outputs._kiara)
assert self._job_registry is not None
controller = SinglePipelineBatchController(
pipeline=pipeline, job_registry=self._job_registry
)
pipeline.set_pipeline_inputs(inputs=inputs)
controller.process_pipeline()
# TODO: resolve values first?
outputs.set_values(**pipeline.get_current_pipeline_outputs())
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
pretty_print.any.value
Documentation
-- n/a --
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo: https://github.com/DHARPA-Project/kiara
documentation: https://dharpa.org/kiara_documentation/
Module config schema
Field Type Description Required Default
───────────────────────────────────────────────────────────────────────────────────
constants object Value constants for this module. no
defaults object Value defaults for this module. no
source_type string The value type of the source value. yes
target_type string The value type of the rendered value. yes
Python class
python_class_name PrettyPrintAnyValueModule
python_module_name kiara.modules.included_core_modules.pretty_print
full_name kiara.modules.included_core_modules.pretty_print.PrettyPrintAnyValueModule
Processing source code ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: ValueMap):
source_type = self.get_config_value("source_type")
target_type = self.get_config_value("target_type")
value = inputs.get_value_obj(source_type)
render_config = inputs.get_value_data("render_config")
func_name = f"pretty_print__{source_type}__as__{target_type}"
func = getattr(self, func_name)
# TODO: check function signature is valid
result = func(value=value, render_config=render_config)
outputs.set_value("rendered_value", result)
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
pretty_print.value
Documentation
-- n/a --
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo: https://github.com/DHARPA-Project/kiara
documentation: https://dharpa.org/kiara_documentation/
Module config schema
Field Type Description Required Default
───────────────────────────────────────────────────────────────────────────────────
constants object Value constants for this module. no
defaults object Value defaults for this module. no
source_type string The value type of the source value. yes
target_type string The value type of the rendered value. yes
Python class
python_class_name ValueTypePrettyPrintModule
python_module_name kiara.modules.included_core_modules.pretty_print
full_name kiara.modules.included_core_modules.pretty_print.ValueTypePrettyPrintModule
Processing source code ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: ValueMap):
source_type = self.get_config_value("source_type")
target_type = self.get_config_value("target_type")
source_value = inputs.get_value_obj(source_type)
render_config = inputs.get_value_obj("render_config")
if not source_value.is_set:
outputs.set_value("rendered_value", "-- none/not set --")
return
data_type_cls = source_value.data_type_class.get_class()
data_type = data_type_cls(**source_value.value_schema.type_config)
func_name = f"pretty_print_as__{target_type}"
func = getattr(data_type, func_name)
render_config_dict = render_config.data
if render_config_dict is None:
render_config_dict = {}
result = func(value=source_value, render_config=render_config_dict)
# TODO: check we have the correct type?
outputs.set_value("rendered_value", result)
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
render.value
Documentation
-- n/a --
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo: https://github.com/DHARPA-Project/kiara
documentation: https://dharpa.org/kiara_documentation/
Module config schema
Field Type Description Required Default
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
constants object Value constants for this module. no
defaults object Value defaults for this module. no
render_instruction_type string The id of the model that describes (and handles) the yes
actual rendering.
target_type string The type of the rendered result. yes
Python class
python_class_name RenderValueModule
python_module_name kiara.modules.included_core_modules.render_value
full_name kiara.modules.included_core_modules.render_value.RenderValueModule
Processing source code ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: ValueMap) -> None:
instruction_type = self.get_config_value("render_instruction_type")
model_registry = ModelRegistry.instance()
instr_info: TypeInfo = model_registry.all_models.get(instruction_type) # type: ignore
instr_model: Type[RenderInstruction] = instr_info.python_class.get_class() # type: ignore
data_type_name = instr_model.retrieve_source_type()
render_instruction: RenderInstruction = inputs.get_value_data(
"render_instruction"
)
if not issubclass(render_instruction.__class__, instr_model):
raise KiaraProcessingException(
f"Invalid type for 'render_instruction': must be a subclass of '{instr_model.__name__}'."
)
result_model_type: str = self.get_config_value("target_type")
value: Value = inputs.get_value_obj(data_type_name)
func_name = f"render_as__{result_model_type}"
func = getattr(render_instruction, func_name)
rendered: Union[RenderValueResult, Any] = func(value=value)
try:
rendered_value = rendered.rendered
metadata = rendered.metadata
except Exception:
rendered_value = rendered
metadata = None
if not metadata:
metadata = RenderMetadata()
outputs.set_values(
**{result_model_type: rendered_value, "render_metadata": metadata}
)
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
unpickle.value
Documentation
-- n/a --
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo: https://github.com/DHARPA-Project/kiara
documentation: https://dharpa.org/kiara_documentation/
Module config schema
Field Type Description Required Default
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
constants object Value constants for this module. no
defaults object Value defaults for this module. no
serialization_profile string The name of the serialization profile used to serialize yes
the source value.
target_profile string The profile name of the de-serialization result data. yes
value_type string The value type of the actual (unserialized) value. yes
Python class
python_class_name UnpickleModule
python_module_name kiara.modules.included_core_modules.serialization
full_name kiara.modules.included_core_modules.serialization.UnpickleModule
Processing source code ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: ValueMap) -> None:
value_type = self.get_config_value("value_type")
serialized_value = inputs.get_value_obj(value_type)
config = inputs.get_value_obj("deserialization_config")
target_profile = self.get_config_value("target_profile")
func_name = f"to__{target_profile}"
func = getattr(self, func_name)
if config.is_set:
_config = config.data
else:
_config = {}
result: Any = func(data=serialized_value.serialized_data, **_config)
outputs.set_value("python_object", result)
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Documentation
Base class to use when writing a module to extract metadata from a file.
It's possible to use any arbitrary kiara module for this purpose, but sub-classing this makes it easier.
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo: https://github.com/DHARPA-Project/kiara
documentation: https://dharpa.org/kiara_documentation/
Module config schema
Field Type Description Required Default
────────────────────────────────────────────────────────────────────────────────────────────
constants object Value constants for this module. no
data_type string The data type this module will be used for. yes
defaults object Value defaults for this module. no
kiara_model_id string The id of the kiara (metadata) model. yes
Python class
python_class_name ExtractMetadataModule
python_module_name kiara.modules.included_core_modules.metadata
full_name kiara.modules.included_core_modules.metadata.ExtractMetadataModule
Processing source code ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: ValueMap) -> None:
value = inputs.get_value_obj("value")
kiara_model_id: str = self.get_config_value("kiara_model_id")
model_registry = ModelRegistry.instance()
metadata_model_cls: Type[ValueMetadata] = model_registry.get_model_cls(kiara_model_id=kiara_model_id, requ…
metadata = metadata_model_cls.create_value_metadata(value=value)
if not isinstance(metadata, metadata_model_cls):
raise KiaraProcessingException(
f"Invalid metadata model result, should be class '{metadata_model_cls.__name__}', but is: {metadat…
)
outputs.set_value("value_metadata", metadata)
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────