module_types
deserialize.file
Documentation
Deserialize data to a 'file' value instance.
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_documentatio…
Module config schema
Field Type Descript… Required Default
─────────────────────────────────────────────────────
constants object Value no
constants
for this
module.
defaults object Value no
defaults
for this
module.
serializ… string The name yes
of the
serializ…
profile
used to
serialize
the
source
value.
target_p… string The yes
profile
name of
the
de-seria…
result
data.
value_ty… string The value yes
type of
the
actual
(unseria…
value.
Python class
python_class_name DeserializeFileModule
python_module_name kiara.modules.included_core_m…
full_name kiara.modules.included_core_m…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: Value…
value_type = self.get_config_value("value_type…
serialized_value = inputs.get_value_obj(value_…
config = inputs.get_value_obj("deserialization…
target_profile = self.get_config_value("target…
func_name = f"to__{target_profile}"
func = getattr(self, func_name)
if config.is_set:
_config = config.data
else:
_config = {}
result: Any = func(data=serialized_value.seria…
outputs.set_value("python_object", result)
─────────────────────────────────────────────────────
deserialize.file_bundle
Documentation
Deserialize data to a 'file' value instance.
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_documentatio…
Module config schema
Field Type Descript… Required Default
─────────────────────────────────────────────────────
constants object Value no
constants
for this
module.
defaults object Value no
defaults
for this
module.
serializ… string The name yes
of the
serializ…
profile
used to
serialize
the
source
value.
target_p… string The yes
profile
name of
the
de-seria…
result
data.
value_ty… string The value yes
type of
the
actual
(unseria…
value.
Python class
python_class_name DeserializeFileBundleModule
python_module_name kiara.modules.included_core_m…
full_name kiara.modules.included_core_m…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: Value…
value_type = self.get_config_value("value_type…
serialized_value = inputs.get_value_obj(value_…
config = inputs.get_value_obj("deserialization…
target_profile = self.get_config_value("target…
func_name = f"to__{target_profile}"
func = getattr(self, func_name)
if config.is_set:
_config = config.data
else:
_config = {}
result: Any = func(data=serialized_value.seria…
outputs.set_value("python_object", result)
─────────────────────────────────────────────────────
deserialize.from_json
Documentation
-- n/a --
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_documentatio…
Module config schema
Field Type Descript… Required Default
─────────────────────────────────────────────────────
constants object Value no
constants
for this
module.
defaults object Value no
defaults
for this
module.
result_p… string The path no "data"
of the
result
dictiona…
to
return.
Python class
python_class_name DeserializeFromJsonModule
python_module_name kiara.modules.included_core_m…
full_name kiara.modules.included_core_m…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: Value…
value: Value = inputs.get_value_obj("value")
serialized: SerializedData = value.serialized_…
chunks = serialized.get_serialized_data(self.g…
assert chunks.get_number_of_chunks() == 1
_data = list(chunks.get_chunks(as_files=False))
assert len(_data) == 1
_chunk: bytes = _data[0] # type: ignore
deserialized = orjson.loads(_chunk)
outputs.set_value("python_object", deserialize…
─────────────────────────────────────────────────────
import.file
Documentation
Import a file from the local filesystem.
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_documentatio…
Module config schema
Field Type Descript… Required Default
─────────────────────────────────────────────────────
constants object Value no
constants
for this
module.
defaults object Value no
defaults
for this
module.
Python class
python_class_name ImportFileModule
python_module_name kiara.modules.included_core_m…
full_name kiara.modules.included_core_m…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: Value…
path = inputs.get_value_data("path")
file = FileModel.load_file(source=path)
outputs.set_value("file", file)
─────────────────────────────────────────────────────
import.file_bundle
Documentation
Import a folder (file_bundle) from the local
filesystem.
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_documentatio…
Module config schema
Field Type Descript… Required Default
─────────────────────────────────────────────────────
constants object Value no
constants
for this
module.
defaults object Value no
defaults
for this
module.
Python class
python_class_name ImportFileBundleModule
python_module_name kiara.modules.included_core_m…
full_name kiara.modules.included_core_m…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: Value…
path = inputs.get_value_data("path")
file_bundle = FileBundle.import_folder(source=…
outputs.set_value("file_bundle", file_bundle)
─────────────────────────────────────────────────────
load.bytes
Documentation
-- n/a --
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_documentatio…
Module config schema
Field Type Descript… Required Default
─────────────────────────────────────────────────────
constants object Value no
constants
for this
module.
defaults object Value no
defaults
for this
module.
serializ… string The name yes
of the
serializ…
profile
used to
serialize
the
source
value.
target_p… string The yes
profile
name of
the
de-seria…
result
data.
value_ty… string The value yes
type of
the
actual
(unseria…
value.
Python class
python_class_name LoadBytesModule
python_module_name kiara.modules.included_core_m…
full_name kiara.modules.included_core_m…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: Value…
value_type = self.get_config_value("value_type…
serialized_value = inputs.get_value_obj(value_…
config = inputs.get_value_obj("deserialization…
target_profile = self.get_config_value("target…
func_name = f"to__{target_profile}"
func = getattr(self, func_name)
if config.is_set:
_config = config.data
else:
_config = {}
result: Any = func(data=serialized_value.seria…
outputs.set_value("python_object", result)
─────────────────────────────────────────────────────
load.internal_model
Documentation
-- n/a --
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_documentatio…
Module config schema
Field Type Descript… Required Default
─────────────────────────────────────────────────────
constants object Value no
constants
for this
module.
defaults object Value no
defaults
for this
module.
serializ… string The name yes
of the
serializ…
profile
used to
serialize
the
source
value.
target_p… string The yes
profile
name of
the
de-seria…
result
data.
value_ty… string The value yes
type of
the
actual
(unseria…
value.
Python class
python_class_name LoadInternalModel
python_module_name kiara.modules.included_core_m…
full_name kiara.modules.included_core_m…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: Value…
value_type = self.get_config_value("value_type…
serialized_value = inputs.get_value_obj(value_…
config = inputs.get_value_obj("deserialization…
target_profile = self.get_config_value("target…
func_name = f"to__{target_profile}"
func = getattr(self, func_name)
if config.is_set:
_config = config.data
else:
_config = {}
result: Any = func(data=serialized_value.seria…
outputs.set_value("python_object", result)
─────────────────────────────────────────────────────
load.string
Documentation
-- n/a --
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_documentatio…
Module config schema
Field Type Descript… Required Default
─────────────────────────────────────────────────────
constants object Value no
constants
for this
module.
defaults object Value no
defaults
for this
module.
serializ… string The name yes
of the
serializ…
profile
used to
serialize
the
source
value.
target_p… string The yes
profile
name of
the
de-seria…
result
data.
value_ty… string The value yes
type of
the
actual
(unseria…
value.
Python class
python_class_name LoadStringModule
python_module_name kiara.modules.included_core_m…
full_name kiara.modules.included_core_m…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: Value…
value_type = self.get_config_value("value_type…
serialized_value = inputs.get_value_obj(value_…
config = inputs.get_value_obj("deserialization…
target_profile = self.get_config_value("target…
func_name = f"to__{target_profile}"
func = getattr(self, func_name)
if config.is_set:
_config = config.data
else:
_config = {}
result: Any = func(data=serialized_value.seria…
outputs.set_value("python_object", result)
─────────────────────────────────────────────────────
pipeline
Documentation
-- n/a --
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_documentatio…
Module config schema
Field Type Descript… Required Default
─────────────────────────────────────────────────────
constan… object Value no
constants
for this
module.
context object Metadata no
for this
workflow.
defaults object Value no
defaults
for this
module.
doc string Document… no "-- n/a
about --"
what the
pipeline
does.
input_a… object A map of yes
input
aliases,
with the
calculat…
(<step_i…
-- double
undersco…
name as
key, and
a string
(the
resulting
workflow
input
alias) as
value.
Check the
document…
for the
config
class for
which
marker
strings
can be
used to
automati…
create
this map
if
possible.
output_… object A map of yes
output
aliases,
with the
calculat…
(<step_i…
-- double
undersco…
name as
key, and
a string
(the
resulting
workflow
output
alias) as
value.
Check the
document…
for the
config
class for
which
marker
strings
can be
used to
automati…
create
this map
if
possible.
pipelin… string The name yes
of this
pipeline.
steps array A list of yes
steps/mo…
of this
pipeline,
and their
connecti…
Python class
python_class_name PipelineModule
python_module_name kiara.modules.included_core_m…
full_name kiara.modules.included_core_m…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: Value…
pipeline_structure: PipelineStructure = self.c…
pipeline = Pipeline(structure=pipeline_structu…
assert self._job_registry is not None
controller = SinglePipelineBatchController(
pipeline=pipeline, job_registry=self._job_…
)
pipeline.set_pipeline_inputs(inputs=inputs)
controller.process_pipeline()
# TODO: resolve values first?
outputs.set_values(**pipeline.get_current_pipe…
─────────────────────────────────────────────────────
pretty_print.any.value
Documentation
-- n/a --
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_documentatio…
Module config schema
Field Type Descript… Required Default
─────────────────────────────────────────────────────
constants object Value no
constants
for this
module.
defaults object Value no
defaults
for this
module.
source_t… string The value yes
type of
the
source
value.
target_t… string The value yes
type of
the
rendered
value.
Python class
python_class_name PrettyPrintAnyValueModule
python_module_name kiara.modules.included_core_m…
full_name kiara.modules.included_core_m…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: Value…
source_type = self.get_config_value("source_ty…
target_type = self.get_config_value("target_ty…
value = inputs.get_value_obj(source_type)
render_config = inputs.get_value_data("render_…
func_name = f"pretty_print__{source_type}__as_…
func = getattr(self, func_name)
# TODO: check function signature is valid
result = func(value=value, render_config=rende…
outputs.set_value("rendered_value", result)
─────────────────────────────────────────────────────
pretty_print.value
Documentation
-- n/a --
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_documentatio…
Module config schema
Field Type Descript… Required Default
─────────────────────────────────────────────────────
constants object Value no
constants
for this
module.
defaults object Value no
defaults
for this
module.
source_t… string The value yes
type of
the
source
value.
target_t… string The value yes
type of
the
rendered
value.
Python class
python_class_name ValueTypePrettyPrintModule
python_module_name kiara.modules.included_core_m…
full_name kiara.modules.included_core_m…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: Value…
source_type = self.get_config_value("source_ty…
target_type = self.get_config_value("target_ty…
source_value = inputs.get_value_obj(source_typ…
render_config = inputs.get_value_obj("render_c…
if not source_value.is_set:
outputs.set_value("rendered_value", "-- no…
return
data_type_cls = source_value.data_type_class.g…
data_type = data_type_cls(**source_value.value…
func_name = f"pretty_print_as__{target_type}"
func = getattr(data_type, func_name)
render_config_dict = render_config.data
if render_config_dict is None:
render_config_dict = {}
result = func(value=source_value, render_confi…
# TODO: check we have the correct type?
outputs.set_value("rendered_value", result)
─────────────────────────────────────────────────────
render.value
Documentation
-- n/a --
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_documentatio…
Module config schema
Field Type Descript… Required Default
─────────────────────────────────────────────────────
constants object Value no
constants
for this
module.
defaults object Value no
defaults
for this
module.
render_i… string The id of yes
the model
that
describes
(and
handles)
the
actual
renderin…
target_t… string The type yes
of the
rendered
result.
Python class
python_class_name RenderValueModule
python_module_name kiara.modules.included_core_m…
full_name kiara.modules.included_core_m…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: Value…
instruction_type = self.get_config_value("rend…
model_registry = ModelRegistry.instance()
instr_info: TypeInfo = model_registry.all_mode…
instr_model: Type[RenderInstruction] = instr_i…
data_type_name = instr_model.retrieve_source_t…
render_instruction: RenderInstruction = inputs…
"render_instruction"
)
if not issubclass(render_instruction.__class__…
raise KiaraProcessingException(
f"Invalid type for 'render_instruction…
)
result_model_type: str = self.get_config_value…
value: Value = inputs.get_value_obj(data_type_…
func_name = f"render_as__{result_model_type}"
func = getattr(render_instruction, func_name)
rendered: Union[RenderValueResult, Any] = func…
try:
rendered_value = rendered.rendered
metadata = rendered.metadata
except Exception:
rendered_value = rendered
metadata = None
if not metadata:
metadata = RenderMetadata()
outputs.set_values(
**{result_model_type: rendered_value, "ren…
)
─────────────────────────────────────────────────────
unpickle.value
Documentation
-- n/a --
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_documentatio…
Module config schema
Field Type Descript… Required Default
─────────────────────────────────────────────────────
constants object Value no
constants
for this
module.
defaults object Value no
defaults
for this
module.
serializ… string The name yes
of the
serializ…
profile
used to
serialize
the
source
value.
target_p… string The yes
profile
name of
the
de-seria…
result
data.
value_ty… string The value yes
type of
the
actual
(unseria…
value.
Python class
python_class_name UnpickleModule
python_module_name kiara.modules.included_core_m…
full_name kiara.modules.included_core_m…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: Value…
value_type = self.get_config_value("value_type…
serialized_value = inputs.get_value_obj(value_…
config = inputs.get_value_obj("deserialization…
target_profile = self.get_config_value("target…
func_name = f"to__{target_profile}"
func = getattr(self, func_name)
if config.is_set:
_config = config.data
else:
_config = {}
result: Any = func(data=serialized_value.seria…
outputs.set_value("python_object", result)
─────────────────────────────────────────────────────
Documentation
Base class to use when writing a module to extract
metadata from a file.
It's possible to use any arbitrary kiara module for
this purpose, but sub-classing this makes it
easier.
Author(s)
Markus Binsteiner markus@frkl.io
Context
Labels package: kiara
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_documentatio…
Module config schema
Field Type Descript… Required Default
─────────────────────────────────────────────────────
constants object Value no
constants
for this
module.
data_type string The data yes
type this
module
will be
used for.
defaults object Value no
defaults
for this
module.
kiara_mo… string The id of yes
the kiara
(metadat…
model.
Python class
python_class_name ExtractMetadataModule
python_module_name kiara.modules.included_core_m…
full_name kiara.modules.included_core_m…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueMap, outputs: Value…
value = inputs.get_value_obj("value")
kiara_model_id: str = self.get_config_value("k…
model_registry = ModelRegistry.instance()
metadata_model_cls: Type[ValueMetadata] = mode…
metadata = metadata_model_cls.create_value_met…
if not isinstance(metadata, metadata_model_cls…
raise KiaraProcessingException(
f"Invalid metadata model result, shoul…
)
outputs.set_value("value_metadata", metadata)
─────────────────────────────────────────────────────