Available module types¶
This page contains a list of all available Kiara module types, and their details.
Note
The formatting here will be improved later on, for now this should be enough to get the important details of each module type.
array.map¶
Documentation
Map a list of values into another list of values.
This module must be configured with the type (and
optional) configuration of another kiara module.
This 'child' module will then be used to compute
the array items of the result.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags array, core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this
module.
defaults object Value defaults no
for this
module.
module_type string The name of the yes
kiara module to
use to filter
the input data.
module_config object The config for no
the kiara
filter module.
input_name string The name of the no
input name of
the module
which will
receive the
items from our
input array.
Can be omitted
if the
configured
module only has
a single input.
output_name string The name of the no
output name of
the module
which will
receive the
items from our
input array.
Can be omitted
if the
configured
module only has
a single
output.
Module config -- no config --
Python class
class_name MapModule
module_name kiara_modules.core.array
full_name kiara_modules.core.array.MapModule
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
import pyarrow as pa
input_array: pa.Array = inputs.get_value_data(…
init_data: typing.Dict[str, typing.Any] = {}
for input_name in self.input_schemas.keys():
if input_name in ["array", self.module_inp…
continue
init_data[input_name] = inputs.get_value_o…
result_list = map_with_module(
input_array,
module_input_name=self.module_input_name,
module_obj=self.child_module,
init_data=init_data,
module_output_name=self.module_output_name,
)
outputs.set_value("array", pa.array(result_lis…
─────────────────────────────────────────────────────
array.metadata¶
Documentation
Extract metadata from an 'array' value.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags array, core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults for no
this module.
value_type string The data type this yes
module will be
used for.
Module config -- no config --
Python class
class_name ArrayMetadataModule
module_name kiara_modules.core.array
full_name kiara_modules.core.array.ArrayMetada…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
input_name = self.value_type
if input_name == "any":
input_name = "value_item"
value = inputs.get_value_obj(input_name)
if self.value_type != "any" and value.type_nam…
raise KiaraProcessingException(
f"Can't extract metadata for value of …
)
# TODO: if type 'any', validate that the data …
outputs.set_value("metadata_item_schema", self…
metadata = self.extract_metadata(value)
if isinstance(metadata, BaseModel):
metadata = metadata.dict(exclude_none=True)
# TODO: validate metadata?
outputs.set_value("metadata_item", metadata)
─────────────────────────────────────────────────────
array.sample¶
Documentation
Sample an array.
Samples are used to randomly select a subset of a
dataset, which helps test queries and workflows on
smaller versions of the original data, to adjust
parameters before a full run.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags array, core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults no
for this module.
sample_type string The sample yes
method.
Module config -- no config --
Python class
class_name SampleArrayModule
module_name kiara_modules.core.array
full_name kiara_modules.core.array.SampleArray…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
sample_size: int = inputs.get_value_data("samp…
sample_type: str = self.get_config_value("samp…
if sample_size < 0:
raise KiaraProcessingException(
f"Invalid sample size '{sample_size}':…
)
input_name = self.get_value_type()
if input_name == "any":
input_name = "value_item"
value: Value = inputs.get_value_obj(input_name)
func = getattr(self, f"sample_{sample_type}")
result = func(value=value, sample_size=sample_…
outputs.set_value("sampled_value", result)
─────────────────────────────────────────────────────
array.store¶
Documentation
Save an Arrow array to a file.
This module wraps the input array into an Arrow
Table, and saves this table as a feather file.
The output of this module is a dictionary
representing the configuration to be used with kira
to re-assemble the array object from disk.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags array, core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults for no
this module.
value_type string The type of the yes
value to save.
Module config -- no config --
Python class
class_name StoreArrayTypeModule
module_name kiara_modules.core.array
full_name kiara_modules.core.array.StoreArrayT…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
value_id: str = inputs.get_value_data("value_i…
if not value_id:
raise KiaraProcessingException("No value i…
field_name = self.get_config_value("value_type…
if field_name == "any":
field_name = "value_item"
value_obj: Value = inputs.get_value_obj(field_…
base_path: str = inputs.get_value_data("base_p…
result = self.store_value(value=value_obj, bas…
if isinstance(result, typing.Mapping):
load_config = result
result_value = value_obj
elif isinstance(result, tuple):
load_config = result[0]
if result[1]:
result_value = result[1]
else:
result_value = value_obj
else:
raise KiaraProcessingException(
f"Invalid result type for 'store_value…
)
load_config["value_id"] = value_id
lc = LoadConfig(**load_config)
if lc.base_path_input_name and lc.base_path_in…
raise KiaraProcessingException(
f"Invalid load config: base path '{lc.…
)
outputs.set_values(
metadata=None, lineage=None, **{"load_conf…
)
─────────────────────────────────────────────────────
bytes.load¶
Documentation
-- n/a --
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core, serialization, bytes
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants for no
this module.
defaults object Value defaults for no
this module.
Module config -- no config --
Python class
class_name LoadBytesModule
module_name kiara_modules.core.bytes
full_name kiara_modules.core.bytes.LoadBytesMo…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
path = inputs.get_value_data("path")
if not os.path.exists(path):
raise KiaraProcessingException(
f"Can't read file, path does not exist…
)
with open(path, "rb") as f:
content = f.read()
outputs.set_value("bytes", content)
─────────────────────────────────────────────────────
bytes.msgpack.from_value¶
Documentation
-- n/a --
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core, serialization, msgpack, bytes
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults for no
this module.
value_type string The value type to yes
serialize/deseria…
Module config -- no config --
Python class
class_name SerializeToMsgPackModule
module_name kiara_modules.core.bytes.msgpack
full_name kiara_modules.core.bytes.msgpack.Ser…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
import msgpack
type_name: str = self.get_config_value("value_…
if not hasattr(self, f"from_{type_name}"):
raise KiaraProcessingException(
f"Value type not supported for msgpack…
)
func = getattr(self, f"from_{type_name}")
value = inputs.get_value_obj("value_item")
metadata = value.get_metadata(also_return_sche…
msg = func(value=value)
data = {"value_type": value.type_name, "metada…
msg = msgpack.packb(data, use_bin_type=True)
outputs.set_value("bytes", msg)
─────────────────────────────────────────────────────
bytes.msgpack.to_value¶
Documentation
-- n/a --
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core, serialization, msgpack, bytes
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults for no
this module.
value_type string The value type to yes
serialize/deseria…
Module config -- no config --
Python class
class_name DeserializeFromMsgPackModule
module_name kiara_modules.core.bytes.msgpack
full_name kiara_modules.core.bytes.msgpack.Des…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
import msgpack
msg = inputs.get_value_data("bytes")
unpacked = msgpack.unpackb(msg, raw=False)
value_type = unpacked["value_type"]
outputs.set_value("value_type", value_type)
metadata = unpacked["metadata"]
outputs.set_value("value_metadata", metadata)
new_data = unpacked["data"]
if not hasattr(self, f"to_{value_type}"):
raise KiaraProcessingException(
f"Value type not supported for msgpack…
)
func = getattr(self, f"to_{value_type}")
obj = func(data=new_data)
outputs.set_value("value_data", obj)
─────────────────────────────────────────────────────
bytes.store¶
Documentation
-- n/a --
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core, serialization, bytes
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults for no
this module.
value_type string The type of the yes
value to save.
Module config -- no config --
Python class
class_name StoreBytesTypeModule
module_name kiara_modules.core.bytes
full_name kiara_modules.core.bytes.StoreBytesT…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
value_id: str = inputs.get_value_data("value_i…
if not value_id:
raise KiaraProcessingException("No value i…
field_name = self.get_config_value("value_type…
if field_name == "any":
field_name = "value_item"
value_obj: Value = inputs.get_value_obj(field_…
base_path: str = inputs.get_value_data("base_p…
result = self.store_value(value=value_obj, bas…
if isinstance(result, typing.Mapping):
load_config = result
result_value = value_obj
elif isinstance(result, tuple):
load_config = result[0]
if result[1]:
result_value = result[1]
else:
result_value = value_obj
else:
raise KiaraProcessingException(
f"Invalid result type for 'store_value…
)
load_config["value_id"] = value_id
lc = LoadConfig(**load_config)
if lc.base_path_input_name and lc.base_path_in…
raise KiaraProcessingException(
f"Invalid load config: base path '{lc.…
)
outputs.set_values(
metadata=None, lineage=None, **{"load_conf…
)
─────────────────────────────────────────────────────
date.extract_from_string¶
Documentation
Extract a date object from a string.
This module is not really smart yet, currently it
uses the following regex to extract a date (which
might fail in a lot of cases):
┌─────────────────────────────────────────────────┐
│ r"_(\d{4}-\d{2}-\d{2})_" │
└─────────────────────────────────────────────────┘
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants for no
this module.
defaults object Value defaults for no
this module.
Module config -- no config --
Python class
class_name ExtractDateModule
module_name kiara_modules.core.date
full_name kiara_modules.core.date.ExtractDateM…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
from dateutil import parser
text = inputs.get_value_data("text")
date_match = re.findall(r"_(\d{4}-\d{2}-\d{2})…
assert date_match
d_obj = parser.parse(date_match[0]) # type: i…
outputs.set_value("date", d_obj)
─────────────────────────────────────────────────────
date.range_check¶
Documentation
Check whether a date falls within a specified date
range.
If none one of the inputs 'earliest' or 'latest' is
set, this module will always return 'True'.
Return True if that's the case, otherwise False.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants for no
this module.
defaults object Value defaults for no
this module.
Module config -- no config --
Python class
class_name DateRangeCheckModule
module_name kiara_modules.core.date
full_name kiara_modules.core.date.DateRangeChe…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
from dateutil import parser
d = inputs.get_value_data("date")
earliest: typing.Optional[datetime.datetime] =…
latest: typing.Optional[datetime.datetime] = i…
if not earliest and not latest:
outputs.set_value("within_range", True)
return
if hasattr(d, "as_py"):
d = d.as_py()
if isinstance(d, str):
d = parser.parse(d)
if earliest and latest:
matches = earliest <= d <= latest
elif earliest:
matches = earliest <= d
else:
matches = d <= latest
outputs.set_value("within_range", matches)
─────────────────────────────────────────────────────
dev.dummy¶
Documentation
Module that simulates processing, but uses
hard-coded outputs as a result.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this
module.
defaults object Value defaults no
for this
module.
documentation string no
input_schema object The input yes
schema for this
module.
output_schema object The output yes
schema for this
module.
outputs object The (dummy) no
output for this
module.
delay number The delay in no
seconds from
processing
start to when
the (dummy)
outputs are
returned.
Module config -- no config --
Python class
class_name DummyModule
module_name kiara_modules.core.dev
full_name kiara_modules.core.dev.DummyModule
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
"""Returns the hardcoded output values that ar…
Optionally, this module can simulate processin…
"""
time.sleep(self.config.get("delay")) # type: …
output_values: typing.Mapping = self.config.ge…
value_dict = {}
for output_name in self.output_names:
if output_name not in output_values.keys():
raise NotImplementedError()
# v = self.output_schemas[output_name]…
# value_dict[output_name] = v
else:
value_dict[output_name] = output_value…
outputs.set_values(**value_dict)
─────────────────────────────────────────────────────
dict.store¶
Documentation
-- n/a --
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults no
for this module.
value_type string The type of the yes
value to save.
options integer The options to no
use for the json
serialization.
Check
https://github.c…
for details.
file_name string The name of the no
serialized file.
Module config -- no config --
Python class
class_name SaveDictModule
module_name kiara_modules.core.dict
full_name kiara_modules.core.dict.SaveDictModu…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
value_id: str = inputs.get_value_data("value_i…
if not value_id:
raise KiaraProcessingException("No value i…
field_name = self.get_config_value("value_type…
if field_name == "any":
field_name = "value_item"
value_obj: Value = inputs.get_value_obj(field_…
base_path: str = inputs.get_value_data("base_p…
result = self.store_value(value=value_obj, bas…
if isinstance(result, typing.Mapping):
load_config = result
result_value = value_obj
elif isinstance(result, tuple):
load_config = result[0]
if result[1]:
result_value = result[1]
else:
result_value = value_obj
else:
raise KiaraProcessingException(
f"Invalid result type for 'store_value…
)
load_config["value_id"] = value_id
lc = LoadConfig(**load_config)
if lc.base_path_input_name and lc.base_path_in…
raise KiaraProcessingException(
f"Invalid load config: base path '{lc.…
)
outputs.set_values(
metadata=None, lineage=None, **{"load_conf…
)
─────────────────────────────────────────────────────
file.import¶
Documentation
Import an external file into a kiara session.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value no
constants for
this module.
defaults object Value defaults no
for this
module.
source_profile string The name of yes
the source
profile. Used
to distinguish
different
input
categories for
the same input
type.
source_type string The type of yes
the source to
import from.
Module config -- no config --
Python class
class_name DefaultFileImportModule
module_name kiara_modules.core.file
full_name kiara_modules.core.file.DefaultFileI…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
source_profile: str = self.get_config_value("s…
source_type: str = self.get_config_value("sour…
source = inputs.get_value_data(source_profile)
if self.get_target_value_type() == "any":
output_key: str = "value_item"
else:
output_key = self.get_target_value_type()
func_name = f"import_from__{source_profile}__{…
if not hasattr(self, func_name):
raise Exception(
f"Can't import '{source_type}' value: …
)
func = getattr(self, func_name)
# TODO: check signature?
result = func(source)
# schema = ValueSchema(type=self.get_target_va…
# value_lineage = ValueLineage.from_module_and…
# module=self, output_name=output_key, inp…
# )
# value: Value = self._kiara.data_registry.reg…
# value_data=result, value_schema=schema, …
# )
outputs.set_value(output_key, result)
─────────────────────────────────────────────────────
file.load¶
Documentation
Load a file and its metadata.
This module does not read or load the content of a
file, but contains the path to the local
representation/version of the file so it can be
read by a subsequent process.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants for no
this module.
defaults object Value defaults for no
this module.
Module config -- no config --
Python class
class_name LoadLocalFileModule
module_name kiara_modules.core.file
full_name kiara_modules.core.file.LoadLocalFil…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
base_path = inputs.get_value_data("base_path")
rel_path = inputs.get_value_data("rel_path")
path = os.path.join(base_path, rel_path)
file_model = KiaraFile.load_file(path)
outputs.set_value("file", file_model)
─────────────────────────────────────────────────────
file.metadata¶
Documentation
-- n/a --
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults for no
this module.
value_type string The data type this yes
module will be
used for.
Module config -- no config --
Python class
class_name FileMetadataModule
module_name kiara_modules.core.file
full_name kiara_modules.core.file.FileMetadata…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
input_name = self.value_type
if input_name == "any":
input_name = "value_item"
value = inputs.get_value_obj(input_name)
if self.value_type != "any" and value.type_nam…
raise KiaraProcessingException(
f"Can't extract metadata for value of …
)
# TODO: if type 'any', validate that the data …
outputs.set_value("metadata_item_schema", self…
metadata = self.extract_metadata(value)
if isinstance(metadata, BaseModel):
metadata = metadata.dict(exclude_none=True)
# TODO: validate metadata?
outputs.set_value("metadata_item", metadata)
─────────────────────────────────────────────────────
file.store¶
Documentation
Save a file to disk.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults for no
this module.
value_type string The type of the yes
value to save.
Module config -- no config --
Python class
class_name StoreFileTypeModule
module_name kiara_modules.core.file
full_name kiara_modules.core.file.StoreFileTyp…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
value_id: str = inputs.get_value_data("value_i…
if not value_id:
raise KiaraProcessingException("No value i…
field_name = self.get_config_value("value_type…
if field_name == "any":
field_name = "value_item"
value_obj: Value = inputs.get_value_obj(field_…
base_path: str = inputs.get_value_data("base_p…
result = self.store_value(value=value_obj, bas…
if isinstance(result, typing.Mapping):
load_config = result
result_value = value_obj
elif isinstance(result, tuple):
load_config = result[0]
if result[1]:
result_value = result[1]
else:
result_value = value_obj
else:
raise KiaraProcessingException(
f"Invalid result type for 'store_value…
)
load_config["value_id"] = value_id
lc = LoadConfig(**load_config)
if lc.base_path_input_name and lc.base_path_in…
raise KiaraProcessingException(
f"Invalid load config: base path '{lc.…
)
outputs.set_values(
metadata=None, lineage=None, **{"load_conf…
)
─────────────────────────────────────────────────────
file_bundle.import¶
Documentation
Import a file bundle into the kiara data store.
This module will support multiple source types and
profiles in the future, but at the moment only
import from local folder is supported. Thus,
requiring the config value 'local' for
'source_profile', and 'folder_path' for
'source_type'.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value no
constants for
this module.
defaults object Value defaults no
for this
module.
source_profile string The name of yes
the source
profile. Used
to distinguish
different
input
categories for
the same input
type.
source_type string The type of yes
the source to
import from.
Module config -- no config --
Python class
class_name DefaultFileBundleImportModule
module_name kiara_modules.core.file_bundle
full_name kiara_modules.core.file_bundle.Defau…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
source_profile: str = self.get_config_value("s…
source_type: str = self.get_config_value("sour…
source = inputs.get_value_data(source_profile)
if self.get_target_value_type() == "any":
output_key: str = "value_item"
else:
output_key = self.get_target_value_type()
func_name = f"import_from__{source_profile}__{…
if not hasattr(self, func_name):
raise Exception(
f"Can't import '{source_type}' value: …
)
func = getattr(self, func_name)
# TODO: check signature?
result = func(source)
# schema = ValueSchema(type=self.get_target_va…
# value_lineage = ValueLineage.from_module_and…
# module=self, output_name=output_key, inp…
# )
# value: Value = self._kiara.data_registry.reg…
# value_data=result, value_schema=schema, …
# )
outputs.set_value(output_key, result)
─────────────────────────────────────────────────────
file_bundle.load¶
Documentation
Load a file bundle and its metadata.
This module does not read or load the content of
all included files, but contains the path to the
local representation/version of them so they can be
read by a subsequent process.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants for no
this module.
defaults object Value defaults for no
this module.
Module config -- no config --
Python class
class_name LoadFileBundleModule
module_name kiara_modules.core.file_bundle
full_name kiara_modules.core.file_bundle.LoadF…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
base_path = inputs.get_value_data("base_path")
rel_path = inputs.get_value_data("rel_path")
path = os.path.join(base_path, rel_path)
included_files = inputs.get_value_data("includ…
excluded_dirs = inputs.get_value_data("exclude…
excluded_files = inputs.get_value_data("exclud…
import_config = FolderImportConfig(
include_files=included_files,
exclude_dirs=excluded_dirs,
excluded_files=excluded_files,
)
bundle = KiaraFileBundle.import_folder(source=…
outputs.set_values(file_bundle=bundle)
─────────────────────────────────────────────────────
file_bundle.metadata¶
Documentation
-- n/a --
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults for no
this module.
value_type string The data type this yes
module will be
used for.
Module config -- no config --
Python class
class_name FileBundleMetadataModule
module_name kiara_modules.core.file_bundle
full_name kiara_modules.core.file_bundle.FileB…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
input_name = self.value_type
if input_name == "any":
input_name = "value_item"
value = inputs.get_value_obj(input_name)
if self.value_type != "any" and value.type_nam…
raise KiaraProcessingException(
f"Can't extract metadata for value of …
)
# TODO: if type 'any', validate that the data …
outputs.set_value("metadata_item_schema", self…
metadata = self.extract_metadata(value)
if isinstance(metadata, BaseModel):
metadata = metadata.dict(exclude_none=True)
# TODO: validate metadata?
outputs.set_value("metadata_item", metadata)
─────────────────────────────────────────────────────
file_bundle.store¶
Documentation
Save a file bundle to disk.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults for no
this module.
value_type string The type of the yes
value to save.
Module config -- no config --
Python class
class_name StoreFileBundleType
module_name kiara_modules.core.file_bundle
full_name kiara_modules.core.file_bundle.Store…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
value_id: str = inputs.get_value_data("value_i…
if not value_id:
raise KiaraProcessingException("No value i…
field_name = self.get_config_value("value_type…
if field_name == "any":
field_name = "value_item"
value_obj: Value = inputs.get_value_obj(field_…
base_path: str = inputs.get_value_data("base_p…
result = self.store_value(value=value_obj, bas…
if isinstance(result, typing.Mapping):
load_config = result
result_value = value_obj
elif isinstance(result, tuple):
load_config = result[0]
if result[1]:
result_value = result[1]
else:
result_value = value_obj
else:
raise KiaraProcessingException(
f"Invalid result type for 'store_value…
)
load_config["value_id"] = value_id
lc = LoadConfig(**load_config)
if lc.base_path_input_name and lc.base_path_in…
raise KiaraProcessingException(
f"Invalid load config: base path '{lc.…
)
outputs.set_values(
metadata=None, lineage=None, **{"load_conf…
)
─────────────────────────────────────────────────────
generic.restore_from_json¶
Documentation
-- n/a --
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants for no
this module.
defaults object Value defaults for no
this module.
Module config -- no config --
Python class
class_name RestoreFromJsonDictModule
module_name kiara_modules.core.generic
full_name kiara_modules.core.generic.RestoreFr…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
base_path = inputs.get_value_data("base_path")
file_name = inputs.get_value_data("file_name")
full_path = os.path.join(base_path, file_name)
if not os.path.exists(full_path):
raise KiaraProcessingException(
f"Can't deserialize dict, path to file…
)
if not os.path.isfile(os.path.realpath(full_pa…
raise KiaraProcessingException(
f"Can't deserialize dict, path is not …
)
with open(full_path, "r") as f:
content = f.read()
data = orjson.loads(content)
outputs.set_value("value_item", data)
─────────────────────────────────────────────────────
generic.restore_scalar¶
Documentation
Utility module, only used internally.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults for no
this module.
value_type string The value type of yes
the scalar to
load.
Module config -- no config --
Python class
class_name RestoreScalarModule
module_name kiara_modules.core.generic
full_name kiara_modules.core.generic.RestoreSc…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
data = inputs.get_value_obj("scalar_data")
outputs.set_value("value_item", data)
─────────────────────────────────────────────────────
generic.store¶
Documentation
-- n/a --
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults for no
this module.
value_type string The type of the yes
value to save.
Module config -- no config --
Python class
class_name StoreScalarModule
module_name kiara_modules.core.generic
full_name kiara_modules.core.generic.StoreScal…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
value_id: str = inputs.get_value_data("value_i…
if not value_id:
raise KiaraProcessingException("No value i…
field_name = self.get_config_value("value_type…
if field_name == "any":
field_name = "value_item"
value_obj: Value = inputs.get_value_obj(field_…
base_path: str = inputs.get_value_data("base_p…
result = self.store_value(value=value_obj, bas…
if isinstance(result, typing.Mapping):
load_config = result
result_value = value_obj
elif isinstance(result, tuple):
load_config = result[0]
if result[1]:
result_value = result[1]
else:
result_value = value_obj
else:
raise KiaraProcessingException(
f"Invalid result type for 'store_value…
)
load_config["value_id"] = value_id
lc = LoadConfig(**load_config)
if lc.base_path_input_name and lc.base_path_in…
raise KiaraProcessingException(
f"Invalid load config: base path '{lc.…
)
outputs.set_values(
metadata=None, lineage=None, **{"load_conf…
)
─────────────────────────────────────────────────────
json.to_json¶
Documentation
Convert arbitrary types into json.
Very early days for this module, it doesn't support
a lot of types yet.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults no
for this module.
source_type string The source type. yes
target_type string The target type. yes
Module config -- no config --
Python class
class_name ToJsonModuleOld
module_name kiara_modules.core.json
full_name kiara_modules.core.json.ToJsonModule…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
value = inputs.get_value_obj("source_value")
if value.value_schema.type != self.source_type:
raise KiaraProcessingException(
f"Can't convert value of source type '…
)
config = inputs.get_value_data("config")
if config is None:
config = {}
target_value = self.convert(value=value, confi…
# TODO: validate value?
outputs.set_value("target_value", target_value)
─────────────────────────────────────────────────────
list.contains¶
Documentation
Check whether an element is in a list.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants for no
this module.
defaults object Value defaults for no
this module.
Module config -- no config --
Python class
class_name IncludedInListCheckModule
module_name kiara_modules.core.list
full_name kiara_modules.core.list.IncludedInLi…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
item_list = inputs.get_value_data("list")
item = inputs.get_value_data("item")
outputs.set_value("is_included", item in item_…
─────────────────────────────────────────────────────
list.store¶
Documentation
-- n/a --
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults no
for this module.
value_type string The type of the yes
value to save.
options integer The options to no
use for the json
serialization.
Check
https://github.c…
for details.
file_name string The name of the no
serialized file.
Module config -- no config --
Python class
class_name StoreDictModule
module_name kiara_modules.core.list
full_name kiara_modules.core.list.StoreDictMod…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
value_id: str = inputs.get_value_data("value_i…
if not value_id:
raise KiaraProcessingException("No value i…
field_name = self.get_config_value("value_type…
if field_name == "any":
field_name = "value_item"
value_obj: Value = inputs.get_value_obj(field_…
base_path: str = inputs.get_value_data("base_p…
result = self.store_value(value=value_obj, bas…
if isinstance(result, typing.Mapping):
load_config = result
result_value = value_obj
elif isinstance(result, tuple):
load_config = result[0]
if result[1]:
result_value = result[1]
else:
result_value = value_obj
else:
raise KiaraProcessingException(
f"Invalid result type for 'store_value…
)
load_config["value_id"] = value_id
lc = LoadConfig(**load_config)
if lc.base_path_input_name and lc.base_path_in…
raise KiaraProcessingException(
f"Invalid load config: base path '{lc.…
)
outputs.set_values(
metadata=None, lineage=None, **{"load_conf…
)
─────────────────────────────────────────────────────
logic.and¶
Documentation
Returns 'True' if both inputs are 'True'.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants for no
this module.
defaults object Value defaults for no
this module.
delay number the delay in no
seconds from
processing start to
when the output is
returned.
Module config -- no config --
Python class
class_name AndModule
module_name kiara_modules.core.logic
full_name kiara_modules.core.logic.AndModule
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
time.sleep(self.config.delay) # type: ignore
outputs.set_value(
"y", inputs.get_value_data("a") and inputs…
)
─────────────────────────────────────────────────────
logic.not¶
Documentation
Negates the input.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants for no
this module.
defaults object Value defaults for no
this module.
delay number the delay in no
seconds from
processing start to
when the output is
returned.
Module config -- no config --
Python class
class_name NotModule
module_name kiara_modules.core.logic
full_name kiara_modules.core.logic.NotModule
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
"""Negates the input boolean."""
time.sleep(self.config.get("delay")) # type: …
outputs.set_value("y", not inputs.get_value_da…
─────────────────────────────────────────────────────
logic.or¶
Documentation
Returns 'True' if one of the inputs is 'True'.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants for no
this module.
defaults object Value defaults for no
this module.
delay number the delay in no
seconds from
processing start to
when the output is
returned.
Module config -- no config --
Python class
class_name OrModule
module_name kiara_modules.core.logic
full_name kiara_modules.core.logic.OrModule
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
time.sleep(self.config.get("delay")) # type: …
outputs.set_value("y", inputs.get_value_data("…
─────────────────────────────────────────────────────
string.deserialize¶
Documentation
-- n/a --
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value no
constants for
this module.
defaults object Value defaults no
for this
module.
serialization… string The yes
serialization
type that was
used to
serialize the
value.
Module config -- no config --
Python class
class_name DeserializeStringModule
module_name kiara_modules.core.string
full_name kiara_modules.core.string.Deserializ…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
serialization_type = self.get_config_value("se…
if serialization_type not in ["json"]:
raise KiaraProcessingException(
f"Can't deserialize string: serialisat…
)
serialized = inputs.get_value_data("serialized…
outputs.set_value("value_item", serialized)
─────────────────────────────────────────────────────
string.magic¶
Documentation
-- n/a --
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants for no
this module.
defaults object Value defaults for no
this module.
Module config -- no config --
Python class
class_name MagicModule
module_name kiara_modules.core.string
full_name kiara_modules.core.string.MagicModule
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
pass
─────────────────────────────────────────────────────
string.match_regex¶
Documentation
Match a string using a regular expression.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value no
constants for
this module.
defaults object Value no
defaults for
this module.
regex string The regex to yes
apply.
only_first_ma… boolean Whether to no
only return
the first
match, or all
matches.
Module config -- no config --
Python class
class_name RegexModule
module_name kiara_modules.core.string
full_name kiara_modules.core.string.RegexModule
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
text = inputs.get_value_data("text")
regex = self.get_config_value("regex")
matches = re.findall(regex, text)
if not matches:
raise KiaraProcessingException(f"No match …
if self.get_config_value("only_first_match"):
result = matches[0]
else:
result = matches
outputs.set_value("text", result)
─────────────────────────────────────────────────────
string.replace¶
Documentation
Replace a string if it matches a key in a mapping
dictionary.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value no
constants for
this module.
defaults object Value defaults no
for this
module.
replacement_m… object A map, yes
containing the
strings to be
replaced as
keys, and the
replacements
as values.
default_value string The default no
value to use
if the string
to be replaced
is not in the
replacement
map. By
default, this
just returns
the string
itself.
Module config -- no config --
Python class
class_name ReplaceStringModule
module_name kiara_modules.core.string
full_name kiara_modules.core.string.ReplaceStr…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
text = inputs.get_value_data("text")
repl_map = self.get_config_value("replacement_…
default = self.get_config_value("default_value…
if text not in repl_map.keys():
if default is None:
result = text
else:
result = default
else:
result = repl_map[text]
outputs.set_value("text", result)
─────────────────────────────────────────────────────
string.serialize¶
Documentation
-- n/a --
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value no
constants for
this module.
defaults object Value defaults no
for this
module.
value_type string The type of yes
the source
value.
serialization… string The type of yes
the converted
value.
Module config -- no config --
Python class
class_name SerializeStringModule
module_name kiara_modules.core.string
full_name kiara_modules.core.string.SerializeS…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
value_type: str = self.get_config_value("value…
value_obj = inputs.get_value_obj("value_item")
serialization_type = self.get_config_value("se…
if value_type != value_obj.type_name:
raise KiaraProcessingException(
f"Invalid type ({value_obj.type_name})…
)
if not hasattr(self, f"to_{serialization_type}…
# this can never happen, I think
raise Exception(
f"Module '{self._module_type_id}' can'…
)
func = getattr(self, f"to_{serialization_type}…
serialized = func(value_obj)
if isinstance(serialized, typing.Mapping):
serialized = DeserializeConfig(**serialize…
if not isinstance(serialized, DeserializeConfi…
raise KiaraProcessingException(
f"Invalid serialization result type: {…
)
outputs.set_values(
deserialize_config=serialized, value_info=…
)
─────────────────────────────────────────────────────
table.create¶
Documentation
Convert an Arrow table.
This module supportes two conversion targets
currently:
• bytes: a memoryview of the byte-representation
of the Table
• string: the base64-encoded byte-representation
of the Table
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value no
constants for
this module.
defaults object Value no
defaults for
this module.
source_profile string The profile yes
of the source
value.
target_type string The type of yes
the value to
convert to.
allow_none_in… boolean Whether to no
allow 'none'
source
values, if
one is
encountered
'none' is
returned.
ignore_errors boolean Whether to no
ignore
convert
errors and
omit the
failed items.
Module config -- no config --
Python class
class_name ConvertToTableModule
module_name kiara_modules.core.table
full_name kiara_modules.core.table.ConvertToTa…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
source_profile: str = self.get_config_value("s…
source_config: typing.Mapping[
str, typing.Mapping[str, typing.Any]
] = self._kiara.type_mgmt.get_type_config_for_…
source_type = source_config["type"]
target_type: str = self.get_config_value("targ…
allow_none: bool = self.get_config_value("allo…
source: Value = inputs.get_value_obj(source_pr…
if source_type != source.type_name:
raise KiaraProcessingException(
f"Invalid type ({source.type_name}) of…
)
if not source.is_set or source.is_none:
if allow_none:
outputs.set_value("value_item", None)
return
else:
raise KiaraProcessingException("No sou…
if not hasattr(self, f"from_{source_profile}"):
raise Exception(
f"Module '{self._module_type_id}' can'…
)
func = getattr(self, f"from_{source_profile}")
converted = func(source)
outputs.set_value(target_type, converted)
─────────────────────────────────────────────────────
table.cut_column¶
Documentation
Cut off one column from a table, returning an
array.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants for no
this module.
defaults object Value defaults for no
this module.
Module config -- no config --
Python class
class_name CutColumnModule
module_name kiara_modules.core.table
full_name kiara_modules.core.table.CutColumnMo…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
import pyarrow as pa
table_value = inputs.get_value_obj("table")
column_name: str = inputs.get_value_data("colu…
available = table_value.get_metadata("table")[…
if column_name not in available:
raise KiaraProcessingException(
f"Invalid column name '{column_name}'.…
)
table: pa.Table = inputs.get_value_data("table…
column = table.column(column_name)
outputs.set_value("array", column)
─────────────────────────────────────────────────────
table.export¶
Documentation
Export a table object to disk.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants for no
this module.
defaults object Value defaults for no
this module.
Module config -- no config --
Python class
class_name ExportArrowTable
module_name kiara_modules.core.table
full_name kiara_modules.core.table.ExportArrow…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
import pyarrow as pa
from pyarrow import feather
table: pa.Table = inputs.get_value_data("table…
full_path: str = inputs.get_value_data("path")
force_overwrite = inputs.get_value_data("force…
format: str = inputs.get_value_data("format")
compression = inputs.get_value_data("compressi…
if compression not in ["zstd", "lz4", "uncompr…
raise KiaraProcessingException(
f"Invalid compression format '{compres…
)
if format != "feather":
raise KiaraProcessingException(
f"Can't export table to format '{forma…
)
if os.path.exists(full_path) and not force_ove…
raise KiaraProcessingException(
f"Can't write table to file, file alre…
)
os.makedirs(os.path.dirname(full_path), exist_…
feather.write_feather(table, full_path, compre…
result = {
"module_type": "table.load",
"base_path_input_name": "base_path",
"inputs": {
"base_path": os.path.dirname(full_path…
"rel_path": os.path.basename(full_path…
"format": format,
},
"value_id": NO_VALUE_ID_MARKER,
"output_name": "table",
}
outputs.set_value("load_config", result)
─────────────────────────────────────────────────────
table.filter.with_mask¶
Documentation
Filter a table using a mask array.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants for no
this module.
defaults object Value defaults for no
this module.
Module config -- no config --
Python class
class_name CreateFilteredTableModule
module_name kiara_modules.core.table.filter
full_name kiara_modules.core.table.filter.Crea…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
import pyarrow as pa
input_table: pa.Table = inputs.get_value_data(…
filter_array: pa.Array = inputs.get_value_data…
filtered = input_table.filter(filter_array)
outputs.set_value("table", filtered)
─────────────────────────────────────────────────────
table.load¶
Documentation
Load a table object from disk.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants for no
this module.
defaults object Value defaults for no
this module.
Module config -- no config --
Python class
class_name LoadArrowTable
module_name kiara_modules.core.table
full_name kiara_modules.core.table.LoadArrowTa…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
from pyarrow import feather
base_path = inputs.get_value_data("base_path")
rel_path = inputs.get_value_data("rel_path")
format = inputs.get_value_data("format")
if format != "feather":
raise NotImplementedError()
path = os.path.join(base_path, rel_path)
table = feather.read_table(path)
outputs.set_value("table", table)
─────────────────────────────────────────────────────
table.map_column¶
Documentation
Map the items of one column of a table onto an
array, using another module.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this
module.
defaults object Value defaults no
for this
module.
module_type string The name of the yes
kiara module to
use to filter
the input data.
module_config object The config for no
the kiara
filter module.
input_name string The name of the no
input name of
the module
which will
receive the
rows from our
input table.
Can be omitted
if the
configured
module only has
a single input.
output_name string The name of the no
output name of
the module
which will
receive the
items from our
input array.
Can be omitted
if the
configured
module only has
a single
output.
Module config -- no config --
Python class
class_name MapColumnModule
module_name kiara_modules.core.table
full_name kiara_modules.core.table.MapColumnMo…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
import pyarrow as pa
table: pa.Table = inputs.get_value_data("table…
column_name = inputs.get_value_data("column_na…
if column_name not in table.column_names:
raise KiaraProcessingException(
f"Table column '{column_name}' not ava…
)
input_array: pa.Array = table.column(column_na…
init_data: typing.Dict[str, typing.Any] = {}
for input_name in self.input_schemas.keys():
if input_name in ["table", "column_name", …
continue
init_data[input_name] = inputs.get_value_o…
result_list = map_with_module(
input_array,
module_input_name=self.module_input_name,
module_obj=self.child_module,
init_data=init_data,
module_output_name=self.module_output_name,
)
outputs.set_value("array", pa.array(result_lis…
─────────────────────────────────────────────────────
table.merge¶
Documentation
Create a table from other tables and/or arrays.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults no
for this module.
input_schema object A dict yes
describing the
inputs for this
merge process.
Module config -- no config --
Python class
class_name MergeTableModule
module_name kiara_modules.core.table
full_name kiara_modules.core.table.MergeTableM…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
import pyarrow as pa
input_schema: typing.Dict[str, typing.Any] = s…
"input_schema"
)
sources = {}
for field_name in input_schema.keys():
sources[field_name] = inputs.get_value_dat…
len_dict = {}
arrays = []
column_names = []
for source_key, table_or_column in sources.ite…
if isinstance(table_or_column, pa.Table):
rows = table_or_column.num_rows
for name in table_or_column.schema.nam…
column = table_or_column.column(na…
arrays.append(column)
column_names.append(name)
elif isinstance(table_or_column, (pa.Array…
rows = len(table_or_column)
arrays.append(table_or_column)
column_names.append(source_key)
else:
raise KiaraProcessingException(
f"Can't merge table: invalid type …
)
len_dict[source_key] = rows
all_rows = None
for source_key, rows in len_dict.items():
if all_rows is None:
all_rows = rows
else:
if all_rows != rows:
all_rows = None
break
if all_rows is None:
len_str = ""
for name, rows in len_dict.items():
len_str = f" {name} ({rows})"
raise KiaraProcessingException(
f"Can't merge table, sources have diff…
)
table = pa.Table.from_arrays(arrays=arrays, na…
outputs.set_value("table", table)
─────────────────────────────────────────────────────
table.metadata¶
Documentation
Extract metadata from a table object.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults for no
this module.
value_type string The data type this yes
module will be
used for.
Module config -- no config --
Python class
class_name TableMetadataModule
module_name kiara_modules.core.table
full_name kiara_modules.core.table.TableMetada…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
input_name = self.value_type
if input_name == "any":
input_name = "value_item"
value = inputs.get_value_obj(input_name)
if self.value_type != "any" and value.type_nam…
raise KiaraProcessingException(
f"Can't extract metadata for value of …
)
# TODO: if type 'any', validate that the data …
outputs.set_value("metadata_item_schema", self…
metadata = self.extract_metadata(value)
if isinstance(metadata, BaseModel):
metadata = metadata.dict(exclude_none=True)
# TODO: validate metadata?
outputs.set_value("metadata_item", metadata)
─────────────────────────────────────────────────────
table.query.graphql¶
Documentation
Execute a graphql aggregation query against an
(Arrow) table.
References: -
https://vaex.io/docs/example_graphql.html
Examples: An example for a query could be:
┌─────────────────────────────────────────────────┐
│ { │
│ df(where: { │
│ Language: {_eq: "German"} │
│ } ) { │
│ │
│ row(limit: 10) { │
│ Label │
│ City │
│ } │
│ } │
│ } │
└─────────────────────────────────────────────────┘
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants for no
this module.
defaults object Value defaults for no
this module.
Module config -- no config --
Python class
class_name QueryTableGraphQL
module_name kiara_modules.core.table.query
full_name kiara_modules.core.table.query.Query…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
import vaex
table = inputs.get_value_data("table")
query = inputs.get_value_data("query")
df = vaex.from_arrow_table(table)
result = df.graphql.execute(query)
outputs.set_value("query_result", result.to_di…
─────────────────────────────────────────────────────
table.query.sql¶
Documentation
Execute a sql query against an (Arrow) table.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this
module.
defaults object Value defaults no
for this
module.
query string The query to no
execute. If not
specified, the
user will be
able to provide
their own.
relation_name string The name the no
table is
referred to in
the sql query.
If not
specified, the
user will be
able to provide
their own.
Module config -- no config --
Python class
class_name QueryTableSQL
module_name kiara_modules.core.table.query
full_name kiara_modules.core.table.query.Query…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
import duckdb
if self.get_config_value("query") is None:
_query: str = inputs.get_value_data("query…
_relation_name: str = inputs.get_value_dat…
else:
_query = self.get_config_value("query")
_relation_name = self.get_config_value("re…
if _relation_name.upper() in RESERVED_SQL_KEYW…
raise KiaraProcessingException(
f"Invalid relation name '{_relation_na…
)
_table = inputs.get_value_data("table")
relation: duckdb.DuckDBPyRelation = duckdb.arr…
result: duckdb.DuckDBPyResult = relation.query…
print(result.arrow())
outputs.set_value("query_result", result.arrow…
─────────────────────────────────────────────────────
table.sample¶
Documentation
Sample a table.
Samples are used to randomly select a subset of a
dataset, which helps test queries and workflows on
smaller versions of the original data, to adjust
parameters before a full run.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults no
for this module.
sample_type string The sample yes
method.
Module config -- no config --
Python class
class_name SampleTableModule
module_name kiara_modules.core.table
full_name kiara_modules.core.table.SampleTable…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
sample_size: int = inputs.get_value_data("samp…
sample_type: str = self.get_config_value("samp…
if sample_size < 0:
raise KiaraProcessingException(
f"Invalid sample size '{sample_size}':…
)
input_name = self.get_value_type()
if input_name == "any":
input_name = "value_item"
value: Value = inputs.get_value_obj(input_name)
func = getattr(self, f"sample_{sample_type}")
result = func(value=value, sample_size=sample_…
outputs.set_value("sampled_value", result)
─────────────────────────────────────────────────────
table.store¶
Documentation
-- n/a --
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults no
for this module.
value_type string The type of the yes
value to save.
compression string The compression no
to use when
saving the table.
Module config -- no config --
Python class
class_name StoreArrowTable
module_name kiara_modules.core.table
full_name kiara_modules.core.table.StoreArrowT…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
value_id: str = inputs.get_value_data("value_i…
if not value_id:
raise KiaraProcessingException("No value i…
field_name = self.get_config_value("value_type…
if field_name == "any":
field_name = "value_item"
value_obj: Value = inputs.get_value_obj(field_…
base_path: str = inputs.get_value_data("base_p…
result = self.store_value(value=value_obj, bas…
if isinstance(result, typing.Mapping):
load_config = result
result_value = value_obj
elif isinstance(result, tuple):
load_config = result[0]
if result[1]:
result_value = result[1]
else:
result_value = value_obj
else:
raise KiaraProcessingException(
f"Invalid result type for 'store_value…
)
load_config["value_id"] = value_id
lc = LoadConfig(**load_config)
if lc.base_path_input_name and lc.base_path_in…
raise KiaraProcessingException(
f"Invalid load config: base path '{lc.…
)
outputs.set_values(
metadata=None, lineage=None, **{"load_conf…
)
─────────────────────────────────────────────────────
value.data_profile¶
Documentation
Generate a data profile report for a dataset.
This uses the DataProfiler Python library, check
out its documentation for more details.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults for no
this module.
value_type string The value type to yes
profile.
Module config -- no config --
Python class
class_name DataProfilerModule
module_name kiara_modules.core.value
full_name kiara_modules.core.value.DataProfile…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
import pyarrow as pa
from dataprofiler import Data, Profiler, Profi…
set_verbosity(logging.WARNING)
value_type = self.get_config_value("value_type…
profile_options = ProfilerOptions()
profile_options.structured_options.data_labele…
profile_options.unstructured_options.data_labe…
if value_type == "table":
table_item: pa.Table = inputs.get_value_da…
pd = table_item.to_pandas()
profile = Profiler(
pd, options=profile_options
) # Calculate Statistics, Entity Recognit…
report = profile.report()
elif value_type == "file":
file_item: KiaraFile = inputs.get_value_da…
data = Data(file_item.path)
profile = Profiler(data, options=profile_o…
report = profile.report()
else:
raise KiaraProcessingException(
f"Data profiling of value type '{value…
)
outputs.set_value("report", report)
─────────────────────────────────────────────────────
yaml.to_yaml¶
Documentation
Convert arbitrary types into YAML format.
Early days for this module, it doesn't support a
whole lot of types yet.
Origin
Authors Markus Binsteiner (markus@frkl.io)
Context
Tags core
Labels package: kiara_modules.core
References source_repo:
https://github.com/DHARPA-Project/kia…
documentation:
https://dharpa.org/kiara_modules.core/
module_doc:
https://dharpa.org/kiara_modules.core…
source_url:
https://github.com/DHARPA-Project/kia…
Module config
Field Type Description Required
─────────────────────────────────────────────────────
constants object Value constants no
for this module.
defaults object Value defaults no
for this module.
source_type string The source type. yes
target_type string The target type. yes
Module config -- no config --
Python class
class_name ToYamlModuleOld
module_name kiara_modules.core.yaml
full_name kiara_modules.core.yaml.ToYamlModule…
Processing source code ─────────────────────────────────────────────────────
def process(self, inputs: ValueSet, outputs: Value…
value = inputs.get_value_obj("source_value")
if value.value_schema.type != self.source_type:
raise KiaraProcessingException(
f"Can't convert value of source type '…
)
config = inputs.get_value_data("config")
if config is None:
config = {}
target_value = self.convert(value=value, confi…
# TODO: validate value?
outputs.set_value("target_value", target_value)
─────────────────────────────────────────────────────