`artemis.core.gate`¶

Framework-level services and helper mixin classes to provide access to metadata, histograms, timers, and stores

Module Contents¶

class artemis.core.gate.ArtemisGateSvc¶

Wrapper class as Singleton type Framework level service Providing access to common data sinks required in artemis and algorithms

meta¶

JobInfo object holding uuids to configuration and menu metadata

Type: JobInfo

hbook¶

OrderedDict of all histograms in framework

Type: ArtemisBook

tbook¶

OrderedDict of all tdigests in framework

Type: TDigestBook

menu¶

Business process graph

Type: Menu

config¶

All configuration meta, including properties of tools and algorithms

Type: Configuration

tools¶

OrderedDict of all tool objects in framework.

Type: ToolStore

store¶

Metadata service and access to underlying data store

Type: BaseObjectStore

tree¶

Execution graph

Type: Tree

configure(self, jobinfo)¶: Configure the gate with jobinfo passed to Artemis.

_finalize_jobstate(self)¶

_finalize_timers(self)¶

_job_report(self)¶

finalize(self)¶: Finalize job. Collects timers and registers remaining content, such as histograms to the store.

class artemis.core.gate.MetaMixin¶

Methods for setting / getting job attributes Meta data required in ArtemisGate to track processing

property job_id(self)¶: UUID of job

property path(self)¶: Absolute path to datastore, e.g. where things are written to

property store_name(self)¶: Name of metastore

property store_uuid(self)¶: UUID of store

property menu_id(self)¶: UUID of menu metadata

property config_id(self)¶: UUID of configuration metadata

property input_id(self)¶: Parent dataset uuid.

property output_id(self)¶: Output dataset uuid

get_tool(self, name)¶: Retrieve a tool via the name

class artemis.core.gate.IOMetaMixin¶

property job_state(self)¶: Job state currently stored in protobuf

property current_file(self)¶: current file UUID held in gate

property processed_ndatums(self)¶: current number of datums processed

property processed_bytes(self)¶: current total bytes processed

register_log(self)¶

register a log file in the store

Returns: logobj
Return type: Context object

register_content(self, buf, info, dataset_id, job_id, partition_key=None)¶

register content in the store

Parameters

buf (protobuf or pyarrow.buffer) – raw bytes
info (Context object) – metadata associated to data object
dataset_id (UUID) – UUID of dataset to associate with content
job_id (UUID) – UUID of current job that produced data object
partition_key (str) – partition name in dataset

set_file_size_bytes(self, filepath_or_buffer, size_)¶: set the size in bytes of an object in the context object

set_file_blocks(self, filepath_or_buffer, blocks)¶

set blocks, chunks or record batches in a file. provides context of how a raw data object is split, read and processed.

Parameters

filepath_or_buffer (bytes) – data object
blocks (List) – offset is zeroth element length is first element

new_partition(self, key)¶

add a partition to the current dataset

Parameters: key (str) – name of partition. Typically a leaf name in the menu

reset_job_summary(self)¶: clears the summary information. Typically used when sampling data.

get_leaves(self)¶: Return leaves from the execution tree

get_node(self, node_id)¶

Return node from execution tree

Parameters: node_id (str) – unique name of node

persist_to_storage(self, obj_id, buf)¶: Writes data object to disk via the store

datastore_flush(self)¶: Clears all nodes in the execution tree

datastore_is_empty(self)¶

checks that the in-memory datastore is empty

Returns
Return type: bool

artemis.core.gate¶

Module Contents¶

`artemis.core.gate`¶