improv.store

Module Contents

Classes

StoreInterface	General interface for a store
PlasmaStoreInterface	Basic interface for our specific data store implemented with apache arrow plasma
LMDBStoreInterface	General interface for a store
LMDBData	Dataclass to store objects and their metadata into LMDB.

Attributes

logger
StoreInterface	General interface for a store

improv.store.logger

class improv.store.StoreInterface

General interface for a store

abstract get()

abstract put()

abstract delete()

abstract replace()

abstract subscribe()

class improv.store.PlasmaStoreInterface(name='default', store_loc='/tmp/store')

Bases: StoreInterface

Basic interface for our specific data store implemented with apache arrow plasma Objects are stored with object_ids References to objects are contained in a dict where key is shortname, value is object_id

connect_store(store_loc)

Connect to the store at store_loc, max 20 retries to connect Raises exception if can’t connect Returns the plasmaclient if successful Updates the client internal

Parameters:

store_loc – store location

put(object, object_name)

Put a single object referenced by its string name into the store Raises PlasmaObjectExists if we are overwriting Unknown error

Parameters:

• object –

• object_name (str) –

• flush_this_immediately (bool) –

Returns:

Plasma object ID

Return type:

class ‘plasma.ObjectID’

Raises:

PlasmaObjectExists – if we are overwriting unknown error

get(object_name)

Get a single object from the store by object name Checks to see if it knows the object first Otherwise throw CannotGetObject to request dict update TODO: update for lists of objects TODO: replace with getID

Returns:

Stored object

getID(obj_id, hdd_only=False)

Get object by object ID

Parameters:

• obj_id (class 'plasma.ObjectID') – the id of the object

• hdd_only (bool) –

Returns:

Stored object

Raises:

ObjectNotFoundError – If the id is not found

getList(ids)

Get multiple objects from the store

Parameters:

ids (list) – of type plasma.ObjectID

Returns:

list of the objects

get_all()

Get a listing of all objects in the store

Returns:

list of all the objects in the store

reset()

Reset client connection

release()

subscribe()

Subscribe to a section? of the ds for singals

Raises:

Exception – Unknown error

notify()

random_ObjectID(number=1)

updateStoreInterfaced(object_name, object_id)

Update local dict with info we need locally Report to Nexus that we updated the store (did a put or delete/replace)

Parameters:

• object_name (str) – the name of the object to update

• () (object_id) – the id of the object to update

getStored()

Returns:

its info about what it has stored

class improv.store.LMDBStoreInterface(path='../outputs/', name=None, load=False, max_size=1000000000000.0, flush_immediately=False, commit_freq=1)

Bases: StoreInterface

General interface for a store

get(key: pyarrow.plasma.ObjectID | bytes | List[pyarrow.plasma.ObjectID] | List[bytes], include_metadata=False)

Get object using key (could be any byte string or plasma.ObjectID)

Parameters:

• key –

• include_metadata (bool) – returns whole LMDBData if true else LMDBData.obj (just the stored object).

Returns:

object or LMDBData

get_keys()

Get all keys in LMDB

put(obj, obj_name, obj_id=None, flush_this_immediately=False)

Put object ID / object pair into LMDB.

Parameters:

• obj – Object to be saved

• obj_name (str) – the name of the object

• obj_id ('plasma.ObjectID') – Object_id from Plasma client

• flush_this_immediately (bool) – Override self.flush_immediately. For storage of critical objects.

Returns:

None

flush(sig=None, frame=None)

Must run before exiting. Flushes buffer to disk.

commit_daemon()

commit()

Commit objects in {self.put_cache} into LMDB.

delete(obj_id)

Delete object from LMDB.

Parameters:

obj_id (class 'plasma.ObjectID') – the object_id to be deleted

Returns:

None

Raises:

ObjectNotFoundError – If the id is not found

replace()

subscribe()

improv.store.StoreInterface

class improv.store.LMDBData

Dataclass to store objects and their metadata into LMDB.

property queue

Returns: Queue name if object is a queue else None

obj: object

time: float

name: str

is_queue: bool = False

exception improv.store.ObjectNotFoundError(obj_id_or_name)

Bases: Exception

Common base class for all non-exit exceptions.

exception improv.store.CannotGetObjectError(query)

Bases: Exception

Common base class for all non-exit exceptions.

exception improv.store.CannotConnectToStoreInterfaceError(store_loc)

Bases: Exception

Raised when failing to connect to store.