Serialization

The serialization subpackage groups all serialization components and serializer implementations.

Base classes are defined which can allow for other serialization implementations beside the built-in specification serializers.

class MsgPackSerializer ( bool timestamps_as_str=False )

Bases: Serializer

Provides a serializer for the MessagePack specification.

Parameters :

timestamps_as_str ( bool ) – If the serializer converts int64_t timestamps to str on serialization, and back to int64_t on deserialization.

deserialize ( self , bytes obj_bytes )

Deserialize the given MessagePack specification bytes to an object.

Parameters :

obj_bytes ( bytes ) – The object bytes to deserialize.

Returns :

Instrument

Raises :

RuntimeError – If obj_bytes cannot be deserialized.

serialize ( self , obj ) → bytes

Serialize the given object to MessagePack specification bytes.

Parameters :

obj ( object ) – The object to serialize.

Returns :

bytes

Raises :

RuntimeError – If obj cannot be serialized.

timestamps_as_str

If the serializer converts timestamp int64_t to str.

Returns :

bool

class ParquetSerializer

Bases: object

Provides an object serializer for the Parquet specification.

static deserialize ( type cls , chunk )

Deserialize the given Parquet specification bytes to an object.

Parameters :

• cls ( type ) – The type to deserialize to.

• chunk ( bytes ) – The chunk to deserialize.

Returns :

object

Raises :

TypeError – If chunk cannot be deserialized.

static serialize ( obj )

Serialize the given instrument to Parquet specification bytes.

Parameters :

obj ( object ) – The object to serialize.

Returns :

bytes

Raises :

TypeError – If obj cannot be serialized.

get_cls_table ( type cls: type )

get_partition_keys ( type cls: type )

get_schema ( type cls: type )

list_schemas ( )

register_parquet ( type cls , serializer: Optional[Callable] = None , deserializer: Optional[Callable] = None , schema: Optional[pa.Schema] = None , bool chunk=False , type table=None , **kwargs )

Register a new class for serialization to parquet.

Parameters :

• cls ( type ) – The type to register serialization for.

• serializer ( Callable , optional ) – The callable to serialize instances of type cls_type to something parquet can write.

• deserializer ( Callable , optional ) – The callable to deserialize rows from parquet into cls_type .

• schema ( pa.Schema , optional ) – If the schema cannot be correctly inferred from a subset of the data (i.e. if certain values may be missing in the first chunk).

• chunk ( bool , optional ) – Whether to group objects by timestamp and operate together (Used for complex objects where we write each object as multiple rows in parquet, i.e. OrderBook or AccountState ).

• table ( type , optional ) – An optional table override for cls . Used if cls is going to be transformed and stored in a table other than its own. (for example, OrderBookSnapshots are stored as OrderBookDeltas , so we use table=OrderBookDeltas ).

class Serializer

Bases: object

The base class for all serializers.

Warning

This class should not be used directly, but through a concrete subclass.

deserialize ( self , bytes obj_bytes )

Abstract method (implement in subclass).

serialize ( self , obj ) → bytes

Abstract method (implement in subclass).

register_serializable_object ( obj , to_dict : Callable [ [ Any ] , dict [ str , Any ] ] , from_dict : Callable [ [ dict [ str , Any ] ] , Any ] ) → void

Register the given object with the global serialization object maps.

Parameters :

• obj ( object ) – The object to register.

• to_dict ( Callable [ [ Any ] , dict [ str , Any ] ] ) – The delegate to instantiate a dict of primitive types from the object.

• from_dict ( Callable [ [ dict [ str , Any ] ] , Any ] ) – The delegate to instantiate the object from a dict of primitive types.

Raises :

• TypeError – If to_dict or from_dict are not of type Callable .

• KeyError – If obj already registered with the global object maps.