Serialization

The serialization subpackage groups all serialization components and serializer implementations.

Base classes are defined which can allow for other serialization implementations beside the built-in specification serializers.

class MsgPackSerializer ( bool timestamps_as_str=False , bool timestamps_as_iso8601=False )

Bases: Serializer

Provides a serializer for the MessagePack specification.

Parameters :

• timestamps_as_str ( bool , default False ) – If the serializer converts uint64_t timestamps to integer strings on serialization, and back to uint64_t on deserialization.

• timestamps_as_iso8601 ( bool , default False ) – If the serializer converts uint64_t timestamps to ISO 8601 strings on serialization, and back to uint64_t on deserialization.

deserialize ( self , bytes obj_bytes )

Deserialize the given MessagePack specification bytes to an object.

Parameters :

obj_bytes ( bytes ) – The object bytes to deserialize.

Returns :

Instrument

Raises :

RuntimeError – If obj_bytes cannot be deserialized.

serialize ( self , obj ) → bytes

Serialize the given object to MessagePack specification bytes.

Parameters :

obj ( object ) – The object to serialize.

Returns :

bytes

Raises :

RuntimeError – If obj cannot be serialized.

timestamps_as_iso8601

If the serializer converts timestamp int64_t to ISO 8601 strings.

Returns :

bool

timestamps_as_str

If the serializer converts timestamp int64_t to integer strings.

Returns :

bool

class ParquetSerializer

Bases: object

Provides an object serializer for the Parquet specification.

static deserialize ( type cls , chunk )

Deserialize the given Parquet specification bytes to an object.

Parameters :

• cls ( type ) – The type to deserialize to.

• chunk ( bytes ) – The chunk to deserialize.

Returns :

object

Raises :

TypeError – If chunk cannot be deserialized.

static serialize ( obj )

Serialize the given instrument to Parquet specification bytes.

Parameters :

obj ( object ) – The object to serialize.

Returns :

bytes

Raises :

TypeError – If obj cannot be serialized.

get_cls_table ( type cls: type )

get_partition_keys ( type cls: type )

get_schema ( type cls: type )

list_schemas ( )

register_parquet ( type cls , serializer: Optional[Callable] = None , deserializer: Optional[Callable] = None , schema: Optional[pa.Schema] = None , bool chunk=False , type table=None , **kwargs )

Register a new class for serialization to parquet.

Parameters :

• cls ( type ) – The type to register serialization for.

• serializer ( Callable , optional ) – The callable to serialize instances of type cls_type to something parquet can write.

• deserializer ( Callable , optional ) – The callable to deserialize rows from parquet into cls_type .

• schema ( pa.Schema , optional ) – If the schema cannot be correctly inferred from a subset of the data (i.e. if certain values may be missing in the first chunk).

• chunk ( bool , optional ) – Whether to group objects by timestamp and operate together (Used for complex objects where we write each object as multiple rows in parquet, i.e. OrderBook or AccountState ).

• table ( type , optional ) – An optional table override for cls . Used if cls is going to be transformed and stored in a table other than its own.

class Serializer

Bases: object

The base class for all serializers.

Warning

This class should not be used directly, but through a concrete subclass.

deserialize ( self , bytes obj_bytes )

Abstract method (implement in subclass).

serialize ( self , obj ) → bytes

Abstract method (implement in subclass).

register_serializable_object ( obj , to_dict : Callable [ [ Any ] , dict [ str , Any ] ] , from_dict : Callable [ [ dict [ str , Any ] ] , Any ] ) → void

Register the given object with the global serialization object maps.

Parameters :

• obj ( object ) – The object to register.

• to_dict ( Callable [ [ Any ] , dict [ str , Any ] ] ) – The delegate to instantiate a dict of primitive types from the object.

• from_dict ( Callable [ [ dict [ str , Any ] ] , Any ] ) – The delegate to instantiate the object from a dict of primitive types.

Raises :

• TypeError – If to_dict or from_dict are not of type Callable .

• KeyError – If obj already registered with the global object maps.