quantpylib.hft.qet

Quantpylib Event Trace (QET) is the append-only binary container for accepted portfolio-state events. The module writes and reads the trace bytes emitted by the production Orders, Fills, and Positions ledgers.

This page defines the file and Python API contracts. For the narrative workflow from portfolio mutation through the performance logger, Grafana Alloy, the QET compiler, and downstream inspection, see the HFT Event Journal tutorial.

Container layout

A QET file contains one header followed by concatenated binary trace records:

Segment	Encoding	Purpose
Magic	b"QET1"	Identifies the QET container version.
Schema count	little-endian uint32	Number of schema-registry entries.
Schema entry	uint32 schema_id, uint32 length, UTF-8 domain	Makes the file's record domains self-describing.
Trace records	concatenated self-length-prefixed byte strings	Preserves the original portfolio traces.

Each trace begins with its own record length and schema ID. Top-level records also carry a session ID (sid) and monotonically assigned session sequence number (seqno). Their identity is therefore:

(sid, seqno)

QETWriter validates the schema and appends the original trace bytes. It does not reorder or deduplicate records; those are compiler responsibilities. QETReader returns records in physical file order and validates the container, record lengths, trace identity, and nested snapshot boundaries while decoding.

Trace schemas

Trace	Role
Order	One accepted order insertion or state transition.
Fill	One accepted fill insertion.
Position delta	One accepted position mutation caused by a fill.
Orders snapshot	Recovery snapshot containing nested order traces.
Positions snapshot	Recovery snapshot containing nested position traces.
Position	Nested position representation used inside a positions snapshot.

The current schema registry is exposed as QET_SCHEMA_REGISTRY. QET_TOP_LEVEL_SCHEMA_IDS identifies records that may be appended directly; nested position records are decoded only as children of a snapshot.

Reading and writing

from quantpylib.hft.qet import QETReader, QETWriter


with QETWriter("portfolio.qet") as writer:
    writer.write(trace)
    writer.flush(durable=True)

reader = QETReader("portfolio.qet")
for record in reader.records():
    print(record.sid,record.seqno,record.schema_id,record.offset)

for event in reader.decoded_records():
    print(event)

The specialized iterators decoded_orders(), decoded_fills(), decoded_position_deltas(), decoded_orders_snapshots(), and decoded_positions_snapshots() filter to one trace family. The generic decoded_records() iterator decodes every supported top-level family.

Malformed magic, truncated headers or records, invalid lengths, unsupported top-level schemas, and trailing snapshot bytes raise QETFormatError.

API reference

Read and write Quantpylib Event Trace (QET) binary journals.

QET stores the original binary traces emitted by the portfolio ledgers. A file contains a self-describing schema registry followed by append-only order, fill, position-delta, and recovery-snapshot records. Top-level records are identified by trading session ID and session sequence number.

The module owns container validation and trace decoding. Ordering, deduplication, log transport, and daily-file partitioning belong to the QET compiler described in the HFT Event Journal tutorial.

QETFormatError

Bases: ValueError

QET container or trace bytes violate the binary format contract.

QETReader

Validate and stream records from one QET file.

Each iterator opens the file independently, validates its header, and reads records in physical file order. Decoded iterators add the record's absolute offset to the normalized trace dictionary.

Parameters:

Name	Type	Description	Default
path	path - like	QET file to read.	required

__init__(path)

Store the source path without opening it.

decoded_fills()

Stream decoded fill insertions.

Yields:

Type	Description
dict	Normalized fill trace with its QET file offset.

decoded_orders()

Stream decoded order insertions and state transitions.

Yields:

Type	Description
dict	Normalized order trace with its QET file offset.

decoded_orders_snapshots()

Stream decoded order-ledger recovery snapshots.

Yields:

Type	Description
dict	Snapshot and nested orders with its QET file offset.

decoded_position_deltas()

Stream decoded position mutations.

Yields:

Type	Description
dict	Normalized position delta with its QET file offset.

decoded_positions_snapshots()

Stream decoded position-ledger recovery snapshots.

Yields:

Type	Description
dict	Snapshot and nested positions with its QET file offset.

decoded_records()

Stream every supported decoded top-level trace family.

Yields:

Type	Description
dict	Normalized trace with its QET file offset.

records()

Stream validated top-level records in physical file order.

Yields:

Type	Description
QETRecord	Trace bytes, file offset, schema, and session identity.

Raises:

Type	Description
FileNotFoundError	If the configured path does not exist.
QETFormatError	If the container header, record envelope, length, schema, or session identity is invalid.

QETRecord dataclass

One top-level trace read from a QET file.

Parameters:

Name	Type	Description	Default
schema_id	int	Numeric trace schema identifier.	required
offset	int	Absolute byte offset at which the trace begins.	required
trace	bytes	Exact self-length-prefixed trace bytes.	required
sid	int	Trading-session identifier encoded in the trace.	required
seqno	int	Monotonic sequence number within sid.	required

QETWriter

Append validated top-level portfolio traces to a QET file.

A new or empty file receives the QET magic and schema registry before the first trace. A non-empty file is opened for append without rewriting or validating its existing header. Use flush() with durable=True when the caller requires an operating-system durability barrier.

Parameters:

Name	Type	Description	Default
path	path - like	Destination QET file. Missing parent directories are created.	required
schema_registry	mapping	Accepted schema IDs mapped to record-domain names. Defaults to QET_SCHEMA_REGISTRY.	None

__enter__()

Return this writer for context-managed use.

Returns:

Type	Description
QETWriter	Open writer instance.

__exit__(exc_type, exc, tb)

Close the writer when leaving a context-manager block.

__init__(path, schema_registry=None)

Open the destination for append and initialize an empty file.

close()

Close the underlying append file.

flush(durable=False)

Flush buffered writes and optionally synchronize them to storage.

Parameters:

Name	Type	Description	Default
durable	bool	Call fsync after flushing the Python file buffer.	False

tell()

Return the current absolute append offset.

Returns:

Type	Description
int	Current file position in bytes.

write(trace)

Append one validated top-level trace without re-encoding it.

Parameters:

Name	Type	Description	Default
trace	bytes - like	Complete portfolio trace to append.	required

Returns:

Type	Description
int	Absolute byte offset at which the trace was written.

Raises:

Type	Description
QETFormatError	If the trace envelope is invalid, its schema is nested-only, or its schema is absent from this writer's registry.

decode_fill_trace(trace)

Decode one accepted fill trace.

Parameters:

Name	Type	Description	Default
trace	bytes - like	Complete fill trace bytes.	required

Returns:

Type	Description
dict	Schema and session identity, field bitmap, exchange, trade and order identity, ticker, amount, price, fees, realized PnL, position start and end amounts, maker flag, and exchange/update timestamps. Bitmap-optional fields are None when absent.

Raises:

Type	Description
QETFormatError	If the fixed header, schema ID, record length, or variable-length payload boundary is invalid.

decode_order_trace(trace)

Decode one order insertion or state-transition trace.

Bitmap-optional fields are returned as None when they were not present in the encoded mutation. Decimal-valued fields are reconstructed from their lossless ASCII representation.

Parameters:

Name	Type	Description	Default
trace	bytes - like	Complete order trace bytes.	required

Returns:

Type	Description
dict	Schema and session identity, field bitmap, exchange and order identity, order attributes, status, filled quantities, and submit, cancel, exchange, and update timestamps.

Raises:

Type	Description
QETFormatError	If the fixed header, schema ID, record length, or variable-length payload boundary is invalid.

decode_orders_snapshot_trace(trace)

Decode an order-ledger recovery snapshot.

Parameters:

Name	Type	Description	Default
trace	bytes - like	Complete top-level orders snapshot trace.	required

Returns:

Type	Description
dict	Schema and session identity, snapshot update timestamp, exchange, and decoded nested orders list.

Raises:

Type	Description
QETFormatError	If the snapshot header, schema, record length, exchange field, nested order boundaries, or final payload boundary is invalid.

decode_position_delta_trace(trace)

Decode one accepted position-delta trace.

Parameters:

Name	Type	Description	Default
trace	bytes - like	Complete position-delta trace bytes.	required

Returns:

Type	Description
dict	Schema and session identity, field bitmap, exchange, ticker, resulting amount, applied delta, entry values, and exchange/update timestamps.

Raises:

Type	Description
QETFormatError	If the fixed header, schema ID, record length, or variable-length payload boundary is invalid.

decode_position_trace(trace)

Decode one nested position snapshot entry.

Position traces do not carry their own session identity because they are children of a top-level positions snapshot.

Parameters:

Name	Type	Description	Default
trace	bytes - like	Complete nested position trace bytes.	required

Returns:

Type	Description
dict	Schema ID, exchange, ticker, amount, and entry price.

Raises:

Type	Description
QETFormatError	If the fixed header, schema ID, record length, or variable-length payload boundary is invalid.

decode_positions_snapshot_trace(trace)

Decode a position-ledger recovery snapshot.

Parameters:

Name	Type	Description	Default
trace	bytes - like	Complete top-level positions snapshot trace.	required

Returns:

Type	Description
dict	Schema and session identity, snapshot update timestamp, exchange, and decoded nested positions list.

Raises:

Type	Description
QETFormatError	If the snapshot header, schema, record length, exchange field, nested position boundaries, or final payload boundary is invalid.

trace_identity(trace)

Return the session identity of a top-level QET trace.

Parameters:

Name	Type	Description	Default
trace	bytes - like	Complete self-length-prefixed QET trace.	required

Returns:

Type	Description
tuple[int, int]	(sid, seqno) encoded in the trace header.

Raises:

Type	Description
QETFormatError	If the envelope is invalid, the schema is not a top-level QET schema, or the identity header is truncated.

trace_schema_id(trace)

Validate a trace envelope and return its schema identifier.

Parameters:

Name	Type	Description	Default
trace	bytes - like	Self-length-prefixed QET trace bytes.	required

Returns:

Type	Description
int	Numeric schema identifier from the trace header.

Raises:

Type	Description
QETFormatError	If the trace is shorter than its header or its declared record length differs from the supplied byte length.