Skip to content

L0 Data Capture / Quantpylib Binary eNcoding (QBN)

Quantpylib Binary eNcoding (QBN) is quantpylib's binary capture format for native market-data events. L0 data capture is the durable, lowest-level event log beneath higher-level functions such as market data replay and analytics. The capture path records the normalized event objects emitted by wrappers and gateway subscriptions. These artefacts can later be used to inspect trading system market data latency, backtesting, or combined with event traces to get markouts, et cetera.

The supported L0 event schemas are:

Use quantpylib.hft.feed.Feed to subscribe to market data and pass quantpylib.hft.l0.L0ArchiveSink into the feed as a global sink or per-feed sink. Feed owns event fanout; L0ArchiveSink owns queued durable capture; Writer and Reader classes are responsible for encoding and decoding QBN files.

Capture Path

wrapper websocket/http message
        |
        v
gateway executor subscription
        |
        v
schema-2 native event
BookUpdate / BBAUpdate / TradeUpdate
        |
        v
Feed fanout
        |
        +--> on_msg callback / materializer sinks
        |
        +--> L0ArchiveSink
                 |
                 v
              writer queue
                 |
                 v
              L0DirectArchiveSink
                 |
                 v
              L0Writer -> QBN file

Typical usage captures every event from a feed while keeping file IO off the feed callback path:

from quantpylib.hft.feed import Feed
from quantpylib.hft.l0 import L0ArchiveSink

sink = L0ArchiveSink(root="./archives")
feed = Feed(gateway=gateway, sinks=[sink])

await feed.add_l2_book_feed(
    exc="binance",
    ticker="BTCUSDT",
    depth=20,
)

Per-feed sinks can also be used when only selected subscriptions should be captured:

book_feed = await feed.add_l2_book_feed(
    exc="binance",
    ticker="BTCUSDT",
    sinks=[L0ArchiveSink(root="./archives")],
)

That's all. A .qbn file will be written. To understand the file layout and custom policies, read on.

Why QBN

  • Native event fidelity: QBN records quantpylib.standards.models.BookUpdate, quantpylib.standards.models.BBAUpdate, and quantpylib.standards.models.TradeUpdate events as they happened with lossless capture for critical fields used for event reconstruction.
  • Append-friendly writes: writers append sealed blocks and can reopen existing files.
  • Partial-tail recovery: readers ignore incomplete trailing blocks; writers can truncate an incomplete tail before appending.
  • Block-level validation: each block carries a CRC over the block header, instrument dictionary, and record payload.
  • Self-describing: the file header stores the schema id, record-domain string, venue, market class, and feed type with the captured data.
  • Compact symbology: records store a local_id; the per-block dictionary stores local_id -> symbol once for that block.
  • Index-friendly time-range scanning: block headers store exchange-time and receive-time bounds, so readers can skip out-of-range blocks before decoding payloads. Applications can design custom indexing based on file format and application semantics.

File Layout

A QBN file starts with one file header followed by zero or more sealed blocks. All fixed-width numeric fields are little-endian.

QBN file
|
+-- FileHeader
|
+-- Block 0
|   |
|   +-- BlockHeader
|   +-- InstrumentDictionary
|   +-- RecordPayload
|       |
|       +-- Record 0
|       +-- Record 1
|       +-- ...
|
+-- Block 1
|   |
|   +-- BlockHeader
|   +-- InstrumentDictionary
|   +-- RecordPayload
|
+-- ...

File Header

Field Type Description
FILE_MAGIC char[4] File magic. Current value: QBN1.
schema_id uint32 Event schema id.
record_domain_length uint32 Byte length of record_domain.
record_domain bytes UTF-8 record-domain string describing the record layout.
venue_length, venue uint16, bytes Venue name.
market_class_length, market_class uint16, bytes Market class.
feed_type_length, feed_type uint16, bytes Feed type.

Current schema ids:

Schema id Event object Reader Writer
20 BookUpdate L0BookUpdateReader L0BookUpdateWriter
21 BBAUpdate L0BBAUpdateReader L0BBAUpdateWriter
22 TradeUpdate L0TradeUpdateReader L0TradeUpdateWriter

Block Layout

Blocks are the unit of validation and partial recovery.

Block
|
+-- BlockHeader
|   magic = QBLK1
|   block_length
|   record_count
|   min_ts_exch_ns / max_ts_exch_ns
|   min_ts_recv_ns / max_ts_recv_ns
|   dictionary_length
|   payload_length
|   crc
|
+-- InstrumentDictionary
|   symbol_count
|   repeated local_id, symbol_length, symbol
|
+-- RecordPayload
    repeated schema-specific QBN records
Field Type Description
BLOCK_MAGIC char[5] Block magic. Current value: QBLK1.
block_length uint32 Total block bytes, including header, dictionary, and payload.
record_count uint32 Number of records in the payload.
min_ts_exch_ns uint64 Minimum exchange-event timestamp in the block.
max_ts_exch_ns uint64 Maximum exchange-event timestamp in the block.
min_ts_recv_ns uint64 Minimum local receive timestamp in the block.
max_ts_recv_ns uint64 Maximum local receive timestamp in the block.
dictionary_length uint32 Instrument dictionary byte length.
payload_length uint32 Record payload byte length.
crc uint32 CRC32 over the zero-CRC header plus dictionary and payload.

Blocks seal when either max_payload_bytes would be exceeded or max_block_seconds has elapsed since the first record in the pending block. Defaults are 32 MiB and 60 seconds.

Instrument Dictionary

Each block has its own compact symbol dictionary.

Field Type Description
symbol_count uint32 Number of symbols in the block.
local_id uint32 Block-local symbol id referenced by records.
symbol_length uint16 Symbol byte length.
symbol bytes UTF-8 symbol.

Runtime event bytes are self-contained and include the ticker string. QBN record bytes are archive-local and replace the ticker string with local_id. The dictionary stores the symbol once per block.

Record Layouts

The payload is a sequence of records for the file schema. A file contains one schema id and one record domain.

BookUpdate

BookUpdate records use schema id 20.

Field Type Description
record_length uint32 Total record bytes.
local_id uint32 Instrument dictionary id.
data_length uint32 Book-level buffer byte length.
flags uint32 Snapshot, timestamp, and ordering flags.
nbids uint32 Number of bid levels.
nasks uint32 Number of ask levels.
seq_type uint32 Sequence mode.
reserved0 uint32 Reserved, must be zero.
ts_exch_ns uint64 Exchange-event timestamp.
ts_disp_ns uint64 Provider dispatch timestamp.
ts_recv_ns uint64 Local receive timestamp.
seq0, seq1, seq2 int64 Sequence fields.
data int64[] Bid price/size pairs followed by ask price/size pairs.

BBAUpdate

BBAUpdate records use schema id 21.

Field Type Description
record_length uint32 Total record bytes.
local_id uint32 Instrument dictionary id.
flags uint32 Timestamp and side-availability flags.
seq_type uint32 Sequence mode.
reserved0, reserved1 uint32 Reserved, must be zero.
ts_exch_ns uint64 Exchange-event timestamp.
ts_disp_ns uint64 Provider dispatch timestamp.
ts_recv_ns uint64 Local receive timestamp.
seq0, seq1, seq2 int64 Sequence fields.
bid_price, bid_size int64 Best bid price and size.
ask_price, ask_size int64 Best ask price and size.

TradeUpdate

TradeUpdate records use schema id 22.

Field Type Description
record_length uint32 Total record bytes.
local_id uint32 Instrument dictionary id.
flags uint32 Timestamp flags.
seq_type uint32 Sequence mode.
ts_exch_ns uint64 Exchange-event timestamp.
ts_disp_ns uint64 Provider dispatch timestamp.
ts_recv_ns uint64 Local receive timestamp.
seq0, seq1, seq2 int64 Sequence fields.
price int64 Trade price.
size int64 Trade size.
side int32 Aggressor side: buy, sell, or unknown.
reserved0 uint32 Reserved, must be zero.

The flags fields use the native model flag semantics documented in quantpylib.standards.models. Price and size fields use the scale documented in quantpylib.standards.models.

File Routing

Archive sinks route each event by calling:

context = policy.resolve(feed_id, event, metrics)

resolve() is the filesystem-policy contract. It must return an L0FileContext with:

Field Meaning
path Destination .qbn file path.
venue Venue string written to the QBN file header.
market_class Market class written to the QBN file header.
feed_type Feed type written to the QBN file header.

Custom policies may route by feed id, event timestamp, current metrics, or another application-specific partitioning rule. The default DailyCaptureFileSystemPolicy uses event.ts_recv_ns and routes files by receive date:

root/venue/market_class/feed_type/YYYY-MM-DD.qbn

FeedTypeFileSystemPolicy is available for feed-type, ticker, or parameter-granular files. It accepts the same resolve(feed_id, event, metrics) call shape, but routes from feed_id and does not require the event.

Reading And Exporting

Use schema-specific readers when the schema is known, or reader_for_file when the file header should select the reader.

from quantpylib.hft.l0 import L0BookUpdateReader, reader_for_file

reader = L0BookUpdateReader("./archives/binance/perp/l2book/2026-07-08.qbn")
records = list(reader.records(symbol="BTCUSDT"))

reader = reader_for_file("./archives/binance/perp/l2book/2026-07-08.qbn")
df = reader.to_pandas(include_offsets=True)
parquet_path = reader.to_parquet()

Reader filters use exchange-event time:

  • start: inclusive ts_exch_ns.
  • end: exclusive ts_exch_ns.
  • symbol: event ticker after dictionary resolution.

Native Objects

Native event objects are documented in quantpylib.standards.models. The L0 reader decodes records back into those same object types; downstream materializers and replay tools should consume the native event objects rather than parsing QBN bytes directly unless they are implementing a new low-level reader.

Capture Sinks

L0ArchiveSink

Bases: FeedSink

Queued feed sink for live L0/QBN capture.

L0ArchiveSink accepts feed events without doing file IO on the feed callback path. Events are queued and written by a background L0DirectArchiveSink. Queue overflow behavior is controlled by backpressure.

Parameters:

Name Type Description Default
root str | Path

Archive root used with the default daily capture policy.

None
policy L0FileSystemPolicy

File routing policy. When not supplied, root is required and DailyCaptureFileSystemPolicy is used.

None
max_payload_bytes int

Maximum unsealed block payload bytes.

DEFAULT_L0_MAX_PAYLOAD_BYTES
max_block_seconds float

Maximum block age before sealing.

DEFAULT_L0_MAX_BLOCK_SECONDS
writer_idle_seconds float

Idle age after which non-active writers are closed. Defaults to three block-age windows.

None
writer_retire_interval_records int

Record interval for checking idle writers.

1000
fsync bool

Whether to fsync after sealed block writes.

False
validate_existing bool

Whether to validate existing blocks before appending to an existing QBN file.

True
maxsize int

Queue capacity.

QUEUE_MAXSIZE
backpressure FeedSinkBackpressure | str

Queue overflow behavior. FAIL raises FeedSinkBackpressureError; DROP_NEW drops the new event and increments drop metrics.

FAIL
logger Logger

Logger for queue, metrics, writer lifecycle, and write failures.

None
metrics_log_interval_sec float | None

Background metrics logging interval. Use None to disable background metrics logging.

DEFAULT_L0_METRICS_LOG_INTERVAL_SECONDS

metrics property

Sink and writer metrics.

writers property

Active QBN writers keyed by path.

close()

Stop the worker, flush, and close active writers. Idempotent.

flush()

Wait for queued events to be written and flush active writers.

write(feed_id, event)

Enqueue one native event for L0 capture.

Parameters:

Name Type Description Default
feed_id FeedId

Feed identity.

required
event BookUpdate | BBAUpdate | TradeUpdate

Native event to capture.

required

Returns:

Name Type Description
bool

True when the event was enqueued, False when the sink is already closed or the event was dropped by DROP_NEW backpressure.

Raises:

Type Description
L0WriteError

If the worker has failed.

FeedSinkBackpressureError

If the queue is full and backpressure mode is FAIL.

L0DirectArchiveSink

Bases: FeedSink

Synchronous feed sink that writes native events to L0/QBN files.

L0DirectArchiveSink owns one writer per resolved QBN path. It is useful when the caller wants file IO on the feed callback path or when another queue already owns backpressure. Most live feed capture should use L0ArchiveSink, which moves file IO to a worker thread.

Parameters:

Name Type Description Default
root str | Path

Archive root used with the default daily capture policy.

None
policy L0FileSystemPolicy

File routing policy. When not supplied, root is required and DailyCaptureFileSystemPolicy is used.

None
max_payload_bytes int

Maximum unsealed block payload bytes.

DEFAULT_L0_MAX_PAYLOAD_BYTES
max_block_seconds float

Maximum block age before sealing.

DEFAULT_L0_MAX_BLOCK_SECONDS
writer_idle_seconds float

Idle age after which non-active writers are closed. Defaults to three block-age windows.

None
writer_retire_interval_records int

Record interval for checking idle writers.

1000
fsync bool

Whether to fsync after sealed block writes.

False
validate_existing bool

Whether to validate existing blocks before appending to an existing QBN file.

True
metrics L0DirectArchiveMetrics

Metrics object to update.

None
logger Logger

Logger for metrics, writer lifecycle, and write failures.

None
metrics_log_interval_sec float | None

Background metrics logging interval. Use None to disable background metrics logging.

DEFAULT_L0_METRICS_LOG_INTERVAL_SECONDS

close()

Flush and close every active writer. Idempotent.

flush()

Seal pending blocks and flush all active writers.

log_metrics()

Emit a metrics snapshot through the configured logger.

write(feed_id, event)

Write one native event to the resolved QBN file.

Parameters:

Name Type Description Default
feed_id FeedId

Feed identity.

required
event BookUpdate | BBAUpdate | TradeUpdate

Native event to capture.

required

Returns:

Name Type Description
bool

True when the event was written, False when the sink is already closed.

Raises:

Type Description
L0WriteError

If the event cannot be written.

Filesystem Policies

L0FileSystemPolicy

Base contract for L0/QBN filesystem routing policies.

Archive sinks call resolve(feed_id, event, metrics) for each captured event to choose the destination QBN file and file-header context. Custom policies should return an L0FileContext and may use feed_id, event timestamps, or current sink metrics to choose a path.

resolve(feed_id, event, metrics)

Resolve an event destination.

Parameters:

Name Type Description Default
feed_id FeedId

Feed identity.

required
event Any

Native schema-2 event being captured.

required
metrics L0DirectArchiveMetrics

Current sink metrics.

required

Returns:

Name Type Description
L0FileContext L0FileContext

Destination file and header context.

DailyCaptureFileSystemPolicy

Bases: L0FileSystemPolicy

Resolve QBN files by receive-time calendar date.

The default layout is root/venue/market_class/feed_type/YYYY-MM-DD.qbn.

Parameters:

Name Type Description Default
root str | Path

Archive root.

'.'
suffix str

File suffix. Defaults to .qbn.

'.qbn'
tz tzinfo

Time zone used to derive the receive-date partition from event.ts_recv_ns.

utc

resolve(feed_id, event, metrics=None)

Resolve a destination from feed identity and receive date.

The receive-date partition is derived from event.ts_recv_ns in this policy's timezone. The returned context path is root/venue/market_class/feed_type/YYYY-MM-DD.qbn by default.

Parameters:

Name Type Description Default
feed_id FeedId

Feed identity.

required
event Any

Native schema-2 event with ts_recv_ns.

required
metrics L0DirectArchiveMetrics

Current sink metrics.

None

Returns:

Name Type Description
L0FileContext L0FileContext

Destination file and header context.

FeedTypeFileSystemPolicy

Bases: L0FileSystemPolicy

Resolve QBN files by feed context instead of receive date.

Parameters:

Name Type Description Default
root str | Path

Archive root.

'.'
suffix str

File suffix. Defaults to .qbn.

'.qbn'
granularity str

File grouping mode. Supported values are feed_type, ticker, and params.

'feed_type'

resolve(feed_id, event=None, metrics=None)

Resolve a destination from feed identity.

event and metrics are accepted for the policy interface but are not used by this policy. The returned context path is rooted at root/venue/market_class and then grouped by granularity.

Parameters:

Name Type Description Default
feed_id FeedId

Feed identity.

required
event Any

Ignored.

None
metrics L0DirectArchiveMetrics

Ignored.

None

Returns:

Name Type Description
L0FileContext L0FileContext

Destination file and header context.

L0FileContext dataclass

File destination and semantic context for an L0 writer.

Parameters:

Name Type Description Default
path Path

QBN file path.

required
venue str

Venue name stored in the file header.

required
market_class str

Market class stored in the file header.

required
feed_type str

Feed type stored in the file header.

required

Writers

L0Writer

Base writer for one L0/QBN event schema.

A writer owns one QBN file and accepts one native event type. The file header is written lazily on the first event. Existing files are reopened by validating complete blocks and truncating any incomplete trailing block before appending.

Parameters:

Name Type Description Default
context L0FileContext

Destination file and header context.

required
max_payload_bytes int

Maximum unsealed block payload bytes.

DEFAULT_L0_MAX_PAYLOAD_BYTES
max_block_seconds float

Maximum block age before sealing.

DEFAULT_L0_MAX_BLOCK_SECONDS
fsync bool

Whether to fsync after sealed block writes.

False
validate_existing bool

Whether to validate existing blocks before appending to an existing QBN file.

True
metrics L0FileMetrics

File metrics object to update.

None

close()

Flush and close the file handle. Idempotent.

flush()

Seal the pending block and flush the file handle.

record_length(event) classmethod

Return the QBN record length for an event.

write(feed_id, event)

Append one event to the current block.

Parameters:

Name Type Description Default
feed_id FeedId

Feed identity used when creating a new file.

required
event BookUpdate | BBAUpdate | TradeUpdate

Native event matching the writer schema.

required

Raises:

Type Description
L0WriteError

If the writer is closed or the event type does not match the writer schema.

L0BookUpdateWriter

Bases: L0Writer

L0/QBN writer for quantpylib.standards.models.BookUpdate events.

L0BBAUpdateWriter

Bases: L0Writer

L0/QBN writer for quantpylib.standards.models.BBAUpdate events.

L0TradeUpdateWriter

Bases: L0Writer

L0/QBN writer for quantpylib.standards.models.TradeUpdate events.

Readers

reader_for_file(path)

Open a QBN file with the reader matching its header schema id.

Parameters:

Name Type Description Default
path str | Path

QBN file path.

required

Returns:

Type Description
L0Reader

Reader instance for the file schema.

L0Reader

Base reader for one L0/QBN event schema.

Readers validate the file schema, skip incomplete trailing blocks, validate complete block CRCs, and decode QBN records back into native event objects.

Parameters:

Name Type Description Default
path str | Path

QBN file path.

required

blocks(*, symbol=None, start=None, end=None)

Iterate decoded QBN blocks.

Parameters:

Name Type Description Default
symbol str

Symbol filter.

None
start int

Inclusive exchange-event timestamp filter in nanoseconds.

None
end int

Exclusive exchange-event timestamp filter in nanoseconds.

None

Yields:

Type Description
dict

Block metadata with decoded symbols and records.

header()

Read the QBN file header.

Returns:

Type Description
dict

Header fields: schema_id, record_domain, venue, market_class, and feed_type.

records(*, symbol=None, start=None, end=None, with_context=False)

Iterate decoded native events.

Parameters:

Name Type Description Default
symbol str

Symbol filter.

None
start int

Inclusive exchange-event timestamp filter in nanoseconds.

None
end int

Exclusive exchange-event timestamp filter in nanoseconds.

None
with_context bool

When true, yield dictionaries containing file context, block offset, record index, symbol, and event.

False

Yields:

Type Description
BookUpdate | BBAUpdate | TradeUpdate | dict

Native event, or a context dictionary when with_context is true.

scan_valid_blocks()

Return all currently valid decoded blocks.

Returns:

Type Description
list

Decoded block dictionaries.

to_pandas(*, symbol=None, start=None, end=None, include_offsets=False)

Materialize decoded records as a pandas DataFrame.

Parameters:

Name Type Description Default
symbol str

Symbol filter.

None
start int

Inclusive exchange-event timestamp filter in nanoseconds.

None
end int

Exclusive exchange-event timestamp filter in nanoseconds.

None
include_offsets bool

Include block_offset and record_index columns.

False

Returns:

Type Description
DataFrame

Decoded records in tabular form.

to_parquet(path=None, *, symbol=None, start=None, end=None, include_offsets=False, **kwargs)

Export decoded records to a derived Parquet file.

Parameters:

Name Type Description Default
path str | Path

Output path. Defaults to the QBN path with a .parquet suffix.

None
symbol str

Symbol filter.

None
start int

Inclusive exchange-event timestamp filter in nanoseconds.

None
end int

Exclusive exchange-event timestamp filter in nanoseconds.

None
include_offsets bool

Include block_offset and record_index columns.

False
**kwargs Any

Forwarded to pandas.DataFrame.to_parquet.

{}

Returns:

Type Description
Path

Parquet output path.

L0BookUpdateReader

Bases: L0Reader

L0/QBN reader for quantpylib.standards.models.BookUpdate files.

L0BBAUpdateReader

Bases: L0Reader

L0/QBN reader for quantpylib.standards.models.BBAUpdate files.

L0TradeUpdateReader

Bases: L0Reader

L0/QBN reader for quantpylib.standards.models.TradeUpdate files.