L0 Data Capture / Quantpylib Binary eNcoding (QBN)
Quantpylib Binary eNcoding (QBN) is quantpylib's binary capture format for native market-data events. L0 data capture is the durable, lowest-level event log beneath higher-level functions such as market data replay and analytics. The capture path records the normalized event objects emitted by wrappers and gateway subscriptions. These artefacts can later be used to inspect trading system market data latency, backtesting, or combined with event traces to get markouts, et cetera.
The supported L0 event schemas are:
- quantpylib.standards.models.BookUpdate
- quantpylib.standards.models.BBAUpdate
- quantpylib.standards.models.TradeUpdate
Use quantpylib.hft.feed.Feed to subscribe to
market data and pass quantpylib.hft.l0.L0ArchiveSink
into the feed as a global sink or per-feed sink. Feed owns event fanout;
L0ArchiveSink owns queued durable capture; Writer and Reader classes are responsible
for encoding and decoding QBN files.
Capture Path

wrapper websocket/http message
|
v
gateway executor subscription
|
v
schema-2 native event
BookUpdate / BBAUpdate / TradeUpdate
|
v
Feed fanout
|
+--> on_msg callback / materializer sinks
|
+--> L0ArchiveSink
|
v
writer queue
|
v
L0DirectArchiveSink
|
v
L0Writer -> QBN file
Typical usage captures every event from a feed while keeping file IO off the feed callback path:
from quantpylib.hft.feed import Feed
from quantpylib.hft.l0 import L0ArchiveSink
sink = L0ArchiveSink(root="./archives")
feed = Feed(gateway=gateway, sinks=[sink])
await feed.add_l2_book_feed(
exc="binance",
ticker="BTCUSDT",
depth=20,
)
Per-feed sinks can also be used when only selected subscriptions should be captured:
book_feed = await feed.add_l2_book_feed(
exc="binance",
ticker="BTCUSDT",
sinks=[L0ArchiveSink(root="./archives")],
)
That's all. A .qbn file will be written. To understand the file layout and custom
policies, read on.
Why QBN
- Native event fidelity: QBN records quantpylib.standards.models.BookUpdate, quantpylib.standards.models.BBAUpdate, and quantpylib.standards.models.TradeUpdate events as they happened with lossless capture for critical fields used for event reconstruction.
- Append-friendly writes: writers append sealed blocks and can reopen existing files.
- Partial-tail recovery: readers ignore incomplete trailing blocks; writers can truncate an incomplete tail before appending.
- Block-level validation: each block carries a CRC over the block header, instrument dictionary, and record payload.
- Self-describing: the file header stores the schema id, record-domain string, venue, market class, and feed type with the captured data.
- Compact symbology: records store a
local_id; the per-block dictionary storeslocal_id -> symbolonce for that block. - Index-friendly time-range scanning: block headers store exchange-time and receive-time bounds, so readers can skip out-of-range blocks before decoding payloads. Applications can design custom indexing based on file format and application semantics.
File Layout
A QBN file starts with one file header followed by zero or more sealed blocks. All fixed-width numeric fields are little-endian.
QBN file
|
+-- FileHeader
|
+-- Block 0
| |
| +-- BlockHeader
| +-- InstrumentDictionary
| +-- RecordPayload
| |
| +-- Record 0
| +-- Record 1
| +-- ...
|
+-- Block 1
| |
| +-- BlockHeader
| +-- InstrumentDictionary
| +-- RecordPayload
|
+-- ...
File Header
| Field | Type | Description |
|---|---|---|
FILE_MAGIC |
char[4] |
File magic. Current value: QBN1. |
schema_id |
uint32 |
Event schema id. |
record_domain_length |
uint32 |
Byte length of record_domain. |
record_domain |
bytes |
UTF-8 record-domain string describing the record layout. |
venue_length, venue |
uint16, bytes |
Venue name. |
market_class_length, market_class |
uint16, bytes |
Market class. |
feed_type_length, feed_type |
uint16, bytes |
Feed type. |
Current schema ids:
| Schema id | Event object | Reader | Writer |
|---|---|---|---|
20 |
BookUpdate | L0BookUpdateReader | L0BookUpdateWriter |
21 |
BBAUpdate | L0BBAUpdateReader | L0BBAUpdateWriter |
22 |
TradeUpdate | L0TradeUpdateReader | L0TradeUpdateWriter |
Block Layout
Blocks are the unit of validation and partial recovery.
Block
|
+-- BlockHeader
| magic = QBLK1
| block_length
| record_count
| min_ts_exch_ns / max_ts_exch_ns
| min_ts_recv_ns / max_ts_recv_ns
| dictionary_length
| payload_length
| crc
|
+-- InstrumentDictionary
| symbol_count
| repeated local_id, symbol_length, symbol
|
+-- RecordPayload
repeated schema-specific QBN records
| Field | Type | Description |
|---|---|---|
BLOCK_MAGIC |
char[5] |
Block magic. Current value: QBLK1. |
block_length |
uint32 |
Total block bytes, including header, dictionary, and payload. |
record_count |
uint32 |
Number of records in the payload. |
min_ts_exch_ns |
uint64 |
Minimum exchange-event timestamp in the block. |
max_ts_exch_ns |
uint64 |
Maximum exchange-event timestamp in the block. |
min_ts_recv_ns |
uint64 |
Minimum local receive timestamp in the block. |
max_ts_recv_ns |
uint64 |
Maximum local receive timestamp in the block. |
dictionary_length |
uint32 |
Instrument dictionary byte length. |
payload_length |
uint32 |
Record payload byte length. |
crc |
uint32 |
CRC32 over the zero-CRC header plus dictionary and payload. |
Blocks seal when either max_payload_bytes would be exceeded or
max_block_seconds has elapsed since the first record in the pending block.
Defaults are 32 MiB and 60 seconds.
Instrument Dictionary
Each block has its own compact symbol dictionary.
| Field | Type | Description |
|---|---|---|
symbol_count |
uint32 |
Number of symbols in the block. |
local_id |
uint32 |
Block-local symbol id referenced by records. |
symbol_length |
uint16 |
Symbol byte length. |
symbol |
bytes |
UTF-8 symbol. |
Runtime event bytes are self-contained and include the ticker string. QBN
record bytes are archive-local and replace the ticker string with local_id.
The dictionary stores the symbol once per block.
Record Layouts
The payload is a sequence of records for the file schema. A file contains one schema id and one record domain.
BookUpdate
BookUpdate records use schema id 20.
| Field | Type | Description |
|---|---|---|
record_length |
uint32 |
Total record bytes. |
local_id |
uint32 |
Instrument dictionary id. |
data_length |
uint32 |
Book-level buffer byte length. |
flags |
uint32 |
Snapshot, timestamp, and ordering flags. |
nbids |
uint32 |
Number of bid levels. |
nasks |
uint32 |
Number of ask levels. |
seq_type |
uint32 |
Sequence mode. |
reserved0 |
uint32 |
Reserved, must be zero. |
ts_exch_ns |
uint64 |
Exchange-event timestamp. |
ts_disp_ns |
uint64 |
Provider dispatch timestamp. |
ts_recv_ns |
uint64 |
Local receive timestamp. |
seq0, seq1, seq2 |
int64 |
Sequence fields. |
data |
int64[] |
Bid price/size pairs followed by ask price/size pairs. |
BBAUpdate
BBAUpdate records use schema id 21.
| Field | Type | Description |
|---|---|---|
record_length |
uint32 |
Total record bytes. |
local_id |
uint32 |
Instrument dictionary id. |
flags |
uint32 |
Timestamp and side-availability flags. |
seq_type |
uint32 |
Sequence mode. |
reserved0, reserved1 |
uint32 |
Reserved, must be zero. |
ts_exch_ns |
uint64 |
Exchange-event timestamp. |
ts_disp_ns |
uint64 |
Provider dispatch timestamp. |
ts_recv_ns |
uint64 |
Local receive timestamp. |
seq0, seq1, seq2 |
int64 |
Sequence fields. |
bid_price, bid_size |
int64 |
Best bid price and size. |
ask_price, ask_size |
int64 |
Best ask price and size. |
TradeUpdate
TradeUpdate records use schema id 22.
| Field | Type | Description |
|---|---|---|
record_length |
uint32 |
Total record bytes. |
local_id |
uint32 |
Instrument dictionary id. |
flags |
uint32 |
Timestamp flags. |
seq_type |
uint32 |
Sequence mode. |
ts_exch_ns |
uint64 |
Exchange-event timestamp. |
ts_disp_ns |
uint64 |
Provider dispatch timestamp. |
ts_recv_ns |
uint64 |
Local receive timestamp. |
seq0, seq1, seq2 |
int64 |
Sequence fields. |
price |
int64 |
Trade price. |
size |
int64 |
Trade size. |
side |
int32 |
Aggressor side: buy, sell, or unknown. |
reserved0 |
uint32 |
Reserved, must be zero. |
The flags fields use the native model flag semantics documented in
quantpylib.standards.models. Price
and size fields use the scale documented in
quantpylib.standards.models.
File Routing
Archive sinks route each event by calling:
resolve() is the filesystem-policy contract. It must return an
L0FileContext with:
| Field | Meaning |
|---|---|
path |
Destination .qbn file path. |
venue |
Venue string written to the QBN file header. |
market_class |
Market class written to the QBN file header. |
feed_type |
Feed type written to the QBN file header. |
Custom policies may route by feed id, event timestamp, current metrics, or
another application-specific partitioning rule. The default
DailyCaptureFileSystemPolicy
uses event.ts_recv_ns and routes files by receive date:
FeedTypeFileSystemPolicy is
available for feed-type, ticker, or parameter-granular files. It accepts the
same resolve(feed_id, event, metrics) call shape, but routes from feed_id
and does not require the event.
Reading And Exporting
Use schema-specific readers when the schema is known, or reader_for_file when the file header should select the reader.
from quantpylib.hft.l0 import L0BookUpdateReader, reader_for_file
reader = L0BookUpdateReader("./archives/binance/perp/l2book/2026-07-08.qbn")
records = list(reader.records(symbol="BTCUSDT"))
reader = reader_for_file("./archives/binance/perp/l2book/2026-07-08.qbn")
df = reader.to_pandas(include_offsets=True)
parquet_path = reader.to_parquet()
Reader filters use exchange-event time:
start: inclusivets_exch_ns.end: exclusivets_exch_ns.symbol: event ticker after dictionary resolution.
Native Objects
Native event objects are documented in quantpylib.standards.models. The L0 reader decodes records back into those same object types; downstream materializers and replay tools should consume the native event objects rather than parsing QBN bytes directly unless they are implementing a new low-level reader.
Capture Sinks
L0ArchiveSink
Bases: FeedSink
Queued feed sink for live L0/QBN capture.
L0ArchiveSink accepts feed events without doing file IO on the feed
callback path. Events are queued and written by a background
L0DirectArchiveSink. Queue
overflow behavior is controlled by backpressure.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
root
|
str | Path
|
Archive root used with the default daily capture policy. |
None
|
policy
|
L0FileSystemPolicy
|
File routing policy. When not
supplied, |
None
|
max_payload_bytes
|
int
|
Maximum unsealed block payload bytes. |
DEFAULT_L0_MAX_PAYLOAD_BYTES
|
max_block_seconds
|
float
|
Maximum block age before sealing. |
DEFAULT_L0_MAX_BLOCK_SECONDS
|
writer_idle_seconds
|
float
|
Idle age after which non-active writers are closed. Defaults to three block-age windows. |
None
|
writer_retire_interval_records
|
int
|
Record interval for checking idle writers. |
1000
|
fsync
|
bool
|
Whether to fsync after sealed block writes. |
False
|
validate_existing
|
bool
|
Whether to validate existing blocks before appending to an existing QBN file. |
True
|
maxsize
|
int
|
Queue capacity. |
QUEUE_MAXSIZE
|
backpressure
|
FeedSinkBackpressure | str
|
Queue overflow behavior.
|
FAIL
|
logger
|
Logger
|
Logger for queue, metrics, writer lifecycle, and write failures. |
None
|
metrics_log_interval_sec
|
float | None
|
Background metrics logging
interval. Use |
DEFAULT_L0_METRICS_LOG_INTERVAL_SECONDS
|
metrics
property
Sink and writer metrics.
writers
property
Active QBN writers keyed by path.
close()
Stop the worker, flush, and close active writers. Idempotent.
flush()
Wait for queued events to be written and flush active writers.
write(feed_id, event)
Enqueue one native event for L0 capture.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feed_id
|
FeedId
|
Feed identity. |
required |
event
|
BookUpdate | BBAUpdate | TradeUpdate
|
Native event to capture. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bool |
|
Raises:
| Type | Description |
|---|---|
L0WriteError
|
If the worker has failed. |
FeedSinkBackpressureError
|
If the queue is full and backpressure
mode is |
L0DirectArchiveSink
Bases: FeedSink
Synchronous feed sink that writes native events to L0/QBN files.
L0DirectArchiveSink owns one writer per resolved QBN path. It is useful
when the caller wants file IO on the feed callback path or when another
queue already owns backpressure. Most live feed capture should use
L0ArchiveSink, which moves file IO to a
worker thread.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
root
|
str | Path
|
Archive root used with the default daily capture policy. |
None
|
policy
|
L0FileSystemPolicy
|
File routing policy. When not
supplied, |
None
|
max_payload_bytes
|
int
|
Maximum unsealed block payload bytes. |
DEFAULT_L0_MAX_PAYLOAD_BYTES
|
max_block_seconds
|
float
|
Maximum block age before sealing. |
DEFAULT_L0_MAX_BLOCK_SECONDS
|
writer_idle_seconds
|
float
|
Idle age after which non-active writers are closed. Defaults to three block-age windows. |
None
|
writer_retire_interval_records
|
int
|
Record interval for checking idle writers. |
1000
|
fsync
|
bool
|
Whether to fsync after sealed block writes. |
False
|
validate_existing
|
bool
|
Whether to validate existing blocks before appending to an existing QBN file. |
True
|
metrics
|
L0DirectArchiveMetrics
|
Metrics object to update. |
None
|
logger
|
Logger
|
Logger for metrics, writer lifecycle, and write failures. |
None
|
metrics_log_interval_sec
|
float | None
|
Background metrics logging
interval. Use |
DEFAULT_L0_METRICS_LOG_INTERVAL_SECONDS
|
close()
Flush and close every active writer. Idempotent.
flush()
Seal pending blocks and flush all active writers.
log_metrics()
Emit a metrics snapshot through the configured logger.
write(feed_id, event)
Write one native event to the resolved QBN file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feed_id
|
FeedId
|
Feed identity. |
required |
event
|
BookUpdate | BBAUpdate | TradeUpdate
|
Native event to capture. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bool |
|
Raises:
| Type | Description |
|---|---|
L0WriteError
|
If the event cannot be written. |
Filesystem Policies
L0FileSystemPolicy
Base contract for L0/QBN filesystem routing policies.
Archive sinks call resolve(feed_id, event, metrics) for each captured
event to choose the destination QBN file and file-header context. Custom
policies should return an L0FileContext and may use feed_id, event
timestamps, or current sink metrics to choose a path.
resolve(feed_id, event, metrics)
Resolve an event destination.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feed_id
|
FeedId
|
Feed identity. |
required |
event
|
Any
|
Native schema-2 event being captured. |
required |
metrics
|
L0DirectArchiveMetrics
|
Current sink metrics. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
L0FileContext |
L0FileContext
|
Destination file and header context. |
DailyCaptureFileSystemPolicy
Bases: L0FileSystemPolicy
Resolve QBN files by receive-time calendar date.
The default layout is
root/venue/market_class/feed_type/YYYY-MM-DD.qbn.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
root
|
str | Path
|
Archive root. |
'.'
|
suffix
|
str
|
File suffix. Defaults to |
'.qbn'
|
tz
|
tzinfo
|
Time zone used to derive the receive-date
partition from |
utc
|
resolve(feed_id, event, metrics=None)
Resolve a destination from feed identity and receive date.
The receive-date partition is derived from event.ts_recv_ns in this
policy's timezone. The returned context path is
root/venue/market_class/feed_type/YYYY-MM-DD.qbn by default.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feed_id
|
FeedId
|
Feed identity. |
required |
event
|
Any
|
Native schema-2 event with |
required |
metrics
|
L0DirectArchiveMetrics
|
Current sink metrics. |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
L0FileContext |
L0FileContext
|
Destination file and header context. |
FeedTypeFileSystemPolicy
Bases: L0FileSystemPolicy
Resolve QBN files by feed context instead of receive date.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
root
|
str | Path
|
Archive root. |
'.'
|
suffix
|
str
|
File suffix. Defaults to |
'.qbn'
|
granularity
|
str
|
File grouping mode. Supported values are
|
'feed_type'
|
resolve(feed_id, event=None, metrics=None)
Resolve a destination from feed identity.
event and metrics are accepted for the policy interface but are not
used by this policy. The returned context path is rooted at
root/venue/market_class and then grouped by granularity.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feed_id
|
FeedId
|
Feed identity. |
required |
event
|
Any
|
Ignored. |
None
|
metrics
|
L0DirectArchiveMetrics
|
Ignored. |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
L0FileContext |
L0FileContext
|
Destination file and header context. |
L0FileContext
dataclass
File destination and semantic context for an L0 writer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
Path
|
QBN file path. |
required |
venue
|
str
|
Venue name stored in the file header. |
required |
market_class
|
str
|
Market class stored in the file header. |
required |
feed_type
|
str
|
Feed type stored in the file header. |
required |
Writers
L0Writer
Base writer for one L0/QBN event schema.
A writer owns one QBN file and accepts one native event type. The file header is written lazily on the first event. Existing files are reopened by validating complete blocks and truncating any incomplete trailing block before appending.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context
|
L0FileContext
|
Destination file and header context. |
required |
max_payload_bytes
|
int
|
Maximum unsealed block payload bytes. |
DEFAULT_L0_MAX_PAYLOAD_BYTES
|
max_block_seconds
|
float
|
Maximum block age before sealing. |
DEFAULT_L0_MAX_BLOCK_SECONDS
|
fsync
|
bool
|
Whether to fsync after sealed block writes. |
False
|
validate_existing
|
bool
|
Whether to validate existing blocks before appending to an existing QBN file. |
True
|
metrics
|
L0FileMetrics
|
File metrics object to update. |
None
|
close()
Flush and close the file handle. Idempotent.
flush()
Seal the pending block and flush the file handle.
record_length(event)
classmethod
Return the QBN record length for an event.
write(feed_id, event)
Append one event to the current block.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feed_id
|
FeedId
|
Feed identity used when creating a new file. |
required |
event
|
BookUpdate | BBAUpdate | TradeUpdate
|
Native event matching the writer schema. |
required |
Raises:
| Type | Description |
|---|---|
L0WriteError
|
If the writer is closed or the event type does not match the writer schema. |
L0BookUpdateWriter
L0BBAUpdateWriter
L0TradeUpdateWriter
Readers
reader_for_file(path)
Open a QBN file with the reader matching its header schema id.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
QBN file path. |
required |
Returns:
| Type | Description |
|---|---|
L0Reader
|
Reader instance for the file schema. |
L0Reader
Base reader for one L0/QBN event schema.
Readers validate the file schema, skip incomplete trailing blocks, validate complete block CRCs, and decode QBN records back into native event objects.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
QBN file path. |
required |
blocks(*, symbol=None, start=None, end=None)
Iterate decoded QBN blocks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
symbol
|
str
|
Symbol filter. |
None
|
start
|
int
|
Inclusive exchange-event timestamp filter in nanoseconds. |
None
|
end
|
int
|
Exclusive exchange-event timestamp filter in nanoseconds. |
None
|
Yields:
| Type | Description |
|---|---|
dict
|
Block metadata with decoded |
header()
Read the QBN file header.
Returns:
| Type | Description |
|---|---|
dict
|
Header fields: |
records(*, symbol=None, start=None, end=None, with_context=False)
Iterate decoded native events.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
symbol
|
str
|
Symbol filter. |
None
|
start
|
int
|
Inclusive exchange-event timestamp filter in nanoseconds. |
None
|
end
|
int
|
Exclusive exchange-event timestamp filter in nanoseconds. |
None
|
with_context
|
bool
|
When true, yield dictionaries containing file
context, block offset, record index, symbol, and |
False
|
Yields:
| Type | Description |
|---|---|
BookUpdate | BBAUpdate | TradeUpdate | dict
|
Native event, or a
context dictionary when |
scan_valid_blocks()
Return all currently valid decoded blocks.
Returns:
| Type | Description |
|---|---|
list
|
Decoded block dictionaries. |
to_pandas(*, symbol=None, start=None, end=None, include_offsets=False)
Materialize decoded records as a pandas DataFrame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
symbol
|
str
|
Symbol filter. |
None
|
start
|
int
|
Inclusive exchange-event timestamp filter in nanoseconds. |
None
|
end
|
int
|
Exclusive exchange-event timestamp filter in nanoseconds. |
None
|
include_offsets
|
bool
|
Include |
False
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
Decoded records in tabular form. |
to_parquet(path=None, *, symbol=None, start=None, end=None, include_offsets=False, **kwargs)
Export decoded records to a derived Parquet file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
Output path. Defaults to the QBN path
with a |
None
|
symbol
|
str
|
Symbol filter. |
None
|
start
|
int
|
Inclusive exchange-event timestamp filter in nanoseconds. |
None
|
end
|
int
|
Exclusive exchange-event timestamp filter in nanoseconds. |
None
|
include_offsets
|
bool
|
Include |
False
|
**kwargs
|
Any
|
Forwarded to |
{}
|
Returns:
| Type | Description |
|---|---|
Path
|
Parquet output path. |