Skip to content

L0 Data Capture / Quantpylib Binary eNcoding (QBN)

HFT systems deal with large volumes of tick data, from BBA to book updates and trades across many instruments and exchanges. These datasets are typically stored in flat binary files, which provide compact storage, high sequential-write throughput, predictable record layouts, and efficient streaming reads without relying on database read-write patterns.

Furthermore, it is critical to store receive (RX) timings alongside exchange and provider-dispatch timestamps. RX time records when the trading process could actually observe an event. Preserving that information boundary allows users to measure live phenomena such as feed latency, bursts, backpressure and message races without speculation.

Quantpylib Binary eNcoding

Quantpylib Binary eNcoding (QBN) is quantpylib's binary capture format for native market-data events. L0 data capture is the durable, lowest-level event log beneath higher-level functions such as market data replay and analytics. The capture path records the normalized event objects emitted by wrappers and gateway subscriptions. These artefacts can later be used to inspect trading system market data latency, backtesting, or combined with event traces to get markouts, et cetera.

This tutorial follows quantpylib's production capture path from a live Feed subscription to durable QBN files. It explains how to attach archival sinks, organize multi-exchange output, and read captured events. For technical documentation and business logic fields, use the quantpylib.hft.l0 technical reference.

In Quantpylib, the supported L0 event schemas are:

We use quantpylib.hft.feed.Feed to subscribe to market data and pass quantpylib.hft.l0.L0ArchiveSink into the feed as a global sink or per-feed sink. Feed owns event fanout; L0ArchiveSink owns queued durable capture; Writer and Reader classes are responsible for encoding and decoding QBN files.

Capture Path

wrapper websocket/http message
        |
        v
gateway executor subscription
        |
        v
schema-2 native event
BookUpdate / BBAUpdate / TradeUpdate
        |
        v
Feed fanout
        |
        +--> on_msg callback / materializer sinks
        |
        +--> L0ArchiveSink
                 |
                 v
              writer queue
                 |
                 v
              L0DirectArchiveSink
                 |
                 v
              L0Writer -> QBN file

Typical usage captures every event from a feed while keeping file IO off the feed callback path:

from quantpylib.hft.feed import Feed
from quantpylib.hft.l0 import L0ArchiveSink

sink = L0ArchiveSink(root="./archives")
feed = Feed(gateway=gateway, sinks=[sink])

await feed.add_l2_book_feed(
    exc="binance",
    ticker="BTCUSDT",
    depth=20,
)

Per-feed sinks can also be used when only selected subscriptions should be captured:

book_feed = await feed.add_l2_book_feed(
    exc="binance",
    ticker="BTCUSDT",
    sinks=[L0ArchiveSink(root="./archives")],
)

That's all. A .qbn file will be written. To understand the file layout and custom policies, read on.

Why QBN

  • Native event fidelity: QBN records quantpylib.standards.models.BookUpdate, quantpylib.standards.models.BBAUpdate, and quantpylib.standards.models.TradeUpdate events as they happened with lossless capture for critical fields used for event reconstruction.
  • Append-friendly writes: writers append sealed blocks and can reopen existing files.
  • Partial-tail recovery: readers ignore incomplete trailing blocks; writers can truncate an incomplete tail before appending.
  • Block-level validation: each block carries a CRC over the block header, instrument dictionary, and record payload.
  • Self-describing: the file header stores the schema id, record-domain string, venue, market class, and feed type with the captured data.
  • Compact symbology: records store a local_id; the per-block dictionary stores local_id -> symbol once for that block.
  • Index-friendly time-range scanning: block headers store exchange-time and receive-time bounds, so readers can skip out-of-range blocks before decoding payloads. Applications can design custom indexing based on file format and application semantics.

Example

Here is an example script of how to easily archive different types of tick data from multiple exchanges.

scripts/journal.py:

import os
import asyncio
import logging
from decimal import Decimal

from dotenv import load_dotenv

from quantpylib.hft.oms import OMS
from quantpylib.logger import Logger
from quantpylib.gateway.master import Gateway


load_dotenv()

exchange = "bybit"
ticker = "HYPEUSDT"
amount = Decimal("1")
price = Decimal("10")

async def main():
    gateway = Gateway({
        exchange: {
            "key": os.environ["TEST_BYBIT_KEY"],
            "secret": os.environ["TEST_BYBIT_SECRET"],
        },
    })
    await gateway.init_clients()

    os.makedirs("logs", exist_ok=True)
    logger = Logger(
        name="perf",
        stdout_register=True,
        filename="journal.log",
        logs_dir="./logs",
        file_level=logging.INFO,
    )

    oms = OMS(gateway, logger=logger)

    try:
        await oms.init()
        cloid = oms.rand_cloid(exc=exchange)
        res = await oms.limit_order(
            exc=exchange,
            ticker=ticker,
            amount=amount,
            price=price,
            cloid=cloid,
        )
        print(res)
        await asyncio.sleep(5)

        res = await oms.cancel_order(exc=exchange, ticker=ticker, cloid=cloid)
        print(res)
    finally:
        await oms.cleanup()
        logger.shutdown()

if __name__ == "__main__":
    asyncio.run(main())