Exchange Wire Formats

Exchange wire formats

Each exchange sends market data in its own format. We map each format to one schema for each data type. The pages in this section show real messages from each exchange next to the exact rows that we publish for them, column by column.

How a message becomes a row

  1. Collect. Our collector subscribes to the public WebSocket channels of the exchange. If an exchange has no stream, the collector polls REST endpoints (for example, for order book snapshots and some open interest feeds). The collector adds our receive time to each message and writes the message to an hourly raw file.
  2. Parse and check. After the hour ends, a postprocessor for the exchange parses the messages from the raw file with the field names of the exchange. It checks the order book sequences. It applies the rules that each exchange page shows.
  3. Write. The postprocessor writes one Parquet file for each exchange, symbol, data type and hour. The files have the same columns for all exchanges.

Conventions

  • received_time is the time on the clock of our collector when the collector received the message. The unit is nanoseconds since the Unix epoch. It is the only column that no exchange sends. We partition the files by this column.
  • We copy exchange timestamps, for example event_time, as the exchange sent them. The unit is milliseconds if the exchange page does not give a different unit.
  • Prices, quantities, rates and volumes are strings. If the exchange sends decimal strings, we keep its exact digits. For example, 85583.10 stays 85583.10. The exchange pages tell you where an exchange sends JSON numbers instead.
  • If the exchange does not supply a column, the column is null. If the schema needs a value that an exchange does not send, the rule tells you which fixed placeholder we write.
  • Order books have one row for each price level. snapshot rows show the full order book at that time. update rows set the quantity at one price, and a zero quantity removes the level. Some messages have sequence IDs but no levels, for example an empty snapshot or an update that only advances the sequence. Each of these messages becomes one row with side noop and with price and quantity 0, so that the sequence stays continuous. An empty noop snapshot clears the order book.

The pages describe the pipeline as it is now. Files that we wrote before a change can be different. For these changes and for corrections to published history, refer to Known Gaps & Corrections.

Coverage by exchange

How we keep these pages correct

We keep each example as the exact text that the exchange sent. Our test suite replays each example through the production collector parser and the postprocessor of the exchange. The test fails if the published rows are different from the rows on the page. It also fails if the source path of a column is not in the message. We cannot release a change to how we map an exchange without an update to its page. For known exceptions and corrections to published history, refer to Known Gaps & Corrections.

If you cannot find the information that you need, tell us. We also help with integrations.

Contact support