Known Gaps & Corrections
This page lists hours that are missing or incomplete in the published archive, behaviour changes that affect how rows should be read, and the corrections we are making to files that are already published. It is based on an audit of every exchange and data type against exchange reference data, completed in September 2026.
Status of corrections
None of the corrections to published history listed below has been applied yet. A corrected file replaces the original under the same object key ({exchange}/{YYYY-MM-DD}/{HH}/{symbol}_{data_type}.parquet) and keeps the same schema, so existing download code keeps working. This page shows when each correction is published; re-download the affected hours after that.
Changes from the 2026-09 data-integrity release
The following changes apply to files published from the 2026-09 data-integrity release onward. Earlier files keep the previous behaviour until the matching correction below is published. The venue notes on each data type page (for example trades and liquidations) explain how to handle older files.
| Exchange ID | Data type | Before the release | From the release |
|---|---|---|---|
okx_spot, okx_futures | trades | One row per taker order per price level: the exchange's trades channel aggregates fills, trade_id is the last fill ID and quantity the summed size | One row per fill (trades-all channel) |
okx_futures | mark_price (funding rows) | event_time is the next settlement time (fundingTime), ahead of when the rate was observed | event_time is the publish time of the funding message (ts); fundingTime stays in next_funding_time |
okx_futures | ticker | base_asset_volume in contracts, quote_asset_volume in base coin | base_asset_volume in base coin (volCcy24h) for linear and inverse swaps; quote_asset_volume = base volume × last price (an approximation of USD volume for inverse swaps) |
kraken_derivatives | mark_price | funding_rate is Kraken's absolute funding rate | funding_rate is Kraken's relative funding rate, a fraction per funding period |
kraken_derivatives | ticker | price_change holds Kraken's 24h change percentage; price_change_percent is not a percentage | price_change = last price − 24h open; price_change_percent = Kraken's 24h change percentage |
kraken_spot | orderbook | Levels pushed out of Kraken's 1,000-level window had no delete row; replay had to truncate | Levels pushed out of the window are published as delete rows, so plain replay stays at 1,000 levels per side |
kraken_derivatives | open_interest (PI_, FI_) | sum_open_interest_value = contracts × price | sum_open_interest_value = contracts × contract size (USD) |
lighter | open_interest | sum_open_interest holds the USD notional | sum_open_interest in base units, sum_open_interest_value = USD notional |
lighter | mark_price | funding_rate in percent; next_funding_time holds the last funding time | funding_rate as a fraction; next_funding_time is the next funding time |
bybit | ticker | price_change_percent as a fraction (-0.0207 = -2.07 %) | price_change_percent in percent |
bybit | ticker, mark_price, open_interest | Change-only deltas: unchanged fields are null; an open interest file exists only for hours in which open interest changed | Full carried-forward state per symbol; every symbol with known open interest gets a row at the hour start (timestamp and received_time = hour start) |
bitget_futures | open_interest | sum_open_interest_value empty | sum_open_interest_value = open interest (base coin) × mark price |
binance_futures | ticker, mark_price, liquidations | COIN-M contracts (e.g. BTCUSD_PERP, BTCUSD_260925, with contract-denominated volumes and quantities) and delisted USDⓈ-M symbols included | Only listed USDⓈ-M symbols published under binance_futures |
binance_spot, binance_futures, aster_futures | orderbook | Each REST depth snapshot published twice (on Binance one copy has received_time = event time × 1e6) | Each REST depth snapshot published once |
hyperliquid_spot, hyperliquid_futures | trades | Trades replayed by the exchange after a reconnect could reappear in a later hour's file | Replayed trades are not republished |
Corrections to published history
These corrections rewrite files that are already published. Unless stated otherwise, they cover the full history of the affected files.
| Correction | Affected files | Status |
|---|---|---|
| Kraken Futures funding rate restated as the relative rate | kraken_derivatives mark_price | Planned |
Kraken Futures ticker price change restated: price_change = last − open, price_change_percent = Kraken's percentage | kraken_derivatives ticker | Planned |
| Kraken inverse open interest value recomputed from contract size | kraken_derivatives open_interest, PI_ and FI_ symbols | Planned |
| Lighter open interest in base units with USD value; funding rate as a fraction; next funding time | lighter open_interest, mark_price | Planned |
OKX funding rows re-timestamped so event_time is no longer the next settlement time. The original publish time was not stored, so repaired rows use our receive time in milliseconds, typically about 140 ms after OKX's publish time | okx_futures mark_price | Planned |
| OKX perpetual swap ticker volumes restated in base and quote units | okx_futures ticker | Planned |
BitMEX liquidations deduplicated; event_time and trade_time converted from nanoseconds to milliseconds | bitmex liquidations | Planned |
| Open interest snapshots repeated from earlier hours removed | binance_futures open_interest | Planned |
| Duplicate copies of REST depth snapshots removed | binance_spot, binance_futures, aster_futures orderbook | Planned |
| Trades repeated from earlier hours removed | hyperliquid_spot, hyperliquid_futures trades | Planned |
Missing BitMEX trades backfilled from the BitMEX REST API. Backfilled rows have no receive time, so received_time = trade_time × 1e6 | bitmex trades | Planned |
| OKX trades rebuilt as one row per fill for roughly the last three months, from the OKX REST API | okx_spot, okx_futures trades | Planned |
| Hour 2026-09-22 21 backfilled from Binance's public trade archives | binance_spot trades, aggTrades | Planned |
Bybit trade gaps backfilled from Bybit's public trade archives. Backfilled rows carry a trade_id derived from the archive's ID, not the WebSocket trade ID | bybit, bybit_spot trades | Planned |
| Empty spot mark price and open interest files removed (spot markets have neither) | bitget_spot mark_price, open_interest, 2025-07-15 → 2026-09-20 | Scheduled for removal |
Missing hours
No files were published for the following hours. Times are UTC and ranges are inclusive hour partitions (YYYY-MM-DD HH).
| Exchange ID | Missing hours (UTC) |
|---|---|
bitget_futures | 2025-09-23 07 → 2025-12-26 20; 2026-01-28 03 → 17; 2026-02-04 07 → 2026-02-14 18; 2026-04-10 10 → 2026-04-11 08; several shorter gaps |
bybit | 2025-11-26 10 → 2025-12-26 19; 2026-04-13 15 → 2026-04-14 21 |
bybit_spot | 2026-04-12 11 → 2026-04-14 21 |
kraken_spot, kraken_derivatives | Hours on 2025-10-21 and 2025-10-22; 2026-07-09 21 → 2026-07-10 09; 2026-08-19 14 → 2026-08-21 14; 2026-09-03 about 15 → 20; several shorter gaps |
lighter | 2026-02-13 12 → 2026-02-14 18 (see also the partial-universe periods below) |
hyperliquid_futures | 2026-04-12 09 → 16 |
bitget_spot | 2025-08-29 03 → 09; outage from 2026-09-20 03:08 (the 2026-09-20 03 files cover 03:00–03:08 only) |
Some other hours are thin: files exist, but only for part of the symbol universe or with less data than usual. If an analysis depends on specific hours, list those hour prefixes with flat-file access and confirm that the symbols you need are present.
Partially affected periods
| Exchange ID | Period (UTC) | Impact | Recovery |
|---|---|---|---|
binance_spot | 2026-09-22, 21:15:30 → 22:00 | All data types for most symbols (about 1,300) are missing for the rest of the hour, although the hour was published as complete. | Trades and aggTrades backfill from Binance's public archives planned; other data types cannot be recovered. |
bitget_futures, bitget_spot | 2026-05-20 03 → 2026-05-24 08 | Order books missing. Trades and tickers were published normally. | Not recoverable (no public order book history). |
lighter | For example 2026-09-22 16:02 → 2026-09-23 about 00:30, and from 2026-09-23 14:32 | Only part of the market universe was published (in the latest episode, 144 of 210 markets stopped, including ETH, HYPE and XRP). About 90 shorter partial-universe episodes occurred earlier. | Not recoverable (no public history for these streams). |
bitmex | Liquid days across the history, e.g. XBTUSD on 2025-11-15, 2026-01-15, 2026-05-15 | 5–20 % of trades missing compared with BitMEX's own trade history. | Backfill from the BitMEX REST API planned. |
binance_futures | Most hours since at least mid-August 2026, around hh:02 → hh:05 | Mark price stream gaps of about 1–2 minutes. | Not recoverable (no public mark price tick history). |
bitget_spot | 2025-07-15 → 2026-09-20 | mark_price and open_interest files exist but every value column is empty; spot markets have no mark price or open interest. | Do not use; scheduled for removal. |
Short reconnect gaps
Collectors reconnect periodically (some venues require it) and after network errors. Data the exchange sends while a connection is being re-established is not recovered, which leaves gaps of seconds:
- Binance (spot and futures): Connections rotate roughly every 24 hours, as Binance requires. Trades sent during the reconnect are missing and show up as trade-ID gaps.
- Bybit trades: Gaps from a few seconds up to about 3 minutes, clustered around a daily reconnect between about 02:00 and 02:50 UTC. Backfill from Bybit's public trade archives is planned.
- Kraken spot: Reconnect gaps of 3–7 seconds, and occasional order book stalls of 10–20 minutes while trades continue.
- Hyperliquid: A small number of trades can be lost around reconnects.
Use sequence fields where a venue provides them (order book update IDs, contiguous Binance trade IDs) to detect these gaps. For order books, discard the local book at a gap and resume from the next snapshot.
BitMEX (historical dataset)
BitMEX has closed: perpetual contracts were settled on 2026-09-16 at 12:00 UTC, spot pairs were delisted on 2026-09-21, and BitMEX announced "BitMEX Exchange Now Closed" on 2026-09-22. Collection has stopped. The bitmex dataset, from 2025-09-29 to 2026-09-22, remains downloadable through the Python SDK, the REST API and flat-file access. Files for the final hours contain little or no activity.
When using it, note the raw per-instrument quantity units and the liquidation duplicates described in the venue notes, and the missing trades listed above.