Flat-File Access with boto3

Use `boto3` when you want direct Python access to the bucket without going through the REST API or the Python SDK download wrapper.

List Objects

list_objects.py

1import os
2import boto3
3
4session = boto3.session.Session(
5    aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
6    aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
7    aws_session_token=os.environ["AWS_SESSION_TOKEN"],
8    region_name="auto",
9)
10
11s3 = session.client(
12    "s3",
13    endpoint_url=os.environ["CHD_ENDPOINT"],
14)
15
16response = s3.list_objects_v2(
17    Bucket="cryptohftdata-processed",
18    Prefix="binance_spot/2025-08-01/20/",
19)
20
21for entry in response.get("Contents", []):
22    print(entry["Key"], entry["Size"])

Download A File

download_file.py

1import boto3
2
3s3 = boto3.client(
4    "s3",
5    endpoint_url="https://e5de051a8af1332b58a11bd5b2e40258.r2.cloudflarestorage.com",
6    aws_access_key_id="<TEMP_ACCESS_KEY_ID>",
7    aws_secret_access_key="<TEMP_SECRET_ACCESS_KEY>",
8    aws_session_token="<TEMP_SESSION_TOKEN>",
9    region_name="auto",
10)
11
12s3.download_file(
13    "cryptohftdata-processed",
14    "binance_spot/2025-08-01/20/BTCUSDT_trades.parquet.zst",
15    "BTCUSDT_trades.parquet.zst",
16)

Security Notes

Do not hardcode the temporary credentials into production code. Load them from environment variables or another short-lived secret source, and refresh them when they expire.