Skip to content
Language: EN
Menu

Dataset storage

datashard

Available

A table format for datasets and blobs that does not need a database.

Documentation Contact us

Storage for datasets, recordings and large files that keeps its history. Records are appended and snapshots taken, and a dataset can be read as it stood at any earlier moment. It replaces loose CSV and JSON dumps, hand-managed folders and databases stood up only to hold files.

Who it is for. Python projects that persist recordings, datasets or analytical data and need versioned, readable storage.

Status
Available
Part of
Open source
Licence
Open source, Apache 2.0
Tags
PythonSnapshotsTime travelOpen formatDisk or object storage

What it does

  • Append-only, with snapshots

    Because records are appended and snapshotted, you can read a dataset as it stood at a point in time.

  • Disk or S3

    The table format works on local disk and in object storage, and moving between the two is a configuration change.

  • Parquet underneath

    Data is stored as Parquet, which any tool that reads Parquet can open.

For documentation, a pilot or an integration, contact us.