Dataset storage
datashard
AvailableA table format fur datasets an blobs that disnae need a database.
Storage for datasets, recordings an lairge files that hauds its history: records is appendit, snapshots is taen, an a dataset can be read as it stud at ony earlier moment. It taks the place o loose CSV an JSON dumps, hand-managed folders, an databases pit up juist tae haud files.
Wha it is for. Python projects that haud ontae recordings, datasets or analytical data an need versioned, readable storage.
- Status
- Available
- Pairt o
- Open soorce
- Licence
- Open soorce, Apache 2.0
- Tags
- PythonSnapshotsTime travelOpen formatDisk or object storage
Whit it daes
-
Append-only, wi snapshots
Records are appendit an snapshottit, sae a dataset can be read as it stude at a pynt in time.
-
Disk or S3
The same table format wrochts on local disk or in object storage, sae muivin atween thaim is a configuration chynge.
-
Parquet unnerneath
Data is stored as Parquet, so ony tool that reads Parquet can read it.
For documentation, a pilot or an integration, contact us.