Data retention¶

PhyCloud keeps every event in its hot store (PostgreSQL or SQLite) until a retention policy says otherwise. One robot streaming at 5 Hz writes about 1 GB a day, so most fleets need a policy.

A retention policy belongs to one tenant. It says how long events stay in the hot store and what happens when they age out:

A tenant without a policy keeps every event. That is the default, so upgrading PhyCloud never deletes data.

This page explains how retention works. Operators should also read the operator guide. Endpoints are listed in the API reference.

The policy¶

An event ages out when it is older than hot_retention, or when keeping it would exceed a cap. Age is measured from ingested_at, the time PhyCloud received the event. Robot clocks do not affect it.

Retention always removes the oldest events first, as a contiguous run of sequence numbers. It never leaves holes in the middle of the chain.

Risky changes wait¶

A change that would make the next run delete more data is staged rather than applied:

• adding a policy to a tenant that has none;
• shortening hot_retention;
• adding a cap, or lowering one;
• switching from archive_then_delete to delete.

A staged change takes effect after a delay, 72 hours by default, or as soon as a super-admin confirms it. It can be cancelled until then. Lengthening retention, removing a cap, switching to archive_then_delete, changing the lake and removing the policy all apply at once.

Every change, staged or applied, is written to the audit log.

Your hash chain stays verifiable¶

Each tenant's events form a hash chain: every event's previous_hash is the content_hash of the event before it. When retention removes the oldest events, PhyCloud records a retention checkpoint: the last removed sequence number and its hash.

GET /api/v1/verify starts at the checkpoint. The first remaining event must link to the checkpoint hash, so the chain still verifies, and the response shows where it starts:

{
  "verified": true,
  "events": 41022,
  "chain_start": 2588101,
  "checkpoint": { "sequence_number": 2588100, "content_hash": "9f2c…", "retained_events": 0 }
}

For an archived tenant, GET /api/v1/verify/archive proves the whole chain: it reads every archived segment from the lake, checks its digest and every hash and link, and then walks the hot store from the checkpoint. See Verifying the full chain.

Legal holds¶

A legal hold stops retention from deleting events, whatever the policy says. A hold covers a whole tenant or one source.

• A tenant-level hold stops retention runs for that tenant.
• A source-level hold keeps that source's events. The rest of the tenant's events age out as usual.
• Under archive_then_delete, held events are still archived, so the lake has a complete chain.

When a hold is released, the next run removes the events it was keeping. Under an archive policy, it removes only events that were archived.

Evidence lockers keep their own copy of each event when they are sealed, so sealed evidence survives retention without a hold.

What you can see as a tenant¶

Any key with events:read can read its own tenant's retention state:

# Your policy, and any staged change
phyware cloud retention get

# Events, bytes, oldest event, checkpoint and archive totals
phyware cloud storage

# Prove the full chain across the lake and the hot store
phyware cloud retention verify-archive

Setting policies, running retention and managing holds need a super-admin key with admin:tenants.

Verifying the full chain¶

phyware cloud retention verify-archive (or GET /api/v1/verify/archive) returns:

Problem kinds:

The verifier reads every archived segment, so it can take a while for a large archive.

Archive format¶

Archived events are stored as NDJSON compressed with zstd, one segment per source per UTC day, in Hive-style partitions that DuckDB, Spark and Athena read directly:

<prefix>/tenant=acme/source=robot-1/date=2026-10-27/seg-00000000000000000001-00000000000000004321.ndjson.zst
<prefix>/tenant=acme/source=robot-1/date=2026-10-27/seg-00000000000000000001-00000000000000004321.manifest.json

Each line is one stored event, including sequence_number, content_hash, previous_hash and the original event_data text. You can read a segment with standard tools:

zstdcat seg-….ndjson.zst | jq -c '{sequence_number, event_type, captured_at}'

The manifest next to each segment records its sequence range, first and last hash, event count, size and SHA-256. PhyCloud keeps the same manifest.

Further reading¶

• Operator guide: configuration, lakes, alerts and the runbook.
• API reference.
• phyware cloud retention, cloud holds and cloud storage.
• Design: design/enhancements/phycloud_data_retention.md.