Retention: operator guide¶

This guide covers configuring PhyCloud for data retention, setting up data lakes, and handling paused runs. Read the overview first.

Configuration¶

Generate a secrets key once and keep it with your other service secrets:

openssl rand -base64 32

Without PHYCLOUD_SECRETS_KEY, policies with lake credentials are rejected. In dev mode (PHYWARE_DEV_MODE=1) a fixed development key is used instead; never rely on it in production.

Setting a policy¶

export PHYCLOUD_API_KEY=phyc_…   # a super-admin key with admin:tenants

# Delete events older than 30 days
phyware cloud retention set --tenant acme --hot-retention 30d --action delete

# See what the first run would do
phyware cloud retention run --tenant acme --dry-run

# Apply the staged change now instead of waiting 72 hours
phyware cloud retention confirm --tenant acme

The same through the API:

curl -X PUT "$PHYCLOUD_URL/api/v1/admin/tenants/acme/retention" \
  -H "Authorization: Bearer $PHYCLOUD_API_KEY" -H "Content-Type: application/json" \
  -d '{"hot_retention": "30d", "action": "delete"}'

A 202 Accepted response with "status": "pending" means the change was staged. 200 OK with "status": "applied" means it is active.

Data lakes¶

S3-compatible storage¶

AWS S3, MinIO, Cloudflare R2 and other S3-compatible stores work:

{
  "kind": "s3",
  "bucket": "acme-telemetry",
  "prefix": "phycloud",
  "region": "us-east-1",
  "credentials": { "access_key_id": "AKIA…", "secret_access_key": "…" }
}

phyware cloud retention set --tenant acme --hot-retention 7d \
  --action archive-then-delete --lake @acme-lake.json

The credentials are encrypted before they are stored and are never returned by the API. To change other policy fields later, send the same lake without credentials; the stored ones are kept as long as the bucket and endpoint are unchanged.

The key needs s3:PutObject and s3:GetObject on the prefix. PhyCloud never deletes objects from the lake. Use bucket lifecycle rules if archived data should eventually expire, and consider object lock for tamper resistance.

Filesystem¶

For development and on-prem installs, a directory under PHYCLOUD_LAKE_FS_ROOT:

{ "kind": "filesystem", "path": "acme" }

The path must be relative and cannot contain ... Each object is written to a temporary file, synced to disk, and renamed into place.

How a run works¶

Each scheduled run handles every tenant that has a policy, one at a time:

1. If a staged change is due, the run applies it and stops. The next run enforces it.
2. A tenant-level legal hold stops the run (held).
3. The run works out which events have aged out, then retires them in batches, oldest first. Under archive_then_delete, each batch is written to the lake, read back, and checked against its SHA-256 before anything is deleted.
4. Each batch's delete and the checkpoint update commit together, so a crash loses at most the batch in progress. The next run continues where it stopped. Re-running a batch is harmless.
5. Events kept by a released hold are removed.

Caps never target events received in the last five minutes, so an insert in flight cannot end up behind the checkpoint.

Run records are kept for every run, including dry runs:

phyware cloud retention runs --tenant acme --limit 10

Every run and every policy or hold change is also written to the audit log (retention_run, retention_policy_*, legal_hold_*).

Paused runs and alerts¶

If a tenant's policy is archive_then_delete and the lake is unreachable, the credentials are wrong, or a write cannot be verified, nothing is deleted. The run is recorded as paused with the error, and:

• PhyCloud logs at error level with alert="retention_paused";
• an audit record is written;
• if PHYCLOUD_ALERT_WEBHOOK_URL is set, PhyCloud posts:

{
  "alert": "retention_paused",
  "severity": "critical",
  "tenant_id": "acme",
  "run_id": "1b6f…",
  "error": "lake unavailable: …",
  "message": "Retention for tenant acme paused: nothing more was deleted. …"
}

• /health counts the tenant in retention.paused_tenants.

The next scheduled run tries again. Retention never falls back to deleting.

To recover:

1. Read the error: phyware cloud retention runs --tenant acme --limit 1.
2. Fix the lake: credentials, bucket permissions, endpoint or disk space.
3. Run it: phyware cloud retention run --tenant acme.

The hot store keeps growing while a tenant is paused, so treat the alert as urgent.

Monitoring¶

GET /health includes a retention block. None of it scans the events table, so it is safe to poll:

"retention": {
  "database_bytes": 52344832000,
  "tenants_with_policy": 12,
  "paused_tenants": 0,
  "held_tenants": 1
}

For per-tenant usage, use phyware cloud storage --all or GET /api/v1/admin/storage. These scan each tenant's events, so run them on demand, not from a frequent probe.

PostgreSQL notes¶

Retention deletes rows in batches by (tenant_id, sequence_number), which uses an existing unique index. PostgreSQL reclaims the space through autovacuum. On busy installs, vacuum events more often:

ALTER TABLE events SET (autovacuum_vacuum_scale_factor = 0.02);

Deleted space is reused by new rows, but the table file does not shrink. Run VACUUM FULL events in a maintenance window if you need the disk back.

We chose batched deletes over time-partitioning events; the design document explains why.

Running more than one PhyCloud replica¶

Every replica runs the scheduler. Runs are safe to overlap across replicas: deletes are idempotent, the checkpoint only moves forward, and re-archiving a batch writes the same objects. To avoid duplicate work, set PHYCLOUD_RETENTION_INTERVAL_SECS=0 on all replicas but one.