Skip to main content

Migrating Storage

The Shift Platform supports two storage backends: file-based (JSON files in dotfile directories) and Convex (cloud or self-hosted). This guide covers migrating between them.

Overview​

  • File-based (SHIFT_STORAGE=file): Each service stores data as JSON files in a dotfile directory (e.g., .yellowpages/services/<id>.json). Git-native -- designed to be committed, diffed, and reviewed in PRs.
  • Convex (SHIFT_STORAGE=convex): All services share a single Convex deployment with table prefixes (yp_, ledger_, pulse_, etc.). The default backend for production and staging.

Migrating File to Convex​

The platform includes a migration script that reads data from local file storage and writes it to Convex:

# Migrate all services
bun run scripts/migrate-to-convex.ts

# Migrate a specific service
bun run scripts/migrate-to-convex.ts --service yellowpages

# Dry run (show what would be migrated without writing)
bun run scripts/migrate-to-convex.ts --dry-run

What Happens During Migration​

  1. The script reads all JSON files from each service's dotfile directory.
  2. For each record, it generates a sid (shift ID) matching the file-based id.
  3. It calls the Convex mutation to insert each record into the appropriate table.
  4. Existing records (matched by sid) are skipped to avoid duplicates.
  5. A summary is printed showing how many records were migrated per service.

Prerequisites​

Before running the migration:

  • Ensure CONVEX_URL or CONVEX_SELF_HOSTED_URL is set.
  • If using self-hosted Convex, ensure CONVEX_SELF_HOSTED_ADMIN_KEY is set.
  • Ensure the Convex schema is deployed (npx convex deploy).
  • The dotfile directories (.yellowpages/, .ledger/, etc.) must exist and contain data.

Example​

# Set up Convex connection
export CONVEX_SELF_HOSTED_URL=http://127.0.0.1:3210
export CONVEX_SELF_HOSTED_ADMIN_KEY=your-admin-key

# Preview what will be migrated
bun run scripts/migrate-to-convex.ts --dry-run

# Run the migration
bun run scripts/migrate-to-convex.ts

Switching Storage Backends​

To switch which backend the platform uses at runtime, change the SHIFT_STORAGE environment variable:

# Use file-based storage
SHIFT_STORAGE=file bun run dev

# Use Convex storage (default)
SHIFT_STORAGE=convex bun run dev

In Kubernetes, update the ConfigMap:

data:
  SHIFT_STORAGE: "convex"  # or "file"

Rollback​

If you need to switch back to file-based storage after migrating to Convex:

  1. Set SHIFT_STORAGE=file in your environment.
  2. The file-based data in the dotfile directories is still intact (migration does not delete it).
  3. Any data created after the migration in Convex will not be present in the file-based store.

There is currently no reverse migration script (Convex to file). If you need to export data from Convex back to files, you can use the Convex dashboard or write a custom script using convexQuery() from @shift/platform-core/convex-client.

sid Bridging​

When data moves between backends, the sid (shift ID) field is the bridge. Every Convex table has a sid field indexed for fast lookup:

  • App-level ID (id): A nanoid(8) string used throughout the application.
  • Convex internal ID (_id): Convex's auto-generated document ID.
  • Shift ID (sid): Stored in every Convex document, equal to the app-level id. Used to look up documents by their application ID.

The ConvexStore implementation translates between id and sid transparently. When migrating, the script sets sid to match the existing file-based id, ensuring continuity.