lapis-notes/lapis

Fork 0

Restore metadata cache from database or portable backup #95

New issue

Closed

opened 2026-05-23 13:45:06 +00:00 by steve · 0 comments

steve commented

2026-05-23 13:45:06 +00:00

Owner

Fix desktop cold boot metadata restore so existing vaults do not appear to rebuild metadata every startup and restored graph views show data after opening a vault.

Problem

Desktop startup currently appears to repeatedly rebuild or refresh the metadata cache. A cold boot from an existing vault can also leave Graph view empty until metadata work finishes or is manually rebuilt.

The existing spec says MetadataCache.load() should restore from AppDatabase, fall back to legacy snapshots, and only rebuild when needed. The observed behavior suggests the restore path is incomplete for desktop native generated state, missing portable metadata backup support, or both.

Imported/copied vaults also need a way to hydrate generated metadata when the host-owned app database does not exist for the new machine/profile.

Goal

Implement a deterministic metadata restore order:

Load metadata from the per-vault app database when an existing DB snapshot exists.
If the app database is missing or has no metadata snapshot, load a portable backup from .lapis/cache/metadata-cache.json and hydrate the app database from it.
If neither source exists, schedule a first-open metadata rebuild.
Ensure Graph view receives loaded metadata and shows nodes/links on cold boot without requiring manual rebuild.

Scope

Add a portable metadata backup file at .lapis/cache/metadata-cache.json.
Treat the app database as the primary generated-state source.
Treat the portable backup as a vault-local fallback for imported/copied vaults.
Keep metadata backup data rebuildable and safe to delete.
Prevent metadata backup writes from triggering filesystem watch reload loops.
Fix Electron native DB load behavior so probing for missing state does not create an empty SQLite DB file.
Ensure restored Graph views rebuild from metadataCache.loaded.
Add regression tests for DB restore, backup restore, no-state rebuild, and desktop graph cold boot.

Non-goals

Do not store canonical note contents in the metadata backup.
Do not persist search chunks, notebook outputs, notifications, or plugin state in the metadata backup.
Do not move the primary desktop app database into the user vault folder.
Do not add cloud sync, conflict resolution, or multi-device sync semantics.

Implementation Details

Metadata restore order

Update MetadataCache.load() so it:

Opens app.appDatabase.
Calls loadMetadataSnapshot().
If a primary snapshot exists:
- Applies it to fileCache, metadataCache, resolvedLinks, and unresolvedLinks.
- Emits loaded.
- Starts a background stale-entry reconciliation.
If no primary snapshot exists:
- Reads .lapis/cache/metadata-cache.json through app.vault.adapter.
- Validates the backup schema.
- Applies the backup snapshot.
- Hydrates AppDatabase by saving the snapshot and upserting indexed file records derived from the snapshot.
- Emits loaded.
- Starts stale-entry reconciliation.
If neither exists:
- Schedules metadataCache.rebuild() after first open.
- Emits loaded only after that rebuild completes.

Backup schema

Create an internal backup shape:

type MetadataCacheBackupV1 = {
  kind: "lapis.metadata-cache.snapshot";
  schemaVersion: 1;
  appDatabaseSchemaVersion: number;
  createdAt: number;
  updatedAt: number;
  sourceVaultId?: string;
  snapshot: MetadataCacheSnapshot;
};

Rules:

sourceVaultId is informational only and must not block restore for imported/copied vaults.
Unknown schema versions are ignored and treated as no backup.
Invalid JSON is ignored and falls through to rebuild.
The backup file is generated state, not canonical user data.

Snapshot persistence

Replace the current debounced-only metadata persistence path with explicit helpers:

saveSnapshotNow():
- Saves the snapshot to AppDatabase.
- Optionally writes the portable backup if the backup throttle allows it or if forced.
scheduleSnapshotSave():
- Debounces primary DB snapshot writes.
- Throttles portable backup writes to at most once every 30 seconds.
flushSnapshotSave():
- Flushes any pending snapshot save during app/session teardown.
- Force-writes the backup after explicit rebuild.

Continue saving metadata changes after create/modify/delete/rename.

Filesystem watch ignore

Update workspace filesystem watching so changes under .lapis/cache/metadata-cache.json do not cause vault reloads or metadata reprocessing.

Prefer a narrow ignore helper, e.g.:

function isGeneratedMetadataBackupPath(path: string): boolean {
  return normalizePath(path) === ".lapis/cache/metadata-cache.json";
}

Use it in native and polling file-watch event handling before reloadVault().

Electron native DB behavior

Update Electron main-process generated-state helpers:

Split "state database path exists" from "open or create state database".
desktop_db_load_state must return null without creating vault-state/<vaultId>.sqlite3 when no SQLite DB or legacy JSON state exists.
If legacy JSON exists, load it and migrate it into SQLite only when saving or explicitly migrating.
Existing SQLite DB state continues to load normally.
Existing legacy JSON migration behavior remains compatible.

Graph cold boot behavior

No public graph API change.

Verify that:

Restored graph leaves subscribe to metadataCache.loaded.
Graph rebuilds after DB snapshot restore.
Graph rebuilds after backup hydration.
Graph does not remain empty when metadata already exists.

If needed, add a graph view guard so the first requestAnimationFrame() rebuild does not permanently win with an empty cache before metadataCache.loaded.

Acceptance Criteria

Desktop relaunch with an existing app database loads metadata without scheduling a full rebuild.
Desktop relaunch with an existing app database shows graph nodes/links after metadataCache.loaded.
Removing the app database but keeping .lapis/cache/metadata-cache.json hydrates metadata and recreates app database state.
Removing both database and backup schedules a first-open metadata rebuild.
Backup writes do not trigger a filesystem reload loop.
Imported/copied vaults can restore metadata from .lapis/cache/metadata-cache.json even when the host user-data DB does not exist.
Invalid backup files are ignored safely and do not block app startup.
Metadata backup is documented as rebuildable generated state.

Suggested Files or Specs To Inspect

packages/api/src/lib/cache.svelte.ts
packages/api/src/lib/storage/app-database.ts
packages/api/src/lib/storage/desktop-native.ts
packages/api/src/lib/storage/vault-session.ts
packages/workspace/src/lib/components/app/App.svelte
packages/workspace/src/lib/hooks/watch-vault.svelte.ts
packages/plugins/plugin-graph/src/graph-view.ts
packages/plugins/plugin-graph/src/graph-data.ts
packages/desktop-electron/src-electron/main.ts
packages/desktop-electron/src-electron/preload.ts
packages/desktop-electron/e2e/metadata-rebuild.spec.ts
spec/src/30-cross-package-contracts/storage-metadata-search.md
spec/src/20-packages/api/index.md
spec/src/20-packages/workspace/index.md
spec/src/20-packages/desktop-electron/index.md
spec/src/20-packages/plugins/graph/index.md

Test Plan

API unit tests

MetadataCache.load() applies an app-database snapshot and does not rebuild.
MetadataCache.load() restores from .lapis/cache/metadata-cache.json when DB snapshot is absent.
Backup restore saves the snapshot back into AppDatabase.
Missing DB and missing backup schedules a rebuild.
Invalid backup JSON/schema falls through to rebuild.
Stale reconciliation removes deleted-file entries and reprocesses changed files.
Snapshot save helpers debounce DB persistence and throttle portable backup writes.
Backup path changes are ignored by file-watch handling.

Desktop/native tests

desktop_db_load_state does not create a SQLite file when no state exists.
Existing SQLite state loads correctly.
Existing legacy JSON state still migrates safely.
Native state save still creates vault-state/<vaultId>.sqlite3.

Graph/desktop e2e

First launch fixture vault, wait for metadata, open graph, assert non-zero nodes and links.
Relaunch with same userDataDir, assert graph restores with nodes/links and no full rebuild progress.
Delete native DB, keep .lapis/cache/metadata-cache.json, relaunch, assert graph hydrates and DB state is recreated.
Delete both DB and backup, relaunch, assert first-open rebuild runs and graph eventually gets links.

Validation Commands

pnpm --filter @lapis-notes/api check
pnpm --filter @lapis-notes/workspace check
pnpm --filter @lapis-notes/desktop-electron check
pnpm --filter @lapis-notes/graph check
make spec-lint
mdbook build spec
pnpm test:smoke

Follow-up Tasks

Consider a broader portable generated-state backup after this lands, covering search documents and notebook outputs separately.
Consider user-facing recovery UI that shows whether metadata was restored from DB, backup, or rebuilt.
Consider a maintenance command to delete and regenerate .lapis/cache/metadata-cache.json.

Implementation Summary

Implemented deterministic metadata restore from app database, portable .lapis/cache/metadata-cache.json backup hydration, legacy fallback, first-open rebuild, throttled/forced backup persistence, generated-backup watch ignores, Electron no-create state load probing, and focused regression coverage.

Fix desktop cold boot metadata restore so existing vaults do not appear to rebuild metadata every startup and restored graph views show data after opening a vault. ## Problem Desktop startup currently appears to repeatedly rebuild or refresh the metadata cache. A cold boot from an existing vault can also leave Graph view empty until metadata work finishes or is manually rebuilt. The existing spec says `MetadataCache.load()` should restore from `AppDatabase`, fall back to legacy snapshots, and only rebuild when needed. The observed behavior suggests the restore path is incomplete for desktop native generated state, missing portable metadata backup support, or both. Imported/copied vaults also need a way to hydrate generated metadata when the host-owned app database does not exist for the new machine/profile. ## Goal Implement a deterministic metadata restore order: 1. Load metadata from the per-vault app database when an existing DB snapshot exists. 2. If the app database is missing or has no metadata snapshot, load a portable backup from `.lapis/cache/metadata-cache.json` and hydrate the app database from it. 3. If neither source exists, schedule a first-open metadata rebuild. 4. Ensure Graph view receives loaded metadata and shows nodes/links on cold boot without requiring manual rebuild. ## Scope - Add a portable metadata backup file at `.lapis/cache/metadata-cache.json`. - Treat the app database as the primary generated-state source. - Treat the portable backup as a vault-local fallback for imported/copied vaults. - Keep metadata backup data rebuildable and safe to delete. - Prevent metadata backup writes from triggering filesystem watch reload loops. - Fix Electron native DB load behavior so probing for missing state does not create an empty SQLite DB file. - Ensure restored Graph views rebuild from `metadataCache.loaded`. - Add regression tests for DB restore, backup restore, no-state rebuild, and desktop graph cold boot. ## Non-goals - Do not store canonical note contents in the metadata backup. - Do not persist search chunks, notebook outputs, notifications, or plugin state in the metadata backup. - Do not move the primary desktop app database into the user vault folder. - Do not add cloud sync, conflict resolution, or multi-device sync semantics. ## Implementation Details ### Metadata restore order Update `MetadataCache.load()` so it: - Opens `app.appDatabase`. - Calls `loadMetadataSnapshot()`. - If a primary snapshot exists: - Applies it to `fileCache`, `metadataCache`, `resolvedLinks`, and `unresolvedLinks`. - Emits `loaded`. - Starts a background stale-entry reconciliation. - If no primary snapshot exists: - Reads `.lapis/cache/metadata-cache.json` through `app.vault.adapter`. - Validates the backup schema. - Applies the backup snapshot. - Hydrates `AppDatabase` by saving the snapshot and upserting indexed file records derived from the snapshot. - Emits `loaded`. - Starts stale-entry reconciliation. - If neither exists: - Schedules `metadataCache.rebuild()` after first open. - Emits `loaded` only after that rebuild completes. ### Backup schema Create an internal backup shape: ```ts type MetadataCacheBackupV1 = { kind: "lapis.metadata-cache.snapshot"; schemaVersion: 1; appDatabaseSchemaVersion: number; createdAt: number; updatedAt: number; sourceVaultId?: string; snapshot: MetadataCacheSnapshot; }; ``` Rules: - `sourceVaultId` is informational only and must not block restore for imported/copied vaults. - Unknown schema versions are ignored and treated as no backup. - Invalid JSON is ignored and falls through to rebuild. - The backup file is generated state, not canonical user data. ### Snapshot persistence Replace the current debounced-only metadata persistence path with explicit helpers: - `saveSnapshotNow()`: - Saves the snapshot to `AppDatabase`. - Optionally writes the portable backup if the backup throttle allows it or if forced. - `scheduleSnapshotSave()`: - Debounces primary DB snapshot writes. - Throttles portable backup writes to at most once every 30 seconds. - `flushSnapshotSave()`: - Flushes any pending snapshot save during app/session teardown. - Force-writes the backup after explicit rebuild. Continue saving metadata changes after create/modify/delete/rename. ### Filesystem watch ignore Update workspace filesystem watching so changes under `.lapis/cache/metadata-cache.json` do not cause vault reloads or metadata reprocessing. Prefer a narrow ignore helper, e.g.: ```ts function isGeneratedMetadataBackupPath(path: string): boolean { return normalizePath(path) === ".lapis/cache/metadata-cache.json"; } ``` Use it in native and polling file-watch event handling before `reloadVault()`. ### Electron native DB behavior Update Electron main-process generated-state helpers: - Split "state database path exists" from "open or create state database". - `desktop_db_load_state` must return `null` without creating `vault-state/<vaultId>.sqlite3` when no SQLite DB or legacy JSON state exists. - If legacy JSON exists, load it and migrate it into SQLite only when saving or explicitly migrating. - Existing SQLite DB state continues to load normally. - Existing legacy JSON migration behavior remains compatible. ### Graph cold boot behavior No public graph API change. Verify that: - Restored graph leaves subscribe to `metadataCache.loaded`. - Graph rebuilds after DB snapshot restore. - Graph rebuilds after backup hydration. - Graph does not remain empty when metadata already exists. If needed, add a graph view guard so the first `requestAnimationFrame()` rebuild does not permanently win with an empty cache before `metadataCache.loaded`. ## Acceptance Criteria - Desktop relaunch with an existing app database loads metadata without scheduling a full rebuild. - Desktop relaunch with an existing app database shows graph nodes/links after `metadataCache.loaded`. - Removing the app database but keeping `.lapis/cache/metadata-cache.json` hydrates metadata and recreates app database state. - Removing both database and backup schedules a first-open metadata rebuild. - Backup writes do not trigger a filesystem reload loop. - Imported/copied vaults can restore metadata from `.lapis/cache/metadata-cache.json` even when the host user-data DB does not exist. - Invalid backup files are ignored safely and do not block app startup. - Metadata backup is documented as rebuildable generated state. ## Suggested Files or Specs To Inspect - `packages/api/src/lib/cache.svelte.ts` - `packages/api/src/lib/storage/app-database.ts` - `packages/api/src/lib/storage/desktop-native.ts` - `packages/api/src/lib/storage/vault-session.ts` - `packages/workspace/src/lib/components/app/App.svelte` - `packages/workspace/src/lib/hooks/watch-vault.svelte.ts` - `packages/plugins/plugin-graph/src/graph-view.ts` - `packages/plugins/plugin-graph/src/graph-data.ts` - `packages/desktop-electron/src-electron/main.ts` - `packages/desktop-electron/src-electron/preload.ts` - `packages/desktop-electron/e2e/metadata-rebuild.spec.ts` - `spec/src/30-cross-package-contracts/storage-metadata-search.md` - `spec/src/20-packages/api/index.md` - `spec/src/20-packages/workspace/index.md` - `spec/src/20-packages/desktop-electron/index.md` - `spec/src/20-packages/plugins/graph/index.md` ## Test Plan ### API unit tests - `MetadataCache.load()` applies an app-database snapshot and does not rebuild. - `MetadataCache.load()` restores from `.lapis/cache/metadata-cache.json` when DB snapshot is absent. - Backup restore saves the snapshot back into `AppDatabase`. - Missing DB and missing backup schedules a rebuild. - Invalid backup JSON/schema falls through to rebuild. - Stale reconciliation removes deleted-file entries and reprocesses changed files. - Snapshot save helpers debounce DB persistence and throttle portable backup writes. - Backup path changes are ignored by file-watch handling. ### Desktop/native tests - `desktop_db_load_state` does not create a SQLite file when no state exists. - Existing SQLite state loads correctly. - Existing legacy JSON state still migrates safely. - Native state save still creates `vault-state/<vaultId>.sqlite3`. ### Graph/desktop e2e - First launch fixture vault, wait for metadata, open graph, assert non-zero nodes and links. - Relaunch with same `userDataDir`, assert graph restores with nodes/links and no full rebuild progress. - Delete native DB, keep `.lapis/cache/metadata-cache.json`, relaunch, assert graph hydrates and DB state is recreated. - Delete both DB and backup, relaunch, assert first-open rebuild runs and graph eventually gets links. ## Validation Commands - `pnpm --filter @lapis-notes/api check` - `pnpm --filter @lapis-notes/workspace check` - `pnpm --filter @lapis-notes/desktop-electron check` - `pnpm --filter @lapis-notes/graph check` - `make spec-lint` - `mdbook build spec` - `pnpm test:smoke` ## Follow-up Tasks - Consider a broader portable generated-state backup after this lands, covering search documents and notebook outputs separately. - Consider user-facing recovery UI that shows whether metadata was restored from DB, backup, or rebuilt. - Consider a maintenance command to delete and regenerate `.lapis/cache/metadata-cache.json`. ## Implementation Summary Implemented deterministic metadata restore from app database, portable .lapis/cache/metadata-cache.json backup hydration, legacy fallback, first-open rebuild, throttled/forced backup persistence, generated-backup watch ignores, Electron no-create state load probing, and focused regression coverage.

steve added this to the Complete Daily-Use App milestone

2026-05-23 13:45:06 +00:00

steve

2026-05-23 13:45:06 +00:00