lapis-notes/lapis

Fork 0

Evaluate AppDatabase-backed Bases query execution for large vault tables instead of renderer-only... #4

New issue

Closed

opened 2026-05-20 18:54:28 +00:00 by steve · 0 comments

steve commented

2026-05-20 18:54:28 +00:00

Owner

Evaluate and implement an AppDatabase-backed query path for Bases views over large vaults instead of relying on renderer-only metadata iteration.

Problem

Bases views currently build their query input in the renderer from app.metadataCache.fileCache. Each open QueryController turns the full metadata cache into a VaultRecord[], asks columnsFor(...) to scan that row set, creates a PEaQL BasesTable, and executes generated SQL synchronously over the renderer-owned array.

That preserves current .base and .bases semantics, but it scales poorly for large vault tables because the renderer repeatedly performs vault-wide row construction, column inference, filtering, sorting, grouping, and formula evaluation. The app already persists generated file, metadata, link, tag, and property state behind AppDatabase; Bases should evaluate whether that generated-state layer can provide a narrower, off-renderer query path without leaking plugin internals into packages/api.

Goal

Deliver a correctness-preserving AppDatabase-backed Bases query path that reduces renderer work for large vault tables while keeping the existing Bases document model, PEaQL-facing semantics, custom view data shape, and markdown-hosted read-only surfaces intact.

The preferred result is a staged implementation: use AppDatabase to load or narrow candidate rows, keep PEaQL ownership in plugin-bases, and fall back to the current in-memory evaluator whenever a query shape cannot be safely lowered.

Scope

Add an AppDatabase read/query contract for indexed files, metadata, frontmatter properties, tags, links, and file records.
Implement that contract for memory and IndexedDB fallback behavior.
Implement SQLite-backed behavior for browser OPFS sessions through SqliteWasmAppDatabaseCore, the worker proxy, and the coordinated browser proxy.
Extend desktop native behavior if the final design requires host-side metadata queries instead of serialized-state-only reads.
Add a Bases query-source abstraction that converts AppDatabase rows back into the existing VaultRecord/BasesEntry shape.
Convert QueryController reloads from synchronous derived array execution to async, generation-tokened reloads that cannot overwrite newer document or metadata changes with stale database responses.
Preserve current Bases behavior for table, card, list, custom registered views, markdown embeds, and fenced base/bases blocks.
Update package-local and rendered specs for the final behavior and any intentional fallback limits.

Non-goals

Do not change the .base or .bases file format.
Do not store canonical note bytes in AppDatabase.
Do not let plugin-bases open SQLite, native stores, or browser storage directly.
Do not move PEaQL or Bases-specific query semantics into packages/api.
Do not replace PEaQL syntax or reimplement the complete PEaQL evaluator in SQL.
Do not solve the separate map-view issue #3.
Do not include the table column drag-and-drop migration tracked by issue #5.

Current State

QueryController.data is derived from app.metadataCache.fileCache and maps every cached file into a VaultRecord.
columnsFor(...) infers frontmatter columns by scanning the current renderer row set, while also materializing explicit note.* properties declared in a Bases document.
QueryController.db creates a PEaQL Context over new BasesTable("bases").withPropertyColumns(...).data(() => this.data).
QueryController.getRecords(...) calls this.db.execute(qs) synchronously and converts the result into the existing BasesEntry/BasesQueryResult pipeline.
AppDatabase already persists files, metadata, links, tags, and properties, but the public contract currently exposes write operations for indexed metadata and does not expose a Bases-suitable read/query surface.
Browser SQLite already runs behind an app-database worker and has coordinated secondary-tab proxying.
NativeDesktopAppDatabase currently persists serialized generated state and mirrors search documents into native search tables, but it does not expose a dedicated metadata query IPC surface.
Metadata invalidation already lives in QueryController and uses dependency tracking over document filters, view filters, visible/sorted properties, group-by state, formulas, and direct file-reference neighborhoods.

Implementation Details

1. Characterize current behavior

Add focused regression coverage or a benchmark fixture that builds a synthetic large metadata set and records the current renderer-only behavior.
Cover query shapes that must remain stable: file fields, inFolder, explicit note.* fields, inferred frontmatter fields, tags, links, sort, limit, grouping, formulas, and search-within-view.
Use the fixture as both a baseline and a parity guard for the AppDatabase-backed path.

2. Design the AppDatabase metadata read contract

Add serializable types in packages/api for a plugin-neutral indexed metadata query surface. The contract should expose data that any structured metadata consumer could use, not Bases-specific PEaQL concepts.

Suggested shape:

type AppDatabaseIndexedMetadataQuery = {
  fileExtensions?: string[];
  pathPrefixes?: string[];
  propertyNames?: string[];
  propertyFilters?: AppDatabasePropertyFilter[];
  tags?: string[];
  linkedPaths?: string[];
  sort?: AppDatabaseIndexedMetadataSort[];
  limit?: number;
};

type AppDatabaseIndexedMetadataRow = {
  file: AppDatabaseFileRecord;
  metadata: AppDatabaseMetadataRecord | null;
  properties: AppDatabasePropertyRecord[];
  tags: AppDatabaseTagRecord[];
  links: AppDatabaseLinkRecord[];
};

The exact API can differ, but it should:

Return enough file stat, metadata, property, tag, and link data to reconstruct a VaultRecord for existing Bases rendering.
Allow callers to request all indexed rows when safe lowering is unavailable.
Allow simple candidate narrowing by extension, folder/path prefix, property existence, property equality/comparison, tags, links, sort, and limit.
Preserve memory and IndexedDB parity with SQLite-backed results.
Avoid importing plugin-bases, PEaQL, or Svelte concepts into packages/api.

3. Implement AppDatabase backends

Add the new method or methods to the AppDatabase interface.
Implement MemoryAppDatabase by reading the existing in-memory files, metadata, properties, tags, and links maps.
Let IndexedDbAppDatabase inherit the memory behavior after opening/restoring state, unless a storage-specific read path is necessary.
Implement SQL-backed filtering in SqliteWasmAppDatabaseCore using the existing files, metadata, properties, tags, and links tables.
Add any missing SQLite indexes needed by the selected filter shape, such as (path, name), (name, value_json), or file extension/path indexes, without overfitting to one Bases view.
Route the new method through sqlite-wasm-app-database.worker.ts, SqliteWasmAppDatabase, and BrowserCoordinatedAppDatabase so browser worker and secondary-tab behavior stays consistent.
Decide whether NativeDesktopAppDatabase can answer from restored serialized state or needs a native IPC query. If native IPC is required, add a narrow host command that returns the same serializable row shape.

4. Add a Bases query-source abstraction

Introduce a small boundary inside packages/plugins/plugin-bases, for example:

interface BasesQuerySource {
  loadRows(request: BasesQuerySourceRequest): Promise<VaultRecord[]>;
}

Responsibilities:

Translate the current Bases document, active view, filters, visible properties, sort, group-by, formulas, and search state into a conservative AppDatabase query request.
Reconstruct TFile or TFile-compatible file objects from returned file records by resolving paths through app.vault.getFileByPath(...) when available and using the existing fallback TFile construction otherwise.
Rebuild CachedMetadata/frontmatter-compatible row data from returned metadata and property records.
Preserve checksum behavior so BasesEntry identity and custom views continue to work.
Fall back to loading all indexed rows and running PEaQL in memory for unsupported query shapes.

5. Convert QueryController reloads to async

Replace the synchronous data derived value and direct this.db.execute(...) reload path with an explicit async reload pipeline.
Track a monotonically increasing reload generation or cancellation token.
Ignore stale database responses when a newer metadata change, document edit, view switch, or filter edit has already requested another reload.
Keep initial render, metadata loaded, metadata changed, and metadata deleted events working.
Continue to call view.onDataUpdated() only when the active view receives a fresh BasesQueryResult.
Ensure errors surface as the existing empty/error result behavior rather than crashing the view.

6. Lower safe constraints first

Start with conservative candidate narrowing rather than full SQL execution of every PEaQL expression.

Good first candidates:

File extension and markdown-only row selection.
file.path, file.name, file.folder, and inFolder path-prefix filters.
Property existence and simple property equality/comparison where the value can be serialized safely.
Tag filters backed by the indexed tags table.
Direct link/embed candidate filters backed by the indexed links table.
Sort and limit when the sorted field maps directly to a stored file or property record.

Fallback cases should continue through the current PEaQL path:

Custom formulas.
Unsupported PEaQL functions.
Complex expressions that combine derived values in ways the AppDatabase contract cannot represent safely.
Query shapes where SQLite and memory semantics would diverge.

7. Preserve column inference and explicit note properties

Keep explicit note.* properties declared in the Bases document even when the current metadata rows do not contain that frontmatter key.
Continue to infer observed frontmatter columns from returned candidate rows.
Make sure frontMatterTypesForColumns(...) still supplies PEaQL property column types for both explicit and inferred note fields.
Add regression tests for generated plugin-owned .base documents that reference structured fields such as note.status before any note currently carries that property.

8. Update invalidation behavior

Reuse the existing shouldReloadForMetadataChange(...) dependency logic.
Make async reload scheduling compatible with metadata loaded, changed, and deleted events.
Ensure AppDatabase-backed reads see the freshly indexed metadata after MetadataCache writes upsertIndexedFile(...), deleteIndexedFile(...), or renameIndexedFile(...).
If write/read ordering is not guaranteed for a backend, await or queue the AppDatabase write before asking Bases to reload.

9. Document the result

Update:

spec/src/20-packages/plugins/bases/index.md
spec/src/30-cross-package-contracts/storage-metadata-search.md
packages/plugins/plugin-bases/spec.md

Document:

Which work moves behind AppDatabase.
Which query shapes are lowered.
Which query shapes intentionally fall back to in-memory PEaQL.
How memory, browser SQLite, coordinated tabs, and desktop native behavior stay equivalent.

Acceptance Criteria

Bases views can load candidate rows through AppDatabase instead of always rebuilding a full renderer-owned VaultRecord[] from metadataCache.fileCache.
Query results match the current renderer-only path for covered filters, sort, limit, grouping, formulas, visible columns, and search-within-view.
Unsupported query shapes fall back to the existing PEaQL behavior without user-visible result divergence.
Explicit note.* fields declared in a Bases document continue to compile and render even when no current row has that frontmatter key.
Browser OPFS SQLite sessions execute the new AppDatabase read path through the existing worker/proxy model.
Coordinated secondary browser tabs delegate the new AppDatabase query method to the owning tab and recover consistently after owner promotion.
Desktop behavior remains correct, either through serialized-state reads or a narrow native metadata-query bridge.
Metadata create, modify, delete, rename, and metadata-cache load events refresh open Bases views without stale async responses winning over newer state.
Custom Bases view registrations receive the same BasesQueryResult shape they receive today.
Markdown file embeds and fenced base/bases code blocks continue to render in read-only mode.
Large synthetic vault tests or benchmarks show reduced renderer-side row construction/query work for lowered query shapes.
Specs describe the final AppDatabase-backed behavior and intentional fallback limits.

Suggested Files or Specs To Inspect

packages/api/src/lib/storage/app-database.ts
packages/api/src/lib/storage/sqlite-wasm-app-database-core.ts
packages/api/src/lib/storage/sqlite-wasm-app-database.ts
packages/api/src/lib/storage/sqlite-wasm-app-database.worker.ts
packages/api/src/lib/storage/browser-coordinated-app-database.ts
packages/api/src/lib/storage/desktop-native.ts
packages/api/src/lib/cache.svelte.ts
packages/plugins/plugin-bases/src/bases-view/bases.svelte.ts
packages/plugins/plugin-bases/src/bases-view/db.ts
packages/plugins/plugin-bases/src/bases-view/columns.ts
packages/plugins/plugin-bases/src/bases-view/filter-parser.ts
packages/plugins/plugin-bases/src/bases-view/file-fields-core.ts
packages/plugins/plugin-bases/spec.md
spec/src/20-packages/plugins/bases/index.md
spec/src/30-cross-package-contracts/storage-metadata-search.md

Test Plan

AppDatabase tests

Memory implementation returns indexed metadata rows for inserted file, metadata, property, tag, and link records.
SQL implementation returns the same rows as memory for the same input records.
Property existence, equality, comparison, tag, link, folder/path, sort, and limit filters match across memory and SQL.
Delete and rename operations update query results.
Browser worker/proxy and coordinated database paths expose the new method.
Desktop native or serialized-state reads return the same serializable row shape.

Bases tests

Existing filters, formulas, sort, limit, grouping, and view search produce the same result set as the renderer-only path.
Explicit note.* properties are materialized without observed frontmatter values.
QueryController ignores stale async reload results.
Metadata loaded, changed, and deleted events trigger correct reloads.
Custom view registrations receive updated BasesQueryResult data.
Markdown-hosted read-only embeds and fenced Bases blocks keep rendering through the same query pipeline.

Manual or smoke checks

Open a large vault table in the workspace app and verify the UI remains responsive while metadata-backed Bases rows load.
Run the same large-vault check in Electron with the native app database path.
Start interactive verification with syntax highlighting disabled first so editor highlighting issues do not mask Bases behavior.

Validation Commands

pnpm --filter @lapis-notes/api test:run src/lib/__tests__/app-database.test.ts
pnpm --filter @lapis-notes/bases test -- src/bases-view
pnpm --filter @lapis-notes/api check
pnpm --filter @lapis-notes/bases check
pnpm check
make spec-lint
mdbook build spec
pnpm test:smoke

Completion Notes

When implementation and validation are complete, close this issue through the backlog helper with a concise implementation summary and commit with Fixes #4 in the commit message body.

Evaluate and implement an AppDatabase-backed query path for Bases views over large vaults instead of relying on renderer-only metadata iteration. ## Problem Bases views currently build their query input in the renderer from `app.metadataCache.fileCache`. Each open `QueryController` turns the full metadata cache into a `VaultRecord[]`, asks `columnsFor(...)` to scan that row set, creates a PEaQL `BasesTable`, and executes generated SQL synchronously over the renderer-owned array. That preserves current `.base` and `.bases` semantics, but it scales poorly for large vault tables because the renderer repeatedly performs vault-wide row construction, column inference, filtering, sorting, grouping, and formula evaluation. The app already persists generated file, metadata, link, tag, and property state behind `AppDatabase`; Bases should evaluate whether that generated-state layer can provide a narrower, off-renderer query path without leaking plugin internals into `packages/api`. ## Goal Deliver a correctness-preserving AppDatabase-backed Bases query path that reduces renderer work for large vault tables while keeping the existing Bases document model, PEaQL-facing semantics, custom view data shape, and markdown-hosted read-only surfaces intact. The preferred result is a staged implementation: use `AppDatabase` to load or narrow candidate rows, keep PEaQL ownership in `plugin-bases`, and fall back to the current in-memory evaluator whenever a query shape cannot be safely lowered. ## Scope - Add an AppDatabase read/query contract for indexed files, metadata, frontmatter properties, tags, links, and file records. - Implement that contract for memory and IndexedDB fallback behavior. - Implement SQLite-backed behavior for browser OPFS sessions through `SqliteWasmAppDatabaseCore`, the worker proxy, and the coordinated browser proxy. - Extend desktop native behavior if the final design requires host-side metadata queries instead of serialized-state-only reads. - Add a Bases query-source abstraction that converts AppDatabase rows back into the existing `VaultRecord`/`BasesEntry` shape. - Convert `QueryController` reloads from synchronous derived array execution to async, generation-tokened reloads that cannot overwrite newer document or metadata changes with stale database responses. - Preserve current Bases behavior for table, card, list, custom registered views, markdown embeds, and fenced `base`/`bases` blocks. - Update package-local and rendered specs for the final behavior and any intentional fallback limits. ## Non-goals - Do not change the `.base` or `.bases` file format. - Do not store canonical note bytes in `AppDatabase`. - Do not let `plugin-bases` open SQLite, native stores, or browser storage directly. - Do not move PEaQL or Bases-specific query semantics into `packages/api`. - Do not replace PEaQL syntax or reimplement the complete PEaQL evaluator in SQL. - Do not solve the separate map-view issue #3. - Do not include the table column drag-and-drop migration tracked by issue #5. ## Current State - `QueryController.data` is derived from `app.metadataCache.fileCache` and maps every cached file into a `VaultRecord`. - `columnsFor(...)` infers frontmatter columns by scanning the current renderer row set, while also materializing explicit `note.*` properties declared in a Bases document. - `QueryController.db` creates a PEaQL `Context` over `new BasesTable("bases").withPropertyColumns(...).data(() => this.data)`. - `QueryController.getRecords(...)` calls `this.db.execute(qs)` synchronously and converts the result into the existing `BasesEntry`/`BasesQueryResult` pipeline. - `AppDatabase` already persists `files`, `metadata`, `links`, `tags`, and `properties`, but the public contract currently exposes write operations for indexed metadata and does not expose a Bases-suitable read/query surface. - Browser SQLite already runs behind an app-database worker and has coordinated secondary-tab proxying. - `NativeDesktopAppDatabase` currently persists serialized generated state and mirrors search documents into native search tables, but it does not expose a dedicated metadata query IPC surface. - Metadata invalidation already lives in `QueryController` and uses dependency tracking over document filters, view filters, visible/sorted properties, group-by state, formulas, and direct file-reference neighborhoods. ## Implementation Details ### 1. Characterize current behavior - Add focused regression coverage or a benchmark fixture that builds a synthetic large metadata set and records the current renderer-only behavior. - Cover query shapes that must remain stable: file fields, `inFolder`, explicit `note.*` fields, inferred frontmatter fields, tags, links, sort, limit, grouping, formulas, and search-within-view. - Use the fixture as both a baseline and a parity guard for the AppDatabase-backed path. ### 2. Design the AppDatabase metadata read contract Add serializable types in `packages/api` for a plugin-neutral indexed metadata query surface. The contract should expose data that any structured metadata consumer could use, not Bases-specific PEaQL concepts. Suggested shape: ```ts type AppDatabaseIndexedMetadataQuery = { fileExtensions?: string[]; pathPrefixes?: string[]; propertyNames?: string[]; propertyFilters?: AppDatabasePropertyFilter[]; tags?: string[]; linkedPaths?: string[]; sort?: AppDatabaseIndexedMetadataSort[]; limit?: number; }; type AppDatabaseIndexedMetadataRow = { file: AppDatabaseFileRecord; metadata: AppDatabaseMetadataRecord | null; properties: AppDatabasePropertyRecord[]; tags: AppDatabaseTagRecord[]; links: AppDatabaseLinkRecord[]; }; ``` The exact API can differ, but it should: - Return enough file stat, metadata, property, tag, and link data to reconstruct a `VaultRecord` for existing Bases rendering. - Allow callers to request all indexed rows when safe lowering is unavailable. - Allow simple candidate narrowing by extension, folder/path prefix, property existence, property equality/comparison, tags, links, sort, and limit. - Preserve memory and IndexedDB parity with SQLite-backed results. - Avoid importing `plugin-bases`, PEaQL, or Svelte concepts into `packages/api`. ### 3. Implement AppDatabase backends - Add the new method or methods to the `AppDatabase` interface. - Implement `MemoryAppDatabase` by reading the existing in-memory `files`, `metadata`, `properties`, `tags`, and `links` maps. - Let `IndexedDbAppDatabase` inherit the memory behavior after opening/restoring state, unless a storage-specific read path is necessary. - Implement SQL-backed filtering in `SqliteWasmAppDatabaseCore` using the existing `files`, `metadata`, `properties`, `tags`, and `links` tables. - Add any missing SQLite indexes needed by the selected filter shape, such as `(path, name)`, `(name, value_json)`, or file extension/path indexes, without overfitting to one Bases view. - Route the new method through `sqlite-wasm-app-database.worker.ts`, `SqliteWasmAppDatabase`, and `BrowserCoordinatedAppDatabase` so browser worker and secondary-tab behavior stays consistent. - Decide whether `NativeDesktopAppDatabase` can answer from restored serialized state or needs a native IPC query. If native IPC is required, add a narrow host command that returns the same serializable row shape. ### 4. Add a Bases query-source abstraction Introduce a small boundary inside `packages/plugins/plugin-bases`, for example: ```ts interface BasesQuerySource { loadRows(request: BasesQuerySourceRequest): Promise<VaultRecord[]>; } ``` Responsibilities: - Translate the current Bases document, active view, filters, visible properties, sort, group-by, formulas, and search state into a conservative AppDatabase query request. - Reconstruct `TFile` or `TFile`-compatible file objects from returned file records by resolving paths through `app.vault.getFileByPath(...)` when available and using the existing fallback `TFile` construction otherwise. - Rebuild `CachedMetadata`/frontmatter-compatible row data from returned metadata and property records. - Preserve checksum behavior so `BasesEntry` identity and custom views continue to work. - Fall back to loading all indexed rows and running PEaQL in memory for unsupported query shapes. ### 5. Convert QueryController reloads to async - Replace the synchronous `data` derived value and direct `this.db.execute(...)` reload path with an explicit async reload pipeline. - Track a monotonically increasing reload generation or cancellation token. - Ignore stale database responses when a newer metadata change, document edit, view switch, or filter edit has already requested another reload. - Keep initial render, metadata `loaded`, metadata `changed`, and metadata `deleted` events working. - Continue to call `view.onDataUpdated()` only when the active view receives a fresh `BasesQueryResult`. - Ensure errors surface as the existing empty/error result behavior rather than crashing the view. ### 6. Lower safe constraints first Start with conservative candidate narrowing rather than full SQL execution of every PEaQL expression. Good first candidates: - File extension and markdown-only row selection. - `file.path`, `file.name`, `file.folder`, and `inFolder` path-prefix filters. - Property existence and simple property equality/comparison where the value can be serialized safely. - Tag filters backed by the indexed `tags` table. - Direct link/embed candidate filters backed by the indexed `links` table. - Sort and limit when the sorted field maps directly to a stored file or property record. Fallback cases should continue through the current PEaQL path: - Custom formulas. - Unsupported PEaQL functions. - Complex expressions that combine derived values in ways the AppDatabase contract cannot represent safely. - Query shapes where SQLite and memory semantics would diverge. ### 7. Preserve column inference and explicit note properties - Keep explicit `note.*` properties declared in the Bases document even when the current metadata rows do not contain that frontmatter key. - Continue to infer observed frontmatter columns from returned candidate rows. - Make sure `frontMatterTypesForColumns(...)` still supplies PEaQL property column types for both explicit and inferred note fields. - Add regression tests for generated plugin-owned `.base` documents that reference structured fields such as `note.status` before any note currently carries that property. ### 8. Update invalidation behavior - Reuse the existing `shouldReloadForMetadataChange(...)` dependency logic. - Make async reload scheduling compatible with metadata `loaded`, `changed`, and `deleted` events. - Ensure AppDatabase-backed reads see the freshly indexed metadata after `MetadataCache` writes `upsertIndexedFile(...)`, `deleteIndexedFile(...)`, or `renameIndexedFile(...)`. - If write/read ordering is not guaranteed for a backend, await or queue the AppDatabase write before asking Bases to reload. ### 9. Document the result Update: - `spec/src/20-packages/plugins/bases/index.md` - `spec/src/30-cross-package-contracts/storage-metadata-search.md` - `packages/plugins/plugin-bases/spec.md` Document: - Which work moves behind `AppDatabase`. - Which query shapes are lowered. - Which query shapes intentionally fall back to in-memory PEaQL. - How memory, browser SQLite, coordinated tabs, and desktop native behavior stay equivalent. ## Acceptance Criteria - Bases views can load candidate rows through `AppDatabase` instead of always rebuilding a full renderer-owned `VaultRecord[]` from `metadataCache.fileCache`. - Query results match the current renderer-only path for covered filters, sort, limit, grouping, formulas, visible columns, and search-within-view. - Unsupported query shapes fall back to the existing PEaQL behavior without user-visible result divergence. - Explicit `note.*` fields declared in a Bases document continue to compile and render even when no current row has that frontmatter key. - Browser OPFS SQLite sessions execute the new AppDatabase read path through the existing worker/proxy model. - Coordinated secondary browser tabs delegate the new AppDatabase query method to the owning tab and recover consistently after owner promotion. - Desktop behavior remains correct, either through serialized-state reads or a narrow native metadata-query bridge. - Metadata create, modify, delete, rename, and metadata-cache load events refresh open Bases views without stale async responses winning over newer state. - Custom Bases view registrations receive the same `BasesQueryResult` shape they receive today. - Markdown file embeds and fenced `base`/`bases` code blocks continue to render in read-only mode. - Large synthetic vault tests or benchmarks show reduced renderer-side row construction/query work for lowered query shapes. - Specs describe the final AppDatabase-backed behavior and intentional fallback limits. ## Suggested Files or Specs To Inspect - `packages/api/src/lib/storage/app-database.ts` - `packages/api/src/lib/storage/sqlite-wasm-app-database-core.ts` - `packages/api/src/lib/storage/sqlite-wasm-app-database.ts` - `packages/api/src/lib/storage/sqlite-wasm-app-database.worker.ts` - `packages/api/src/lib/storage/browser-coordinated-app-database.ts` - `packages/api/src/lib/storage/desktop-native.ts` - `packages/api/src/lib/cache.svelte.ts` - `packages/plugins/plugin-bases/src/bases-view/bases.svelte.ts` - `packages/plugins/plugin-bases/src/bases-view/db.ts` - `packages/plugins/plugin-bases/src/bases-view/columns.ts` - `packages/plugins/plugin-bases/src/bases-view/filter-parser.ts` - `packages/plugins/plugin-bases/src/bases-view/file-fields-core.ts` - `packages/plugins/plugin-bases/spec.md` - `spec/src/20-packages/plugins/bases/index.md` - `spec/src/30-cross-package-contracts/storage-metadata-search.md` ## Test Plan ### AppDatabase tests - Memory implementation returns indexed metadata rows for inserted file, metadata, property, tag, and link records. - SQL implementation returns the same rows as memory for the same input records. - Property existence, equality, comparison, tag, link, folder/path, sort, and limit filters match across memory and SQL. - Delete and rename operations update query results. - Browser worker/proxy and coordinated database paths expose the new method. - Desktop native or serialized-state reads return the same serializable row shape. ### Bases tests - Existing filters, formulas, sort, limit, grouping, and view search produce the same result set as the renderer-only path. - Explicit `note.*` properties are materialized without observed frontmatter values. - QueryController ignores stale async reload results. - Metadata `loaded`, `changed`, and `deleted` events trigger correct reloads. - Custom view registrations receive updated `BasesQueryResult` data. - Markdown-hosted read-only embeds and fenced Bases blocks keep rendering through the same query pipeline. ### Manual or smoke checks - Open a large vault table in the workspace app and verify the UI remains responsive while metadata-backed Bases rows load. - Run the same large-vault check in Electron with the native app database path. - Start interactive verification with syntax highlighting disabled first so editor highlighting issues do not mask Bases behavior. ## Validation Commands - `pnpm --filter @lapis-notes/api test:run src/lib/__tests__/app-database.test.ts` - `pnpm --filter @lapis-notes/bases test -- src/bases-view` - `pnpm --filter @lapis-notes/api check` - `pnpm --filter @lapis-notes/bases check` - `pnpm check` - `make spec-lint` - `mdbook build spec` - `pnpm test:smoke` ## Completion Notes When implementation and validation are complete, close this issue through the backlog helper with a concise implementation summary and commit with `Fixes #4` in the commit message body.

steve added the

labels

2026-05-20 18:54:28 +00:00

steve

2026-05-29 11:18:24 +00:00