Reduce notebook plugin release size by externalizing heavy runtime assets #300

New issue

Closed

opened 2026-06-05 17:08:58 +00:00 by steve · 0 comments

steve commented

2026-06-05 17:08:58 +00:00

Owner

Problem

The current notebook packaging path stages the full plugin dist output and publishes all generated files. For notebook, this includes very large chunks where DuckDB binaries/workers are inlined and repeated across UMD/ES/CJS outputs. The result is a release payload far larger than expected for an on-demand plugin.

Goal

Reduce notebook official release payload substantially by eliminating duplicate heavy blobs from the default package and moving heavyweight runtime assets to verified on-demand downloads, while keeping notebook behavior reliable on both web/PWA and desktop Electron.

Scope

Notebook plugin packaging/build output changes needed to remove oversized duplication.
Notebook DuckDB asset handling changes so worker/WASM can be fetched as external assets instead of inlined bundle payloads.
Signed external asset publication and runtime download/install path reuse for notebook heavy assets.
Web/PWA and Electron runtime handling for downloaded/cached notebook heavy assets.
Tests and spec updates for new loading behavior.

Non-goals

Replacing DuckDB with another query engine.
Redesigning notebook language semantics or notebook execution model beyond asset loading boundaries.
Broad plugin runtime architecture rewrites unrelated to notebook payload size.

Acceptance Criteria

Notebook release zip size drops significantly from current baseline and no longer contains 100MB-class JS chunks.
Release manifest no longer includes duplicated heavy inlined DuckDB blob payloads across multiple runtime formats.
Opening/editing non-SQL notebook files does not require downloading heavyweight external assets.
First notebook SQL/DuckDB execution triggers verified on-demand asset retrieval when assets are absent.
Downloaded assets are integrity-checked (signed manifest/hash) before use.
Cached downloaded assets are reused on subsequent runs in both web/PWA and Electron.
Missing-network first-run behavior surfaces a clear actionable error when heavy assets are required but unavailable.
Relevant specs are updated to reflect the new packaging and runtime asset-loading contract.

Implementation Notes

Current baseline evidence:
- release-artifacts/official-plugin-assets/forgejo-assets/lapis-notebook-2026.6.1-artifact.json reports a 633331962-byte notebook zip.
- lapis-notebook-2026.6.1-release.json includes very large files such as main.umd.js, notebook-duckdb-service-*.{js,cjs}, and duplicated notebook runtime workers.
Key packaging flow currently copies complete plugin dist into release inputs:
- scripts/publish-official-plugin-assets.mjs (stageReleaseInput)
- scripts/plugin-release.mjs (release file enumeration/signing)
Notebook DuckDB service currently resolves bundles via @lapis-notes/notebook-duckdb-assets and should keep that boundary while changing payload source strategy.
Reuse existing plugin distribution verification/install infrastructure for external heavy asset downloads where practical.

Suggested Files or Specs To Inspect

packages/notebook/vite.config.ts
packages/notebook/duckdb-assets/src/assets.ts
packages/notebook/src/lib/runtime/duckdb/notebook-duckdb-service.ts
scripts/publish-official-plugin-assets.mjs
scripts/plugin-release.mjs
packages/api/src/lib/plugin-distribution/installer.ts
packages/api/src/lib/plugin-asset-server.ts
spec/src/20-packages/plugins/notebook/index.md
spec/src/20-packages/notebook-duckdb-assets/index.md
spec/src/30-cross-package-contracts/plugin-registry.md

Validation Commands

pnpm --filter @lapis-notes/notebook check:all
pnpm --filter @lapis-notes/api check:all (if API/plugin-distribution paths change)
pnpm check:all (final cross-package handoff)
pnpm registry-assets:publish -- --plugins lapis-notebook --dry-run (size/regression check)

Follow-up Tasks

Optional: externalize additional notebook runtime payloads (for example TypeScript compiler artifacts) after DuckDB-heavy path stabilization.

Implementation Summary

Externalized notebook DuckDB wasm/workers to dist/duckdb/, dropped UMD output, marked heavy runtime files optional in signed releases, skip optional files on install, added ensureReleaseFiles() for verified on-demand download before first SQL execution, and wired notebook worker/host lazy asset loading. Dry-run publish: required install payload ~45MB (max chunk ~22MB) vs prior 100MB+ required chunks; optional lazy assets ~734MB fetched on demand. Validated notebook/api check:all+tests and registry-assets dry-run.

Notebook official release artifacts are currently oversized (about 600MB zip), dominated by duplicated generated bundles and inlined DuckDB worker/WASM payloads that are embedded as base64 data URLs across multiple output variants. This makes release publishing heavy and slows install/update paths for the official notebook plugin. ## Problem The current notebook packaging path stages the full plugin dist output and publishes all generated files. For notebook, this includes very large chunks where DuckDB binaries/workers are inlined and repeated across UMD/ES/CJS outputs. The result is a release payload far larger than expected for an on-demand plugin. ## Goal Reduce notebook official release payload substantially by eliminating duplicate heavy blobs from the default package and moving heavyweight runtime assets to verified on-demand downloads, while keeping notebook behavior reliable on both web/PWA and desktop Electron. ## Scope - Notebook plugin packaging/build output changes needed to remove oversized duplication. - Notebook DuckDB asset handling changes so worker/WASM can be fetched as external assets instead of inlined bundle payloads. - Signed external asset publication and runtime download/install path reuse for notebook heavy assets. - Web/PWA and Electron runtime handling for downloaded/cached notebook heavy assets. - Tests and spec updates for new loading behavior. ## Non-goals - Replacing DuckDB with another query engine. - Redesigning notebook language semantics or notebook execution model beyond asset loading boundaries. - Broad plugin runtime architecture rewrites unrelated to notebook payload size. ## Acceptance Criteria - Notebook release zip size drops significantly from current baseline and no longer contains 100MB-class JS chunks. - Release manifest no longer includes duplicated heavy inlined DuckDB blob payloads across multiple runtime formats. - Opening/editing non-SQL notebook files does not require downloading heavyweight external assets. - First notebook SQL/DuckDB execution triggers verified on-demand asset retrieval when assets are absent. - Downloaded assets are integrity-checked (signed manifest/hash) before use. - Cached downloaded assets are reused on subsequent runs in both web/PWA and Electron. - Missing-network first-run behavior surfaces a clear actionable error when heavy assets are required but unavailable. - Relevant specs are updated to reflect the new packaging and runtime asset-loading contract. ## Implementation Notes - Current baseline evidence: - `release-artifacts/official-plugin-assets/forgejo-assets/lapis-notebook-2026.6.1-artifact.json` reports a 633331962-byte notebook zip. - `lapis-notebook-2026.6.1-release.json` includes very large files such as `main.umd.js`, `notebook-duckdb-service-*.{js,cjs}`, and duplicated notebook runtime workers. - Key packaging flow currently copies complete plugin dist into release inputs: - `scripts/publish-official-plugin-assets.mjs` (`stageReleaseInput`) - `scripts/plugin-release.mjs` (release file enumeration/signing) - Notebook DuckDB service currently resolves bundles via `@lapis-notes/notebook-duckdb-assets` and should keep that boundary while changing payload source strategy. - Reuse existing plugin distribution verification/install infrastructure for external heavy asset downloads where practical. ## Suggested Files or Specs To Inspect - `packages/notebook/vite.config.ts` - `packages/notebook/duckdb-assets/src/assets.ts` - `packages/notebook/src/lib/runtime/duckdb/notebook-duckdb-service.ts` - `scripts/publish-official-plugin-assets.mjs` - `scripts/plugin-release.mjs` - `packages/api/src/lib/plugin-distribution/installer.ts` - `packages/api/src/lib/plugin-asset-server.ts` - `spec/src/20-packages/plugins/notebook/index.md` - `spec/src/20-packages/notebook-duckdb-assets/index.md` - `spec/src/30-cross-package-contracts/plugin-registry.md` ## Validation Commands - `pnpm --filter @lapis-notes/notebook check:all` - `pnpm --filter @lapis-notes/api check:all` (if API/plugin-distribution paths change) - `pnpm check:all` (final cross-package handoff) - `pnpm registry-assets:publish -- --plugins lapis-notebook --dry-run` (size/regression check) ## Related Issues - N/A ## Follow-up Tasks - Optional: externalize additional notebook runtime payloads (for example TypeScript compiler artifacts) after DuckDB-heavy path stabilization. ## Implementation Summary Externalized notebook DuckDB wasm/workers to dist/duckdb/, dropped UMD output, marked heavy runtime files optional in signed releases, skip optional files on install, added ensureReleaseFiles() for verified on-demand download before first SQL execution, and wired notebook worker/host lazy asset loading. Dry-run publish: required install payload ~45MB (max chunk ~22MB) vs prior 100MB+ required chunks; optional lazy assets ~734MB fetched on demand. Validated notebook/api check:all+tests and registry-assets dry-run.

steve

2026-06-05 17:08:58 +00:00