Stabilize official plugin asset publish observability and memory usage #346

New issue

Closed

opened 2026-06-06 19:03:29 +00:00 by steve · 0 comments

steve commented

2026-06-06 19:03:29 +00:00

Owner

The publish-official-plugin-assets.yml workflow and registry-assets:publish
script remain too opaque and too memory-hungry under Forgejo CI.

Problem

Publish and verify runs scan Forgejo releases slowly, and the current logging
leaves long gaps after Forgejo releases page ... without showing asset-page
progress or whether the scan is still active.
The availability check indexes all assets across matching
official-plugin-assets-* releases even though it only needs to know whether
the selected plugin/version prefixes already exist.
The current packaging and upload paths buffer large artifacts into JS heap,
especially notebook/docs/canvas assets, which increases OOM risk.
Turbo remote cache may already be wired correctly, but publish logs do not
prove whether the build ran with remote cache env configured or whether the
current build shape is making poor use of it.

Goal

Make official plugin asset publish runs faster to understand, lower-memory, and
easier to debug in Forgejo CI without changing the published asset contract.

Scope

Targeted Forgejo availability scanning for selected plugin/version prefixes.
Better publish logging around scan progress, Turbo env, build, staging, and
upload batch sizes.
Lower-memory packaging and upload implementation for large release assets.
Workflow inputs for GitHub mirror toggling and upload batch size.
Spec and workflow documentation updates that describe the new behavior.

Non-goals

Changing release tag naming, asset naming, signed manifest schema, or plugin
release file layout.
Changing official-registry metadata semantics.
Replacing the hosted Turbo remote cache service or changing its secrets.

Acceptance Criteria

registry-assets:publish --publish and --verify scan only
official-plugin-assets-* releases and stop once all requested
plugin/version prefixes are found.
Publish logs show release-page progress, asset-page progress, Turbo env
status, batch sizes, and per-asset upload progress with sizes.
Official plugin builds use one turbo run build invocation with repeated
--filter flags plus --concurrency=1.
Packaging no longer assembles ZIPs fully in memory before writing them.
Forgejo and GitHub upload scripts no longer read whole release assets into
memory before upload.
The workflow exposes publish_github and upload_batch_size inputs and uses
a Forgejo-compatible artifact upload action.
Specs stay in sync with the workflow and script behavior.

Implementation Notes

Keep the duplicate-detection result shape unchanged so publish/verify behavior
remains skip/fail compatible with existing callers.
Add targeted scan helpers instead of building a global asset index first.
Use Node stream or file-backed blob APIs available in the CI image rather than
introducing a new archive or upload dependency.
Dry runs still stage real assets locally; the goal is observability and lower
heap pressure, not eliminating staging work.

Suggested Files or Specs To Inspect

.forgejo/workflows/publish-official-plugin-assets.yml
scripts/publish-official-plugin-assets.mjs
scripts/plugin-release.mjs
scripts/forgejo-release.mjs
scripts/github-release.mjs
scripts/publish-release-assets.mjs
scripts/publish-official-plugin-assets.test.mjs
scripts/plugin-release.test.mjs
spec/src/80-maintenance/monorepo-scripts.md
spec/src/80-maintenance/release-management.md
spec/src/30-cross-package-contracts/plugin-registry.md

Validation Commands

pnpm test:scripts
pnpm check:all
mdbook build spec

Follow-up to #345 — Reduce registry-assets publish CI memory pressure

Follow-up Tasks

Implementation Summary

Added targeted Forgejo asset scanning, single-invocation Turbo builds, streamed packaging/uploads, CI workflow controls, and release publish observability docs.

The `publish-official-plugin-assets.yml` workflow and `registry-assets:publish` script remain too opaque and too memory-hungry under Forgejo CI. ## Problem - Publish and verify runs scan Forgejo releases slowly, and the current logging leaves long gaps after `Forgejo releases page ...` without showing asset-page progress or whether the scan is still active. - The availability check indexes all assets across matching `official-plugin-assets-*` releases even though it only needs to know whether the selected plugin/version prefixes already exist. - The current packaging and upload paths buffer large artifacts into JS heap, especially notebook/docs/canvas assets, which increases OOM risk. - Turbo remote cache may already be wired correctly, but publish logs do not prove whether the build ran with remote cache env configured or whether the current build shape is making poor use of it. ## Goal Make official plugin asset publish runs faster to understand, lower-memory, and easier to debug in Forgejo CI without changing the published asset contract. ## Scope - Targeted Forgejo availability scanning for selected plugin/version prefixes. - Better publish logging around scan progress, Turbo env, build, staging, and upload batch sizes. - Lower-memory packaging and upload implementation for large release assets. - Workflow inputs for GitHub mirror toggling and upload batch size. - Spec and workflow documentation updates that describe the new behavior. ## Non-goals - Changing release tag naming, asset naming, signed manifest schema, or plugin release file layout. - Changing official-registry metadata semantics. - Replacing the hosted Turbo remote cache service or changing its secrets. ## Acceptance Criteria - `registry-assets:publish --publish` and `--verify` scan only `official-plugin-assets-*` releases and stop once all requested plugin/version prefixes are found. - Publish logs show release-page progress, asset-page progress, Turbo env status, batch sizes, and per-asset upload progress with sizes. - Official plugin builds use one `turbo run build` invocation with repeated `--filter` flags plus `--concurrency=1`. - Packaging no longer assembles ZIPs fully in memory before writing them. - Forgejo and GitHub upload scripts no longer read whole release assets into memory before upload. - The workflow exposes `publish_github` and `upload_batch_size` inputs and uses a Forgejo-compatible artifact upload action. - Specs stay in sync with the workflow and script behavior. ## Implementation Notes - Keep the duplicate-detection result shape unchanged so publish/verify behavior remains skip/fail compatible with existing callers. - Add targeted scan helpers instead of building a global asset index first. - Use Node stream or file-backed blob APIs available in the CI image rather than introducing a new archive or upload dependency. - Dry runs still stage real assets locally; the goal is observability and lower heap pressure, not eliminating staging work. ## Suggested Files or Specs To Inspect - `.forgejo/workflows/publish-official-plugin-assets.yml` - `scripts/publish-official-plugin-assets.mjs` - `scripts/plugin-release.mjs` - `scripts/forgejo-release.mjs` - `scripts/github-release.mjs` - `scripts/publish-release-assets.mjs` - `scripts/publish-official-plugin-assets.test.mjs` - `scripts/plugin-release.test.mjs` - `spec/src/80-maintenance/monorepo-scripts.md` - `spec/src/80-maintenance/release-management.md` - `spec/src/30-cross-package-contracts/plugin-registry.md` ## Validation Commands - `pnpm test:scripts` - `pnpm check:all` - `mdbook build spec` ## Related Issues - Follow-up to #345 — [Reduce registry-assets publish CI memory pressure](../closed/345-reduce-registry-assets-publish-ci-memory-pressure.md) ## Follow-up Tasks ## Implementation Summary Added targeted Forgejo asset scanning, single-invocation Turbo builds, streamed packaging/uploads, CI workflow controls, and release publish observability docs.

steve added this to the Full Registry V1 milestone

2026-06-06 19:03:29 +00:00

steve

2026-06-06 19:03:29 +00:00