Publish: per-item timing and size logs, ETA for large sites, parallel checks and uploads #6

Open
opened 2026-10-10 18:23:09 +00:00 by rhelwig · 3 comments
Owner

Publishing speed is acceptable for a small site but won't scale. At today's per-item pace, a large site could take hours.

Seen on 2026-10-10: ronhelwig.com (11 pages) took a long time to publish. Auth runs about 36 proof reads of inspection before every write, all of it serially.

Wanted:

  1. Log the duration and byte size of every stored or published item (IPFS upload, each Platform check, each write and confirmation). Keep the logs where we can analyze them later.
  2. Use those logs to show an ETA before and during push/publish.
  3. Investigate parallel or concurrent work:
    • concurrent IPFS uploads;
    • parallel proof reads in checks;
    • checking all items up front once, then re-checking only what's needed per write;
    • pipelining independent writes, if nonce rules allow it.
  4. Optional: record actual fees per write for future cost estimates. Today's estimates were good enough.

Related: only-changed versioning and continue-on-fail.

Follow-up notes: https://git.ronhelwig.com/rhelwig/DecentSite/src/branch/main/docs/PUBLISH-FOLLOWUPS-2026-10-10.md

Publishing speed is acceptable for a small site but won't scale. At today's per-item pace, a large site could take hours. **Seen on 2026-10-10:** ronhelwig.com (11 pages) took a long time to publish. Auth runs about 36 proof reads of inspection before *every* write, all of it serially. **Wanted:** 1. Log the duration and byte size of every stored or published item (IPFS upload, each Platform check, each write and confirmation). Keep the logs where we can analyze them later. 2. Use those logs to show an ETA before and during push/publish. 3. Investigate parallel or concurrent work: - concurrent IPFS uploads; - parallel proof reads in checks; - checking all items up front once, then re-checking only what's needed per write; - pipelining independent writes, if nonce rules allow it. 4. Optional: record actual fees per write for future cost estimates. Today's estimates were good enough. Related: only-changed versioning and continue-on-fail. Follow-up notes: https://git.ronhelwig.com/rhelwig/DecentSite/src/branch/main/docs/PUBLISH-FOLLOWUPS-2026-10-10.md
Author
Owner

Started on branch feat/publish-timing-logs (c1fd3ee, not merged yet).

Done so far: the Auth engine's executeRelease now records timing for each item: bytes (size of the reviewed properties), totalMs, and stagesMs for checking, signing, broadcasting and verifying. It also records a total for the whole run. onStep includes the timing with each outcome. The timing data contains no SDK text and no record contents. New test added; 167/167 pass.

Next: keep timing in Auth's Publication history and add an export. Log Admin's IPFS upload time and bytes per blob. Then build the ETA from past runs. Then try the speed experiments: one inspection per run, parallel proof reads, and concurrent uploads. The plan, including design notes for #5, is in docs/IMPLEMENTATION-PLAN.md on that branch.

Started on branch `feat/publish-timing-logs` (c1fd3ee, not merged yet). Done so far: the Auth engine's `executeRelease` now records `timing` for each item: `bytes` (size of the reviewed properties), `totalMs`, and `stagesMs` for checking, signing, broadcasting and verifying. It also records a total for the whole run. `onStep` includes the timing with each outcome. The timing data contains no SDK text and no record contents. New test added; 167/167 pass. Next: keep timing in Auth's Publication history and add an export. Log Admin's IPFS upload time and bytes per blob. Then build the ETA from past runs. Then try the speed experiments: one inspection per run, parallel proof reads, and concurrent uploads. The plan, including design notes for #5, is in `docs/IMPLEMENTATION-PLAN.md` on that branch.
Author
Owner

Progress update. PR #9 (feat/publish-timing-logs @ 3a523e4, on top of c1fd3ee) has these pieces:

  • Auth engine: per-step timing. For each step it records the bytes of the reviewed record, the total ms, and ms spent checking, signing, broadcasting and verifying. It also records a total for the run.
  • Auth 0.1.19:
    • Publication history keeps the newest 20 timing runs for each release, and keeps them after publish.
    • It shows how long the last run took, plus each step's stage times under technical details.
    • Export timing log (CSV) sends all runs through the share sheet.
    • The fee screen shows an estimated publishing time, based on the median confirmed-step time.
  • Admin:
    • Review prep logs how long each phase took, and the IPFS upload and read-back time and size for each blob.
    • It writes these to <dataRoot>/logs/publish-timing.jsonl. QR handoff upload is logged too.
    • Prepare review shows an estimate based on earlier preparations.
  • Privacy: only type, status, sizes and durations are kept. No record IDs, CIDs, contents, keys or SDK text.

Tests: npm test 169/169, typecheck clean, :protocol:test green, and the mainnet debug APK builds. Nothing is installed on a device yet, and no Dash was spent.

Still open here:

  • A per-stage ETA model and a live ETA during publishing.
  • Trying parallel proof reads, inspecting once per run, and concurrent IPFS uploads.
  • Collecting real data from the next publish.

The design for only-changed writes (#5) is in PR #10, as a doc-only change.

Progress update. PR #9 (`feat/publish-timing-logs` @ 3a523e4, on top of c1fd3ee) has these pieces: - **Auth engine:** per-step timing. For each step it records the bytes of the reviewed record, the total ms, and ms spent checking, signing, broadcasting and verifying. It also records a total for the run. - **Auth 0.1.19:** - Publication history keeps the newest 20 timing runs for each release, and keeps them after publish. - It shows how long the last run took, plus each step's stage times under technical details. - **Export timing log (CSV)** sends all runs through the share sheet. - The fee screen shows an estimated publishing time, based on the median confirmed-step time. - **Admin:** - Review prep logs how long each phase took, and the IPFS upload and read-back time and size for each blob. - It writes these to `<dataRoot>/logs/publish-timing.jsonl`. QR handoff upload is logged too. - Prepare review shows an estimate based on earlier preparations. - **Privacy:** only type, status, sizes and durations are kept. No record IDs, CIDs, contents, keys or SDK text. Tests: `npm test` 169/169, typecheck clean, `:protocol:test` green, and the mainnet debug APK builds. Nothing is installed on a device yet, and no Dash was spent. Still open here: - A per-stage ETA model and a live ETA during publishing. - Trying parallel proof reads, inspecting once per run, and concurrent IPFS uploads. - Collecting real data from the next publish. The design for only-changed writes (#5) is in PR #10, as a doc-only change.
Author
Owner

Timing MVP merged to main in PR #9 (merge 2b95a21; Auth 0.1.19 / versionCode 20, now installed on the tablet).

Done:

  • Engine: per-item timing (bytes, total ms, checking/signing/broadcasting/verifying stage ms) plus run total
  • Auth: keeps the last 20 timing runs per receipt, shows them in Publication history, and has an "Export timing log (CSV)" option
  • Admin: prepare-phase and per-blob IPFS upload timing appended to /logs/publish-timing.jsonl
  • ETA stub: Auth fee screen shows an estimated publishing time based on the median step time; Admin Prepare review shows an upload estimate

Leaving this open for the remaining work:

  • Collect real timing logs from the next publish and analyze where the time goes
  • Per-stage ETA model and a live progress ETA during publish
  • Inspect once per run, then re-check only each item's own dependencies
  • Parallel proof reads in checks
  • Concurrent IPFS uploads in Admin
  • Pipelining independent writes, if nonce rules allow
  • (Optional) record the actual fee per write for cost estimates

Only-changed writes (#5) is the biggest win for write count and is in progress.

Timing MVP merged to main in PR #9 (merge 2b95a21; Auth 0.1.19 / versionCode 20, now installed on the tablet). Done: - Engine: per-item timing (bytes, total ms, checking/signing/broadcasting/verifying stage ms) plus run total - Auth: keeps the last 20 timing runs per receipt, shows them in Publication history, and has an "Export timing log (CSV)" option - Admin: prepare-phase and per-blob IPFS upload timing appended to <dataRoot>/logs/publish-timing.jsonl - ETA stub: Auth fee screen shows an estimated publishing time based on the median step time; Admin Prepare review shows an upload estimate Leaving this open for the remaining work: - [ ] Collect real timing logs from the next publish and analyze where the time goes - [ ] Per-stage ETA model and a live progress ETA during publish - [ ] Inspect once per run, then re-check only each item's own dependencies - [ ] Parallel proof reads in checks - [ ] Concurrent IPFS uploads in Admin - [ ] Pipelining independent writes, if nonce rules allow - [ ] (Optional) record the actual fee per write for cost estimates Only-changed writes (#5) is the biggest win for write count and is in progress.
Sign in to join this conversation.
No labels
feature request
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
rhelwig/DecentSite#6
No description provided.