Under the Hood: Headless Web Worker Pools and Cached Navigation Panel Ingestion

TLDR

Transitioning document processing pipelines into platform-agnostic IP assets requires decoupling batch worker delegation from main-thread UI state. In this deep dive, we detail migrating core batch modules (batchGraph.js, batchQueue.js, workerPool.js) into /assets/pdf-processor/batch/ using Vite @batch aliases, embedding the document queue into the navigation panel (nav-view-batch), and building an instant (<10ms) cached memory mount pipeline that eliminates main-thread worker re-extractions upon card focus.

Batch Architecture Component	Naive Implementation	Platform IP Implementation
Worker Task Dispatch	Direct inline Web Worker instantiation	WorkerPool hardware-concurrency pool
Asset Path Resolution	Relational relative pathing (../../)	Vite path alias @batch $\rightarrow$ /assets/pdf-processor/batch
Focus State Hydration	Re-run handleFile() parsing on click	Direct memory mount from cached gxDoc IR & HTML

---

Technical problem statements & solutions

1. Platform IP module isolation via Vite path aliasing

Hardcoding relative paths across separate developer utility tools creates tight coupling and makes shared IP assets difficult to package into standalone packages.

Configure Vite path aliases to point to central IP asset directories:

// vite.config.js
import { defineConfig } from 'vite';
import path from 'path';
export default defineConfig({   resolve: {     alias: {       '@batch': path.resolve(__dirname, '../../assets/pdf-processor/batch')     }   } });

Using @batch allows UI controllers (batchViewController.js) to import queue managers cleanly:

import { BatchQueueManager } from '@batch/batchQueue.js';
import { WorkerPool } from '@batch/workerPool.js';

---

2. Off-thread Web Worker pool task distribution

Running multi-document extraction (PDF, DOCX, HTML, Markdown, JSON) on the main UI thread freezes DOM rendering and produces frame drops during drag-and-drop ingestion.

Delegate extraction tasks to a Web Worker pool sized dynamically to hardware concurrency:

export class WorkerPool {
    constructor(poolSize) {
        this.poolSize = poolSize || Math.max(1, (navigator.hardwareConcurrency || 4) - 1);
        this.workers = [];
        this.idleWorkers = [];
        this.taskQueue = [];
        this._initPool();
    }
processItem(item) {         return new Promise((resolve, reject) => {             this.taskQueue.push({ item, resolve, reject });             this._dispatchNext();         });     }
_dispatchNext() {         if (this.taskQueue.length === 0 || this.idleWorkers.length === 0) return;
const workerEntry = this.idleWorkers.shift();         const task = this.taskQueue.shift();         workerEntry.busy = true;
task.item.file.arrayBuffer().then(arrayBuffer => {             workerEntry.worker.postMessage({                 type: 'process_batch_item',                 id: task.item.id,                 name: task.item.name,                 format: task.item.format,                 buffer: arrayBuffer             }, [arrayBuffer]); // Transferable ArrayBuffer to prevent main thread copy         });     } }

---

3. Instant cached memory mounting without re-extraction

Re-parsing raw document files when switching focus between batch cards causes memory growth (~8GB) and introduces latency.

Mount pre-extracted gxDoc IR and pre-computed HTML directly from the batch cache store:

export async function focusBatchItem(itemId, slotNum = 1) {
    const item = batchQueue.getItem(itemId);
    if (!item || item.status !== 'completed') return;
const slot = slotNum === 1 ? state.pdf1 : state.pdf2;
// 1. Mount pre-extracted cached results directly into slot state     slot.file = item.file;     slot.extractedHTML = item.extractedHTML || '';     slot.extractedText = item.extractedText || '';     slot.bytes = item.bytes || null;     slot.gxDoc = item.gxDoc || null;
// 2. Hydrate DOM view surfaces instantly from cached IR/HTML (<10ms)     applyHtmlEverywhere(slot.extractedHTML || '<p>No extracted content available.</p>', null);
if (item.format === 'pdf' && item.bytes) {         await renderPDFToCanvas(item.bytes, 'pdf-canvas-container');         switchView('pdf');     } else {         switchView('html');     } }

Rule of thumb: Offload multi-document parsing to Web Workers using Transferable ArrayBuffers, and mount pre-computed IR cache directly to state on focus events to achieve instant UI responsiveness.