AUTO-GRADING SYSTEM COMPREHENSIVE AUDIT REPORT
Date: May 8, 2026
Scope: Full architecture audit covering Engine, Workers, UI, WASM, PNG handling, and compilation pipeline
1. CRITICAL INITIALIZATION COMPATIBILITY ISSUES
1.1 Two Competing Worker Implementations - COLLISION RISK [WARN]
Location: GradingPage.jsx line 3
import GradingWorker from '../worker/grading-engine.worker.ts?worker';Problem:
- File conflict:
grading-worker.ts(v3.9) vsgrading-engine.worker.ts(v3.9) both exist - GradingPage only imports v3.9 (dynamic loader), never uses v3.9 (direct WASM)
- v3.9 is orphaned: No UI is calling it, but it's maintained and has different error handling
- Import statement inconsistency: v3.9 uses dynamic imports; v3.9 uses static imports
- Message handler incompatibility: Slightly different log message formats
Impact:
- If developers accidentally call v3.9 worker, it will fail silently with missing emulator exports
- Maintenance nightmare: Two implementations doing similar work but diverging
What happens if both workers initialize:
- v3.9 tries direct imports:
import { FullCircuitValidator } from "@openhw/emulator/..." - v3.9 tries dynamic imports:
emulatorExports = await import("@openhw/emulator") - Both compete for WASM initialization, one will crash with "emulatorExports is undefined"
1.2 WASM Initialization Race Condition - Engine Crash 🔴
Location: Both workers (grading-worker.ts line 35, grading-engine.worker.ts line 45)
Problem:
let isInitialized = false;
async function initEngine() {
if (isInitialized) return; // ← RACE CONDITION HERE
console.log("Initializing WASM Engine...");
await init(wasmUrl);
init_panic_hook();
isInitialized = true;
}Issue:
- If two messages arrive before
isInitializedis set totrue, both will callawait init(wasmUrl) - This causes double initialization, leading to:
- Memory leaks (WASM module loaded twice)
- Conflicting panic hooks
- Undefined behavior in WASM bindings
v3.9 Dynamic Loader Issue:
async function initEngine() {
if (isInitialized) return; // ← SAME PROBLEM
wasmExports = await import('../wasm/grading/...'); // First caller sets this
const wasmUrl = wasmUrlMod.default;
await wasmExports.default(wasmUrl);
if (wasmExports.init_panic_hook) wasmExports.init_panic_hook(); // Can fire twice
isInitialized = true;
}Fix Needed: Use a Promise to serialize initialization
let initPromise: Promise<void> | null = null;
async function initEngine() {
if (initPromise) return initPromise;
initPromise = (async () => {
// ... actual init code
})();
return initPromise;
}1.3 Emulator Exports Not Guaranteed at Message Time - v3.9 WASM Reference Crash
Location: grading-engine.worker.ts line 320+
Problem:
onmessage = async (e: MessageEvent<GradingMessage>) => {
await initEngine(); // ← initEngine only initializes WASM, not emulatorExports in all cases
// ...
const validator = new emulatorExports.FullCircuitValidator(studentMeta);
// ^^^^^^^^^^^^^^^^ CRASH: emulatorExports might still be undefined!Why it crashes:
initEngine()in v3.9 does:emulatorExports = await import("@openhw/emulator");- But if
import()fails silently or returns empty object,emulatorExportsis still falsy - Later code tries to use
emulatorExports.FullCircuitValidator→ TypeError
v3.9 Comparison (More Robust):
// Direct imports at top level catch errors immediately:
import { FullCircuitValidator } from "@openhw/emulator/src/circuit-validation/engine.js";Fix Needed:
- v3.9 should throw if emulatorExports is missing
- v3.9 should be the primary implementation
- Remove v3.9 or properly validate initialization
1.4 Polyfill Side-Effects Breaking Worker Context - v3.9
Location: grading-engine.worker.ts lines 1-10
Problem:
if (typeof window === 'undefined') {
(self as any).window = self; // ← Creates artificial window object
(self as any).document = {
createElement: () => ({ style: {} }),
// ... incomplete mock
};
}
(self as any).$RefreshReg$ = () => {}; // ← React refresh mocks
(self as any).$RefreshSig$ = () => () => (type: any) => type;Issues:
- Mocking
windowin a Worker context can confuse libraries that check fortypeof window - Some libraries specifically check
typeof window === 'undefined'to detect Workers - Incomplete
documentmock might break dynamic feature detection - React refresh mocks are unnecessary in a Worker context
Risk:
- If a library detects
windowexists, it might try DOM operations → crash @xenova/transformers(used in AI worker) might have different behavior with fake window
2. PNG IMPORT/EXPORT COMPATIBILITY ISSUES
2.1 PNG Metadata Marker Inconsistency - Import/Export Mismatch
Location:
- Export: SimulatorPage.jsx line 7853
const MARKER = '\x00OPENHW_META\x00'; - Import: SimulatorPage.jsx line 8678
const marker = '\x00OPENHW_META\x00';
Good News: Markers match [YES]
But Issue: Metadata appending happens AFTER PNG encode:
// export
out.toBlob(async (blob) => {
const pngBuf = await blob.arrayBuffer();
const pngBytes = new Uint8Array(pngBuf);
const metaBytes = new TextEncoder().encode(jsonPayload);
const combined = new Uint8Array(pngBytes.length + metaBytes.length);
combined.set(pngBytes);
combined.set(metaBytes, pngBytes.length);Problem: If PNG is > 2GB (unlikely but worth noting), or contains binary data that looks like UTF-8 text:
- Import searches from END backwards:
for (let i = bytes.length - markerBytes.length; i >= 0; i--) - This is actually correct and safe [YES]
However: UTF-8 Decoding Issue:
const jsonStr = new TextDecoder('utf-8', { fatal: false }).decode(payloadBytes);- Using
fatal: falsemeans invalid UTF-8 is silently replaced with replacement character - Then
JSON.parse(jsonStr)silently fails or returns garbage - No error handling after decode!
Fix Needed:
try {
const jsonStr = new TextDecoder('utf-8', { fatal: true }).decode(payloadBytes);
const meta = JSON.parse(jsonStr);
} catch (err) {
alert('PNG metadata is corrupted or invalid UTF-8. ' + err.message);
return;
}2.2 PNG Export Cache Not Transferred Between Sessions
Location: SimulatorPage.jsx line 7760
const _exportPngResultCache = new Map(); // ← Session-only cache
// ...
if (cached && (Date.now() - cached.createdAt) < CACHE_TTL) {
// Returns cached PNG
}Problem:
- Cache is per-session, lost on page refresh
- WASM PNG export might be slower than expected because cache resets
- No localStorage fallback for large exports
Not a blocker, but inefficiency
2.3 SVG Sanitization for WASM Export Missing Edge Case
Location: SimulatorPage.jsx line 8085
Code:
assets[type] = svg.outerHTML
.replace(/ /g, ' ')
.replace(/<br\s*\/?>/gi, ' ');Problem:
- Only handles
and<br>tags - USVG (Rust XML parser) is strict and might reject:
- HTML entities like
©,®(only XML standard entities allowed:<,>,&,",') - SVG with namespaced attributes like
xlink:hrefif not properly declared - Comments or processing instructions
- HTML entities like
Fix: More comprehensive SVG sanitization needed before WASM
3. PROJECT COMPILATION PIPELINE ISSUES
3.1 projectCompilerUtils.js - Board Discovery Fragility
Location: projectCompilerUtils.js lines 40-50
export function getBoardCompileFiles(project, boardId) {
const projectFiles = normalizeProjectFiles(project.projectFiles);
let targetBoardId = boardId;
// Safety fallback for board discovery
const hasFilesForBoard = projectFiles.some(f => f.path.startsWith(`project/${targetBoardId}/`));
if (!hasFilesForBoard) {
const firstBoardFile = projectFiles.find(f => f.path.startsWith('project/'));
if (firstBoardFile) {
const match = firstBoardFile.path.match(/^project\/([^/]+)\//);
if (match) targetBoardId = match[1]; // ← Silently changes boardId!
}
}Problems:
- Silent fallback: If requested board doesn't exist, silently uses first board found
- No warning logged to worker
- Student submission for "arduino:uno" gets compiled as "rp2040:pico" without notification
- Inconsistent with grading: Teacher graded on "uno" but student compiled on "pico"
- No error propagation: Worker doesn't know compilation unit changed
Impact in Grading:
// In worker:
const boardCompId = boardComp?.id || 'uno1'; // Assumes 'uno1'
const compileUnit = getBoardCompileFiles(meta, boardCompId);
// If projectCompilerUtils silently uses 'pico1', comparison is between different circuits!Fix: Throw error or explicitly warn when board changes
if (!hasFilesForBoard && firstBoardFile) {
const msg = `Requested board '${boardId}' not found. Using '${newId}' instead.`;
console.warn(msg);
// Return { warning: msg, mainCode: "", ... } or throw
}3.2 Main File Detection Order Can Pick Wrong File
Location: projectCompilerUtils.js lines 53-61
const preferredMainName = `${targetBoardId}.ino`;
const main = boardFiles.find((f) => f.name === preferredMainName || f.path.endsWith(preferredMainName))
|| boardFiles.find((f) => fileExt(f.name) === '.ino')
|| boardFiles[0] // ← Picks ANY first file if no .ino found
|| null;Problem:
- If project has files:
[board.h, board.cpp](no .ino), picks.has main - Arduino compiler expects
.inoas entry point - Compilation fails with misleading error about .h syntax
Risk: Silent compilation failure if no .ino file exists
3.3 Disabled Files Not Filtered in Compilation Payload
Location: projectCompilerUtils.js line 68
const files = boardFiles
.filter((f) => !(main && f.path === main.path))
.map((f) => ({ name: f.name, content: f.content || '' }));Problem:
- Files ARE filtered to exclude main file
- But the
isFileDisabled()check is done on boardFiles (line 39) - If a file was re-enabled AFTER import, the disabled flag is stale
- Compiler might include .disabled files in the compilation payload
4. WORKER MESSAGE PROTOCOL INCOMPATIBILITIES
4.1 v3.9 vs v3.9 Response Format Drift
v3.9 Final Response:
postMessage({
type: 'GRADING_COMPLETE',
result,
teacherBinaryKey: teacherBinaryKey
});v3.9 Final Response:
(self as any).postMessage({ type: 'REPORT', report: finalResult });
postMessage({
type: 'GRADING_COMPLETE',
result,
teacherBinaryKey: teacherBinaryKey
});Issues:
- v3.9 sends TWO different message types (
REPORTandGRADING_COMPLETE) - GradingPage.jsx only listens for
GRADING_COMPLETE(line 44) REPORTmessage is lost/ignored- v3.9 sends extra data in
finalResultthat v3.9 doesn't:typescriptteacher_key: teacherBinaryKey, teacher_telemetry: teacher instanceof ArrayBuffer ? ... : null, student_telemetry: JSON.stringify(studentTelemetry), ai_status: "Skipped (Simulation Match Mode)", ai_score: 0
GradingPage.jsx expects:
if (e.data.type === 'GRADING_COMPLETE') {
if (e.data.result.logs) { // ← v3.9 has result.logs?
e.data.result.logs.forEach(log => addLog(log, 'info'));
}
// ...
const finalReport = e.data.result;Compatibility Failure: v3.9 result structure doesn't have .logs array
4.2 LOG Message Format Inconsistency
v3.9 Logs:
postMessage({ type: 'LOG', msg: `[v2.4] ${label} simulation timed out!` });v3.9 Logs:
postMessage({ type: 'LOG', msg: `[TRACE] ${label}: Starting capture behavior...` });Problem:
- Log parsing in GradingPage might expect specific format
- Timestamps inconsistent
- Severity levels different (no logType in some places)
4.3 KEY_GENERATED Message Not Handled Identically
v3.9:
postMessage({ type: 'KEY_GENERATED', key });v3.9:
postMessage({ type: 'KEY_GENERATED', key });Same format, but v3.9 might send additional data in future updates while v3.9 doesn't.
5. WASM MODULE INITIALIZATION SEQUENCING
5.1 Panic Hook Setup Order Issue
Location: Both workers after WASM init
await init(wasmUrl);
init_panic_hook(); // ← Must be called AFTER init() completesProblem:
- If
init()fails silently,init_panic_hook()still executes - Panic hook might not be attached to the WASM module if init failed
- Later panic crashes the worker with no diagnostic info
5.2 Extract Project Meta Called Before WASM Guarantee
Location: grading-worker.ts line 235 (right after initEngine)
await initEngine();
if (type === 'GENERATE_KEY') {
// ...
projectJson = extract_project_meta(new Uint8Array(buf as any));
// ^^^^^^^^^^^^^^^^^^^^^^ What if initEngine failed silently?Problem:
initEngine()doesn't throw on failure in either version- If WASM wasn't loaded,
extract_project_metafunction doesn't exist - Calls
undefined()→ TypeError
6. CROSS-WORKER DEPENDENCY ISSUES
6.1 AI-Audit Worker Spawned Without Guarantee of Results
Location: GradingPage.jsx line 52-83
const aiWorker = new Worker(new URL('../worker/ai-audit-final.worker.ts', import.meta.url), { type: 'module' });
aiWorker.onmessage = (aiEvent) => {
const aiData = aiEvent.data;
if (aiData.type === 'STATUS') { ... }
else if (aiData.type === 'RESULT') { ... }
};
aiWorker.postMessage({
type: 'GRADE_SEMANTICS',
teacherTelemetry: finalReport.teacher_telemetry,
studentTelemetry: finalReport.student_telemetry,
idMapping: finalReport.id_mapping
});Issues:
- No timeout for AI worker
- If AI model fails to load, no recovery
- AI worker references
idMappingthat might not exist in v3.9 response
6.2 Emulator Exports Used But Not Validated
Location: grading-engine.worker.ts line 313+
// After dynamic import:
const validator = new emulatorExports.FullCircuitValidator(studentMeta);Problem:
emulatorExportscould be{ FullCircuitValidator: undefined }- No null check before usage
- Crashes with "Cannot create an instance of undefined"
7. TELEPORTABLE OBJECTS & WASM BOUNDARY ISSUES
7.1 Binary Key Type Confusion
GradingPage.jsx line 303:
let teacherData;
if (teacherKeyCacheRef.current.hash === fileHash && teacherKeyCacheRef.current.key) {
teacherData = teacherKeyCacheRef.current.key; // ← What type is this?
} else {
teacherData = await teacherFile.arrayBuffer(); // ← ArrayBuffer
}Problem:
- First time:
teacherDataisArrayBuffer - After caching:
teacherDataisUint8Array(fromgenerate_binary_key()return) - Worker expects consistent type for
GENERATE_KEYvsGRADE
In Worker:
if (teacher instanceof ArrayBuffer) { // ← This check assumes ArrayBuffer
const teacherMetaJson = wasmExports.extract_project_meta(new Uint8Array(teacher));
} else {
teacherBinaryKey = new Uint8Array(teacher as any); // ← Falls back to Uint8Array
}Issue: Logic works but is fragile. If cache stores wrong type, extraction fails silently.
7.2 Transferable Objects Transfer Rules Violated
GradingPage.jsx line 310:
const transferables = [];
if (teacherData instanceof ArrayBuffer) transferables.push(teacherData);
else if (teacherData && teacherData.buffer instanceof ArrayBuffer) transferables.push(teacherData.buffer);
if (studentBuf instanceof ArrayBuffer) transferables.push(studentBuf);
workerRef.current.postMessage({
type: 'GRADE',
teacher: teacherData,
student: studentBuf,
options
}, transferables);Problem:
- If
teacherDataisUint8Array, the buffer is transferred but the Uint8Array reference in UI becomes detached - Subsequent cache operations on detached
Uint8Arraymight fail - Should deep-clone instead of transferring for cache data
8. ERROR RECOVERY & CRASH HANDLING
8.1 No Recovery Path for Partial WASM Load Failure
Location: Both workers initEngine()
Scenario:
- WASM blob fails to download (network error)
init(wasmUrl)silently fails or throws- No try-catch around
init()call - Worker crashes
async function initEngine() {
if (isInitialized) return;
await init(wasmUrl); // ← No error handling
init_panic_hook();
isInitialized = true;
}8.2 Silent Failure in captureBehavior Function
Location: Both workers lines 131+
catch (err) {
postMessage({ type: 'LOG', msg: `[v2.3] Warning: ${label} simulation failed: ${err}`, logType: 'warning' });
return telemetry; // ← Returns EMPTY telemetry!
}Problem:
- If simulation crashes, telemetry is
{ events: [], serial: "", duration_ms: ... } - Grading proceeds with empty telemetry → all scores become 0
- UI shows "0% behavioral match" but doesn't know simulation crashed
- No way to distinguish "perfect code" from "compilation error"
9. PROJECTCOMPILERUTILS PAYLOAD STRUCTURE ISSUES
9.1 Source Code Detection Fragile
Location: grading-worker.ts lines 120-125
const sourceCode = compileUnit.mainCode || "";
const isSourceCode = sourceCode.startsWith('#') ||
sourceCode.includes('void setup') ||
sourceCode.includes('void loop') ||
sourceCode.includes('int main');Problems:
- Comments starting with
#are not C/C++ directives (should be only#include,#define, etc.) - String literals containing "void setup" trigger false positive:
const msg = "void setup() {...}" - No check for Arduino
.inosyntax (which is C++)
Better detection:
const hasArduinoMarkers = /\b(void\s+setup|void\s+loop|pinMode|digitalWrite|Serial\.begin)\s*\(/.test(sourceCode);
const hasInclude = sourceCode.includes('#include');9.2 File Extension Check Case Sensitivity
Location: projectCompilerUtils.js line 38
const allowed = new Set(['.ino', '.h', '.hpp', '.c', '.cpp']);
// ...
.filter((f) => allowed.has(fileExt(f.path)));
export function fileExt(path) {
const idx = String(path || '').lastIndexOf('.');
return idx >= 0 ? path.substring(idx).toLowerCase() : ''; // ← Converts to lowercase
}This is good - case insensitive [YES]
But issue: What if Windows file system has .INO (uppercase)?
fileExt()converts to.ino[YES]- File is included [YES]
- Arduino compiler might have issues with uppercase extensions on Linux [WARN]
10. AI SEMANTIC WORKER INITIALIZATION
10.1 Transformers Library Might Fail to Load
Location: ai-audit-final.worker.ts line 30+
async function initTransformers() {
if (pipeline) return;
const transformers = await import('@xenova/transformers');
pipeline = transformers.pipeline;
env = transformers.env;Problems:
- If
@xenova/transformersis not bundled, import fails - No error message guides user to install it
- AI worker silently dies
10.2 WASM Model Path Configuration
Location: ai-audit-final.worker.ts line 41
env.backends.onnx.wasm.wasmPaths = '/'; // ← Assumes WASM files at root
env.localModelPath = '/models/'; // ← Assumes models in public/models/Problem:
- If app is served from subpath (e.g.,
/app/grading), paths are wrong - WASM files and models will 404
- No fallback URL resolution
11. SUMMARY TABLE OF CRITICAL vs HIGH ISSUES
| Issue | Severity | Component | Impact |
|---|---|---|---|
| Two competing workers | CRITICAL | Architecture | Engine crash if wrong worker selected |
| WASM init race condition | CRITICAL | Workers | Double initialization, undefined behavior |
| Missing emulatorExports | CRITICAL | v3.9 worker | TypeError when grading |
| PNG metadata decode no error handling | HIGH | SimulatorPage | Silent data loss |
| Silent board discovery fallback | HIGH | projectCompilerUtils | Teacher vs student mismatch |
| No recovery for WASM load failure | HIGH | Workers | Worker crash on network error |
| Empty telemetry on simulation crash | HIGH | Workers | Misleading scores (0% for both success and crash) |
| Key type confusion (ArrayBuffer vs Uint8Array) | MEDIUM | GradingPage | Fragile caching, potential crashes |
| Panic hook not guaranteed | MEDIUM | Workers | Silent crashes in WASM |
| AI worker no timeout | MEDIUM | GradingPage | UI hangs if AI model fails |
12. RECOMMENDED FIXES (Priority Order)
IMMEDIATE (Before Next Production Deployment)
Remove v3.9 or make v3.9 compatible
- Delete grading-worker.ts OR
- Make both identical implementations
Fix WASM initialization race
- Use Promise serialization
- Add explicit error handling
Fix emulatorExports undefined
- Add null checks before usage
- Throw clear error if missing
Add PNG metadata decode error handling
- Try-catch around JSON.parse
- Log corruption errors
SHORT TERM (Next Sprint)
- Fix silent board discovery
- Improve source code detection
- Add recovery for WASM load failures
- Distinguish simulation crash from success in telemetry
MEDIUM TERM
- Refactor to single worker implementation
- Add comprehensive error recovery in all workers
- Improve AI worker timeout and fallbacks
13. EXPECTED FAILURE SCENARIOS
These scenarios WILL crash the engine:
User runs grading twice in quick succession
- Race condition in WASM init causes double initialization
- Panic hooks conflict
- Worker dies
PNG with UTF-8 metadata encoding issues
- Invalid characters in project data
- Silent JSON parse failure
- Wrong circuit loaded
Project missing .ino file
getBoardCompileFilespicks .h as main- Compiler rejects .h syntax
- Confusing error message
Network latency during WASM load
await init(wasmUrl)times out- No error propagation
- Worker hangs silently
Teacher uploaded for board "uno", student uploaded for board "pico"
- Silent board discovery fallback
- Both compiled as different boards
- Comparison fails with 0% match
- No warning that boards don't match
FILE LOCATIONS REFERENCE
openhw-studio-grading-engine/
├── src/lib.rs # Rust grading logic
├── Cargo.toml # WASM build config
└── pkg/ # Compiled WASM outputs
├── openhw_studio_grading_engine.js # WASM bindings
└── openhw_studio_grading_engine_bg.wasm # Binary WASM
OpenHW-studio-frontend/
├── src/worker/
│ ├── grading-worker.ts # v3.9 (Primary)
│ ├── grading-engine.worker.ts # v3.9 (Dynamic - PROBLEM)
│ └── ai-audit-final.worker.ts # AI semantic worker
├── src/wasm/
│ ├── grading/openhw_studio_grading_engine.js
│ └── grading/openhw_studio_grading_engine_bg.wasm
├── src/pages/
│ ├── GradingPage.jsx # Main UI coordinator
│ └── simulationpage/
│ ├── SimulatorPage.jsx # PNG import/export logic
│ └── utils/projectCompilerUtils.js # Compilation pipeline
└── src/utils/projectCompilerUtils.js # Source of truth for filesCONCLUSION
The auto-grading system has multiple critical compatibility issues that can cause silent failures or crashes:
- Architecture level: Two worker implementations competing
- Initialization level: Race conditions in WASM setup
- Data pipeline level: Silent fallbacks and missing error handling
- Protocol level: Message format drift between implementations
Recommend immediate action on Critical items before any student-facing use.
