OpenHW Studio — Compiler Backend

The Express.js REST API server that compiles Arduino C++ sketches into .hex machine code using arduino-cli, and handles user authentication and data storage via MongoDB.

Overview
Tech Stack
Project Structure
API Endpoints
Key Features
Setup & Running Locally
Environment Variables

Overview

The Compiler Backend is the central API server for OpenHW Studio. It:

Accepts Arduino C++ source code from the frontend
Invokes arduino-cli to compile it into an AVR .hex file
Returns the .hex payload to the frontend for simulation
Handles user authentication (login / registration) using JWT + bcrypt
Manages library installation via arduino-cli library commands
Connects to MongoDB for user and project data persistence
Receives and reviews custom component submissions from users

The server runs on http://localhost:5001.

Note on offline usage: The frontend now caches compiled .hex results in IndexedDB (browser-side). Once a sketch has been compiled at least once, subsequent runs with the same code bypass this server entirely and run from the local cache. See OFFLINE_AND_STORAGE.md for details.

Tech Stack

Technology	Purpose
Node.js + Express	Web server and REST API
arduino-cli	Compiling Arduino C++ to AVR `.hex`
Mongoose	MongoDB object modelling
JSON Web Tokens (JWT)	Stateless authentication
bcryptjs	Password hashing
dotenv	Environment variable management
cors	Cross-origin request handling
nodemon	Auto-reload during development

Project Structure

openhw-studio-backend/
├── src/
│   ├── server.js                   # Entry point — Express app setup & startup
│   ├── controllers/
│   │   ├── compileController.js    # POST /api/compile — runs arduino-cli
│   │   ├── libController.js        # Library search & install via arduino-cli
│   │   ├── componentController.js  # Asset Registry & Approval pipeline
│   │   └── userController.js       # Admin & User profile management
│   ├── routes/
│   │   └── api.js                  # Route definitions
│   ├── models/                     # Mongoose data models (users, projects, etc.)
│   ├── db/                         # MongoDB connection setup
│   └── middleware/                 # Auth middleware (JWT verification)
├── temp/                           # Temporary .ino/.hex files (auto-cleaned)
├── env                             # Environment variables file (not committed)
├── nodemon.json                    # Nodemon config (ignores temp/)
├── package.json
└── .gitignore

API Endpoints

Compilation

Method	Endpoint	Description
`POST`	`/api/compile`	Compile Arduino C++ code → returns `.hex`
`GET`	`/api/compile/pico/micropython-uf2`	Fetch cached default Pico MicroPython UF2 payload (CORS-safe via backend)

Request body:

json

{
  "code": "#include <Arduino.h>\nvoid setup() { ... }\nvoid loop() { ... }"
}

Response:

json

{
  "hex": ":100000000C945C000C9479000C94790..."
}

The frontend caches successful responses in IndexedDB (openhw-offline / hexCache). Repeated runs with unchanged code skip this endpoint entirely.

Library Management

Method	Endpoint	Description
`GET`	`/api/lib-search?q=Servo`	Search arduino-cli library index
`POST`	`/api/lib-install`	Install a named library via arduino-cli
`POST`	`/api/lib-uninstall`	Uninstall a named library
`GET`	`/api/lib-list`	List all installed libraries

Component Registry & Pipeline

Method	Endpoint	Description
`POST`	`/api/components/submit`	Users upload a custom component (ZIP content) for review
`GET`	`/api/admin/components/pending`	List all submissions waiting for admin approval
`POST`	`/api/admin/components/approve`	Permanently merge a submission into the emulator library
`DELETE`	`/api/admin/components/reject/:submissionId`	Reject a specific submission by its unique ID
`GET`	`/api/admin/components/installed`	List all manually installed custom components
`DELETE`	`/api/admin/components/installed/:id`	Remove an installed component from the emulator
`GET`	`/api/admin/components/backup`	Full export of all installed components with source files

Component submissions made while offline are queued in the browser's IndexedDB and auto-submitted when connectivity is restored. The backend endpoint is unchanged.

Authentication

Method	Endpoint	Description
`POST`	`/api/user/signup`	Register a new user
`POST`	`/api/user/signin`	Authenticate and receive JWT
`POST`	`/api/user/logout`	Invalidate session
`GET`	`/api/user/profile`	Get current user profile details
`PUT`	`/api/user/profile`	Update user profile

Key Features

Compilation Pipeline (`compileController.js`)

Receives raw C++ code via POST /api/compile
Creates a uniquely named temporary .ino sketch file in temp/
Executes arduino-cli compile targeting the arduino:avr:uno FQBN via Node's child_process.execFile
Extracts the generated .hex content from the build output
Sends the .hex string back to the frontend
Cleans up the temporary directory recursively to prevent disk bloat

The frontend caches this response in IndexedDB. Subsequent Run clicks with the same code are served from the browser cache and never reach this endpoint.

Library Management (`libController.js`)

Wraps arduino-cli lib search and arduino-cli lib install as API endpoints
Allows the frontend's Library Manager UI to search and install Arduino libraries at runtime

Component Review Pipeline (`componentController.js`)

Unique submissionIds: Each upload gets a timestamped ID so rejecting one doesn't affect other pending versions.
Permanent Integration: Approval physically writes the ui.tsx, logic.ts, etc., to the emulator's component directory and updates its registry.
Atomic Rejection: One-click removal of specific submissions from the in-memory pending store.
Offline-Tolerant: The frontend queues failed submissions locally and retries automatically — no data is lost if the backend is temporarily unreachable.

Stability: nodemon + temp/ Ignore

The nodemon.json explicitly ignores the temp/ directory. Without this, file changes inside temp/ (created during active compilation) would cause nodemon to restart the server mid-compilation, resulting in ERR_CONNECTION_RESET errors on the frontend.

json

{
  "ignore": ["temp/"]
}

Setup & Running Locally

Prerequisites

Node.js 18+
npm 9+
MongoDB running locally (or a MongoDB Atlas connection string) for authentication and classroom features

arduino-cli installed and on your system PATH (or placed in the project root)

Download: https://arduino.github.io/arduino-cli/latest/installation/

After installing, run:

bash

arduino-cli config add board_manager.additional_urls https://github.com/earlephilhower/arduino-pico/releases/download/global/package_rp2040_index.json
arduino-cli core update-index
arduino-cli core install arduino:avr
arduino-cli core install rp2040:rp2040

If MongoDB is unavailable, the backend now starts in degraded mode so compile, library, examples, and component endpoints can still be used.

Installation

bash

cd openhw-studio-backend
npm install

Configure Environment

Create a file named .env in the project root (this file is gitignored):

env

PORT=5001
MONGO_URI=mongodb://localhost:27017/openhw-studio
JWT_SECRET=your_secret_key_here
SESSION_SECRET=your_session_secret_here
EXAMPLES_PATH=../path-to-openhw-studio-examples/examples
EMULATOR_PATH=../path-to-openhw-studio-emulator
FRONTEND_URL=http://localhost:5173

Start the Server

bash

# Development (auto-reload with nodemon)
npm run dev

# Production
npm start

Server will be available at http://localhost:5001

Environment Variables

The backend uses a .env (or env) file for configuration. Create one in the root directory based on the table below:

Variable	Description	Default
`PORT`	Port the Express server listens on	`5001`
`MONGO_URI`	MongoDB connection string (local or Atlas)	—
`JWT_SECRET`	Secret key for signing JSON Web Tokens	—
`JWT_EXPIRES_IN`	JWT expiration time (e.g., `7d`)	`7d`
`SESSION_SECRET`	Secret key for Passport/express-session (required)	—
`GOOGLE_CLIENT_ID`	Google OAuth 2.0 Client ID	—
`GOOGLE_CLIENT_SECRET`	Google OAuth 2.0 Client Secret	—
`GOOGLE_CALLBACK_URL`	Authorised redirect URI for Google Login	`http://localhost:5001/auth/google/callback`
`FRONTEND_URL`	Frontend URL for CORS configuration	`http://localhost:5173`
`FRONTEND_URLS`	Comma-separated frontend origins for CORS allow-list	`http://localhost:5173,http://127.0.0.1:5173`
`EXAMPLES_PATH`	Path to the examples directory served at `/examples` (absolute or relative to the backend root)	`../openhw-studio-examples/examples`
`EMULATOR_PATH`	Path to the emulator repo root; the backend uses `src/components` inside it	`../openhw-studio-emulator`
`EMULATOR_COMPONENTS_PATH`	Optional direct path to the emulator component directory	—
`PICO_MICROPYTHON_UART0_UF2_URL`	Preferred source (URL or local file path) for UART0-enabled Pico MicroPython UF2	—
`PICO_MICROPYTHON_UF2_URL`	Backward-compatible alias for Pico MicroPython UF2 source (URL or local path)	—
`PICO_MICROPYTHON_CACHE_TTL_MS`	Cache lifetime (milliseconds) for backend-fetched Pico UF2 asset	`21600000`

Notes:

Paths may be absolute or relative to the backend repository root.
Use EMULATOR_COMPONENTS_PATH if your emulator layout differs from the standard src/components structure.
For rp2040js MicroPython simulation, configure a UART0 REPL UF2 build. The official micropython.org Pico UF2 uses USB CDC REPL and is intentionally rejected.

Sample `.env` Setup:

env

PORT=5001
MONGO_URI=mongodb://localhost:27017/openhw-studio
JWT_SECRET=your_secret_key_here
JWT_EXPIRES_IN=7d
SESSION_SECRET=your_session_secret_here
GOOGLE_CLIENT_ID=your_id.apps.googleusercontent.com
GOOGLE_CLIENT_SECRET=your_secret_here
GOOGLE_CALLBACK_URL=http://localhost:5001/auth/google/callback
FRONTEND_URL=http://localhost:5173
EXAMPLES_PATH=../path-to-openhw-studio-examples/examples
EMULATOR_PATH=../path-to-openhw-studio-emulator

OpenHW Studio — Compiler Backend ​

Table of Contents ​

Overview ​

Tech Stack ​

Project Structure ​

API Endpoints ​

Compilation ​

Library Management ​

Component Registry & Pipeline ​

Authentication ​

Key Features ​

Compilation Pipeline (compileController.js) ​

Library Management (libController.js) ​

Component Review Pipeline (componentController.js) ​

Stability: nodemon + temp/ Ignore ​

Setup & Running Locally ​

Prerequisites ​

Installation ​

Configure Environment ​

Start the Server ​

Environment Variables ​

Sample .env Setup: ​

OpenHW Studio — Compiler Backend

Table of Contents

Overview

Tech Stack

Project Structure

API Endpoints

Compilation

Library Management

Component Registry & Pipeline

Authentication

Key Features

Compilation Pipeline (`compileController.js`)

Library Management (`libController.js`)

Component Review Pipeline (`componentController.js`)

Stability: nodemon + temp/ Ignore

Setup & Running Locally

Prerequisites

Installation

Configure Environment

Start the Server

Environment Variables

Sample `.env` Setup: