Spid3r Logo
Spid3r Documentation
WASM Loading...
/
โšก In-Browser Runtime

01. Introduction to Spid3r

Spid3r is a modern, high-performance, memory-safe, embeddable general-purpose programming language built from scratch in Rust with zero external dependencies.

It combines clean, expressive C/JavaScript-style syntax with the safety guarantees of managed memory, the mathematical power of NumPy/BLAS-style contiguous matrices and vectors, the concurrency model of Erlang/Go channels, and an instant WebAssembly compiler that lets you run complete programs inside any browser.

๐Ÿ“– How to read this manual

Each chapter gives you the essentials up front โ€” what it is, how to use it, and what matters. Dense reference material (full contracts, wire formats, security models, build internals) is collapsed into deep-dive blocks marked โ–ธ Deep dive โ€” โ€ฆ: the main topic stays visible, and the complete detail is one click away. Nothing is left out; it is just not in your face. All code samples are runnable as shown.

๐Ÿ’ก Key Design Tenets
  • Zero External Dependencies: The entire compiler, bytecode virtual machine, AST interpreter, standard library (including cryptographic primitives, regex engines, date-time calendars, and compression algorithms) is written in pure, self-contained Rust.
  • Dual Execution Engines: Run programs with the high-speed --engine vm (bytecode register/stack virtual machine) or the pure reference --engine tree (AST tree-walk interpreter) with 100% byte-for-byte behavioral parity (program stdout, exit codes, and side effects).
  • Zero-Leak Tracing Garbage Collection: Deterministic mark-and-sweep cycle collector with interior-mutable cell headers and weak references.
  • Native WebAssembly Core: Zero-overhead compilation target exportable as a 100% self-contained linear-memory engine running in web browsers, Node.js, Deno, and Edge Workers.

๐Ÿ“ฆ Installing Spid3r

Spid3r is lightweight and self-contained: one binary, no runtime to install, and no Visual C++ Redistributable (the C runtime is linked statically). Only GUI mode borrows the system webview โ€” WebView2 on Windows, WebKit on macOS/Linux. You can install it on Windows, Linux, or macOS in under a minute:

Supported Platforms & Installation Methods Release

๐Ÿง / ๐Ÿ 1. Linux & macOS (One-Line Automated Installer)

Run this single command in your terminal. It detects your operating system and CPU architecture, downloads the release tarball, installs both spid3r and the self-hosted compiler spid3r-self into ~/.spid3r/bin (override with --dir <path> or SPID3R_INSTALL_DIR), deletes any stdlib/ folder left by an older install (the standard library is compiled into the binary), adds that directory to your shell PATH (--no-modify-path to skip), and verifies execution:

curl -fsSL https://raw.githubusercontent.com/needyamin/spid3r/main/install.sh | sh

๐ŸชŸ 2. Windows (Interactive GUI Setup Installer)

Download and run Spid3r-Setup-<version>.exe from the official releases. The automated installer configures:

  • Adds spid3r-self.exe (self-hosted native AOT compiler & dev CLI), spid3r.exe (VM/interpreter/REPL/LSP), and spid3r-control.exe to your system PATH.
  • Bundles LLVM-C.dll, LLVM-C.lib, and the SHA-256-pinned runtime staticlib (spid3r_lang.lib + spid3r_lang.lib.sha256 + runtime-manifest.json) for out-of-the-box AOT compilation.
  • Registers .matrix source files so double-clicking executes them, with all 29 stdlib modules compiled directly into the binaries.

๐Ÿฆ€ 3. Build from Source via Cargo (Any Platform)

If you have the Rust toolchain installed, build and install directly into your user binary folder (~/.cargo/bin):

# Install zero-dependency core toolchain
cargo install --path . --no-default-features

# Or build with Desktop GUI window support (Tao/Wry)
cargo install --path . --no-default-features --features gui

โœ… 4. Verify Installation

Open a new terminal or command prompt window and verify that the spid3r command is accessible:

$ spid3r version

Rust host, no runtime dependencies.

Your First Spid3r Program

Entry point: a program executed with spid3r run must define fn main() { ... } โ€” it is called automatically, so do not invoke it yourself. The โšก Try it Yourself runner and playground execute top-level code directly (they synthesize an empty main if needed). If you copy a snippet to a .matrix file and run it on the CLI, wrap the statements in fn main() { ... return 0; }. Helper fn/class/enum declarations stay at top level.

Let's start with a classic "Hello, World!" program. You can execute this code immediately in your browser by clicking โšก Try it Yourself ยป below:

โšก Example: Hello World & String Output
// 1. Hello World in Spid3r fn main() { println("Hello, World!"); let language = "Spid3r"; let year = 2026; println("Welcome to the " + language + " developer guide!"); return 0; }

02. How Spid3r Works: Compiler & VM Pipeline

Understanding the Spid3r internal execution pipeline helps you write highly optimized code and leverage both the VM and AOT native backends.

Source passes through lexer, parser, semantic checker and lowering into a checked AST; four backends consume it โ€” tree walker, bytecode VM, LLVM native, and WebAssembly.
One front end, four backends. The tree walker is the reference oracle; the VM, native and WebAssembly backends must reproduce its stdout, error phase and exit code exactly.
Phase Component Responsibility Output Artifact
1. Lexical Analysis src/lexer/mod.rs DFA-based tokenizer recognizing radix literals (0x, 0o, 0b), scientific float notation (1.5e3), Unicode escapes (\u{1F600}), and desugaring string interpolation holes ({var}). Stream of Token structs with line/col spans
2. Syntax Parsing src/parser/mod.rs Top-down recursive descent + Pratt precedence parser with statement-level error recovery (parse_recovering reports up to 25 syntax errors in check/lsp, while run/build fail fast on the first). Enforces strict 64-level nesting depth. Abstract Syntax Tree (ast::Program)
3. Semantic Analysis src/semantic/mod.rs Scope resolution, constant folding, flow-sensitive nullability narrowing (T? → T), scrutinee-driven enum match exhaustiveness, member visibility checks (public / private), interface compliance, lints, and E1xxx codes with end spans. Validated AST or coded compiler diagnostic
4. Lowering Pass src/lower/mod.rs Desugars high-level syntax into core primitives: converts ?? into null checks, value-position if into branch blocks, for (k, v) in map into iterator lookups, and destructuring tuples into temporary bindings. Desugared Low-level AST
5. Execution Backends src/vm/, src/interpreter/ & src/native/ Bytecode VM: 60-opcode VM with optimizer and per-opcode + per-function self-time profiler.
Tree Interpreter: Reference AST oracle for differential parity.
Native AOT (LLVM 23): Scalar unboxing, direct field GEPs, and call-scoped bump-slab arena reclamation.
Live execution, captured stdout, exit status
6. Memory Management src/value/gc.rs & src/value/mod.rs Tracing mark-and-sweep garbage collection with cell tracking, weak references, and zero-leak cyclic object reclamation. Reclaimed heap cells

The Spid3r Toolchain CLI

The spid3r command-line executable provides a complete developer environment:

One .matrix source file feeds five delivery paths: spid3r run for direct execution, spid3r build for an AOT native executable, bundle --target wasm for a wasm module, spid3r gui for a desktop app on a native WebView, and the spid3r_lang crate for embedding in a Rust host.
One source, five outputs. Every path is the same front end; only the backend and the packaging change.
Command Usage Description
spid3r run <file.matrix> spid3r run --engine vm main.matrix Compile and execute program. Flags: --engine tree|vm, --prof (prints per-opcode execution counts plus per-function self-time and inclusive instruction attribution in ==== function profile (self-time) ====), --budget N (stop after roughly N steps with the fatal error execution budget exhausted โ€” for untrusted code), --max-steps N (documented alias of --budget; specify only one of the two), --max-call-depth N (cap the call stack depth โ€” VM and tree engines, default 1000, propagated to tasks and modules), --coverage (record line coverage), --no-opt (disable the compile-time bytecode optimizer and the VM runtime fused-superinstruction peeks; per-VM inline caches remain enabled), --sandbox (deny filesystem, network, process and environment access; denied calls raise a catchable sandboxed: โ€ฆ access denied error).
spid3r check <file.matrix> spid3r check main.matrix --deny-warnings Lex, parse (recovering up to 25 syntax errors in one pass), and run semantic checks (types, scopes, flow-sensitive nullability narrowing, scrutinee-driven enum exhaustiveness, unused/unreachable lints) without executing. Emits E1xxx codes, [line:col .. end_line:end_col] spans, and TTY-gated ANSI colors (respecting NO_COLOR).
spid3r compile <file.matrix> spid3r compile app.matrix -o app.matrixc Compile source file into standalone binary bytecode (.matrixc). Use --dump to inspect disassembled bytecode opcodes. Output is written in the version-3 container format with a SHA-256 integrity checksum over the payload.
spid3r disasm <file.matrixc> spid3r disasm app.matrixc Load a compiled bytecode file and print human-readable disassembly. Exits 4 if the file is corrupt or fails its integrity checksum.
spid3r build <file.matrix> spid3r build app.matrix --release Compile into a standalone native machine-code executable for your operating system. Flags: --release, -o <path>, --target <triple>, --icon app.ico, --subsystem console|windows, --link-lib <name|path>, --allow-gui, --embed <path>, --bytecode <file.matrixc>, --bundle-runtime, --product-name, --version.
spid3r exec <file.matrixc> spid3r exec app.matrixc Directly execute a pre-compiled bytecode file on the virtual machine with instant startup. Flags: --sandbox (same capability denials as run --sandbox โ€” this is the untrusted-bytecode distribution path). A corrupt or checksum-invalid .matrixc exits with code 4.
spid3r fmt <files...> spid3r fmt --check src/ Canonical code formatter. Idempotent and preserves code semantics. --check exits 1 if files require formatting.
spid3r doc [paths...] spid3r doc src/ -o docs/api.html Extract /// and // doc comments above declarations (sharing doc_above / strip_comment with spid3r lsp) and emit a self-contained searchable HTML reference or deterministic JSON (--json). Pass --private to include non-public symbols.
spid3r repl spid3r repl Interactive Read-Eval-Print-Loop with raw-mode TTY cursor editing (Home, End, Left/Right, Delete, Ctrl-A, Ctrl-E, Ctrl-U, Ctrl-K, Ctrl-L) and 500-entry history recall (Up/Down preserving drafts), falling back cleanly to line mode when stdin/stdout is not a terminal.
spid3r debug <file.matrix> spid3r debug app.matrix --break 12 Interactive CLI debugger: breakpoints, step, next, locals inspection, backtrace, and source listing.
spid3r gui <file.matrix> spid3r gui window.matrix Run an interactive desktop GUI application backed by native WebViews. Flags: --html <path>, --dry-run, --engine tree|vm (the VM also runs a .matrixc argument directly); or --payload <exe> to run a program embedded by build --embed straight out of the executable, with its assets served from memory (--persist keeps the webview profile on disk). A payload carrying a .matrixc entry runs on the VM.
spid3r bundle <file> spid3r bundle app.matrix --target wasm Bundle project into a distribution package (wasm, desktop, or android).
spid3r test [--filter S] [--list] spid3r test --filter math Run every examples/*.matrix (or a project's tests/) through the three-engine differential: tree, VM, and compile+exec must agree.
spid3r bench [--runs N] [--compare] spid3r bench --compare --runs 5 Time benchmarks/*.matrix (best of N); --compare pits tree against VM, --gate fails on a regression.
spid3r lsp spid3r lsp Language Server over stdio: diagnostics, hover, definition, references, rename, symbols, completion, semantic tokens, code actions and formatting.
spid3r info spid3r info --json Runtime information, one self-describing report. Flags: --json, --html, --no-redact (local operator only). Same collector as spid3rinfo().
spid3r config spid3r config get agent.port Read and write spid3r.ini. Subcommands: get KEY, set KEY VALUE, list, path, validate, reload, init.
spid3r config validate spid3r config validate Print validation problems (exit 1 when any, 0 when clean); a broken SPID3R_INI pin is reported as one of those problems. Local diagnostics keep policy values such as security.basedir and paths.* raw so operators can see which entry is wrong; credential-like values are hashed. An unknown flag (e.g. --bogus) is a usage error (exit 2).
spid3r config init [PATH] [--force] spid3r config init ./spid3r.ini --force Write a commented default ini. --help prints the config usage instead of writing a file named --help; without --force it refuses to overwrite an existing ini (exit 2). config save refuses to persist while the ini pin is broken.
spid3r control config validate spid3r control config validate Agent-side validation; exits 1 and prints errors to stderr when the config is valid:false (it no longer prints a blank line and exits 0).
spid3r agent spid3r agent start Local management agent. Subcommands: start, pair [--role R] [--name N], status, ops. Listens on [agent] bind:port (default 127.0.0.1:9527). No unrestricted shell.
spid3r admin spid3r admin --host web-1 ping Remote administration client. Pair first with spid3r admin pair <host:port> <code>, then call operations against the paired name.
spid3r init [dir] spid3r init my_project Create a project skeleton (spider.toml + src/main.matrix).
spid3r install [--update] spid3r install Resolve path, git, local-directory, and remote http(s):// registry dependencies (SPID3R_REGISTRY / packages.registry) into vendor/ and write spider.lock. Verifies mandatory sha256.txt hashes, enforces tar path-traversal safety, verifies Ed25519 sha256.txt.sig signatures against packages.trust_pk / SPID3R_TRUST_PK (or registry.pub), and unifies diamond version constraints across single-project and [workspace] manifests; --update re-resolves instead of honouring locked versions/commits.
spid3r add <name> spid3r add mathlib --path ../mathlib Declare a dependency in spider.toml and vendor it in one step. Sources: --path P, --git URL [--rev R], or --version V for the local registry. --version alongside a path/git source is a constraint. If the dependency does not resolve the manifest is restored byte-for-byte, so a typo cannot leave the project broken.
spid3r remove <name> spid3r remove mathlib Delete the declaration from [dependencies], prune vendor/<name>/, and rewrite spider.lock so the project stays verifiable.
spid3r tree spid3r tree Print the declared dependency tree with each package's locked version and origin. A path dependency's own manifest is followed, so transitive path dependencies appear nested.
spid3r verify spid3r verify Re-hash every vendored package against the sha256 recorded in spider.lock. Exits 1 and names the package, with both hashes, if a vendored file was edited by hand; also reports a vendored directory that no lock entry covers.
spid3r search <query> spid3r search math --local Find packages by name in a configured local directory registry (SPID3R_REGISTRY) and in vendor/ (remote http(s):// registries are fetched on spid3r install).
spid3r publish --to-dir <dir> spid3r publish --to-dir ../registry Write <dir>/<name>/<version>/ containing spider.toml, src/, package.tar, sha256.txt, and a detached Ed25519 signature sha256.txt.sig, plus sorted <dir>/<name>/index.txt and <dir>/registry.key / registry.pub (or sign with SPID3R_SIGN_KEY). Republishing an existing version is refused rather than overwritten.
spid3r plugin abi|list|check spid3r plugin check examples/plugins/hello-wasm Inspect the plugin contract: abi prints the version string this build implements, list shows what the plugin search path would load, and check <dir> validates one directory against ABI v1 using the loader's own code path, without registering anything. Exit codes: 0 valid, 1 invalid, 2 usage.
spid3r modules <file> spid3r modules main.matrix --tree List every module a program imports as origin<TAB>path, where origin is disk, vendor, embedded or plugin; --tree prints the dependency graph. Nothing is executed โ€” this is the command to reach for when a build fails and you need to know which file an import actually named.
spid3r dist [-o out] spid3r dist Compile the project entry named by spider.toml into target/<package>.matrixc.
spid3r pack / unpack spid3r pack -o dist/app.tar Archive spider.toml + src/ into a tarball, and extract one again with spid3r unpack <archive> [dir].
spid3r container <subcmd> spid3r container run alpine:latest /bin/sh OCI container engine (built with --features container; Linux runtime isolation). Subcommands: pull [--require-sig --pubkey <hex>], create [--name ID], start, run, stop, kill, rm, ps, inspect, logs, gc.
spid3r kernel build spid3r kernel build --out kernel.bin Build the zero-dependency no_std x86_64-unknown-none kernel crate (kernel/Cargo.toml) via cargo build --release and copy the ELF binary to --out (default kernel/build/kernel.bin).
spid3r version / help spid3r version Print the toolchain and language versions, or the full command reference.

Automated GitHub Actions Workflows

Every change, tag, and schedule triggers automated GitHub Actions workflows (.github/workflows) to guarantee multi-platform reproducibility, strict clippy checks, differential engine validation, and automated multi-architecture releases:

Spid3r GitHub Actions workflows: 6-job CI matrix covering Core, GUI, WASM, LLVM 23 Native AOT with Linux self-host bootstrap, Phase 0 measurement gates, and Self-host Parity Gate over the frozen corpus; Self-hosted Release pipeline building 4 tarballs and the Windows NSIS installer packaging spid3r-self, spid3r, and pinned runtime artifacts; and scheduled/PR cargo-fuzz regression suite.
Spid3r automated workflow architecture: Continuous Integration (ci.yml), self-hosted multi-architecture releases (release.yml), and scheduled/PR libfuzzer (fuzz.yml).

Module Resolution

Every backend resolves import "name" through one implementation, so the tree interpreter, the bytecode VM and the AOT compiler cannot disagree about which file an import names. The search order is fixed, and spid3r modules prints which root won:

# Root Origin label
1 The importing file's own directory disk
2 SPID3R_PATH (a ;-separated list) disk
3 ./stdlib, relative to the working directory disk
4 stdlib/ beside the executable, or above it disk
5 vendor/ from the nearest enclosing package vendor
6 The stdlib table compiled into the binary embedded

Each candidate is tried as written and then with a .matrix suffix. Disk always wins over the embedded table, which is what lets a project pin a patched stdlib/semver.matrix; the embedded table is the last resort that makes a single-file app work on a machine with no stdlib/ at all. An import spec may not escape these roots: an absolute path, a drive letter, angle brackets or a .. component is rejected with error[semantic]: invalid module path on every backend. (Only the tree walker enforced that before resolution was unified, so import "../x" used to load on two engines and fail on the third.)

A vendored package is a directory: vendor/mathlib/lib.matrix is imported as import "mathlib/lib". Which root a module came from โ€” and therefore how much you should trust it โ€” is the subject of the next section.

The Ecosystem: Packages, Plugins and Trust

Third-party code reaches a program through seven routes, and they do not carry the same authority. The tiers below are the whole trust model; anything not on this list is not a supported extension point.

A Spid3r program receives code through seven routes: the embedded standard library, project source files, Spid3r plugins, vendored packages, WASM plugins, host plugins and native FFI โ€” every route ships. Only the WASM plugin route is sandboxed; the Spid3r, host and FFI routes are trusted code by definition.
Seven routes in, all of them shipped. Only one โ€” WASM plugins โ€” can be sandboxed; the rest are trusted code, and the diagram marks which is which.
Tier What Guarantee Status
T0 Built-in stdlib Embedded, immutable, no disk override from the install tree shipped
T1 Project .matrix files Same trust as the project's own source shipped
T1 Spid3r plugins (kind = "matrix") Written in the language and loaded from disk; script authority, like your own source shipped
T2 Vendored packages Checksummed against spider.lock; spid3r verify re-checks shipped
T3 WASM plugins Sandboxed โ€” no ambient authority, manifest-declared permissions, fuel-limited shipped
T4 Host plugins Trusted code. Not sandboxed, and cannot be shipped
T5 FFI (extern fn) Trusted code, AOT-only; spid3r run rejects it at compile time shipped

Two things this page will not overstate. Capability filtering is a policy filter, not a sandbox: it replaces known builtin natives with stubs that return a catchable capability denied error, and a host function a plugin registers keeps whatever authority it was given โ€” no filter can take that back. And dependency installation never executes package code โ€” there are no build scripts, unlike Cargo โ€” which is a real security advantage and is deliberate.

Packages

A package is a directory with a spider.toml and a src/ tree. spid3r add declares it and vendors it in one step, spid3r tree shows what that resolved to, spid3r verify proves the vendored bytes are the ones the lock file describes, and spid3r publish --to-dir writes a registry entry with package.tar, index.txt, sha256.txt, and an Ed25519 signature sha256.txt.sig (registry.pub). spid3r install supports both local directory mirrors and remote http(s):// registries (SPID3R_REGISTRY / packages.registry, cached under .spid3r/registry/), verifies SHA-256 and Ed25519 signatures (pinned via packages.trust_pk / SPID3R_TRUST_PK), intersects compatible diamond semver ranges across both single-project and workspace dependency graphs, and never executes package code. Note: release tarballs built without tls support http:// mirrors and local directories; enable --features tls (included in Windows installers) for https:// registries.

Registry deep dive โ€” resolution, layout, integrity, remote mirrors & signing

The full registry contract: a registry dependency pulls a package by name + version constraint (strlib = { version = "^1.2" }, optionally registry = "internal"), reusing the same vendoring and spider.lock machinery as path/git dependencies โ€” only resolution is new. spid3r install vendors the resolved src/ tree into vendor/<name>/ and pins source = "registry+<resolved-version>" plus the content SHA-256; re-running install is idempotent. A registry is a directory named by SPID3R_REGISTRY (or packages.registry) โ€” local files, a committed index, or a shared drive; there is no default remote registry in the base build, and an unset variable fails with a message telling you to set it.

Version constraint Meaning
^1.2 / ^1.2.3 same major, at least the given minor/patch
>=1.2.3 at least this version
=1.2.3 or bare 1.2.3 exactly this version

The chosen directory's spider.toml version must also satisfy the constraint (checked after resolution). The layout is $SPID3R_REGISTRY/<name>/<version>/ with a spider.toml, a src/ tree, and a required sha256.txt; version directories are major.minor.patch (or major.minor) and the highest satisfying version wins. Named sub-registries put packages under $SPID3R_REGISTRY/<name-from-registry-key>/<package>/<version>/. The sha256.txt hash covers every file under src/ over raw bytes โ€” for each file in sorted order, relative/path + 0x00 + contents + 0x00, concatenated and hashed once, with / separators on every platform โ€” and install refuses a missing, unreadable, empty or mismatched file. spid3r publish --to-dir <dir> writes the layout offline and refuses to republish an existing name/version (a registry entry is immutable; bump the version instead).

Integrity: spid3r verify re-hashes every vendored tree against the sha256 in spider.lock โ€” a lock entry with no recorded hash also fails, and so does an undeclared directory under vendor/. Editing one vendored file therefore fails the build gate (exit 1) instead of being silently compiled. spid3r search <query> --local matches package names in SPID3R_REGISTRY and vendor/ without network access.

Remote registries, signing & dependency unification (R8.1โ€“R8.4): SPID3R_REGISTRY / packages.registry also accept http:// / https:// URLs serving <base>/<name>/index.txt, package.tar and sha256.txt, cached under .spid3r/registry/<url-hash>/<name>/<version>/ after verifying tree hashes and path-traversal safety. Portable Unix tarballs are built without tls (so they support http:// mirrors and local directories but diagnose https:// clearly); Windows installers enable tls. publish --to-dir writes an Ed25519 sha256.txt.sig plus registry.key/registry.pub; when packages.trust_pk / SPID3R_TRUST_PK is set, install verifies the signature and refuses unsigned or tampered packages, otherwise it verifies against registry.pub or warns. Diamond dependencies across single-project and [workspace] manifests intersect all semver constraints per package name and re-select the highest satisfying version (e.g. ^1.0 + =1.2.0 โ†’ 1.2.0), reporting conflicts cleanly. No semver pre-releases or build metadata โ€” plain major.minor.patch. A spider.lock entry records name, version, source, used-by (the workspace members that depend on it), files and sha256.

Package flow: spider.toml plus src tree, spid3r add vendors and pins it in spider.lock, spid3r verify re-hashes every file against the lock so hand edits fail, publish --to-dir writes package.tar, index.txt, sha256.txt and Ed25519 sha256.txt.sig, spid3r install supports local or HTTP(S) registries with signature verification and diamond semver unification. Installing never executes package code.
Signed and checksummed end to end: declare, vendor, verify, publish, install. Supports local and HTTP(S) registries, Ed25519 packages.trust_pk verification, and diamond constraint unification โ€” with zero build scripts executed.
# Which file does `import "semver"` actually name?
spid3r modules src/main.matrix --tree
# Declare, inspect, then prove the tree is intact
spid3r add mathlib --path ../mathlib
spid3r tree
spid3r verify
# Publish into a static registry directory
spid3r publish --to-dir ../registry

Plugins

Host plugins work today. A plugin declares itself in a plugin.toml, implements one trait, and is registered by its host at startup. There is no dynamic loading: a plugin is compiled into the host that uses it, because a native function is a Rust function pointer over Value, and Value has no stable representation that could cross a C ABI boundary.

The payoff of the design is that registration merges the plugin's functions into stdlib::native_defs() โ€” the one list the tree interpreter, the bytecode VM and the AOT backend all install from. A plugin function needs no per-engine code.

Host plugin flow: plugin.toml declares the ABI string and one block per function with arity and permission, the plugin implements one Rust trait over Value, the host registers it at startup with plugin::register, and the functions merge into the native table all three engines read โ€” no per-engine code. Warning strip: trusted code, same address space, compiled into its host, no dynamic loading.
Declare, implement, register. The plugin's functions land in the one native table every engine installs from โ€” and because it is compiled into its host, it is trusted code by definition.
# plugin.toml โ€” the manifest is the contract
[plugin]
name = "hello"
abi = "spid3r-plugin-abi-1"
capabilities = ["environment"]
# one block per function, with its arity and permission
[[function]]
name = "hello_env"
arity = 1
permission = "environment"

register rejects, before any plugin code runs: an abi that is not exactly spid3r-plugin-abi-1; a function the manifest never declared, or one whose arity disagrees; a permission the manifest's capabilities does not announce; a duplicate name; and a function that would shadow a builtin โ€” write_file must not be interceptable. A plugin whose start() fails is recorded as Failed and simply never offers its functions, so one broken plugin cannot take the host down.

The permission gate is the same mechanism as for builtins: Engine::deny(Capability::Environment) replaces hello_env with a stub returning a catchable capability denied error, while the plugin's other functions keep working.

A complete sample lives at examples/plugins/hello-host/ โ€” a manifest, a PluginHost implementation, a host binary and two scripts, including one that demonstrates the gate. The full plugin contract: status is every route works โ€” Spid3r-source, WASM and host plugins plus native FFI โ€” full contract in the deep dive below.

Plugin deep dive โ€” three kinds, manifest contract, lifecycle, WASM guest ABI, status

The three kinds and their trust stories:

kind How it loads Trust
matrix a .matrix file the toolchain loads and runs script code โ€” the authority a script has
wasm a .wasm module loaded and interpreted by wasmi sandboxed โ€” no imports, so no ambient authority
host compiled into a Rust host that calls plugin::register trusted native code โ€” same address space and authority

matrix is the default (write public fns, declare them, done); wasm is for code you did not write; host is for a Rust host exposing its own internals. A host plugin cannot be loaded from disk โ€” a native function is (&'static str, fn(&NativeCtx, Vec<Value>) -> Result<Value, String>), and Value is a reference-counted enum with no stable C ABI, so dynamic loading would need a serialisation hop or handle table plus dlopen, which ABI v1 deliberately does not do. The spid3r binary itself registers no plugins, so spid3r run cannot call a host plugin function; an embedding binary calls plugin::register(Box::new(Hello))? at startup, before creating an engine.

Manifest contract. [plugin] carries name, version, an exact abi = "spid3r-plugin-abi-1", kind (host|wasm), and capabilities. Each [[function]] declares name, arity and an optional permission. register rejects before any plugin code runs: a non-exact abi; a function the manifest does not declare; a disagreeing arity; a permission the capabilities list omits; a duplicate plugin/function name or one that shadows a builtin (write_file must not be interceptable).

Lifecycle. Loaded โ€”start()โ†’ Started โ€”stop()โ†’ Stopped, with a failed start() moving to Failed. start runs during register with the registry lock released (so it may open a pool or read config) and a failure is recorded but never offered โ€” the host keeps running. stop runs in plugin::shutdown_all(), must tolerate being called twice, and returns errors rather than failing fast so one bad teardown cannot leave the rest running. Enforcement of permission happens at call time for all three kinds, so it does not matter whether the plugin was registered before or after deny() was configured; a pure function with no declared permission still runs under deny_all.

Spid3r-source plugins. A directory with plugin.toml (kind = "matrix", optional entry defaulting to <name>.matrix) and a .matrix file. The loader parses and checks it with the ordinary front end, runs it as a module (no fn main required), wraps each declared global function as a native closure, and registers it. Each plugin gets its own interpreter and globals (it cannot clobber the caller's), can call the whole standard library, and a throw reaches the caller's catch intact. Trust is tier T1 script authority; no feature flag is needed. Sample: examples/plugins/hello-matrix/.

WASM plugins. Discovery order: SPID3R_PLUGINS (;-separated directories containing plugin directories), ./plugins, <exe>/plugins, ~/.spidr/plugins; the load order is sorted so a name clash reports the same plugin every time. run, check, compile, exec, test, bench, debug, gui, magic and repl load plugins; build and dist do not (a standalone binary cannot link one, so such a call is rejected at compile time). A plugin that fails to load is a warning, not a fatal error. The guest exports memory, alloc(i32)->i32, optional dealloc(i32,i32), and one (i32,i32)->i64 per function, named exactly as in the manifest (no dispatch table). Wire format (little-endian, defined in src/plugin/abi.rs): request := u8 argc, argc ร— value; response := value | (0xFF, u32 len, utf8 message); value := u8 tag, payload with 0 null ยท 1 i64 ยท 2 f64 ยท 3 u8 bool ยท 4 (u32 len, utf8) ยท 5 (u32 count, values). A map, class instance or closure is refused rather than coerced. The host builds an empty Linker โ€” a module importing anything (WASI, a clock, randomness) fails to instantiate; the capability never existed, it is not revoked. Every call gets a fuel budget (DEFAULT_FUEL, 50M instructions, or a tighter host limit), so loop {} traps as a catchable script error. Sample: examples/plugins/hello-wasm/.

Status of every route. Spid3r plugins (kind = "matrix"): works, no feature flag or build step. Host plugins: works. WASM plugins (kind = "wasm", disk-loaded): works, wasmi behind the default plugins-wasm feature (without it, a wasm plugin reports the missing feature rather than being silently ignored). FFI (extern fn): works, AOT only. spid3r plugin abi|list|check: works โ€” abi prints the version, list reports the search path (name, kind and function count per directory, not what is registered in this process), check validates a directory with the loader's own code path. enable/disable is not implemented (discovery is per run, so there is no persistent enable list) and native dlopen of host plugins is deliberately absent. To write one: copy examples/plugins/hello-host, edit plugin.toml (names are &'static str compile-time constants), implement PluginHost with manifest() using include_str! so the shipped manifest cannot drift, return one PluginFnDef per declared function, and register before creating the engine.

WASM plugins: the sandboxed route

A WASM plugin is the opposite of a host plugin: it exists only as a file. The toolchain reads plugin.toml and plugin.wasm from the plugin search path and interprets the module, so spid3r run can call it without the binary being rebuilt.

# discovery order
SPID3R_PLUGINS ยท ./plugins ยท <exe>/plugins ยท ~/.spidr/plugins
# run a program that calls a sandboxed plugin
SPID3R_PLUGINS=examples/plugins spid3r run app.matrix
# the guest must export its memory and two helpers
memory ยท alloc(i32) -> i32 ยท dealloc(i32, i32)

Each manifest function is one export with the signature (i32, i32) -> i64: the host writes the arguments into linear memory, calls the export, and reads the result back. There is no dispatch table inside the guest. The wire format is a small typed binary encoding โ€” null, int, float, bool, string, list โ€” rather than JSON, so a plugin author does not have to ship a JSON parser into the sandbox.

The sandbox is structural: the host builds an empty linker, so a module that imports anything โ€” fd_write, a clock, randomness โ€” fails to instantiate. Nothing is revoked at run time, because nothing was ever granted. Every call also carries a fuel budget, so a loop {} traps instead of hanging the interpreter; that is the one place a limit is enforceable below the language, and it is what makes running an untrusted plugin safe.

spid3r build and spid3r dist deliberately do not load plugins: a standalone native executable cannot link one. A program that calls a plugin function is rejected at compile time, with a note explaining why โ€” not compiled into a binary that throws at run time.

Route Status
Spid3r plugins (kind = "matrix", loaded from disk) works โ€” no feature flag, no build step
Host plugins (compiled into the host) works
WASM plugins (kind = "wasm", loaded from disk) works โ€” wasmi, behind the default plugins-wasm feature
FFI (extern fn) works โ€” AOT only; spid3r run rejects it at compile time
spid3r plugin abi|list|check works โ€” abi prints the version, list reports the search path, check validates a directory against ABI v1
spid3r plugin enable/disable not implemented โ€” a plugin is discovered per run, so there is no persistent enable list
Dynamic loading of native host plugins (dlopen) deliberately not implemented for host plugins (no stable ABI over Rust Value); AOT C-ABI FFI is the opt-in native route โ€” see the FFI deep dive below.
FFI deep dive โ€” extern fn types, marshalling, linking, callbacks, --allow-dlopen

The native C route (AOT only). extern fn declarations name the C symbol verbatim โ€” no mangling, no aliasing โ€” and only six types cross the boundary: int โ†’ int64_t, float โ†’ double (an int argument is widened), bool โ†’ 1-byte _Bool, string โ†’ UTF-8 NUL-terminated const char*, ptr โ†’ opaque void* handle, usize โ†’ 64-bit size_t. Lists, maps, classes and closures are refused at build time with the type named; an untyped parameter is an error (C has none). A ptr is a boxed address โ€” nothing is copied or freed around it, and passing a handle C did not hand you is undefined behavior on the C side.

Marshalling & linking. Every call copies string arguments into temporary C buffers and frees them when the call returns; a string returned from C is copied into a Spid3r string and the C buffer is not freed โ€” a C function must not retain a pointer to a string argument. --link-lib is repeatable: a bare name becomes -l<name> on Unix and <name>.lib under MSVC; paths ending in .a/.lib/.so/.dylib pass through as given. A missing symbol is a link error, so no binary is produced โ€” a bad symbol can never reach a user as a crash.

Where it runs & trust. The tree interpreter and bytecode VM reject extern fn outright (exit 2, before any statement runs) because they link nothing. AOT FFI is tier-T5 trusted code: a C function runs with full process authority, no sandbox can exist at this layer, and --sandbox does not apply to a compiled executable. Untrusted third-party code belongs in a WASM plugin.

Extensions. POD structs (parent-less classes with 1โ€“2 scalar fields) marshal by value (Win64/SysV ABIs); larger or non-POD classes are refused at build time. Variadics work (extern fn printf(fmt: string, ...) -> int;, bool promoted to int). ffi_callback(fn_val, "fn(int, int) -> int") emits a C-ABI thunk for passing Spid3r closures to C APIs โ€” the slot lives for the process lifetime, a throw inside returns a zero/null sentinel to C and re-raises through try/catch. Runtime dynamic loading โ€” dl_open/dl_sym/dl_call/dl_close โ€” is disabled by default and requires spid3r build --allow-dlopen (refused at build time otherwise, and rejected on tree/vm/--sandbox). C++ symbols (name mangling) and non-POD/large structs are not implemented.

The distinction that matters: a host plugin is trusted code โ€” same address space, same authority, and no filter can revoke what it already has. A WASM plugin is the only route that can be sandboxed, and it is the one to prefer for anything you did not write. Both samples are in the repository: examples/plugins/hello-host and examples/plugins/hello-wasm.

The full developer guide — the manifest contract, the guest ABI, the wire format and a troubleshooting table — is its own page: Writing Spid3r Plugins →

Native Machine-Code Compilation (AOT)

Spid3r includes an Ahead-Of-Time (AOT) native compiler powered by LLVM. With spid3r build, any .matrix script can be compiled directly into a standalone machine-code executable that runs natively without any external runtime dependencies:

# Compile script to a native standalone executable
spid3r build examples/hello.matrix -o hello.exe
# Run the generated native binary
./hello.exe
// Output: hello

Desktop Executables: Custom Icon, No Terminal

A plain spid3r build produces a console binary: on Windows it allocates a terminal window for the process, and the file carries the default toolchain icon. Three flags turn that into an ordinary desktop application (a fourth adds version metadata):

# Windowed desktop build with an embedded icon
spid3r build app.matrix -o App.exe --release --icon app.ico --subsystem windows
Flag Effect
--icon <path.ico> Embeds the icon into the PE resource directory as RT_ICON plus RT_GROUP_ICON. Every size stored in the .ico is preserved, so Explorer, the taskbar and Alt-Tab each pick the resolution they need. Without the flag the icon shipped beside spid3r.exe is embedded, so a build started from any directory still gets one; SPID3R_ICON overrides that default.
--subsystem windows Passes -subsystem:windows and pins the entry point with -entry:mainCRTStartup, setting the PE optional header's subsystem field to 2 (GUI) instead of 3 (console). Windows then allocates no console for the process. --subsystem console is the default. Windows-only; ignored by the ELF and Mach-O linkers.
--allow-gui Permits a build of a program that calls GUI natives even though the AOT runtime does not link the GUI bridge. Without it the compiler rejects such a program before linking.
--embed <path> Appends a file or directory to the finished executable, after linking, so --icon and --version still apply. Paths stay relative to the source file, and the body is XOR-obfuscated โ€” obfuscation, not encryption. spid3r gui --payload <exe> runs the result with no sidecar files. Repeatable.
--bytecode <file.matrixc> Ships a compiled program instead of its source: appends the .matrixc to the payload and compiles the launcher from a three-line run_payload_strict() stub, so no application code, symbol names or string literals reach the native image. The strict form refuses to run when the payload is missing or truncated instead of falling back to an installed toolchain. gui --payload prefers a .matrixc entry and runs it on the VM.
--bundle-runtime Also appends the running spid3r.exe to the payload, so the built app starts its own runtime copy and runs on a machine with no Spid3r installed. Extracted to %LOCALAPPDATA%\Spid3r\runtime\<hash>\ on first launch. Implies a payload.
--product-name / --version Writes RT_VERSION metadata, displayed in the file's Properties โ†’ Details tab.

--allow-gui exists because the AOT runtime registers the standard library natives but not the GUI bridge, so gui_on, gui_eval and friends resolve to null in a native binary. The supported pattern is a launcher: the executable does its startup work, then hands the program off to the installed toolchain's spid3r gui runtime. One native call does all of it โ€” run_payload() finds a runtime (payload first, then %SPID3R_EXE%, the normal install, PATH), extracts the embedded copy when there is one, anchors the working directory to the executable's folder, and starts gui --payload hidden. Writing the handoff by hand is only necessary for custom startup behaviour.

# The whole launcher: one call. It finds a runtime, extracts the
# embedded copy if there is one, anchors the working directory and
# starts `gui --payload` with no console.
fn main() {
return run_payload();
}

Two entry points, and the difference matters for a self-contained build. run_payload() falls back to an installed spid3r when the executable carries no payload at all โ€” which is what a hand-written launcher that expects a global install wants. run_payload_strict() treats a missing payload as a hard error ("payload is missing or truncated; rebuild it"); it is what --bytecode generates, and what a launcher should call when the build is supposed to carry its own runtime, so that deleting the trailer cannot redirect the launch to whatever spid3r.exe is on PATH.

โš ๏ธ Two details that decide whether the window ever appears

run_payload() already does both. They matter when you write the handoff yourself, which is only necessary for custom startup behaviour.

  • Working directory โ‰  program folder. Windows starts a process with an inherited directory, so a shortcut, the Run dialog, a scheduled task or Start-Process all begin somewhere else. Relative paths like ui/index.html then miss, and a GUI-subsystem process exits with no visible message because it has no console to print to. Call exe_path() and chdir(dirname(exe_path())) first.
  • The handoff inherits the console. Spawning without map("hidden", true) opens a terminal window for the child. That option passes CREATE_NO_WINDOW on Windows; on other platforms it is ignored.

Shipping a Single File

Two flags turn a build into one self-contained executable: --bundle-runtime appends the running spid3r.exe to the payload, so the target machine needs no Spid3r install, and --bytecode ships the compiled program instead of its source.

# 1. compile the program
spid3r compile app.matrix -o app.matrixc
# 2. one executable: icon, no terminal, bytecode, UI, runtime
spid3r build app.matrix -o App.exe --release --icon app.ico \
--subsystem windows --bytecode app.matrixc \
--embed ui --bundle-runtime
Payload entry What it is
app.matrixc The compiled program. No .matrix is shipped.
ui/index.html, ui/app.js, ui/styles.css The interface, served from memory โ€” nothing is unpacked to disk.
__spid3r_runtime/spid3r.exe The runtime, extracted to %LOCALAPPDATA%\Spid3r\runtime\<hash>\ on first launch and reused afterwards.

--allow-gui is not needed with --bytecode: the launcher is compiled from the three-line run_payload() stub above, which mentions no GUI natives. It is only required when the application itself is compiled into the binary (--embed app.matrix).

โš ๏ธ What --bytecode does and does not buy
  • It removes the source text. With --embed app.matrix the payload carries your source, and the obfuscation is not encryption โ€” anyone can read it back out. With --bytecode the payload carries bytecode, and the launcher carries no application code, symbol names or string literals.
  • It is not secrecy. The VM is open source, so a determined person can write a decompiler for the bytecode. It raises the cost from "open a text file" to "reverse-engineer a bytecode format". Do not put secrets in a client binary.
  • One system dependency remains. The WebView2 runtime (default on Windows 11, on Windows 10 via Edge) โ€” and the matching system webview on macOS/Linux. Everything else is self-contained: the toolchain and every executable spid3r build produces link the C runtime statically and import only system DLLs, so no Visual C++ Redistributable is needed. LLVM-C.dll is delay-loaded and is only touched by spid3r build, so a GUI app never needs it.

03. Syntax, Variables & Data Types

Spid3r syntax is clear, familiar, and unambiguous. Semicolons (;) terminate statements, and variables are declared with let or const.

Variables & Constants

Variables are declared with let and can be reassigned with new values. Constants are declared with const and evaluated at compile-time:

โšก Example: Variables, Constants, and Dynamic Types
// Variables are mutable through reassignment: fn main() { let counter = 10; counter += 5; println("counter: " + string(counter)); // Constants are immutable and resolved at compile time: const MAX_BUFFER = 1024; const APP_NAME = "Spid3rApp"; println("Buffer: " + string(MAX_BUFFER) + " for " + APP_NAME); // Optional, nullable (T?), and union (A | B) type annotations are supported: let port: int | string = 8080; let host: string? = "127.0.0.1"; let is_secure: bool = false; println("Server listening on " + host + ":" + string(port)); return 0; }

Primitive Data Types & Literals

Spid3r features six core primitive value types:

Type Literal Syntax Examples Description
int Decimal, Hex, Octal, Binary 42, 0xFF, 0o755, 0b1010_1011 64-bit signed two's complement integer (-2^63 to 2^63 - 1). Supports digit separators (1_000_000) and an n suffix for arbitrary-precision big integers (12345678901234567890n).
float Standard, Scientific 3.14159, 1.5e3, 1e-5, 2E+4 64-bit IEEE 754 double precision. Overflow produces clean compile errors. Special values: nan(), inf().
bool Keywords true, false Boolean truth values. Conditionals evaluate truthiness: the falsy values are null, false, 0, 0.0 and "" (empty string); everything else โ€” including empty lists [] โ€” is truthy.
string Double quotes, Raw r"..." "hello", r"C:\path\file", r#"raw "quote""# Immutable UTF-8 scalar sequence. Supports negative indexing (s[-1]), character slices, raw literals, and interpolation (use \{ / \} for literal braces).
null Keyword null Represents the intentional absence of any object value. Supported by null-coalescing (??) and optional chaining (?.).
bytes Constructor bytes([65, 66, 67]), bytes("text") Contiguous 8-bit unsigned byte buffer (0..255). Capped at 16 MiB. Direct hex and base64 conversions.

Type Casting with as

Spid3r supports explicit, safe type casting using the as keyword (similar to Rust). It desugars into the corresponding conversion builtins:

  • x as int: Desugars to int(x)
  • x as float: Desugars to float(x)
  • x as string: Desugars to string(x)
  • x as bool: Desugars to bool(x)
โšก Example: Radix Literals & Type Casting
// Hex, binary, and scientific float literals: fn main() { let hex_mask = 0xFF; let bin_flags = 0b1010_1100; let scientific = 1.25e4; println("Hex 0xFF as decimal: " + string(hex_mask)); println("Binary flags: " + string(bin_flags)); println("Scientific 1.25e4: " + string(scientific)); // 12500.0 // Explicit 'as' casting: let raw_str = "128"; let cast_int = raw_str as int; let float_val = cast_int as float; let is_valid = cast_int as bool; println("cast_int + 2: " + string(cast_int + 2)); println("float_val / 2: " + string(float_val / 2.0)); // 64.0 println("is_valid: " + string(is_valid)); return 0; }

Spid3r Reserved Keywords (Complete Reference)

Spid3r defines exactly 40 reserved keywords. Reserved keywords have dedicated syntactic meaning in the compiler and cannot be used as variable, parameter, function, class, or module identifiers.

Category Keyword Syntax / Pattern Description
Declaration let let x = 10; Declares a mutable variable. Reassignable with = or compound operators.
Declaration const const MAX = 100; Declares an immutable compile-time constant.
Declaration fn fn add(a, b) { ... } Defines a named function, method, or anonymous closure.
Control Flow if if x > 0 { ... } Branching statement and expression evaluating truthiness.
Control Flow elif elif x == 0 { ... } Alternative conditional branch (also supports else if).
Control Flow else else { ... } Default fallback branch when preceding conditions are falsy.
Control Flow while while cond { ... } Pre-condition loop executing while condition remains truthy.
Control Flow for for x in list Collection iteration (for in) and C-style 3-clause loops.
Control Flow in for item in items Iterator membership keyword in for-in loops.
Control Flow break break; Terminates loop execution immediately.
Control Flow continue continue; Skips remaining loop body to start next iteration.
Control Flow return return value; Returns from function or closure with optional value.
Control Flow match match val { ... } Pattern matching expression with pattern guards and destructuring.
Error Handling try try { ... } Guards block against runtime throws and exceptions.
Error Handling catch catch err { ... } Catches thrown values or errors from a preceding try.
Error Handling finally finally { ... } Guaranteed cleanup block run regardless of throws or returns.
Error Handling throw throw "error"; Raises a runtime exception carrying any value or string.
Logic & Literals true true Boolean truth literal.
Logic & Literals false false Boolean falsity literal.
Logic & Literals null null Literal representing absence of a reference or object value.
Logic & Literals and a and b Logical short-circuit conjunction (equivalent to &&).
Logic & Literals or a or b Logical short-circuit disjunction (equivalent to ||).
Logic & Literals not not flag Logical negation prefix operator (equivalent to !).
OOP class class Shape { ... } Declares an object-oriented class template.
OOP interface interface Drawable { ... } Defines a method contract implemented by classes.
OOP new new Dog("Rex") Instantiates a class and specifies constructor fn new.
OOP self self.field = 10; Explicit instance receiver inside class methods and constructors.
OOP super super(name); super.m() Parent class constructor and method delegation.
OOP extends class Sub extends Super Establishes single class inheritance.
OOP implements class C implements I Declares compliance with one or more interfaces.
OOP public public fn run(self) Marks a class field or method as publicly accessible.
OOP private private let secret; Restricts field or method access strictly to defining class.
OOP abstract abstract class Base Declares an abstract class or uninstantiated method signature.
Generators & Enums yield yield val; Yields from a generator; consume with import "gen" and next(it) (a bare for x in gen() iterates the generator object, not the stream).
Generators & Enums enum enum Color { Red, Blue } Defines an algebraic sum type with unit or payload variants.
Concurrency spawn spawn fn() { ... } Spawns an asynchronous background task.
Concurrency async async fn fetch() { ... } Declares an asynchronous cooperative function.
Concurrency await await task; Suspends execution until a spawned task completes.
Modules & Casting import import "math" as m; / import { A, B as C } from "mod"; Imports external scripts or standard library modules (whole-module or selective symbol imports).
Modules & Casting as val as int Type casting and namespace aliasing (import x as y).

04. Operators, Null Safety & Expressions

Spid3r operators follow standard mathematical and C-precedence rules. It features modern null-coalescing (??) and optional navigation (?.) for bulletproof null safety.

Category Operators Precedence Semantics & Behavior
Postfix ., ?., (), [] Highest Method call, optional field access, indexing, invocation. null?.field safely returns null without throwing.
Unary not, -, await Very High Logical negation, arithmetic negation, coroutine join.
Type Cast as High Cast to int, float, string, bool (binds tighter than multiplication).
Multiplicative *, /, % Upper Mid Multiplication, division, modulo. Integer division truncates towards zero; float division preserves decimals.
Additive +, - Mid Addition (and string concatenation), subtraction.
Bitwise Shift <<, >> Lower Mid Shift left/right on 64-bit integers. Shift amounts must be in 0..=63 (verified by runtime).
Comparison ==, !=, <, <=, >, >= Low Mid Value equality and ordering. Strings compare lexicographically.
Bitwise Logic &, ^, | Low Bitwise AND, XOR, OR (strictly follows standard C precedence: & > ^ > |).
Logical and, &&, or, || Very Low Short-circuiting boolean conjunction and disjunction.
Null Coalesce ?? Very Low Returns the right operand only if the left operand is strictly null.
Assignment =, +=, -=, *=, /=, %= Lowest Right-associative target assignment.
โšก Example: Operators, Bitwise Logic & Null Safety
// Bitwise arithmetic with C precedence: fn main() { let mask = (1 << 4) | (0b0011 ^ 0b0001) & 0xFF; println("Calculated bit mask: " + string(mask)); // Null-coalescing '??' (evaluates right side ONLY when left is null): let user_name = null; let display_name = user_name ?? "Guest User"; println("Welcome, " + display_name + "!"); // Optional navigation '?.' prevents null dereference crashes: let account = null; let balance = account?.wallet?.balance ?? 0.0; println("Balance: $" + string(balance)); // 0.0 // Chained boolean comparisons: let x = 15; if (x >= 10 and x <= 20) { println("x is within [10, 20] range"); }return 0; }

05. Strings, Slicing & String Interpolation

Strings in Spid3r are immutable, UTF-8 encoded sequences with full negative indexing, Unicode escape sequences, and a complete suite of native methods.

String Interpolation & Raw Strings

Spid3r supports concise string interpolation using {identifier} or {object.field}. Note: there must be no whitespace inside the braces (e.g. "{name}" works, while "{ name }" remains literal text for templating compatibility). Escape braces with \{ and \} when literal braces are needed, or use raw string literals (r"..." and r#"..."#) where backslashes and braces are never interpreted.

Method Signature Returns Description
.len() s.len() or len(s) int Count of characters in the string.
.slice(start, end) s.slice(0, 3) string Character-indexed slice. Bounds-checked; negative indices supported on [].
.upper() / .lower() s.upper() string Uppercase / lowercase copy of the string.
.trim() s.trim() string Strips leading and trailing whitespace. (Also .ltrim() and .rtrim()).
.split(sep) s.split(",") list Splits string into a list of substrings by delimiter.
.replace(from, to) s.replace("-", "_") string Replaces all occurrences of from with to.
.contains(sub) s.contains("key") bool Returns true if substring exists.
.starts_with(p) s.starts_with("pre") bool Returns true if string begins with prefix.
.ends_with(p) s.ends_with("suf") bool Returns true if string ends with suffix.
.index_of(sub) s.index_of("abc") int First character position of match, or -1 if not found.
.repeat(n) s.repeat(3) string Repeats string n times.
.format(args...) "{:.2}".format(88.6875) string Positional formatting with precision rounding ({:.N} rounds half-to-even).
โšก Example: String Slicing, Interpolation & Methods
fn main() { let text = " Spid3r Language "; println("Trimmed: [" + text.trim() + "]"); println("Upper: " + text.upper()); println("Slice(2, 8): [" + text.slice(2, 8) + "]"); println("Negative index [-1]: " + "Spid3r"[-1]); // 'r' // String Interpolation: let user = "Dev"; let score = 98.5; println("Welcome {user}! Your score is {score}."); // Formatting precision with tie-to-even rounding: let formatted = "Result: {:.2}".format(88.6875); println(formatted); // "Result: 88.69" // Split & Join: let parts = "apple,banana,cherry".split(","); println("Split count: " + string(len(parts)) + ", second: " + parts[1]); return 0; }

06. Control Flow: Conditionals & Loops

Control execution flow using if / elif / else, value-position conditional expressions, C-style for loops, and collection iterators.

If-Else as Statements and Expressions

In Spid3r, if can be used both as a traditional control statement and as an expression yielding a value:

โšก Example: If / Elif / Else Chains & Expression If
fn main() { let marks = 85; let letter_grade = ""; // Traditional conditional chain: if (marks >= 90) { letter_grade = "A"; } elif (marks >= 80) { letter_grade = "B"; } elif (marks >= 70) { letter_grade = "C"; } else { letter_grade = "F"; } // 'if' in value-position (conditional expression): let status = if (marks >= 60) { "Passed" } else { "Failed" }; println("Student Grade: " + letter_grade + " (" + status + ")"); return 0; }

Loops: while and for

Spid3r supports three versatile loop constructs:

  1. while (condition) { ... }: Standard condition-tested loop.
  2. for let i = 0; i < n; i += 1 { ... }: C-style loop with lexical index binding.
  3. for item in collection { ... } / for (k, v) in map { ... }: Iterator loop over lists, sets, tuples, and maps.
โšก Example: While, C-style For, and Map Iteration
// 1. C-Style For with break and continue: let sum_evens = 0; for let i = 0; i <= 10; i +=1 { if (i % 2 !=0) { continue; } // skip odds if (i> 8) { break; } // early stop sum_evens += i; } println("Sum of evens up to 8: " + string(sum_evens)); // 2. While loop: let steps = 0; let n = 12; while (n > 1) { if (n % 2 == 0) { n = n / 2; } else { n = 3 * n + 1; } steps += 1; } println("Collatz steps for 12: " + string(steps)); // 3. For-in over lists and maps: let fruits = ["Apple", "Orange", "Mango"]; for fruit in fruits { println("Fruit: " + fruit); } let roles = map("Alice", "Admin", "Bob", "Developer"); for (name, role) in roles { println("{name} has role {role}"); }

07. Functions, Closures & Functional Methods

Functions in Spid3r are first-class values. They can be stored in variables, passed to other functions, returned from closures, and chained using higher-order collection methods.

Function Syntax, Arrow Closures & Default Parameters

Declare functions using fn name(params) -> return_type { ... }, or write concise inline closures with arrow syntax: x => x * 2, (a, b) => a + b, or () => "ready". Trailing parameters may specify default scalar or collection literals (such as items = [] or opts = {}, which allocate a fresh mutable collection on every call). Unused parameters prefixed with an underscore (such as _event) suppress unused-parameter compiler warnings.

โšก Example: First-Class Functions, Closures & Defaults
// Function with default parameter: fn power(base: int, exp: int = 2) -> int { let res = 1; for let i = 0; i < exp; i +=1 { res *=base; } return res; } println("power(3): " + string(power(3))); // 3^2 = 9 println(" power(2, 4): " + string(power(2, 4))); // 2^4 = 16 // Lexical closure that captures outer variable: fn make_accumulator(start: int) { let total = start; return fn(val: int) { total += val; return total; }; } let acc = make_accumulator(100); println(" acc(20): " + string(acc(20))); // 120 println(" acc(15): " + string(acc(15))); // 135 // Functional higher-order methods with concise arrow closures (.map, .filter, .reduce): let nums = [1, 2, 3, 4, 5, 6]; let squared_evens = nums .filter(x => x % 2 == 0) .map(x => x * x); println(" Squared evens: " + string(squared_evens)); // [4, 16, 36] let sum_all = nums.reduce(0, (acc, n) => acc + n); println(" Sum via reduce: " + string(sum_all)); // 21

08. Enums, Pattern Matching & Destructuring

Model complex data structures using algebraic sum types (enum) and inspect them with exhaustive match expressions with pattern guards.

Algebraic Data Types (Sum Types)

Pattern Matching & Destructuring: Spid3r supports deep pattern matching with guards and unpacking tuple/list structures (e.g. let (x, y) = [100, 200]; or let (ip, port, proto) = get_server();).

Enums in Spid3r can define both empty unit variants and data-carrying variants. When dispatched with match, the compiler verifies exhaustiveness and binds inner fields directly:

Unit variants: a unit variant is built with parentheses (Stop()); the bare name Stop is the constructor function and is only valid inside a pattern. Using Stop as a value produces a confusing "function does not support indexing" error at the match, so always construct with ().

๐Ÿ” Nullability Narrowing, Scrutinee Exhaustiveness & E1xxx Diagnostics
  • Flow-Sensitive Nullability Narrowing: Nullable bindings (T?) narrow automatically to T inside if (x != null) / if (!is_null(x)) branches, after early-exit guards (if (x == null) { return ...; }), after ? error/null propagation, through ?. optional fields, inside while (x != null) loops (while un-reassigned to nullable), and on catch (e) error binders.
  • Scrutinee-Driven Enum Exhaustiveness: When a match scrutinee has a static enum type, the semantic checker enforces every variant of that enum (unless covered by _ or a variable binding), falling back to arm inference when untyped.
  • Multi-Error Diagnostics: spid3r check and spid3r lsp recover across up to 25 syntax errors in one pass with E1xxx error codes, [line:col .. end_line:end_col] spans, and TTY-gated ANSI colors (NO_COLOR supported).
โšก Example: Enums, Pattern Guards & Destructuring
// 1. Enum with payload variants and unit variants: enum Command { Move(x, y), Rotate(degrees), Stop } fn describe_command(cmd) { return match (cmd) { Move(x, y) if x == 0 and y == 0 => "Hold position", Move(x, y) => "Move to ({x}, {y})", Rotate(deg) => "Rotate by {deg} deg", Stop => "Emergency Stop!", _ => "Unknown command" }; } println(describe_command(Move(10, 20))); println(describe_command(Move(0, 0))); println(describe_command(Rotate(90))); println(describe_command(Stop())); // 2. Destructuring assignments (lists & tuples): let (x, y) = [100, 200]; println("Destructured list: x=" + string(x) + ", y=" + string(y)); fn get_server_info() { return tuple("10.0.0.1", 443, "https"); } let (ip, port, proto) = get_server_info(); println("Server: {proto}://{ip}:{port}");

09. Error Handling: Exceptions & Option / Result

Spid3r combines structured exceptions (try-catch-finally) with functional Rust-style error combinators (Option and Result).

Try-Catch-Finally Execution Guarantees

The catch block accepts both parenthesized (catch (err) { ... }) and unparenthesized (catch err { ... }) bindings. The finally block is guaranteed to execute, even if the try block issues a return, break, continue, or an unhandled throw.

โšก Example: Try, Catch, Finally & Result Types
// 1. Structured try-catch-finally: fn parse_positive_int(s: string) { try { let val = int(s); if (val <= 0) { throw "value must be positive, got " + string(val); } return val; } catch (err) { println("Caught error: " + string(err)); return -1; } finally { println("Cleanup: parse_positive_int completed"); } } println("Result 1: " + string(parse_positive_int("42"))); println("Result 2: " + string(parse_positive_int("-5")));

10. Object-Oriented Programming (OOP)

Spid3r provides first-class object-oriented programming: classes, single inheritance (extends), constructor and method delegation (super), encapsulation (public / private), and interfaces (implements).

Core OOP Rules & Visibility

  • Constructors: The constructor is declared as public fn new(params) { ... }. Inside the constructor, self is implicitly bound to the newly allocated instance.
  • Explicit Receiver: Instance methods explicitly take self as their first parameter: public fn method(self, arg) { ... }.
  • Default Visibility is Private: Any member without an explicit public keyword is private to the defining class. The checker enforces this whenever the receiver's class is statically known โ€” an annotated binding (let c: C = C()), new C(), or self โ€” and leaves a receiver whose class is only known at run time unchecked, the same gradual rule the rest of the checker follows. Annotate the binding when you need the guarantee: let c = C(); c.secret is accepted, let c: C = C(); c.secret is rejected.
  • Single Inheritance: Classes inherit from a single parent class using class Sub extends Parent.
  • Super Delegation: Subclasses call super(args) in their constructor to initialize the parent fields, and super.method(args) to delegate to the parent method.
  • Abstract Classes & Static Methods: Declared with abstract class Base, abstract classes define method contracts that cannot be instantiated directly. Classes may also declare class-level methods with public static fn name(args) { ... } (called directly on the class as ClassName.name(args) without self).
โšก Example: Classes, Inheritance & Super Delegation
class Animal { public let name; private let energy; public fn new(name: string) { self.name = name; self.energy = 100; } public fn speak(self) -> string { return self.name + " makes a sound"; } public fn play(self, cost: int) { self.energy -= cost; return self.energy; } public fn get_energy(self) { return self.energy; } } class Dog extends Animal { public let breed; public fn new(name: string, breed: string) { super(name); // delegate to Animal constructor self.breed = breed; } public fn speak(self) -> string { return super.speak() + " -> Woof! (Breed: " + self.breed + ")"; } } let pet = new Dog("Rex", "German Shepherd"); println(pet.speak()); println("Initial Energy: " + string(pet.get_energy())); println("After playing: " + string(pet.play(25)));

Interfaces, Default Methods & Polymorphism

Interfaces define structural method contracts that implementing classes must satisfy. Interfaces can also provide default method implementations (fn describe(self) -> string { ... }) that implementing classes inherit automatically unless overridden:

โšก Example: Interfaces & Polymorphic Dispatch
interface Drawable { fn draw(self) -> string; fn area(self) -> float; } class Circle implements Drawable { public let radius; public fn new(r: float) { self.radius = r; } public fn draw(self) -> string { return "Drawing Circle with radius " + string(self.radius); } public fn area(self) -> float { return 3.14159 * self.radius * self.radius; } } class Rectangle implements Drawable { public let width; public let height; public fn new(w: float, h: float) { self.width = w; self.height = h; } public fn draw(self) -> string { return "Drawing Rectangle " + string(self.width) + "x" + string(self.height); } public fn area(self) -> float { return self.width * self.height; } } let shapes = [new Circle(5.0), new Rectangle(4.0, 6.0)]; for s in shapes { println(s.draw() + " -> Area: " + string(s.area())); // floats keep .0 }

Generics, Constraints & AOT Monomorphization

Functions, classes, and interfaces support multi-argument generic type parameters (such as <T> or <K, V>) with optional trait/interface constraints:

  • Built-in & Interface Constraints: <T: Numeric>, <T: Comparable>, <T: Printable>, any declared interface <T: Drawable>, or intersection bounds with + (<T: Numeric + Comparable>).
  • Explicit & Inferred Type Arguments: Call sites and constructors accept explicit type arguments (identity<int>(42), new Box<string>("hi")) or infer them from value arguments. Unannotated generic parameters stay backward-compatible via type erasure in the Tree and VM engines.
  • AOT Monomorphization & Unboxed Collections: When compiling natively with spid3r build, concrete generic specializations are monomorphized (spid3r_mono_*) and homogeneous numeric lists (list<int>, list<float>) use unboxed i64/f64 storage backed by 12 specialized runtime trampolines (spid3r_ulist_i64_*, spid3r_ulist_f64_*).
Generics design spec (R10.1 / DP-1) โ€” Value v2, monomorphization & unboxed containers

Status: approved โ€” contract for parser/checker instantiation and the AOT monomorphizer + unboxed containers. The RED LINE was amended to permit a Value v2 representation alongside AOT monomorphization: a compact 64-bit tagged value (NaN-boxed or low-tag u64 inline scalars for small Int, Float, Bool, Null, tagged pointers for heap objects) that removes Box<Value> allocations for scalars; typed container annotations lowering to unboxed backing buffers (Vec<i64>, Vec<f64>, typed hash map) without per-element boxing or Mutex locking on unshared paths, with identical language semantics; and safety guardrails that confine all unsafe pointer/tag operations to src/native/rt.rs (and src/value/ if inline-tagged Value v2 uses NonNull/usize bitpacking), each with a written // SAFETY: invariant, with the runtime ABI version bumping cleanly if trampoline signatures change. This targets the 8โ€“18ร— container gap on json/map versus Node/Rust caused by boxing + Arc<Mutex<โ€ฆ>> overhead.

Erased-syntax backward compatibility. Existing programs use erased generic syntax with no explicit type arguments (class Box<T>, fn first<T>(xs), fn identity<T>(x), fn head<T>(xs) -> T). Type arguments at call and construction sites (identity<int>(7), new Box<int>(7)) remain optional; when omitted the checker infers concrete arguments from call arguments where possible or falls back to Any (dynamic Value). Unconstrained <T> with untyped parameters keeps type-checking and executing identically across Tree, VM, WASM and AOT.

Parser & AST. TypeExpr.arg: Option<Box<TypeExpr>> becomes args: Vec<TypeExpr> so multi-parameter types (map<string, int>, result<int, string>, Pair<int, bool>) are represented directly; a TypeParam { name, bound, span } replaces the bare name list, with a .names() helper for a minimal diff. parse_type_params() parses comma-separated parameters with optional bounds (<T>, <T: Comparable>, <K, V>), parse_single_type() parses <T1, T2, โ€ฆ>, and explicit arguments on calls and constructors are disambiguated from less-than by lookahead (< followed by a type list then >().

Checker instantiation & constraints. Declared TypeParams bind with their optional Comparable/Numeric/Printable/interface bound during function/class checking; at instantiation the checker verifies each concrete type satisfies the bound and emits a Phase::Semantic diagnostic on mismatch or wrong arity. A deduplicated InstantiationSet records MonoFnKey { name, type_args }, MonoClassKey { name, type_args } and typed container annotations propagated to local, parameter and return type tables. Tree and VM execute generics polymorphically on Value with zero behavioral divergence from AOT monomorphized code.

AOT monomorphization & unboxed containers. For each MonoFnKey whose type args are scalar/known, emit a specialized spid3r_mono_<fn>__<mangled> taking and returning unboxed LLVM types (i64, double, i1, typed pointer) where safe, with bounded depth (e.g. 64 specializations) and automatic fallback to the generic *mut Value entry on overflow or Any/heterogeneous types. Typed-buffer handles or specialized ListI64/ListF64/MapStrI64 trampolines are used only when the checker proves the value never escapes to untyped Value contexts (auto-boxing on escape); xs[i], .push(v), .len() and for x in xs lower to direct unboxed loads/stores. Gates: HLC corpus codegen parity โ‰ค 1.25ร—, a Tier-B improvement on json and map, and unchanged selfhost native parity (selfhost sources use no generics).

Surface Impact Action
selfhost/corpus/ (134 programs) 0 existing programs use <T> Frozen corpus parity unchanged; add generic instantiation cases under the SH-4 ledger entry
selfhost/*.matrix (17 modules) 0 modules use generics All selfhost parity scripts stay green without syntax changes
tests/constructs.rs (GENERICS) uses erased Box<T>, first<T>, identity<T>, head<T> Must keep passing identically in Tree, VM and AOT
tests/engine_matrix.rs needs new instantiation & constraint rows Add tree == vm == native parity rows
RUNTIME_ABI.md new spid3r_list_i64_* / spid3r_map_str_i64_* trampolines Update trampoline counts & ABI docs when the AOT stage lands

11. Built-in Collections & Advanced Data Structures

Spid3r comes with over 20 built-in, native data structures designed for maximum algorithmic performance. Every container is created via a globally available constructor function.

Container Constructor Core Native Methods Big-O Complexity
List list(1, 2) or [1, 2] push, pop, get, set, remove(value), remove_at(i), insert, sort, reverse, slice, map, filter, reduce, argsort O(1) append/pop, O(1) index, O(n log n) sort
Map map("k", 1) or {"k": 1} get, set, m[k], remove, has, keys, values, items, len, clear O(1) average lookup/insert/remove(k) via FxHash index table (tombstone compaction + insertion order preserved)
Set set(1, 2) add, remove, contains, union, intersect/intersection, difference, is_subset O(1) average membership and insertion (FxHash scalar index for int/string/bool; unique members)
Bag bag() add, remove, count_of, contains, len, is_empty, clear Multiset (counts duplicates). No set algebra โ€” use set for that.
Tuple tuple(1, "a", true) t[0], len(t), immutability, destructuring O(1) index, zero-overhead fixed-size sequence
Bytes bytes([65, 66]) / bytes("s") get, set, slice, to_hex, to_base64 O(1) byte index, contiguous memory up to 16 MiB
Stack stack() push(x), pop(), peek(), len(), is_empty() O(1) LIFO push/pop (throws on empty pop)
Queue queue() push(x), pop(), peek(), len(), is_empty() O(1) FIFO enqueue/dequeue
Deque deque() push_front, push_back, pop_front, pop_back, peek_front, peek_back O(1) double-ended queue operations
Heap heap("min") / heap("max") push(x), pop(), peek(), change_priority, merge O(log n) push/pop, O(1) peek
Trie / Tree trie() insert(word), contains(word), starts_with(prefix), with_prefix(prefix) O(L) string prefix matching where L is word length
Graph graph("directed") add_node, add_edge, neighbors, find_path, bfs, dfs, cycle Adjacency list with native Dijkstra & BFS pathfinding
RingBuffer ring_buffer(capacity) push, pop, peek, len O(1) bounded circular buffer (overwrites oldest)
Also built in linked_list(), hash_table(n), bst(), binary_tree(), priority_queue(), vec(), singly_linked_list(), doubly_linked_list() linked_list(): push_back/front, pop_front/back, get, remove, to_list ยท hash_table(): 8 growing buckets, bucket_count, load_factor ยท bst()/binary_tree(): insert, inorder/preorder/postorder, root ยท priority_queue() (heap-backed) ยท vec() ยท singly_linked_list() / doubly_linked_list() โ€” see the full data-structures reference below Same native containers as the highlighter; constructor details and the complete method vocabulary follow

Every container prints with a type prefix, so a logged value is never ambiguous: [1, 2] for a list, map{a: 1} for a map, set{1, 2} for a set, stack() for an empty stack. Note that bag() takes no constructor arguments โ€” add members with add and read multiplicities with count_of; a bag deliberately has no set algebra, so use set when you need union or difference.

Data-structures deep dive โ€” complete constructor & method reference, comparator sorting, float precision

Constructor & Method Reference (Complete)

Every container is a value created by a constructor function โ€” ordinary globals that can be shadowed โ€” and all share a common vocabulary: len, is_empty, clear, plus push/pop/peek/get/contains where they make sense. Methods use dot syntax; a wrong argument type throws a descriptive, catchable runtime error. Constructors take initial elements where natural:

let xs = list(10, 20, 30); let m = map("a", 1, "b", 2); let q = queue(); let g = graph("undirected"); let h = heap("min"); let mx = matrix(2, 3, 0.0); let rb = ring_buffer(4); let cell = mutex(0); let atom = atomic_int(0); let ch = channel(); let box = ref(1); // a shared, interior-mutable cell let yes = some(5); // option let no = none; let okv = ok(1); let errv = err("bad"); // result
Group Constructors and methods
Sequences list(a, b, โ€ฆ) (indexable; get/set, push/pop, remove/remove_at, contains, sort/reverse + sorted/reversed, slice, join(sep), append, index_of, higher-order map/filter/reduce/for_each/all/any, sort_by/sorted_by/argsort) ยท vec() (clone) ยท deque() (push_front/back, pop_front/back, peek_front) ยท tuple(a, b, โ€ฆ) (fixed, immutable length) ยท bytes(...) (slice, to_list/string, to_hex/from_hex, to_base64/from_base64, copy) ยท ring_buffer(n) (overwrites the oldest when full)
Stacks & queues stack() (LIFO; pop on empty throws) ยท queue() (FIFO) ยท priority_queue("min"|"max") (heap-backed)
Strings Immutable, with len/is_empty, character-indexed slice, upper/lower/trim/ltrim/rtrim, split/join, contains/starts_with/ends_with/index_of (-1 when absent), replace/repeat/reverse/chars, pad_left/pad_right, format(argsโ€ฆ) with {} and {{/}} escapes plus precision {:.N}
Associative map(k, v, โ€ฆ) (insertion-order; index-assign m[k] = v, remove/delete, keys/values/items) ยท set(a, b, โ€ฆ) (union/intersection/difference, is_subset/is_superset) ยท bag() (multiset: add/count_of/remove) ยท hash_table() (separate chaining, no constructor args, bucket_count/load_factor)
Linked nodes linked_list() / singly_linked_list() ยท doubly_linked_list() (fast pop_back, both-end push)
Trees & heaps binary_tree(a, b, โ€ฆ) (level-order constructor, traversals) ยท bst() (insert/remove/contains, traversals) ยท trie() (with_prefix/all_words) ยท heap("min"|"max") (change_priority/merge)
Numeric vector(n, fill) / vector(list) and matrix(rows, cols, fill) / matrix(rows_list) โ€” contiguous 64-bit float storage capped at 16,777,216 elements; elementwise arithmetic, reductions, linear algebra (dot/norm/solve/inverse/det/eigen_sym). See the Linear Algebra chapter.
Graphs graph("directed"|"undirected") โ€” adjacency map (may be cyclic): add/remove_node, add/remove_edge, neighbors, n_nodes/n_edges, bfs/dfs, path_exists/find_path, is_connected/cycle, replace/clear
Shared state ref(v) (get/set, safe across tasks) ยท weak(r)/deref(w) (non-owning; null if collected) ยท mutex(v) (RAII lock()) ยท atomic_int(init?) (load/store/cas/exchange/fetch_add/fetch_sub) ยท channel() (send/recv/try_recv/state/len) ยท task (from spawn, joined with await)
Sum types option (some/none, unwrap/unwrap_or) ยท result (ok/err, unwrap/unwrap_err) ยท enum Name { Variant1, Variant2(field) } โ€” each variant lowers to an ordinary public constructor function (works across import); a value is a map carrying __tag, __fields and one key per payload field; match arms use the bare variant name
Binary I/O & formats read_bytes/write_bytes ยท save_npy/load_npy (NumPy v1.0 little-endian <f8) ยท parse_csv/read_csv/write_csv (RFC-4180, quoted fields, type inference) ยท regex_match/find/find_all/replace (Thompson NFA) ยท unix_time/unix_ms/datetime_parts/datetime_from_parts/format_datetime

Sorting with a Comparator & Float Precision

xs.sort_by(cmp) sorts in place (returns null), xs.sorted_by(cmp) returns a new list, and xs.argsort() returns the original indices in ascending order (list(30, 10, 20).argsort() โ†’ [1, 2, 0]). A comparator cmp(a, b) returns either a bool (true = a before b) or a number (negative/0/positive = before/equal/after). Sorts are stable: the list is snapshotted before comparison, no lock is held while a comparator runs, and the sorted snapshot is written back after the last call โ€” so a comparator that throws, mutates the receiver or touches a self-referential list gets a clean result, never a deadlock or host panic. Because false cannot distinguish "after" from "equal", the mirrored question cmp(b, a) is asked whenever cmp(a, b) is false; keep comparators side-effect free. Non-bool/int/float or NaN returns throw a catchable error.

A format placeholder may carry precision {:.N} for N in 0..=15 โ€” "{:.2}".format(88.6875) โ†’ "88.69", "{:.0}".format(2.5) โ†’ "2" (tie-to-even, unlike round(x), which rounds halves away from zero). An int is widened to fixed point; precision > 15, a non-number placeholder, or a malformed spec such as {:.}/{:x} is a catchable error. Methods validate arguments and throw on violation (empty stack.pop(), out-of-range get(i), a method the collection does not have); these are ordinary catchable thrown values.

โšก Example: Lists, Maps, Sets, Stacks, Heaps & Tries
// 1. Lists with in-place sorting and slicing: fn main() { let items = [40, 10, 50, 20, 30]; items.sort(); println("Sorted items: " + string(items)); println("First: " + string(items[0]) + ", Last: " + string(items[-1])); // 2. Map (keeps insertion order): let student = map("name", "David", "grade", 95, "major", "Computer Science"); student["advisor"] = "Dr. Smith"; println("Keys: " + string(student.keys())); println("Has advisor: " + string(student.has("advisor"))); // 3. Set algebra: let set_a = set(1, 2, 3, 4); let set_b = set(3, 4, 5, 6); println("Union: " + string(set_a.union(set_b))); println("Intersection: " + string(set_a.intersect(set_b))); // 4. Min-Heap Priority Queue: let min_heap = heap("min"); min_heap.push(50); min_heap.push(10); min_heap.push(30); println("Min element popped: " + string(min_heap.pop())); // 10 // 5. Trie (Prefix String Tree): let words = trie(); words.insert("apple"); words.insert("app"); words.insert("application"); words.insert("banana"); println("Words starting with 'app': " + string(words.with_prefix("app"))); return 0; }

12. Linear Algebra: Vector & Matrix Computing

Spid3r provides high-performance, contiguous 64-bit float memory buffers for 1-D vectors and 2-D matrices with zero external dependencies. It replaces NumPy with zero installation required.

Matrix Indexing Architecture

A Spid3r matrix stores its elements in one contiguous row-major buffer. Note:

  • 2-D Indexing: Use m.get(row, col) and m.set(row, col, value) to address cells by row and column.
  • Flat Indexing: m[i] reads or writes a flat cell starting from the top-left (i.e. cell row * cols + col). m[i] returns a float, so m[i][j] is not used.
  • Dimensions: Read with m.rows() and m.cols(). Total cell count is len(m) == rows * cols.
Operation Matrix Method Vector Method Description
Multiplication A.mul(B) v1.dot(v2) Matrix product ($A \cdot B$) / Vector dot product ($\sum u_i v_i$).
Hadamard A.hadamard(B) v1.mul(v2) Element-wise Schur product ($A \odot B$).
Transpose A.transpose() โ€” Returns transposed matrix ($A^T$).
Determinant A.det() โ€” Matrix determinant $\det(A)$ using Gaussian pivoting.
Inverse A.inverse() โ€” Computes $A^{-1}$ via LU decomposition. Throws if matrix is singular.
Linear Solver A.solve(b) โ€” Solves $A X = B$ with partial pivoting.
Statistics A.mean(), A.sum() v.mean(), v.variance(), v.std() High-precision scalar and axis-wise statistics.
NumPy I/O save_npy(path, A) load_npy(path) Save / Load standard NumPy v1.0 .npy binary files.
โšก Example: Vectors, Matrix Multiplication & Inversion
// 1. Vector arithmetic and Euclidean L2 norm: fn main() { let v1 = vector([1.0, 2.0, 3.0]); let v2 = vector([4.0, 5.0, 6.0]); println("Vector dot product: " + string(v1.dot(v2))); // 1*4 + 2*5 + 3*6 = 32.0 println("Vector Euclidean Norm: " + string(v1.norm())); println("Vector mean: " + string(v1.mean())); // 2. Matrix creation & row-major indexing: let A = matrix(2, 2, 0.0); A.set(0, 0, 4.0); A.set(0, 1, 7.0); A.set(1, 0, 2.0); A.set(1, 1, 6.0); println("Matrix A determinant: " + string(A.det())); // 4*6 - 7*2 = 10.0 println("Matrix A trace: " + string(A.trace())); // 4 + 6 = 10.0 // 3. Matrix Inverse via LU decomposition: let A_inv = A.inverse(); println("A^-1 top-left: " + string(A_inv.get(0, 0))); // 0.6 println("A^-1 top-right: " + string(A_inv.get(0, 1))); // -0.7 // 4. Solving linear systems A * x = b: let b = vector([18.0, 10.0]); let x = A.solve(b); println("Solution x: [" + string(x[0]) + ", " + string(x[1]) + "]"); // [3.8, 0.4] return 0; }

13. Concurrency, Tasks & Message Channels

Spid3r runs tasks on an M:N scheduler: a pool of OS threads (sized to the machine, growing on demand up to SPID3R_MAX_WORKERS, default 256) pulling from one shared ready queue, with no global interpreter lock โ€” two CPU-bound tasks really do run at the same time. Spawn background tasks, communicate over buffered channels, and synchronize with mutexes.

Tasks, Join, and Async-Await

  • Task Spawning: spawn fn() { ... } spawns a concurrent task and immediately returns a task handle.
  • Joining Tasks: task.join() blocks until the task finishes and returns its result (or throws if the task encountered an unhandled exception).
  • Async / Await: Functions marked async fn name() { ... } can use await task or await channel.
  • Channels: let ch = channel(); creates a buffered message channel โ€” an unbounded FIFO queue. ch.send(val) appends and returns immediately (it never blocks, so a producer cannot deadlock on a full queue), and ch.recv() blocks until a value is available. Values are dynamically typed, like every other Spid3r binding.
  • Mutex: let mu = mutex(initial_value); guards critical sections with mu.lock() and mu.unlock().
  • Atomic Integers: let a = atomic_int(0); (argument defaults to 0) provides lock-free 64-bit atomic operations across tasks: .load(), .store(val), .exchange(val), .cas(expected, desired), .fetch_add(delta), and .fetch_sub(delta).
โšก Example: Spawn, Task Joining & Channels
// 1. Spawning a concurrent background task: fn main() { let worker = spawn fn() { let acc = 0; for let i = 1; i <= 100; i += 1 { acc += i; } return acc; }; // Wait for task completion: let result = worker.join(); println("Sum 1..100 from worker task: " + string(result));return 0; }

14. Standard Library & System API Reference

Spid3r includes a comprehensive set of built-in global functions and 29 first-party modules covering networking, cryptography, formats, compression, civil calendars, and machine learning.

Console, System & Filesystem Built-ins

Spid3r provides essential global primitives for terminal output, user input, filesystem operations, and mathematical calculations:

Category Function Signature Description
Console println / print println(msg) / print(msg) Prints message to standard output (with or without newline).
Console eprint eprint(msg) Prints message to standard error stream.
Console read_line read_line() Reads a line of input from standard input (returns string).
Runtime spid3rinfo spid3rinfo(mode?) Comprehensive runtime information (version, OS, config, modules, memory). Default returns a map. Modes: "html", "json", "text", "map". Optional second truthy argument disables redaction.
Runtime config_get config_get(key) Live spid3r.ini value as a string, or null. Sensitive-looking keys are redacted.
Runtime config_set config_set(key, value) Set a known, validated key in the live configuration.
Runtime config_reload config_reload() Re-read spid3r.ini; returns the path that was loaded (or "").
Filesystem read_file read_file(path) Reads complete file contents into a UTF-8 string; returns result (ok(string) or err(string)).
Filesystem write_file write_file(path, data) Writes string or bytes to specified path; returns result (ok(null) or err(string)).
Filesystem file_exists file_exists(path) Returns true if path exists on disk (bool).
Filesystem is_dir / is_file is_dir(path) / is_file(path) Type predicates: true only when the path is a directory / a regular file (symlinks followed). Any other outcome is false โ€” never throws.
Filesystem file_size / file_mtime file_size(path) / file_mtime(path) Size in bytes (ok(int)) and modification time in Unix seconds (ok(float)); a missing path or a directory is err(string).
Filesystem list_dir list_dir(path) Entry names of a directory as ok(list) โ€” sorted, excluding . and ..; err(string) for a missing path or a non-directory.
Filesystem remove_file / file_delete remove_file(path) Deletes file at path from disk; returns result (ok(null) or err(string)).
Filesystem cwd / chdir cwd() / chdir(path) Read (ok(string)) or change (ok(null)) the process working directory. Process-global; embedding hosts should treat chdir as a startup-time call.
Filesystem exe_path exe_path() Absolute path of the running executable as result (ok(string) / err(string)). Use it to locate files shipped next to the binary: the working directory is not the program's folder when a process is started from a shortcut, the Run dialog, a scheduled task, or Start-Process.
Filesystem (Handles) file_open / file_close file_open(path, mode?) / file_close(id) Open a streaming file handle (modes "r", "w", "a", "r+", "w+") returning a monotonic non-reused integer handle ok(id) (max 1,024 open handles; path guarded by security.basedir).
Filesystem (Handles) file_read / file_read_line / file_write file_read(id, max_bytes) / file_read_line(id) / file_write(id, data) Buffered handle I/O returning ok(bytes), ok(string?) (null at EOF), or bytes written ok(int) for string/bytes payloads.
Filesystem (Handles) file_seek / file_tell / file_flush / file_fsync file_seek(id, pos, whence?) / file_tell(id) / file_flush(id) / file_fsync(id) Cursor positioning ("start", "current", "end"), byte offset query, stream flush, and durable disk sync (Windows FlushFileBuffers / Unix fsync).
Filesystem (Handles) file_lock / file_unlock file_lock(id, exclusive?) / file_unlock(id) Advisory/OS region file locking (Windows LockFileEx/UnlockFileEx, Unix flock; shared when exclusive == false).
Process spawn_process spawn_process(cmd, args[, opts]) Starts a child process and waits for it, returning its exit code as result (ok(int) / err(string)). opts is an optional map supporting: cwd (string, checked against security.basedir), env (map of environment variables passed to the child), and hidden (bool; passes CREATE_NO_WINDOW on Windows, ignored elsewhere).
Signals & Shutdown signal_requested / shutdown_requested signal_requested(name, clear?) / shutdown_requested() Cooperative signal polling for "INT", "TERM", "HUP", "USR1", "USR2" (Unix signal handlers coordinated with agent SIGHUP reload; Windows console control handler for shutdown).
Event Loop & Timers poll_create / poll_add / poll_wait poll_create() / poll_wait(ep, ms) / after_ms(ms) / timer_check(id) / timer_cancel(id) OS readiness polling over epoll (Linux), kqueue (macOS/BSD), or wepoll (Windows), paired with monotonic millisecond timers (also poll_mod, poll_del, poll_close).
Networking & UDS http_fetch / http_write_response_chunk / uds_* http_fetch(m, url, hdrs?, body?) / uds_listen(path) / uds_connect(path) HTTP(S) client returns response map with both UTF-8 "body" and exact "body_bytes" (bytes); chunked response primitive http_write_response_chunk(conn, bytes, final); Unix domain sockets uds_listen / uds_connect (unlinked on close on Unix; usage error on Windows; paths guarded by security.basedir).
Filesystem env / env_set env(name) / env_set(name, value) Read an environment variable as option, or set one for env() and later spawned processes (ok(null) / err(string)).
Paths path_join / basename / dirname / path_ext path_join(a, b, ...) etc. Pure lexical path helpers: no filesystem access; both / and \ count as separators on every OS and results normalize to /. A later absolute component resets path_join; dirname("/") = "/"; a leading dot is not an extension.
Result is_ok / is_err is_ok(res) / is_err(res) Tests whether a result value is ok or err (returns bool).
Math abs / sqrt abs(n) / sqrt(n) Absolute value and square root.
Math pow / clamp pow(b, e) / clamp(x, lo, hi) Exponentiation and range bounding.
Math min / max min(a, b) / max(a, b) Minimum / maximum of two values.
Math random_int random_int(min, max) Clock-seeded pseudo-random integer in [min, max] (use random_bytes for cryptographic tokens).
Math random_float random_float([min, max]) Uniform pseudo-random 64-bit float in [0.0, 1.0) (or [min, max) when specified).
Runtime (Monotonic) now / now_ms / elapsed_ms now() / now_ms() / elapsed_ms() Monotonic elapsed timer in milliseconds (starts at 0 on first call; for measuring intervals and benchmarks).
Runtime (Wall clock) unix_time / unix_ms unix_time() / unix_ms() Current Unix epoch timestamp in whole seconds / milliseconds (for real-world dates, times, and expiration).
Runtime type_of / len type_of(val) / len(container) Runtime type inspection and element count. Calling len() on a result prompts calling .unwrap().

Cryptography, Passwords & Encodings

Function Signature Standard Description
sha256 sha256(str_or_bytes) FIPS 180-4 Returns 64-character lowercase hexadecimal SHA-256 digest.
hmac_sha256 hmac_sha256(key, message) RFC 2104 HMAC-SHA256 signature as lowercase hex string.
pbkdf2_sha256 pbkdf2_sha256(pw, salt, iters) RFC 8018 Password key derivation returning 32-byte key as hex.
sha512 sha512(str_or_bytes) FIPS 180-4 Returns 128-character lowercase hexadecimal SHA-512 digest. Accepts the same string-or-bytes input convention as sha256.
ed25519_keypair ed25519_keypair(seed) RFC 8032 Derive a keypair from a 32-byte hex seed (or call with no arguments for OS entropy). Returns a map with public (64 hex chars) and secret (128 hex chars = seedโ€–public).
ed25519_sign ed25519_sign(secret_hex, message) RFC 8032 Sign a message; returns the 64-byte signature as 128 lowercase hex characters.
ed25519_verify ed25519_verify(public_hex, message, sig_hex) RFC 8032 Verify a signature; returns true/false, never an error โ€” malformed input verifies as false. Strict verification: rejects non-canonical S, small-order public keys and off-curve points. Not constant-time; fine for verifying third-party data, avoid secret-key operations in hostile environments.
bcrypt_hash bcrypt_hash(pw, cost=10) OpenBSD Generates canonical $2b$... bcrypt password hash.
bcrypt_verify bcrypt_verify(pw, hash) Constant-Time Verifies password against bcrypt hash in constant time (bool).
argon2 argon2(pw, salt, t, m, p, var, len) RFC 9106 Argon2 ("i", "d", or "id") password hasher.
secure_compare secure_compare(a, b) Constant-Time Timing-attack resistant byte equality comparison (bool).
base64_encode / decode base64_encode(d) / decode(s) RFC 4648 Standard Base64 encoder/decoder (decode returns bytes).
hex_encode / decode hex_encode(d) / decode(s) Two hex chars/byte Converts string or bytes to lowercase hex (and vice versa).
โš ๏ธ Cryptography Trust Statement (R12.1)

Spid3r's cryptographic primitives (sha256, sha512, hmac_sha256, pbkdf2_sha256, ed25519_*, bcrypt_*, argon2) and TLS 1.3 client in src/stdlib/net.rs are self-contained pure-Rust implementations with zero external dependencies, and have not yet undergone an independent third-party cryptographic audit. The primitives are verified by internal RFC test vectors and negative/malleability tests โ€” Ed25519 uses a constant-time cswap Montgomery ladder with canonical S < L and small-order-key rejection โ€” but are not formally proven side-channel hardened, and R12.1 remains open pending an external review report. For high-stakes production workloads or side-channel-hostile environments, terminate TLS or run secret-key operations behind an audited reverse proxy or HSM. Cryptographic randomness comes only from random_bytes(n) (OS CSPRNG: /dev/urandom, RtlGenRandom); random_int/random_float/random_normal are clock-seeded xorshift and never suitable for tokens. jwt_verify is constant-time with a 60-second exp/nbf skew but checks HS256 only and does not pin alg to a policy โ€” do not feed it tokens from an issuer you do not control.

Serialization, Compression & Regular Expressions

Function Signature Format Description
json_parse json_parse(json_str) RFC 8259 Parses JSON text into Spid3r maps, lists, numbers, booleans, and null. Supports \uXXXX surrogate pairs.
json_write json_write(val) RFC 8259 Serializes any Spid3r structure into a compact JSON string.
parse_csv parse_csv(text, opts?) RFC 4180 Parses CSV with automatic type inference into {"headers": [...], "rows": [...]}.
gzip_compress / gunzip gzip_compress(bytes) RFC 1952 Compresses and decompresses GZIP streams with CRC-32 integrity checking.
regex_match regex_match(pat, text) Thompson NFA Linear-time regex matching (supports \d, \w, \s, groups, ranges, anchors).
regex_find_all regex_find_all(pat, text) Thompson NFA Returns a list of all matching substrings in text.
datetime_parts datetime_parts(epoch_sec) Civil Calendar Decomposes Unix epoch timestamp into year, month, day, hour, minute, second, weekday.
format_datetime format_datetime(sec, fmt) Strftime Formats timestamp with %Y-%m-%d %H:%M:%S in UTC.
โšก Example: JSON, Crypto, Regex & DateTime
// 1. JSON serialization and parsing: fn main() { let config = map("app", "Spid3r", "port", 8080, "features", ["wasm", "gc", "blas"]); let json_text = json_write(config); println("JSON Output: " + json_text); let parsed = json_parse(json_text); println("Parsed app name: " + parsed["app"]); // 2. Cryptographic hashing: let hash = sha256("spid3r-token-secret"); println("SHA-256: " + hash); let sig = hmac_sha256("api-secret-key", "user=100&action=login"); println("HMAC-SHA256: " + sig); // 3. Regular Expressions (Thompson NFA): let email = "developer@spid3r.dev"; let is_valid = regex_match("^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$", email); println("Is valid email: " + string(is_valid)); let numbers = regex_find_all("\\d+", "ports 8080, 8081, 9090"); println("Extracted port numbers: " + string(numbers)); // 4. Date & Time (Howard Hinnant Civil Calendar): let parts = datetime_parts(0); // Epoch 0 println("Epoch 0 date: " + string(parts["year"]) + "-0" + string(parts["month"]) + "-0" + string(parts["day"])); println("Formatted: " + format_datetime(0, "%Y-%m-%d %H:%M:%S UTC")); return 0; }

The 29 Standard Library Modules (stdlib/)

Spid3r includes 29 zero-dependency standard library modules in stdlib/, automatically discoverable via import "name" as alias; or selective symbol imports (import { ClassA, helper as h } from "name";). In Spid3r, top-level symbols in imported files must be marked with the public modifier (e.g., public fn, public class, public interface, public enum, public let) to be exported; unexported names are private to their module and cannot be accessed externally.

Module Import Statement Primary Purpose
web.matrix import "web" as w; Express-style HTTP/HTTPS server (bind_tls), keep-alive idle eviction, Transfer-Encoding: chunked request decoding, streaming responses (stream_response), and URL param decoding.
dataframe.matrix import "dataframe" as df; Pandas-style tabular dataframes: column selection, filtering, group-by, joining.
stats.matrix import "stats" as st; Descriptive statistics, normal/poisson distributions, linear regression, ANOVA.
ml.matrix import "ml" as ml; K-Means clustering, logistic regression, decision trees, k-nearest neighbors.
mathx.matrix import "mathx" as mx; Advanced math: complex numbers, polynomial roots, fast Fourier transform (FFT).
toml.matrix import "toml" as toml; TOML v1.0 configuration parser and serializer.
yaml.matrix import "yaml" as yaml; YAML document parser and validator.
xml.matrix import "xml" as xml; XML element tree parser, attributes, text extraction, serialization.
tz.matrix import "tz" as tz; IANA tzdata 2026e timezone conversions (598 authoritative zones).
unicode.matrix import "unicode" as uc; UCD 15.1 NFC/NFD normalization, full case folding, combining mark stripping.
smtp.matrix import "smtp" as smtp; SMTP client with authentication, TLS upgrade, MIME multi-part attachments.
ftp.matrix import "ftp" as ftp; FTP client with active/passive transfers, directory listings, binary downloads.
session.matrix import "session" as sess; Web session management with signed cookies and HMAC tamper protection.
middleware.matrix import "middleware" as mw; CORS headers, rate limiting, request logging, authentication middleware.
testkit.matrix import "testkit" as tk; Unit test runner with assertions, test suites, and colorized reporting.
assert.matrix import "assert" as a; Minimal test assertions (`ok`, `eq`, `fail`) for scripts and tests.
mock.matrix import "mock" as mk; Test doubles โ€” stubs and spies โ€” for isolating units under test.
http.matrix import "http" as http; HTTP/1.1 web framework: router, request/response helpers, chunked parsing, keep-alive timers, static files, and streaming responses (http_stream over lists, callbacks, sockets, and file_open handles).
fastcgi.matrix import "fastcgi" as fcgi; FastCGI responder for running a Spid3r app behind nginx/Apache.
multipart.matrix import "multipart" as mp; `multipart/form-data` parsing for file uploads.
event.matrix import "event" as ev; Event loop backed by OS readiness primitives (epoll/kqueue/wepoll plus non-blocking timers).
gen.matrix import "gen" as gen; Generators built on the scheduler (`yield`-style lazy sequences).
bigint.matrix import "bigint" as bi; Arbitrary-precision integers (also behind the `123n` literal).
complex.matrix import "complex" as cx; Complex numbers with arithmetic and magnitude helpers.
decimal.matrix import "decimal" as dec; Exact fixed-point decimals for money-style arithmetic.
rational.matrix import "rational" as rat; Exact rational numbers (fractions) with normalization.
semver.matrix import "semver" as sv; Semantic-version parsing, comparison and range matching.
uuid.matrix import "uuid" as uuid; UUID v4 generation (CSPRNG-backed).
ssh.matrix import "ssh" as ssh; SSH/SFTP by driving the system OpenSSH client.

A module is imported by name, but its exported functions keep the module's own prefix โ€” do not guess a bare name. The ones most often mistyped:

Import Real export names
import "uuid" as uuid; uuid.uuid_v4() โ€” not uuid4()
import "toml" as toml; toml.toml_parse(text), toml.toml_get(m, path) โ€” not parse()
import "event" as ev; ev.start(), ev.run(loop, timeout_ms), ev.stop(loop), ev.on_readable(loop, conn, cb) โ€” not new()
bcrypt_hash (builtin) bcrypt_hash(pw, cost=10), bcrypt_verify(pw, hash) โ€” no import needed; cost must be 4โ€“31

14b. Runtime Configuration & Remote Management

Spid3r has spid3r.ini, spid3rinfo(), and an encrypted management agent. One installed binary can be inspected, reconfigured, and operated from another machine โ€” without a shell.

The host runs spid3r agent start on [agent] bind:port; an operator's machine pairs once with a one-time 48-hex code and then speaks spid3r-agent/1 over TCP with ChaCha20 plus HMAC-SHA256. Every request is role-checked (viewer, operator, admin) against an ops allow-list and dispatched to a predefined operation; calls land in audit.log and config is saved atomically. Below, the spid3r.ini precedence chain and the remote config.set deny-list.
Runtime configuration & remote management. The wire never reaches a shell: an unknown operation is refused, every call is role-checked and audited, and the schema-derived frozen set โ€” every security.* and logging.* key, agent.role_*, the agent listener switches and the capability deny lists โ€” is read-only over the wire.

The spid3r.ini File

A single INI file configures the runtime. The discovered candidates are tried in this order (first hit wins) โ€” except for the authoritative SPID3R_INI pin described below:

  1. $SPID3R_INI โ€” when set and non-empty, the only ini candidate
  2. spid3r.ini next to the spid3r executable
  3. platform config dir (%APPDATA%\Spid3r, $XDG_CONFIG_HOME/spid3r, ~/.config/spid3r)
  4. ./spid3r.ini

SPID3R_INI is authoritative: when it is set and non-empty it is the only ini candidate. If that path is missing or unreadable the loader uses compiled defaults only (it never falls through to the machine or working-directory ini), records a typed pin failure for local diagnostics, and program-executing commands (run, check, compile, exec, test, bench, debug, gui, repl, magic) refuse to start with exit 1. --version, help, info, config path and config validate remain usable to diagnose the pin, and remote config.set refuses to persist while it is broken โ€” the local control config set / spid3r config init paths are how an operator repairs it. Restart a long-lived process that initialized its policy while the pin was broken; its fail-closed policy remains sticky until restart. spid3r config validate reports a broken pin as a problem (exit 1) and treats an unknown flag as a usage error (exit 2); local diagnostics keep policy values raw so the bad entry is visible. spid3r control config validate exits 1 with errors on stderr when valid:false.

Precedence, highest first: process environment โ†’ [env.<name>] (selected by SPID3R_ENV or [spid3r] env) โ†’ the INI file โ†’ compiled defaults. Every layer is validated with the same rules: a value that fails its type/range check is ignored with a note and the compiled default stays in effect (a bool must be true/false/1/0/on/off/yes/no; a typo like ture is an error, not a silent true). A UTF-8 BOM at the start of the file is tolerated. Unknown keys are reported by spid3r config validate and never abort a run. Live reload: spid3r config reload, SIGHUP on Unix, or the agent's reload operation. The [logging] keys enabled, file and audit are honored (the SPID3R_LOG/SPID3R_LOG_FILE env vars override the INI). Config and state files are written atomically (temp file + rename).

โšก Example: A Minimal spid3r.ini
[spid3r] engine = vm max_workers = 256 gc = true [runtime] wall_clock_limit_ms = 0 ; fatal execution deadline in ms; 0 = unlimited memory_limit_mb = 0 ; soft cap on script-value heap (catchable); 0 = unlimited timezone = UTC ; UTC, a fixed offset like +06:00, or an IANA name [net] timeout_ms = 30000 ; HTTP framing deadline in ms (slowloris guard); 0 disables max_body = 8388608 ; largest accepted request body in bytes (413 above it) max_headers = 64 ; most request headers accepted (431 above it) keepalive_ms = 5000 ; idle window between keep-alive requests; 0 = close after each [errors] level = info ; error|warn|info โ€” minimum surfaced diagnostic level display = false ; mirror diagnostic log lines to stderr log_file = ; dedicated error-log path; empty = shared log only [agent] enabled = false bind = 127.0.0.1 ; 0.0.0.0 to accept other hosts port = 9527 [security] allow_remote = false basedir = ; open_basedir analog: paths scripts may touch disable_natives = ; natives replaced by a catchable denial ; WARNING: basedir alone is NOT a jail - Process ; natives (spawn_process/exec_capture) reach any ; path the OS user can. Pair them, e.g.: ; disable_natives = spawn_process, exec_capture, run_payload, run_payload_strict [modules] disable = ; module names that fail to import assert_active = true ; false makes assert()/assert_eq() no-ops [packages] registry = ; local dir or http(s):// URL (alias: SPID3R_REGISTRY) trust_pk = ; 64-hex Ed25519 public key for registry signatures (alias: SPID3R_TRUST_PK) allow_insecure_git = false allow_local_git = false [paths] cache_dir = ; extracted-runtime cache home (alias: SPID3R_CACHE_DIR) state_dir = ; agent store, audit log, pid file (alias: SPID3R_STATE_DIR) sqlite_path = ; SQLite library tried first (alias: SPID3R_SQLITE_PATH) [cache] bytecode_dir = ; bytecode cache for `spid3r run`; empty = disabled [env.production] agent.enabled = true agent.bind = 0.0.0.0
Section Key Default Effect
[spid3r] env default Configuration environment name; selects [env.<name>] overrides (also set by SPID3R_ENV).
[spid3r] engine vm Default engine for spid3r run when --engine is omitted (tree or vm).
[spid3r] max_workers 256 Ceiling for the grow-on-demand task pool.
[spid3r] queue_cap 65536 Bound on the shared ready queue.
[spid3r] deadlock_abort true Abort when no task can ever wake a wait.
[spid3r] task_drain_ms 2000 How long exit waits for still-running tasks.
[spid3r] gc true Automatic cycle collection at statement boundaries.
[memory] max_steps 0 Execution budget in engine steps; 0 disables the limit.
[memory] max_call_depth 1000 Interpreter call-depth limit.
[memory] gc_threshold 0 Live-cell threshold that triggers an extra collection; 0 uses the default.
[memory] allocation_cap_mb 0 Soft cap on a single allocation, in MiB; 0 disables.
[runtime] wall_clock_limit_ms 0 Fatal wall-clock deadline for spid3r run; expiry raises the uncatchable wall-clock limit exceeded (N ms) on the tree and VM engines (checked at statement boundaries, the VM poll, and around blocking native calls). 0 = unlimited. Not wired to the native AOT backend. Applied live by config reload.
[runtime] memory_limit_mb 0 Catchable soft cap (out of memory: script exceeded memory_limit_mb (N)) on the total heap used by script values: strings by byte length, containers by element count; approximate (refreshes at most every 25 ms, and values in untracked environments are not counted). 0 = unlimited. Applied live by config reload.
[runtime] timezone UTC Default zone for the datetime natives: UTC, a fixed offset like +06:00, or an IANA name from tz_names(). Unknown values are rejected at load time with a note.
[net] timeout_ms 30000 Deadline for HTTP framing in ms (the slowloris guard); 0 disables it. SPID3R_NET_TIMEOUT_MS is its env alias. Applied live: re-read per accepted connection, so config reload applies to connections accepted afterwards.
[net] max_body 8388608 Largest accepted HTTP request body, in bytes. A declared Content-Length above it is rejected with 413 Payload Too Large before the handler runs.
[net] max_headers 64 Most request headers accepted per request; more are rejected with 431 Request Header Fields Too Large.
[net] keepalive_ms 5000 Idle window between keep-alive requests on a connection; 0 closes after each response (Connection: close semantics).
[debug] enabled false Master debug switch (verbose diagnostics, extra checks).
[debug] log_level info Minimum level written to the diagnostic log (error|warn|info|debug|trace).
[debug] log_stdout false Mirror diagnostic log lines to stderr (alias of errors.display).
[logging] enabled true Write the append-only diagnostic log. Frozen for remote config.set.
[logging] file empty Log path override (empty = platform default; SPID3R_LOG_FILE env alias). Frozen for remote config.set.
[logging] rotate_mb 1 Rotate the log once it passes this size (1โ€“1024 MiB). Frozen for remote config.set.
[logging] audit true Append remote-management actions to the audit log. Frozen for remote config.set.
[errors] level info Minimum level for script-visible diagnostics (error/warn/info); diagnostics below the level are not surfaced.
[errors] display false Mirror diagnostic log lines to stderr. Canonical spelling of the documented debug.log_stdout alias (an explicit errors.display in any layer wins, otherwise the alias applies).
[errors] log_file empty Dedicated error-log path for error-level diagnostics (they also keep flowing to the shared log); empty = shared log only.
[modules] path empty Extra module search directories, semicolon-separated (alias of SPID3R_PATH).
[modules] plugins empty Plugin search path (defaults to SPID3R_PLUGINS).
[modules] allow_host true Allow host-registered plugins (compiled into the embedding host).
[modules] allow_wasm true Allow loading sandboxed .wasm plugins.
[security] deny_filesystem false Deny filesystem natives to scripts (embed-style policy). Frozen for remote config.set.
[security] deny_process false Deny process natives to scripts. Frozen for remote config.set.
[security] deny_network false Deny network natives to scripts. Frozen for remote config.set.
[security] deny_environment false Deny environment natives to scripts. Frozen for remote config.set.
[security] allow_remote false Serves the management agent when true; the agent also serves when [agent] enabled is true (either switch starts it, bound to agent.bind).
[security] basedir empty open_basedir analog: when non-empty, every native that takes a script-controlled path may only touch paths inside these directories (; or , separated). Symlinks and junctions are resolved before comparing, the match is component-wise (never a raw string prefix), and unresolvable paths are rejected. Denials are catchable (path outside basedir: โ€ฆ), raised before the filesystem is touched. Honest limits: the check canonicalizes the nearest existing ancestor for not-yet-created write targets, matches component-wise (case-insensitive on Windows), resolves symlinks/junctions, and refuses a basedir entry that does not exist; TOCTOU is inherent (a file or link swapped between check and open is not covered โ€” an empirical 200,000-iteration race produced 0 escapes, an observation not a guarantee), hard links already pointing outside are indistinguishable, and the policy is a per-process filter over built-in natives, not a kernel privilege boundary or a resource limit. basedir alone is not a jail: Process natives (spawn_process/exec_capture) reach any path the OS user can โ€” pair with disable_natives = spawn_process, exec_capture, run_payload, run_payload_strict or run under --sandbox.
[security] disable_natives empty Named natives replaced by the catchable disabled by configuration denial. Can only narrow capability; composes with --sandbox; unknown names are noted and ignored.
[modules] disable empty Module names that fail to import with module disabled by configuration: 'โ€ฆ', on every engine (the gate lives in the shared resolver). The name comparison follows the platform's default file-name case rules: case-insensitive on Windows and macOS (like NTFS and the APFS default), case-sensitive on Linux.
[modules] assert_active true When false, assert()/assert_eq() become no-ops returning true (the zend.assertions analog).
[packages] registry empty Canonical home of the SPID3R_REGISTRY env alias: local registry root directory or remote http(s):// base URL.
[packages] trust_pk empty Canonical home of SPID3R_TRUST_PK: 64-character hex Ed25519 public key used to verify sha256.txt.sig on spid3r install (refuses unsigned or tampered packages when set). Frozen for remote config.set.
[packages] allow_insecure_git false Canonical home of SPID3R_ALLOW_INSECURE_GIT: permit http:// and git:// git dependencies. Frozen for remote config.set.
[packages] allow_local_git false Canonical home of SPID3R_ALLOW_LOCAL_GIT_DEPS: permit bare-path and file:// git dependencies. Frozen for remote config.set.
[paths] cache_dir empty Canonical home of SPID3R_CACHE_DIR (generic spelling SPID3R_PATHS_CACHE_DIR): cache root for the extracted bundle runtime (runtime/). Empty = platform default. Frozen for remote config.set.
[paths] state_dir empty Canonical home of SPID3R_STATE_DIR (generic spelling SPID3R_PATHS_STATE_DIR): paired devices, admin-client book, audit log, pid file. Empty = platform default. Frozen for remote config.set.
[paths] sqlite_path empty Canonical home of SPID3R_SQLITE_PATH (generic spelling SPID3R_PATHS_SQLITE_PATH): full path of the SQLite shared library tried before the default lookup. Read once per process. Frozen for remote config.set.
[cache] bytecode_dir empty Bytecode cache for spid3r run: the compiled .matrixc container is keyed by the source's SHA-256, verified by its v3 checksum on load, and recompiled fresh when corrupt โ€” a hit is byte-identical to compiling. Empty = disabled. Frozen for remote config.set.
[agent] enabled false Start the management agent when spid3r agent start runs. Frozen for remote config.set.
[agent] bind 127.0.0.1 Listen address. 127.0.0.1 is local-only; 0.0.0.0 is required for other hosts. Bind the local address unless you actually need other hosts. Frozen for remote config.set.
[agent] port 9527 TCP port of the encrypted management API (1โ€“65535). Frozen for remote config.set.
[agent] max_connections 16 Concurrent management connections (1โ€“256). Frozen for remote config.set.
[agent] request_timeout_ms 10000 Per-request deadline in ms (1000โ€“300000). Frozen for remote config.set.
[agent] max_body 262144 Largest accepted (decrypted) request body, in bytes (1024โ€“8388608). Frozen for remote config.set.
[agent] pairing_ttl_sec 300 How long a one-time pairing code remains valid (30โ€“3600 s). Remotely settable.
[agent] name empty Friendly name advertised to administrators (empty = hostname). Remotely settable.
[agent] role_viewer status,ping,info,config.get,logs.tail,modules.list,devices.list Operations granted to the viewer role. Frozen for remote config.set.
[agent] role_operator status,ping,info,config.get,config.set,logs.tail,modules.list,devices.list,ops.run,config.reload Operations granted to the operator role. Frozen for remote config.set.
[agent] role_admin * Operations granted to the admin role (* = all). Frozen for remote config.set.

Scripts read and write the live values with config_get(key), config_set(key, value) and config_reload(). The CLI is spid3r config get|set|list|path|validate|reload|init.

The spid3rinfo() Builtin

Spid3r's own runtime introspection. One collector, three renderers, one redaction pass.

Call Result
spid3rinfo() Map of section -> {key: value}
spid3rinfo("html") Standalone HTML page
spid3rinfo("json") JSON string
spid3rinfo("text") Aligned plain text
spid3rinfo("map", true) Map without redaction (local operator choice)

Sections: Spid3r (version, language, Rust, profile), System (OS, arch, memory, CPUs, Process RSS / Process RSS bytes), Installation (exe, ini, log, state), Features (gui, native (LLVM AOT), tls, plugins-wasm, agent, spid3rinfo), Modules (embedded stdlib, plugins), Configuration, Environment, Memory (interpreter), AOT Memory (bump-slab arena allocation & slab counters), and Agent. On wasm32 (the playground build) the Environment and Agent sections are omitted โ€” there is no OS process or environment there. Keys containing or ending in password, passwd, pwd, passphrase, token, secret, shared, credential, authorization, cookie, api_key, apikey, private, or any โ€ฆ_key/โ€ฆ-key name are replaced with [redacted sha256:<12 hex>]. Remote callers never receive unredacted output.

Local spid3rinfo() includes the configuration notes and any broken-pin details for operators. Remote info and dump_diagnostics omit the configuration-notes row, and remote info redacts its Installation path rows. Local validation diagnostics keep policy values such as security.basedir and paths.* raw so an operator can see which entry is wrong; credential-like values are hashed.

โšก Example: spid3rinfo() from a Script
fn main() { let m = spid3rinfo(); println("version: " + m["Spid3r"]["Spid3r version"]); println("os: " + m["System"]["OS"]); println("json is " + type_of(spid3rinfo("json"))); let port = config_get("agent.port"); println("agent.port = " + port); println(config_set("agent.name", "docs-demo")); println("name is now " + config_get("agent.name")); return 0; }

The Management Agent

spid3r agent start listens on [agent] bind:port and speaks spid3r-agent/1 over TCP. There is no unrestricted shell. Every operation is a predefined name checked against the caller's role. The same protocol runs on Windows, Linux and macOS with no certificate store and no third-party crates.

โšก Example: Pair and Operate Another Host
# On the host that will be managed: $ spid3r agent start $ spid3r agent pair --role operator --name laptop-1 pairing code (operator): 9f2cโ€ฆ # 48 hex chars, single use, short TTL # On the administrator's machine: $ spid3r admin pair 192.168.1.10:9527 9f2cโ€ฆ --name laptop-1 $ spid3r admin --host laptop-1 ping $ spid3r admin --host laptop-1 status $ spid3r admin --host laptop-1 info $ spid3r admin --host laptop-1 config.get --key agent.port $ spid3r admin --host laptop-1 config.set --key agent.name --value web-1 $ spid3r admin --host laptop-1 logs.tail --lines 80 $ spid3r admin --host laptop-1 modules.list $ spid3r admin --host laptop-1 devices.list $ spid3r admin --host laptop-1 ops.run --op-name self_test $ spid3r admin --host laptop-1 reload

Predefined operations (ops.run --name โ€ฆ): ping, reload_config, rotate_log, gc_collect, self_test, dump_diagnostics, restart_runtime. restart_runtime acknowledges the caller first, then exits and spawns the same executable; the new process retries its bind for a few seconds so the handover cannot lose the port. Run the agent under a service manager (Windows service, systemd) in production.

Wire Protocol

Pairing is out-of-band: the host prints a one-time code; the client sends it once. Both sides derive the long-term shared key as SHA-256(code). The shared key never crosses the wire. After pairing, every data frame is:

magic "SPID3R01" | version | kind | device[16] | nonce[12] | len u32 | payload

kind = pair is plaintext JSON (the code is the secret). kind = data is ChaCha20 encrypted and HMAC-SHA256 authenticated (encrypt-then-MAC). Subkeys come from HKDF-SHA256. The frame header is AAD, so a swapped payload fails the tag. Nonces are random 96-bit values; the server keeps a server-wide replay window (device + nonce, entries expire after ~1 hour, capped at 4096 frames) โ€” a captured frame cannot be replayed on a fresh connection or after a restart. A frame whose declared size exceeds agent.max_body (+16 bytes of tag) is rejected straight after the header, before the body is read.

Roles, Devices and Audit

Role May call
viewer status, ping, info, config.get, logs.tail, modules.list, devices.list
operator viewer + config.set, ops.run, config.reload
admin everything, including devices.revoke and pair.create

Role lists are configurable via agent.role_viewer / role_operator / role_admin (* = all). Every accepted or rejected operation is one JSON line in <state>/audit.log. Disable with [logging] audit = false โ€” locally only: the remote config.set op refuses a schema-derived set of security-relevant keys โ€” every security.* and logging.* key, agent.role_*, the agent enabled/bind/port/max_connections/request_timeout_ms/max_body switches, the package/module policy keys (modules.disable, packages.allow_insecure_git, packages.allow_local_git, packages.trust_pk), and the path keys (paths.cache_dir, paths.state_dir, paths.sqlite_path, cache.bytecode_dir) โ€” so a paired operator cannot grant itself more operations, silence the audit trail, re-bind the listener, widen the script deny lists, or relocate the host's write targets. Benign keys (e.g. agent.name, packages.registry, modules.assert_active) stay remotely settable; change the rest by editing spid3r.ini on the host.

State files: devices.json (paired devices), clients.json (admin-client known hosts), audit.log. <state> is %LOCALAPPDATA%\Spid3r\state, $XDG_STATE_HOME/spid3r, the paths.state_dir ini key, or SPID3R_STATE_DIR (env alias; process env beats the INI). Those files are 0600 on Unix.

Security notes: scripts still run with process authority (see the trust model). The agent is a management plane, not a sandbox. There is no exec, no spawn_process, and no path-based file write. Remote info and config.get are always redacted. Devices and config state are saved atomically (temp file + rename). Pair on a network you trust: the code is high-entropy and short-lived, but it is not a PAKE.

One more pairing rule for operators: security.basedir confines only the guarded path-taking natives. A script with Process capability reaches arbitrary paths through spawn_process/exec_capture no matter what basedir says โ€” deny those natives (disable_natives = spawn_process, exec_capture, run_payload, run_payload_strict) or run under --sandbox for a meaningful jail.

Operator deep dive โ€” run-time & HTTP limits, diagnostics, audit records, cross-platform paths

Operator Reference: Limits, Audit and Cross-Platform Details

Run-time limits. runtime.wall_clock_limit_ms arms a wall-clock deadline for spid3r run โ€” expiry is a fatal, uncatchable wall-clock limit exceeded (N ms) error on the tree and VM engines, checked at statement boundaries, at the VM's instruction poll, and around blocking native calls (residual granularity is one blocking operation). The native AOT backend is not wired to it. Deadline and cap are Arc-shared with spawned tasks and module interpreters, and config reload applies new values live. runtime.memory_limit_mb is a catchable soft cap on total script-value heap (out of memory: script exceeded memory_limit_mb (N) โ€” a program may catch it, drop the offending container, and continue). Accounting covers every tracked heap cell (containers, instances, closures, classes), counts strings by byte length and containers by element count, ignores free-standing scalars/strings in untracked environments, and refreshes at most every 25 ms โ€” so the cap is approximate by design. runtime.timezone sets the datetime default: UTC, a fixed offset like +06:00, or an IANA name from tz_names(); anything else is rejected at load with a note and UTC stays in effect.

HTTP serving limits. net.timeout_ms is the per-connection framing deadline (slowloris guard; 0 disables; alias SPID3R_NET_TIMEOUT_MS); net.max_body caps the request body (larger โ†’ 413 before the handler runs); net.max_headers caps header count (larger โ†’ 431); net.keepalive_ms is the idle window between keep-alive requests (0 closes after each response). A server may override all four per listen via its options map (timeout_ms, max_body_bytes, max_headers, idle_timeout_ms); an explicit option beats the INI default, and all four are re-read per accepted connection so config reload applies to later connections.

Diagnostics and script policy. errors.level (error|warn|info, default info) gates the script-visible diagnostics channel; errors.display mirrors diagnostic log lines to stderr (canonical spelling of the debug.log_stdout alias); errors.log_file routes error-level diagnostics to a dedicated file when set. security.disable_natives replaces named natives with a catchable disabled by configuration denial (unknown names noted and ignored); modules.disable makes importing a named module fail with module disabled by configuration: 'โ€ฆ' under the platform's default filename case rules (case-insensitive on Windows/macOS, case-sensitive on Linux); modules.assert_active = false turns assert()/assert_eq() into no-ops returning true. The policy keys (security.basedir, security.disable_natives, modules.disable) are read once per process (a script cannot widen them mid-run with config_set) and are frozen for remote config.set. packages.registry is the canonical home of SPID3R_REGISTRY; packages.allow_insecure_git / packages.allow_local_git are the canonical homes of their env aliases. Scripts read and write live values with config_get, config_set and config_reload(); sensitive-looking values are redacted on read unless the caller is local. The config.set freeze set is derived from the schema (frozen on each KeySpec in src/config/schema.rs) โ€” a single enforcement point, not a hand-written deny list. packages.registry is deliberately remote-settable: a compromised channel could redirect the install source, and a first install from a redirected registry trusts its sha256.txt (spider.lock protects re-installs only); operators who reject that trade-off should restrict config.set in agent.role_operator.

Audit record. Every accepted or rejected operation is one JSON line in <state>/audit.log, e.g. {"ts":1762252800,"device":"laptop-1","role":"operator","op":"config.set","ok":true,"peer":"192.168.1.4:51000","detail":""}. The audit log rotates at 1 MiB (current file renamed to audit.log.1, a fresh audit.log starts). rotate_log rotates both spid3r.log and audit.log immediately (each renamed to <name>.1, replacing any previous .1) and returns both new paths in its response. A stolen devices.json is game over for that host: rotate by revoking the device and re-pairing, and keep the file off unencrypted backups.

Concern Windows Linux macOS
Config dir %APPDATA%\Spid3r $XDG_CONFIG_HOME/spid3r ~/.config/spid3r
State dir %LOCALAPPDATA%\Spid3r\state $XDG_STATE_HOME/spid3r ~/.local/state/spid3r
Reload config reload / agent SIGHUP + those SIGHUP + those
CSPRNG RtlGenRandom /dev/urandom /dev/urandom
Memory info GlobalMemoryStatusEx /proc/meminfo sysctl hw.memsize

No third-party crates were added for the management plane: the agent, INI parser, ChaCha20, HMAC, HKDF and Poly1305 (unused, kept with a vector) are all in-tree.

15. Virtual Machine Internals, GC & WebAssembly

Spid3r has three execution engines โ€” a tree-walking interpreter, a bytecode VM, and an LLVM ahead-of-time compiler โ€” built on one parser and one semantic checker. This chapter is about what the VM does with your program, how the cycle collector reclaims memory that reference counting cannot, and how the same VM runs inside a browser.

Three Engines, One Front End

Every engine consumes the same parsed and checked program, so anything that type-checks runs on all three. spid3r test is what keeps them honest: it runs every example on the tree interpreter, the VM, and the compiled .matrixc artifact, and fails if the output differs.

๐Ÿงฌ Self-hosting status

The compiler and core developer tools are written in Spid3r across 17 compiler/CLI modules (21 .matrix files) in selfhost/: lexer, parser, semantic checker & lints, lowering, bytecode compiler (.matrixc), LLVM-C codegen (with default<O3> PassBuilder optimizations), cross-OS linker/driver, and the test, fmt, and package tools (pack/unpack/verify/dist). Compiled via Stage 0 (spid3r build selfhost/main.matrix -o selfhost/main_native.exe), spid3r-self reaches a bit-identical Stage 1 → Stage 2 → Stage 3 object fixed point across the 134-program frozen corpus, verifies a SHA-256-pinned runtime staticlib (spid3r-self verify-runtime, RUNTIME_ABI.md), and records frozen Stage-0 vs selfhost throughput baselines in Benchmarks. See the self-hosting hub, ADR, and retirement review.

Self-Hosted Compiler Architecture & Bootstrap Boundary

Rebuilding selfhost/ or running compiler and dev-CLI commands with spid3r-self requires zero cargo or rustc calls. Rust Stage 0 (src/) stays in place permanently as the cold bootstrap seed, differential oracle (src/oracle.rs), pinned runtime staticlib (87 spid3r_* trampolines + 257 stdlib natives), and host for non-AOT engines/tools (tree, vm, repl, debug, lsp, agent, gui, wasm32, plugin).

Stage 0 Rust compiles Stage 1 spid3r-self, Stage 1 compiles Stage 2, and Stage 2 compiles Stage 3 with bit-identical spid3r_main.o fixed point across the 134-program frozen corpus.
Bootstrap chain (SH-1 ยท SH-5 ยท SH-8): Stage 2 and Stage 3 emit bit-identical spid3r_main.o objects and pass 124/124 runnable corpus parity (111/111 object match).
17-module compiler and dev-CLI architecture under selfhost/ driven by main.matrix across front end, bytecode, LLVM-C AOT, and test/fmt/pkg tools.
Internal architecture of selfhost/ (SH-3 ยท SH-7): one flat module root dispatching front-end parity, .matrixc emission, LLVM-C native AOT codegen, SHA-256-pinned linking, and test / fmt / pkg commands.
Ownership boundary between pure self-hosted compiler/CLI builds (zero cargo/rustc), the SHA-256-pinned runtime staticlib ABI (87 trampolines, 257 natives), and permanently retained Rust Stage 0 oracle, runtime, and non-AOT tooling.
Pure self-hosted compiler work vs permanently retained Rust Stage 0 (SH-2 ยท SH-6 ยท SH-8), coupled only through the SHA-256-verified spid3r_lang runtime staticlib ABI.
Generics, constraint bounds, AOT monomorphization, and unboxed list<int> / list<float> specialization in Spid3r.
Generics & unboxing (R10 ยท DP-1): multi-arg type parameters, constraint checking, AOT function specialization (spid3r_mono_*), and unboxed list<int> / list<float> storage.
Spid3r OCI container engine architecture: Registry v2 pull, SHA-256 verification, whiteout layer cache, snapshotter, and DP-7 Linux syscall shims.
OCI container engine (R11 ยท DP-7): Registry v2 pull + SHA-256 verification, .wh. whiteout layer pipeline, copy-up / overlayfs snapshotter, and --features container Linux syscall shims.
Engine Selected by What it does
Tree interpreter spid3r run (default) Walks the AST directly. The reference semantics: every other engine is expected to match it.
Bytecode VM spid3r run --engine vm, spid3r exec app.matrixc, spid3r gui --engine vm Compiles to bytecode, optimizes it, and runs a dispatch loop. This is the engine inside .matrixc artifacts and inside the WebAssembly build.
Native (AOT) spid3r build (Stage 0) / spid3r-self build (selfhost) LLVM 23 compiles to machine code (linked against SHA-256-pinned spid3r_lang). Two documented trades: the GUI bridge is not linked, and the execution budget is not enforced (a counter in generated code is not free).

What a Compiled Chunk Holds

Each function compiles to a Chunk: a Vec<u8> of encoded instructions, a constant pool, the prototypes of every nested function, and the class descriptors. Two parallel arrays map instruction indices to byte offsets and source spans, which is how a runtime error can point at the exact line that raised. Nested prototypes are shared through Arc, so creating a closure at run time is a refcount bump rather than a copy of the function's code.

The instruction set is 60 opcodes, from Constant and GetLocal up through the object-model and concurrency additions (SuperCtor, SuperMethod, CaptureSync, FieldGetSoft). Opcodes are byte-encoded, and OpCode::from_u8 rejects any value it does not know.

A .matrixc file is that chunk graph serialized. Loading it is defensive by construction: a magic check, a format-version check, a bounds-checked deserializer, and validate_chunk all run before the first instruction executes, so a corrupt or truncated artifact is a clean error and never a misread. The compatibility promise is one-directional: an artifact written by any 1.x build loads on every later 1.x build, while an older build reading a newer artifact must reject it cleanly rather than guess.

Frames, Calls and Unwinding

A CallFrame holds the function being run, its instruction pointer, its local slots, a display name used in stack traces, and โ€” deliberately โ€” its own globals map: name resolution uses the module that defined the function, not the caller's. Method frames carry the qualified Class::method form and constructors Class::new, so a trace reads the same on the VM as on the tree interpreter.

Call depth is capped at 1,000 frames. Exceeding it produces an ordinary runtime error (stack overflow: call depth limit reached) with a source span, not a host crash โ€” a runaway recursive function is a diagnosable mistake rather than a dead process.

try/catch/finally is not host unwinding. Each handler records the catch target, the finally target, the catch slot, and the operand-stack length and frame depth that were live when it was pushed. On a throw the VM restores both to exactly those values before jumping, so a handler sees a clean stack no matter how deep the failure happened.

The Optimizer

After compilation and before serialization, every reachable chunk is optimized: constant folding, peephole rewrites, and dead-code removal. The design rule is that no transformation may change observable behaviour, and the implementation enforces it three ways:

  • Folding uses the interpreter's own helper. Constant a; Constant b; <binop> is evaluated through the same vm_eval_binary the dispatch loop calls, so numeric promotion and string concatenation produce byte-identical results.
  • Failing operations are not folded. 1 / 0 and "a" - 1 stay in the bytecode, so the error still fires at run time with the same message and span.
  • Jumps win over folding. A fold is skipped when any jump or handler target points into the middle of the sequence being folded โ€” a target must keep landing on a real instruction.

A fourth pass rotates canonical while loop back-edges. A loop emitted as <cond>; JumpIfFalse end; <body>; Jump start; end: becomes <cond>; JumpIfFalse end; <body>; <cond>; JumpIfTrue body_start; end:. The condition is duplicated, never reordered, so evaluation count, order and stack effect are unchanged; steady-state iterations simply skip one unconditional jump. The pass lives only in src/vm/opt.rs, so --no-opt never sees it. It adds no opcode and does not change the .matrixc format, and it is skipped for chunks containing Try, for conditions with internal jumps or side effects, and when a jump from outside could land inside the loop. Because it removes work at compile time, --prof on the default path reports exactly one fewer executed instruction per loop iteration than --prof --no-opt; the runtime fused superinstructions remain disabled under instrumentation and, since Phase 5, under --no-opt as well (Vm::set_opt_enabled gates the VM peeks); per-VM inline caches are memoization and stay on. In both cases no opcode is hidden.

The Execution Budget

Embedders that run untrusted code can set a step allowance. It is a shared, saturating counter: the tree engine charges a step per statement and per loop iteration, the VM charges in batches at its instruction poll, and spawned tasks draw from the same allowance as the code that spawned them. When it runs out, the error is fatal โ€” it bypasses try/catch in both engines, so a script cannot swallow the stop and keep running. The native backend does not enforce it at all, which is why embedders running untrusted code use the tree or VM engine.

The budget bounds time; a separate policy bounds authority. An embedding host can deny whole capabilities, and each native the capability covers is replaced by a stub that raises a catchable error instead of reaching the OS:

let engine = Engine::new().deny(Capability::FileSystem);
let engine = Engine::new().deny_all();
# Capability::FileSystem | Process | Network | Environment
A script reaches the operating system only through built-in natives, which can be filtered opt-in via --sandbox on script engines or Engine::deny when embedding.
By default a program runs with user authority; opt-in capability gates filter built-in natives and matching plugin permissions via spid3r run --sandbox, spid3r exec --sandbox, or Engine::deny (not applied to Native AOT binaries). For OS isolation, run untrusted code as a separate user or in a container.

Garbage Collection: Cycles

Values are reference-counted. That handles the common case for free: the moment the last reference to an acyclic list drops, its memory is gone, with no collector involved. What reference counting cannot do is reclaim a cycle โ€” two values pointing at each other, or a value that contains itself โ€” because every member of the cycle keeps the next one's count above zero. Those cells would leak until the process exits.

Left: two owners share a value whose reference count is two; when both drop the count reaches zero and the value is freed immediately. Right: two values point at each other so each count stays at one and neither is freed; the cycle collector reclaims them.
Reference counting frees the acyclic case with no collector involved. Only the cycle on the right needs the mark-sweep pass.

The collector is a mark-sweep that deliberately never enumerates roots. Enumerating every place a live value can hide โ€” interpreter environments, VM frame slots, globals, upvalue cells, task captures โ€” is exactly where a collector gets it wrong, and one missed root means a use-after-free. Instead it computes, for each tracked cell, how many references come from outside the tracked set:

external(cell) = strong_count(cell) - internal_edges_into(cell)
# saturating at zero; internal_edges is counted by tracing every tracked cell

Anything holding a strong reference from outside the tracked set โ€” an environment binding, a Rust local mid-expression, a global, a captured upvalue โ€” shows up as extra strong_count, so that cell has external > 0 and is a root. The collector marks everything reachable from a root, then sweeps what is left: those cells are reachable only from each other, which is the definition of a closed cycle.

โš ๏ธ The safety asymmetry that makes this correct

If tracing misses an edge, internal_edges comes out too small, external too large, and the cell is treated as a root and kept โ€” a leak, never a crash. If tracing ever over-counted an edge, a live cell could compute external = 0 and be swept while still in use. So the rule is absolute: count each strong child exactly once, never count weak references, and leave a container untraced if you are unsure of its children. Being conservative costs memory; being optimistic costs correctness.

Reading strong_count is only sound while nothing else is mutating it, so collection runs only when no other thread is executing Spid3r code. Tasks parked in a blocking wait (recv, join, mutex.lock, sleep) do not count as running โ€” their values sit on their own stacks and are therefore roots. If you call gc_collect() while another thread is running, it fails with a clean message naming how many threads are in the way, rather than collecting under a race.

Detail Behaviour
Automatic collection On by default. Checked at statement boundaries and at the VM's dispatch point, but triggered by allocation count: the first pass runs after 1024 tracked allocations, then at the number of cells that survived the previous pass. SPID3R_GC=0 turns it off.
Manual control gc_collect() runs a collection; gc_live_cells() reports the tracked cell count and doubles as a leak probe.
Coverage 27 GcTrace implementations: every container type, class and function objects, class instances on both engines, and the environment itself. Containers register through the single allocation choke point value::new_ref, so a new one is tracked by construction.
Cost The next threshold is the surviving cell count (never below 1024), and the registry-prune watermark starts at 8192 and doubles. A linear build therefore costs O(log N) collections instead of O(N) at a fixed threshold; a test pins the linear behaviour.
On the native backend The AOT runtime allocates via a thread-local bump-slab arena (ARENA_SLAB_LEN = 4096) with call-scoped reclamation (spid3r_arena_enter / spid3r_arena_leave and spid3r_return_leave evacuating return values cleanly); gc_collect(), gc_live_cells(), and spid3rinfo()["AOT Memory"] report engine-internal arena and slab counters.

WebAssembly: The Same VM in a Browser

The browser build is not a separate implementation. web/spid3r.wasm is this VM compiled to wasm32-unknown-unknown with no dependencies, exposed through a small C ABI over linear memory: the JavaScript side allocates a buffer, copies source in, and reads a packed result back out.

WASM Export JavaScript SDK Method Description
spid3r_wasm_eval(src, eng) runtime.eval(source, engine="vm") Executes Spid3r code in the browser and captures stdout, errors, thrown values, and line numbers.
spid3r_wasm_check(src) runtime.check(source) Lexes, parses, and semantic-checks source code without running it.
spid3r_wasm_format(src) runtime.format(source) Canonically formats code according to the official Spid3r style guide.
spid3r_wasm_version() runtime.version() Retrieves the toolchain and language version strings.
spid3r_wasm_alloc(len) / spid3r_wasm_free(ptr, len) used internally by the SDK Hands the caller a buffer in linear memory to write source into, and releases it afterwards.
โœ… Engine divergence fixed: not (A or B)

Earlier 1.0 builds miscompiled not (A or B) on the VM when A was falsy. Two holes were closed: (1) short-circuit vs right-evaluated, pinned as not_over_or; (2) the right operand evaluating to true, which the [Not][JumpIfFalse] peephole still got wrong until the Not was no longer deleted when it was an or merge target. Both are pinned in tests/engine_matrix.rs (not_over_or, not_over_or_b_true). Tree, VM, exec and WASM now agree. The let is_item = ...; if not is_item workaround is still correct โ€” just no longer necessary.

Systems FFI, POD Structs & Freestanding AOT

The LLVM AOT backend (src/native/ir.rs) supports systems-level C interop and bare-metal code generation:

  • Static & Shared Linking: Pass --link-lib <name|path> to spid3r build (repeatable) so extern fn declarations resolve against system libraries (-l<name> on Unix, <name>.lib under MSVC).
  • Variadic FFI, Callbacks & POD Structs: extern fn supports C variadics, ffi_callback(fn, "fn(...)->...") generates C-ABI trampolines for passing Spid3r functions to C APIs, and POD structs marshal by value across the FFI boundary.
  • Runtime dlopen Gate & Freestanding Mode: In BuildOptions, dynamic loading primitives (dl_open, dl_sym, dl_close, dl_call) are gated off by default unless allow_dlopen is enabled, and freestanding emits host-independent IR without linking the standard runtime.

OCI Container Engine (spid3r container)

Built behind the container Cargo feature (--features container), Spid3r includes an OCI Image & Container engine (src/oci/ and src/container/). Status: R11.1โ€“R11.9 implemented across src/oci/, src/container/, src/native/sys.rs (DP-7) and spid3r container; Linux kernel namespace/overlayfs/cgroup execution is gated behind #[cfg(all(target_os = "linux", feature = "container"))].

  • OCI Image Pull & Verification: src/oci/ implements OCI Distribution pulls, SHA-256 manifest and layer digest verification, optional Ed25519 signature enforcement (--require-sig --pubkey <hex>), and OCI .wh. whiteout layer extraction.
  • Linux Isolation: src/native/sys.rs (Linux-only, gated behind #[cfg(all(target_os = "linux", feature = "container"))]) provides namespace isolation, pivot_root, cgroups v2 resource limits, and veth/NAT networking for spid3r container <pull|create|start|run|stop|kill|rm|ps|inspect|logs|gc>.
Container deep dive โ€” quickstart commands, module architecture & security model

Quickstart

# Pull an OCI image (verifies manifest, config, and every layer SHA-256) spid3r container pull http://127.0.0.1:5000/app/demo:v1 # Create & inspect a container bundle spid3r container create --name web http://127.0.0.1:5000/app/demo:v1 spid3r container ps spid3r container inspect web # Run on Linux (built with --features container) spid3r container run --name web http://127.0.0.1:5000/app/demo:v1 /bin/echo ok # Stop, remove, and garbage-collect unreferenced layers spid3r container stop web spid3r container rm web spid3r container gc

Architecture

  • src/oci/ โ€” Cross-platform OCI Image Spec v1 parser (manifest.rs), Registry v2 HTTP(S) client with anonymous Bearer token flow and multi-arch host selection (client.rs), and hardened gzip/ustar layer pipeline with .wh. whiteouts, path-traversal refusal, and atomic cache commits (layers.rs).
  • src/container/ โ€” Snapshotter (snapshot.rs: Linux overlayfs + copy-up fallback + refcounted GC), runtime state machine & OCI config.json bundle loader (runtime.rs), spid3r0 bridge IPAM pool (net.rs), and daemon state recovery (daemon.rs).
  • src/native/sys.rs (DP-7) โ€” Sanctioned unsafe home for Linux syscall shims (unshare, setns, mount, umount2, pivot_root, chroot, sethostname, prctl(PR_SET_NO_NEW_PRIVS), fork, waitpid, kill, setuid/setgid). Zero external crates; gated out of --no-default-features and wasm32 builds.

Security Model & Honest Limitations

  • Digest & signature verification: Every config and layer blob is SHA-256 verified before caching or extraction; mismatches fail closed with exit code 4. Optional Ed25519 manifest signature enforcement (--require-sig --pubkey <hex>) reuses crate::pkg::ed25519.
  • Tar hardening: Absolute paths, drive prefixes, and .. components are refused before writing any file. Uncompressed layer buffering is capped at 64 MiB (MAX_LAYER_TAR_BYTES).
  • Platform scope: Image pull, verification, whiteout assembly, IPAM, OCI bundle creation, and CLI management (pull/create/ps/inspect/logs/rm/gc) run on Windows, macOS, and Linux. Kernel namespace/cgroup/overlayfs execution (spid3r container start/run) requires Linux built with --features container.

Bare-Metal Kernel Track (spid3r kernel build)

The kernel/ workspace member is a zero-dependency no_std x86_64-unknown-none kernel foundation (R13 / DP-8). Status: DP-8 chartered and the R13.1 bare-metal kernel/ foundation + host-testable subsystems implemented; QEMU 10m/30m soak tests (R13.2/R13.5), live ring-3 sysret execution (R13.6), virtio-tap packet roundtrip (R13.7), and external unsafe review (R13.8) remain open until run on Linux QEMU CI.

Kernel deep dive โ€” boot & core subsystems, build & verify commands

Boot & Core Subsystems

  • Boot: The 24-byte Multiboot2 header (0xE85250D6) is embedded in .multiboot2; COM1 (0x3F8) serial UART is initialized, Spid3rOS v0.1 (x86_64 multiboot2) ready is emitted, and the CPU idles via hlt.
  • Memory: An audited 4 KiB physical bitmap frame allocator (double-free refusal, exact frame accounting) plus a checked-arithmetic bump heap (KernelBumpHeap, clean error on exhaustion without integer wrap).
  • Interrupts & Console: 64-bit GDT/IDT descriptors, exception vector table (#DE, #UD, #DF, #GP, #PF), PS/2 scancode set 1 decoder, and help/ps/meminfo shell commands.
  • Userland & Syscall ABI: ELF64 x86-64 validator (0x7F ELF, ELFCLASS64, EM_X86_64) with documented syscall numbers (SYS_WRITE = 1, SYS_GETPID = 39, SYS_EXIT = 60); kernel/init/init.matrix targets the R9.4 freestanding Spid3r build profile (--freestanding --entry _start --target x86_64-unknown-none --emit-obj).
  • Drivers: PCI Type-0 configuration address encoder (0xCF8) and modern virtio device classification (virtio-net = 0x1041, virtio-blk = 0x1042).
  • Toolchain Integration: Run spid3r kernel build [--out <path>] to build kernel/Cargo.toml for x86_64-unknown-none --release (requires rustup target add x86_64-unknown-none). The 24-byte Multiboot2 header and kernel/link.ld link at 1 MiB physical with -C disable-redzone=yes and -C relocation-model=static; zero external crate dependencies.

Build & Verify

# Host unit tests for all kernel subsystems (Windows / Linux / macOS) cargo test --manifest-path kernel/Cargo.toml # Bare-metal x86_64-unknown-none kernel build + SHA-256 sidecar rustup target add x86_64-unknown-none spid3r kernel build
Bare-metal x86_64 kernel architecture: Multiboot2 boot, GDT/IDT, paging, allocators, serial/VGA console, scheduler, syscalls, and initrd.
Bare-metal kernel architecture (kernel/, x86_64-unknown-none, zero external crate dependencies).