JavaScript API

The objects behind the components, for your own UI or game loop. Names mirror the Unity SDK (SmartAgent is NeocortexSmartAgent, and so on).

Entry points

Every class takes endpoint (default /api/neocortex) or a client, which wins.

Example: voice in, voice out

import { SmartAgent, AudioReceiver } from "@neocortex-link/web/client";

const agent = new SmartAgent({ endpoint: "/api/neocortex", characterId: "your-character-id" });

agent.on("lineStarted", ({ line }) => subtitle.show(line.text, line.emotion));
agent.on("emotionChanged", ({ emotion }) => avatar.setExpression(emotion));
agent.on("composing", (thinking) => spinner.toggle(thinking));
agent.registerAction("FOLLOW_PLAYER", () => npc.follow(player));

// Typed input, spoken reply
sendButton.onclick = () => {
  agent.primeAudio(); // browsers only allow audio after a click
  agent.textToAudio(input.value);
};

// Spoken input, spoken reply
const mic = new AudioReceiver({ maxWaitTime: 1.2 });
mic.on("recorded", (wav) => agent.audioToAudio(wav));

talkButton.onclick = async () => {
  agent.primeAudio();
  await mic.start(); // asks for microphone permission, then records when the player speaks
};

Speech streams line by line, prefetching the next line. A line that fails to synthesize is shown as text. Input during a turn is queued. Call primeAudio() from a user gesture so later speech can play.

Events

Every class except EventLogger and Interactables is an emitter:

const off = agent.on("lineStarted", (detail) => { /* ... */ }); // returns a function that removes the handler
agent.once("replyFinished", (reply) => { /* ... */ });
agent.off("lineStarted", handler);

A throwing handler is logged and doesn't stop the others.

SmartAgent

One character in conversation with the player.

const agent = new SmartAgent({ endpoint: "/api/neocortex", characterId: "your-character-id", voice: true });
await agent.send("Where is the blacksmith?");

Conversation options

SmartAgent and GroupDirector share these options:

LocalSessionStore uses the Unity SDK's keys (neocortex-session-id_<characterId>) and falls back to memory where localStorage is unavailable. An unknown session is cleared and the next turn starts fresh.

Methods

The send methods resolve with the character's message { characterId, name, lines, actions } once it has been played, or null if the turn failed. Failures are reported through the error event, not thrown.

Properties

characterId, sessionId, isBusy, isSpeaking, client, player, and the settable voice, paceLines, actionTrigger and language.

Events

Emotions are camelCase: neutral, happy, pleased, disappointed, upset, amazed, curious, confused, alarmed, fascinated, impressed, annoyed, angry, confident, reassured, concerned, scared. Anything else is read as neutral.

GroupDirector

Several characters in one conversation, where the AI picks who answers. See Group Chat for a walkthrough.

import { GroupDirector } from "@neocortex-link/web/client";

const group = new GroupDirector({ endpoint: "/api/neocortex", characterIds: ["aria", "bram"], voice: true });
group.on("speaker", ({ name }) => highlight(name));
await group.send("What happened at the mill?");

Takes the conversation options, plus characterIds and an optional maxTurns (how many characters may speak in one turn). The proxy allows up to 6 characters a turn by default.

The send methods resolve with the whole reply { sessionId, messages }, or null. members lists the current cast. Everything else (events, registerAction, history, abort and so on) works as on SmartAgent.

QuizAgent

A quiz host that asks questions, judges answers and keeps score (Team plan). See Quiz for a walkthrough.

import { QuizAgent } from "@neocortex-link/web/client";

const quiz = new QuizAgent({ endpoint: "/api/neocortex", characterId: "host-id", questionSetId: "set-id", voice: true });
quiz.on("hostLine", ({ line }) => subtitle.show(line.text));
quiz.on("questionChanged", (question) => renderOptions(question?.options ?? []));
quiz.on("finished", () => showResults(quiz.leaderboard));

startButton.onclick = () => {
  quiz.primeAudio();
  quiz.begin();
};

The state of the run is kept on the agent: sessionId, question, progress ({ asked, total }), expecting ("answer", "confirmation" or "nothing"), leaderboard, lastResult, lastIntent, attempt, attemptsLeft, hintGiven, pending, answered, isFinished and isBusy.

AudioReceiver

Records the player's voice with automatic speech detection or push-to-talk, on an AudioWorklet. In detection mode it listens continuously, starts recording when loudness crosses the threshold, and finishes after maxWaitTime seconds of silence. Recordings come out as 16 kHz mono WAV, ready for audioToAudio.

import { AudioReceiver } from "@neocortex-link/web/client";

const mic = new AudioReceiver({ pushToTalk: true });
mic.on("recorded", (wav) => agent.audioToText(wav));

talkButton.onpointerdown = () => mic.press();
talkButton.onpointerup = () => mic.release();

StreamingAudioPlayer

Plays a character's voice while it is still being synthesized, through Web Audio. Every agent has one as its player, so you can read the speech level for lip sync:

function animate() {
  avatar.mouthOpen = agent.player.getLevel(); // 0 to 1
  requestAnimationFrame(animate);
}
animate();

Pass one player to several agents with the player option to share one volume. A player plays one thing at a time, so a new line stops whatever it was playing.

UsageGate

Checks credits before offering a conversation, so you can fall back to scripted lines. Fails open.

import { UsageGate } from "@neocortex-link/web/client";

const gate = new UsageGate({ endpoint: "/api/neocortex", characterId: "your-character-id" });
await gate.refresh();

if (gate.canUse()) await agent.send(text);
else showScriptedLine();

Events: updated, lowCredits, creditsEmpty, playerOverLimit, characterOverLimit, each with the usage, and error when the check itself failed.

Behind the proxy, usage is checked for the signed-in user. Caps are set on the API Keys page.

EventLogger

Collects game events for characters to know about, such as "the player opened the vault". Each entry is cut to 64 characters, and a turn sends at most 20, highest priority first, then newest.

import { EventLogger, SmartAgent } from "@neocortex-link/web/client";

const events = new EventLogger();
const agent = new SmartAgent({ endpoint: "/api/neocortex", characterId: "guard", eventLogger: events });

events.push("The player picked the vault lock", 2);

Interactables

The things in a scene a character can perceive and act on. Actions come back with the targetId of the entity they apply to. See Interactables for a walkthrough.

import { Interactables, SmartAgent } from "@neocortex-link/web/client";

const scene = new Interactables()
  .set({ id: "door-1", type: "OBJECT", name: "Cellar door", properties: [{ name: "locked", value: "true" }] })
  .set({ id: "lamp-1", type: "OBJECT", name: "Oil lamp" })
  .focus("door-1");

const agent = new SmartAgent({ endpoint: "/api/neocortex", characterId: "guard", interactables: scene });
agent.registerAction("UNLOCK", (action) => unlock(action.targetId));

set, update, remove and focus return the same object, so calls can be chained.

The proxy drops browser metadata and events unless allowClientContext: true.

NeocortexClient

A typed wrapper over every v3 endpoint. Create one to add headers, change timeouts, or call endpoints directly.

import { NeocortexClient } from "@neocortex-link/web";
import { SmartAgent } from "@neocortex-link/web/client";

const client = new NeocortexClient({
  baseUrl: "/api/neocortex",
  headers: () => ({ authorization: `Bearer ${getToken()}` }),
});
const agent = new SmartAgent({ client, characterId: "your-character-id" });

Every method takes an optional second argument { signal, timeoutMs }. quizTurn retries the server's "still busy" refusals the same way the Unity SDK does.

Server-to-server

On a backend that calls Neocortex directly, give the client your key. baseUrl then defaults to the Neocortex API:

import { NeocortexClient } from "@neocortex-link/web/server";

const neocortex = new NeocortexClient({ apiKey: process.env.NEOCORTEX_API_KEY });
const { sessionId, messages } = await neocortex.chat({
  characterIds: ["your-character-id"],
  message: "Hi",
  playerId: "user-1",
});

Errors

Every failure is a NeocortexError. Agents emit it on error; NeocortexClient throws it.

Codes raised in the browser are timeout, network_error, aborted, client_error and browser_api_key. Codes from the proxy are listed in Proxy Options.

Use isNeocortexError(error) rather than instanceof, so errors from a second copy of the package (CJS next to ESM) match too.

import { isNeocortexError } from "@neocortex-link/web";

try {
  await client.chat({ characterIds: ["your-character-id"], message: "Hi" });
} catch (error) {
  if (isNeocortexError(error) && error.code === "rate_limited") {
    await wait((error.retryAfter ?? 5) * 1000);
  }
}