All projects

Voice & agent control · Explicit actions · Trace & replay

PIM: Voice & Agent Control

Control applications through voice, AI proposals, and scripts using one shared command model. PIM resolves incoming words to declared application callbacks and records how each command was handled. The model supports context, arguments, and command completion, with behavior you can inspect and replay.

The wider worker interface in use: changing views and adjusting reading position and speed. The standalone repository contains PIM's interaction runtime and command model.

From a proposal to an explicit action Link to heading

A command has to resolve through the declared interaction model to reach a callback. The bundled proof makes this concrete with fixed proposals and recording adapters:

Proposals meet the declared command model Verified with fixed inputs and recording callbacks

  • Send a release prompt to Ada

    Select Ada's worker and ask it to tighten the release notes.

    Selects worker-ada and sends “tighten release notes” in its release context.

  • Approve for this session

    Answer the pending command approval with a session scope.

    Records acceptForSession for approval-tests, with scope session.

  • Delete production

    Propose a command absent from this interaction model.

    Rejects the undeclared path and records no target action.

Replaying this batch reproduced the same decisions, actions, and trace summary.

The application still owns its effects and permission checks. PIM makes command resolution and dispatch explicit, so a reviewer can inspect the route from input to callback.

One interaction model across input channels Link to heading

Commands live in the interaction model; input and effects connect through adapters. A speech recognizer, model, or script supplies words. PIM resolves them in the current interaction context, and application callbacks implement the resulting actions.

Language enters; application callbacks act

  1. Incoming wordsSpeech recognizer, model proposal, or script
  2. Declared model + runtimeResolve a path in the current interaction context
  3. Application callbackInvoked through a resolved, declared action

Processing events expose matches, continuations, and rejections for inspection and replay.

The host owns input adapters and application effects. PIM owns interaction resolution and its observable execution.

Nested interaction spaces keep commands in context. The runtime can return unused words to a parent interaction, find a valid command in an enclosing context, and expose the options available next. A $stop marker distinguishes a completed phrase from a longer command with the same prefix.

This lets an application reuse command semantics across entry points. The host supplies speech recognition, model inference, UI integration, and effectful services.

Declare the action, observe the execution Link to heading

A minimal command model with a recording callback; imports are omitted:

const actions: string[] = [];
const trace: ProcessingEvent[] = [];

const interpreter = runtime({
  deploy: {
    preview: {
      onUtter: () => actions.push("deploy-preview"),
    },
  },
}, {
  onProcessingEvent: event => trace.push(event),
});

interpreter.next();
interpreter.next(["deploy", "preview"]);
// actions: ["deploy-preview"]

The declaration connects a resolved path to an ordinary application callback. Observers receive processing events separately from the callback, including matched options, path changes, queued actions, continuations, and rejections.

Test behavior without calling a model Link to heading

Fix the inputs and inspect the exact actions. Tests can supply words, runtime state, and recording callbacks directly, then check the resolved path, arguments, and processing trace. A failure can be investigated from the intermediate events as well as its final effect.

With the same interaction model, state, normalized words, and callback results, PIM reproduces its resolution and event sequence. The bundled proof evaluates the complete proposal batch twice and compares decisions, actions, and trace summaries byte for byte. Inference and external services remain outside that deterministic boundary.

The public repository is a standalone, tested runtime preview. Explore the source and executable proof, the architecture, or the original demo post.