--- { "n": 12, "title": "Plugin system design", "abstract": "Install a plugin from a GitHub URL: manifest, install lifecycle, bus runtime, supervisor-owned services, viewer pages, and the workflows plugin", "refs": [], "seen": [ "agent", "user" ], "data": {}, "created": 1789853423.7873821, "updated": 1789866325.242545, "deleted": 0.0, "completed": 0.0, "outcome": "", "type": "doc" } --- Four Opus architects designed this on 2026-09-19, each owning one part: format and install, the bus runtime, supervisor-run services, and the workflows integration. This doc reconciles their four designs into one. Where they disagreed, the decision is stated below with its reason. The build order is plan 3. ## What a plugin is A plugin is a git repo with `.journal-plugin/plugin.json`. You paste its URL: a GitHub URL, `owner/repo`, a `#ref`, or a local path. The journal then: 1. shows exactly what the plugin will run; 2. clones it at a pinned commit into `.journal/plugins//`; 3. runs its setup steps; 4. feeds it bus events; 5. starts its services under the session's supervisor; 6. lists its pages in the viewer's sidebar. A plugin can: - answer events with a small set of instructions (tell the agent something, notify, open a to-do, hold writes, refuse a tool use); - call the journal's HTTP API itself; - run its own servers. There is no sandbox: a plugin runs as you. Install says so, and install happens only after you've seen every command it will run. ## The manifest: `.journal-plugin/plugin.json` The smallest valid manifest: ```json {"name": "hello", "on": {"todo.created": "python3 hello.py"}} ``` The workflows plugin in full: ```json { "name": "workflows", "title": "Workflows", "version": "0.9.0", "journal": "2.14.0", "description": "Visual workflow engine that runs pipelines from journal events", "requires": { "php": {"check": "php -r 'exit(version_compare(PHP_VERSION,\"8.5\",\">=\")?0:1);'", "hint": "PHP 8.5+ with pdo_sqlite, dom, posix, pcntl (brew install php)"}, "composer": {"check": "composer --version", "hint": "Composer 2 (brew install composer)"} }, "env": { "JOURNAL_PLUGIN_DATA": "{data}", "DB_CONNECTION": "sqlite", "DB_DATABASE": "{data}/workflows.sqlite", "APP_URL": "http://127.0.0.1:{ports.web}", "QUEUE_CONNECTION": "database", "WORKFLOWS_PORT_OFFSET": "0", "WORKFLOWS_BROADCAST_PORT": "{ports.realtime}" }, "setup": [ {"name": "storage", "run": "bash .journal-plugin/bin/prepare.sh"}, {"name": "composer", "cwd": ".journal-plugin/host", "run": "composer install --no-interaction --no-dev --prefer-dist"}, {"name": "key", "cwd": ".journal-plugin/host", "run": "php artisan journal:key"}, {"name": "migrate", "cwd": ".journal-plugin/host", "run": "php artisan migrate --force"}, {"name": "install", "cwd": ".journal-plugin/host", "run": "php artisan workflows:install --no-migrate --mcp=none --oauth=no --no-interaction"}, {"name": "assets", "cwd": ".journal-plugin/host", "run": "php artisan workflows:assets"}, {"name": "integration", "cwd": ".journal-plugin/host", "run": "php artisan journal:integration"} ], "services": { "web": {"cwd": ".journal-plugin/host", "run": "php artisan serve --host=127.0.0.1 --port={port} --no-reload", "port": "auto", "env": {"PHP_CLI_SERVER_WORKERS": "6"}, "ready": {"path": "/up"}, "restart": "on-failure"}, "queue": {"cwd": ".journal-plugin/host", "run": "php artisan queue:work --queue=workflows,workflows-agent,default --sleep=1", "restart": "always"}, "realtime": {"cwd": ".journal-plugin/host", "run": "php artisan workflows:broadcast --port={port}", "port": "auto", "restart": "always"}, "schedule": {"cwd": ".journal-plugin/host", "run": "php artisan schedule:work", "restart": "always"} }, "on": { "todo.created": {"post": "http://127.0.0.1:{ports.web}/journal/events"}, "todo.completed": {"post": "http://127.0.0.1:{ports.web}/journal/events"}, "message.created": {"post": "http://127.0.0.1:{ports.web}/journal/events"} }, "pages": [ {"name": "workflows", "title": "Workflows", "icon": "workflow", "service": "web", "path": "/workflows"}, {"name": "engine", "title": "Engine", "icon": "activity", "service": "web", "path": "/workflows", "status": "/workflows/api/engine/health"} ] } ``` **Keys:** - `name` is the only required key. It is 2–32 lowercase letters, digits or dashes, and must not be the name of a built-in feature. - `version`, `title`, `description`, and `journal` (the minimum journal version). - `requires`: checks that run before anything else. Install stops at the first one that fails and prints its `hint`. - `env`: variables given to every setup step, handler and service. - `setup`: ordered, idempotent steps. Each is `{name, run, cwd?}`, where `run` is a string (run through `sh -c`) or an argv list. - `services`: long-running processes (see Services). - `on`: event pattern → handler (see Runtime). - `refuse`: an optional command that may refuse a tool use. - `pages`: sidebar entries that show a service's path. - `settings`: values set on the Plugins panel. An `env` setting only names one of your own environment variables and is never stored. **Placeholders** in any string: - `{dir}`: the plugin's folder. - `{data}`: `.journal/plugin-data//`. It survives reinstalls and upgrades, and holds `.env`, the sqlite database and storage. - `{port}`: this service's own port. - `{ports.}`: another service's port. - `{root}`: the `.journal` folder. - `{journal.url}`: the viewer's URL. - `{token}`: the plugin's API token. **Validation is strict.** Unknown keys, a pattern no event can match (checked against the known types and actions), a page naming an undeclared service, or a bad name are each refused with one plain sentence. Example: `plugin.json: unknown key 'listens'; known: name, version, …`. ## Install, upgrade, remove **The record.** Installed plugins are a project-scoped resource type `plugin`, like tools and connections. Each row holds: - `source`, `ref`, `commit`, `version` - `linked` (installed from a local path) - `enabled` - `manifest` (the validated snapshot) - `settings` - `token` Because a plugin is a row, its lifecycle emits `plugin.*` events. The CLI and the viewer's generic routes cover it for free. The lifecycle logic lives in one fixed feature, `features/plugins/`, as `@command("plugin")` methods. **One clone funnel.** `install.fetch(repository, ref) -> (folder, commit, error)` does `git init`, then `git fetch --depth 1 origin `, then `checkout FETCH_HEAD`. That works for a branch, a tag or a bare SHA. `journal upgrade` moves onto the same funnel. A private GitHub repo uses your git credentials. **Install**, in order: 1. Fetch into `.journal/plugins/.staging-`. 2. Validate the manifest, and refuse a name clash. 3. Run `requires`. 4. Show the preview: source, exact SHA, every command verbatim, pages, settings, and the environment variables the plugin reads. 5. Stop and wait. Without `--yes` nothing is installed; it refuses with "run again with `--yes --ref `". So what you approved is exactly what installs, even if the branch moved in the meantime. 6. With `--yes`: run the setup steps in order. On the first failure, show the step name, its command, its last 40 lines and the log path, delete staging, and install nothing. 7. Rename staging to `plugins/`, create the row, and notify you. **A local path is linked, not copied.** The folder becomes a symlink, and `upgrade` only re-reads the manifest. This is how the workflows plugin is developed in place, without copying `vendor/`. **Upgrade:** 1. Fetch to staging. 2. Show which commands changed; confirmation is bound to the SHA again. 3. Run setup in staging. 4. Stop the plugin's services. 5. Swap the folders. 6. Start the services again. **Enable and disable** are project-wide. **Remove** completes the row. A listener stops the services and deletes the folder, or unlinks it for a linked install. `{data}` stays until you run `journal plugin remove --purge`. **Commands:** - `journal plugin preview [--ref]` - `journal plugin install [--ref] [--yes]` - `journal plugin all` - `journal plugin show ` - `journal plugin upgrade [--ref] [--yes]` - `journal plugin enable ` and `journal plugin disable ` - `journal plugin remove [--purge]` In the viewer, a **Plugins** panel in Settings has: - a URL box with a Preview button; - a confirmation that lists everything the plugin will run, then an Install button that sends the SHA; - one row per plugin with its version and commit, an on/off switch, Upgrade, Remove, and its settings. ## Runtime: how a plugin hears the bus and answers Events are written by several processes: the CLI, hooks, the engine and the server. The in-process bus only hears its own, so the one funnel is the event log itself. **Delivery:** - **The host.** It runs inside the viewer server, started from `serve()` the way the updates announcement is. It tails every environment's `events.jsonl` with a cursor per plugin. - **Waking up.** It wakes at once on the server's own events and otherwise polls every 0.5 s. - **One host per project.** An `flock` on `runtime/plugins.lock` ensures only one server delivers. - **What gets delivered.** On first start nothing old is replayed. After downtime, only events younger than 10 minutes are. - **Hook events.** They go on the log as facts first: `hooks.handle` puts `{hook, tool, file, session}` on the agent's event. Plugins see them as `hook.PreToolUse`, `hook.PostToolUse` and so on. **Handlers.** A handler is either: - **A command.** The payload goes in as JSON on stdin, and at most one JSON reply comes out on stdout. This is at-most-once delivery, which suits a script. - **`{"post": url}`.** The payload is POSTed to one of the plugin's own services, and the reply is the response body. The cursor only advances on a 2xx, so delivery is at-least-once: an event is delivered late, never lost, while the service restarts. This is what workflows uses, since its server is already running. Handlers get `JOURNAL_ROOT`, `JOURNAL_ENV`, `JOURNAL_URL`, `JOURNAL` (the command's path), `JOURNAL_PLUGIN`, `JOURNAL_TOKEN` and the manifest's `env`. Commands run with the plugin folder as their working directory. Plugins should always call `$JOURNAL`, never a bare `journal`. **The payload (v1):** ```json {"v": 1, "event": "todo.created", "id": 4812, "at": 1790000000.1, "type": "todo", "n": 12, "action": "created", "actor": "agent", "data": {}, "env": "main", "project": "/path", "resource": {"type": "todo", "n": 12, "title": "…", "brief": "…", "ref": "todo:12"}, "agent": {"session": "…", "status": "working", "model": "…"}, "plugin": {"name": "workflows", "dir": "…"}} ``` Every record event and the nine hook events are exposed. The agent's private nudges are not. **Answers.** Every key is optional, and `{}` or no output means nothing to do. The runtime applies them through one funnel, as the `plugin` actor, stamping `plugin=` on each row, with at most 20 per answer: | key | effect | |---|---| | `whisper` | a private note for the agent, handed over on its next tool use | | `say` | typed to the agent when it is idle | | `notify` | a notification | | `notice` | a band for the user in the viewer | | `todo` | a new to-do | | `hold` | holds the agent's writes with a reason; `""` releases it | | `refuse` | only from the `refuse` command: refuses this tool use | For events that don't come from an agent, `whisper` and `say` go to the environment's primary agent. **Loop guard.** Events on rows a plugin created are not sent back to that same plugin. **Refusing synchronously.** This is possible only on PreToolUse, and only for writes unless the manifest sets `reads: true`. One `@refuses` method runs each plugin's `refuse` command within a budget: 1.5 s each (3 s at most) and 5 s in total. It fails open: a timeout, crash or bad output means "no refusal". Anything slower should answer `hold` from an async handler instead. **Isolation:** - Each plugin gets its own worker thread and a bounded queue: events stay in order, and plugins never block one another or a hook. - Each call runs in its own process group with a timeout: 10 s by default, 60 s at most. The whole group is killed if it overruns. - Replies are capped at 64 KB. - Failures go to `runtime/plugins/.log`. - After five failures in a row, you (never the agent) get one notice ("Plugin workflows is failing: …"), and the plugin backs off up to 5 minutes. The next success clears the notice. **Calling the journal's API.** Plugins that call the API send `X-Journal-Token`. Their writes are then attributed to the `plugin` actor, which the loop guard and the viewer can tell apart. ## Services: owned by the session, dying with it **Who owns them.** `journal claude` runs a launcher that lives for the whole session in your terminal. The supervisor under it is replaced whenever the code changes, so the services belong to the launcher, not to a supervisor that would restart them on every edit. The mechanism is a lifeline pipe: - The launcher opens the pipe and keeps the write end until it dies. - Every supervisor and every service keeper inherits the read end. - The kernel closes the write end on any death: a clean exit, a closed terminal, even SIGKILL. The lifeline therefore reads end-of-file at once, on macOS and Linux alike. **The keeper** (`engine/keeper.py`) is one small process per service. It: - takes a lease: an `flock` on `runtime/service-.lock`. If another keeper holds it, this keeper exits; - starts the service as the leader of its own process group, so `php artisan serve` and the `php -S` it spawns are one group; - watches the lifeline and probes readiness: HTTP below 500 on `ready.path`, otherwise a TCP connect; - tears down on end-of-file or SIGTERM: `killpg` TERM, a grace period, then KILL, with deadline loops. **Orphan sweep.** Each status file records the keeper, the process group and the owner. On every tick, the manager kills any group whose keeper or owner is dead, to cover a SIGKILLed keeper. Services must stay in the foreground and must not daemonize themselves; the skill says so. **The manager** (`engine/services.py`) ticks every second in the supervisor. It: - reconciles what should run against what does; - restarts with backoff: 1, 2, 4 … 30 s, and five crashes in 60 s mark a service `failed` until it is restarted by hand; - adopts running keepers after a supervisor reload, so there are no duplicates. Services are project-wide, because a port belongs to the machine. If the owning session ends while another session is live, the survivor starts them again after a few seconds. With no session running, services are down. `journal services up` is a foreground owner without an agent, and Ctrl-C ends everything. **Ports.** `auto` reuses the last port if it is free, and otherwise takes the first free one in 8440–8499, beside the viewer's 8420–8439. A fixed port that is taken shows as `blocked`. **Files, all flat in `runtime/`:** - `service-..log`: combined output, trimmed by housekeeping like the other logs. - `service-.json`: status, written only by the keeper and atomically. It holds the state (`starting|ready|exited|stopped|failed|blocked`), port, url and pids. - `service-.want`: `up` or `down` plus a nonce. It is written only through `services.want()`, which the CLI and HTTP share; a new nonce restarts the service. **Control:** - CLI: `journal services [list|up|start|stop|restart|log ]` - HTTP: `GET /api/services`, `GET /api/services/{id}/log`, and `POST /api/services/{id}` with `{want}` - The viewer: a Services page plus a small state chip in the status bar. A failed service notifies you. ## In the viewer **The sidebar** gets a **Plugins** group after Project, listing every installed plugin's pages. Each entry has a dot: green when its service is ready, amber when its `status` check reports trouble. **A page** is `#//plugin/.` and shows the service's path in a full-height iframe: - It uses the viewer's own hostname, so cookies and CSRF work: 127.0.0.1 on another port is still same-site. - `?at=` keeps a deep link across reloads. - When the service is down, the page shows start and log controls instead of the iframe, and "Open in a new tab" is always there. ## The workflows plugin Workflows is a Laravel package (a library), not an app. So the plugin ships a small **host app** in the workflows repo at `.journal-plugin/host/`. It requires the package through a Composer path repository pointing at the clone itself, and commits its `composer.lock`. The editor bundle `dist/` is already committed, so Node is not needed. **The pages:** - Workflows: the dashboard at `/workflows`. - Engine: the same app, with the journal's service states and `/workflows/api/engine/health` above it. **The bridge both ways:** - **Journal to workflows.** The host app's `POST /journal/events` checks the token and fires one of three triggers, which appear in the editor with no package changes: `journal.todo_created`, `journal.todo_completed` and `journal.message_received`. - **Workflows to journal.** `journal:integration` writes an integration manifest whose actions are "Journal: notify" (`POST /api/{env}/notifications`) and "Journal: create to-do". A workflow answers into the environment it was triggered from. **Changes in the workflows repo:** - Add `.journal-plugin/` with `plugin.json`, `bin/prepare.sh` and the host app: bootstrap, routes, the journal triggers and controller, the `journal:key` and `journal:integration` commands, the jobs, cache and sessions migrations, and `composer.lock`. - Stop tracking `.journal/`. The repo still tracks 867 files there, and a clone inside `.journal/plugins/` would carry a nested journal that the `journal` command finds first. **What is kept where:** - `APP_KEY`, the database and storage live in `{data}`, so they survive reinstalls. `journal:key` writes the key only when it is missing. - The ports are the journal's, so the plugin never fights your own checkout's rig on 8000/8085. ## Two hazards fixed first 1. The supervisor walks every `*.py` under `.journal` every 5 s, and the launcher does so every 0.5 s. A plugin's `vendor/` would make every tick crawl, and a `.py` inside a plugin would restart the engine. The walk will scan only `.journal/src`. 2. The workflows repo's tracked `.journal/`, described above. ## Settled defaults (tell me to change any) - Plugins are installed per machine, like the record: there is no committed lockfile yet. - On/off is per project. - Events missed while the viewer was down are replayed only if they are younger than 10 minutes. - A surviving session takes over the services. - A failed service notifies you. - Writes from plugins get their own `plugin` actor. - Workflows runs as four explicit services instead of `workflows:dev`: the journal owns each process, and each dies with the session.