# Pilot prototype

Pilot turns a task into screenshot-based actions, and useful actions into reusable macros.

## Try it

- Web: https://obstudio.org/tools/pilot/
- Windows: unzip `Pilot-Windows.zip` and open `Pilot.exe`. Windows 10/11 x64; no installer, administrator rights, Node, Python, or separate .NET installation required.
- Browser: unzip `pilot-browser.zip`, open `chrome://extensions` or `edge://extensions`, enable Developer mode, choose **Load unpacked**, and select the folder containing `manifest.json`. Open Pilot from its toolbar icon, select **Browser tab**, and pick a regular website tab.

Managed computers may block unpacked extensions or unsigned programs. Pilot does not bypass those restrictions. The Windows app cannot control administrator windows or the Windows secure desktop.

## First run

1. Use **Local demo** in the web Quiz lab, or **Try local quiz demo** in Windows. These solve visible arithmetic without AI or an API key.
2. Get a Gemini API key at https://aistudio.google.com/apikey. Use a project without paid billing to stay on the free tier. Paste it into **Connections** on the web, or the **Task** tab in Windows. Keys stay in memory, never in saved macros or usage files.
3. Choose **Gemini 3.5 Flash-Lite** for speed or **Gemini 3.8 Flash** for more complex tasks. Both tasks and macro drafting use this choice.
4. Web: choose a quiz or extension tab, then **Run task**. Windows: click **Select target (3 sec)**, select the application during the countdown, optionally **Preview target screen**, enter the goal, and **Run AI task**.
5. Stop with **Esc** in the web workspace, **Ctrl+Shift+Y** in the extension, or **Ctrl+Alt+Esc** anywhere while the Windows app is running. Switching away from the Windows target also pauses its actions.

## Make a macro

In **My macros / Macros**, describe the task and click **Draft from current screen / Draft macro with AI**. Pilot takes a screenshot and prepares JSON steps; nothing runs until you choose **Play**. Review and save the steps. You can also save a completed run as a macro, edit the steps directly, and replay it without API calls.

The no-key draft demo only answers the current arithmetic question in the web quiz. General prompts require your API key. Macros are fixed sequences; keep window size and layout consistent. The AI task mode takes a fresh screenshot between actions and is better for changing pages. Quiz recordings keep the round seed and check the question before clicking.

## Usage and privacy

Connections (web) and Usage (Windows) show AI requests used today, requests left in **your local cap**, and counts by model. Default local cap: 150 requests/day, adjustable from 1 to 10,000. It counts attempted requests, including failures, and resets at midnight Pacific. Web, extension, and Windows counts are separate; this is not Google's project-wide quota. Actual limits: https://aistudio.google.com/rate-limit.

Google limits requests per minute/day and tokens per minute. A full hour of continuous AI is not guaranteed. Pilot stops on quota errors and never switches providers or retries into a paid service. Local macros and the quiz demo make no API calls. A billing-enabled Google project can still charge for API use; Pilot cannot change or verify its billing tier.

AI tasks and drafts send the selected screenshot, task, recent actions, and (in the extension) visible page text directly to Google. No Pilot backend receives your key or screen. Google may use free-tier content to improve products. Do not capture sensitive content. Local macros, model preference, and request counters are saved on your device; Windows counters live under `%LOCALAPPDATA%\Pilot`.

The regular website cannot control your PC or other browser tabs. Use the extension for tabs or the Windows app for desktop control. The native app also includes **Open web version**, which serves its built-in quiz workspace on a random loopback port; that page does not expose a remote desktop-control API.

## Scope and test evidence

The quiz is an independent Kahoot-style arithmetic benchmark. Pilot needs visible question and option text for other quizzes; colored shapes alone are not enough. No real Kahoot session or live Gemini request has been tested without a supplied API key.

Automated tests cover arithmetic, action validation, model routing, mocked screenshot requests, malformed model responses, macro drafting, quota handling, usage reset, cancellation, and the extension bridge. The Windows self-test and demo commands produce JSON evidence in the specified file.

Verified in this build: browser quiz 10/10, recorded quiz replay, prompt-to-macro local draft, model selection, and usage controls; 18 automated JavaScript tests and 16 native assertions pass. The Windows interface renders successfully. Live Windows capture/click verification is blocked in the build environment because Windows reports no foreground desktop window (handle 0); this part needs a run on an unlocked interactive desktop. The app stops without sending input in that condition.

## Build

Requires Node.js and .NET 10 SDK for development only.

```text
npm test
npm start
node package-extension.mjs
dotnet publish desktop/Pilot.csproj -c Release -r win-x64 --self-contained true -o release/windows -p:PublishSingleFile=true -p:IncludeNativeLibrariesForSelfExtract=true
release/windows/Pilot.exe --self-test test-results/desktop-self-test.json
release/windows/Pilot.exe --demo-test test-results/desktop-demo.json
```

`package-release.ps1` packages the executable and browser extension, then copies web assets and downloads to `../obstudio/public/pilot`. Build obstudio with `npm run build`, then run its existing `deploy.py` from the obstudio directory to publish. Only deploy after checking the complete static export.

API models and free-tier details checked against Google's official model, pricing, and rate-limit documentation on 2026-09-20. Screenshots use ordinary multimodal generation with validated JSON actions; this prototype does not call Google's separate paid computer-use tool.

## Jev mode (0.2)

Jev is available in the Chrome/Edge extension and Windows app. Paste your TypeSafe key into the Jev field and choose Jev. The ordinary hosted webpage cannot call Jev directly because TypeSafe blocks its browser origin; use the extension or Windows app instead. Jev's key stays in memory. There is no automatic switch to Gemini.

Jev receives visible text and a list of currently available controls, then selects one allowed action. Screenshots stay local in this mode. Browser controls come from visible page elements; Windows controls come from the selected window's accessibility tree. Canvas-only games, unlabeled controls, and apps without accessible text need Gemini screenshot mode. Jev does not independently browse the internet: Pilot opens/reads the selected page, Jev chooses an action, and Pilot performs it.

For search or text entry, put the exact text to type in double quotes: `Click the search box, type "cats", press Enter, and stop when the results appear.` Jev does not generate free-form text. A Jev macro draft contains one grounded step; for a longer macro, run an AI task and save the executed steps. Confidence below 65% pauses the run; this threshold is a prototype setting, not a measured accuracy guarantee.

Published Jev pricing is $0.042 per million input tokens, output free. Your account may include a free allowance. If its dashboard confirms 5,000 free requests per day, set your local cap to 5,000; Pilot does not verify that plan or show a live account balance. The local counter combines Gemini and Jev requests in the same installation and resets at midnight Pacific, which may differ from TypeSafe's account reset. Actual allowance: https://console.typesafe.ai/.

Official references: https://docs.typesafe.ai/models, https://docs.typesafe.ai/api, https://docs.typesafe.ai/model-jaggedness/jev-1.13. Jev's own docs advise using code for arithmetic; the local quiz solver remains the reliable no-key arithmetic benchmark. No live Jev request has been verified without your key.
