RemotePilot: An Autonomous Remote Agent Daemon That Plans, Acts, and Self-Corrects

What it is

RemotePilot is an autonomous remote agent system: a Python FastAPI daemon that takes a goal in plain language, plans a multi-step execution strategy, runs each step (shell commands, UI clicks, keystrokes, browser actions), verifies the result — including visually — and re-plans when something fails. A Flutter client app in ui/ provides the remote interface. It streams task logs over WebSocket, supports cron-scheduled goals, and can expose the daemon remotely through a Cloudflare tunnel.

The repository (github.com/Shivay00001/RemotePilot) was verified directly from source: the full agent pipeline, task lifecycle, auth middleware, scheduler, tunnel manager, and sandbox all exist as working code. Notably, the project's own README documents its hard limits up front — including the fact that its "sandbox" is not actually a sandbox — which is refreshingly honest and will be treated as a design fact here, not a footnote.

The problem it solves

Remote operations — maintaining servers, running reports, driving desktop software — still require human hands on keyboards. The available automation landscape is two unsatisfying extremes:

1. Brittle scripts. Cron jobs and shell scripts that fail silently or break when the UI changes.
2. Unattended LLM agents with no guardrails. Powerful, but with no verification loop, no safety screen, and no way to watch what they're doing.

RemotePilot targets the middle: an agent that plans in steps, screens each plan for danger, executes one step at a time, verifies the outcome (including by looking at the screen), and re-plans from the actual current state when verification fails. Every one of those behaviors exists in the code.

How it works: the execution lifecycle

The heart of the system is process_task() in daemon/main.py, which runs as a background asyncio task after POST /task/submit:

POST /task/submit  {"goal": "..."}
   │
   ▼
┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐
│ PLANNING │→│ SECURITY │→│   ACT    │→│ VERIFY   │→│ DONE /   │
│ (LLM)    │  │ (screen) │  │ (agent)  │  │(vision)  │  │ FAILED   │
└──────────┘  └──────────┘  └──────────┘  └──────────┘  └──────────┘
                                    │ fail
                                    ▼
                              RE-PLAN (vision context)

1. Planning — with memory

PlannerAgent (daemon/agents/planner.py) converts the goal into a strict JSON list of atomic actions. Before generating, it pulls semantically similar successful plans from memory and includes them as few-shot context:

Similar successful plans from history:
- Goal: ...  Plan: [...]

Available actions: COMMAND, TYPE, HOTKEY, CLICK, WAIT, BROWSE, CLICK_BROWSER. The LLM call goes to a local Ollama server (http://localhost:11434, default model llama3.2, format: "json"), parsed defensively — invalid JSON returns an error status rather than crashing the pipeline.

2. Security screening

Every plan passes SecurityAgent before execution. It runs two checks:

• Heuristic denylist — regex patterns for rm -rf, mkfs, sudo, fork bombs, format, etc. Match → BLOCKED with the pattern named in the reason.
• LLM intent check — the plan is sent to Ollama with an "analyze for malicious intent, return SAFE or MALICIOUS" prompt. If the LLM call fails or times out (2s), the agent falls back to the heuristic result rather than blocking everything.

3. Action and verification loop

ActionAgent executes one step at a time; then VerifierAgent checks the result against the expectation. The verifier is wired to the VisionAgent, which uses a vision-language model (moondream/llava via Ollama) to describe the actual UI state. On verification failure, the loop triggers a re-plan with vision context:

RE-PLANNING REQUIRED.
Original Goal: ...
The step ... FAILED with error: ...
Current Screen State: <vision model description>
Generate a NEW plan ... starting from this state.

Up to 10 retries, resetting on each success. This is the most interesting design decision in the codebase: the agent doesn't retry blindly — it re-observes the world and plans from there.

4. Synthesis and memory

If the execution produced research fragments, ResearchAgent synthesizes them into a summary. The completed (or failed) plan is then stored: memory_store.add_interaction(task.goal, task.plan) and the MemoryAgent, so future similar goals retrieve past plans — the retrieval-augmented planning loop in section 1.

The 12-agent roster

daemon/agents/ contains the full cast, registered by Coordinator in daemon/coordinator.py:

The coordinator also supports a text command interface (plan <goal>, run <cmd>, see <...>), with the legacy run path audit-logged rather than safety-screened — documented honestly as "UNCHECKED for Phase 1/2 compatibility."

Remote access, scheduling, and auth

• REST + WebSocket. POST /task/submit, GET /task/state/{id}, POST /task/schedule (cron via APScheduler), GET /metrics, and WS /ws/logs for live log streaming via asyncio queues.
• Cloudflare tunnels. tunnels.py starts/stops a cloudflared tunnel so the daemon can be reached from outside the LAN; the tunnel token comes from the request body.
• Token auth. daemon/auth.py requires Authorization: Bearer <token> (the REMOTEPILOT_TOKEN env var, constant-time comparison, never logged) on every route except /health. The websocket takes ?token=. If REMOTEPILOT_TOKEN is unset, the service runs in openly-reported dev mode (/health says so).

The sandbox situation — read this

The README states it plainly and the code confirms it: daemon/sandbox/local.py (ProcessSandbox) is a plain subprocess shell, not a sandbox. It runs the command with an empty environment (deliberately, to avoid leaking host secrets) and tracks processes for cleanup, but there is no containerization, no chroot, no capability restriction. The action.py code itself admits COMMAND actions run "un-sandboxed in this poC."

Practical consequence: do not expose this daemon to untrusted goals. It is a personal-automation daemon for machines you control, not a multi-tenant agent runtime. The README's honesty about this is a feature — evaluate accordingly.

The Flutter client

ui/ is a full Flutter project (Android, iOS, Linux, macOS, Windows, web) with ui/lib/main.dart as its entry point. The README notes it was not exercised in the verification pass — treat the mobile/web client as present-but-unverified, and drive the daemon via its REST API for now.

How to run it

git clone https://github.com/Shivay00001/RemotePilot.git
cd RemotePilot/daemon
pip install -r requirements.txt
export REMOTEPILOT_TOKEN=<redacted>
uvicorn main:app --host 127.0.0.1 --port 8000

Then:

curl localhost:8000/health
# {"status":"ok","auth":"token"}

curl -X POST localhost:8000/task/submit \
  -H "Authorization: Bearer $REMOTEPILOT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"goal": "summarize the repo"}'

Without a token: 401 {"detail": "unauthorized: bad or missing token"}.

For the agent pipeline to actually plan and see, you need a local Ollama (llama3.2 for planning, a vision model for the VisionAgent) — without it, tasks run the lifecycle and mark themselves FAILED with a logged error rather than hanging.

Honest status and roadmap

Status: working proof-of-concept daemon with documented limits. The task lifecycle, planning loop, security screening, vision-verified self-correction, scheduling, tunnels, and auth all exist as real code. The honest gaps:

• No real sandboxing. The biggest one. Container or VM isolation is the obvious next milestone.
• LLM-dependent. Planning, vision, and intent checks all need local Ollama; there is no cloud-LLM fallback in the current code.
• UI automation needs a display. pyautogui-based actions fail cleanly headless (lazy import), but full UI driving is a desktop-with-display scenario.
• Flutter client unverified. Present in the tree, not exercised end-to-end.
• Tunnels shell out to cloudflared — an external binary dependency, not bundled.

A natural roadmap: real sandbox execution (Docker), pluggable LLM backends, screenshot-grounded verification with coordinates, and a signed audit log. The architecture — particularly the vision-context re-planning and the retrieval-augmented planner — is the part worth keeping no matter how it's rebuilt.

Why it matters

Most "autonomous agent" demos stop at "the LLM wrote a plan." RemotePilot goes further: screen the plan, execute one step, verify against reality (including visually), and re-plan from the observed state. That closed loop — plan → act → verify → re-plan — is the difference between a demo and a system, and the code implements it faithfully within its sandbox limits. For anyone building personal AI automation they can run on their own hardware, this is a solid, readable reference architecture.

Source: github.com/Shivay00001/RemotePilot · MIT license

Frequently asked questions

What does RemotePilot do?

It accepts a plain-language goal, plans a multi-step execution with a local LLM, screens each plan for dangerous commands, executes steps one at a time (shell, UI clicks, keystrokes, browser), verifies each outcome — including visually — and re-plans from the observed state when a step fails. It exposes REST endpoints, live logs over WebSocket, cron scheduling, and a Flutter client.

Does it need cloud services?

No. Planning and vision run against a local Ollama server (llama3.2 for planning by default). Remote access is provided by shelling out to the cloudflared binary for Cloudflare tunnels.

Is it safe to run?

It has token auth (except /health), a security agent with a heuristic denylist plus an LLM intent check, and audit logging. But its sandbox is a plain subprocess shell — no containerization — so it should only ever run goals you trust, on machines you control. The project README states this limit up front.

What are its hard limits?

Planning and vision need local Ollama models; UI actions need pyautogui and a display; tunnels need the cloudflared binary installed separately; and the Flutter client in ui/ ships unexercised — drive the daemon via its REST API for now.