VQ Agent Core: A Three-Layer, Phone-First AI Agent Engine (Open Source)

TL;DR: VQ Agent Core is an open-source Python engine that runs AI agents as a locked three-step loop — THINK → ACT → DECIDE → THINK. A local LLM understands and composes, Needle 2 handles fast grammar-constrained tool routing, and a swappable judgment layer (Jev cloud, or Strands Decider 2B / Laya / prompt-based LLM locally) gates every low-confidence or ambiguous action. Every step is written as a JSONL trace. It degrades honestly when models are unreachable — factual summaries, never fake prose.

The problem: most agent frameworks are one giant guess

A typical agent framework works like this: the LLM decides everything — what the user meant, which tool to call, what arguments to pass, whether the result looks right — in one uninterrupted stream of text. That works until it doesn't. The tool call is slightly wrong, the judgment is slightly off, and there's no structural way to catch it. Cloud-only frameworks add two more problems: your data leaves the machine on every judgment call, and the whole system dies without connectivity.

VQ Agent Core answers with structure instead of scale. Instead of one model doing everything, three specialised layers each do one job:

• THINK — understand the request, reason over it, compose the final answer. Any OpenAI-compatible local LLM.
• ACT — pick the tool, fill the arguments, report a confidence. Needle 2: fast, local, grammar-constrained function calling.
• DECIDE — approve, clarify, or escalate. A purpose-built decision model (Jev in the cloud, or Strands Decider 2B / Laya on-device) that only fires when ACT's confidence is below threshold or the request is ambiguous.

The loop always returns to THINK, which composes the final answer from the executed steps. Ambiguity never flows silently downstream — it gets stopped at the DECIDE gate.

How the three layers work

THINK: the understanding and composition engine

The THINK layer lives in think.py with local_model.py providing the OpenAI-compatible client. It exposes two entry points the agent loop uses:

• understand(request, media_paths) — parses the request and any attached media, returning understanding, actionable_request (what ACT will route on), a plan_hint, and a via field naming the path taken.
• compose(request, steps) — turns the executed steps into the final answer.

The important design choice is what happens when no local model is reachable. Both functions degrade honestly: understand() returns the raw request tagged via="skipped" (and logs that it did), while compose() returns a factual structured summary of what actually happened — never synthetic LLM-sounding prose. The code comments are explicit: "Never fakes content." Text files under 100KB get inlined (head of file); binary media (audio, video, images, scanned documents) is described by metadata only unless the configured model is multimodal-capable. This "phone-first" design is deliberate — the engine is built to run on a phone-class device, where the model behind the endpoint might be a Qwen2.5-Omni-class or Gemma-3n-class model, or might not exist at all.

ACT: fast, grammar-constrained tool routing with Needle 2

The ACT layer is needle_router.py, backed by cactus-needle==2.0.15. Needle 2 does grammar-constrained function calling: instead of hoping the LLM emits valid tool JSON, the router constrains generation so the tool name and arguments are structurally valid by construction. Two backends exist: ctypes (in-process, for Linux/macOS/Windows) and serve (HTTP to the needle CLI binary — the Termux/Android path, so the same engine runs on a phone).

The tools themselves live in a registry under tools/:

The confidence threshold lives in config.yaml (router.confidence_threshold, default 0.7). When ACT routes a tool below that threshold — or the request is ambiguous — the agent doesn't guess. It calls the DECIDE gate.

DECIDE: a swappable judgment layer, cloud or fully offline

deciders.py is a factory over four judgment engines, selected by the decider: key in config.yaml (jev | local | auto):

1. Jev (TypeSafe cloud) — the sharpest judge, a purpose-built decision model exposing choice / score / noul in one call. Used for reply classification, lead scoring (temperature + fit 1–5), and escalate-vs-proceed decisions.
2. Strands Decider 2B (local) — the first local backend: a Qwen3.5-2B base plus pointer head plus LoRA adapter, Apache-2.0, giving typed choice/score/noul answers in one forward pass, calibrated by design. Needs ~4.5GB of weights, so PC/VM only.
3. Laya (local) — the second local backend, a purpose-built ~1GB decision model. Same interface, lighter footprint.
4. Prompt-based local LLM — last resort: a general chat model asked to output strict JSON. Private, offline, free, but weaker and uncalibrated.

decider: auto tries Jev first and falls back to local when it's unreachable. If every backend is unavailable, the error propagates and the agent escalates to a human — the safe default, coded rather than hoped for.

The decision jobs are concrete, not abstract: classify_reply() turns a prospect's email into a label like demo_requested with confidence; score_lead() returns temperature and fit; should_escalate() returns an escalation probability and a recommended action. Tools read the same VQ_DECIDER environment variable that agent.py sets from the config, so they stay consistent with the running agent even when invoked standalone.

The loop and its trace

agent.py wires the three layers into the Agent class: input → THINK (understand) → ACT (route + execute via the tools registry) → DECIDE (only through _decide_gate, on low confidence or ambiguity) → THINK (compose the final answer). Every event — tool start, tool result, decide gate — is appended as a line to runs/run-<timestamp>.jsonl, so any run is fully auditable after the fact. Usage is a single command:

python agent.py "score this lead: ..."        # default config.yaml
python agent.py --config my.yaml "do X"       # custom config

Code-level specifics worth noting

• Honest degradation is a feature, not a fallback: think.py refuses to fabricate prose when the model is down, and the media describer marks binary files as "described by metadata only" rather than inventing contents.
• The DECIDE backends share one interface (decide / classify_reply / score_lead / should_escalate), so swapping Jev for Strands or Laya changes no calling code — only the config value.
• Local backends degrade in order: LocalDecider._backends = ("strands", "laya", "prompt"), each raising a typed *DeciderUnavailable error that the runner catches and moves past. You can also pin one backend explicitly.
• Phone-first is real: Needle 2's serve backend, the Termux note in the docs (phone-setup.md), and the prompt-based local decider as the "last resort" on devices where torch can't install all point to an engine designed to work on a phone, not just a server.
• Repo contents (28 files, verified in the tree): the loop, the three layers, the tool registry, config.yaml, requirements.txt, fetch_model.sh / start_server.sh helpers, test_agent.py and test_strands_backend.py test harnesses, and dated engineering notes under notes/.

How to run it

1. Clone and install: git clone https://github.com/Shivay00001/vq-agent-core.git, then pip install -r requirements.txt (key pinned dependency: cactus-needle==2.0.15 for the ACT layer; torch plus model weights only if you want the local DECIDE backends).
2. Point THINK at a model: set the OpenAI-compatible endpoint in config.yaml under local_model (base_url, model). fetch_model.sh helps with the download.
3. Choose your judge: decider: jev (needs a TypeSafe key) for the sharpest judgments, decider: local for fully offline (Strands Decider 2B first if its weights are present, then Laya, then prompt-based), or decider: auto for Jev-with-local-fallback.
4. Run: python agent.py "classify this reply: ...". Read the trace in runs/ to see every THINK/ACT/DECIDE step.
5. On a phone: follow phone-setup.md — the Needle serve backend and prompt-based local decider are the Termux path.

Honest status and roadmap

Working. The core loop, all three layers, the tools registry, and the JSONL tracing are implemented and present in the repository. The repo ships dated engineering notes (late September 2026) from end-to-end test runs, and the README documents measured Strands Decider 2B outputs on a real machine — e.g. classifying a "Yes, let's do a demo" reply as demo_requested at 0.91 confidence, offline, at ~25 seconds per decision on a 2-CPU box. The honest-fallback behavior (no model → factual summary, never fake prose) is coded into think.py, not just promised.

What it isn't: a polished product with a UI, a managed service, or benchmarks beyond the ones documented. The local DECIDE backends need multi-GB weights and a PC-class machine (not Termux), the tool registry is small and clearly meant to be extended, and THINK's multimodal ability depends entirely on which model sits behind the endpoint. Sensible next steps from the code's own structure: more tool modules, a streaming composition path, and a packaged phone build.

The bottom line

VQ Agent Core is a small, opinionated engine with one big idea: separate understanding, action, and judgment into three layers, and make uncertainty stop at a gate instead of flowing downstream. The code backs the claim — honest fallbacks, a swappable calibrated decision layer, full run tracing, and a genuine phone-class deployment path. It's a working foundation for builders who want agents they can audit and run offline, not another demo repo.

Repository: github.com/Shivay00001/vq-agent-core · Want something like this built for your business? VisionQuantech designs and deploys AI agents for clinics, home services, real estate and more. Contact: [email protected]