Website Optimizer Agent: A Self-Hosted, Multi-LLM Technical SEO and CRO Auditor

TL;DR: The Website Optimizer Agent is an open-source audit service: submit any website URL, pick an LLM (OpenAI, Anthropic, Gemini, Zhipu GLM, or local Ollama), and get a scored technical SEO + CRO audit report. It's a FastAPI backend (v1.1.0) with a Next.js dashboard, multi-LLM routing through LiteLLM, async background jobs logged to a database, an SSRF guard that refuses to audit private network targets, and API-key auth that fails closed. We read the server code to verify — real job pipeline, real security controls, a 15-check smoke test. It's working, self-hostable software; you bring your own LLM key, and the "sellable audit service" framing (₹5–10k per audit+fix) is the stated business model, not a deployed product.

The problem: broken websites, black-box audits

Most small-business websites are slow, missing meta tags, and leaking conversions — and the owner has no idea. The audit tools that exist are either expensive SaaS with black-box scoring, or free checkers that spit out a generic checklist. What agencies actually sell is the interpreted audit: a human reads the site, writes up what's wrong in priority order, and charges ₹5,000–10,000 for the audit plus the fix.

The Website Optimizer Agent automates the reading-and-writing part. An LLM does the interpretation; the code around it handles fetching, safety, queuing, and reporting. The repo is at github.com/Shivay00001/website-optimizer-agent.

What it actually is

Two halves: a FastAPI backend (backend/) that fetches a target site's HTML, sends it to a configured LLM via LiteLLM for a technical SEO + CRO audit, and stores the report; and a Next.js + Tailwind frontend (frontend/) where you enter the backend API key, pick a model, enter a target URL, and run the audit. OpenAPI docs are served at /docs.

The API surface is small and clean:

How it works: the audit pipeline

Submit a job and this happens, all visible in backend/server.py:

1. Validation. target_url must be a public http(s) URL and model_id must start with one of the allowed prefixes: gpt-, claude-, gemini/, zhipu/, or ollama/. Anything else is a 422 with structured JSON.
2. Background execution. The endpoint returns 202 immediately with a task_id; the audit runs as a background task. An ExecutionLog row (SQLAlchemy, SQLite by default, Postgres supported) tracks the job through pending → running → success/error, and the frontend polls /api/tasks/{task_id}.
3. Fetching. fetch_website() pulls the page with httpx (15s timeout, follows redirects), strips <script> and <style> with BeautifulSoup, and truncates to 15,000 characters (MAX_HTML_CHARS) — bounding both cost and context.
4. LLM audit. The cleaned HTML goes to LiteLLM with the chosen model. No stubs, no mock fallbacks — the code comment says so explicitly, and the pipeline genuinely calls the provider. The response is capped at 2,000 tokens (LITELLM_MAX_TOKENS).
5. Report storage. The audit report lands in the execution log; the client fetches it from the task endpoint.

The security engineering is the real story

Plenty of demo AI wrappers skip this. This one doesn't:

• SSRF guard. _assert_public_host() resolves the target's DNS and refuses any host that resolves to a private or loopback IP — so the audit endpoint can't be abused to probe someone's internal network. There's a DISABLE_SSRF_GUARD=1 escape hatch documented as test-only, never for production.
• Fail-closed auth. Every non-public endpoint requires X-API-Key, compared with hmac.compare_digest. If API_KEY isn't set on the server, it returns 503 instead of serving unauthenticated.
• Secrets discipline. Everything is env vars — no hardcoded secrets, keys, or URLs. LLM provider keys can be supplied per-request via headers (X-OpenAI-Key, X-Anthropic-Key, …); the frontend keeps them in browser localStorage and they are never stored server-side. The smoke test even strips inherited LLM keys from its environment to stay hermetic.
• Structured errors. All errors are {"code": "...", "message": "..."} — stack traces never leave the server.
• Sane CORS. Explicit origin allowlist only; allow_credentials is never combined with *.

A smoke_test.py runs 15 checks against the real app: health, auth (missing/wrong key → 401), input validation (bad URL/model/loopback target → 422), submit → 202 + task_id, unknown task → 404, no stack-trace leakage, no wildcard CORS. That's a real test suite, not a README claim.

What the audit covers

The README frames the output as a scored technical SEO + CRO audit report with critical issues and recommendations. In practice that means the LLM is asked to read the site's cleaned HTML the way an auditor would: missing or duplicate titles and meta descriptions, heading hierarchy, image alt attributes, internal linking, page-speed signals visible in markup, mobile-friendliness cues — on the SEO side; and value proposition clarity, call-to-action placement, trust signals, and friction points — on the CRO side. The "scored" framing matters for the sellable-service angle: a score gives the prospect a number to react to, and the critical-issues list gives the agency the fix backlog to quote against. The quality ceiling here is the model you point it at — which is exactly why the multi-provider design exists.

The data layer: small but deliberate

The backend's persistence is a single ExecutionLog model via SQLAlchemy async — task id, target URL, model id, status, timestamps, and the report payload. SQLite is the default (a file next to the app, zero setup), Postgres a one-line DATABASE_URL change. There's no user table, no session store, no analytics pipeline — the data model is exactly as big as the product needs and not a row bigger. For a self-hosted tool this is a feature: backup is copying a file, and there is no personal data to leak beyond the URLs you chose to audit.

How to run it

Backend:

cd backend
cp .env.example .env          # fill in API_KEY + at least one LLM provider key
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
uvicorn server:app --port 8010
# check: curl http://localhost:8010/health → {"status":"ok",...}

Frontend:

cd frontend
cp .env.example .env          # set NEXT_PUBLIC_API_URL to your backend URL
npm install
npm run dev                   # http://localhost:3000

Enter the backend API key (must match the server's API_KEY), optionally add your own LLM provider keys, enter a target URL, run the audit. For production, the README documents a Render free-tier deployment (web service for the backend, static site for the frontend) and a Dockerfile is included. Run the smoke test with cd backend && API_KEY=<redacted> python smoke_test.py.

Who this is for, and the provider math

Agencies are the obvious fit: run audits for every prospect, attach the report to the pitch, charge for the fix. The per-request key design means you can even run it client-side in a sales call — the prospect's URL, your API key, their screen. Freelancers selling website fixes get a repeatable audit product without building one. SMB owners comfortable with a terminal get a second opinion on their site before hiring anyone.

The provider choice is a cost lever, not just a quality one. Flagship models produce the most polished reports; local Ollama models cost nothing per audit and keep the target URL entirely on your machine — relevant if you're auditing client sites under NDA. The 15,000-character HTML cap and 2,000-token response cap keep each audit to a predictable, small inference cost either way. And because the model id is just a string prefix away, swapping providers is a one-field change, not a migration.

Honest status and roadmap

The honest read: the engineering is genuinely production-minded — the security controls alone put it ahead of most open-source AI wrappers. What's missing is everything around the code: a hosted instance, the audit report's client-facing polish, and the business wrapper (pricing page, delivery format, the monthly care-plan upsell). If you're a developer, the SSRF guard + per-request key pattern is worth reading. If you're an agency, this is an audit engine waiting for a brand.

Frequently asked questions

The bottom line

The Website Optimizer Agent is a serious piece of self-hostable AI infrastructure: a real async audit pipeline, real multi-LLM routing, and real security controls, with the test suite to back the claims. The code is public at github.com/Shivay00001/website-optimizer-agent — clone it, point it at a site you own, and read the audit it produces. If you run an agency, the audit-plus-fix business model is sitting right there in the README.

Want website audits or AI agents for your business? VisionQuantech builds custom AI tools, website optimization, and automation for SMBs — with working demos before you pay anything. Contact: [email protected]