VQ OpenMemory: Encrypted, Audited Memory Infrastructure for AI Applications

What it is

VQ OpenMemory is an open-source memory service for AI applications — a self-hosted system that gives your agents and LLM apps long-term, structured memory with encryption, rate limiting, audit trails, and an adversarial output auditor. Think of it as the memory layer your RAG pipeline forgot to secure: it stores what the model should remember, compresses what no longer fits in context, encrypts it per tenant, and then checks the model's answers against its own stored facts before anything reaches the user.

The repository (github.com/Shivay00001/vq-openmemory) is a Python package with a FastAPI service, a Python client wrapper, and Docker Compose packaging. It was verified directly from source: the memory store, encryption layer, rate limiter, compressor, billing meter, and auditor all exist as working code with smoke tests, not as README promises.

The problem it solves

Every serious AI application eventually hits the same wall: the model forgets. Session state evaporates between calls, fine-tuned knowledge goes stale, and developers bolt on memory as an afterthought — an unencrypted vector store, a JSON file, or a messages array that silently eats the context window. Three concrete problems follow:

1. Context rot. Long conversations overflow the context window. Most developers either truncate (losing facts) or summarize (introducing drift) with no discipline.
2. Unprotected recall. Memory content — often PII, credentials, or proprietary facts — sits plaintext in vector databases and SQL tables.
3. Hallucination at scale. Even with memory, nothing checks whether the generated answer contradicts the stored facts. Memory becomes a suggestion the model can ignore.

OpenMemory's answer is architectural: separate the memory system from the generation system, encrypt everything per tenant, compress context deterministically, and run an adversarial auditor model over every output before it ships.

Architecture

The system has four layers — core/, security/, api/, and audit/ — wired together by a FastAPI service and a drop-in client.

vq-openmemory/
├── core/            # MemoryStore, models, ContextCompressor
│   ├── memory_store.py
│   ├── models.py
│   └── compressor.py
├── security/        # Envelope encryption (Fernet + AWS KMS)
│   └── encryption.py
├── api/             # FastAPI server, rate limiter, billing meter
│   ├── server.py
│   ├── rate_limiter.py
│   └── billing.py
├── audit/           # Panopticon LLM-as-a-judge auditor
│   ├── panopticon.py
│   └── prompts.py
├── client.py        # Drop-in request wrapper
├── requirements.txt
├── Dockerfile / docker-compose.yml
└── tests/           # smoke tests

1. Dual memory: episodic + semantic

MemoryStore in core/memory_store.py keeps two kinds of memory:

• Episodic memory — a SQLAlchemy-backed table (episodic_memory) storing time-ordered interactions: role, content, session, timestamp, tenant. This is the conversation log, queryable as recent history.
• Semantic memory — a ChromaDB collection (semantic_facts) storing vector-embedded key/value facts: "the customer prefers morning delivery", "the clinic closes Sundays". Queries run vector search with a strict tenant filter:

results = self.semantic_collection.query(
    query_texts=[query_text],
    n_results=n_results,
    where={"tenant_id": tenant_id}   # Strictly isolate by tenant
)

Every fact's metadata carries an encrypted payload, so even the vector store's metadata is opaque to anyone with raw DB access.

2. Envelope encryption, per tenant

security/encryption.py implements genuine envelope encryption:

1. Each tenant gets a data key (Fernet, i.e. AES-256-CBC + HMAC) stored in the tenants table.
2. The data key itself is encrypted by a master key — real AWS KMS when AWS_KMS_KEY_ID is set, via boto3.client('kms').encrypt/decrypt. Without AWS credentials it falls back to a locally generated mock master key (clearly commented as testing-only).
3. Every payload is encrypted with the tenant's decrypted data key, so one compromised key leaks one tenant, never the whole platform.

The critical design decision is in the right place: encryption happens in the store layer, not the API layer, so every consumer — the API server, the compressor, the auditor — gets encrypted-at-rest data automatically.

3. The Panopticon auditor: adversarial LLM-as-a-judge

The most distinctive piece is audit/panopticon.py. After the primary model generates a response, a separate auditor model (default gpt-4o-mini, temperature 0.0 for determinism) evaluates the output against the stored semantic facts:

is_approved, reasoning, corrected = self.auditor.audit_response(
    user_prompt=user_prompt,
    generator_response=generated_text,
    context_facts=current_facts
)

If the auditor rejects, the generator is forced to retry with the rejection reason appended — up to max_retries times — after which the auditor's corrected response is returned with an explicit safety-intervention flag. The architecture is fail-safe: if the auditor itself errors, the verdict defaults to REJECT, never APPROVE. That's the kind of default you only get when someone has thought about adversarial failure.

4. Context compression with facts injection

core/compressor.py implements "Lossless Context Management": when history exceeds the token budget (using ~4 chars/token as the proxy), it keeps the system prompt and the most recent messages, and replaces the dropped middle with a compression notice that injects the semantic facts:

[SYSTEM ALERT: N older messages were compressed to save context.
Essential semantic facts retained: ... If you need older details,
they are stored in Episodic Memory.]

This is the right trade: recency for fluency, facts for accuracy, and a pointer back to episodic memory for drill-down.

5. API layer: auth, rate limits, billing

api/server.py is an async FastAPI service (title: "OpenMemory Enterprise API", v3.0.0) with:

• API-key auth — X-API-Key header validated against hashed keys in the tenants table via a FastAPI dependency (verify_api_key); invalid key → 401.
• Token-bucket rate limiting — RateLimiter(requests_per_minute=100) on every write path: Redis-backed when available, with a graceful in-memory fallback (including a memory-leak guard).
• Billing meter — BillingMeter logs every storage write (context_compression, storage_write events) to billing_logs, exposed at GET /v1/billing/usage.
• Async Postgres via create_async_engine, with an SQLite fallback for local runs, and Chroma persistence to ./chroma_db.

The drop-in client

client.py is the intended integration path. OpenMemoryClient wraps your existing LLM client and runs the full lifecycle on every call:

1. Log the user interaction to episodic memory
2. Compress the context if needed
3. Inject the Panopticon warning into the system prompt
4. Generate with your model
5. Audit the output (retry up to max_retries on rejection)
6. Log the assistant response to episodic memory

Usage, per the README:

from client import MemoryClient
client = MemoryClient()
client.store("context", "value")

Note: the README's MemoryClient name vs. the code's OpenMemoryClient class name — use the class name from client.py (OpenMemoryClient), which takes your llm_client as a constructor argument.

How to run it

Prerequisites: Python 3.10+, and optionally Redis, Postgres, and AWS KMS.

git clone https://github.com/Shivay00001/vq-openmemory.git
cd vq-openmemory
pip install -r requirements.txt
# or
docker-compose up

The service entry point is the FastAPI app:

uvicorn api.server:app --reload

Endpoints available out of the box:

For local development with no infrastructure, it degrades gracefully: SQLite instead of Postgres, in-memory rate limiting instead of Redis, and a local mock master key instead of AWS KMS.

The billing meter and multi-tenancy

The tenants table is the spine of the multi-tenant design: tenant_id, a hashed API key (api_key_hash), and the tenant's encrypted data key. BillingMeter in api/billing.py records storage_write and context_compression events with byte quantities to billing_logs, and GET /v1/billing/usage returns the tenant's aggregated consumption — the seed of a usage-based SaaS pricing model. Combined with the per-tenant rate limiter (100 requests/minute default on write paths) and the strict tenant filter on every Chroma query, the system is designed to host multiple customers on one deployment with isolated data, isolated keys, and metered usage. The missing piece is Postgres row-level security to enforce that isolation at the database layer rather than purely in application code — the current where={"tenant_id": tenant_id} filters are correct but rely on every code path remembering to apply them.

Honest status and roadmap

Status: working prototype, not audited production software. What's genuinely done: the dual memory store with encryption, the compressor, the rate limiter, billing metering, the fail-safe auditor, the FastAPI service, and Docker packaging — all verified as real code. What to know before trusting it:

• KMS is a local mock unless you configure AWS. The envelope structure is real, but AWS_KMS_KEY_ID must be set for true KMS-backed keys.
• The auditor doubles LLM spend per call (one generation + one audit), and retries multiply it. Budget accordingly.
• Tests are smoke-level (tests/test_smoke.py verifies files parse; test_enterprise.py/test_openmemory.py exist but coverage is thin). Run your own integration tests before production.
• The README has a class-name mismatch (MemoryClient vs OpenMemoryClient) — the code is authoritative.

Roadmap items a contributor could pick up: Postgres row-level security for tenant isolation at the DB layer, configurable compression strategies (summarization vs. extraction), auditor prompt versioning, and Prometheus metrics on the /v1/memory/* paths.

Why it matters

Memory is becoming the load-bearing layer of every AI product, and most of it is still being built like it's 2023 — a vector index with a prayer. VQ OpenMemory treats memory as infrastructure: encrypted per tenant, metered, compressed deterministically, and adversarially audited. It's a prototype with a production shape, and for builders who've been burned by context rot or memory-shaped security holes, the design decisions are worth studying even if you adapt rather than adopt.

Source: github.com/Shivay00001/vq-openmemory · MIT license

Frequently asked questions

What does VQ OpenMemory do?

It is an open-source memory service for AI applications: encrypted episodic (SQL) and semantic (vector) memory, deterministic context compression, a rate-limited FastAPI with billing, and an adversarial LLM auditor that checks every output against stored facts before it reaches the user.

How is the stored memory encrypted?

Through envelope encryption: each tenant gets a Fernet (AES-256) data key, which is itself encrypted by a master key — real AWS KMS when AWS_KMS_KEY_ID is set, with a clearly-marked local mock for development. Vector queries are strictly tenant-isolated.

What is the Panopticon auditor?

An LLM-as-a-judge layer (default gpt-4o-mini at temperature 0.0) that evaluates each generated response against the stored semantic facts. Rejections force regeneration with the failure reason appended, up to a retry limit — and if the auditor itself fails, it defaults to REJECT, never APPROVE.

Is it production-ready?

It is a working prototype with a production shape. Before production you should configure real AWS KMS, budget for the doubled LLM cost the auditor implies, add integration tests beyond the included smoke tests, and fix the README's MemoryClient vs OpenMemoryClient class-name mismatch.