Shivacon-Energy: The 0.5B Energy LLM That Refuses to Hallucinate
TL;DR: Shivacon-Energy is a LoRA fine-tune of Qwen2.5-0.5B-Instruct for energy-domain Q&A (solar, wind, storage, grids) that passed a brutal 4-gate sellability harness: 9/10 domain tasks, 5/5 unknown-honesty, 5/5 refusal, 5/5 JSON formatting, 0/10 hallucinations. It was the only survivor of 9 domain adapters trained on a free Colab T4 — and the repo publishes the 8 failures too. Public at Shivay00001/shivacon-energy.
The problem it solves
Small language models have a specific, expensive failure mode in agentic pipelines: they hallucinate with confidence and they mangle structured output. Most tiny fine-tunes make this worse — the model memorizes narrow patterns, invents facts fluently, and breaks the JSON your tool pipeline depends on.
Shivacon-Energy attacks the problem from the calibration side. Its headline isn't "knows more energy facts" — the author is explicit that it doesn't. The headline is: it would rather say "I don't know" than invent a number, and it emits valid JSON every time. For an agent or tool pipeline, a model that refuses to fake the figures it can't verify is worth more than a model that knows slightly more and lies slightly more.
What the repository actually contains
I verified the tree on GitHub. Everything needed to use and audit the model is on main:
shivacon-energy/
├── adapter_model.safetensors # LoRA weights (35MB)
├── shivacon_energy_adapter.zip # full adapter bundle (34MB)
├── adapters/shivacon_energy_adapter.zip
├── adapter_config.json # LoRA r=16, alpha=32
├── tokenizer.json / tokenizer_config.json
├── chat_template.jinja
├── MODEL_CARD.md # full card with limitations
├── EVAL_REPORT.md # v1 failure report (2026-10-06)
├── V2_TRAINING_REPORT.md # training config + metrics
├── BENCHMARK.md # gate score summary
└── energy-eval.md # full eval transcriptsThe training facts
- Base: Qwen2.5-0.5B-Instruct
- Method: LoRA, rank 16, alpha 32 — ~8.8M trainable parameters (1.75% of the model)
- Data: 1,596 training rows (v2 dataset: refusal rows, anti-hallucination examples, calibration pairs, tool trajectories)
- Compute: free Google Colab Tesla T4 — $0
- Run: 300 steps, 3 epochs, lr 2e-4 — train loss 0.4018, eval loss 0.08377
Total training cost: zero dollars. The adapter itself is 34MB — small enough to ship inside a mobile app or an edge agent.
The 4-gate sellability harness
This is the part that separates the repo from a typical "I fine-tuned a model" post. Nine domain adapters (energy, education, health, sales, marketing, research, core, general, and more) went through the same harness. The gates, run 2026-10-07 on real CPU inference with greedy decoding and per-prompt timeouts:
| Gate | Score | Meaning |
|---|---|---|
| G1 Base vs. adapter (6 prompts) | 0W / 0L / 6T | Ties the base model everywhere — no knowledge lift, zero regressions |
| G2 Domain tasks | 9/10 | One garbled practical-advice answer |
| G3 Unknown honesty | 5/5 | Admits unknown solar figures, points to sources |
| G3 Refusal | 5/5 | Refuses to forecast oil prices or invent facts |
| G4 JSON / tool format | 5/5 | Clean, well-formed JSON every time |
| Hallucination rate | 0/10 | — |
The honest headline, stated in the README itself: the adapter's value is calibration, not knowledge. It doesn't know more energy facts than the base model. It refuses to fake the ones it doesn't know.
The failures are published too
EVAL_REPORT.md documents the v1 run from the day before (2026-10-06), and it's brutal: 0–0–6 against the baseline, 4/4 hallucination failures (invented countries, invented laws, invented drugs answered with fabricated tool calls), wrong physics (10 kW ≠ 10,000 kWh/hour), Chinese token leakage, fabricated file paths. The v2 dataset was rebuilt specifically around those failure modes — refusal rows, anti-hallucination examples, calibration — and v2 passed.
Eight of the nine adapters failed their gates (data contamination, over-refusal epidemics, one adapter that denied a real court case exists). Only energy shipped. Publishing the failure report alongside the winner is unusual and valuable: the 8 failures genuinely teach more than the 1 success, because they map exactly which training-data mistakes produce which pathology in tiny LoRAs.
How to use it
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = PeftModel.from_pretrained(base, "./shivacon-energy") # unzip shivacon_energy_adapter.zipAsk it an energy question it can't verify and watch it say "unknown" instead of hallucinating. The repo also ships chat_template.jinja — the Qwen chat template the eval used — so your prompts are formatted exactly the way the harness graded them, and the model card documents the training-native system prompt (including the tool list) that both base and adapter ran under during evaluation. Reproducing the gates means reusing those exact prompts, which is precisely why they're published.
Honest status and roadmap
Status: production, sellable — with precisely documented limits. The model card's own verdict is SELLABLE ✅, and the limitations section is the most honest part of the repo:
- No measurable knowledge lift over base (0W/0L/6T on G1). If you need deeper energy expertise, this isn't it.
- Energy domain only — off-distribution questions are off-distribution.
- 0.5B parameters — not a substitute for engineering analysis or professional energy audits.
- One known garbled answer on practical home electricity-saving advice.
- Minor: a strict "output ONLY the JSON" phrasing can occasionally leak a
Thought:line instead of pure JSON (observed once; gate prompts scored 5/5).
Roadmap: v3 retraining is in progress for the 8 failed domains, and for energy specifically the next data round targets the weak spots — more practical-advice rows (the G2 failure) and deeper domain coverage so a future version can earn knowledge lift, not just calibration.
The v1 → v2 diff: a recipe for fixing tiny LoRAs
The most instructive document in the repo isn't the model card — it's the pair of reports read together. EVAL_REPORT.md (v1, 2026-10-06) shows a model that fabricated tool calls on invented inputs, invented "key principles" for an invented law, emitted wrong physics, and leaked Chinese tokens. V2_TRAINING_REPORT.md (2026-10-06) shows what changed: the dataset was rebuilt to 1,596 rows around the exact failure modes — refusal rows (questions the model must decline), anti-hallucination pairs (invented entities it must not elaborate on), calibration examples ("I don't know" as a correct answer), and tool trajectories (correct tool selection with real paths).
One day later, the same harness that scored v1 at 0–0–6 with 4/4 hallucination failures scored v2 at 9/10 domain, 5/5 honesty, 5/5 refusal, 0/10 hallucinations. Same base model, same LoRA rank, same 300 steps. The only variable that moved was the data composition. If you take one lesson from this repo, it's that one: for small fine-tunes, the dataset's failure-mode coverage matters more than its size.
Why calibration beats knowledge at 0.5B
There's a structural reason the "calibration, not knowledge" framing is right. A 0.5B model cannot hold deep energy expertise — no 34MB adapter changes that, and the G1 result (0W/0L/6T vs. base) proves it. What a small model can do in a pipeline is be a reliable component: take a bounded question, answer within its competence, refuse outside it, and emit machine-readable output.
That's the actual job description for SLMs in agentic systems. The orchestrator holds the knowledge (retrieval, tools, larger models); the small model handles the high-volume, latency-sensitive, well-scoped turns. A 0.5B adapter that hallucinates 40% of the time is a liability in that slot. One that hallucinates 0% and formats JSON 100% of the time is infrastructure. The eval harness — domain tasks, unknown-honesty, refusal, JSON format, base-vs-adapter comparison — is really a job interview for that infrastructure role, and energy is the candidate that passed.
The eval methodology, briefly
Worth noting because most tiny-model READMEs report vibes: the gates ran on real CPU inference (not API sampling), greedy decoding, max_new_tokens=100, the Qwen chat template, with a hang-resistant harness and per-prompt hard timeouts. Every failure in energy-eval.md was manually adjudicated — the transcripts are in the repo, so you can disagree with the grading. Keyword rubrics plus human review of every fail is the right standard for claims like "0/10 hallucinations," and publishing the transcripts is what makes the claim checkable.
Frequently asked questions
Does it know more about energy than the base model?
No — 0 wins, 0 losses, 6 ties against Qwen2.5-0.5B-Instruct on domain prompts. Its advantage is behavioral: honesty calibration and JSON reliability. The repo states this plainly; buy it for pipelines, not for expertise.
Why publish the failures?
Because tiny-LoRA failure modes (contamination, over-refusal, hallucinated tool calls) are under-documented, and the v1→v2 diff is a reproducible recipe: add refusal rows, anti-hallucination examples, and calibration pairs, and a failing adapter can become a passing one.
What does it cost to reproduce?
$0 in compute (free Colab T4), 1,596 training rows, 300 steps. The adapter is 34MB. The expensive part was the eval harness design, not the training.
Can I use it commercially?
Adapter weights carry the base model's license (Qwen2.5); code and docs are MIT. Check the Qwen2.5 license terms for your use case.
The bottom line
Shivacon-Energy is a small, honest artifact: a 34MB adapter that knows its limits and formats its outputs. In agentic pipelines, that's the behavior that actually matters — and the published failure reports make it one of the more instructive tiny-fine-tune repos on GitHub. Public at github.com/Shivay00001/shivacon-energy — read EVAL_REPORT.md before MODEL_CARD.md.