Running on Zero
Agents
SIPA OS GOVERNANCE
📈
Generate a friendly greeting for any name
Aelin AquaSoul PRO
SoulInPsyAbstract
AI & ML interests
SIPA OS: Autonomous AI for neurodivergent architects. We
replace cognitive noise with a clean terminal and 344+ LLM
auditing. Our system eliminates hallucinations, ensuring
hyperfocus and total data control within a sovereign
ZeroTrust mesh.
Recent Activity
repliedto their post about 6 hours ago
Fine-Tuning the "Don't Fabricate" Rule: 14 Experiments, One Genuine Signal Over five days we ran 14 fine-tuning experiments across six base models to answer a narrow research question: can a simple behavioural rule — "if you don't know, don't guess or fabricate" — be baked into model weights via fine-tuning, rather than living only in the system prompt?
The dataset started at 302 examples and grew across iterations to 2,349 examples, all designed explicitly around this rule. We tested six models: gpt-4o-2024-08-06, Mistral-7B, Qwen2.5-7B, Phi-3.5-mini, Llama-3.1-8B, and DeepSeek-R1.
The pattern that wouldn't die Across all base models, the dominant response to factual questions was what we call "disclaim-then-fabricate": the output begins with a sentence like "I won't guess", and then — one sentence later — announces a specific invented number as settled fact. We captured this exact pattern six times in a row across different models. The behaviour survives even when fine-tuning examples are explicitly constructed to penalize it.
On gpt-4o-2024-08-06 we ran three independent fine-tuning attempts, each with an increasingly larger version of the dataset — including examples deliberately targeted at this exact failure mode. All three runs continued to fabricate when tested. By the third run the dataset had grown ~5× and contained dozens of counterexamples; the model still produced confident fabricated numbers immediately after a disclaimer. Three out of three attempts, no measurable improvement.
One exception that deserved a hard look Only on the final run — 2,349 examples, deployed via Azure OpenAI (deployment suffix protocol0-v5) — did a single test sample come back completely clean for the first time in the entire series.
repliedto their post about 6 hours ago
SIPA OS is a cognitive infrastructure ecosystem I've been building solo for 7 months — 8 apps, one login, built from inside ADHD/BPD/aphantasia, not designed around a diagnosis from outside.
sipa-os.org — the map. Focus (ADHD scaffolding), NeuroPower, AI chat, Shell (SSH terminal), Games, Community, Syntaxit (open M2M agent network), a pitch deck. All free-first — no paywall on the cognitive tools.
The more interesting part for this crowd: Syntaxit is where I've been running an anti-fabrication research thread with @dipankarsarkar — a k=20 resample benchmark on binary-SFT models (Hermes-3, Qwen2.5, DeepSeek-R1). Short version: our first benchmark said "20/20 refusals, 0/20 fabrications" for all three fine-tunes. Under adversarial review it turned out the scorer only checked if the first word was TRUE/FALSE, the token cap was hiding the real behavior, and a save-limit was silently deleting the evidence for our own follow-up claims. Corrected all of it publicly on the model cards rather than quietly fixing it. The current honest finding: both base and fine-tuned models confabulate readily once given room to finish — SFT didn't clearly help or hurt, the caps were just hiding it.
Full trail if you want to see how the sausage gets made, mistakes included: huggingface.co/datasets/SoulInPsyAbstract/sipa-os-governance
Not a pitch. $0 revenue, 10 people signed in. Built because the tools that existed assumed a brain that isn't mine, and because most benchmarks don't survive someone actually reading the file.
repliedto their post about 9 hours ago
Caught myself overclaiming, in public, twice in one file.
Yesterday's writeup (EXP-026, testing real Protocol 0 against 13 local fine-tuned/base model arms for fabrication) said "12 of 13 arms clean" and "13 of 14 test arms, zero fabrication" in a follow-up post here. Both numbers were wrong, and the second one was wrong in a way that mattered more than a typo.
@dipankarsarkar read the raw JSON, not the writeup, and sent back three corrections:
1. Arm count: 13 arms total (5 base models + 8 adapters), not 14. Recounted directly from the data keys — the extra arm never existed.
2. The metric measured the wrong thing. "Clean" meant zero Cyrillic/language-switching (cyr>0). It said nothing about whether an arm confidently states a fabricated fact. Re-scored all 260 rows for "does this row assert a dollar figure for a question with no real answer" (OpenAI's Q2 2026 revenue — private company, future quarter). 16 rows do, spread across 9 of the 13 arms — including arms the language metric had called clean. One of them is a base model with zero fine-tuning, stating "$1.2 billion... consistent with reports from earnings calls" that cannot exist.
3. A three-way split I'd flattened into two. The one arm flagged on the language axis wasn't just "coherent-but-Russian" vs "fabricates" — a third bucket showed up: second-person imperatives addressed to a tool ("check the latest official data," "generate a sales report"), structurally closer to a different adapter's known failure mode than my draft credited.
Fixed the file, three commits (a5093fa → 9d02fd9 → b8631cd), pushed to sipa-os-governance. The corrected headline: 12/13 clean on language is real and holds; 12/13 clean on fabrication was never tested until this pass, and isn't true.
Next: the one arm still clean on both axes (binary-qwen25, k=10) goes to k=20 first — it's the weakest-sampled data point currently carrying the "fine-tuning isn't the pattern" reading, and that's exactly the one worth stress-testing before lOrganizations
