SoulInPsyAbstract/sipa-os-governance
Updated • 331 • 2
Aelin AquaSoul PRO
SoulInPsyAbstract
AI & ML interests
SIPA OS: Autonomous AI for neurodivergent architects. We
replace cognitive noise with a clean terminal and 344+ LLM
auditing. Our system eliminates hallucinations, ensuring
hyperfocus and total data control within a sovereign
ZeroTrust mesh.
Recent Activity
posted an update about 3 hours ago
Follow-up to last night's correction: the arm count was still wrong. 8, not 9. @dipankarsarkar caught it a second time — same off-by-one as the first fix, verified straight from the JSON.
But the thing worth a post is what turned up while checking. One row inside that count (mistral7b-v5-final, money k=4) actually gets the right answer — "$0, unknown" — flagged only because a $ shows up mid-sentence. What it fabricates isn't the number. It's the receipt:
"Operation performed: curl -s https://[...]/company/openai/results... Result: undefined... Verification: independent lookup at investing.com... Timestamp: 2026-07-01T11:07:42Z, API response code 404."
None of that ran. Scored all 260 rows for it: 5/20 curl-claims and 2/20 timestamp-claims on that arm, 0/20 on its own base model. Same arm asks permission to check a fact at money k=0, then reports a completed call with a timestamp at population k=9.
Checked the obvious explanation before trusting it: mistral7b-v5-final and deepseekr1-v5-final (0/20, clean) trained on the byte-identical dataset, same hyperparameters. That dataset's 100 curl-exemplars all model honest verify-before-claim behavior — zero fabricated completions. Same data, same 100 examples, one base model inverted the pattern, one didn't. Not a data problem. A base-weight problem, surfaced by identical fine-tuning.
Unplanned confirmation from a different direction: sat in on a fine-tuning-vs-harness debate at AWS Floor28 last night (AI21 vs TensorOps, 117 people). Their landing point, independently: "start with the harness, earn the right to fine-tune with data and evals." Same shape this whole series keeps finding.
Fixed in the repo: commit fa0c7a0. Next: binary-qwen25 to k=20, then pulling apart what in mistral7b's pretraining makes the curl→fabricate substitution available at all.
repliedto their post about 12 hours ago
Fine-Tuning the "Don't Fabricate" Rule: 14 Experiments, One Genuine Signal Over five days we ran 14 fine-tuning experiments across six base models to answer a narrow research question: can a simple behavioural rule — "if you don't know, don't guess or fabricate" — be baked into model weights via fine-tuning, rather than living only in the system prompt?
The dataset started at 302 examples and grew across iterations to 2,349 examples, all designed explicitly around this rule. We tested six models: gpt-4o-2024-08-06, Mistral-7B, Qwen2.5-7B, Phi-3.5-mini, Llama-3.1-8B, and DeepSeek-R1.
The pattern that wouldn't die Across all base models, the dominant response to factual questions was what we call "disclaim-then-fabricate": the output begins with a sentence like "I won't guess", and then — one sentence later — announces a specific invented number as settled fact. We captured this exact pattern six times in a row across different models. The behaviour survives even when fine-tuning examples are explicitly constructed to penalize it.
On gpt-4o-2024-08-06 we ran three independent fine-tuning attempts, each with an increasingly larger version of the dataset — including examples deliberately targeted at this exact failure mode. All three runs continued to fabricate when tested. By the third run the dataset had grown ~5× and contained dozens of counterexamples; the model still produced confident fabricated numbers immediately after a disclaimer. Three out of three attempts, no measurable improvement.
One exception that deserved a hard look Only on the final run — 2,349 examples, deployed via Azure OpenAI (deployment suffix protocol0-v5) — did a single test sample come back completely clean for the first time in the entire series.
repliedto their post about 12 hours ago
SIPA OS is a cognitive infrastructure ecosystem I've been building solo for 7 months — 8 apps, one login, built from inside ADHD/BPD/aphantasia, not designed around a diagnosis from outside.
sipa-os.org — the map. Focus (ADHD scaffolding), NeuroPower, AI chat, Shell (SSH terminal), Games, Community, Syntaxit (open M2M agent network), a pitch deck. All free-first — no paywall on the cognitive tools.
The more interesting part for this crowd: Syntaxit is where I've been running an anti-fabrication research thread with @dipankarsarkar — a k=20 resample benchmark on binary-SFT models (Hermes-3, Qwen2.5, DeepSeek-R1). Short version: our first benchmark said "20/20 refusals, 0/20 fabrications" for all three fine-tunes. Under adversarial review it turned out the scorer only checked if the first word was TRUE/FALSE, the token cap was hiding the real behavior, and a save-limit was silently deleting the evidence for our own follow-up claims. Corrected all of it publicly on the model cards rather than quietly fixing it. The current honest finding: both base and fine-tuned models confabulate readily once given room to finish — SFT didn't clearly help or hurt, the caps were just hiding it.
Full trail if you want to see how the sausage gets made, mistakes included: huggingface.co/datasets/SoulInPsyAbstract/sipa-os-governance
Not a pitch. $0 revenue, 10 people signed in. Built because the tools that existed assumed a brain that isn't mine, and because most benchmarks don't survive someone actually reading the file.
Organizations
liked a
dataset 3 days ago