- cross-posted to:
- privacy@programming.dev
- cross-posted to:
- privacy@programming.dev
Hello!
As a handsome local AI enjoyer™ you’ve probably noticed one of the big flaws with LLMs:
It lies. Confidently. ALL THE TIME.
(Technically, it “bullshits” - https://link.springer.com/article/10.1007/s10676-024-09775-5
I’m autistic and extremely allergic to vibes-based tooling, so … I built a thing. Maybe it’s useful to you too.
The thing: llama-conductor
llama-conductor is a router that sits between your frontend (OWUI / SillyTavern / LibreChat / etc) and your backend (llama.cpp + llama-swap, or any OpenAI-compatible endpoint). Local-first (because fuck big AI), but it should talk to anything OpenAI-compatible if you point it there (note: experimental so YMMV).
I tried to make a glass-box that makes the stack behave like a deterministic system, instead of a drunk telling a story about the fish that got away.
TL;DR: “In God we trust. All others must bring data.”
Three examples:
1) KB mechanics that don’t suck (1990s engineering: markdown, JSON, checksums)
You keep “knowledge” as dumb folders on disk. Drop docs (.txt, .md, .pdf) in them. Then:
>>attach <kb>— attaches a KB folder>>summ new— generatesSUMM_*.mdfiles with SHA-256 provenance baked in- `>> moves the original to a sub-folder
Now, when you ask something like:
“yo, what did the Commodore C64 retail for in 1982?”
…it answers from the attached KBs only. If the fact isn’t there, it tells you - explicitly - instead of winging it. Eg:
The provided facts state the Commodore 64 launched at $595 and was reduced to $250, but do not specify a 1982 retail price. The Amiga’s pricing and timeline are also not detailed in the given facts.
Missing information includes the exact 1982 retail price for Commodore’s product line and which specific model(s) were sold then. The answer assumes the C64 is the intended product but cannot confirm this from the facts.
Confidence: medium | Source: Mixed
No vibes. No “well probably…”. Just: here’s what’s in your docs, here’s what’s missing, don’t GIGO yourself into stupid.
And when you’re happy with your summaries, you can:
>>move to vault— promote those SUMMs into Qdrant for the heavy mode.
2) Mentats: proof-or-refusal mode (Vault-only)
Mentats is the “deep think” pipeline against your curated sources. It’s enforced isolation:
- no chat history
- no filesystem KBs
- no Vodka
- Vault-only grounding (Qdrant)
It runs triple-pass (thinker → critic → thinker). It’s slow on purpose. You can audit it. And if the Vault has nothing relevant? It refuses and tells you to go pound sand:
FINAL_ANSWER:
The provided facts do not contain information about the Acorn computer or its 1995 sale price.
Sources: Vault
FACTS_USED: NONE
[ZARDOZ HATH SPOKEN]
Also yes, it writes a mentats_debug.log, because of course it does. Go look at it any time you want.
The flow is basically: Attach KBs → SUMM → Move to Vault → Mentats. No mystery meat. No “trust me bro, embeddings.”
3) Vodka: deterministic memory on a potato budget
Local LLMs have two classic problems: goldfish memory + context bloat that murders your VRAM.
Vodka fixes both without extra model compute. (Yes, I used the power of JSON files to hack the planet instead of buying more VRAM from NVIDIA).
!!stores facts verbatim (JSON on disk)??recalls them verbatim (TTL + touch limits so memory doesn’t become landfill)- CTC (Cut The Crap) hard-caps context (last N messages + char cap) so you don’t get VRAM spikes after 400 messages
So instead of:
“Remember my server is 203.0.113.42” → “Got it!” → [100 msgs later] → “127.0.0.1 🥰”
you get:
!! my server is 203.0.113.42?? server ip→ 203.0.113.42 (with TTL/touch metadata)
And because context stays bounded: stable KV cache, stable speed, your potato PC stops crying.
There’s more (a lot more) in the README, but I’ve already over-autism’ed this post.
TL;DR:
If you want your local LLM to shut up when it doesn’t know and show receipts when it does, come poke it:
- Primary (Codeberg): https://codeberg.org/BobbyLLM/llama-conductor
- Mirror (GitHub): https://github.com/BobbyLLM/llama-conductor
PS: Sorry about the AI slop image. I can’t draw for shit.
PPS: A human with ASD wrote this using Notepad++. If it the formatting is weird, now you know why.
You must log in or # to comment.
THIS IS AWESOME!!! I’ve been working on using an obsidian vault and a podman ollama container to do something similar, with VSCodium + continue as middleware. But this! This looks to me like it is far superior to what I have cobbled together.
I will study your codeberg repo, and see if I can use your conductor with my ollama instance and vault program. I just registered at codeberg, if I make any progress I will contact you there, and you can do with it what you like.
On an unrelated note, you can download wikipedia. Might work well in conjunction with your conductor.
Please enjoy :) Hope it’s of use to you!
EDIT: Please don’t yeet all of wikipedia into it. It will die. And you will be sad.
Super interesting build
And if programming doesn’t pan out please start writing for a magazine, love your style (or was this written with your AI?)
Once again: I am a meat popsicle (with ASD), not AI. All errors and foibles are mine :)
meat popsicle
( ͡° ͜ʖ ͡°)
Anyway, the other person is right. Your writing style is great !
I successfully read your whole post and even the README. Probably the random outbursts grabbed my attention back to te text.
Anyway version 2, this Is a very cool idea ! I cannot wait to either :
- incorporate it to my workflows
- let it sit in a tab to never be touched ever again
- tgeoryceaft, do tests and request features so much as to burnout
Last but not least, thank you for not using github as your primary repo
Hmm. One of those things is not like the other, one of those things just isn’t the same…
About the random outburst: caused by TOO MUCH FUCKING CHATGPT WASTING HOURS OF MY FUCKING LIFE, LEADING ME DOWN BLIND ALLEYWAYS, YOU FUCKING PIEC…
…sorry, sorry…
Anyway, enjoy. Don’t spam my Github inbox plz :)
Don’t spam my Github inbox plz
I can spam your codeberg’s then ? :)
About the random outburst: caused by TOO MUCH FUCKING CHATGPT WASTING HOURS OF MY FUCKING LIFE, LEADING ME DOWN BLIND ALLEYWAYS, YOU FUCKING PIEC… …sorry, sorry…
Understandable, have a great day.
Don’t spam my Codeberg either.
Just send nudes.
In ASCII format.
By courier pigeon
Awesome work. And I agree that we can have good and responsible AI (and other tech) if we start seeing it for what it is and isn’t, and actually being serious about addressing its problems and limitations. It’s projects like yours that can demonstrate pathways toward achieving better AI.
I’m probably going to give this a try, but I think you should make it clearer for those who aren’t going to dig through the code that it’s still LLMs all the way down and can still have issues - it’s just there are LLMs double-checking other LLMs work to try to find those issues. There are still no guarantees since it’s still all LLMs.
Fair point on setting expectations, but this isn’t just LLMs checking LLMs. The important parts are non-LLM constraints.
The model never gets to “decide what’s true.” In KB mode it can only answer from attached files. Don’t feed it shit and it won’t say shit.
In Mentats mode it can only answer from the Vault. If retrieval returns nothing, the system forces a refusal. That’s enforced by the router, not by another model.
The triple-pass (thinker → critic → thinker) is just for internal consistency and formatting. The grounding, provenance, and refusal logic live outside the LLM.
So yeah, no absolute guarantees (nothing in this space has those), but the failure mode is “I don’t know / not in my sources, get fucked” not “confidently invented gibberish.”
I haven’t tried this tool specifically, but I do on occasion ask both Gemini and ChatGPT’s search-connected models to cite sources when claiming stuff and it doesn’t seem to even slightly stop them bullshitting and claiming a source says something that it doesn’t.
I want to believe you, but that would mean you solved hallucination.
Either:
A) you’re lying
B) you’re wrong
C) KB is very small
D) None of the above.
I didn’t “solve hallucination”. I changed the failure mode. The model can still hallucinate internally. The difference is it’s not allowed to surface claims unless they’re grounded in attached sources.
If retrieval returns nothing relevant, the router forces a refusal instead of letting the model free-associate. So the guarantee isn’t “the model is always right.”
The guarantee is “the system won’t pretend it knows when the sources don’t support it.” That’s it. That’s the whole trick.
KB size doesn’t matter much here. Small or large, the constraint is the same: no source, no claim. GTFO.
That’s a control-layer property, not a model property. If it helps: think of it as moving from “LLM answers questions” to “LLM summarizes evidence I give it, or says ‘insufficient evidence.’”
Again, that’s the whole trick.
You don’t need to believe me. In fact, please don’t. Test it.
I could be wrong…but if I’m right (and if you attach this to a non-retarded LLM), then maybe, just maybe, this doesn’t suck balls as much as you think it might.
Maybe it’s even useful to you.
I dunno. Try it?
So… Rag with extra steps and rag summarization? What about facts that are not rag retrieval?
Parts of this are RAG, sure
RAG parts:
- Vault / Mentats is classic retrieval + generation.
- Vector store = Qdrant
- Embedding and reranker
So yes, that layer is RAG with extra steps.
What’s not RAG -
KB mode (filesystem SUMM path)
This isn’t vector search. It’s deterministic, file-backed grounding. You attach folders as needed. The system summarizes and hashes docs. The model can only answer from those summaries in that mode. There’s no semantic retrieval step. It can style and jazz around the answer a little, but the answer is the answer is the answer.
If the fact isn’t in the attached KB, the router forces a refusal. Put up or shut up.
Vodka (facts memory)
That’s not retrieval at all, in the LLM sense. It’s verbatim key-value recall.
- JSON on disk
- Exact store (!!)
- Exact recall (??)
Again, no embeddings, no similarity search, no model interpretation.
“Facts that aren’t RAG”
In my set up, they land in one of two buckets.
-
Short-term / user facts → Vodka. That for things like numbers, appointments, lists, one-off notes etc. Deterministic recall, no synthesis.
-
Curated knowledge → KB / Vault. Things you want grounded, auditable, and reusable.
In response to the implicit “why not just RAG then”
Classic RAG failure mode is: retrieval is fuzzy → model fills gaps → user can’t tell which part came from where.
The extra “steps” are there to separate memory from knowledge, separate retrieval from synthesis and make refusal a legal output, not a model choice.
So yeah; some of it is RAG. RAG is good. The point is this system is designed so not everything of value is forced through a semantic search + generate loop. I don’t trust LLMs. I am actively hostile to them. This is me telling my LLM to STFU and prove it, or GTFO. I know that’s a weird way to operate maybe (advesarial, assume worst, engineer around issue) but that’s how ASD brains work.
The system summarizes and hashes docs. The model can only answer from those summaries in that mode
Oh boy. So hallucination will occur here, and all further retrievals will be deterministically poisoned?
Woof, after reading your “contributions” here, are you this fucking insufferable IRL or do you keep it behind a keyboard?
Goddamn. I’m assuming you work in tech in some capacity? Shout-out to anyone unlucky enough to white-knuckle through a workday with you, avoiding an HR incident would be a legitimate challenge, holy fuck.
Okay pardon the double comment, but I now have no choice but to set this up after reading your explainations. Doing what TRILLIONS of dollars hasn’t cooked up yet… I hope you’re ready by whatever means you deam, when someone else “invents” this
It’s copyLEFT (AGPL-3.0 license). That means, free to share, copy, modify…but you can’t roll a closed source version of it and sell it for profit.
In any case, I didn’t build this to get rich (fuck! I knew I forgot something).
I built this to try to unfuck the situation / help people like me.
I don’t want anything for it. Just maybe a fist bump and an occasional “thanks dude. This shit works amazing”
Doing gods work
o7
Friendship drive activated.
<3
This is awesome. Definitely gonna dig into this later.
I wouldn’t know how to get this going, but I very much enjoyed reading it and your comments and think that it looks like a great project. 👍
(I mean, as a fellow autist I might be able to hyperfocus on it for a while, but I’m sure that the ADHD would keep me from finishing to go work on something else. 🙃)
Ah - ASD, ADHD and Lemmy. You’re a triple threat, Harry! :)
Glad if it was entertaining, if even a little!
Very impressive! Do you have benchmark to test the reliability? A paper would be awesome to contribute to the science.
Just bush-league ones I did myself, that have no validation or normative values. Not that any of the LLM benchmarks seem to have those either LOL
I’m open to ideas, time wiling. Believe it or not, I’m not a code monkey. I do this shit for fun to get away from my real job
I understand, no idea on how to do it. I heard about SWE‑Bench‑Lite that seems to focus on real-world usage. Maybe try to contact “AI Explained” on YT, he’s the best IMO. Your solution might be novel or not but he might help you figuring that. If it is indeed novel, it might be worth it to share it with the larger community. Of course, I totally get that you might not want to do any of that. Thank you for your work!
I really need this. Each time I try messing with GPT4All’s “reasoning” model, it pisses me off. I’m selective on my inputs, low temperature, local docs, and it’ll tell me things like tension matters for a coil’s magnetic field. Oh and it spits out what I assume is unformatted LATEX so if anyone has an interface/stack recommendation please let me know
This seems astonishingly more useful than the current paradigm, this is genuinely incredible!
I mean, fellow Autist here, so I guess I am also… biased towards… facts…
But anyway, … I am currently uh, running on Bazzite.
I have been using Alpaca so far, and have been successfully running Qwen3 8B through it… your system would address a lot of problems I have had to figurr out my own workarounds for.
I am guessing this is not available as a flatpak, lol.
I would feel terrible to ask you to do anything more after all of this work, but if anyone does actually set up a podman installable container for this that actually properly grabs all required dependencies, please let me know!
Indeed. And have you heard? That makes the normies think were clankers (bots). How delightful.
Re: the Linux stuff…please, if someone can do that, please do. I have no idea how to do that. I can figure it out but making it into a “one click install” git command took several years off my life.
Believe it or not, I’m not actually a IT / CS guy. My brain just decided to latch onto this problem one day 6 months ago and do an autism.
I’m 47 and I still haven’t learned how to operate this vehicle…and my steering is getting worse, not better, with age.
Oh I entirely believe you.
Hell hath no wrath like an annoyed high functioning autist.
I’ve … had my own 6 month black out periods where I came up with something extremely comprehensive and ‘neat’ before.
Seriously, bootstrapping all this is incredibly impressive.
I would… hope that you can find collaborators, to keep this thing alive in the event you get into a car accident (metaphorical or literal), or, you know, are completely burnt out after this.
… but yeah, it is… yet another immensely ironic aspect of being autistic that we’ve been treated and maligned as robots our whole lives, and then when the normies think they’ve actually built the AI from sci fi, no, turns out its basically extremely talented at making up bullshit and fudging the details and being a hypocrite, which… appalls the normies when they have to look into a hyperpowered mirror of themselves.
And then, of course, to actually fix this, its some random autist no one has ever heard of (apologies if you are famous and i am unaware of this), who is putting in an enormous of effort, that… most likely, will not be widely recognized.
… fucking normies man.
Based AF. Can anyone more knowledgeable explain how it works? I am not able to understand.
Hell yes I can explain. What would you like to know.
As I understand it, it corrects the output of LLMs. If so, how does it actually work?
Good question.
It doesn’t “correct” the model after the fact. It controls what the model is allowed to see and use before it ever answers.
There are basically three modes, each stricter than the last. The default is “serious mode” (governed by serious.py). Low temp, punishes chattiness and inventiveness, forces it to state context for whatever it says.
Additionally, Vodka (made up of two sub-modules - “cut the crap” and “fast recall”) operate at all times. Cut the crap trims context so the model only sees a bounded, stable window. You can think of it like a rolling, summary of what’s been said. That summary is not LLM generated summary either - it’s concatenation (dumb text matching), so no made up vibes.
Fast recall OTOH stores and recalls facts verbatim from disk, not from the model’s latent memory.
It writes what you tell it to a text file and then when you ask about it, spits it back out verbatim ((!! / ??)
And that’s the baseline
In KB mode, you make the LLM answer based on the above settings + with reference to your docs ONLY (in the first instance).
When you >>attach <kb>, the router gets stricter again. Now the model is instructed to answer only from the attached documents.
Those docs can even get summarized via an internal prompt if you run >>summ new, so that extra details are stripped out and you are left with just baseline who-what-where-when-why-how.
The SUMM_*.md file come SHA-256 provenance, so every claim can be traced back to a specific origin file (which gets moved to a subfolder)
TL;DR: If the answer isn’t in the KB, it’s told to say so instead of guessing.
Finally, Mentats mode (Vault / Qdrant). This is the “I am done with your shit" path.
It’s all of the three above PLUS a counter-factual sweep.
It runs ONLY on stuff you’ve promoted into the vault.
What it does is it takes your question and forms in in a particular way so that all of the particulars must be answered in order for there to BE an answer. Any part missing or not in context? No soup for you!
In step 1, it runs that past the thinker model. The answer is then passed onto a “critic” model (different llm). That model has the job of looking at the thinkers output and say “bullshit - what about xyz?”.
It sends that back to the thinker…who then answers and provides final output. But if it CANNOT answer the critics questions (based on the stored info?). It will tell you. No soup for you, again!
TL;DR:
The “corrections” happen by routing and constraint. The model never gets the chance to hallucinate in the first place, because it literally isn’t shown anything it’s not allowed to use. Basic premise - trust but verify (and I’ve given you all the tools I could think of to do that).
Does that explain it better? The repo has a FAQ but if I can explain anything more specifically or clearly, please let me know. I built this for people like you and me.
That is much clearer. Thank you for making this. It actually makes LLMs useful with much lesser downsides.
God, I hope so. Else I just pissed 4 months up the wall and shouted a lot of swears at my monitor for nada :)
Let me know if it works for you
Will do.
Hey Human,
Thank you!
Meep Morp
I have no remarks, just really amused with your writing in your repo.
Going to build a Docker and self host this shit you made and enjoy your hard work.
Thank you for this!
Thank you <3
Please let me know how it works…and enjoy the >>FR settings. If you’ve ever wanted to trolled by Bender (or a host of other 1990s / 2000s era memes), you’ll love it.
Same sentiment. Tonight it run on my systems XD.
