No Body

You're absolutely correct. For me, choosing DoRA over LoRA dramatically reduces the size of the model I can train locally on my 4090 - by creating a bnb nf4 of a model and then loading that into my trainer I have been able to LoRA train up to 24B models on my 24gb of VRAM but I can't get nearly as large with DoRA. But since I'm only training on a smaller dataset - I use NobodyExistsOnTheInternet/ToxicQAFinal - I don't mind renting an A100 for $1.50/hr on RunPod for a short run. I'm sure the costs would be a lot higher if I weren't strictly limiting myself to sub-100B models. But also since I'm only focusing on a small adjustment to direction, I can take some of the edge off by using a lower-rank adapter like 16, whereas with LoRA training I prefer a higher rank with rank-stabilized LoRA.

So far my only proof-of-concepts have been variants of Gemma 12B and 27B - Nabbers1999/gemma-3-27b-it-abliterated-refined-vision - which I'm relying on community feedback to counter my own personal bias, but the few people on Reddit who have given me their feedback have suggested that they feel the model is less "cranky" about providing helpful responses to toxic prompts. The main complaint that started me down this path is, I was hearing in the Reddit threads about abliterated models being compliant but frequently still being opinionated when presented with toxic prompts.