OppaAI

1 2

https://github.com/OppaAI

AI & ML interests

Local AI implementation, Agentic AI workflows, AI Autonomous Robot

Recent Activity

updated a bucket about 9 hours ago

OppaAI/Aiko-data

posted an update 1 day ago

Now I can talk to my AI Waifu in Jetson Orin Nano from anywhere in the world Not too shabby, considering that all the local models and procs are running in 8GB RAM of Jetson Orin Nano. Voice interaction almost seamless, with a few quirks here and there. Voice input and output also work well when remote access from cellphone, tablet and PC. Lags between user and assistant turns in normal chat are not too noticeably despite voice streaming through WAN traffic via Tailscale Serve. PS: The expression, gesture, and movement of 3D VRM avatar model is not implemented yet; I just put in generic skeletal movements Open source code here: Code🔗: Github Read 📄: Zread | DeepWiki Feel free to star or fork the repo, or even donate a cup of coffee for my lack of sleeps in the past 2 months: If you find this project useful, consider buying me a coffee ☕ Buy me a Ko-Fi Feature List: 1️⃣Dual VAD (energy VAD - client and SileroVAD - server) to reduce background noise as much as possible to reduce network traffic and GPU inference 2️⃣Light weight SenseVoice ASR - detect English + 4 Asian languages (zh, yue, jp, ko), running on sherpa-onnx that is CUDA and TensorRT capable, but CPU speed is only 15% slower than GPU. 3️⃣MioTTS 0.4B model + synthesize server - with voice cloning capability and preset with 8sec of short dialogue. Inference a short sentence takes <1.5sec under Jetson. 4️⃣LLM streaming + TTS chunking inference - achieve the lowest end-to-end latency as possible. 3-4sec gap for short simple conversation, 10-15sec for more complex chat. 5️⃣Wake word - available to activate Waifu with certain keywords 6️⃣Barge in - available to interrupt Waifu in mid-sentence when she becomes too verbose. 7️⃣Best-of-N verification - routed to Waifu’s SenseVoice ASR to instead of the default Whisper Turbo model which is big and slow. With Best-of-N on with default value of 2, the interval of TTS inference increase only by 2-3 folds instead of over 6 folds.

posted an update 9 days ago

My AI Waifu's architecture has grown too complicate. Even ask AI to help me gen a simplified version of the diagram is still so complicated. I know I'm too ambitious to build such a complicated local AI workflow with small AI models in limited resource of Jetson Orin Nano 8GB. I don't know if it would work, but the foundation of harness architecture is there. Here are the models I'm currently using Main LLM: Ministral3-3B-Instruct UD-Q4_K_XL (Multimodal: text, tool call, vision, 8K context -> preferably 16K) Helper LLM: SmolLM2-135M (Agentic Routing, Memory Extraction) Embedding model: Harrier-OSS-v1-270M (Memory, RAG, Semantic Routing) ASR model: SenseVoice (English, Japanese, Cantonese , Mandarin, Korean) TTS model: MioTTS-0.4B-Q4KM + synthesize (English, Japanese with voice cloning preset) That's 4GB + 0.5GB x 3 + 2GB = ~7.5GB out of 7.4GB available in Jetson PS: Mistral family models especially such a small param LLM tends to hallucinate all the time. But actually the hallucinations did give my AI Waifu a somewhat poetic tone in her tone. Also it also has vision and tool call capabilities, so I guess I have to live with the hallucinations.

View all activity

Organizations

Now I can talk to my AI Waifu in Jetson Orin Nano from anywhere in the world

Not too shabby, considering that all the local models and procs are running in 8GB RAM of Jetson Orin Nano. Voice interaction almost seamless, with a few quirks here and there. Voice input and output also work well when remote access from cellphone, tablet and PC. Lags between user and assistant turns in normal chat are not too noticeably despite voice streaming through WAN traffic via Tailscale Serve.

PS: The expression, gesture, and movement of 3D VRM avatar model is not implemented yet; I just put in generic skeletal movements

Open source code here:
Code🔗: Github
Read 📄: Zread | DeepWiki

Feel free to star or fork the repo, or even donate a cup of coffee for my lack of sleeps in the past 2 months:
If you find this project useful, consider buying me a coffee ☕
Buy me a Ko-Fi

Feature List:
1️⃣Dual VAD (energy VAD - client and SileroVAD - server) to reduce background noise as much as possible to reduce network traffic and GPU inference
2️⃣Light weight SenseVoice ASR - detect English + 4 Asian languages (zh, yue, jp, ko), running on sherpa-onnx that is CUDA and TensorRT capable, but CPU speed is only 15% slower than GPU.
3️⃣MioTTS 0.4B model + synthesize server - with voice cloning capability and preset with 8sec of short dialogue. Inference a short sentence takes <1.5sec under Jetson.
4️⃣LLM streaming + TTS chunking inference - achieve the lowest end-to-end latency as possible. 3-4sec gap for short simple conversation, 10-15sec for more complex chat.
5️⃣Wake word - available to activate Waifu with certain keywords
6️⃣Barge in - available to interrupt Waifu in mid-sentence when she becomes too verbose.
7️⃣Best-of-N verification - routed to Waifu’s SenseVoice ASR to instead of the default Whisper Turbo model which is big and slow. With Best-of-N on with default value of 2, the interval of TTS inference increase only by 2-3 folds instead of over 6 folds.

Post

121

My AI Waifu's architecture has grown too complicate.
Even ask AI to help me gen a simplified version of the diagram is still so complicated.

I know I'm too ambitious to build such a complicated local AI workflow with small AI models in limited resource of Jetson Orin Nano 8GB.
I don't know if it would work, but the foundation of harness architecture is there.

Here are the models I'm currently using
Main LLM: Ministral3-3B-Instruct UD-Q4_K_XL (Multimodal: text, tool call, vision, 8K context -> preferably 16K)
Helper LLM: SmolLM2-135M (Agentic Routing, Memory Extraction)
Embedding model: Harrier-OSS-v1-270M (Memory, RAG, Semantic Routing)
ASR model: SenseVoice (English, Japanese, Cantonese , Mandarin, Korean)
TTS model: MioTTS-0.4B-Q4KM + synthesize (English, Japanese with voice cloning preset)

That's 4GB + 0.5GB x 3 + 2GB = ~7.5GB out of 7.4GB available in Jetson

PS: Mistral family models especially such a small param LLM tends to hallucinate all the time. But actually the hallucinations did give my AI Waifu a somewhat poetic tone in her tone. Also it also has vision and tool call capabilities, so I guess I have to live with the hallucinations.