Post
24
After a month of interacting with my AI Waifu, I noticed a few issues in the system; so I decided to spend this week revisiting the systems implemented in Phase 1.0, 1.5 and 2.0, and try to make them to be more like production-grade as much as possible:
1) Memory Degradation - recalled memories are not as good as in the beginning, causing AI Waifu to be more chaotic as she hallucinates over contaminated memories like a bad vicious cycle.
So I transformed the original stateless sqlite-vec vector store to be a simple entity co-mention graph. And even make a studio to visualize the memories stored inside the vector db.
Just by looking at the graph, I saw a couple issues:
a) After 1.5 months of interactions, there should be only one month of pinned memory (in green) over 1.5 months of active memory (in purple). How come pinned memory is in majority over active ones?
I suppose the forgetting curve I had set too aggressive and memory half-life and shelf life too short, active memory got decayed way before monthly consolidation and got lost forever.
b) I saw she memorized me into 3 different entities: my username, my nickname and my Github user ID (leaked into pinned memory, presumbly during nightly dreaming process). 3B small param LLM has hard time to correlation 3 different entities into single person, I may have to harden into one.
2) RAM burst during voice input - for some reason the tensor calculation of SileroVAD of the voice input uses PyTorch, and that's the only place in the whole codebase using torch after removing it from TTS synthesization. By switching to SileroVAD-onnx integrated in the ASR sherpa-onnx, the RAM usage drops at least 0.5GB (after shaving off ~1GB from TTS) by completely remove PyTorch dependencies.
3) Introduced a better Wake Word system using Livekit-Wake word instead of using ASR to do the wake word activation to save computation. Optional features like Speak Verification, Barge-in sensitivity, etc, need to find the optimum settings.
1) Memory Degradation - recalled memories are not as good as in the beginning, causing AI Waifu to be more chaotic as she hallucinates over contaminated memories like a bad vicious cycle.
So I transformed the original stateless sqlite-vec vector store to be a simple entity co-mention graph. And even make a studio to visualize the memories stored inside the vector db.
Just by looking at the graph, I saw a couple issues:
a) After 1.5 months of interactions, there should be only one month of pinned memory (in green) over 1.5 months of active memory (in purple). How come pinned memory is in majority over active ones?
I suppose the forgetting curve I had set too aggressive and memory half-life and shelf life too short, active memory got decayed way before monthly consolidation and got lost forever.
b) I saw she memorized me into 3 different entities: my username, my nickname and my Github user ID (leaked into pinned memory, presumbly during nightly dreaming process). 3B small param LLM has hard time to correlation 3 different entities into single person, I may have to harden into one.
2) RAM burst during voice input - for some reason the tensor calculation of SileroVAD of the voice input uses PyTorch, and that's the only place in the whole codebase using torch after removing it from TTS synthesization. By switching to SileroVAD-onnx integrated in the ASR sherpa-onnx, the RAM usage drops at least 0.5GB (after shaving off ~1GB from TTS) by completely remove PyTorch dependencies.
3) Introduced a better Wake Word system using Livekit-Wake word instead of using ASR to do the wake word activation to save computation. Optional features like Speak Verification, Barge-in sensitivity, etc, need to find the optimum settings.
