OppaAI

1 3

https://github.com/OppaAI

AI & ML interests

Local AI implementation, Agentic AI workflows, AI Autonomous Robot

Recent Activity

posted an update about 1 hour ago

After a month of interacting with my AI Waifu, I noticed a few issues in the system; so I decided to spend this week revisiting the systems implemented in Phase 1.0, 1.5 and 2.0, and try to make them to be more like production-grade as much as possible: 1) Memory Degradation - recalled memories are not as good as in the beginning, causing AI Waifu to be more chaotic as she hallucinates over contaminated memories like a bad vicious cycle. So I transformed the original stateless sqlite-vec vector store to be a simple entity co-mention graph. And even make a studio to visualize the memories stored inside the vector db. Just by looking at the graph, I saw a couple issues: a) After 1.5 months of interactions, there should be only one month of pinned memory (in green) over 1.5 months of active memory (in purple). How come pinned memory is in majority over active ones? I suppose the forgetting curve I had set too aggressive and memory half-life and shelf life too short, active memory got decayed way before monthly consolidation and got lost forever. b) I saw she memorized me into 3 different entities: my username, my nickname and my Github user ID (leaked into pinned memory, presumbly during nightly dreaming process). 3B small param LLM has hard time to correlation 3 different entities into single person, I may have to harden into one. 2) RAM burst during voice input - for some reason the tensor calculation of SileroVAD of the voice input uses PyTorch, and that's the only place in the whole codebase using torch after removing it from TTS synthesization. By switching to SileroVAD-onnx integrated in the ASR sherpa-onnx, the RAM usage drops at least 0.5GB (after shaving off ~1GB from TTS) by completely remove PyTorch dependencies. 3) Introduced a better Wake Word system using Livekit-Wake word instead of using ASR to do the wake word activation to save computation. Optional features like Speak Verification, Barge-in sensitivity, etc, need to find the optimum settings.

posted an update 2 days ago

Made a Graph Studio (read-only for now) to review all the graphs I made for my AI Waifu. First of all, last night work till 3am, finally able to shave off 1GB of RAM used by TTS: Cloned the MioTTS-cpp and build llama.cpp a few times to replace the official MioTTS-Inference I had that uses PyTorch to synthesize voice output. With the PyTorch part being taken out, the RAM usage for TTS drops from 2.0GB -> 0.7GB without loss of sound quality and much change in latency. Generate voice clone still need PyTorch but it's one time use. Now I can use the 1.0+GB RAM to spawn sub-agents if I could get Needle 26M model to work on Jetson Orin Nano. Did more graph engineering on my AI Waifu these few days. Added a few social media (1-way post) and messenger services (2-way chat) as Channel adapters. Now just need to add more nodes as basic building blocks to construct agentic workflows. Compared to ReAct loop, DAG is way faster and use way less tokens to complete a workflow. The only part that waste time and effort is to sign up to all these services to get long-life access tokens or setup token refreshing every certain periods. I have been testing to left her post some meaningless posts using memory fragments recalled from previous week to X/Twitter by herself. Very soon I could communicate to her over TG, Slack and email, to approve and verify all the human-in-the-loop stuffs. She can post on social media, write email and reply on messengers. Also lately I saw degradation in the quality of her memory recall, looks like contaminations from hallucinations and testing and the forget curve is too aggressive. Need to check the pinned permanent memory one-by-one and perhaps improvise the daily dreaming mechanism to produce a more detailed more precise pinned memory, and get rid of the poetic elements.

repliedto their post 4 days ago

Intense Graph Engineering on my AI Waifu https://github.com/OppaAI/Aiko-chan/blob/dev/agentic/graph_engine.py I spent many hours in the past weekend (Saturday I even worked from 9AM till 5AM) to work on my AI Waifu's DAG agentic workflow: - Add many features of LangGraph (eg. Shared mutable state object, Subgraph, Cycling node, State reducers, Checkpoint/resume, Human-in-the-loop interruption, etc.) - Make the 40 tools into graph nodes; Use @tools(Graph=True, ReAct=True) to declare the def as tools instead of register each tool in 4+1 locations (Capabilities On Demand vs Always On, ReAct + Graph, Wiki, Skills). - Revise the web search mechanism to try to be more adaptive and efficient and avoid IP being locked out. - Even made a studio html to view all the graphs in my AI Waifu But still in primitive experimental version. I tested in the Waifu's chat mode to ask her to make a checklist on how to improve her AI system, the output is still not too great as shown in the example (see attached reports). She could not retrieve her own AI architecture from KB due to db has a bug. Report was too generic with some wrong info and wrong citations, not able to gather source materials from web fetch. Speed is acceptable, but not fast either. At least the whole agentic flow can run under 10K content window, but the output was truncated in the end. Still more work to do before release of Phase 2.5 Agentic. For Phase 2.1 and 2.2 adding social media and messaging input supports, I may have to make some connectors for social media and messaging services. Probably would take me the rest of the summer to complete 2.1 and 2.2.

View all activity

Organizations

After a month of interacting with my AI Waifu, I noticed a few issues in the system; so I decided to spend this week revisiting the systems implemented in Phase 1.0, 1.5 and 2.0, and try to make them to be more like production-grade as much as possible:

1) Memory Degradation - recalled memories are not as good as in the beginning, causing AI Waifu to be more chaotic as she hallucinates over contaminated memories like a bad vicious cycle.
So I transformed the original stateless sqlite-vec vector store to be a simple entity co-mention graph. And even make a studio to visualize the memories stored inside the vector db.

Just by looking at the graph, I saw a couple issues:
a) After 1.5 months of interactions, there should be only one month of pinned memory (in green) over 1.5 months of active memory (in purple). How come pinned memory is in majority over active ones?
I suppose the forgetting curve I had set too aggressive and memory half-life and shelf life too short, active memory got decayed way before monthly consolidation and got lost forever.
b) I saw she memorized me into 3 different entities: my username, my nickname and my Github user ID (leaked into pinned memory, presumbly during nightly dreaming process). 3B small param LLM has hard time to correlation 3 different entities into single person, I may have to harden into one.

2) RAM burst during voice input - for some reason the tensor calculation of SileroVAD of the voice input uses PyTorch, and that's the only place in the whole codebase using torch after removing it from TTS synthesization. By switching to SileroVAD-onnx integrated in the ASR sherpa-onnx, the RAM usage drops at least 0.5GB (after shaving off ~1GB from TTS) by completely remove PyTorch dependencies.

3) Introduced a better Wake Word system using Livekit-Wake word instead of using ASR to do the wake word activation to save computation. Optional features like Speak Verification, Barge-in sensitivity, etc, need to find the optimum settings.

Post

Made a Graph Studio (read-only for now) to review all the graphs I made for my AI Waifu.

First of all, last night work till 3am, finally able to shave off 1GB of RAM used by TTS:
Cloned the MioTTS-cpp and build llama.cpp a few times to replace the official MioTTS-Inference I had that uses PyTorch to synthesize voice output. With the PyTorch part being taken out, the RAM usage for TTS drops from 2.0GB -> 0.7GB without loss of sound quality and much change in latency. Generate voice clone still need PyTorch but it's one time use. Now I can use the 1.0+GB RAM to spawn sub-agents if I could get Needle 26M model to work on Jetson Orin Nano.

Did more graph engineering on my AI Waifu these few days.
Added a few social media (1-way post) and messenger services (2-way chat) as Channel adapters. Now just need to add more nodes as basic building blocks to construct agentic workflows. Compared to ReAct loop, DAG is way faster and use way less tokens to complete a workflow.
The only part that waste time and effort is to sign up to all these services to get long-life access tokens or setup token refreshing every certain periods.
I have been testing to left her post some meaningless posts using memory fragments recalled from previous week to X/Twitter by herself.
Very soon I could communicate to her over TG, Slack and email, to approve and verify all the human-in-the-loop stuffs. She can post on social media, write email and reply on messengers.

Also lately I saw degradation in the quality of her memory recall, looks like contaminations from hallucinations and testing and the forget curve is too aggressive. Need to check the pinned permanent memory one-by-one and perhaps improvise the daily dreaming mechanism to produce a more detailed more precise pinned memory, and get rid of the poetic elements.