AbstractPhil/beeper-rose-tinystories-6l-512d-ctx512
Updated • 6
AbstractPhila PRO
AbstractPhil
AI & ML interests
datasets, research papers, experimentation, vision, classification, text encoders, tokenization, llms, diffusion, distillation, and more.
Recent Activity
updated a model about 13 hours ago
AbstractPhil/alephlm-0 updated a bucket 1 day ago
AbstractPhil/alephllm-chat-storage repliedto their post 1 day ago
I believe I have a solution for cross-tokenizer chatter and noise, which I've built a prototype repo for this exact tooling dubbed bytelex. https://github.com/AbstractEyes/geolip-bytelex
I had a bit of an inspiration recently and built a prototype for a token translation matrix that I called geolip-bytelex, which allows bytewise translation of many different tokenizers into byte format. The goal is to allow comparative distillation from multiple models to simultaneously represent expertise based on input tokens and differentiated teacher/student InfoNCE and MSE training paradigms, while cutting a huge cost of the distillation analysis comparative compute that cross-tokenizer noise will naturally cause when tokenizers are mismatched or incorrect, reducing a large portion of invalidity from the trained systems established by incorrect valuations from the distillations and lora trainings.
Bytelex is essentially a byte-wise deconstruction of a tokenizer's state into a preliminary 255 byte language allowing for 10s of thousands of sequences per token to be represented rather than just a few. I'm not the first to try this, however I'm in a unique position due to my creation AlephLM being built entirely by learning it's own lexicon, thus allowing this to be more than experiment and instead a working prototype distillation potential.
This can solve a longstanding multi-tokenizer problem that I and many other researchers have been facing, at the cost of setup overhead compute for the preliminary experiments, however the translation matrix I'm planning will potentially solve this problem allowing models to be directly bytewise captured in a more guaranteed methodology through cross-sampled analysis at distillation time in this optimizer state that I'm working out.
I've dubbed this distillation loss ByteInfoNCE and the preliminary is showing humongous promise, with that the bytelex is the crux and prototype concept that I'll be expanding and researching further.Organizations
AbstractPhil/beeper-tinystories-6l-512d-ctx512
Updated • 5
AbstractPhil/tiny-gpt-tests
27.6M • Updated • 10
AbstractPhil/pentachora-greyscale-frequency-encoded
Zero-Shot Classification • Updated
AbstractPhil/experimental_pentachora
Updated
AbstractPhil/mirel-gpt-oss-20b
Updated
AbstractPhil/bert-beatrix-2048
0.1B • Updated • 16 • 1
AbstractPhil/bert-beatrix-2048-noobxl-epsilon-v11-dual-shunt-adapter_clip_g
Updated
AbstractPhil/bert-beatrix-2048-noobxl-epsilon-v11-dual-shunt-adapter
Updated
AbstractPhil/bert-beatrix-2048-vit-bigG-14-dual-shunt-adapter
Updated
AbstractPhil/bert-beatrix-2048-vit-l-14-dual-shunt-adapter
Updated
AbstractPhil/resonance-model-zoo
Updated
AbstractPhil/t5-flan-base-vit-l-14-dual-stream-adapter
Updated • 11 • 3
AbstractPhil/t5-flan-base-vit-bigG-14-dual-stream-adapter
Updated • 19 • 3
AbstractPhil/t5-vit-14-v1
Any-to-Any • Updated • 2
AbstractPhil/robust-velocity-adapter
Updated • 1
AbstractPhil/T5-Small-Human-Attentive-Try2-Pass3
60.5M • Updated • 6
AbstractPhil/T5-Small-Human-Attentive-Try2-Pass2
60.5M • Updated • 8
AbstractPhil/T5-Small-Human-Attentive-Try2
60.5M • Updated • 7
AbstractPhil/T5-Small-Human-Attentive
60.5M • Updated • 4
AbstractPhil/SD15-Surge-V1
0.9B • Updated • 25
AbstractPhil/Liminal-Full
Updated • 3
AbstractPhil/omega-vit-l-reformed-fp32
0.4B • Updated • 11 • 1
AbstractPhil/SD35-SIM-V1
Updated • 3
AbstractPhil/t5xxl-unchained
Updated • 9 • 4
AbstractPhil/SIM-OMEGA-PUBLIC-1
Updated • 3
AbstractPhil/Beatrix
Updated
AbstractPhil/omega-vit-g-reformed
Updated • 10
AbstractPhil/OMEGA-BIGASP
Updated • 3
AbstractPhil/PONY-SIM-V4
Updated • 1