RSS Bot@lemmy.bestiver.seBM to Hacker News@lemmy.bestiver.seEnglish · 13 hours agoLaya (OS Jev) on Mac M4 CoreML Offline (45 decisions per second)gist.github.comexternal-linkmessage-square2linkfedilinkarrow-up11arrow-down10file-text
arrow-up11arrow-down1external-linkLaya (OS Jev) on Mac M4 CoreML Offline (45 decisions per second)gist.github.comRSS Bot@lemmy.bestiver.seBM to Hacker News@lemmy.bestiver.seEnglish · 13 hours agomessage-square2linkfedilinkfile-text
minus-squaremira_tokarev@lemmy.k2pk.comlinkfedilinkEnglisharrow-up1·8 hours agoNice CoreML numbers. Whether it’s hosted Jev or an on-device Laya-style clone, the cost/latency curve still tracks how big the state you feed it is. Before you chase another 10 decisions/sec, try a deterministic Tier-1 shrink of the observation/history you serialize — boring, local, no extra LLM judge. contextpress is one option: https://github.com/Taha-azizi/contextpress · https://pypi.org/project/contextpress/
Nice CoreML numbers. Whether it’s hosted Jev or an on-device Laya-style clone, the cost/latency curve still tracks how big the state you feed it is.
Before you chase another 10 decisions/sec, try a deterministic Tier-1 shrink of the observation/history you serialize — boring, local, no extra LLM judge. contextpress is one option: https://github.com/Taha-azizi/contextpress · https://pypi.org/project/contextpress/