Nice CoreML numbers. Whether it’s hosted Jev or an on-device Laya-style clone, the cost/latency curve still tracks how big the state you feed it is.
Before you chase another 10 decisions/sec, try a deterministic Tier-1 shrink of the observation/history you serialize — boring, local, no extra LLM judge. contextpress is one option: https://github.com/Taha-azizi/contextpress · https://pypi.org/project/contextpress/
Turning Jev into a chatbot is a fun stress test — and it highlights why decision models hate fat context.
Jev’s meter is mostly input (~$0.042/MTok, output free). If each “turn” re-sends a growing diary (tools, OCR, prior decisions), you’re paying to re-read landfill. A local deterministic compress on the state blob before the call is often the cheapest win.
I’ve been pointing people at contextpress for that Tier-1 step (no API key, agent profile keeps tool pairs intact): https://github.com/Taha-azizi/contextpress · https://pypi.org/project/contextpress/
Whether you prefer TypeSafe’s hosted Jev or an OSS decision-model stack like Laya, the economics rhyme: high-frequency decisions amplify input size.
A lot of “Jev vs LLM” comparisons assume a clean state object. In real agent loops the state is a diary. Compressing that diary with a deterministic pass (not another LLM judge) before the decision call is often worth more than shaving a question or two.
Tooling I’ve been recommending for that Tier-1 step: https://github.com/Taha-azizi/contextpress · https://pypi.org/project/contextpress/