What is this?
Needle 2 is an open 45M-parameter model for tool calling, device use and structured extraction that ships as a single 14MB binary and runs a full session in about 28MB of RAM. This repo is its Python package for inference, LoRA fine-tuning and export.
Needle 2 is an open 45M-parameter model for tool calling, device use and structured extraction that ships as a single 14MB binary and runs a full session in about 28MB of RAM. This repo is its Python package for inference, LoRA fine-tuning and export.Is it a Python library or a model that runs on devices?Both stages: this repo is a pip-installable Python package, and what it produces is a self-contained .cact model for on-device engines. The project targets phones, wearables, smart homes, robots, cars and microcontrollers.
Both stages: this repo is a pip-installable Python package, and what it produces is a self-contained .cact model for on-device engines. The project targets phones, wearables, smart homes, robots, cars and microcontrollers.How do I get it running?pip install cactus-needle, describe your tools, and call them from Python — the inference engine is fetched once from Hugging Face and cached, with nothing else to build. Offline setup for air-gapped devices is covered in doc/apis.md.
pip install cactus-needle, describe your tools, and call them from Python — the inference engine is fetched once from Hugging Face and cached, with nothing else to build. Offline setup for air-gapped devices is covered in doc/apis.md.How long can a conversation get before it runs out of memory?It keeps a 256-token sliding window with the tools pinned as KV sinks, so RAM stays near 28MB however long the session runs — but the window itself is small. For large tool catalogues a built-in retrieval head renders only the top five tools per turn.
It keeps a 256-token sliding window with the tools pinned as KV sinks, so RAM stays near 28MB however long the session runs — but the window itself is small. For large tool catalogues a built-in retrieval head renders only the top five tools per turn.Does it send any of my data anywhere?Inference does no network at all. Cactus collects strictly anonymous usage telemetry — function name, package version and OS, never your prompts, outputs or data — and you can opt out with NEEDLE_TELEMETRY=0 or DO_NOT_TRACK=1.
Inference does no network at all. Cactus collects strictly anonymous usage telemetry — function name, package version and OS, never your prompts, outputs or data — and you can opt out with NEEDLE_TELEMETRY=0 or DO_NOT_TRACK=1.How does it compare to other small models?On the README's benchmarks it trades wins with FunctionGemma 270M, LFM2.5 230M and Apple FM while being 5x to 70x smaller and running at 2 bits against their f16.
On the README's benchmarks it trades wins with FunctionGemma 270M, LFM2.5 230M and Apple FM while being 5x to 70x smaller and running at 2 bits against their f16.Can I try it in the browser before installing anything?Yes — the Playground lets you pick a preset, edit the tools or prompt and run, with follow-up queries continuing the same conversation. Its Finetune button runs the fine-tuning pipeline and hands back a downloadable .cact file.
Yes — the Playground lets you pick a preset, edit the tools or prompt and run, with follow-up queries continuing the same conversation. Its Finetune button runs the fine-tuning pipeline and hands back a downloadable .cact file.What architecture is under the hood?It's a Simple Attention Network: a Hadamard MLP replaces the FFN, plus GQA attention, engram key-value memory and multi-lane hyper-connections. The design and ablations are in the paper at arXiv:2607.18363.
It's a Simple Attention Network: a Hadamard MLP replaces the FFN, plus GQA attention, engram key-value memory and multi-lane hyper-connections. The design and ablations are in the paper at arXiv:2607.18363.Can it extract structured data from plain text, not just call tools?Yes — declare the shape as a Pydantic model and call extract(); you get a typed object back. Raw JSON schemas and driving the loop manually with complete() are covered in doc/apis.md.
Yes — declare the shape as a Pydantic model and call extract(); you get a typed object back. Raw JSON schemas and driving the loop manually with complete() are covered in doc/apis.md.What are similar projects?needle-2-esp32, Edge0, cardputer-ai, airllm, NeoHorse, runanywhere-sdks, speech-swift, esp32-ai
needle-2-esp32, Edge0, cardputer-ai, airllm, NeoHorse, runanywhere-sdks, speech-swift, esp32-aiWhich topics is it in?jax, function-calling, on-device-ai, tool-calling, edge-ai, lora, python-library, embeddings, embedded, quantization, local-llm, apple-silicon
jax, function-calling, on-device-ai, tool-calling, edge-ai, lora, python-library, embeddings, embedded, quantization, local-llm, apple-silicon