omnibin README

The project's README from github.com/fzakaria/omnibin. As one markdown file: README.md.

omnibin

Every binary nixpkgs ever shipped, on your PATH.

$ ls /omnibin/bin | wc -l
51468

$ python3 --version
Python 3.14.6

$ [email protected] --version
Python 3.6.2

That machine had none of those packages a moment ago and still has almost none of them. /omnibin/bin is a filesystem, /nix/store underneath it is a filesystem, and a package's bytes are fetched from cache.nixos.org the first time something reads a file inside it.

Every command it knows about is browsable at https://omnibin.dev/, with every version of each, what it costs to fetch, and what is inside it.

Documentation: Design · Using it · Building the index · Caveats

Status

check workflow update workflow

system executables name@version package versions store paths
aarch64-linux 47,795 795,110 225,262 626,275
x86_64-linux 51,584 885,282 254,873 626,275

Store paths from 2012-07-05 to 2026-09-29, 61.1 TB unpacked. Commands are nameable from 2017-03-23 on, which is when Hydra started publishing file listings. Built from nixpkgs-multiverse data-20261001.

Why

Every sandbox an agent works in begins with somebody guessing which packages it will need. Guess short and the agent is stuck halfway through a task; guess long and you ship a ten gigabyte image so that something can run jq twice. Either way the agent's first move on a real task is to install something, which is what a sealed environment exists to prevent.

omnibin removes the guess. The machine starts with every version of every package on its PATH, weighing nothing, and pays only for what gets run.

The other half is history. nixpkgs-multiverse resolves any (attribute, version) in thirteen years of nixpkgs to the store path Hydra built for it, so an old version costs a download rather than a build. 610 packages have shipped something called python3, and reaching any of them is one command.

The store paths reach back to 2012. The commands inside them can only be named from March 2017, because that is when Hydra started publishing a file listing beside each narinfo. Older paths still fetch and still run; they just cannot be searched for by command name. See caveats.

How it works

Hydra publishes a .ls file beside every narinfo on cache.nixos.org: the complete listing of a store path's contents as JSON, with each entry's type, size, executable bit and symlink target. That is the metadata half of a Nix store, already served, at about nine kilobytes per path. omnibin crawls those listings once, which is the only data this project adds to what nixpkgs-multiverse already publishes, and keeps them in a database. That crawl is 233,690 store paths and 282 million files, and it is what every answer on this page and on the site is made of.

flowchart LR
  subgraph build ["built once, published as a release"]
    LS["cache.nixos.org<br/>.ls listings"] -->|"crawl, 12 min"| JSONL["listings<br/>jsonl.zst"]
    MV["nixpkgs-multiverse<br/>outpaths + narinfo graph"] --> DB[("omnibin.db<br/>50,462 names")]
    JSONL --> DB
  end

  subgraph run ["at runtime"]
    DB --> TREE["/omnibin/bin<br/>names and versions"]
    DB --> STORE["/nix/store<br/>lazy, passthrough to the real one"]
    STORE -.->|"only when a file is read"| NAR["cache.nixos.org<br/>NARs"]
  end

The filesystem answers every question about what exists out of that database, with no network at all:

$ ls /nix/store/2lb6nn8ivk1alhckv43n7734lqwbw7h9-python3-3.6.2/bin
2to3      idle     pydoc     python   python3.6          python3-config  pyvenv
2to3-3.6  idle3    pydoc3    python3  python3.6-config   python-config   pyvenv-3.6
          idle3.6  pydoc3.6           python3.6m         python3.6m-config

That is CPython from 2017 and it is not on the machine. Listing it cost nothing. Reading one byte out of python3.6 fetches the NAR, once, after which the path is served from the unpacked copy like any other directory.

Finding things

/omnibin/bin is the second filesystem and the one people use. ls shows the bare names, one per executable anybody ever shipped, each resolving to the newest package that provides it. The versioned forms resolve too but are deliberately not listed, because there are 881,933 of them and a directory nothing can read is worse than one that answers the questions you ask it.

So ask. omnibin which is the ergonomic way in and costs no downloads:

$ omnibin which python3
/nix/store/gxzhl7aaiid7zp3y47jqqiq7zg5mqpwp-python3-3.14.6/bin/python3

$ omnibin which --all python3 | wc -l
610

$ omnibin which --all ffmpeg | head -2
[email protected]  ffmpeg  0.4 MB  /nix/store/0adpc3…-ffmpeg-3.1.7-bin/bin/ffmpeg
[email protected]  ffmpeg  0.4 MB  /nix/store/nhfgdv…-ffmpeg-3.2.4-bin/bin/ffmpeg

The size is what running that version costs the first time. When you want to ask something which does not answer, the database is a plain SQLite file at /omnibin/index.db:

$ sqlite3 /omnibin/index.db \
    "SELECT name FROM latest WHERE name LIKE 'gcc%'"

Three ways in

A shell, on any Linux. The lazy store is mounted over /nix/store inside a user and mount namespace belonging to this shell, which dies with it. Your real store is served through it untouched, so the software you already have keeps working, including the shell you are typing into.

$ nix run github:fzakaria/omnibin
omnibin: tree at /run/user/1000/omnibin, cache at /home/you/.cache/omnibin

$ [email protected] -c 'import sys; print(sys.version.split()[0])'
3.6.2

$ jq --version
jq-1.8.1

$ exit

A NixOS machine. One module, and every package is installed on it.

{
  imports = [ inputs.omnibin.nixosModules.default ];
  services.omnibin.enable = true;
}

A container. fmzakari/omnibin mounts on start, so it works as a base for anything that wants a full toolbox without choosing one:

# syntax=docker/dockerfile:1
FROM fmzakari/omnibin:latest

COPY <<'SH' /demo.sh
[email protected] -c 'import sys; print(sys.version.split()[0])'
jq --version
[email protected] --version | head -1
SH

CMD ["bash", "/demo.sh"]
$ docker build -t example .
$ docker run --rm --device /dev/fuse --cap-add SYS_ADMIN example
3.6.2
jq-1.8.1
gcc (GCC) 10.2.0

Three eras of toolchain in one image that contains none of them.

$ docker run --rm -it --device /dev/fuse --cap-add SYS_ADMIN fmzakari/omnibin
$ ls /omnibin/bin | wc -l
51468

FUSE inside a container needs the device and the capability, and nothing else. SYS_ADMIN is for mount and /dev/fuse is for FUSE.

The packages arrive when the container runs, not when it builds. docker build gives a RUN step neither /dev/fuse nor the capability to mount, so a RUN that calls jq fails the same way it would on any base image without it. Put the work in CMD or ENTRYPOINT, where the mount is up.

What it costs

Measured on a machine that had never seen CPython 3.6.2:

$ time [email protected] -c 'import sys; print(sys.version.split()[0])'
3.6.2
real    0m2.690s

$ time [email protected] -c 'print(6*7)'
42
real    0m0.035s

The cold run fetched three store paths and 92 MB: the interpreter, its glibc, and one more. The warm run touched the network zero times. That is the trade, disk and bandwidth for exactly the packages that got used, against an image sized for the packages somebody guessed would be.

License

MIT, please see LICENSE.

The Nix expressions and tooling are original work. The published listings are generated metadata about store paths, being names, sizes and modes crawled from cache.nixos.org, and are not nixpkgs source.

omnibin (fzakaria/omnibin)

nixos
Searching Database…

What is this?omnibin is a FUSE-based lazy filesystem that puts every binary ever published across 13+ years of nixpkgs on your PATH, fetching each package's bytes from cache.nixos.org only when a file inside it is actually read.
Is this a package manager, or something else?It's two overlaid filesystems: /omnibin/bin lists every executable name anyone ever shipped, and /nix/store underneath is a lazy store that pulls a package's bytes from cache.nixos.org the first time a file in it is read. A SQLite index backs the search side.
How do I get omnibin running?There are three entry points: run it in a shell on any Linux box where it mounts the lazy store inside a user/mount namespace that dies with the shell, install the NixOS module to have every package available system-wide, or run the fmzakari/omnibin Docker image as a base for a full toolbox.
What does the container setup need to actually mount the store?Running omnibin inside Docker needs the SYS_ADMIN capability for mounting and access to /dev/fuse for FUSE — nothing else.
Can I use omnibin inside a Dockerfile's RUN steps?No — docker build gives a RUN step neither /dev/fuse nor the mount capability, so calling a tool there fails just like on any base image without it; the mount is only up once the container is running, so the work has to go in CMD or ENTRYPOINT instead.
Does this need NixOS, or will it run on any Linux machine?The shell mode works on any Linux machine: the lazy store is mounted over /nix/store inside a user and mount namespace tied to that shell, leaving your real store untouched.
Why did the author build omnibin instead of just pre-installing packages in a sandbox image?Agent sandboxes either guess short and get stuck mid-task, or guess long and ship a ten-gigabyte image. omnibin starts with every nixpkgs binary already on the PATH and you only pay in bandwidth and disk for what actually runs.
What is nixpkgs-multiverse and how does omnibin use it?nixpkgs-multiverse resolves any (package, version) pair across 13 years of nixpkgs history to the exact store path Hydra built; omnibin crawls the file listings Hydra publishes beside those store paths to build its own searchable index.
How do I find which package provides a given command?Run 'omnibin which ' to look it up in the crawled SQLite index at /omnibin/index.db at zero network cost, or browse the full searchable index on the project's site.
What does it actually cost to run a package you've never used before?Only the first read pays: fetching CPython 3.6.2 for the first time pulled three store paths totaling 92 MB (interpreter, glibc, and one more); every run after that is served from the unpacked copy with zero network calls.
What are similar projects?nixpkgs, home-manager, computer, nix, microdnf, kubo, mirage, uv
Creator
fzakaria
Size
0.3 MB
Stars
145
Age
23d
Commits · 90 Days