What is this?
A Windows toolkit that lets Codex or other computer-use agents play Minecraft by reading the screen and sending keyboard and mouse input to move, gather, craft, explore, and build.
A Windows toolkit that lets Codex or other computer-use agents play Minecraft by reading the screen and sending keyboard and mouse input to move, gather, craft, explore, and build.Does this include the AI agent that plays the game, or just the controls?Just the controls and optional visual checks. The agent itself (Codex or another computer-use agent) decides what to do; this toolkit has no agent model, game client, world save, or game-memory interface built in.
Just the controls and optional visual checks. The agent itself (Codex or another computer-use agent) decides what to do; this toolkit has no agent model, game client, world save, or game-memory interface built in.How do I get it running?Download or clone the repo keeping the agents and scripts folders, open the codex-minecraft-gameplay folder as a local Codex project on the Windows PC running Minecraft, launch Minecraft into a world, and give Codex the repo plus an in-game goal.
Download or clone the repo keeping the agents and scripts folders, open the codex-minecraft-gameplay folder as a local Codex project on the Windows PC running Minecraft, launch Minecraft into a world, and give Codex the repo plus an in-game goal.Do I need Codex specifically, or can I use another agent?It is built for Codex but works with other computer-use agents too, since it just exposes Windows input events and screenshots for any agent to drive.
It is built for Codex but works with other computer-use agents too, since it just exposes Windows input events and screenshots for any agent to drive.Does it work on Mac or Linux?No, the runtime relies on Windows input events and screenshots, so it only works on a Windows PC running Minecraft.
No, the runtime relies on Windows input events and screenshots, so it only works on a Windows PC running Minecraft.What actions can the agent perform in-game?It can move, look around, gather materials, craft items, explore, and build, all through the normal visible game interface.
It can move, look around, gather materials, craft items, explore, and build, all through the normal visible game interface.Why doesn't it include the agent or game memory itself?The toolkit only handles input and screen checks; it intentionally leaves the agent model, game client, world save, and any memory system to the user, keeping the project focused purely on controls.
The toolkit only handles input and screen checks; it intentionally leaves the agent model, game client, world save, and any memory system to the user, keeping the project focused purely on controls.Does the game window need to stay visible while it runs?Yes, Minecraft must be open with the game window visible, since the agent only perceives the game through screenshots of that window.
Yes, Minecraft must be open with the game window visible, since the agent only perceives the game through screenshots of that window.What are similar projects?robotgo, typesafe-computer-use, collab-agent, agent-desktop, midscene, pyautogui, bytebot, self-operating-computer
robotgo, typesafe-computer-use, collab-agent, agent-desktop, midscene, pyautogui, bytebot, self-operating-computerWhich topics is it in?computer-use-agent, input, mouse, desktop-automation, gui-automation, keyboard, toolkit, screenshot, computer-use, game, gaming, agentic-ai
computer-use-agent, input, mouse, desktop-automation, gui-automation, keyboard, toolkit, screenshot, computer-use, game, gaming, agentic-ai