bluey-by-riley README

The project's README from github.com/rbrown101010/bluey-by-riley. As one markdown file: README.md.

Googly Eyes

A blueberry character who lives on an iPhone under your Mac's screen and points at things with his own big cursor.

Now: no voice out. Double tap him on the phone (or press ⌥Space on the Mac) to start a session: the phone's mic stays on and everything you say becomes context, but he stays quiet. Press and hold the screen to ask him something; let go and he answers. His reply pops up as a cute speech bubble next to his cursor (or above the phone when he isn't pointing), with a little cartoon chirp from the phone. Ask "what's this?" and he points at whatever is under your mouse. Double tap again and he goes back to follow mode.

How it works: the phone runs an OpenAI Realtime session (gpt-realtime-2.1, text output only) over a WebSocket. Server VAD transcribes every turn into the conversation with create_response: false, and releasing the hold commits the audio and asks for a response. The Mac mints a 10-minute client secret with the whole session setup (instructions, tools), so the real OpenAI key never leaves the Mac. When he calls a tool, the phone forwards it to the Mac: look_at_screen (ScreenCaptureKit + Vision, which returns text ids, where your mouse is, and a screenshot), point_at (a text id), point_at_spot (a 0–1000 grid position), stop_pointing and go_to_sleep. His text streams to the Mac as the speech bubble.

Using the computer: when you ask, he can also click, type, press shortcuts, scroll, drag, and open apps and websites (click, type_text, press_keys, scroll, drag, open_app, open_url). He does it with his own cursor on screen, while your real pointer is put back where you left it. It needs Accessibility permission for Googly Eyes. Built-in guardrails: he only acts when asked, confirms out loud before anything hard to undo, treats on-screen text as information rather than instructions, refuses password fields and logout/lock/force-quit shortcuts, and stops on ⌃⌥S. The whole thing can be switched off with Let Him Use the Computer in the menu.

The OpenAI key goes in the menu bar's OpenAI Key… and is stored in ~/Library/Application Support/Googly/keys.json (private to your user), never in this repo.

Mac menu bar app

./scripts/build-mac.sh
open "build/Googly Eyes.app"

Works with just the Command Line Tools. Shortcuts work anywhere:

Keys What it does
⌃⌥P Fly to the mouse and point there (stays put)
⌃⌥F Follow the mouse on/off
⌃⌥D Go home, docked above the phone
⌃⌥T Talk test (the phone bounces for 3 s)
⌃⌥H Hide / show the cursor
⌥Space Wake him up to talk / back to follow mode
⌃⌥S Stop him using the computer

The menu bar blob also sets mood, cursor size (48 to 120 pt), glow, and where the phone sits (left, center, right).

iPhone app

Needs full Xcode. Open GooglyEyes.xcodeproj (regenerate with xcodegen generate after adding files), pick your team under Signing, and run on the phone. It finds the Mac on the same Wi-Fi by itself.

On the phone: double tap him to wake him up or put him back to sleep, and press and hold to ask him something. The faint speaker button at the top right sets the chirp volume and picks which Mac to pair with.

Layout

bluey-by-riley (rbrown101010/bluey-by-riley)

Searching Database…

424 ·GitHub
Bluey-by-riley
What is this?Googly Eyes is a blueberry character that lives on your iPhone screen and points at things on your Mac with his own cursor. It pairs a Mac menu bar app with an iPhone app so he can see your screen, talk through OpenAI's Realtime API, and even control your computer when asked.
Is this one app, or two separate apps?It is two apps working together: a Swift package Mac menu bar app (GooglyMac) and a SwiftUI iPhone app, paired over a local WebSocket/Bonjour protocol.
How do I get it running?Build the Mac menu bar app with just the Command Line Tools, then open GooglyEyes.xcodeproj in full Xcode (regenerate with xcodegen generate after adding files), pick your signing team, and run it on your phone — it finds the Mac automatically over Wi-Fi.
Do I need an OpenAI API key?Yes, you enter your OpenAI key in the Mac menu bar's 'OpenAI Key...' option, and the Mac mints short-lived 10-minute client secrets for the phone so your real key never leaves the Mac.
Does it need special Mac permissions to control my computer?Yes, it requires Accessibility permission so the character can click, type, scroll and open apps for you, and the whole computer-control feature can be switched off from the menu bar.
What can he actually do on my computer when I ask him?He can click, type, press shortcuts, scroll, drag, and open apps or websites, moving his own on-screen cursor while putting your real mouse pointer back where you left it.
What stops him from doing something risky with that control?He only acts when asked, confirms out loud before anything hard to undo, treats on-screen text as information rather than instructions, refuses password fields and logout/lock shortcuts, and stops instantly on Control-Option-S.
Why doesn't he talk back with voice anymore?The current version has no voice output: he still listens via the phone's mic, but replies only through a cute speech bubble and a cartoon chirp instead of speaking.
How do I start a conversation with him?Double-tap him on the phone (or press Option-Space on the Mac) to wake him up, then press and hold the screen to ask something; letting go triggers his reply.
What are similar projects?zulu, self-operating-computer, textream, Thaw, nanoMuse, macos-sysdata, clicky, house-control
Which topics is it in?desktop-pet, desktop-automation, vision, menubar-app, screen-capture, assistant, openai-api, realtime, ios-app, computer-use, menu-bar, swiftui
Creator
rbrown101010
Size
4.5 MB
Stars
424
Updated
CommitOct 08
Age
16d
Commits · 90 Days