bluey-by-riley README

The project's README from github.com/rbrown101010/bluey-by-riley. As one markdown file: README.md.

Googly Eyes

A blueberry character who lives on an iPhone under your Mac's screen and points at things with his own big cursor.

Now: no voice out. Double tap him on the phone (or press ⌥Space on the Mac) to start a session: the phone's mic stays on and everything you say becomes context, but he stays quiet. Press and hold the screen to ask him something; let go and he answers. His reply pops up as a cute speech bubble next to his cursor (or above the phone when he isn't pointing), with a little cartoon chirp from the phone. Ask "what's this?" and he points at whatever is under your mouse. Double tap again and he goes back to follow mode.

How it works: the phone runs an OpenAI Realtime session (gpt-realtime-2.1, text output only) over a WebSocket. Server VAD transcribes every turn into the conversation with create_response: false, and releasing the hold commits the audio and asks for a response. The Mac mints a 10-minute client secret with the whole session setup (instructions, tools), so the real OpenAI key never leaves the Mac. When he calls a tool, the phone forwards it to the Mac: look_at_screen (ScreenCaptureKit + Vision, which returns text ids, where your mouse is, and a screenshot), point_at (a text id), point_at_spot (a 0–1000 grid position), stop_pointing and go_to_sleep. His text streams to the Mac as the speech bubble.

Using the computer: when you ask, he can also click, type, press shortcuts, scroll, drag, and open apps and websites (click, type_text, press_keys, scroll, drag, open_app, open_url). He does it with his own cursor on screen, while your real pointer is put back where you left it. It needs Accessibility permission for Googly Eyes. Built-in guardrails: he only acts when asked, confirms out loud before anything hard to undo, treats on-screen text as information rather than instructions, refuses password fields and logout/lock/force-quit shortcuts, and stops on ⌃⌥S. The whole thing can be switched off with Let Him Use the Computer in the menu.

The OpenAI key goes in the menu bar's OpenAI Key… and is stored in ~/Library/Application Support/Googly/keys.json (private to your user), never in this repo.

Mac menu bar app

./scripts/build-mac.sh
open "build/Googly Eyes.app"

Works with just the Command Line Tools. Shortcuts work anywhere:

Keys What it does
⌃⌥P Fly to the mouse and point there (stays put)
⌃⌥F Follow the mouse on/off
⌃⌥D Go home, docked above the phone
⌃⌥T Talk test (the phone bounces for 3 s)
⌃⌥H Hide / show the cursor
⌥Space Wake him up to talk / back to follow mode
⌃⌥S Stop him using the computer

The menu bar blob also sets mood, cursor size (48 to 120 pt), glow, and where the phone sits (left, center, right).

iPhone app

Needs full Xcode. Open GooglyEyes.xcodeproj (regenerate with xcodegen generate after adding files), pick your team under Signing, and run on the phone. It finds the Mac on the same Wi-Fi by itself.

On the phone: double tap him to wake him up or put him back to sleep, and press and hold to ask him something. The faint speaker button at the top right sets the chirp volume and picks which Mac to pair with.

Layout

bluey-by-riley (rbrown101010/bluey-by-riley)

Searching Database…

412 ·GitHub
Bluey-by-riley
这是什么？Googly Eyes 是一个住在 iPhone 屏幕上的蓝莓角色，会用它自己的光标指向你 Mac 上的东西。它由一个 Mac 菜单栏应用和一个 iPhone 应用配对组成，让它能看到你的屏幕、通过 OpenAI 的 Realtime API 说话，甚至在你开口时控制你的电脑。
这是一个应用，还是两个独立的应用？它是两个协同工作的应用：一个 Swift 包形式的 Mac 菜单栏应用（GooglyMac）和一个 SwiftUI iPhone 应用，通过本地 WebSocket/Bonjour 协议配对。
怎么让它跑起来？只用 Command Line Tools 构建 Mac 菜单栏应用，然后在完整版 Xcode 里打开 GooglyEyes.xcodeproj（添加文件后用 xcodegen generate 重新生成），选择你的签名团队，并在手机上运行——它会通过 Wi-Fi 自动找到 Mac。
需要 OpenAI API 密钥吗？需要，你在 Mac 菜单栏的“OpenAI Key...”选项里输入你的 OpenAI 密钥，Mac 会为手机铸造有效期 10 分钟的短期客户端密钥，所以你的真实密钥从不离开 Mac。
控制我的电脑需要特殊的 Mac 权限吗？需要，它要求辅助功能（Accessibility）权限，这个角色才能替你点击、输入、滚动和打开应用；整个电脑控制功能也可以从菜单栏关闭。
我开口时它能在我的电脑上实际做什么？它能点击、输入、按快捷键、滚动、拖拽，以及打开应用或网站，移动它自己的屏幕光标，同时把你真实的鼠标指针放回你离开时的位置。
是什么防止它用这些控制权做危险的事？它只在被要求时才行动，在做任何难以撤销的事之前会出声确认，把屏幕上的文字当作信息而不是指令，拒绝密码输入框和登出/锁定快捷键，并且一按 Control-Option-S 立即停下。
它为什么不再用语音回话了？当前版本没有语音输出：它仍然通过手机的麦克风聆听，但只用可爱的对话气泡和卡通啾啾声来回应，不再开口说话。
怎么和它开始对话？在手机上双击它（或在 Mac 上按 Option-Space）把它唤醒，然后按住屏幕提问；松手就会触发它的回答。
What are similar projects?zulu, self-operating-computer, textream, Thaw, nanoMuse, macos-sysdata, clicky, house-control
Which topics is it in?desktop-pet, desktop-automation, vision, menubar-app, screen-capture, assistant, openai-api, realtime, ios-app, computer-use, menu-bar, swiftui
Creator
rbrown101010
Size
4.5 MB
Stars
412
Updated
CommitOct 08
Age
15d
Commits · 90 Days