kitesurf README

The project's README from github.com/cloudflare/kitesurf. As one markdown file: README.md.

Kitesurf logo

Kitesurf

Kitesurf is an experimental browser engine that runs on Cloudflare Workers. It parses HTML, applies CSS, runs page JavaScript, and renders screenshots without Chrome or a browser container.

Kitesurf speaks the Chrome DevTools Protocol (CDP), so you can use it with CDP clients such as Puppeteer.

[!IMPORTANT] Kitesurf requires Cloudflare Dynamic Workers to create isolated workers for pages. Dynamic Workers are part of the Workers Paid plan. Local development uses wrangler dev. A Cloudflare Workers Paid plan is required to run Kitesurf on Cloudflare.

Requirements

Install pnpm separately if it is not already available:

npm install --global [email protected]

Build and run locally

Clone the repository and install its dependencies:

git clone https://github.com/cloudflare/kitesurf.git
cd kitesurf
pnpm install --frozen-lockfile

Build and start the local engine:

pnpm run dev

The engine listens on http://127.0.0.1:8787. Visit http://127.0.0.1:8787/health to confirm that it is running.

The root URL shows the landing page. It has a form to open a URL in Chrome DevTools, example pages, and the CDP endpoints for agents and other clients. See Use Chrome DevTools.

pnpm run dev builds the WebAssembly modules and the page and renderer bundles with Bazel, then starts wrangler dev. The first build compiles the Rust core and its vendored crates, which takes a while; later builds reuse Bazel's cache.

Set KITESURF_PORT when port 8787 is already in use:

KITESURF_PORT=8799 pnpm run dev

Run many navigations locally

Each navigation starts a new page isolate and stops the previous one. A local engine keeps a stopped isolate in memory until the engine's garbage collector runs. If you run many navigations in parallel, these isolates can use many gigabytes of memory.

pnpm run dev starts the runtime with the V8 flag --expose-gc. The engine then runs garbage collection after it stops a page isolate. Pages do not get gc(). In one test, five parallel connections loaded 940 pages in total. Without the flag, the runtime stopped with a V8 out-of-memory error after about 300 pages. With the flag, all 940 pages loaded and memory stayed below 4 GB.

If you start wrangler dev directly, for example in a container, set the flag yourself. Run this from packages/engine and add your other options:

MINIFLARE_WORKERD_V8_FLAGS=--expose-gc wrangler dev --env local

If you set MINIFLARE_WORKERD_V8_FLAGS for pnpm run dev, your value replaces the default. Include --expose-gc in it.

Connect with Puppeteer

Install puppeteer-core in the project that will control Kitesurf. Then connect to the browser WebSocket endpoint:

import puppeteer from "puppeteer-core";

const browser = await puppeteer.connect({
  browserWSEndpoint: "ws://127.0.0.1:8787/devtools/browser",
});

const page = await browser.newPage();
await page.goto("https://example.com/");
await page.screenshot({ path: "page.png" });

await browser.disconnect();

Use the port from KITESURF_PORT if you selected a different port. HTTP discovery is available at http://127.0.0.1:8787/json/version.

Use Chrome DevTools

With pnpm run dev, open http://127.0.0.1:8787/ in Chrome. Enter a URL to inspect, or select one of the example pages. DevTools opens in the tab on that page. To go to DevTools directly, open http://127.0.0.1:8787/?url=https://example.com, or http://127.0.0.1:8787/?inspect=1 for a blank page.

If you built the bundled frontend, pnpm run dev serves it from the engine's own origin. See Use the bundled DevTools frontend. Otherwise, the DevTools frontend comes from chrome-devtools-frontend.appspot.com, the build that Google hosts for Chrome. The hosted frontend has these limits:

You can also use chrome://inspect:

  1. Open chrome://inspect/#devices in Chrome. Type or paste it into the address bar because web pages cannot link to it.
  2. Select Discover network targets.
  3. Select Configure... and add 127.0.0.1:8787.
  4. Under Remote Target, select inspect next to kitesurf.
  5. Enter a page URL in the address bar above the screencast.

Use the port from KITESURF_PORT when you select a different local port. Chrome discovers targets with plain HTTP and WebSocket connections. The chrome://inspect workflow cannot reach an HTTPS engine. Use it with a local engine instead.

Kitesurf provides a CDP endpoint, not a full Chrome implementation. Some Chrome DevTools features do not work yet. We are working to add more support over time. Puppeteer and other CDP clients can connect to ws://127.0.0.1:8787/devtools/browser.

Use the bundled DevTools frontend

Kitesurf can serve the Chrome DevTools frontend from its own origin. Then DevTools works without the hosted frontend, its permission prompt and the 127.0.0.1 limit. A deployed engine always uses this copy. The frontend is the prebuilt build that Google publishes for Chromium. The default build does not include it.

Build the frontend

The --config=devtools Bazel option adds the frontend to the build. The first build with it downloads the frontend (about 250 MB). Bazel keeps the download in its cache for later builds.

To build the frontend once, run this command from the repository root:

bazel run --config=devtools //packages/engine:dev_build

To include the frontend in every local build, add the option to .bazelrc.user. Git ignores this file. Then pnpm run build and pnpm run dev include the frontend.

echo 'common --config=devtools' >> .bazelrc.user

The build copies the frontend to packages/engine/public/inspector/. Git ignores this directory. A build without the option removes it. To turn the frontend off, remove the line from .bazelrc.user.

Run the frontend locally

  1. Build the frontend as described above.
  2. Start the engine with pnpm run dev. The engine serves the bundled frontend when the build includes it, and the hosted frontend when it does not.
  3. Open http://127.0.0.1:8787/ in Chrome. Enter a URL to inspect, or select one of the example pages. DevTools opens in the tab on that page.

To start on a page, open http://127.0.0.1:8787/?url=https://example.com.

Rebuild the frontend

Use the smallest step that fixes the problem:

Do not use bazel clean --expunge for this. It also removes the WebAssembly build and the rest of the Bazel cache.

Update the frontend version

Two pins select the frontend version:

Keep the two pins on the same Chrome version. After you change MODULE.bazel, the next build downloads the new archive.

Build and test

The pnpm scripts are front ends to Bazel:

pnpm run build                    # bazel run //packages/engine:dev_build
pnpm run check                    # type checks, bundles, and repository checks
pnpm run test:unit                # engine vitest and Rust host tests
pnpm run test:e2e <file-substring>
pnpm run format:check             # oxfmt; pnpm run format rewrites the same files
pnpm run format:rust              # format Rust with Bazel's pinned rustfmt
pnpm run lint                     # oxlint, the custom lint rules, and Rust lint

The Rust formatter rewrites tracked and non-ignored untracked files, excluding paths in .oxfmtrc.json.

pnpm run build copies the Bazel-built WebAssembly, bundles, and RBI files into the source tree, where wrangler dev reads them. Other targets run directly, for example bazel test //tests/e2e:gate for the full e2e suite. Another Bazel workspace can depend on this repository as a module.

Technology stack

Kitesurf combines several open-source Rust projects:

Kitesurf patches its Rust dependencies. The patch series are in scripts/patches/, and Bazel applies them itself.

Cloudflare deployment

[!WARNING] Do not deploy Kitesurf without authentication in front of it. Use Cloudflare Access (Cloudflare Zero Trust) or another authentication layer. The engine has no authentication of its own. A person who has the URL can:

  • Open DevTools and browse the web through your engine.
  • Connect Puppeteer and other CDP clients to it.
  • Send outbound requests that come from your Cloudflare account.
  • Use CPU time and Dynamic Workers that your account pays for.

The workers.dev URL is public by default. Its name follows a predictable pattern: kitesurf.<your-subdomain>.workers.dev.

To deploy the engine as a Worker on your Cloudflare account:

  1. Deploy the engine. Run the command from the repository root. In a package directory, use pnpm -w run deploy.

    pnpm run deploy
    
  2. Protect the Worker URL with Cloudflare Access immediately. Follow Manage access to workers.dev. For a custom domain or route, protect it with a self-hosted Access application.

  3. Open the Worker URL in Chrome. Access asks you to sign in. Then the landing page opens. Enter a URL to open it in DevTools.

Between steps 1 and 2, the engine is public. To prevent this gap, turn on Access for all Workers in the account before you deploy. The workers.dev page in step 2 tells you how.

The command builds the engine with Bazel and uploads it with Wrangler. Wrangler uses the account of wrangler login, or CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN. The command sends other arguments to wrangler deploy, for example --name <worker-name>. Without --name, the Worker name is kitesurf.

To build the deploy without an upload, run pnpm run deploy --dry-run.

To upload a version without sending traffic to it, run pnpm run deploy:preview. The command runs wrangler versions upload. If the Worker has Version URLs turned on, Wrangler prints the URL of the new version. Version URLs are public too. Protect them with Access as well. Follow Manage access to Version URLs.

The deploy always builds with --config=devtools, so it includes the Chrome DevTools frontend. You do not need .bazelrc.user for it. The first deploy downloads the frontend, as the first local build does. The deploy sets KITESURF_INSPECTOR_BASE_URL to /inspector/inspector.html. To use a different frontend, add --var KITESURF_INSPECTOR_BASE_URL:<url> to the command.

DevTools loads from the Worker's own origin, so its WebSocket sends the Access cookie. A DevTools frontend on another origin cannot connect through Access.

Connect a CDP client through Access

Access blocks a CDP client that has no browser session. To connect Puppeteer:

  1. Create an Access service token.

  2. Add a policy with the Service Auth action that allows this token.

  3. Send the token headers with the WebSocket connection:

    const browser = await puppeteer.connect({
      browserWSEndpoint: "wss://<worker-host>/devtools/browser",
      headers: {
        "CF-Access-Client-Id": process.env.CF_ACCESS_CLIENT_ID,
        "CF-Access-Client-Secret": process.env.CF_ACCESS_CLIENT_SECRET,
      },
    });
    

Keep the service token secret. The token gives the same access as a signed-in user.

The engine configuration includes a Worker Loader binding. Kitesurf uses this binding to start a separate worker for each page and renderer. Cloudflare Dynamic Workers provide this capability, and they require the Workers Paid plan. See the Dynamic Workers pricing documentation before planning a Cloudflare deployment.

Packages

Package Description
engine Cloudflare Worker and CDP server
page-script Page DOM and JavaScript runtime
page-renderer Renderer for recorded page scenes
dom-shim DOM APIs inside the page worker
core Rust layout, paint, and JavaScript engine code
renderer-core Rust scene rasterizer for page rendering
scene-host-wasm Local scene rasterizer for the terminal client
kitesurf-cli Terminal client

Contributing

Read CONTRIBUTING.md before opening an issue or pull request. Report security problems as described in SECURITY.md.

License

Kitesurf is licensed under Apache 2.0. Bundled fonts, data, and dependencies keep their own licenses. See NOTICE.

kitesurf (cloudflare/kitesurf)

Searching Database…

What is this?Kitesurf is an experimental browser engine that runs on Cloudflare Workers. It parses HTML, applies CSS, runs page JavaScript and renders screenshots without Chrome or a browser container.
Is it a full Chrome replacement?Not yet. It provides a CDP endpoint rather than a full Chrome implementation, so some DevTools features do not work.
How do I get it running locally?Clone the repo, install dependencies, then run pnpm run dev. It builds with Bazel and starts wrangler dev on port 8787. The first build compiles the Rust core and takes a while. Set KITESURFPORT to use another port.
What do I need installed first?Bazelisk (installed as bazel), Node.js 22 or newer and pnpm 9.15.4. On macOS you also need GNU tar. Rust via rustup is only needed for editing vendored crate patches, and Bazel fetches everything else itself.
How are pages kept isolated from each other?Each page and renderer runs in its own worker, started through Cloudflare Dynamic Workers. That is why the Workers Paid plan is needed.
What can it output besides the live page?It renders screenshots from recorded page scenes and can generate PDF output through Krilla.
Which parts of the repo matter most?The engine package is the Cloudflare Worker and CDP server, page-script holds the page DOM and JavaScript runtime, and core contains the Rust layout, paint and JavaScript code. kitesurf-cli is a terminal client.
Creator
cloudflare
Size
42.6 MB
Stars
2.3K
Updated
CommitNov 17
Age
1mo
Commits · 90 Days