gpt-crawler README

The project's README from github.com/BuilderIO/gpt-crawler. As one markdown file: README.md.

GPT Crawler

Deutsch | Español | français | 日本語 | 한국어 | Português | Русский | 中文

Crawl a site to generate knowledge files to create your own custom GPT from one or multiple URLs

Gif showing the crawl run

Example

Here is a custom GPT that I quickly made to help answer questions about how to use and integrate Builder.io by simply providing the URL to the Builder docs.

This project crawled the docs and generated the file that I uploaded as the basis for the custom GPT.

Try it out yourself by asking questions about how to integrate Builder.io into a site.

Note that you may need a paid ChatGPT plan to access this feature

Get started

Running locally

Clone the repository

Be sure you have Node.js >= 16 installed.

git clone https://github.com/builderio/gpt-crawler

Install dependencies

npm i

Configure the crawler

Open config.ts and edit the url and selector properties to match your needs.

E.g. to crawl the Builder.io docs to make our custom GPT you can use:

export const defaultConfig: Config = {
  url: "https://www.builder.io/c/docs/developers",
  match: "https://www.builder.io/c/docs/**",
  selector: `.docs-builder-container`,
  maxPagesToCrawl: 50,
  outputFileName: "output.json",
};

See config.ts for all available options. Here is a sample of the common configuration options:

type Config = {
  /** URL to start the crawl, if sitemap is provided then it will be used instead and download all pages in the sitemap */
  url: string;
  /** Pattern to match against for links on a page to subsequently crawl */
  match: string;
  /** Selector to grab the inner text from */
  selector: string;
  /** Don't crawl more than this many pages */
  maxPagesToCrawl: number;
  /** File name for the finished data */
  outputFileName: string;
  /** Optional resources to exclude
   *
   * @example
   * ['png','jpg','jpeg','gif','svg','css','js','ico','woff','woff2','ttf','eot','otf','mp4','mp3','webm','ogg','wav','flac','aac','zip','tar','gz','rar','7z','exe','dmg','apk','csv','xls','xlsx','doc','docx','pdf','epub','iso','dmg','bin','ppt','pptx','odt','avi','mkv','xml','json','yml','yaml','rss','atom','swf','txt','dart','webp','bmp','tif','psd','ai','indd','eps','ps','zipx','srt','wasm','m4v','m4a','webp','weba','m4b','opus','ogv','ogm','oga','spx','ogx','flv','3gp','3g2','jxr','wdp','jng','hief','avif','apng','avifs','heif','heic','cur','ico','ani','jp2','jpm','jpx','mj2','wmv','wma','aac','tif','tiff','mpg','mpeg','mov','avi','wmv','flv','swf','mkv','m4v','m4p','m4b','m4r','m4a','mp3','wav','wma','ogg','oga','webm','3gp','3g2','flac','spx','amr','mid','midi','mka','dts','ac3','eac3','weba','m3u','m3u8','ts','wpl','pls','vob','ifo','bup','svcd','drc','dsm','dsv','dsa','dss','vivo','ivf','dvd','fli','flc','flic','flic','mng','asf','m2v','asx','ram','ra','rm','rpm','roq','smi','smil','wmf','wmz','wmd','wvx','wmx','movie','wri','ins','isp','acsm','djvu','fb2','xps','oxps','ps','eps','ai','prn','svg','dwg','dxf','ttf','fnt','fon','otf','cab']
   */
  resourceExclusions?: string[];
  /** Optional maximum file size in megabytes to include in the output file */
  maxFileSize?: number;
  /** Optional maximum number tokens to include in the output file */
  maxTokens?: number;
};

Run your crawler

npm start

Alternative methods

Running in a container with Docker

To obtain the output.json with a containerized execution, go into the containerapp directory and modify the config.ts as shown above. The output.jsonfile should be generated in the data folder. Note: the outputFileName property in the config.ts file in the containerapp directory is configured to work with the container.

Running as an API

To run the app as an API server you will need to do an npm install to install the dependencies. The server is written in Express JS.

To run the server.

npm run start:server to start the server. The server runs by default on port 3000.

You can use the endpoint /crawl with the post request body of config json to run the crawler. The api docs are served on the endpoint /api-docs and are served using swagger.

To modify the environment you can copy over the .env.example to .env and set your values like port, etc. to override the variables for the server.

Upload your data to OpenAI

The crawl will generate a file called output.json at the root of this project. Upload that to OpenAI to create your custom assistant or custom GPT.

Create a custom GPT

Use this option for UI access to your generated knowledge that you can easily share with others

Note: you may need a paid ChatGPT plan to create and use custom GPTs right now

  1. Go to https://chat.openai.com/
  2. Click your name in the bottom left corner
  3. Choose "My GPTs" in the menu
  4. Choose "Create a GPT"
  5. Choose "Configure"
  6. Under "Knowledge" choose "Upload a file" and upload the file you generated
  7. if you get an error about the file being too large, you can try to split it into multiple files and upload them separately using the option maxFileSize in the config.ts file or also use tokenization to reduce the size of the file with the option maxTokens in the config.ts file

Gif of how to upload a custom GPT

Create a custom assistant

Use this option for API access to your generated knowledge that you can integrate into your product.

  1. Go to https://platform.openai.com/assistants
  2. Click "+ Create"
  3. Choose "upload" and upload the file you generated

Gif of how to upload to an assistant

Contributing

Know how to make this project better? Send a PR!



Made with love by Builder.io

gpt-crawler (BuilderIO/gpt-crawler)

Searching Database…

22.4K ·GitHub
Gpt-crawler
What is this?A crawler that scrapes a website (one or multiple URLs) and turns the pages into a knowledge file you can upload to OpenAI to build a custom GPT or assistant.
Is this a CLI tool or does it run as a server too?Both: by default it runs locally as a CLI/script, but it can also run as an Express API server with a /crawl endpoint and Swagger docs, or inside a Docker container.
How do I get it running?Clone the repo, install dependencies, edit the url and selector properties in config.ts to target your site, then run the crawler to produce output.json.
What do I need installed before running it?Node.js 16 or higher; Playwright is installed automatically via the preinstall script.
Is there a size limit on the file I can upload to OpenAI?Yes, if the generated output.json is too large for a custom GPT, you need to split it into multiple files or shrink it using the maxFileSize or maxTokens config options.
Do I need a paid ChatGPT plan to use the output?To create and use a custom GPT in the ChatGPT UI, you currently may need a paid ChatGPT plan; using the output via the Assistants API doesn't require that.
What does the crawler actually produce?It outputs an output.json file at the project root containing the crawled content, ready to upload to OpenAI as knowledge for a custom GPT or assistant.
Can I run this without touching code?You still need to edit config.ts to set the target URL and CSS selector before running, though Docker and API modes let you avoid managing the Node process directly.
What crawling engine does it use under the hood?It's built on Crawlee with Playwright for browser automation, and Express for the optional API server.
Can I crawl more than one site at once?Yes, the README explicitly supports generating knowledge files from one or multiple URLs.
What are similar projects?crawl4ai, maxun, apify-mcp-server, crawlee, deep-research, firecrawl, gpt4free, MediaCrawler
Which topics is it in?swagger, scraper, crawler, express, playwright, gpt, web-scraping, knowledge-base, chatbot, rest-api, openai, api
Creator
BuilderIO
Size
0.9 MB
Stars
22.4K
Updated
CommitJul 07Releasev1.5.1Jan 23
Age
2y
Commits · 90 Days