prediction-market-analysis README

The project's README from github.com/Jon-Becker/prediction-market-analysis. As one markdown file: README.md.

Prediction Market Analysis

A framework for analyzing prediction market data, including the largest publicly available dataset of Polymarket and Kalshi market and trade data. Provides tools for data collection, storage, and running analysis scripts that generate figures and statistics.

Overview

This project enables research and analysis of prediction markets by providing:

Currently supported features:

Installation & Usage

Requires Python 3.9+. Install dependencies with uv:

uv sync

Download and extract the pre-collected dataset (36GiB compressed):

make setup

This downloads data.tar.zst from Cloudflare R2 Storage and extracts it to data/.

Data Collection

Collect market and trade data from prediction market APIs:

make index

This opens an interactive menu to select which indexer to run. Data is saved to data/kalshi/ and data/polymarket/ directories. Progress is saved automatically, so you can interrupt and resume collection.

Running Analyses

make analyze

This opens an interactive menu to select which analysis to run. You can run all analyses or select a specific one. Output files (PNG, PDF, CSV, JSON) are saved to output/.

Packaging Data

To compress the data directory for storage/distribution:

make package

This creates a zstd-compressed tar archive (data.tar.zst) and removes the data/ directory.

Project Structure

├── src/
│   ├── analysis/           # Analysis scripts
│   │   ├── kalshi/         # Kalshi-specific analyses
│   │   └── polymarket/     # Polymarket-specific analyses
│   ├── indexers/           # Data collection indexers
│   │   ├── kalshi/         # Kalshi API client and indexers
│   │   └── polymarket/     # Polymarket API/blockchain indexers
│   └── common/             # Shared utilities and interfaces
├── data/                   # Data directory (extracted from data.tar.zst)
│   ├── kalshi/
│   │   ├── markets/
│   │   └── trades/
│   └── polymarket/
│       ├── blocks/
│       ├── markets/
│       └── trades/
├── docs/                   # Documentation
└── output/                 # Analysis outputs (figures, CSVs)

Documentation

Contributing

If you'd like to contribute to this project, please open a pull-request with your changes, as well as detailed information on what is changed, added, or improved.

For more information, see the contributing guide.

Issues

If you've found an issue or have a question, please open an issue here.

Research & Citations

If you have used or plan to use this dataset in your research, please reach out via email or Twitter -- i'd love to hear about what you're using the data for! Additionally, feel free to open a PR and update this section with a link to your paper.

prediction-market-analysis (Jon-Becker/prediction-market-analysis)

datasets
Searching Database…

3.9K ·GitHub
Prediction-market-analysis
What is this?It's a framework for collecting and analyzing prediction market data from Polymarket and Kalshi, bundled with the largest publicly available dataset of their market and trade history.
Is this a ready-made dataset or a toolkit I run myself?It's both: a Python toolkit with interactive data-collection indexers and an extensible analysis script framework, plus a pre-collected 36GiB dataset you can download and use directly without collecting anything.
How do I get the data and run an analysis?Install dependencies with uv, then either download and extract the pre-collected dataset or run the interactive indexer menu to collect fresh market/trade data; a separate interactive menu lets you pick and run analysis scripts.
What do I need installed before running this?You need Python 3.9+ and the uv package manager; collecting fresh data also requires reaching the Polymarket/Kalshi APIs and blockchain data sources.
Any catch with the dataset or collection process?The full pre-collected dataset is 36GiB compressed, so grabbing and extracting it takes real bandwidth and disk space; if you collect fresh data yourself, progress saves automatically so you can safely interrupt and resume.
What file formats does the data and output come in?Markets and trades are stored in Parquet files with automatic progress saving, while analysis scripts output PNG, PDF, CSV, and JSON files to an output/ directory.
Can I add my own analysis scripts?Yes, the project ships an extensible analysis script framework with a dedicated 'Writing Analyses' guide, so custom analyses can plug into the same interactive menu.
Why was this dataset built?It was built to support academic research into prediction market behavior; the README lists over a dozen papers from SSRN and arXiv that already analyze wisdom-of-crowds effects and trading patterns using this data.
Is the project actively maintained?Yes, it has CI workflows for pull-request validation and explicitly invites contributions of new indexers, analysis scripts, and links to new research papers that use the dataset.
What are similar projects?akshare, COVID-19, awesome-quant, anti-gambling-trader-tw, polyledger, miller, datasette, xsv
Which topics is it in?data-collection, market-data, datasets, data, csv, quantitative-finance, fintech, statistics, data-analysis, blockchain, dataset, finance
Creator
Jon-Becker
Size
158.2 MB
Stars
3.9K
Updated
CommitOct 16
Age
10mo
Commits · 90 Days