aggregator README

The project's README from github.com/wzdnzd/aggregator. As one markdown file: README.md.

Aggregator - 免费代理池构建工具

GitHub stars GitHub forks GitHub issues License

🎯 项目简介

一个强大的免费代理池构建工具,通过爬取多个平台/网页的代理资源,自动验证、聚合并转换为各种客户端所需格式。

✨ 核心特性

🌐 支持协议

VMess | Trojan | SS | SSR | Snell | Hysteria2 | VLESS | Hysteria | TUIC | AnyTLS | HTTP | SOCKS

🚀 两种使用方式

方式一:process.py(推荐)

完整功能版本 - 支持复杂配置、多源爬取、自定义规则

# 1. 准备配置文件
cp subscribe/examples/config.default.json my-config.json

# 2. 设置环境变量
export PUSH_TOKEN=your_github_token

# 3. 运行处理
python subscribe/process.py -s my-config.json

适用场景:

方式二:collect.py

简化版本 - 快速收集机场订阅

# 直接运行,自动收集并上传到 Gist
python subscribe/collect.py \
    -g username/gist-id \
    -k your-github-token \
    -t clash v2ray singbox

适用场景:

🎁 共享订阅

可前往 Issue #91 获取现成的共享订阅,量大质优。请勿浪费

📊 工作流程

graph LR
    A[开始] --> B{选择方式}
    B -->|简单| C[collect.py]
    B -->|完整| D[process.py]

    C --> E[自动收集机场]
    E --> F[注册账号]
    F --> G[获取订阅]
    G --> H[验证代理]
    H --> I[上传到Gist]

    D --> J[加载配置]
    J --> K[多源爬取]
    K --> L[聚合处理]
    L --> M[质量检测]
    M --> N[格式转换]
    N --> O[推送存储]

⚡ 快速配置

最小配置示例

process.py 配置:

{
    "sites": [
        {
            "name": "example-airport",
            "domain": "example.com",
            "push_to": ["free"]
        }
    ],
    "crawl": {
        "enable": true,
        "telegram": {
            "enable": true,
            "users": {
                "proxy_channel": {
                    "push_to": ["free"]
                }
            }
        }
    },
    "groups": {
        "free": {
            "targets": {"clash": "free-clash"}
        }
    },
    "storage": {
        "engine": "gist",
        "items": {
            "free-clash": {
                "username": "your-username",
                "gist_id": "your-gist-id",
                "filename": "clash.yaml"
            }
        }
    }
}

环境变量:

export PUSH_TOKEN=your_github_token

常用命令

# 快速收集(推荐新手)
python subscribe/collect.py -g username/gist-id -k token

# 完整处理(推荐进阶)
python subscribe/process.py -s config.json

# 仅检查代理活性
python subscribe/process.py -s config.json --check

# 高性能模式
python subscribe/process.py -s config.json -n 128

📚 相关文档

文档 说明 适用人群
完整文档 详细配置说明 进阶用户
English Docs English version International users

🔧 常见问题

问题 解决方案
配置文件错误 python -m json.tool config.json 验证语法
Token 无效 检查 GitHub Token 权限和有效期
网络超时 增加超时 -t 15000 或减少线程 -n 16
无代理输出 检查爬取源配置和网络连接

插件开发

拥有灵活的插件系统,支持自定义爬取目标。欢迎贡献高质量的爬取插件!

🚧 TODO 路线图

架构重构

插件化架构

配置系统优化

代码质量提升

性能与稳定性


⚖️ 免责声明

🙏 致谢

核心依赖

赞助支持

感谢以下组织的赞助支持:

社区贡献

感谢所有为项目贡献代码、提出建议和报告问题的开发者们!

如果这个项目对你有帮助,请给它一个 ⭐

报告问题 · 功能请求 · 贡献代码

aggregator (wzdnzd/aggregator)

Searching Database…

What is this?A free proxy pool builder. It crawls proxy sources, checks which proxies work, aggregates them, and converts them into formats for clients like Clash, V2Ray and SingBox.
Is it a script I run myself or a hosted service?It is a Python command-line tool you run yourself. It has two entry points: process.py for full control and collect.py for a quick grab of airport subscriptions.
How do I get it running?Follow the README quick configuration: write a config.json for process.py, check it with python -m json.tool, set the listed environment variables, then run the script.
What do I need besides Python?The README names Subconverter as the subscription conversion core and Mihomo as the proxy testing engine. It also mentions a GitHub Token, which matters for GitHub-based storage.
What commonly goes wrong?If you get no proxies, check your crawl sources and network connection. If you hit timeouts, raise the timeout with -t 15000 or lower the thread count with -n 16.
Which entry point should I use, process.py or collect.py?process.py is for users who need fine control over crawl rules, multiple output groups, custom storage or scheduled runs. collect.py is for quick personal use with no complex setup.
Can I add my own proxy sources?Yes. The README describes a plugin system for custom crawl targets and invites contributors to add new crawler plugins.
Which sites does it crawl for proxies?Telegram, GitHub, Google, Yandex, Twitter and other web pages. Results can be stored in GitHub Gist, PasteGG or Imperial.
How does it check that proxies actually work?It automatically tests proxy liveness and quality. The README credits Mihomo as the proxy testing engine.
What is planned next?The roadmap covers turning crawlers and storage backends into plugins, adding core interfaces, Pydantic-based config models, structured logging, and better concurrency and retries.
Creator
wzdnzd
Size
1.1 GB
Stars
7.6K
Updated
CommitOct 23
Age
7y
Commits · 90 Days