What is this?
Photon is a fast web crawler built for OSINT (open-source intelligence) gathering, extracting URLs, emails, social media accounts, secret keys, subdomains and more while it crawls a target site.
Photon is a fast web crawler built for OSINT (open-source intelligence) gathering, extracting URLs, emails, social media accounts, secret keys, subdomains and more while it crawls a target site.Is this a library or a command-line tool?It's a Python command-line crawler you run against a target URL, not a library you import into code.
It's a Python command-line crawler you run against a target URL, not a library you import into code.How do I run it?Run it as a Python script pointed at a target, or launch it via the official lightweight Python-Alpine Docker image and check results in the mounted loot folder.
Run it as a Python script pointed at a target, or launch it via the official lightweight Python-Alpine Docker image and check results in the mounted loot folder.Do I need anything besides Python to use it?Just Python to run the script, or Docker if you prefer the containerized version; no API keys are required for basic crawling.
Just Python to run the script, or Docker if you prefer the containerized version; no API keys are required for basic crawling.Is crawling resource-heavy?Yes, Photon relies on smart thread management to stay fast, but the README itself notes that crawling can still be resource intensive on large targets.
Yes, Photon relies on smart thread management to stay fast, but the README itself notes that crawling can still be resource intensive on large targets.What kind of data can Photon pull out while crawling?It extracts in-scope and out-of-scope URLs, URLs with parameters, intel like emails and social media accounts, files (pdf, png, xml etc.), secret/API keys, JavaScript files and their endpoints, custom regex matches, and subdomain/DNS data, saved organized or exportable as JSON.
It extracts in-scope and out-of-scope URLs, URLs with parameters, intel like emails and social media accounts, files (pdf, png, xml etc.), secret/API keys, JavaScript files and their endpoints, custom regex matches, and subdomain/DNS data, saved organized or exportable as JSON.What are Photon's plugins for?Built-in plugins include wayback (pull archived URLs from archive.org as seeds), dnsdumpster, and an Exporter for saving results.
Built-in plugins include wayback (pull archived URLs from archive.org as seeds), dnsdumpster, and an Exporter for saving results.How do I keep Photon updated?Run it with the --update option, which checks for and installs updates without losing any of your saved crawl data.
Run it with the --update option, which checks for and installs updates without losing any of your saved crawl data.Why would I use the --wayback option?It fetches URLs archived by archive.org to use as additional seeds, helping Photon discover pages that may no longer be linked live on the site.
It fetches URLs archived by archive.org to use as additional seeds, helping Photon discover pages that may no longer be linked live on the site.What are similar projects?social-analyzer, bbot, maigret, theHarvester, phoneinfoga, Osintgram, sherlock, Sublist3r
social-analyzer, bbot, maigret, theHarvester, phoneinfoga, Osintgram, sherlock, Sublist3rWhich topics is it in?osint-tools, spider, web-scraper, information-gathering, reconnaissance, crawler, osint, pentesting, hacking, web-scraping, json, security-tools
osint-tools, spider, web-scraper, information-gathering, reconnaissance, crawler, osint, pentesting, hacking, web-scraping, json, security-tools