Description:
Our fetchers in 'fetchers/' directory pulls from aggregators (Bigevent, dev.events, papercall.io, sessionize.com) plus the hardcoded flagship watchlist, using standard Python requests. Some of these calls have hardcoded Chrome User-Agents, others have none. Running this nightly via GitHub Actions will eventually get us blocked with 429 Too Many Requests by Cloudflare or similar.
Tasks:
To claim: comment below so we don't get duplicate work.
Description:
Our fetchers in 'fetchers/' directory pulls from aggregators (Bigevent, dev.events, papercall.io, sessionize.com) plus the hardcoded flagship watchlist, using standard Python
requests. Some of these calls have hardcoded Chrome User-Agents, others have none. Running this nightly via GitHub Actions will eventually get us blocked with429 Too Many Requestsby Cloudflare or similar.Tasks:
utils/user_agents.py) containing an array of 10-15 modern, real-world browser User-Agent strings (Chrome, Firefox, Safari, Edge — mix of desktop/mobile)get_random_user_agent()) that returns a random one usingrandom.choice()requestscalls instead of hardcoded or missing User-Agent headersrequests-based aggregator scrapersTo claim: comment below so we don't get duplicate work.