data-scraper-agent
Build a fully automated AI-powered data collection agent for any public source — job boards, prices, news, GitHub, sports, anything. Runs on a schedule, enriches data with a free LLM (Gemini Flash), stores results in Notion/Sheets/Supabase, and learns from user feedback. Runs 100% free on GitHub Actions.
- Kind
- Skill
- Works with
- Claude Code, Codex
- Source
- affaan-m/ECC @ d3b8a3e90890
- Path
- skills/data-scraper-agent/SKILL.md
- License
- MIT
- Jobs
- Automation, Implementation
InstallCopy it from the source repository, or sign in and let the assistant adapt it for your project.
# Data Scraper Agent
Build a production-ready, AI-powered data collection agent for any public data source.
Runs on a schedule, enriches results with a free LLM, stores to a database, and improves over time.
**Stack: Python · Gemini Flash (free) · GitHub Actions (free) · Notion / Sheets / Supabase**
## When to Activate
- User wants to gather or monitor any public website or API
- User says "build a bot that checks...", "monitor X for me", "collect data from..."
- User wants to track jobs, prices, news, repos, sports scores, events, listings
- User asks how to automate data collection without paying for hosting
- User wants an agent that gets smarter over time based on their decisions
## Core Concepts
### The Three Layers
Every data collection agent has three layers:
```
COLLECT → ENRICH → STORE
│ │ │
Scraper AI (LLM) Database
runs on scores/ Notion /
schedule summarises Sheets /
& classifies Supabase
```
### Free Stack
| Layer | Tool | Why |
|---|---|---|
| **Scraping** | `requests` + `BeautifulSoup` | No cost, covers 80% of public sites |
| **JS-rendered sites** | `playwright` (free) | When HTML fetching fails |
…