ScraperService
Modular high-performance Python scraper with specialized extractors for News, Finance, and generic sites.
Advanced Build
Interactive Preview
Initialising Operative...
Architecture
Component Architecture
ScraperService/
Prop Usage
| Prop | Type | Default | Description |
|---|---|---|---|
targetUrlrequired | string | — | The URL to scrape. |
scraperType | "generic" | "news" | "finance" | "generic" | Selects the specialised extractor pipeline. |
onResult | (data: PageData) => void | — | Callback fired with the extracted page data. |
apiBase | string | "http://localhost:8000" | Base URL of the running Python scraper service. |
Usage
import { ScraperService } from '@/ui/features/ScraperService';
<ScraperService
targetUrl="https://news.ycombinator.com"
scraperType="news"
onResult={(data) => console.log(data)}
/>Technical Setup Guide
Follow these steps to initialize the Python scraping engine on your local machine.
01
Environment
cd scraper-service
python3 -m venv venv
source venv/bin/activate02
Dependencies
pip install -r requirements.txt03
Launch Node
python api/main.pyPre-configured Handshake
The frontend Scraper component is already mapped to localhost:8000. Once your local node is active, use the interactive preview above to trigger real-world extractions and modular data ingestion logs.