Scraping Improvements — Plan¶

Motivation¶

The current scraping stack uses plain requests + BeautifulSoup for NAVER Finance and raw Finnhub article URLs. Two pain points have emerged: (1) Korean financial sites increasingly apply bot-detection that blocks standard requests TLS fingerprints, causing silent fetch failures in naver_finance.py; (2) article bodies fed to FinBERT are often noisy — full HTML pages with nav, ads, and boilerplate — which degrades sentiment quality. A small set of targeted library swaps can address both without restructuring the pipeline.

Scope¶

In scope: - Swap requests for curl_cffi in scraper/naver_finance.py to bypass TLS fingerprint bot detection - Add trafilatura for clean article body extraction in scraper/finnhub.py (fetch full article text from URLs Finnhub returns, strip boilerplate, feed to FinBERT) - Replace BeautifulSoup with selectolax in naver_finance.py for faster HTML parsing - Add newspaper4k as a fallback extractor when trafilatura returns empty/short text

Out of scope: - Playwright / headless browser scraping (not needed yet — NAVER is server-rendered) - Scrapy framework migration (too much structural churn for the current scraper volume) - New scraping targets (separate plan per source) - Rate-limit or retry architecture changes

Approach¶

1. Dependencies — `requirements.txt`¶

Add:

curl_cffi>=0.7
trafilatura>=1.12
selectolax>=0.3
newspaper4k>=0.9

2. `curl_cffi` swap — `scraper/naver_finance.py`¶

Replace:

import requests
response = requests.get(url, headers=HEADERS, timeout=10)

With:

from curl_cffi import requests as cffi_requests
response = cffi_requests.get(url, impersonate="chrome", timeout=10)

impersonate="chrome" mimics Chrome TLS fingerprint — the main signal NAVER uses to detect scrapers
Drop the custom HEADERS dict (curl_cffi sets realistic browser headers automatically)
Apply to all requests.get calls in the file; keep the same response interface (.text, .status_code, .raise_for_status())

3. `selectolax` swap — `scraper/naver_finance.py`¶

Replace BeautifulSoup HTML parsing with selectolax Lexbor parser:

# Before
from bs4 import BeautifulSoup
soup = BeautifulSoup(response.text, "html.parser")
items = soup.select("ul.newsList > li")

# After
from selectolax.parser import HTMLParser
tree = HTMLParser(response.text)
items = tree.css("ul.newsList > li")

.text → .text() on nodes; .get("href") stays the same
Benchmark first — if page sizes are small (<50 KB), skip this step (BeautifulSoup is fine)

4. Article body extraction — `scraper/finnhub.py`¶

Finnhub returns headline + URL but no body text. Add an optional body-fetch step:

Create scraper/article_extractor.py:

import trafilatura
from newspaper import Article  # newspaper4k

def extract_article_text(url: str, timeout: int = 8) -> str | None:
    """Fetch and extract clean article body from a URL. Returns None on failure."""
    try:
        from curl_cffi import requests as cffi_requests
        html = cffi_requests.get(url, impersonate="chrome", timeout=timeout).text
    except Exception:
        return None

    # Primary: trafilatura
    text = trafilatura.extract(html, include_comments=False, include_tables=False)
    if text and len(text) > 100:
        return text

    # Fallback: newspaper4k
    try:
        article = Article(url)
        article.set_html(html)
        article.parse()
        return article.text or None
    except Exception:
        return None

In scraper/finnhub.py, after fetching article metadata: - Call extract_article_text(article_url) for articles where body is empty or absent - Store result in NewsArticle.content (existing column) before upsert - Wrap in try/except — body extraction is best-effort, never block the main fetch loop - Add a config flag FINNHUB_FETCH_ARTICLE_BODY = True in config.py to toggle (off by default until tested)

5. Scheduler impact¶

No scheduler changes needed — the extraction runs inline within job_fetch_us_news(). Monitor job duration after enabling body fetch; if it adds >2 min, move extraction to a separate async post-processing step.

6. Testing¶

scripts/test_naver_scrape.py — run NAVER fetch manually, confirm articles returned with curl_cffi
scripts/test_article_extract.py — pass 5 Finnhub URLs through extract_article_text(), inspect output length and quality before/after
Confirm FinBERT sentiment scores change (or improve) on articles that previously had only headlines

Long-Term: Scrapling as Unified Foundation¶

Scrapling is worth revisiting when a second or third scraping target is added (e.g. Capitol Trades from the political trades plan). It replaces curl_cffi + selectolax + partial Playwright with a single library, adds smart element tracking (relocates elements after site redesigns by similarity — directly addresses the NAVER selector fragility risk), and bundles Cloudflare bypass for US financial sites.

Hold off for now — Scrapling is relatively young and the NAVER scraper is working. The forcing function to migrate is adding a new scraping target, not fixing the current one.

MCP server (scrapling[ai]): Scrapling ships a built-in MCP server that pre-extracts targeted content from pages before passing to an AI model. This reduces token consumption significantly vs. dumping raw HTML at Claude. Relevant for FinTrack if we ever build agentic scraping tasks (e.g. letting Claude drive a research fetch) or want to pipe scraped content into the daily briefings pipeline more efficiently. Install with pip install "scrapling[ai]" and wire into .claude/settings.json when the time comes.

Open Questions¶

curl_cffi Windows compatibility — verify it installs cleanly on Windows (pre-built wheels available for most platforms, but worth confirming in dev env)
Body fetch rate limits — fetching full articles from publisher sites (WSJ, Reuters, etc.) may trigger their own bot detection or paywalls. Start with FINNHUB_FETCH_ARTICLE_BODY = False; enable after testing a sample of URLs.
selectolax worth it? — only valuable if NAVER parsing is a measured bottleneck. Profile before committing the swap.
newspaper4k maintenance — active fork of newspaper3k; API-compatible but confirm import is from newspaper import Article (not newspaper4k).

Status¶

draft