Architecture — Spec¶

Data Flow¶

Finnhub API ─────────→ scraper/finnhub.py ──────────┐
NAVER Finance ──────→ scraper/naver_finance.py ──────┤
YouTube Channels ───→ scraper/youtube.py ────────────┼──→ NewsArticle (DB)
Reddit (public JSON)→ scraper/reddit.py ─────────────┤
Hacker News Algolia → scraper/hackernews.py ──────────┤
StockTwits API ─────→ scraper/stocktwits.py [DISABLED]─┘

yfinance ───────────→ data/stock_fetcher.py ──────────┐
FinanceDataReader ──→ data/stock_fetcher.py ──────────┼──→ StockPrice (DB)
                                                      │
                                                      │
                    NewsArticle → sentiment/analyzer.py → SentimentScore (DB)
                                       ↓
                                Azure Translator (optional: translate headlines)

                    StockPrice + SentimentScore ──→ models/predictor.py → Prediction (DB)
                                                  → models/trend_analyzer.py → TrendingTopic (DB)
                                                  → Alert (DB, high-confidence signals)

                    Prediction + SentimentScore + TrendingTopic + Alert
                        → reports/daily_summary.py → Claude Haiku API
                        → DailySummary (DB)

                    Alert (high-confidence) → notifications/telegram.py → Telegram Bot API
                    TrendingTopic (sentiment spike) → notifications/telegram.py → Telegram Bot API
                        → TelegramNotification (DB, audit log)

M-STOCK screenshots → portfolio/parser.py (Gemini vision) → PortfolioSnapshot + PortfolioHolding (DB)

financialdatasets.ai → scraper/financialdatasets.py → Fundamentals (DB, market="US")
DART (Korea FSC) ──→ scraper/dart_fetcher.py ─────→ Fundamentals (DB, market="KR")

Capitol Trades ─────→ scraper/political_trades.py ──┐
SEC EDGAR Form 4 ───→ scraper/political_trades.py ──┴──→ PoliticalTrade (DB)

yfinance (earnings) → data/earnings_fetcher.py ──────────→ EarningsEvent (DB, US only)
yfinance (^VIX/^TNX/^IRX) → data/macro_fetcher.py ───→ MacroIndicator (DB, US macro)

Prediction + SentimentScore + Fundamentals (user-triggered)
    → models/debate.py → Claude Haiku API (2 calls: bull + bear)
    → returned directly to frontend (not persisted)

All DB tables ──────→ api/main.py (FastAPI :8000)
                       │  all routes under /api prefix
                       │  serves React SPA from frontend/dist (production)
                       ↓
                    React frontend (Vite :5173 dev / :8000 prod)
                    TanStack Query · Recharts · Tailwind v4

All DB tables ──────→ mcp_server/server.py (stdio MCP server)
                       │  9 tools: list_tickers, get_prediction, get_all_predictions,
                       │           get_sentiment, get_news, get_political_trades,
                       │           get_price_history, get_debate, get_prediction_accuracy
                       ↓
                    Claude Desktop / any MCP client

All pipeline consumers (scrapers, price fetcher, predictor, scheduler) read the active ticker list from the Ticker table via db/tickers.py at runtime — not from config.py. Adding a ticker in the Manage Tickers UI takes effect on the next pipeline run with no restart.

Module Dependencies¶

Module	Depends on	Notes
`config.py`	`.env` via dotenv	No internal deps; holds API keys, thresholds, schedules, seed ticker data, and `POLITICAL_WATCHLIST`
`db/database.py`	`config.py`	SQLAlchemy engine + session factory; seeds `Ticker` table from config on `init_db()`
`db/models.py`	`db/database.py`	ORM models; `utcnow()` defined here
`db/tickers.py`	`db/models.py`	`get_active_tickers()`, `get_ticker_map()` — single source of truth for live ticker lists
`data/stock_fetcher.py`	`db/`	US via yfinance; KR via FinanceDataReader; caller provides ticker list
`scraper/finnhub.py`	`db/`, `config.py`	Finnhub REST API; 60 req/min free tier; reads ticker list from DB
`scraper/naver_finance.py`	`db/`	BeautifulSoup scrape; reads ticker list from DB
`scraper/youtube.py`	`db/`, `config.py`	YouTube transcript fetch; markitdown primary, yt-dlp + Whisper fallback; channel list from `YOUTUBE_CHANNELS` in `config.py`
`scraper/reddit.py`	`db/`	Reddit public JSON API (no auth); scrapes r/wallstreetbets + r/stocks; word-boundary ticker matching; stores one `NewsArticle` per (post, ticker) with URL fragment `#TICKER` for uniqueness
`scraper/hackernews.py`	`db/`	HN Algolia search API (no auth); fetches last-24h stories mentioning 14 US tech tickers; deduped by URL; `source="HackerNews"`
`scraper/stocktwits.py`	`db/`	StockTwits public stream — DISABLED (API restricted); code preserved for future rework
`scraper/translator.py`	`db/`, `config.py`	Azure Translator batch translation with DB caching
`scraper/financialdatasets.py`	`db/`, `config.py`	financialdatasets.ai REST API for quarterly EPS/revenue/debt data (US tickers)
`scraper/dart_fetcher.py`	`db/`, `config.py`, `requests`	Korea FSC DART API for annual revenue/net income/debt/equity data (KR tickers); corp code XML cached locally 30 days; no-op when `DART_API_KEY` unset
`scraper/political_trades.py`	`db/`, `config.py`, `requests`, `bs4`	Capitol Trades HTML scrape + SEC EDGAR Form 4 XML for `POLITICAL_WATCHLIST`; 2 s inter-request delay on Capitol Trades; no API key required
`data/earnings_fetcher.py`	`db/`, `config.py`, `yfinance`	yfinance `get_earnings_dates()` for all US tickers; 0.3 s inter-ticker sleep; KR tickers excluded (yfinance has no KRX earnings data); timestamps normalised to UTC midnight before upsert so DST offset changes don't create duplicate rows
`data/macro_fetcher.py`	`db/`, `yfinance`	yfinance `^VIX` (CBOE VIX), `^TNX` (10Y Treasury), `^IRX` (3M T-bill); computes `yield_spread = 10Y − 3M`; 180-day backfill on first run, 7-day incremental refresh thereafter
`sentiment/analyzer.py`	`db/`	EN: `ProsusAI/finbert` (VADER fallback); KO: `snunlp/KR-FinBert-SC` (keyword lexicon fallback)
`models/volatility_analyzer.py`	`db/`	Realized vol (5/20/60-day), GARCH(1,1) 1-day forecast, K-means volatility regime
`models/predictor.py`	`db/`, `config.py`, `models/volatility_analyzer.py`, `data/macro_fetcher.py`	RF training + inference + rule-based fallback
`models/buffett.py`	`db/`, `models/volatility_analyzer.py`	Value-investing scorecard
`models/trend_analyzer.py`	`db/`, `config.py`	Sector/keyword trend aggregation, ticker heat ranking, trending alerts
`models/debate.py`	`db/`, `config.py`, `anthropic` SDK	User-triggered bull/bear debate: queries Prediction + SentimentScore + Fundamentals, calls Claude Haiku twice (cached system prompt) to generate bull and bear cases; no-op when `ANTHROPIC_API_KEY` unset
`models/sector_rotation.py`	`db/`, `config.py`	Sector rotation signals: aggregates latest `Prediction` rows by sector, computes 5d/20d relative strength from `StockPrice`, ranks sectors by composite score (60% bullish predictions + 40% normalized RS), tags each `rotate_in` / `hold` / `rotate_out`
`reports/daily_summary.py`	`db/`, `config.py`, `anthropic` SDK	Collects 24h signals → prompts Claude Haiku → saves `DailySummary` row; no-op when `ANTHROPIC_API_KEY` unset
`notifications/telegram.py`	`db/`, `config.py`, `requests`	Sends Telegram messages via Bot API (no SDK); records every attempt in `TelegramNotification`; no-op when `TELEGRAM_BOT_TOKEN`/`TELEGRAM_CHAT_ID` unset
`portfolio/parser.py`	`db/models.py`, `config.py`, `google-generativeai` SDK	Gemini 2.5 Flash vision parser for Mirae Asset screenshots
`portfolio/risk.py`	`db/models.py`, `config.py`, `numpy`	Portfolio risk calculations: Sharpe, historical + parametric VaR, max drawdown, sector concentration
`db/auth.py`	`db/models.py`	Auth helpers: bcrypt verification, `UserSession` token creation/validation
`scheduler/jobs.py`	all modules above	17 APScheduler `BackgroundScheduler` jobs (1 disabled)
`api/main.py`	`api/auth.py`, `api/routes/*`, `db/`	FastAPI app; all routers mounted under `/api` prefix; CORS origins configurable via `CORS_ORIGINS` env var; serves React SPA from `frontend/dist` when present
`api/auth.py`	`db/auth.py`, `db/`	`/auth/login`, `/auth/me`, `/auth/logout`
`api/routes/`	domain modules	`prices`, `news`, `sentiment`, `predictions`, `trends`, `portfolio`, `risk`, `tickers`, `jobs`, `admin`, `simulation`, `reports`, `backtest`, `notifications`, `trades`, `debate`, `political_trades`, `earnings`, `sectors`
`frontend/src/App.tsx`	React Router, TanStack Query, `api/auth.ts`	SPA entry point; `AuthGuard`; routes all pages
`frontend/src/api/`	`client.ts`, domain modules	Thin `fetch` wrapper; domain modules call FastAPI endpoints
`frontend/src/pages/`	React components	Overview, Predictions, NewsFeed, SectorTrends, StockDetail, MyPortfolio, ManageTickers, Account, Login, Backtest, TradeJournal, PoliticalTrades, EarningsCalendar, SectorRotation
`main.py`	`scheduler/`, `api/`, `db/`	Init DB → seed tickers → initial load → start scheduler → launch FastAPI

Scheduler Job Interactions¶

Jobs share the same DB session factory and run independently at fixed intervals. Each job queries Ticker at runtime so newly added tickers are included automatically.

Job ID	Interval	Writes	Reads
`fetch_us_news`	15 min	`NewsArticle`, `SentimentScore`	Finnhub API
~~`fetch_stocktwits`~~	disabled	—	StockTwits public API restricted; job commented out
`fetch_kr_news`	30 min	`NewsArticle`, `SentimentScore`	NAVER HTML
`fetch_hn_news`	6 hours	`NewsArticle`, `SentimentScore`	HN Algolia search API (14 US tech tickers)
`fetch_reddit_sentiment`	1 hour	`NewsArticle`, `SentimentScore`	Reddit public JSON API
`update_prices`	1 hour	`StockPrice`	yfinance / FDR
`compute_trends`	30 min	`TrendingTopic`, `TelegramNotification` (on spike)	`NewsArticle`
`run_predictions`	6 hours	`Prediction`, `Alert`, `TelegramNotification` (on alert)	`StockPrice`, `SentimentScore` (inference only)
`check_retrain`	6 hours	`rf_*.pkl`	`Prediction`, `PredictionResult` (accuracy-gated)
`fetch_youtube`	6 hours	`NewsArticle`, `SentimentScore`	YouTube transcript API
`fetch_political_trades`	6 hours	`PoliticalTrade`	Capitol Trades HTML; SEC EDGAR Form 4 XML
`fetch_earnings`	24 hours	`EarningsEvent` (US only)	yfinance `get_earnings_dates()`
`evaluate_predictions`	24 hours	`PredictionResult`	`Prediction`, `StockPrice`
`fetch_fundamentals`	24 hours	`Fundamentals` (US)	financialdatasets.ai API
`fetch_kr_fundamentals`	24 hours	`Fundamentals` (KR)	DART API (Korea FSC)
`generate_daily_summary`	24 hours	`DailySummary`	`Prediction`, `SentimentScore`, `TrendingTopic`, `Alert`; calls Claude Haiku API
`fetch_macro`	24 hours	`MacroIndicator`	yfinance `^VIX`, `^TNX`, `^IRX`; 180-day backfill on first run
`prune_old_data`	24 hours	deletes from all tables	all tables

News fetch jobs call score_unscored_articles() in a batched while-loop (batch_size=500) immediately after each ingestion cycle.

Key Design Decisions¶

DB-backed ticker registry — the Ticker table is the single source of truth; config.py holds seed data only; all runtime consumers read from DB via db/tickers.py
Soft delete on tickers — active=False excludes from pipeline while preserving historical data
DATABASE_URL env var selects the backend — production uses Supabase PostgreSQL; falls back to SQLite when unset (local dev only)
One RF model per horizon, shared across all tickers — individual ticker models would have insufficient training samples
Rule-based fallback — prediction pipeline works on day one before any trained model exists
URL as unique key on news_articles — prevents duplicates across repeated scrape cycles; Reddit appends #TICKER fragment so a post mentioning multiple tickers creates one scoreable row per ticker
Ticker + market discriminator pattern — always filter by both ticker and market in cross-market queries
Batch sentiment scoring — scoring runs once per prediction cycle rather than inline during ingestion
No .KS suffix — FinanceDataReader accepts bare KRX codes natively
Translation cache in DB — title_en and title_ko avoid repeated API calls
Token-based session auth — UserSession rows; FastAPI sets httponly; samesite=lax cookie on login
Retention-based pruning — prune_old_data daily; retention windows tuned so ML always has sufficient history
LLM summary is optional — generate_daily_summary is a no-op when ANTHROPIC_API_KEY is unset; the dashboard's briefing card renders nothing when no summary row exists
Debate is user-triggered, not persisted — models/debate.py runs two Claude Haiku calls on-demand per user request and returns the result directly; no DB table; result is ephemeral and regenerated each time the button is clicked
Debate uses prompt caching — the signal context block is marked cache_control: ephemeral so the second API call (bear case) gets a cache hit on the system prompt, halving the billable input tokens
Political trades require no API key — Capitol Trades is scraped as public HTML (2 s delay, robots.txt respected); EDGAR Form 4 XML is a public SEC endpoint; no credentials needed
STOCK Act delay is structural — congressional trades can be filed up to 45 days after execution; the 6-hour polling interval is conservative by design since the data is inherently delayed