Market Pulse — End-to-End Architecture¶
A personal stock prediction tracker that watches markets, reads the news, and tells you which way a stock is likely to move.
The Big Picture¶
INTERNET YOUR SERVER YOU
───────── ─────────── ───
Finnhub API ──────────┐
NAVER Finance ────────┤ ┌─────────────────────────────┐
Reddit ───────────────┤ │ │
YouTube ──────────────┼─►│ DATA COLLECTION LAYER │
yfinance (prices) ────┤ │ scheduler/jobs.py │
KRX (KR prices) ──────┤ │ │
Capitol Trades ───────┘ └──────────┬────────────────-─┘
│
▼
┌──────────────────────┐
│ DATABASE (Supabase) │
│ PostgreSQL / SQLite │
└──────────┬───────────┘
│
┌─────────────────┼──────────────────┐
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────────┐
│ SENTIMENT │ │ ML │ │ AI AGENTS │
│ PIPELINE │ │ PREDICTION │ │ (LLM-powered) │
│ FinBERT EN │ │ Random │ │ Daily summary │
│ FinBERT KO │ │ Forest │ │ Debate │
└──────┬───────┘ └──────┬───────┘ └────────┬─────────┘
│ │ │
└────────┬────────┘ │
▼ │
┌─────────────────┐ │
│ FastAPI │◄───────────────────┘
│ :8000 │
└────────┬────────┘
│
┌────────────┼──────────────┐
▼ ▼ ▼
┌────────────┐ ┌─────────┐ ┌────────────┐
│ React UI │ │Telegram │ │ Claude │
│ :5173 │ │ Bot │ │ Desktop │
│(your │ │(phone │ │ (MCP) │
│ browser) │ │ alerts) │ │ │
└────────────┘ └─────────┘ └────────────┘
| What | Plain English |
|---|---|
| 15 min | How often fresh US news is fetched |
| 1 hour | How often stock prices are updated |
| 6 hours | How often ML predictions are refreshed |
| 3 horizons | Predictions made for 1 day, 7 days, and 30 days out |
| 29 features | Inputs the ML model uses to make each prediction |
| 2 markets | US stocks (NYSE/NASDAQ) and KR stocks (KRX) |
How It Works¶
Step 1 — Collect the Data¶
PRICE DATA NEWS DATA
────────── ─────────
yfinance ──────────┐ Finnhub API ─────────────┐
(US stocks) │ (US financial news) │
▼ │
FinanceDataReader ─┤ NAVER Finance scraper ────┤
(KR stocks) │ (Korean news, no API key) │
│ │
▼ Reddit (r/WSB, r/stocks) ─┤
┌────────────────┐ (retail investor chatter) │
│ StockPrice │ │
│ table (DB) │ YouTube transcripts ───────┤
└────────────────┘ (channel videos → text) │
│
Capitol Trades / SEC ──────┘
(politician stock trades)
│
▼
┌────────────────┐
│ NewsArticle │
│ table (DB) │
└────────────────┘
- Input: External APIs and public websites
- Output:
StockPricerows,NewsArticlerows in the database - Who does it:
data/stock_fetcher.py,scraper/finnhub.py,scraper/naver_finance.py, etc. - When: APScheduler runs these jobs every 15 min – 24 hours depending on the source
Step 2 — Score the Sentiment¶
NewsArticle (DB)
│
│ Is the article in English or Korean?
│
┌────┴─────────────────────────────────────────┐
│ │
▼ English Korean ▼
┌────────────────────┐ ┌─────────────────────┐
│ FinBERT (EN) │ │ KR-FinBERT (KO) │
│ AI model trained │ │ AI model trained │
│ on financial text │ │ on Korean finance │
│ │ │ │
│ VADER fallback │ │ Keyword lexicon │
│ (if model fails) │ │ fallback │
└────────┬───────────┘ └──────────┬──────────┘
│ │
└──────────────┬──────────────────────┘
▼
┌──────────────────────┐
│ SentimentScore (DB) │
│ │
│ score: -1.0 to +1.0 │
│ label: pos/neg/neu │
└──────────────────────┘
- Input: Raw article text from
NewsArticle - Output:
SentimentScorerows — one number (-1 to +1) per article - Who does it:
sentiment/analyzer.py - Rule: Score ≥ 0.15 = positive, ≤ -0.15 = negative, in between = neutral
Step 3 — Predict the Direction¶
StockPrice (DB) SentimentScore (DB) MacroIndicator (DB)
│ │ │
│ price history │ recent sentiment │ VIX, yield spread
└──────────────────────────┴───────────────────────────┘
│
▼
┌─────────────────────────────┐
│ Feature Engineering │
│ 29 inputs per stock: │
│ • Momentum (5d, 20d) │
│ • RSI, MACD, Bollinger │
│ • Sentiment avg/std/count │
│ • Volatility regime │
│ • EPS surprise │
│ • Days to earnings │
│ • VIX level │
└──────────────┬──────────────┘
│
▼
┌──────────────────────────────┐
│ Random Forest Model │
│ 200 trees, depth 8 │
│ One model per horizon │
│ │
│ rf_1d.pkl │
│ rf_7d.pkl │
│ rf_30d.pkl │
└──────────────┬───────────────┘
│
┌──────────────┴───────────────┐
│ │
▼ ▼
┌─────────────────┐ ┌──────────────────┐
│ Prediction(DB) │ │ Alert (DB) │
│ │ │ │
│ direction: up │ │ High-confidence │
│ horizon: 7d │ │ signals only │
│ confidence: % │ │ (→ Telegram) │
└─────────────────┘ └──────────────────┘
- Input: Price history + sentiment scores + macro indicators
- Output:
Predictionrows (up/down/neutral, per stock, per horizon) - Who does it:
models/predictor.py - Fallback: If no trained model exists, rule-based heuristics kick in automatically
Step 4 — The Scheduler (The Engine Room)¶
┌──────────────────────────────────────────────────────────┐
│ APScheduler │
│ │
│ Every 15 min ── fetch_us_news │
│ Every 30 min ── fetch_kr_news │
│ Every 30 min ── compute_trends │
│ Every 1 hr ── update_prices │
│ Every 1 hr ── fetch_reddit_sentiment │
│ Every 6 hr ── run_predictions │
│ Every 6 hr ── check_retrain │
│ Every 6 hr ── fetch_youtube │
│ Every 6 hr ── fetch_political_trades │
│ Every 24 hr ── generate_daily_summary (Claude Haiku) │
│ Every 24 hr ── fetch_fundamentals │
│ Every 24 hr ── fetch_earnings │
│ Every 24 hr ── fetch_macro │
│ Every 24 hr ── evaluate_predictions │
│ Every 24 hr ── prune_old_data │
└──────────────────────────────────────────────────────────┘
- Who does it:
scheduler/jobs.py - Runs inside: The main Python process (
main.py) - No manual intervention needed — everything above runs automatically
Step 5 — Serve the Data (API Layer)¶
Database (all tables)
│
▼
┌───────────────────────────────────────────────┐
│ FastAPI :8000 │
│ │
│ /api/prices /api/predictions │
│ /api/news /api/sentiment │
│ /api/trends /api/portfolio │
│ /api/debate /api/political-trades │
│ /api/earnings /api/sectors │
│ /auth/login /auth/me │
└──────────────────────────┬────────────────────┘
│
┌──────────────┴──────────────┐
│ │
▼ ▼
┌─────────────────┐ ┌──────────────────┐
│ React Frontend │ │ MCP Server │
│ (browser) │ │ (Claude │
│ │ │ Desktop) │
└─────────────────┘ └──────────────────┘
- Who does it:
api/main.py+ all files inapi/routes/ - Auth: Cookie-based login —
mp_sessioncookie, 30-day TTL - Port: 8000 in production, same port serves the built React app
Step 6 — The Dashboard (What You See)¶
Browser → http://localhost:5173 (dev) or :8000 (prod)
┌──────────────────────────────────────────────────────────┐
│ React SPA │
│ │
│ ┌─────────────┐ ┌────────────────┐ ┌──────────────┐ │
│ │ Overview │ │ Predictions │ │ News Feed │ │
│ │ (market │ │ (up/down/ │ │ (articles + │ │
│ │ summary) │ │ neutral) │ │ sentiment) │ │
│ └─────────────┘ └────────────────┘ └──────────────┘ │
│ │
│ ┌─────────────┐ ┌────────────────┐ ┌──────────────┐ │
│ │ My │ │ Sector │ │ Political │ │
│ │ Portfolio │ │ Trends │ │ Trades │ │
│ │ (holdings) │ │ (AI/semicon/ │ │ (congress) │ │
│ └─────────────┘ │ energy...) │ └──────────────┘ │
│ └────────────────┘ │
│ ┌─────────────┐ ┌────────────────┐ │
│ │ Earnings │ │ Bull vs Bear │ │
│ │ Calendar │ │ Debate │ │
│ │ │ │ (AI-generated)│ │
│ └─────────────┘ └────────────────┘ │
└──────────────────────────────────────────────────────────┘
- Tech: React 19 + TypeScript + Tailwind CSS + Recharts
- Data fetching: TanStack Query (auto-refresh, caching)
- Auth guard: All pages require login except
/login
AI Agents¶
Four AI systems run on top of the data pipeline. Three are always-on; one runs overnight.
You (Telegram) Users (chat UI) You (dev)
│ │ │
▼ ▼ ▼
Secretary Agent Chat Agent Night Agent
(ticket triage) (primary product) (overnight dev)
│ │ │
│ GitHub Projects │ DeepSeek 4 │ Claude Code
▼ │ via OpenRouter ▼
Night Agent picks Supabase DB agent/integrated
up tickets (read-only) branch → PR
Secretary Agent (Telegram → GitHub Projects)¶
You ──► Telegram message ("add earnings heatmap")
│
▼
secretary/bot.py (10s poll loop)
│
▼
intent.py ──► DeepSeek V3 API
│ model: deepseek-chat
│ ~500 in / 200 out tokens per call
│ ~$0.0004 per message
▼
{ intent, title, body, priority }
│
┌────┴─────────────────────────────┐
│ new_feature / new_bug │ status / question
▼ ▼
github_projects.py board query → Telegram reply
gh issue create
gh project item-add
GraphQL set Status = Backlog
│
▼
Telegram: "✅ Ticket #42 added to Backlog"
Night Agent (overnight autonomous dev)¶
GitHub Projects "Ready" column
│ manual trigger or cron
▼
Claude Code session (.claude/agent-prompt.md → /night-flow skill)
│
│ KST 06:00 hard stop
│ runs at most ONE code-producing stage:
│
├──► Bug Triage → debugger sub-agent (Opus)
├──► Code Review → code-reviewer (Sonnet)
├──► Feature Dev → domain sub-agent (Sonnet/Opus)
└──► QA → test-engineer (Haiku)
│ then always:
├──► Security scan (Haiku)
├──► Update docs/agent/agent-log.md
├──► Write ~/.night-mem vault
└──► Merge → agent/integrated → push
│
▼
PR open for your morning review
Feature Dev sub-agent routing:
| Ticket touches | Sub-agent | Model |
|---|---|---|
| scraper / DB / scheduler | data-engineer | Sonnet |
| ML / sentiment | ml-analyst | Opus |
| React / frontend | frontend-engineer | Sonnet |
| CI / Docker / Makefile | devops-engineer | Sonnet |
| auth / security / RLS | security-reviewer | Sonnet |
| crashes / stack traces | debugger | Opus |
Platform Chat Agent (⚠️ not yet built — primary product feature)¶
The chat agent is the core user-facing product. Users interact with it conversationally to get trade signals, manage their portfolio, parse breaking news, and surface market trends. It is the only agent that scales with user growth — every conversation turn costs money, making it the primary cost driver at scale.
User ──► "is NVDA a buy right now?"
│
▼
┌────────────────────────────────────────────┐
│ Chat API endpoint │
│ (to be built — FastAPI route │
│ or standalone service) │
└──────────────────┬─────────────────────────┘
│
▼
┌────────────────────────────────────────────┐
│ Context assembly (before LLM call) │
│ │
│ • conversation history (this session) │
│ • user portfolio holdings │
│ • latest ML predictions (1d/7d/30d) │
│ • recent news headlines + sentiment │
│ • sector trends + macro indicators │
│ • relevant stock prices │
│ │
│ → ~5k–20k tokens assembled per turn │
└──────────────────┬─────────────────────────┘
│
▼
┌────────────────────────────────────────────┐
│ DeepSeek 4 via OpenRouter │
│ model: deepseek/deepseek-chat │
│ (deepseek-4 when available) │
│ │
│ $0.27/MTok in · $1.10/MTok out │
│ + ~10% OpenRouter markup │
│ → ~$0.004–$0.006 per turn │
└──────────────────┬─────────────────────────┘
│
│ may issue tool calls back to DB:
│ • fetch latest price for a ticker
│ • pull recent news for a ticker
│ • get prediction for a horizon
│ • read portfolio holdings
▼
┌────────────────────────────────────────────┐
│ Supabase DB (read-only) │
│ same database the dashboard reads │
└──────────────────┬─────────────────────────┘
│
▼
Response streamed to user
What users can ask¶
| Capability | Example query |
|---|---|
| Trade signals | "Is NVDA a buy right now?" |
| Portfolio review | "How is my portfolio positioned vs the market?" |
| News parsing | "What's moving Samsung today?" |
| Sentiment summary | "What's the mood on Korean tech stocks this week?" |
| Sector trends | "Which sectors are showing the strongest momentum?" |
| Macro context | "How does the current VIX level affect my holdings?" |
| Earnings awareness | "Which of my stocks have earnings coming up?" |
| Predictions | "What does the model think TSLA does over the next month?" |
Memory model¶
| Source | What it provides | Scope |
|---|---|---|
| Supabase DB | All market data, predictions, portfolio, news | Persistent, shared across sessions |
| Conversation history | What was said earlier in this chat | Session-only, not persisted |
| Night agent vault | (not used) — engineering memory, not user context | — |
Per-user persistent memory (e.g. "remember I'm bullish on semis") is not yet built — would live as a user preferences table in Supabase when added.
What is TBD¶
- Chat API endpoint design (FastAPI route vs standalone service)
- Authentication — tying chat sessions to user accounts
- Conversation history storage and retrieval strategy
- Tool call schema — which DB queries the model can trigger and how
- Streaming implementation
- Frontend chat UI integration
In-Pipeline Agents (scheduler-driven)¶
| Agent | Model | When | Cost |
|---|---|---|---|
| Daily Summary | Claude Haiku | 24h scheduler | ~$0.001/day |
| Bull/Bear Debate | Claude Haiku ×2 | User button click | ~$0.002/click |
| Portfolio Parser | Gemini 2.5 Flash | Screenshot upload | ~$0.01/screenshot |
| Sentiment (EN) | FinBERT (local) | News ingestion | Free |
| Sentiment (KO) | KR-FinBERT (local) | News ingestion | Free |
Agent Cost Summary¶
| Agent | Per-event cost | Notes |
|---|---|---|
| Chat Agent (per turn) | ~$0.004–$0.006 | DeepSeek 4 via OpenRouter — ~5k–20k tokens in / 1k–3k out + 10% markup |
| Secretary (per message) | ~$0.005–$0.02 | DeepSeek V3 — classification + context read + ticket generation |
| Night Agent — code review | ~$0.50 | Sonnet code-reviewer + Haiku QA + security scan |
| Night Agent — Sonnet feature | ~$0.70 | Sonnet feature dev + Haiku QA + security scan |
| Night Agent — Opus session | ~$2.80 | Opus bug triage or ML feature dev + QA + security |
| Daily Summary | ~$0.001/day | Haiku, small prompt |
| Bull/Bear Debate | ~$0.002/click | Haiku ×2 |
| Portfolio Parser | ~$0.01/screenshot | Gemini 2.5 Flash |
Night agent pricing basis: Opus ($15/MTok in, $75/MTok out), Sonnet ($3/MTok in, $15/MTok out), Haiku ($0.80/MTok in, $4/MTok out). Chat agent pricing basis: DeepSeek 4 via OpenRouter (~$0.27/MTok in, $1.10/MTok out + ~10% OpenRouter markup).
Daily Cost by Traffic Level¶
Traffic assumptions:
| Low | Medium | High | |
|---|---|---|---|
| Chat agent turns | 30 | 200 | 800 |
| Secretary messages | 2 | 5 | 15 |
| Night agent stage | Code review (Sonnet) | Feature dev (Sonnet) | Bug triage or ML (Opus) |
| Bull/Bear debate clicks | 1 | 3 | 8 |
| Portfolio screenshot uploads | 0 | 1 | 2 |
| Daily summary | 1 | 1 | 1 |
Cost breakdown (out-of-pocket only — night agent runs on work Claude account token allocation):
| Component | Low | Medium | High |
|---|---|---|---|
| Chat agent (DeepSeek via OpenRouter) | $0.15 | $1.00 | $4.40 |
| Secretary (DeepSeek V3) | $0.02 | $0.06 | $0.22 |
| Night agent | $0 ¹ | $0 ¹ | $0 ¹ |
| Bull/Bear debate | $0.00 | $0.01 | $0.02 |
| Portfolio parser (Gemini) | $0.00 | $0.01 | $0.02 |
| Daily summary (Haiku) | $0.00 | $0.00 | $0.00 |
| Out-of-pocket total | ~$0.17 | ~$1.08 | ~$4.66 |
¹ Night agent uses leftover tokens from work Claude account — not billed separately.
Out-of-pocket daily cost (USD)
$4.50 ┤ ████
│ ████
$3.50 ┤ ████
│ ████
$2.50 ┤ ████
│ ████
$1.50 ┤ ████
│ ████ ████
$0.50 ┤ ████ ████
│ ████ ████ ████ ████
$0.17 ┤ ████ ████ ████ ████ ████
│ ████ ████ ████ ████ ████
$0.00 └─────────────────────────────────────────────
Low Medium High
████ Chat agent (dominant) remainder = secretary + misc
The chat agent is the only cost that scales with usage. Everything else is fixed or near-zero. At low traffic out-of-pocket spend is trivial (~$0.17/day); at high traffic it's driven almost entirely by chat volume.
Notifications¶
Two types of alerts trigger a Telegram message to your phone:
1. HIGH-CONFIDENCE PREDICTION
┌─────────────────────────────────────────────┐
│ run_predictions job │
│ → confidence > threshold? │
│ → yes → Alert (DB) → Telegram message sent │
└─────────────────────────────────────────────┘
2. SENTIMENT SPIKE
┌───────────────────────────────────────────────┐
│ compute_trends job │
│ → sector sentiment jumped significantly? │
│ → yes → TrendingTopic (DB) → Telegram sent │
└───────────────────────────────────────────────┘
Every send is logged in TelegramNotification (DB)
for deduplication — same alert won't spam you twice.
What Can Go Wrong¶
| Failure | What you'd see | How it's handled |
|---|---|---|
| Finnhub rate limit (60 req/min) | US news stops updating | Fetcher stays under limit by design; free tier covers all tickers in one cycle |
No ANTHROPIC_API_KEY |
Daily summary and debate disappear | Both features silently no-op — dashboard just shows nothing |
| No trained ML model | Predictions still appear | Rule-based fallback activates automatically |
| Supabase connection drops | API errors, empty dashboard | SQLite fallback available in local dev; prod relies on Supabase uptime |
| KR ticker wrong format | No prices fetched for that stock | Bare 6-digit code required — no .KS suffix |
| FinBERT model load fails | EN sentiment stops scoring | VADER lexicon fallback activates automatically |
Glossary¶
| Term | Plain English |
|---|---|
| APScheduler | A Python library that runs tasks on a timer automatically — like a cron job inside the app |
| FinBERT | An AI model specifically trained to understand financial news and rate it as positive, negative, or neutral |
| FastAPI | The web server that handles all data requests from the browser and other apps |
| Finnhub | A financial data company whose API we use to pull US stock news |
| FDR (FinanceDataReader) | A Python library for pulling Korean stock prices from KRX |
| Gemini | Google's AI model — used here to read screenshots of your brokerage account |
| GraphQL | A way to query APIs using flexible, structured questions — used for GitHub Projects |
| horizon | How far into the future a prediction looks: 1 day, 7 days, or 30 days |
| KRX | Korea Exchange — the stock market for Korean companies |
| MCP | Model Context Protocol — lets Claude Desktop talk directly to our app's data |
| ML | Machine learning — the app trains a model on historical data to predict future stock direction |
| NAVER Finance | Korea's main financial news website — we scrape it since there's no public API |
| Random Forest | The ML algorithm used for predictions — it builds 200 decision trees and averages their votes |
| React | The JavaScript framework powering the dashboard you see in your browser |
| RSI | Relative Strength Index — a number (0–100) that indicates if a stock is overbought or oversold |
| sentiment | A score (-1 to +1) measuring whether news about a stock is positive, negative, or neutral |
| Supabase | The hosted PostgreSQL database service where all data is stored in production |
| TanStack Query | A React library that keeps the dashboard data fresh without manual refreshing |
| ticker | The short code that identifies a stock — e.g. AAPL for Apple, 005930 for Samsung |
| UTC | Universal Coordinated Time — all timestamps in the app are stored in UTC to avoid timezone confusion |
| VIX | CBOE Volatility Index — measures how much fear/uncertainty exists in the US market overall |
| yfinance | A Python library for pulling US stock prices from Yahoo Finance |
Last updated: 2026-06-02 · Owner: Michael Ko