--- title: Crypto Narrative Terminal emoji: ๐Ÿ“ˆ colorFrom: yellow colorTo: gray sdk: docker app_port: 7860 pinned: false --- # Crypto News & Narrative Analysis A fully free, local tool that monitors crypto narratives in real-time by aggregating news, Reddit, and market data โ€” then scoring and ranking narratives using NLP. > **Zero paid APIs. No subscriptions. Everything runs on your machine.** --- ## What It Does Crypto markets are narrative-driven. This tool: 1. **Collects** news from 10 RSS feeds + Reddit (10 subreddits) + CoinGecko + Fear & Greed Index 2. **Detects** 16 pre-defined narratives (ETF, AI tokens, RWA, Memecoins, Regulation, etc.) 3. **Scores** each narrative by volume ร— recency ร— sentiment ร— engagement ร— trending-coin bonus 4. **Extracts** coin mentions and emerging keywords using NLP (VADER + spaCy + TF-IDF) 5. **Displays** a live Rich terminal dashboard 6. **Exports** JSON reports for further analysis --- ## Dashboard Preview ``` โ”โ”โ”โ”โ”โ”โ”โ” Crypto News & Narrative Analysis โ”โ”โ”โ” 2024-01-15 14:32 UTC | 847 items analysed โ”โ”โ”โ” Market Cap 24h Volume Dominance Fear & Greed $2.41T +3.2% $98.7B BTC 52.1% ETH 17.3% ๐Ÿš€ 78 โ€” Greed Trending: #1 Bittensor (TAO) ยท #2 dogwifhat (WIF) ยท #3 Render (RNDR) ยท #4 Injective (INJ) Top Narratives # Narrative Score Vol Sentiment Bar Sentiment% Velocity Trending 1 AI & Crypto 8.42 143 โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ”‚ +0.312 47% Yes 2 ETF Approval 7.91 98 โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ”‚ +0.241 38% Yes 3 Memecoin 6.83 212 โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ”‚ +0.198 61% No 4 RWA (Real World Assets) 5.44 67 โ–ˆโ–ˆโ–ˆโ–ˆโ”‚ +0.167 29% No 5 Regulation 4.21 89 โ”‚โ–ˆโ–ˆโ–ˆโ–ˆ -0.223 21% No ``` --- ## Free Data Sources | Source | What it provides | Key? | |--------|-----------------|------| | CoinDesk, CoinTelegraph, Decrypt, Bitcoin Magazine, The Block, Blockworks, CryptoSlate, NewsBTC, AMBCrypto, CryptoNews | News articles via RSS | None | | Reddit (r/CryptoCurrency, r/Bitcoin, r/ethereum + 7 more) | Social sentiment, posts, comments | Free app | | CoinGecko Public API | Trending coins, prices, market data | None | | alternative.me Fear & Greed Index | Market sentiment index | None | --- ## NLP Stack (all free/open-source) | Library | Purpose | |---------|---------| | **Ensemble: CryptoBERT + VADER** | Sentiment scoring (~90% reliable). CryptoBERT is a transformer fine-tuned on 3.2M crypto posts; blended 65/35 with VADER and weighted by source credibility | | **spaCy en_core_web_sm** | Named Entity Recognition (coins, orgs, people) | | **scikit-learn TF-IDF** | Emerging keyword extraction from corpus | | **BERTopic** (optional) | Deep topic clustering for unknown narratives | ### Sentiment reliability Each news/social item is scored by **two models** and blended: - **CryptoBERT** (`ElKulako/cryptobert`) โ€” understands crypto slang ("wagmi", "rugged", "diamond hands"), context, and negation. Runs locally, no API. - **VADER** + crypto lexicon โ€” fast lexicon sanity-check. - **Source weighting** โ€” CoinDesk/Bloomberg count more than anonymous Reddit posts (see `SOURCE_WEIGHTS` in `config.py`). - Each item carries a **confidence** score based on model agreement. If CryptoBERT isn't downloaded, the system transparently falls back to VADER-only. ### 24-hour market sentiment The **S24 panel** aggregates every item from the last 24h into: - A **0โ€“100 sentiment index** + label (Extreme Bearish โ†’ Extreme Bullish) - **Bull / Neutral / Bear split** (counts + percentages) - **Trend arrow** vs the prior 24h (rising/falling/flat with delta) - **Per-narrative net sentiment** (ETF, AI, Regulation, etc. scored separately) Items are weighted by recency (exponential decay), source credibility, confidence, and engagement. --- ## Installation ### 1. Clone & install ```bash git clone cd News-Narrative-Analysis # Recommended: create a virtual environment python -m venv venv venv\Scripts\activate # Windows # source venv/bin/activate # macOS/Linux python install.py # installs all deps + spaCy model ``` ### 2. Reddit credentials (free, 2 minutes) 1. Go to https://www.reddit.com/prefs/apps 2. Click **"create another app"** 3. Choose **"script"**, give it any name, redirect URI: `http://localhost` 4. Copy the client ID and secret ```bash cp .env.example .env # Edit .env and fill in: # REDDIT_CLIENT_ID=your_id # REDDIT_CLIENT_SECRET=your_secret # REDDIT_USER_AGENT=CryptoNarrativeBot/1.0 by YourUsername ``` > The tool works **without** Reddit credentials (falls back to keyless JSON API), but PRAW gives richer data including comments. --- ## Usage ```bash # Single snapshot + dashboard python main.py # Live mode โ€” auto-refreshes every 30 minutes python main.py --live # Custom refresh interval (minutes) python main.py --live --interval 15 # Export JSON report to ./reports/ python main.py --export # Check all dependencies python main.py --setup # Skip Reddit (faster, news-only) python main.py --no-reddit # Verbose logging python main.py --verbose ``` --- ## Narratives Tracked | Narrative | Key Signals | |-----------|------------| | ETF Approval | "spot etf", "sec approval", "blackrock etf", inflows | | AI & Crypto | "ai token", "bittensor", "fetch.ai", "render", "agentfi" | | RWA (Real World Assets) | "tokenized bonds", "real world asset", "ondo finance" | | Memecoin | "pepe", "bonk", "wif", "meme rally", high velocity | | DeFi | "yield farming", "tvl", "uniswap", "aave", "dex" | | Layer 2 Scaling | "arbitrum", "zk rollup", "base", "optimism" | | Institutional Adoption | "microstrategy", "wall street", "treasury", "hedge fund" | | Regulation | "sec", "lawsuit", "crackdown", "legislation", "cftc" | | Exchange Listing | "listed on", "coinbase listing", "binance listing" | | Security Breach / Hack | "exploit", "rug pull", "drained", "flash loan attack" | | Ecosystem Upgrade | "hard fork", "dencun", "pectra", "eip", "protocol upgrade" | | Bitcoin Halving | "halving", "block reward", "miner reward", "halvening" | | Staking & Yield | "liquid staking", "eigenlayer", "lido", "restaking" | | Cross-chain | "bridge", "layerzero", "wormhole", "cosmos ibc" | | NFT | "ordinals", "nft market", "blue chip nft" | | GameFi / P2E | "play to earn", "blockchain game", "axie infinity" | --- ## Scoring Formula ``` score = ( 0.40 ร— log(volume) + 0.30 ร— normalised_sentiment + 0.30 ร— engagement_weight ) ร— trending_coin_bonus ร— (1 + velocity) ``` - **Volume**: article/post count (log-scaled) - **Sentiment**: VADER compound score with crypto-specific lexicon - **Engagement**: Reddit upvotes + comments (0 for news) - **Trending bonus**: ร—1.5 if a related coin is trending on CoinGecko - **Velocity**: fraction of items published in last 6 hours --- ## Project Structure ``` News-Narrative-Analysis/ โ”œโ”€โ”€ main.py # CLI entry point โ”œโ”€โ”€ config.py # all config (feeds, narratives, weights) โ”œโ”€โ”€ install.py # one-shot dependency installer โ”œโ”€โ”€ requirements.txt โ”œโ”€โ”€ .env.example โ”œโ”€โ”€ collectors/ โ”‚ โ”œโ”€โ”€ rss_collector.py # 10 RSS feeds โ”‚ โ”œโ”€โ”€ reddit_collector.py # PRAW + keyless JSON fallback โ”‚ โ”œโ”€โ”€ coingecko_collector.py # trending, prices, global market โ”‚ โ””โ”€โ”€ fear_greed_collector.py โ”œโ”€โ”€ analyzers/ โ”‚ โ”œโ”€โ”€ sentiment.py # VADER + crypto lexicon โ”‚ โ”œโ”€โ”€ narrative_detector.py # keyword matching + TF-IDF โ”‚ โ”œโ”€โ”€ entity_extractor.py # spaCy NER + regex tickers โ”‚ โ””โ”€โ”€ trend_scorer.py # composite narrative scoring โ”œโ”€โ”€ storage/ โ”‚ โ””โ”€โ”€ database.py # SQLite cache (articles, snapshots) โ”œโ”€โ”€ dashboard/ โ”‚ โ””โ”€โ”€ display.py # Rich terminal dashboard โ””โ”€โ”€ reports/ # JSON exports (auto-created) ``` --- ## Extending the Tool ### Add a new narrative In `config.py`, add to the `NARRATIVES` dict: ```python "Layer 3": [ "layer 3", "l3", "app-specific chain", "orbit chain", ], ``` ### Add a new RSS feed ```python RSS_FEEDS["MySource"] = "https://example.com/feed.rss" ``` ### Adjust scoring weights ```python SCORING = { "news_volume_weight": 0.4, # increase to favor volume "sentiment_weight": 0.3, "reddit_upvote_weight": 0.3, "trending_coin_bonus": 1.5, # increase for stronger trending bonus } ``` --- ## License MIT โ€” free to use, modify, and distribute.