Omni Terminal

News & AI Analysis

A news pipeline built for trading context, not headline scrolling.

Omni Terminal treats news as market structure input. Raw Telegram, RSS, and source feeds are normalized, deduped, clustered, summarized, routed to tradable symbols, and then delivered into the terminal and Ask Omni so event flow can be read beside positions, liquidations, funding, and chart state.

At a glance

Pipeline shape
Ingest -> GPT -> terminal Raw feeds are filtered, deduped, clustered, summarized, and adapted before reaching the UI.
Dedupe model
Multi-layer Exact matching, embedding gates, context shrinking, and event clustering reduce repeated alerts.
Delivery
REST + WS The terminal reads history through `/api/terminal/news` and receives live GPT news over websocket.

Why It Matters

What traders and researchers should take away

  • Raw Telegram and RSS feeds are noisy; Omni filters repeated, promotional, and low-signal items before they become terminal context.
  • Symbol-linked AI news is more useful when it can open next to the chart, order ticket, liquidation stats, and current portfolio exposure.
  • The pipeline is cost and latency aware: cheap filters and embedding/event dedupe run before expensive GPT work where possible.

How it works

What the product uses behind the scenes

  • telegram-reader captures raw feed messages and publishes normalized raw news events without calling GPT.
  • gpt-news performs noise filtering, embedding/event dedupe, cluster updates, GPT summarization, topic tagging, and terminal contract validation.
  • omni-api adapts GPT news documents for /api/terminal/news and websocket gpt_news; the terminal adds display-oriented symbol resolution.
  • Ask Omni loads the same backend news source into compact account and market risk snapshots.

Notes

Things to know

  • News interpretation is decision support, not a guarantee that an event will move a market.
  • The terminal UI and Ask Omni share upstream news sources, but they do not use the exact same final symbol-resolution implementation.
  • Public docs intentionally avoid private channel inventory, prompt text, exact scoring thresholds, hostnames, keys, and deploy details.

End-to-end pipeline

The public contract is simple, but the internal path is deliberately layered so the terminal receives cleaner, tradable, context-rich events instead of raw feed spam.

Implementation flow example
flowchart LR
  RAW[Telegram, RSS, and source feeds] --> TR[telegram-reader]
  TR --> R1[(Redis telegram_news)]
  TR --> CH1[(ClickHouse raw messages)]
  R1 --> GN[gpt-news instant worker]
  GN --> FILTER[noise and exact filters]
  FILTER --> EMB[embedding near-duplicate gate]
  EMB --> EVENT[event clustering skip/update/full]
  EVENT --> GPT[OpenAI instant or cluster-update prompt]
  GPT --> STORE[(ClickHouse gpt_news_analysis + Redis gpt_news)]
  STORE --> API[omni-api adapt_gpt_news_doc]
  API --> UI[Terminal news, chart context, Ask Omni]
  • telegram-reader stores raw messages and publishes a common raw-news channel; it is intentionally not the GPT layer.
  • gpt-news decides whether a headline is noise, a near duplicate, an update to an existing event, or a full new analysis candidate.
  • omni-api adapts the GPT news document into the terminal contract used by REST history, websocket toasts, terminal tables, and Ask Omni.

Why the pipeline is this complex

Trading news has a different quality bar than a generic news feed: repetition, latency, bad ticker routing, and irrelevant context all become trader problems.

  • Multiple channels often repeat the same event with slightly different wording; dedupe prevents terminal noise and repeated model calls.
  • Macro, tradfi, energy, and crypto headlines need different symbol-routing behavior than a simple ticker regex.
  • Cost control matters because cheap deterministic filters and vector/event gates should remove low-value work before GPT is called.
  • Terminal context has to stay compact enough for Ask Omni and fast enough for live alerts.

Dedupe, embeddings, and event context

Omni uses several dedupe passes because no single method catches exact reposts, paraphrases, rolling updates, and multi-source event clusters.

  • Exact/source normalization catches obvious reposts and formatting changes before heavier processing.
  • Embedding dedupe catches near-duplicate headlines and shrinks the history window that gets offered as model context.
  • Event clustering keeps a rolling thesis for developing stories, allowing the pipeline to skip, update, or fully analyze a headline.
  • Topic tags such as macro, liquidation, policy, exchange, energy, and event type let the UI and Ask Omni filter context without raw prompt stuffing.

How the terminal consumes news

The terminal does not subscribe directly to raw ingest systems. It consumes an adapted Omni API contract with history, live websocket updates, and display-side symbol resolution.

  • GET /api/terminal/news loads paginated history for news lists, symbol filters, topic filters, and Ask Omni context.
  • Websocket gpt_news delivers live GPT news envelopes for terminal toasts and real-time updates.
  • src/lib/utils/newsSymbolResolution.ts performs display-oriented symbol inference, alias steering, fuzzy matching, and preferred-symbol selection.
  • Ask Omni reuses backend news loading and adaptation, then compacts symbol and global news into its risk snapshot.

What public docs intentionally leave out

The public page explains the architecture enough for users and LLMs to understand the product, but avoids operational details that would make the system easier to copy or abuse.

  • No private feed inventory, channel list, server hostnames, service credentials, or access headers.
  • No exact production thresholds, prompt text, model routing secrets, or deploy topology.
  • No private user balances, positions, wallet addresses, or account-specific news snapshots.
  • Repo-scoped paths and canonical docs URLs are the citation surface for public answers.

References

Implementation references

  • AI news pipeline

    omni-terminal/docs/AI_NEWS_PIPELINE.md

    Public-safe local summary of the AI news ingest, dedupe, GPT analysis, and terminal adaptation flow.

  • Terminal news display

    omni-terminal/docs/NEWS_DISPLAY.md

    Documents REST history, websocket live news, topic tags, and terminal-side symbol resolution.

  • GPT news pipeline

    gpt-news/docs/NEWS_PIPELINE.md

    Documents the raw ingest to GPT analysis to terminal contract path, including dedupe and event update stages.

  • GPT news deduplication

    gpt-news/NEWS_DEDUPLICATION.md

    Documents why exact, embedding, and event-level dedupe are separate layers.

  • Raw feed integration

    telegram-reader/docs/NEWS_INTEGRATION.md

    Documents raw message storage and Redis publication before the GPT analysis layer.

  • Ask Omni documentation

    omni-terminal/docs/ASK_OMNI.md

    Explains how symbol news, global news, and risk snapshots are constructed for AI responses.

  • Omni API news adapters

    hyperliquid-data-api/crates/omni_api/src/routes/terminal.rs

    Shows backend news loading, ClickHouse reads, GPT-news adaptation, and backend symbol normalization.

  • Omni API news websocket

    hyperliquid-data-api/crates/omni_api/src/routes/ws.rs

    Shows live `gpt_news` envelope building for instant, bulk, and event rollup messages.

Next