Master Architecture · The Engineering Deck

The platform,
with the hood open.

Every service, model, database and deployment path behind Midas — statuses included. This is the companion to the product deck: fewer adjectives, more architecture.

4 products · 3 teams 24/7 engine · hourly data derived from Master Architecture v2.7 06 Jul 2026

Strictly private & confidential

Presented by Michele De Filippo, PhD

AI Lead · Midas Analytics

How to read this deck

An honest map — statuses included.

Every component carries three signals, used identically on every diagram. Nothing is dressed up: what's live is marked live, what's mid-build says so.

Status

liverunning in production

in progressbuilt, being hardened / rolled out

designedspecified in the architecture, not yet deployed

Ownership

Data teamscrapers, enrichment, feeds

AI teamengine models, Vox

Software teammidas-web, midas-backend

Cadence

hourlynews ingestion & engine passes

dailydata enrichment batches

24/7serving & monitoring

Stack MongoDB · Docker · FastAPI · Qdrant · Hugging Face · Mistral 7B · Selenium · Airflow · Grafana · Slack · AWS EC2 + on-prem

Products · four surfaces on one hub

Four products, one Insights Hub underneath.

Shared hub Processed documents + resolved entities — news, social, earnings calls, financial statements — read by all four.

Oculus

Real-time sentiment UI over the hub

The web platform: midas-web + backend reading engine-scored sentiment and topics straight from the Insights Hub.

Auris

External data API surface

A FastAPI endpoint that streams hub signals — scores, entities, topics — into a client's own stack.

Vox

RAG endpoint over the hub

Retrieval-augmented Q&A: Qdrant retriever + Mistral 7B behind /ask and query-article. Answers cite sources.

Cortex

Agentic analysis layer

An agent that chains hub retrieval, risk indices and the stock recommender into a per-company analysis. designed

The rest of this deck walks the machinery that powers them — data layer first, then the engine, then Vox, then how it all ships and runs.

System overview · hover any node for detail

One pipeline, five layers, three teams.

Ingestionmidas-data · on-prem + AWS

midas-news-scraper hourly bloomberg-ticker daily ld-updater / turbo-v2 glassdoor-updater-v2 chatgpt + yahoo-finance genie-api

StoresMongoDB

financial-news · raw-data ed-org-strings · companies Engine-Data Insights Hub

AI enginemidas-engine · 24/7

news-translator sentiment-analysis topic-modeling entity-extractor relationship-extractor risk-indices entity-refresher 24/7 document-refresher

Deliveryuser-facing

midas-web · midas-backend Vox endpoints Auris API

MonitoringGrafana + Slack

data-monitoring stat-notifier engine-monitoring vox-monitoring

livein progressdesigned — statuses per component, as in the internal architecture deck

Target architecture · data · mirrors the internal diagram

Hybrid by design: on-prem scrapes, cloud serves.

data team services livein progressdesigned Prod / UAT split on both sides — the Software team consumes stores only through midas-backend, never from scrapers.

midas-news-scraper · every hour

One model, many mastheads, every hour.

Numbers below measured on a 70-source dataset, rolling window — all live on Grafana. Ask for the tour.

The daily funnel

Articles parsed

~9,000 / day

Good articles

~4,000 (~44%)

Net-new added

~500 (~6%)

Aggressive dedup and quality gates by design — the funnel is the feature: only clean, novel articles reach the engine.

Operating envelope

Sources covered70 · measured dataset

Cadencehourly DAG (Airflow)

Continuous operation100+ days

Overall completeness~45%

Consistency delta~3%

Public-API rate limit2,000 req/h/IP · 48K/day

Designone model · per-source configs

The entity backbone · four enrichment services

Before AI reads the news, it must know who's who.

in progressdaily

bloomberg-ticker

Selenium webdriver googles "«name» Bloomberg company profile" — verbatim search, so junk strings never resolve. Ticker length 11 ⇒ flagged private; company_type falls back to LinkedIn data.

livedaily

ld-updater

Queries companies with no LinkedIn data, scrapes and inserts profiles. Deliberately throttled: 20 companies/day.

livedaily

ld-updater-turbo-v2

The scaled successor — proxy rotation, daily limits, and LinkedIn sector data mapped onto GICS for consistent classification.

livedaily

glassdoor-updater-v2

Same pattern for Glassdoor signals — reviews, headcount texture — capped at 100 companies/day.

ed-org-strings-inputraw candidate names

bloomberg check

ed-org-strings-improvedverified companies

enrich

companiesLinkedIn · Glassdoor · GICS

no Bloomberg profile at the check ↳

ed-org-strings-not-entparked — not treated as a company

Data quality · measured like an SLA

Four dimensions, per feed, per day.

Dimension	News scraping	bloomberg-ticker	LinkedIn / Glassdoor
Volume	# of articles · daily running time	# of new companies · running time	# companies updated · running time
Completeness	share of articles with title / content / date	good attempts / total attempts	good attempts / total attempts
Timeliness	Δ completeness within predefined windows	Δ new companies in predefined windows	Δ updates in predefined windows
Consistency	Δ vs trailing baseline (~3% observed)	Δ failed attempts vs baseline	Δ failed attempts vs baseline

alphamine-stat-pulsecomputes daily stats

Grafana Atlas dashboardsalphamine-stats · run-stats

stat-notifier → Slackanomalies reach humans

Every feed is judged on the same four axes — drift is caught by the notifier before users ever see it.

Target architecture · engine · mirrors the internal diagram

midas-engine: an idempotent hourly loop.

livein progressdesigned ① query → ② write → ③ stamp last_processed. Crash-safe: an interrupted pass re-runs.

sentiment-analysis · sentence-level, finance-tuned

Two transformer heads read the tone.

Sentiment

DistilRoBERTa fine-tuned on financial news sentiment — positive / neutral / negative per sentence, aggregated to the document.

model: mrm8488/distilroberta-finetuned-
financial-news-sentiment-analysis

Emotion

A second DistilRoBERTa head classifies emotions in English text; the document keeps its most_common_emotion by aggregated counts.

model: j-hartmann/emotion-english-
distilroberta-base

Document score · normalized to 0–100

sentiment_score = (1·positive − 1·negative + 0·neutral + 1) × 50 // 0 = uniformly negative · 50 = neutral · 100 = uniformly positive

topic-modelling · training · step through it

A taxonomy grown from a million sentences.

01

~1M-sentence corpus

02

Embed · 512-d

03

Reduce · PCA 200 → UMAP 5

04

~1,000 dense clusters

05

4-level taxonomy

Step 1 / 5

~1M-sentence corpus

Training starts from roughly a million sentences sampled from the financial-news corpus — real market language, not generic web text.

topic-modelling · inference · recursive over 4 levels

Every article lands on the tree, with confidence.

Seeded from IPTC Media Topic NewsCodes, extended over the corpus — 4 levels, confidence at each.

One real candidate, as stored

"423": [{ "cluster_keyword": "impeachment", // from corpus "taxonomy_keyword": "Impeachment", // best match "similarity_score": 0.8473, "taxonomy_keyword_level": 3 }]

Output per document: Layer Topic 1–4, each with a confidence level — recursive, explainable, auditable.

entity-extractor · NER + resolution

Names in text become ids in the graph.

Model flair/ner-english-ontonotes-large entity classes: companies · people · locations · products · events · dates · geopolitical entities

1 · DetectNER over unprocessed articles

2 · Validatecompany strings vs ed-org-strings

3 · Resolvecompany id assigned from entities

New companies are written back as candidates — the backbone grows from reading.

entity-refresher · deduplication · rolling 24/7

Eight passes turn aliases into one company.

Phase 1 · initial groupings

Group by logo image-hash → expand via shared LinkedIn URL → shared homepage URL → shared simplified name; then sweep ungrouped companies by each key in turn.

Phase 2 · merging groups

Merge groups whose parents or children share a LinkedIn URL, homepage URL or simplified name — six merge passes, until stable.

Parent election

The member with the highest LinkedIn follower count becomes parent: its official_name → the group's official name; its LinkedIn name → the display name.

Worked example · two records, one company

// record A official_name: "Ford Motor Co" followers: 3,696,299 linkedin_url: …/ford-motor-company // record B official_name: "Ford Motor Pvt Ltd" followers: 3,696,964 linkedin_url: …/ford-motor-company // same → merged simplified: "Ford Motor Co" → "ford motor" display_name: "Ford Motor Company" // parent B's LinkedIn name

Ships as a Docker image on EC2 (engine-a · Hong Kong, image entity-refresher-0.19) — the algorithm above runs continuously against the full entity set.

relationship-extractor · in progress

From sentences to who-did-what-to-whom.

The model

REBEL-large (Babelscape) — seq2seq relation extraction that emits (head, relation, tail) triples straight from raw sentences.

model: Babelscape/rebel-large

The integration detail that matters

REBEL's head_span / tail_span are re-parsed with the engine's own NER model, so extracted relations bind to the same entity ids as everything else — one graph, no orphan nodes.

Rolling out now — once dense, these edges turn the Insights Hub into a knowledge graph.

risk-indices · end-to-end spec

Six lenses, an evaluator, a recommender. designed — not yet built

This is the spec, not running code — the diagram below is a target architecture, nothing here is in production yet.

Target architecture · Vox endpoints · mirrors the internal diagram

RAG in production: ask, and query-article.

DockerFastAPIQdrantHugging FaceMistral 7B Retrieval is the guardrail: the LLM never sees anything the retriever didn't hand it.

The underlying pattern · reusable per client use case

Two steps: build the base, then chat.

Step 1 · build the knowledge base

Documentsany data source

Embeddingsencoder model

Qdrantvector database

Any data can be embedded — documents, company records, client databases — into one searchable space.

Step 2 · Q&A over it

Query + history

Similar chunksshortlist context

Completionprompt-engineered

Answer + sources

Prompt engineering + retrieval orchestrated in a LangChain-style chain; the LLM and embedder are swappable behind the endpoint.

The same two-step pattern generalizes to client deployments — integration points are specified per use case, the chassis stays identical.

Release discipline · every service, same road

DEV → UAT → PROD, no shortcuts.

Who runs it Data team · scrapers & genie AI team · engine & Vox Software team · web integration testing

Infrastructure · Confidential

All-in serving cost today: ~$3.3k/yr.

~$3,334/yr live only (mongo + web pair) → ~$4,276/yr at full build + scrapers & engine envs

AWS estate

mongo-instance liver6g.xlarge · Hong Kong

web-dev-env + web-prod-env livet3.medium ×2 · Singapore

scrapers-env designedt3.medium · Singapore

engine-env designedt3.medium · Singapore

On-prem scraping estateAirflow-orchestrated

Order-of-magnitude cost

mongo-instance$0.27/h ≈ $199/mo ≈ $2,392/yr

web envs (pair)$0.0528/h ×2 ≈ $78.5/mo ≈ $942/yr

each added t3.medium pair≈ $942/yr

The entire serving footprint costs less than one Bloomberg terminal^* — deliberate: spend goes to data and models, not idle compute. Region choice keeps stores in Hong Kong, latency-close to the markets Midas reads.

* Bloomberg Terminal list price ≈ $24–30k/yr/seat.

Ownership · three teams, clean seams

Every box on every diagram has an owner.

Data team

Scrapers, enrichment fleet, entity backbone, genie, data-quality monitoring. Owns everything upstream of the engine — volume with discipline.

AI team

midas-engine models, refreshers, Vox endpoints and their monitoring. Owns everything that reads, understands and answers.

Software team

midas-web + midas-backend, UAT integration testing, the user-facing surface. Owns what the customer touches.

Interfaces between teams are databases and APIs, not meetings — the Software team consumes the hub through midas-backend; the AI team consumes feeds the Data team guarantees.

The engineering thesis

AI on top of the pipeline — never inside the data layer.

Traceable

Every answer resolves to documents, entities and sources you can open. Grounding is architectural, not a prompt promise.

Swappable

Models are components behind endpoints — embedder, LLM, NER can each be upgraded without touching the pipeline.

Honest

Statuses live on the diagrams. What's designed says designed — and ships through the same UAT gate as everything else.

Go deeper

Three ways to kick the tires.

01 · Architecture session

Bring your engineers

A working session on any layer of this deck — data, engine, or Vox — with the people who built it.

02 · Live tour

Grafana, not slides

Watch the dashboards while the pipeline runs: scraper stats, engine throughput, Vox logs — production, live.

03 · API pilot

Integrate Auris

A scoped pilot streaming Midas signals into your stack — integration points specified per use case.

Book a technical walkthrough

Or ask for the live Grafana tour — the pipeline running, not slides. Write to michele@midasanalytics.ai.

Thank you.

AI on top of the pipeline, never inside it — built to be opened, inspected, and trusted.

midasanalytics.ai

The platform,with the hood open.

An honest map — statuses included.

Four products, one Insights Hub underneath.

One pipeline, five layers, three teams.

Hybrid by design: on-prem scrapes, cloud serves.

One model, many mastheads, every hour.

Before AI reads the news, it must know who's who.

Four dimensions, per feed, per day.

midas-engine: an idempotent hourly loop.

Two transformer heads read the tone.

A taxonomy grown from a million sentences.

Every article lands on the tree, with confidence.

Names in text become ids in the graph.

Eight passes turn aliases into one company.

From sentences to who-did-what-to-whom.

Six lenses, an evaluator, a recommender. designed — not yet built

RAG in production: ask, and query-article.

Two steps: build the base, then chat.

DEV → UAT → PROD, no shortcuts.

All-in serving cost today: ~$3.3k/yr.

Every box on every diagram has an owner.

AI on top of the pipeline — never inside the data layer.

Three ways to kick the tires.

Thank you.

The platform,
with the hood open.