Master Architecture · The Engineering Deck

The platform,
with the hood open.

Every service, model, database and deployment path behind Midas — statuses included. This is the companion to the product deck: fewer adjectives, more architecture.

4 products · 3 teams 24/7 engine · hourly data derived from Master Architecture v2.7 06 Jul 2026

Strictly private & confidential

Presented by Michele De Filippo, PhD

AI Lead · Midas Analytics

How to read this deck

An honest map — statuses included.

Every component carries three signals, used identically on every diagram. Nothing is dressed up: what's live is marked live, what's mid-build says so.

Status

liverunning in production

in progressbuilt, being hardened / rolled out

designedspecified in the architecture, not yet deployed

Ownership

Data teamscrapers, enrichment, feeds

AI teamengine models, Vox

Software teammidas-web, midas-backend

Cadence

hourlynews ingestion & engine passes

dailydata enrichment batches

24/7serving & monitoring

The stack, in one strip

MongoDB · storesDocker · servicesFastAPI · endpointsQdrant · vectorsHugging Face · modelsMistral 7B · LLMSelenium · enrichmentAirflow · schedulingGrafana · monitoringSlack · alertingAWS EC2 + on-prem

Products · AI that emulates human senses

Four products. One knowledge base. 10× the analyst.

Knowledge News · Social media · Earnings calls · Financial statements · Network experts & more data sources

Oculus

Monitor sentiment & trends, 24/7

A one-stop digital platform offering real-time tracking and analysis of market sentiment and trends.

Auris

Integrate market data & insights

An API designed to seamlessly aggregate and deliver the precise data and insights you require.

Vox

Ask anything, get everything

A GenAI chatbot designed to comprehensively respond to your inquiries and uncertainties.

Cortex

Delegate to a robo-analyst

An intelligent digital assistant that conducts tailored investment analysis to your specific requirements.

The rest of this deck walks the machinery that powers them — data layer first, then the engine, then Vox, then how it all ships and runs.

System overview · hover any node for detail

One pipeline, five layers, three teams.

Ingestionmidas-data · on-prem + AWS

midas-news-scraper hourly bloomberg-ticker daily ld-updater / turbo-v2 glassdoor-updater-v2 chatgpt + yahoo-finance genie-api

StoresMongoDB

financial-news · raw-data ed-org-strings · companies Engine-Data Insights Hub

AI enginemidas-engine · 24/7

news-translator sentiment-analysis topic-modeling entity-extractor relationship-extractor risk-indices entity-refresher 24/7 document-refresher

Deliveryuser-facing

midas-web · midas-backend Vox endpoints Auris API

MonitoringGrafana + Slack

data-monitoring stat-notifier engine-monitoring vox-monitoring

livein progressdesigned — statuses per component, as in the internal architecture deck

Target architecture · data · mirrors the internal diagram

Hybrid by design: on-prem scrapes, cloud serves.

data team services livein progressdesigned Prod / UAT split on both sides — the Software team consumes stores only through midas-backend, never from scrapers.

midas-news-scraper · measured on a 70-source dataset

One model, many mastheads, every hour.

The daily funnel

Articles parsed

~9,000 / day

Good articles

~4,000 (~44%)

Net-new added

~500 (~6%)

Aggressive dedup and quality gates by design — the funnel is the feature: only clean, novel articles reach the engine.

Operating envelope

Sources covered70 · measured dataset

Cadencehourly DAG (Airflow)

Continuous operation100+ days

Overall completeness~45%

Consistency delta~3%

Public-API rate limit2,000 req/h/IP · 48K/day

Designone model · per-source configs

The entity backbone · four enrichment services

Before AI reads the news, it must know who's who.

in progressdaily

bloomberg-ticker

Selenium webdriver googles "«name» Bloomberg company profile" — verbatim search, so junk strings never resolve. Ticker length 11 ⇒ flagged private; company_type falls back to LinkedIn data.

livedaily

ld-updater

Queries companies with no LinkedIn data, scrapes and inserts profiles. Deliberately throttled: 20 companies/day.

livedaily

ld-updater-turbo-v2

The scaled successor — proxy rotation, daily limits, and LinkedIn sector data mapped onto GICS for consistent classification.

livedaily

glassdoor-updater-v2

Same pattern for Glassdoor signals — reviews, headcount texture — capped at 100 companies/day.

ed-org-strings-inputraw candidate names

bloomberg check

ed-org-strings-improvedverified companies

enrich

companiesLinkedIn · Glassdoor · GICS

no Bloomberg profile at the check ↳

ed-org-strings-not-entparked — not treated as a company

Data quality · measured like an SLA

Four dimensions, per feed, per day.

Dimension	News scraping	bloomberg-ticker	LinkedIn / Glassdoor
Volume	# of articles · daily running time	# of new companies · running time	# companies updated · running time
Completeness	share of articles with title / content / date	good attempts / total attempts	good attempts / total attempts
Timeliness	Δ completeness within predefined windows	Δ new companies in predefined windows	Δ updates in predefined windows
Consistency	Δ vs trailing baseline (~3% observed)	Δ failed attempts vs baseline	Δ failed attempts vs baseline

alphamine-stat-pulsecomputes daily stats

Grafana Atlas dashboardsalphamine-stats · run-stats

stat-notifier → Slackanomalies reach humans

Every feed is judged on the same four axes — drift is caught by the notifier before users ever see it.

Target architecture · engine · mirrors the internal diagram

midas-engine: an idempotent hourly loop.

livein progressdesigned ① query unprocessed → ② process & write → ③ stamp last_processed. Crash-safe: an interrupted pass simply re-runs.

sentiment-analysis · sentence-level, finance-tuned

Two transformer heads read the tone.

Sentiment

DistilRoBERTa fine-tuned on financial news sentiment — positive / neutral / negative per sentence, aggregated to the document.

model: mrm8488/distilroberta-finetuned-
financial-news-sentiment-analysis

Emotion

A second DistilRoBERTa head classifies emotions in English text; the document keeps its most_common_emotion by aggregated counts.

model: j-hartmann/emotion-english-
distilroberta-base

Document score · normalized to 0–100

sentiment_score = (1·positive − 1·negative + 0·neutral + 1) × 50 // 0 = uniformly negative · 50 = neutral · 100 = uniformly positive

topic-modelling · training · step through it

A taxonomy grown from a million sentences.

01

~1M-sentence corpus

02

Embed · 512-d

03

Reduce · PCA 200 → UMAP 5

04

~1,000 dense clusters

05

4-level taxonomy

Step 1 / 5

~1M-sentence corpus

Training starts from roughly a million sentences sampled from the financial-news corpus — real market language, not generic web text.

topic-modelling · inference · recursive over 4 levels

Every article lands on the tree, with confidence.

Taxonomy seeded from IPTC Media Topic NewsCodes, pruned and extended over the financial corpus — 4 levels, confidence at each.

One real candidate, as stored

"423": [{ "cluster_keyword": "impeachment", // from corpus "taxonomy_keyword": "Impeachment", // best match "similarity_score": 0.8473, "taxonomy_keyword_level": 3 }]

Output per document: Layer Topic 1–4, each with a confidence level — recursive, explainable, auditable.

entity-extractor · NER + resolution

Names in text become ids in the graph.

Model flair/ner-english-ontonotes-large entity classes: companies · people · locations · products · events · dates · geopolitical entities

1 · DetectNER over unprocessed articles

2 · Validatecompany strings vs ed-org-strings

3 · Resolvecompany id assigned from entities

New companies met in the wild are written back as candidates (engine write-mode → ed-org-strings-input) — the backbone grows from reading.

entity-refresher · deduplication · rolling 24/7

Eight passes turn aliases into one company.

Phase 1 · initial groupings

Group by logo image-hash → expand via shared LinkedIn URL → shared homepage URL → shared simplified name; then sweep ungrouped companies by each key in turn.

Phase 2 · merging groups

Merge groups whose parents or children share a LinkedIn URL, homepage URL or simplified name — six merge passes, until stable.

Parent election

The member with the highest LinkedIn follower count becomes parent: its official_name → the group's official name; its LinkedIn name → the display name.

Worked example · two records, one company

// record A official_name: "Ford Motor Co" followers: 3,696,299 linkedin_url: …/ford-motor-company // record B official_name: "Ford Motor Pvt Ltd" followers: 3,696,964 linkedin_url: …/ford-motor-company // same → merged simplified: "Ford Motor Co" → "ford motor" display_name: "Ford Motor Company" // parent B's LinkedIn name

Ships as a Docker image on EC2 (engine-a · Hong Kong, image entity-refresher-0.19) — the algorithm above runs continuously against the full entity set.

relationship-extractor · in progress

From sentences to who-did-what-to-whom.

The model

REBEL-large (Babelscape) — seq2seq relation extraction that emits (head, relation, tail) triples straight from raw sentences.

model: Babelscape/rebel-large

The integration detail that matters

REBEL's head_span / tail_span are re-parsed with the engine's own NER model, so extracted relations bind to the same entity ids as everything else — one graph, no orphan nodes.

Rolling out across the corpus now. Once dense enough, this is the edge set that turns the Insights Hub from a library into a knowledge graph.

risk-indices · designed · end-to-end spec

Six lenses, an evaluator, a recommender.

Target architecture · Vox endpoints · mirrors the internal diagram

RAG in production: ask, and query-article.

DockerFastAPIQdrantHugging FaceMistral 7B Retrieval is the guardrail: the LLM never sees anything the retriever didn't hand it.

The underlying pattern · reusable per client use case

Two steps: build the base, then chat.

Step 1 · build the knowledge base

Documentsany data source

Embeddingsencoder model

Qdrantvector database

Any data can be embedded — documents, company records, client databases — into one searchable space.

Step 2 · Q&A over it

Query + history

Similar chunksshortlist context

Completionprompt-engineered

Answer + sources

Prompt engineering + retrieval orchestrated in a LangChain-style chain; the LLM and embedder are swappable behind the endpoint.

The same two-step pattern generalizes to client deployments — integration points are specified per use case, the chassis stays identical.

Release discipline · every service, same road

DEV → UAT → PROD, no shortcuts.

Who runs it Data team · scrapers & genie AI team · engine & Vox Software team · web integration testing

Infrastructure · Confidential

Small, close to the data, fully costed.

AWS estate

mongo-instance liver6g.xlarge · Hong Kong

web-dev-env + web-prod-env livet3.medium ×2 · Singapore

scrapers-env designedt3.medium · Singapore

engine-env designedt3.medium · Singapore

On-prem scraping estateAirflow-orchestrated

Order-of-magnitude cost

mongo-instance$0.27/h ≈ $199/mo ≈ $2,392/yr

web envs (pair)$0.0528/h ×2 ≈ $78.5/mo ≈ $942/yr

each added t3.medium pair≈ $942/yr

The entire serving footprint costs less than one Bloomberg terminal — deliberate: spend goes to data and models, not idle compute. Region choice keeps stores in Hong Kong, latency-close to the markets Midas reads.

Ownership · three teams, clean seams

Every box on every diagram has an owner.

Data team

Scrapers, enrichment fleet, entity backbone, genie, data-quality monitoring. Owns everything upstream of the engine — volume with discipline.

AI team

midas-engine models, refreshers, Vox endpoints and their monitoring. Owns everything that reads, understands and answers.

Software team

midas-web + midas-backend, UAT integration testing, the user-facing surface. Owns what the customer touches.

Interfaces between teams are databases and APIs, not meetings — the Software team consumes the hub through midas-backend; the AI team consumes feeds the Data team guarantees.

The engineering thesis

AI on top of the pipeline — never inside the data layer.

Traceable

Every answer resolves to documents, entities and sources you can open. Grounding is architectural, not a prompt promise.

Swappable

Models are components behind endpoints — embedder, LLM, NER can each be upgraded without touching the pipeline.

Honest

Statuses live on the diagrams. What's designed says designed — and ships through the same UAT gate as everything else.

Go deeper

Three ways to kick the tires.

01 · Architecture session

Bring your engineers

A working session on any layer of this deck — data, engine, or Vox — with the people who built it.

02 · Live tour

Grafana, not slides

Watch the dashboards while the pipeline runs: scraper stats, engine throughput, Vox logs — production, live.

03 · API pilot

Integrate Auris

A scoped pilot streaming Midas signals into your stack — integration points specified per use case.

Write to michele@midasanalytics.ai — the product-side story lives in the companion deck.

Thank you.

Instant insights, faster decisions — built on a pipeline you can open, inspect, and trust.

midasanalytics.ai

The platform,with the hood open.

An honest map — statuses included.

Four products. One knowledge base. 10× the analyst.

One pipeline, five layers, three teams.

Hybrid by design: on-prem scrapes, cloud serves.

One model, many mastheads, every hour.

Before AI reads the news, it must know who's who.

Four dimensions, per feed, per day.

midas-engine: an idempotent hourly loop.

Two transformer heads read the tone.

A taxonomy grown from a million sentences.

Every article lands on the tree, with confidence.

Names in text become ids in the graph.

Eight passes turn aliases into one company.

From sentences to who-did-what-to-whom.

Six lenses, an evaluator, a recommender.

RAG in production: ask, and query-article.

Two steps: build the base, then chat.

DEV → UAT → PROD, no shortcuts.

Small, close to the data, fully costed.

Every box on every diagram has an owner.

AI on top of the pipeline — never inside the data layer.

Three ways to kick the tires.

Thank you.

The platform,
with the hood open.