Kafka Ingestion and PySpark Clustering of Stock Prices
Kafka ingestion of six months of daily prices into 55 per-ticker topics, plus a separate PySpark MLlib K-means stage on one ticker's price series.
55
Kafka topics
Apr to Sep 2023
Window
4
Chosen K
~0.74
Silhouette at K = 4
Hypothesis
Can a single notebook stand up a Kafka + Zookeeper broker, load 55 tickers of daily prices into it, and run a K-means analysis in PySpark with K chosen from silhouette and elbow plots?
- Problem
- A course exercise in standing up a message-ingestion layer and a distributed machine-learning library, with K in K-means chosen from evidence rather than by default.
- My role
- Team of 2 (Hardware and Software for Big Data, Federico II) with Farshad Farahtaj; I built the pipeline.
- Result
- 6,875 daily price records into 55 per-stock Kafka topics; K = 4 chosen from silhouette and elbow plots.
Honest limits
- Nothing reads the topics back: the clustering uses the last stock downloaded, from memory, and Spark ran in local mode on one machine.
- The four price features correlate at 0.97 to 0.99, so the clusters are price-level bands over time, not a market finding.
Kafka ingestion for 55 tickers, plus a separate clustering exercise
A single-broker Apache Kafka 3.3.1 plus Zookeeper setup, started from a notebook. Producers sent daily OHLCV data (April to September 2023, yfinance) for 55 US tickers into one topic per ticker (stock_AAPL, stock_AMZN, and so on; the topics were auto-created by the producers): 6,875 messages, 55 tickers x 125 trading days.
Scope: this is a batch backfill of historical data, not a live stream. Nothing consumes the topics, and Spark does not read from Kafka. The K-means analysis ran separately, in PySpark local mode, on a single ticker's 125 daily rows.
University team project (Hardware and Software for Big Data, Federico II) with Farshad Farahtaj; I built the pipeline.
Choosing K for one ticker's prices
The PySpark MLlib stage runs VectorAssembler → KMeans (seed=1) → ClusteringEvaluator on the raw Open, High, Low and Close prices; PCA (k=2) is used only to plot the clusters. K was swept from 2 to 10 and judged on:
- Silhouette score (cluster cohesion + separation)
- Within-cluster sum of squares, the classic Elbow test
- 2D PCA visualisation, to eyeball how clusters fragment
Silhouette peaks at K = 3 but stays close at K = 4 (about 0.76 vs about 0.74, read off the notebook's plot); Elbow shows the largest drop between K=2 and K=3, then diminishing returns; in the PCA plots K=5 and K=6 start to split bands. K=4 was picked by eye from these plots. With one stock and unscaled prices, this is an exercise in choosing K, not a finding about the market.
Two interactive views of why K = 4
The silhouette and elbow curves below are rebuilt from the notebook outputs as interactive charts. Hover any point for the approximate value at that K. Amber rings mark K = 4, the chosen partition.
K selection (1 of 2)
Silhouette score across K
Higher is better.
How to read it: silhouette measures cluster cohesion plus separation. The global peak is at K = 3 (about 0.76) and K = 4 sits close behind (about 0.74, amber marker). Either is defensible from this metric alone, which is why we cross-checked against Elbow and the PCA grid before committing.
K selection (2 of 2)
Within-cluster sum of squares (Elbow)
Lower is tighter clusters; the bend is the natural K.
How to read it: WCSS always falls as K grows; what matters is where the curve bends. The drop from K = 2 to K = 3 is the biggest single step, then K = 4 (amber marker) flattens. After K = 4 each new cluster buys very little, the signature of over-fragmenting.
What the clusters actually look like
Two notebook outputs that don't translate cleanly to interactive charts. The grid below shows the K-means assignment at K = 2 through 6, and the 3D scatter shows the final K = 4 partition in raw Open / High / Low price space.


Frameworks and infrastructure
Source code on GitHub.