Skip to content
All projects
2024

Kafka Ingestion and PySpark Clustering of Stock Prices

Kafka ingestion of six months of daily prices into 55 per-ticker topics, plus a separate PySpark MLlib K-means stage on one ticker's price series.

55

Kafka topics

Apr to Sep 2023

Window

4

Chosen K

~0.74

Silhouette at K = 4

Hypothesis

Can a single notebook stand up a Kafka + Zookeeper broker, load 55 tickers of daily prices into it, and run a K-means analysis in PySpark with K chosen from silhouette and elbow plots?

Problem
A course exercise in standing up a message-ingestion layer and a distributed machine-learning library, with K in K-means chosen from evidence rather than by default.
My role
Team of 2 (Hardware and Software for Big Data, Federico II) with Farshad Farahtaj; I built the pipeline.
Result
6,875 daily price records into 55 per-stock Kafka topics; K = 4 chosen from silhouette and elbow plots.

Honest limits

  • Nothing reads the topics back: the clustering uses the last stock downloaded, from memory, and Spark ran in local mode on one machine.
  • The four price features correlate at 0.97 to 0.99, so the clusters are price-level bands over time, not a market finding.
Overview

Kafka ingestion for 55 tickers, plus a separate clustering exercise

A single-broker Apache Kafka 3.3.1 plus Zookeeper setup, started from a notebook. Producers sent daily OHLCV data (April to September 2023, yfinance) for 55 US tickers into one topic per ticker (stock_AAPL, stock_AMZN, and so on; the topics were auto-created by the producers): 6,875 messages, 55 tickers x 125 trading days.

Scope: this is a batch backfill of historical data, not a live stream. Nothing consumes the topics, and Spark does not read from Kafka. The K-means analysis ran separately, in PySpark local mode, on a single ticker's 125 daily rows.

University team project (Hardware and Software for Big Data, Federico II) with Farshad Farahtaj; I built the pipeline.

Methodology

Choosing K for one ticker's prices

The PySpark MLlib stage runs VectorAssembler → KMeans (seed=1) → ClusteringEvaluator on the raw Open, High, Low and Close prices; PCA (k=2) is used only to plot the clusters. K was swept from 2 to 10 and judged on:

  • Silhouette score (cluster cohesion + separation)
  • Within-cluster sum of squares, the classic Elbow test
  • 2D PCA visualisation, to eyeball how clusters fragment

Silhouette peaks at K = 3 but stays close at K = 4 (about 0.76 vs about 0.74, read off the notebook's plot); Elbow shows the largest drop between K=2 and K=3, then diminishing returns; in the PCA plots K=5 and K=6 start to split bands. K=4 was picked by eye from these plots. With one stock and unscaled prices, this is an exercise in choosing K, not a finding about the market.

K selection

Two interactive views of why K = 4

The silhouette and elbow curves below are rebuilt from the notebook outputs as interactive charts. Hover any point for the approximate value at that K. Amber rings mark K = 4, the chosen partition.

K selection (1 of 2)

Silhouette score across K

Higher is better.

How to read it: silhouette measures cluster cohesion plus separation. The global peak is at K = 3 (about 0.76) and K = 4 sits close behind (about 0.74, amber marker). Either is defensible from this metric alone, which is why we cross-checked against Elbow and the PCA grid before committing.

K selection (2 of 2)

Within-cluster sum of squares (Elbow)

Lower is tighter clusters; the bend is the natural K.

How to read it: WCSS always falls as K grows; what matters is where the curve bends. The drop from K = 2 to K = 3 is the biggest single step, then K = 4 (amber marker) flattens. After K = 4 each new cluster buys very little, the signature of over-fragmenting.

Cluster visualisation

What the clusters actually look like

Two notebook outputs that don't translate cleanly to interactive charts. The grid below shows the K-means assignment at K = 2 through 6, and the 3D scatter shows the final K = 4 partition in raw Open / High / Low price space.

Notebook output
K-means visualisation grid across K = 2 to 6 in PCA space
K-means on one ticker's daily prices, projected into 2D PCA for K = 2, 3, 4, 5, 6.How to read it: each panel is the same data (one ticker's 125 trading days), coloured by the K-means assignment at a different K. PCA is used only to draw the plot. At K = 5 and K = 6 one band splits into smaller sub-clusters; K = 4 itself was chosen from the silhouette and elbow plots. The features are raw price levels of a single stock, so the clusters are an exercise in choosing K, not a finding about the market.
Notebook output
3D scatter plot of the chosen K=4 clusters in Open/High/Low space
Final K = 4 clusters in raw Open / High / Low price space.How to read it: each point is one trading day of a single ticker, positioned by its Open, High, and Low prices, coloured by its cluster. Because the input is unscaled price levels, the clusters mostly separate periods with different price levels.
Tech stack

Frameworks and infrastructure

Apache Kafka 3.3.1ZookeeperPySpark MLlibKMeansPCAyfinance

Source code on GitHub.