1.3.5 — Clustering and Dimensionality Reduction

Learning outcomes

After this lesson, you will be able to:

• Choose among K-means, hierarchical clustering, and DBSCAN based on shape, density, and scale.
• Explain how scaling, distance metrics, and algorithm parameters affect results.
• Use the elbow method and silhouette score as diagnostics rather than final answers.
• Test cluster stability under changes in samples, initialisation, and parameters.
• Explain what PCA preserves and why components are not automatically meaningful domain features.
• Distinguish dimensionality reduction for modelling from two-dimensional visual exploration.

1. Why no single algorithm fits every dataset

Clustering does not answer “which classes exist?” It asks “what structure appears under a chosen way of measuring similarity?” Start by describing the geometry of the data. Compact groups of similar size may suit K-means. Elongated, nested, or irregular regions may require another method. If density varies substantially, one global neighbourhood threshold may not fit any DBSCAN setting well.

2. Feature preparation: distance is shaped before the algorithm runs

An algorithm compares observations in feature space. If one feature is measured in thousands of kilowatt-hours and another ranges from zero to one, Euclidean distance may be dominated by the first. Standardisation makes scales more comparable, commonly by subtracting the mean and dividing by the standard deviation. It is not mandatory in every task: original units may matter, and outliers can distort the mean and standard deviation. Robust scaling based on the median and interquartile range may be more appropriate.

3. Choosing and tuning an algorithm

K-means minimises the sum of squared distances to centroids. It is fast and useful as a baseline, but favours compact, convex groups and requires k in advance. Use multiple initialisations and fix random_state for reproducibility. Agglomerative hierarchical clustering builds a tree of merges. Cutting the tree at different heights produces different numbers of groups; the result depends on linkage, the rule for measuring distances between clusters. DBSCAN groups points through density reachability, identifies noise, and can find irregular shapes. Its eps and min_samples parameters interact: increasing eps expands neighbourhoods, while min_samples sets the required local density. With strongly varying densities, DBSCAN may split one structure or merge distinct ones.

4. Do not choose the number of clusters by eye alone

The elbow method compares within-cluster sum of squares across candidate k values. Look for a point after which adding clusters brings diminishing improvement. The elbow is often ambiguous and depends on feature scaling. Silhouette score compares an observation's average distance to its own cluster with its distance to the nearest alternative cluster. Values closer to 1 indicate clearer geometric separation, around 0 indicate a boundary, and negative values may indicate a poor assignment. A high score does not mean the groups are useful for the domain task. For DBSCAN, also inspect the noise fraction and sensitivity to eps; noise is not automatically bad data.

5. Stability: does the discovered structure repeat?

Clustering often has no single “correct answer”. Repeat the analysis with different initial states, subsamples, or nearby parameter values. Compare partitions, not cluster numbers: labels 0 and 1 can swap without changing the structure. The Adjusted Rand Index (ARI) compares two partitions while correcting for chance agreement: 1 means identical partitions, around 0 means chance-level agreement, and a negative value means less agreement than expected by chance. Stability does not prove truth: an algorithm can consistently discover an artefact caused by poor data.

6. PCA: what exactly becomes smaller?

PCA constructs new, mutually orthogonal axes called principal components, each a linear combination of the original features. The first component points in the direction of maximum variance; the next captures as much remaining variance as possible while staying orthogonal to earlier components. Explained variance helps quantify how much variation is retained by a chosen number of components. But variance is not the same as usefulness: a rare but important signal may have low variance. Features are usually centred before PCA; scaling depends on whether units are comparable and whether their original variability should be preserved.

7. Dimensionality reduction is not the same as a 2D picture

PCA can be used both as a feature transformation before modelling and as a way to display data on a plane. These are different goals. For modelling, choose the number of components by evaluating downstream performance and acceptable information loss. A 2D projection for visualisation inevitably discards directions. t-SNE and UMAP can preserve local neighbourhoods and reveal complex structure, but distances between far-apart groups, island sizes, and apparent densities should not be treated as direct quantitative measurements. Visual separation is a reason to investigate, not proof of natural classes.

8. Worked example: building operating modes

Suppose each hourly observation contains average power, outdoor temperature, hour of day, and the share of active equipment. First inspect units, missing values, seasonality, and correlated features. Then choose features for the question: do we want similar load profiles, or operating modes independent of time of day? After scaling, run K-means as a baseline, DBSCAN if irregular shapes and noise are expected, and a hierarchical method if nested structure matters. Compare silhouette, noise fraction, subsample stability, and group profiles in original units. Only then assign interpretable names with a domain engineer.

9. Python lab: compare methods and inspect PCA

The reproducible experiment below uses synthetic profiles. It cannot prove which algorithm is best for a real building; it demonstrates a workflow: scaling, baseline clustering, diagnostics, partition comparison, and PCA. Requirements: Python 3.10+, NumPy, and scikit-learn. Install with `python -m pip install numpy scikit-learn`.

Code example: K-means, DBSCAN, and PCA

The code prints diagnostics; exact values may vary with library versions and random-number implementation.

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.decomposition import PCA
from sklearn.metrics import adjusted_rand_score, silhouette_score
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(7)
centres = np.array([[0.0, 0.0, 0.0], [4.0, 4.0, 1.0], [-3.0, 3.0, 4.0]])
clouds = [rng.normal(loc=centre, scale=scale, size=(90, 3))
          for centre, scale in zip(centres, (0.45, 0.75, 0.55))]
X = np.vstack(clouds)
X = np.vstack([X, rng.uniform(-5, 7, size=(12, 3))])

X_scaled = StandardScaler().fit_transform(X)

kmeans_a = KMeans(n_clusters=3, n_init=20, random_state=7)
labels_a = kmeans_a.fit_predict(X_scaled)
kmeans_b = KMeans(n_clusters=3, n_init=20, random_state=21)
labels_b = kmeans_b.fit_predict(X_scaled)

dbscan = DBSCAN(eps=0.55, min_samples=6)
db_labels = dbscan.fit_predict(X_scaled)
non_noise = db_labels != -1

print("K-means clusters:", len(set(labels_a)))
print("K-means silhouette:", round(silhouette_score(X_scaled, labels_a), 3))
print("K-means stability (ARI):", round(adjusted_rand_score(labels_a, labels_b), 3))
print("DBSCAN clusters:", len(set(db_labels) - {-1}))
print("DBSCAN noise share:", round(1 - non_noise.mean(), 3))

pca = PCA(n_components=2, random_state=7)
X_2d = pca.fit_transform(X_scaled)
print("PCA explained variance:", np.round(pca.explained_variance_ratio_, 3))
print("Total retained variance:", round(pca.explained_variance_ratio_.sum(), 3))

Independent task

10. Limitations and common mistakes

• Ignoring units and feature meaning when scaling—or scaling everything automatically without checking outliers.
• Treating an elbow plot as an unambiguous answer for the number of clusters.
• Comparing cluster IDs directly instead of comparing partitions.
• Treating a high silhouette score as proof that groups are meaningful or true.
• Interpreting PCA components as original features or causal factors.
• Treating a 2D embedding as an exact map of distances in the original space.
• Deleting DBSCAN noise points as errors without domain review.

11. Key takeaways

Clustering is an exploratory hypothesis shaped by features, scaling, metrics, and algorithms. A sound analysis compares justified alternatives, tests stability, and returns to the original features for interpretation. PCA compresses variation into new coordinates but does not guarantee that every important signal is preserved. Neither a metric nor an attractive visualisation replaces domain validation.