← CSC 416 Machine Learning and Data Science
Unsupervised Learning and Smallholder Farmer Segmentation
Notes
Fundamentals of Unsupervised Learning
Unsupervised learning discovers intrinsic structures, manifolds, and latent groupings within unlabeled datasets. Unlike supervised models that optimize parameters toward an explicit supervisory target, unsupervised methods optimize intrinsic geometric or density-based objectives. Common formulations include dimensionality reduction algorithms like Principal Component Analysis and partitioning algorithms such as K-Means and Gaussian Mixture Models.
The K-Means Clustering Algorithm
K-Means partitions an unlabeled dataset containing n observations into k distinct clusters. The algorithm minimizes the within-cluster sum of squares, also known as cluster inertia. It iteratively alternates between an assignment step, where each observation is mapped to its nearest centroid using squared Euclidean distance, and an update step, where centroids are recomputed as the arithmetic mean of all assigned points. The initialization method K-Means Plus Plus accelerates convergence and guards against suboptimal local minima by dispersing initial cluster centroids probabilistically.
Determining Cluster Quality
Selecting the optimal value of k requires rigorous heuristics because ground truth labels are absent. The Elbow Method tracks inertia reduction as k increases, identifying inflection points where marginal variance explained diminishes. The Silhouette Coefficient measures how similar an observation is to its assigned cluster compared to neighboring clusters, producing scores between negative one and positive one.
Practical example
An agricultural technology company based in Nairobi, operating a mobile platform similar to DigiFarm or Twiga Foods, seeks to segment smallholder farmers across Nakuru and Uasin Gishu counties to tailor input financing packages.
The dataset collects four continuous features: acreage under cultivation, annual fertilizer expenditure in Kenya Shillings, historical yield in ninety-kilogram bags of maize, and mobile input voucher redemption frequency. Because these features occupy vastly different numerical scales, the data science team applies z-score standardization to ensure Euclidean distances remain scale-invariant.
Running K-Means with k selected as three via silhouette analysis reveals three actionable agricultural archetypes. Cluster One comprises subsistence farmers with under two acres, minimal commercial fertilizer use, and low voucher engagement. Cluster Two represents emerging commercial farmers with three to eight acres and frequent voucher usage. Cluster Three represents medium-scale commercial farmers cultivating over ten acres with significant capital expenditure.
By leveraging these cluster centroids, the enterprise deploys targeted SMS advisory messages and custom credit limits, matching input supply guarantees directly to each farmer segment's actual operational scale.
Quiz
1. What is indicated by an average Silhouette Coefficient close to negative one?
2. How does the K-Means Plus Plus initialization scheme improve standard K-Means?
3. Why must features be standardized before applying distance-based clustering like K-Means?
Sign in to submit and securely grade your quiz.