CampusLearn
← CSC 114 Calculus for Computing

Differentiation and Optimization in Machine Learning

Notes

Fundamentals of Differentiation Differentiation measures the sensitivity of a dependent variable to changes in an independent variable. In computing, the derivative represents the instantaneous rate of change of a system state, such as network throughput, rendering pipelines, or model prediction error. Given a function f(x), its first derivative f prime of x expresses the gradient or slope of the tangent line at any point along its domain. Optimization and Critical Points Optimization is central to software engineering and data science, where the objective is to maximize efficiency or minimize system loss. A differentiable function achieves a local extremum, such as a minimum or maximum, at critical points where the first derivative equals zero or fails to exist. To establish whether a critical point represents a minimum or a maximum, developers apply the second derivative test. If the second derivative at the critical point is strictly positive, the curve is concave upward, confirming a local minimum. If the second derivative is negative, the curve is concave downward, confirming a local maximum. Gradient Descent in Computing Modern computing tasks, including training neural networks and tuning search ranking algorithms, rely on multivariate differentiation. Algorithms compute partial derivatives across high-dimensional cost surfaces to form a gradient vector. By stepping iteratively in the opposite direction of the gradient, machine learning models minimize their error functions. Practical example An e-commerce startup based in Nairobi wants to minimize operational cloud infrastructure costs for its delivery logistics platform during the busy Black Friday shopping week. The infrastructure cost function C(x) in thousands of Kenya Shillings is modeled by C(x) = 2x squared - 160x + 4200, where x represents the number of virtual container instances allocated across local data centers. To find the instance count that minimizes cost, the system architect takes the first derivative with respect to x: C prime of x = 4x - 160. Setting the first derivative to zero to find the critical point yields: 4x - 160 = 0 4x = 160 x = 40 container instances. To verify whether this critical point produces minimum rather than maximum cost, the architect calculates the second derivative: C double prime of x = 4. Because 4 is strictly greater than zero, the cost curve is concave upward at x = 40, verifying an absolute minimum cost. Substituting x = 40 back into the cost function gives C(40) = 2(1600) - 160(40) + 4200 = 3200 - 6400 + 4200 = 1000 thousand Kenya Shillings, or 1,000,000 KES. Allocating exactly 40 containers guarantees optimal cloud expenditure.

Quiz

1. According to the second derivative test, what does a positive second derivative at a critical point indicate?

2. How does gradient descent utilize differentiation during model optimization?

3. What condition must be satisfied for a differentiable function to have a critical point at x?

Sign in to submit and securely grade your quiz.