Ridge regression differs from ordinary least-squares linear regression chiefly by adding, to the objective being minimised,

an L2 penalty on the coefficients

an L1 penalty on the coefficients

an L2 penalty on the coefficients

a penalty on the number of training examples used

a constraint forcing every coefficient to be positive

Logistic regression is used for

classification, via a probability estimated by the sigmoid function

predicting a continuous numeric response by least squares

classification, via a probability estimated by the sigmoid function

unsupervised clustering of unlabelled data

dimensionality reduction by finding directions of maximum variance

Using k = 1 in k-nearest neighbour classification, compared with a larger k, tends to give

low bias and high variance, since a single noisy neighbour can flip the prediction

zero training time and zero prediction-time cost

a decision boundary that is always linear

The naive Bayes classifier makes which simplifying assumption?

Features are conditionally independent of each other, given the class

The classes must be linearly separable

Every class must have exactly the same prior probability

Which of the following are true of a support vector machine's maximum-margin hyperplane?

It is determined only by the support vectors sitting on the margin boundary

The kernel trick allows a nonlinear decision boundary without explicitly computing the higher-dimensional features

Soft-margin SVM training is a convex optimisation problem

It is determined only by the support vectors sitting on the margin boundary

Moving a correctly classified point that lies well outside the margin can change the hyperplane

The kernel trick allows a nonlinear decision boundary without explicitly computing the higher-dimensional features

Soft-margin SVM training is a convex optimisation problem

Linear discriminant analysis (LDA), compared with logistic regression, additionally assumes

classes share a common covariance matrix and features are approximately normal within each class

features are conditionally independent given the class

classes share a common covariance matrix and features are approximately normal within each class

the decision boundary must be nonlinear

there must be more than two classes present

A decision tree grown to full depth on the training data, with no pruning, typically

overfits — low training error, but higher variance and worse accuracy on unseen data

achieves the best possible accuracy on unseen test data

overfits — low training error, but higher variance and worse accuracy on unseen data

cannot be used for classification, only for regression

Leave-one-out cross-validation is the special case of k-fold cross-validation in which

k equals the number of training examples, n

k is selected by an inner grid search

A feed-forward neural network with a single hidden layer of sufficient width can, by the universal approximation property,

only represent linear functions

guarantee zero training error on any dataset whatsoever

operate without any nonlinear activation function at all

At every iteration, k-means updates each cluster's centre to the

mean of the points currently assigned to that cluster

nearest actual data point (a medoid)

mean of the points currently assigned to that cluster

point with the maximum distance from every other point

median of the previous iteration's centroids only

Compared with k-means, k-medoid clustering is generally

more robust to outliers, at a higher computational cost

faster, and less robust to outliers

more robust to outliers, at a higher computational cost

incapable of forming more than two clusters

identical to k-means in every respect

Single-linkage hierarchical clustering defines the distance between two clusters as

the minimum distance between any pair, one point from each cluster

the average of all pairwise distances between the two clusters

the maximum distance between any pair, one point from each cluster

the minimum distance between any pair, one point from each cluster

the distance between the two clusters' centroids only

Compared with complete-linkage, single-linkage hierarchical clustering is more prone to

the 'chaining' effect, where a sequence of close points links otherwise distant clusters

producing perfectly spherical clusters

the 'chaining' effect, where a sequence of close points links otherwise distant clusters

requiring k to be fixed in advance, unlike complete-linkage

The principal components produced by PCA are computed from

the roots of the characteristic polynomial of the class-label vector

a random orthogonal projection chosen independently of the data

the medians of each feature considered separately

Before applying PCA to features on very different numeric scales — say income in rupees and age in years — standard practice is to

standardize the features first, so no feature dominates purely due to its units

discard the smaller-scale feature entirely

standardize the features first, so no feature dominates purely due to its units

apply PCA only to the larger-scale feature

run k-means clustering first, before any PCA

Reducing a dataset's dimensionality before feeding it to a distance-based classifier such as k-NN chiefly helps by

mitigating the curse of dimensionality, where distances become less meaningful as dimensions grow

artificially increasing the number of features

mitigating the curse of dimensionality, where distances become less meaningful as dimensions grow

guaranteeing higher classification accuracy on every dataset without exception

removing the need for any labelled training data at all

In ridge regression, as the penalty weight λ is increased toward infinity, the fitted coefficients tend toward

all zero, giving a constant-mean model

the ordinary least-squares solution

all zero, giving a constant-mean model

the exact values of the true population coefficients

Which of the following are true of naive Bayes and k-nearest neighbour?

k-NN performs no explicit training phase, pushing the work to prediction time

Naive Bayes estimates a joint distribution from independent per-feature distributions

k-NN performs no explicit training phase, pushing the work to prediction time

Naive Bayes estimates a joint distribution from independent per-feature distributions

k-NN's prediction cost is independent of the size of the training set

Naive Bayes requires its conditional-independence assumption to hold exactly, or it cannot be used at all

Running k-means several times from different random initial centroids and keeping the best result mainly helps because

k-means converges only to a local optimum, which depends on the initial centroids

k-means is guaranteed to find the global optimum on its very first run

k-means converges only to a local optimum, which depends on the initial centroids

it removes the need to choose the number of clusters k in advance

it forces every resulting cluster to be perfectly spherical

Machine Learning

Section 6 of the GATE Data Science and Artificial Intelligence (DA) paper, and the largest single section by the number of named methods. It is really two subjects sharing one heading. Supervised learning trains on examples that each carry the answer they are graded against — regression when that answer is a number, classification when it is a category — and this chapter works through simple, multiple and ridge regression, logistic regression, k-nearest neighbour, naive Bayes, linear discriminant analysis, the support vector machine, decision trees, and the bias-variance trade-off and cross-validation that decide whether any of them actually generalise, plus the multi-layer perceptron. Unsupervised learning trains on examples with no answer attached at all, and gets its own full treatment here rather than an afterthought: clustering (k-means, k-medoid, hierarchical) finds structure in unlabelled data, and dimensionality reduction, principal component analysis above all, finds a smaller set of directions that still describes most of it.

1. Regression: simple, multiple and ridge

Simple linear regression fits y = β₀ + β₁x by choosing β₀, β₁ to minimise the sum of squared residuals (ordinary least squares) between the fitted line and the observed points. Multiple linear regression extends this to several predictors, y = β₀ + β₁x₁ + ... + βₖxₖ; each coefficient is then read as the change in y for a one-unit change in that predictor, holding every other predictor fixed — a very different statement from that predictor's effect on its own.

Ridge regression adds an L2 penalty, λΣβᵢ², to the least-squares objective, shrinking every coefficient toward zero by an amount that grows with λ. This trades a little bias for a often-large reduction in variance, which is exactly the useful move when predictors are numerous or correlated (multicollinearity) — in that setting ordinary least squares can produce wildly unstable coefficient estimates that swing on small changes to the data, and ridge's shrinkage stabilises them at the cost of no longer being exactly unbiased.

🎯 Ridge shrinks every coefficient; it never zeroes one out

Because the L2 penalty term βᵢ² is smooth and differentiable everywhere including at zero, the optimum it pushes toward shrinks coefficients continuously without ever forcing one exactly to zero — ridge regression keeps every predictor in the model, just with a smaller weight. This is precisely why it is described as a shrinkage method rather than a feature-selection method.

2. Classification I: logistic regression, k-nearest neighbour, naive Bayes

Despite its name, logistic regression is a classifier: it models P(y=1 | x) as the sigmoid of a linear combination of the predictors, and is trained by maximising likelihood (equivalently, minimising cross-entropy loss) rather than by least squares. k-nearest neighbour classifies a new point by a majority vote among the k closest labelled points in the training set, by some distance metric — it does no explicit training at all (a 'lazy' learner), pushing all the work to prediction time, and its cost per prediction scales with the size of the training set rather than with model complexity.

Naive Bayes applies Bayes' theorem to classification with one simplifying ('naive') assumption: every feature is conditionally independent of every other, given the class. That assumption is rarely exactly true, but it reduces estimating one large joint distribution to estimating several small per-feature distributions, which is cheap, needs little training data, and in practice classifies well even when the independence it assumes is visibly violated.

⚠️ A very small k in k-NN is not automatically 'more accurate'

k = 1 fits the training data closest of all, giving low bias, but it also gives the highest variance: the decision at any point depends entirely on whichever single training example happens to be nearest, so a noisy or mislabelled neighbour changes the prediction outright. Larger k averages over more neighbours, raising bias (the boundary becomes smoother, and can blur a genuinely sharp one) while lowering variance — the choice is a direct instance of the bias-variance trade-off, not a search for the smallest possible k.

3. Classification II: linear discriminant analysis, support vector machine, decision trees

Linear discriminant analysis (LDA) finds the linear combination of features that best separates the classes — maximising the ratio of between-class scatter to within-class scatter — under the assumption that classes are (approximately) normally distributed and share a common covariance matrix. Where that assumption holds, LDA can outperform logistic regression, and it doubles as a dimensionality-reduction technique, projecting data onto the directions that best discriminate the classes rather than, as PCA does, the directions of highest variance regardless of class.

A support vector machine (SVM) finds the hyperplane that separates the classes with the maximum margin — the widest possible gap to the nearest point of either class. Only the points exactly on that margin, the support vectors, determine where the hyperplane sits; every other correctly classified point could be moved or removed without changing it at all. The kernel trick lets an SVM find a nonlinear boundary in the original feature space by implicitly computing distances as if the data had been mapped into a much higher-dimensional space, without ever constructing that mapping explicitly.

🧠 Only the support vectors matter

A quick way to check whether a point influenced an SVM's boundary: if it sits strictly outside the margin and is correctly classified, it did not, and could be deleted from the training set with the fitted hyperplane staying exactly where it is. This is the opposite of, say, ordinary least squares, where every single point contributes to the fit.

A decision tree recursively splits the feature space on the attribute and threshold that most reduces impurity (measured by Gini impurity or entropy) at each node, until a stopping rule is reached. It is easy to read off as a set of rules, but a tree grown to full depth on the training data fits its noise as readily as its signal — low training error paired with much higher variance on unseen data — which is why depth limits and pruning are the standard countermeasure.

4. Bias-variance trade-off, cross-validation, and the multi-layer perceptron

A model's expected prediction error on new data decomposes into three parts: bias (error from a model too simple to capture the true relationship — underfitting), variance (error from a model sensitive enough to fit the training data's noise — overfitting), and irreducible noise (error no model can remove). Increasing model complexity generally lowers bias and raises variance, so the two trade against each other rather than both improving together, and the goal is the complexity that minimises their sum, not either one alone.

💡 The archery-target picture of bias and variance

Think of several shots at a target. High bias, low variance is a tight cluster of arrows that all land off-centre in the same spot — consistent, but consistently wrong. Low bias, high variance is arrows scattered widely but centred around the bullseye on average — right on average, but any one shot is unreliable. The model you actually want lands both close together and close to the centre, and complexity is the dial that moves you along this trade-off rather than an escape from it.

Cross-validation estimates out-of-sample error without setting aside a separate, never-touched test set for that purpose alone. k-fold cross-validation splits the data into k roughly equal folds, trains on k-1 of them and validates on the remaining one, repeats this k times so every fold serves as the validation set exactly once, and averages the k error estimates. Leave-one-out (LOO) cross-validation is the extreme case k = n: it trains on all but one example, n times over, giving an estimate with very low bias but higher variance across folds and a training cost of n full model fits — expensive on a large dataset, though it uses almost all the data for training on every single fold.

A multi-layer perceptron (a feed-forward neural network) stacks layers, each a weighted sum of the previous layer's outputs passed through a nonlinear activation function; with even a single hidden layer of sufficient width, it can approximate any continuous function on a compact domain to arbitrary accuracy — the universal approximation property. It is trained by backpropagation: the chain rule applied layer by layer to compute how the loss changes with each weight, feeding a gradient-descent update.

5. Unsupervised learning I: clustering

k-means partitions data into k clusters by iterating two steps: assign every point to its nearest cluster centroid, then recompute each centroid as the mean of the points now assigned to it — repeated until assignments stop changing. It converges, but only to a local optimum that depends on the initial centroid positions, and because it uses the mean it is pulled noticeably by outliers. k-medoid does the same two-step iteration but restricts a cluster's centre to be an actual data point (the medoid) rather than a computed mean, which is more robust to outliers at the cost of a more expensive search for the best medoid at each step.

⚠️ k-means assumes roughly spherical, similarly sized clusters

Because it assigns points by distance to a single centroid, k-means implicitly assumes clusters that are compact and roughly spherical, of comparable size and density — it does badly on elongated, unevenly sized, or non-convex clusters, and k must be chosen before it runs rather than discovered by it. None of these assumptions are announced by the algorithm itself; they only surface when it is applied to data that violates them and produces clusters that look wrong to the eye.

Hierarchical clustering builds a full tree of nested clusters (a dendrogram) without requiring k in advance — the number of clusters is chosen afterward by cutting the tree at a chosen height. Agglomerative (bottom-up) clustering starts with every point as its own cluster and repeatedly merges the closest pair; divisive (top-down) clustering starts with one cluster containing everything and repeatedly splits the least cohesive one.

Linkage criteria for agglomerative hierarchical clustering
Linkage	Distance between two clusters	Tendency
Single-linkage	Minimum distance between any pair, one point from each cluster	Can chain distant clusters together through a sequence of close points
Complete-linkage (multiple-linkage)	Maximum distance between any pair, one point from each cluster	Produces more compact, roughly equal-sized clusters

6. Unsupervised learning II: dimensionality reduction and principal component analysis

Dimensionality reduction re-expresses data in fewer dimensions while keeping as much of its structure as possible, both to fight the curse of dimensionality (distances between points become less meaningful as dimensions grow, and the amount of data needed to fill a space grows exponentially with it) and to make high-dimensional data visualisable at all.

Principal component analysis (PCA) finds the orthogonal directions of maximum variance in the data, ranked from most to least variance explained; these principal components are the eigenvectors of the data's covariance matrix (equivalently obtainable via the singular value decomposition of the data matrix), and keeping only the first few of them — the ones that explain most of the total variance — is what actually achieves the reduction, discarding the rest with the least possible loss of information as measured by variance.

⚠️ PCA does not look at class labels — LDA is the technique that does

PCA is entirely unsupervised: it finds the directions of maximum variance in the data as a whole, with no notion of which points belong to which class, so the direction it ranks first can easily be one that mixes two classes together while a lower-variance direction would have separated them cleanly. LDA (Section 3) is the supervised counterpart built specifically to find the direction that best separates known classes — the two are easy to conflate because both project data onto a smaller set of directions, but they optimise for entirely different things.

ℹ️ Standardize features before PCA when their scales differ

Because PCA ranks directions purely by variance, a feature measured in units that happen to produce large numeric values — income in rupees versus age in years, say — will dominate the variance and hence the first principal component for no substantive reason at all. Standardizing every feature to zero mean and unit variance first removes this purely-a-units artefact before PCA ever runs.

Key takeaways

Ridge regression adds an L2 penalty that shrinks every coefficient toward zero, trading a little bias for reduced variance when predictors are numerous or correlated — but it never sets a coefficient to exactly zero.
Logistic regression, k-NN and naive Bayes reach classification three different ways — a parametric probability model, a lazy distance-based vote, and Bayes' theorem plus a conditional-independence assumption — and only the support vectors on an SVM's margin determine its boundary.
Model complexity trades bias against variance; k-fold cross-validation, and its extreme case leave-one-out, estimate out-of-sample error without setting aside a separate held-out set.
A feed-forward neural network with even one sufficiently wide hidden layer can approximate any continuous function, trained by backpropagating the loss's gradient layer by layer.
k-means and k-medoid both assume roughly spherical clusters and need k fixed in advance; hierarchical clustering needs no k in advance, but its shape depends heavily on whether the linkage rule is single (chains) or complete (compact).
PCA finds the directions of maximum variance from the covariance matrix's eigenvectors and is entirely unsupervised — it can discard the very direction that would best separate classes, which only a supervised technique like LDA looks for.

Practice questions (23)

Attempt each one before opening the answer. Every explanation names the tempting wrong option as well as the right one, because that is where marks are lost.

Ridge regression differs from ordinary least-squares linear regression chiefly by adding, to the objective being minimised,
1. an L1 penalty on the coefficients
2. an L2 penalty on the coefficients
3. a penalty on the number of training examples used
4. a constraint forcing every coefficient to be positive
Show answer
Answer: B — an L2 penalty on the coefficients
Ridge regression adds λΣβᵢ², the squared (L2) magnitude of the coefficients, to the least-squares objective, shrinking them toward zero.
Logistic regression is used for
1. predicting a continuous numeric response by least squares
2. classification, via a probability estimated by the sigmoid function
3. unsupervised clustering of unlabelled data
4. dimensionality reduction by finding directions of maximum variance
Show answer
Answer: B — classification, via a probability estimated by the sigmoid function
Despite its name, logistic regression is a classifier: it models P(y=1|x) as the sigmoid of a linear combination of the predictors.
Using k = 1 in k-nearest neighbour classification, compared with a larger k, tends to give
1. high bias and low variance
2. low bias and high variance, since a single noisy neighbour can flip the prediction
3. zero training time and zero prediction-time cost
4. a decision boundary that is always linear
Show answer
Answer: B — low bias and high variance, since a single noisy neighbour can flip the prediction
With k=1 the prediction at any point depends entirely on the single nearest training example, which fits the training data closely (low bias) but makes the prediction highly sensitive to noise in that one neighbour (high variance).
The naive Bayes classifier makes which simplifying assumption?
1. All features must be numeric
2. Features are conditionally independent of each other, given the class
3. The classes must be linearly separable
4. Every class must have exactly the same prior probability
Show answer
Answer: B — Features are conditionally independent of each other, given the class
The 'naive' in naive Bayes is exactly this conditional-independence assumption, which lets a joint distribution be approximated from independent per-feature distributions.
Which of the following are true of a support vector machine's maximum-margin hyperplane?
1. It is determined only by the support vectors sitting on the margin boundary
2. Moving a correctly classified point that lies well outside the margin can change the hyperplane
3. The kernel trick allows a nonlinear decision boundary without explicitly computing the higher-dimensional features
4. Soft-margin SVM training is a convex optimisation problem
Show answer
Answer: A — It is determined only by the support vectors sitting on the margin boundary; C — The kernel trick allows a nonlinear decision boundary without explicitly computing the higher-dimensional features; D — Soft-margin SVM training is a convex optimisation problem
A point well outside the margin, correctly classified, contributes nothing to the hyperplane and can be moved or removed without effect — the second option is the false one; the other three are standard properties of the SVM formulation.
Linear discriminant analysis (LDA), compared with logistic regression, additionally assumes
1. features are conditionally independent given the class
2. classes share a common covariance matrix and features are approximately normal within each class
3. the decision boundary must be nonlinear
4. there must be more than two classes present
Show answer
Answer: B — classes share a common covariance matrix and features are approximately normal within each class
LDA's derivation assumes classes are approximately normally distributed with a shared covariance matrix, an assumption logistic regression does not need to make.
A decision tree grown to full depth on the training data, with no pruning, typically
1. underfits, having high bias
2. achieves the best possible accuracy on unseen test data
3. overfits — low training error, but higher variance and worse accuracy on unseen data
4. cannot be used for classification, only for regression
Show answer
Answer: C — overfits — low training error, but higher variance and worse accuracy on unseen data
A fully grown tree fits the training data's noise as readily as its signal, giving low training error but higher variance and worse generalisation — the standard reason for depth limits and pruning.
In the bias-variance decomposition of expected prediction error, a model that is too simple to capture the true relationship suffers mainly from
1. high variance
2. high bias
3. irreducible noise alone
4. zero error, by definition
Show answer
Answer: B — high bias
An overly simple model underfits consistently, regardless of which training sample it saw — that consistent, systematic error is exactly what bias measures.
In 5-fold cross-validation on a dataset of 100 examples, split into equal folds, how many examples are used for validation in each single fold?
Numerical answer — type the value.
Show answer
Answer: 20
100 examples split into 5 equal folds gives 20 examples per fold, and one fold at a time is held out for validation.
Leave-one-out cross-validation is the special case of k-fold cross-validation in which
1. k = 1
2. k = 2
3. k equals the number of training examples, n
4. k is selected by an inner grid search
Show answer
Answer: C — k equals the number of training examples, n
LOO trains on all but one example and validates on that one, repeated n times over — exactly k-fold cross-validation with k = n.
A feed-forward neural network with a single hidden layer of sufficient width can, by the universal approximation property,
1. only represent linear functions
2. approximate any continuous function on a compact domain
3. guarantee zero training error on any dataset whatsoever
4. operate without any nonlinear activation function at all
Show answer
Answer: B — approximate any continuous function on a compact domain
The universal approximation property states that a single sufficiently wide hidden layer, with a nonlinear activation, can approximate any continuous function on a compact domain to arbitrary accuracy.
At every iteration, k-means updates each cluster's centre to the
1. nearest actual data point (a medoid)
2. mean of the points currently assigned to that cluster
3. point with the maximum distance from every other point
4. median of the previous iteration's centroids only
Show answer
Answer: B — mean of the points currently assigned to that cluster
k-means recomputes each centroid as the arithmetic mean of the points currently in its cluster, which is also why it is pulled by outliers.
Compared with k-means, k-medoid clustering is generally
1. faster, and less robust to outliers
2. more robust to outliers, at a higher computational cost
3. incapable of forming more than two clusters
4. identical to k-means in every respect
Show answer
Answer: B — more robust to outliers, at a higher computational cost
Using an actual data point (the medoid) instead of the mean as a cluster centre resists the pull of outliers, but searching for the best medoid at each step costs more than simply averaging points.
Single-linkage hierarchical clustering defines the distance between two clusters as
1. the average of all pairwise distances between the two clusters
2. the maximum distance between any pair, one point from each cluster
3. the minimum distance between any pair, one point from each cluster
4. the distance between the two clusters' centroids only
Show answer
Answer: C — the minimum distance between any pair, one point from each cluster
Single-linkage takes the closest pair of points across the two clusters as the inter-cluster distance, which is what lets it chain distant clusters together through a sequence of close points.
Compared with complete-linkage, single-linkage hierarchical clustering is more prone to
1. producing perfectly spherical clusters
2. the 'chaining' effect, where a sequence of close points links otherwise distant clusters
3. requiring k to be fixed in advance, unlike complete-linkage
4. ignoring outliers entirely
Show answer
Answer: B — the 'chaining' effect, where a sequence of close points links otherwise distant clusters
Because single-linkage only needs one close pair to merge two clusters, a chain of near-neighbours can link clusters that are otherwise far apart overall — the defect complete-linkage's max-distance rule avoids.
Which clustering technique does NOT require the number of clusters k to be fixed before it runs?
1. k-means
2. k-medoid
3. Hierarchical clustering
4. None of the listed methods
Show answer
Answer: C — Hierarchical clustering
Hierarchical clustering builds the full dendrogram first and only afterward chooses how many clusters to read off by cutting it at a chosen height; k-means and k-medoid both need k as an input before they start.
Principal component analysis re-expresses data along directions of
1. maximum class separation
2. maximum variance
3. minimum variance
4. a random orthogonal projection
Show answer
Answer: B — maximum variance
PCA's principal components are ranked directions of decreasing variance in the data, found without any reference to class labels.
The principal components produced by PCA are computed from
1. the eigenvectors of the data's covariance matrix
2. the roots of the characteristic polynomial of the class-label vector
3. a random orthogonal projection chosen independently of the data
4. the medians of each feature considered separately
Show answer
Answer: A — the eigenvectors of the data's covariance matrix
The principal components are the eigenvectors of the covariance matrix, equivalently obtainable from the singular value decomposition of the (centred) data matrix.
Before applying PCA to features on very different numeric scales — say income in rupees and age in years — standard practice is to
1. discard the smaller-scale feature entirely
2. standardize the features first, so no feature dominates purely due to its units
3. apply PCA only to the larger-scale feature
4. run k-means clustering first, before any PCA
Show answer
Answer: B — standardize the features first, so no feature dominates purely due to its units
Since PCA ranks directions purely by variance, a feature with naturally larger numeric values dominates that variance for no substantive reason unless every feature is first standardized to a common scale.
Reducing a dataset's dimensionality before feeding it to a distance-based classifier such as k-NN chiefly helps by
1. artificially increasing the number of features
2. mitigating the curse of dimensionality, where distances become less meaningful as dimensions grow
3. guaranteeing higher classification accuracy on every dataset without exception
4. removing the need for any labelled training data at all
Show answer
Answer: B — mitigating the curse of dimensionality, where distances become less meaningful as dimensions grow
In very high dimensions, distances between points become less discriminating (the curse of dimensionality), and dimensionality reduction is one standard way to counter that before a distance-based method like k-NN is applied.
In ridge regression, as the penalty weight λ is increased toward infinity, the fitted coefficients tend toward
1. the ordinary least-squares solution
2. all zero, giving a constant-mean model
3. infinity themselves
4. the exact values of the true population coefficients
Show answer
Answer: B — all zero, giving a constant-mean model
As λ grows, the penalty on the coefficients' size dominates the objective more and more, shrinking every coefficient toward zero — in the limit, the model predicts only the mean response.
Which of the following are true of naive Bayes and k-nearest neighbour?
1. k-NN performs no explicit training phase, pushing the work to prediction time
2. Naive Bayes estimates a joint distribution from independent per-feature distributions
3. k-NN's prediction cost is independent of the size of the training set
4. Naive Bayes requires its conditional-independence assumption to hold exactly, or it cannot be used at all
Show answer
Answer: A — k-NN performs no explicit training phase, pushing the work to prediction time; B — Naive Bayes estimates a joint distribution from independent per-feature distributions
k-NN's prediction cost actually grows with training-set size, since it must find the nearest neighbours among all of it, and naive Bayes classifies well in practice even when its independence assumption is visibly violated — the third and fourth options are the false ones.
Running k-means several times from different random initial centroids and keeping the best result mainly helps because
1. k-means is guaranteed to find the global optimum on its very first run
2. k-means converges only to a local optimum, which depends on the initial centroids
3. it removes the need to choose the number of clusters k in advance
4. it forces every resulting cluster to be perfectly spherical
Show answer
Answer: B — k-means converges only to a local optimum, which depends on the initial centroids
k-means only guarantees convergence to A local optimum, and which one depends on where the centroids started — running it multiple times and keeping the best (lowest within-cluster variance) result is the standard way to reduce this sensitivity.