Machine Learning & Supervised Learning Fundamentals
Subject: Machine Learning
High-yield concepts, mathematical formulas, and practice questions for Machine Learning and Supervised Learning.
Concept Summary
Machine Learning algorithms learn predictive patterns from data without being explicitly programmed. Supervised Learning handles labeled target outputs via Regression (predicting continuous numerical quantities) and Classification (predicting categorical labels). Core linear models include Linear Regression (optimized via MSE) and Logistic Regression (using log-loss and sigmoid activation for probability estimation).
Model generalization is governed by the Bias-Variance Tradeoff: underfitting stems from insufficient model complexity (high bias), while overfitting arises from excessive model capacity capturing sample noise (high variance). Overfitting is mitigated using Regularization (L1 Lasso for sparse feature selection, L2 Ridge for weight decay), Cross-Validation, Dropout, Early Stopping, and Ensemble Methods (Random Forests, Gradient Boosted Trees).
Model performance must be evaluated using domain-appropriate metrics: Accuracy can be misleading on imbalanced datasets, requiring Precision, Recall, F1-Score, and ROC-AUC curves. Feature scaling (standardization/normalization) is mandatory for distance-based models (KNN, SVM) and gradient-based solvers to ensure stable and unbiased convergence.
Key Revision Rules & Formulas
- 🧠 Supervised vs. Unsupervised vs. Reinforcement Learning: Supervised learning maps labeled inputs X -> Y (Regression & Classification). Unsupervised learning discovers hidden patterns/clusters in unlabeled data X (K-Means, PCA). Reinforcement learning optimizes actions via rewards and penalties in an environment (Q-Learning).
- 📈 Linear Regression & Mean Squared Error (MSE): Models continuous output y = w^T x + b. Loss function: MSE = (1/n) * ∑ (y_i - y_hat_i)^2. Convex loss function guarantees a single global minimum.
- 🎯 Logistic Regression & Sigmoid Function: Models probability of binary outcomes using the sigmoid activation σ(z) = 1 / (1 + e^-z), mapping real numbers z to probabilities (0, 1). Loss function: Binary Cross-Entropy L = -[y log(y_hat) + (1-y) log(1-y_hat)].
- ⚖️ Bias-Variance Tradeoff: High Bias -> Underfitting (model too simple, high training & validation error). High Variance -> Overfitting (model fits noise, low training error but high validation error). Total Error = Bias^2 + Variance + Irreducible Error.
- 🛑 Regularization (L1 Lasso vs. L2 Ridge): L1 (Lasso) adds λ ∑ |w_i| penalty, forcing irrelevant feature weights to exactly ZERO (performing feature selection). L2 (Ridge) adds λ ∑ w_i^2 penalty, shrinking weights towards zero without making them zero.
- 📊 Confusion Matrix & Evaluation Metrics: Precision = TP / (TP + FP) (Quality of positive predictions). Recall (Sensitivity) = TP / (TP + FN) (Coverage of actual positives). F1-Score = 2 * (Precision * Recall) / (Precision + Recall) (Harmonic mean).
- ⚡ Gradient Descent Optimization: Weight update rule: w^(t+1) = w^(t) - α ∇L(w). Learning rate α controls step size. Too large α -> divergence/oscillation; too small α -> extremely slow convergence.
- 🌳 Decision Trees & Impurity Metrics: Splits nodes by maximizing Information Gain or minimizing impurity. Gini Impurity = 1 - ∑ p_i^2. Entropy H(S) = -∑ p_i log2(p_i). Decision trees are non-parametric and prone to overfitting if not pruned.
- 📐 K-Nearest Neighbors (KNN) & Scaling: Non-parametric, instance-based algorithm. Classification by majority vote of k closest neighbors. Highly sensitive to feature scales, requiring Z-score normalization or Min-Max scaling prior to distance calculation.
- 🔄 K-Fold Cross-Validation: Partitions dataset into K equal subsets (folds). Trains model on K-1 folds and tests on the remaining 1 fold, repeating K times to provide unbiased performance estimates without data leakage.
Common Exam Pitfalls
- Mistake: Relying solely on Classification Accuracy for imbalanced datasets (e.g., 99% accuracy on a dataset with 99% negative cases while missing all positive cases). Use Precision, Recall, or F1-Score instead.
- Mistake: Confusing L1 (Lasso) and L2 (Ridge) regularization. L1 produces sparse weights (zeroing parameters for feature selection), whereas L2 shrinks weights smoothly without setting them strictly to zero.
- Mistake: Forgetting to scale features before running distance-based algorithms like KNN or SVM. Unscaled features with large numerical ranges will dominate distance metrics.
- Mistake: Setting the Gradient Descent learning rate α too high, causing loss to explode or oscillate endlessly around the minimum instead of converging.
- Mistake: Performing feature normalization or imputation on the ENTIRE dataset before splitting into Train/Test sets, causing severe Data Leakage.
Sample Practice Questions
Question 1: Which of the following machine learning algorithms is an UNSUPERVISED learning algorithm used for clustering data into k distinct groups?
- Logistic Regression
- K-Means Clustering
- Support Vector Machine (SVM)
- Random Forest Classifier
Explanation: K-Means is an unsupervised clustering algorithm that groups unlabeled data points into K clusters by iteratively updating cluster centroids.
Question 2: In the Bias-Variance Tradeoff, what typically happens when a model's complexity is excessively high (e.g., a deep decision tree with no depth limit)?
- High Bias and Low Variance (Underfitting)
- Low Bias and High Variance (Overfitting)
- Low Bias and Low Variance (Ideal Generalization)
- High Bias and High Variance
Explanation: Excessively complex models capture training noise, leading to low training error (low bias) but poor test generalization (high variance / overfitting).
Question 3: Which mathematical activation function is used in Logistic Regression to map any real-valued input z into a probability range between 0 and 1?
- ReLU (Rectified Linear Unit)
- Softmax Function
- Sigmoid (Logistic) Function
- Hyperbolic Tangent (tanh)
Explanation: The Sigmoid function σ(z) = 1 / (1 + e^-z) outputs values in (0, 1), interpreting outputs as probabilities for binary classification.
Question 4: How does L1 Regularization (Lasso) differ from L2 Regularization (Ridge) in linear models?
- L1 adds squared weights penalty; L2 adds absolute weights penalty
- L1 forces some weights strictly to zero (feature selection); L2 shrinks weights near zero
- L1 is used only for classification; L2 is used only for regression
- L1 increases model variance; L2 increases model bias
Explanation: L1 regularization (λ ∑|w|) produces sparse weight vectors by driving uninformative feature weights to exactly 0, serving as feature selection.
Question 5: In a medical diagnostic ML model for detecting a rare life-threatening disease, which evaluation metric is MOST critical to maximize to minimize missed positive cases (False Negatives)?
- Precision
- Recall (Sensitivity)
- Overall Accuracy
- Specificity
Explanation: Recall = TP / (TP + FN). Maximizing Recall minimizes False Negatives, ensuring sick patients are not incorrectly missed.
Question 6: What happens during Gradient Descent optimization if the learning rate α is set too large?
- The algorithm converges to the global minimum very slowly
- The loss function oscillates or diverges, missing the minimum entirely
- The model underfits due to zero weight updates
- The gradient becomes strictly zero at the first step
Explanation: A learning rate α that is too large causes parameter updates to overshoot the minimum, leading to loss oscillation or divergence.
Question 7: Which metric measures node impurity in a Decision Tree using the formula H(S) = -∑ p_i * log2(p_i)?
- Gini Impurity
- Mean Absolute Error
- Shannon Entropy
- Variance Reduction
Explanation: Shannon Entropy measures expected information / impurity in a dataset. Information Gain is calculated by the reduction in Entropy after a split.
Question 8: Why is feature scaling (Standardization / Normalization) CRITICAL before applying distance-based algorithms like K-Nearest Neighbors (KNN)?
- KNN cannot process non-integer values
- Features with larger magnitude ranges will dominate the Euclidean distance calculation
- Scaling converts non-linear boundaries into linear ones
- Scaling eliminates the need to choose the hyperparameter k
Explanation: Euclidean distance d = √(∑(x_i - y_i)^2) is heavily skewed by features with large scale ranges (e.g. Salary in 100,000s vs Age in 10s).
Question 9: Which of the following techniques is NOT an effective method to combat overfitting in Machine Learning models?
- Increasing dataset size through Data Augmentation
- Applying L1 or L2 Regularization penalties
- Increasing model parameters and polynomial feature degree without bound
- Using K-Fold Cross-Validation and Early Stopping
Explanation: Increasing model complexity (higher degree polynomials / unbounded parameters) amplifies model capacity and worsens overfitting.
Question 10: What is the primary purpose of performing K-Fold Cross-Validation during machine learning model development?
- To automatically calculate optimal gradient descent learning rates
- To evaluate model generalization across K distinct validation folds without data leakage
- To compress the feature space into orthogonal principal components
- To convert supervised dataset labels into unsupervised cluster centers
Explanation: K-Fold Cross-Validation splits data into K subsets to evaluate model stability and generalization across multiple unseen validation splits.