|
|
{% extends "layout.html" %}
|
|
|
|
|
|
{% block content %}
|
|
|
<!DOCTYPE html>
|
|
|
<html lang="en">
|
|
|
<head>
|
|
|
<meta charset="UTF-8">
|
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
|
<title>Study Guide: Voting Ensemble</title>
|
|
|
|
|
|
<script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script>
|
|
|
<script id="MathJax-script" async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
|
|
|
<style>
|
|
|
|
|
|
body {
|
|
|
background-color: #ffffff;
|
|
|
color: #000000;
|
|
|
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
|
|
|
font-weight: normal;
|
|
|
line-height: 1.8;
|
|
|
margin: 0;
|
|
|
padding: 20px;
|
|
|
}
|
|
|
|
|
|
|
|
|
.container {
|
|
|
max-width: 800px;
|
|
|
margin: 0 auto;
|
|
|
padding: 20px;
|
|
|
}
|
|
|
|
|
|
|
|
|
h1, h2, h3 {
|
|
|
color: #000000;
|
|
|
border: none;
|
|
|
font-weight: bold;
|
|
|
}
|
|
|
|
|
|
h1 {
|
|
|
text-align: center;
|
|
|
border-bottom: 3px solid #000;
|
|
|
padding-bottom: 10px;
|
|
|
margin-bottom: 30px;
|
|
|
font-size: 2.5em;
|
|
|
}
|
|
|
|
|
|
h2 {
|
|
|
font-size: 1.8em;
|
|
|
margin-top: 40px;
|
|
|
border-bottom: 1px solid #ddd;
|
|
|
padding-bottom: 8px;
|
|
|
}
|
|
|
|
|
|
h3 {
|
|
|
font-size: 1.3em;
|
|
|
margin-top: 25px;
|
|
|
}
|
|
|
|
|
|
|
|
|
strong {
|
|
|
font-weight: 900;
|
|
|
}
|
|
|
|
|
|
|
|
|
p, li {
|
|
|
font-size: 1.1em;
|
|
|
border-bottom: 1px solid #e0e0e0;
|
|
|
padding-bottom: 10px;
|
|
|
margin-bottom: 10px;
|
|
|
}
|
|
|
|
|
|
|
|
|
li:last-child {
|
|
|
border-bottom: none;
|
|
|
}
|
|
|
|
|
|
|
|
|
ol {
|
|
|
list-style-type: decimal;
|
|
|
padding-left: 20px;
|
|
|
}
|
|
|
|
|
|
ol li {
|
|
|
padding-left: 10px;
|
|
|
}
|
|
|
|
|
|
|
|
|
ul {
|
|
|
list-style-type: none;
|
|
|
padding-left: 0;
|
|
|
}
|
|
|
|
|
|
ul li::before {
|
|
|
content: "•";
|
|
|
color: #000;
|
|
|
font-weight: bold;
|
|
|
display: inline-block;
|
|
|
width: 1em;
|
|
|
margin-left: 0;
|
|
|
}
|
|
|
|
|
|
|
|
|
pre {
|
|
|
background-color: #f4f4f4;
|
|
|
border: 1px solid #ddd;
|
|
|
border-radius: 5px;
|
|
|
padding: 15px;
|
|
|
white-space: pre-wrap;
|
|
|
word-wrap: break-word;
|
|
|
font-family: "Courier New", Courier, monospace;
|
|
|
font-size: 0.95em;
|
|
|
font-weight: normal;
|
|
|
color: #333;
|
|
|
border-bottom: none;
|
|
|
}
|
|
|
|
|
|
|
|
|
.story-voting {
|
|
|
background-color: #f0f9ff;
|
|
|
border-left: 4px solid #0d6efd;
|
|
|
margin: 15px 0;
|
|
|
padding: 10px 15px;
|
|
|
font-style: italic;
|
|
|
color: #555;
|
|
|
font-weight: normal;
|
|
|
border-bottom: none;
|
|
|
}
|
|
|
|
|
|
.story-voting p, .story-voting li {
|
|
|
border-bottom: none;
|
|
|
}
|
|
|
|
|
|
.example-voting {
|
|
|
background-color: #f3f8fe;
|
|
|
padding: 15px;
|
|
|
margin: 15px 0;
|
|
|
border-radius: 5px;
|
|
|
border-left: 4px solid #4dabf7;
|
|
|
}
|
|
|
|
|
|
.example-voting p, .example-voting li {
|
|
|
border-bottom: none !important;
|
|
|
}
|
|
|
|
|
|
|
|
|
.quiz-section {
|
|
|
background-color: #fafafa;
|
|
|
border: 1px solid #ddd;
|
|
|
border-radius: 5px;
|
|
|
padding: 20px;
|
|
|
margin-top: 30px;
|
|
|
}
|
|
|
.quiz-answers {
|
|
|
background-color: #f3f8fe;
|
|
|
padding: 15px;
|
|
|
margin-top: 15px;
|
|
|
border-radius: 5px;
|
|
|
}
|
|
|
|
|
|
|
|
|
table {
|
|
|
width: 100%;
|
|
|
border-collapse: collapse;
|
|
|
margin: 25px 0;
|
|
|
}
|
|
|
th, td {
|
|
|
border: 1px solid #ddd;
|
|
|
padding: 12px;
|
|
|
text-align: left;
|
|
|
}
|
|
|
th {
|
|
|
background-color: #f2f2f2;
|
|
|
font-weight: bold;
|
|
|
}
|
|
|
|
|
|
|
|
|
@media (max-width: 768px) {
|
|
|
body, .container {
|
|
|
padding: 10px;
|
|
|
}
|
|
|
h1 { font-size: 2em; }
|
|
|
h2 { font-size: 1.5em; }
|
|
|
h3 { font-size: 1.2em; }
|
|
|
p, li { font-size: 1em; }
|
|
|
pre { font-size: 0.85em; }
|
|
|
table, th, td { font-size: 0.9em; }
|
|
|
}
|
|
|
</style>
|
|
|
</head>
|
|
|
<body>
|
|
|
|
|
|
<div class="container">
|
|
|
<h1>🗳️ Study Guide: Voting Ensembles</h1>
|
|
|
|
|
|
<h2>🔹 1. Introduction</h2>
|
|
|
<div class="story-voting">
|
|
|
<p><strong>Story-style intuition: The Panel of Judges</strong></p>
|
|
|
<p>Imagine a talent show with a panel of three judges. Each judge (a <strong>base model</strong>) has a different background: one is an expert in singing, one in dancing, and one in comedy. After a performance, each judge gives their vote for whether the contestant should pass.
|
|
|
<br>• <strong>Hard Voting:</strong> The final decision is based on a simple majority. If two out of three judges vote "Pass," the contestant passes. This is a democratic vote where every judge has an equal say.
|
|
|
<br>• <strong>Soft Voting:</strong> Instead of a simple "yes" or "no," each judge provides a confidence score (e.g., "I'm 90% confident they should pass"). The final decision is based on the average confidence score across all judges. This method is often better because it accounts for the *certainty* of each judge's vote.
|
|
|
<br>A <strong>Voting Ensemble</strong> is this panel of judges, combining their diverse opinions to make a final decision that is often more robust and accurate than any single judge's opinion.</p>
|
|
|
</div>
|
|
|
<p>A <strong>Voting Ensemble</strong> is one of the simplest and most effective ensemble learning techniques. It works by training multiple different models on the same data and combining their predictions to generate a final output. Unlike Stacking, it does not use a meta-learner; instead, it relies on simple statistical methods like majority vote or averaging.</p>
|
|
|
|
|
|
<h2>🔹 2. How Voting Works</h2>
|
|
|
<p>The process is straightforward and can be run in parallel.</p>
|
|
|
|
|
|
<ol>
|
|
|
<li><strong>Train Diverse Base Models:</strong> Train several different machine learning models (e.g., a Logistic Regression, a Decision Tree, and a KNN) independently on the entire training dataset.</li>
|
|
|
<li><strong>Make Predictions:</strong> For a new data point, get a prediction from each of the trained models.</li>
|
|
|
<li><strong>Aggregate the Predictions:</strong> Combine the predictions using a voting rule.
|
|
|
<ul>
|
|
|
<li><strong>Hard Voting (for Classification):</strong> The final prediction is the class label that was predicted most frequently by the base models.</li>
|
|
|
<li><strong>Soft Voting (for Classification):</strong> The final prediction is the class label with the highest average predicted probability. This requires that the base models can output class probabilities.</li>
|
|
|
<li><strong>Averaging (for Regression):</strong> The final prediction is simply the average of the predictions from all the base models.</li>
|
|
|
</ul>
|
|
|
</li>
|
|
|
</ol>
|
|
|
|
|
|
<h2>🔹 3. Key Points</h2>
|
|
|
<ul>
|
|
|
<li><strong>Simplicity:</strong> It's one of the easiest ensemble methods to implement and understand.</li>
|
|
|
<li><strong>Model Diversity is Crucial:</strong> Voting works best when the base models are diverse and make different types of errors. Combining three identical models provides no benefit.</li>
|
|
|
<li><strong>Parallelizable:</strong> Since all base models are trained independently, the training process can be fully parallelized, making it computationally efficient.</li>
|
|
|
</ul>
|
|
|
|
|
|
<h2>🔹 4. Advantages & Disadvantages</h2>
|
|
|
<table>
|
|
|
<thead>
|
|
|
<tr>
|
|
|
<th>Advantages</th>
|
|
|
<th>Disadvantages</th>
|
|
|
</tr>
|
|
|
</thead>
|
|
|
<tbody>
|
|
|
<tr>
|
|
|
<td>✅ Very <strong>easy to implement</strong> and interpret.</td>
|
|
|
<td>❌ Often <strong>less powerful</strong> than more advanced ensembles like Boosting or Stacking.</td>
|
|
|
</tr>
|
|
|
<tr>
|
|
|
<td>✅ Can <strong>improve predictive accuracy</strong> and create a more robust model.</td>
|
|
|
<td>❌ It doesn't have a mechanism to explicitly correct the errors of its base models.</td>
|
|
|
</tr>
|
|
|
<tr>
|
|
|
<td>✅ Allows for the combination of <strong>different types of models</strong> (heterogeneous ensemble).</td>
|
|
|
<td>❌ The performance is highly dependent on the quality and diversity of the base models.</td>
|
|
|
</tr>
|
|
|
</tbody>
|
|
|
</table>
|
|
|
|
|
|
<h2>🔹 5. Python Implementation (Sketches)</h2>
|
|
|
<div class="story-voting">
|
|
|
<p>Scikit-learn provides convenient `VotingClassifier` and `VotingRegressor` classes that make building a voting ensemble very simple. You just need to provide a list of the models you want to include in the panel.</p>
|
|
|
</div>
|
|
|
<div class="example-voting">
|
|
|
<h3>Voting Classifier Example</h3>
|
|
|
<pre><code>
|
|
|
from sklearn.ensemble import VotingClassifier
|
|
|
from sklearn.linear_model import LogisticRegression
|
|
|
from sklearn.tree import DecisionTreeClassifier
|
|
|
from sklearn.svm import SVC
|
|
|
# Assume X_train, y_train, X_test are defined
|
|
|
|
|
|
# 1. Define the panel of judges (base models)
|
|
|
estimators = [
|
|
|
('lr', LogisticRegression(random_state=42)),
|
|
|
('dt', DecisionTreeClassifier(random_state=42)),
|
|
|
('svc', SVC(probability=True, random_state=42)) # probability=True is needed for soft voting
|
|
|
]
|
|
|
|
|
|
# 2. Create the Voting Ensemble
|
|
|
# 'soft' voting uses predicted probabilities and is often better
|
|
|
voting_clf = VotingClassifier(estimators=estimators, voting='soft')
|
|
|
|
|
|
# 3. Train and predict
|
|
|
voting_clf.fit(X_train, y_train)
|
|
|
y_pred = voting_clf.predict(X_test)
|
|
|
</code></pre>
|
|
|
|
|
|
<h3>Voting Regressor Example</h3>
|
|
|
<pre><code>
|
|
|
from sklearn.ensemble import VotingRegressor
|
|
|
from sklearn.linear_model import LinearRegression
|
|
|
from sklearn.ensemble import RandomForestRegressor
|
|
|
from sklearn.svm import SVR
|
|
|
# Assume X_train, y_train, X_test are defined
|
|
|
|
|
|
# 1. Define the panel of regressors
|
|
|
regressors = [
|
|
|
('lr', LinearRegression()),
|
|
|
('rf', RandomForestRegressor(random_state=42)),
|
|
|
('svr', SVR())
|
|
|
]
|
|
|
|
|
|
# 2. Create the Voting Ensemble (averages the predictions)
|
|
|
voting_reg = VotingRegressor(estimators=regressors)
|
|
|
|
|
|
# 3. Train and predict
|
|
|
voting_reg.fit(X_train, y_train)
|
|
|
y_pred_reg = voting_reg.predict(X_test)
|
|
|
</code></pre>
|
|
|
</div>
|
|
|
|
|
|
<h2>🔹 6. Applications</h2>
|
|
|
<ul>
|
|
|
<li><strong>Quick Ensemble Baseline:</strong> It's an excellent way to quickly build a baseline ensemble model to see if combining models is likely to improve performance on a given problem.</li>
|
|
|
<li><strong>Production Models:</strong> Due to its simplicity and robustness, a voting ensemble of a few strong, diverse models is often a good candidate for a reliable production system.</li>
|
|
|
<li>Used across many domains, including fraud detection, medical diagnosis, and customer churn prediction, just like other ensemble methods.</li>
|
|
|
</ul>
|
|
|
|
|
|
<div class="quiz-section">
|
|
|
<h2>📝 Quick Quiz: Test Your Knowledge</h2>
|
|
|
<ol>
|
|
|
<li><strong>What is the difference between Hard Voting and Soft Voting? Which one is usually preferred and why?</strong></li>
|
|
|
<li><strong>Does a Voting Ensemble learn from the mistakes of its base models?</strong></li>
|
|
|
<li><strong>You create a Voting Classifier with three identical, perfectly-trained Decision Tree models. Will this ensemble perform better than a single Decision Tree?</strong></li>
|
|
|
</ol>
|
|
|
<div class="quiz-answers">
|
|
|
<h3>Answers</h3>
|
|
|
<p><strong>1.</strong> <strong>Hard Voting</strong> uses a simple majority vote of the predicted class labels. <strong>Soft Voting</strong> averages the predicted probabilities for each class and chooses the class with the highest average probability. Soft voting is usually preferred because it accounts for how confident each model is in its prediction.</p>
|
|
|
<p><strong>2.</strong> No, it does not. A Voting Ensemble trains its models independently and combines their outputs with a fixed rule (voting/averaging). It does not have a mechanism to sequentially correct errors like Boosting does.</p>
|
|
|
<p><strong>3.</strong> No, it will perform exactly the same. Since all three models are identical, they will always produce the same output, and the majority vote will always be the same as the single model's prediction. Diversity is essential for a voting ensemble to be effective.</p>
|
|
|
</div>
|
|
|
</div>
|
|
|
|
|
|
<h2>🔹 Key Terminology Explained</h2>
|
|
|
<div class="story-voting">
|
|
|
<p><strong>The Story: Decoding the Judge's Scorecard</strong></p>
|
|
|
</div>
|
|
|
<ul>
|
|
|
<li>
|
|
|
<strong>Hard Voting:</strong>
|
|
|
<br>
|
|
|
<strong>What it is:</strong> A simple majority vote. The class with the most votes wins.
|
|
|
<br>
|
|
|
<strong>Story Example:</strong> Three judges vote. Judge 1: "Pass". Judge 2: "Fail". Judge 3: "Pass". The majority is "Pass" (2 out of 3), so the contestant passes.
|
|
|
</li>
|
|
|
<li>
|
|
|
<strong>Soft Voting:</strong>
|
|
|
<br>
|
|
|
<strong>What it is:</strong> A weighted vote based on predicted probabilities.
|
|
|
<br>
|
|
|
<strong>Story Example:</strong> Three judges give confidence scores. Judge 1: "90% Pass". Judge 2: "70% Fail". Judge 3: "60% Pass". The average probability for "Pass" is (0.9 + (1-0.7) + 0.6) / 3 = 0.6. The average for "Fail" is 0.4. Since 0.6 > 0.4, the contestant passes. This method captured the uncertainty of Judge 2.
|
|
|
</li>
|
|
|
</ul>
|
|
|
|
|
|
</div>
|
|
|
|
|
|
</body>
|
|
|
</html>
|
|
|
{% endblock %}
|
|
|
|
