2 Estimation
In Probability, you calculated probabilities of events by assuming a probability model for data and then assuming you knew the value of the parameters in that model.
Suppose that \(X\) and \(Y\) represent independent flips of a coin (0 = tails, 1 = heads) with probability \(0.7\) of landing heads. If you flip a coin two times, what’s the probabiity of observing a tail and then a head (\(X = 0, Y = 1\))?
In Mathematical Statistics, we will similarly write down a probability model but then we will use observed data to estimate the value of the parameters in that model.
Suppose we flipped a coin two times and observed one tail and then one head. Would you be more willing to believe that the probability of getting a head (\(p\)) is 0.7 or 0.4? If \(p\) could be anywhere between 0 and 1, what’s your best guess about what that value of \(p\) is based on the observed data?
There is more than one technique that you can use to estimate the value of an unknown parameter. You’re already familiar with one technique—least squares estimation—from STAT 155. We’ll review the ideas behind that approach later in the course. To start, we’ll explore two other widely used estimation techniques: maximum likelihood estimation and the method of moments.
Learning Goals
- Understand the distinction between common parameter estimation methods (e.g., least squares, maximum likelihood, method of moments)
- Be able to implement common parameter estimation methods in a variety of settings and models
2.1 Maximum Likelihood Estimation
Textbook Reading Guide
Read: Section 5.1 and the first half of Section 5.2 (pages 278–288)
Definitions:
- parameter
- statistic/estimator
- estimate
- likelihood function – Definition 5.2.1
- maximum likelihood estimate – Definition 5.2.2
- log-likelihood
Note: wherever applicable, you should write down definitions for important vocab both “in words” and in mathematical notation.
Questions:
- What is the intuition behind the maximum likelihood estimation (MLE) approach?
- What are the typical steps to find a MLE? (see Ex 5.2.1, 5.2.2, and Case Study 5.2.1; work through at least one of these examples in detail, filling in any steps that the textbook left out)
- Are there ever situations when the typical steps to finding a MLE don’t work? If so, what can we do instead to find the MLE? (see Ex 5.2.3, 5.2.4)
- How do the steps to finding a MLE change when we have more than one unknown parameter? (see Ex 5.2.5)
Corresponding Videos
- MLE Intro
- MLE Example
- MLE Numerical Optimization
2.2 The Method of Moments
Textbook Reading Guide
Read: the second half of Section 5.2 (pages 289–293)
Definitions:
- theoretical moment
- sample moment
- method of moments estimates – Definition 5.2.3
Remember: wherever applicable, you should write down definitions for important vocab both “in words” and in mathematical notation.
Questions:
- What is the intuition behind the method of moments (MOM) procedure for estimating unknown parameters?
- What are the typical steps to find a MOM estimator? (see Ex 5.2.6, 5.2.7, and Case Study 5.2.2; work through at least one of these examples in detail, filling in any steps that the textbook left out)
- What advantages does the MOM approach offer compared to MLE?
- Do the MOM and MLE approaches always yield the same estimate? (look through the examples in Section 5.2 and try using the other approach — do you always get the same answer?)
Corresponding Videos
- MOM Intro
- MOM Example