TESTING HYPOTHESES WITH DATA

ST101 – DR. ARIC LABARR


A hypothesis test uses data to help evaluate an initial claim about a parameter from the population.



HYPOTHESIS TESTING


I have a coin that you believe is fair to start.

To test if this coin is fair, you ask me to flip the coin repeatedly and record the results.



HYPOTHESIS TESTING THROUGH EXAMPLE


I have a coin that you believe is fair to start.

To test if this coin is fair, you ask me to flip the coin repeatedly and record the results.



HYPOTHESIS TESTING THROUGH EXAMPLE

Flip Number

Result

Probability

1

Heads

0.50


I have a coin that you believe is fair to start.

To test if this coin is fair, you ask me to flip the coin repeatedly and record the results.



HYPOTHESIS TESTING THROUGH EXAMPLE

Flip Number

Result

Probability

1

Heads

0.50

2

Heads


I have a coin that you believe is fair to start.

To test if this coin is fair, you ask me to flip the coin repeatedly and record the results.



HYPOTHESIS TESTING THROUGH EXAMPLE

Flip Number

Result

Probability

1

Heads

0.50

2

Heads

0.25

Do you still think the coin is fair?


I have a coin that you believe is fair to start.

To test if this coin is fair, you ask me to flip the coin repeatedly and record the results.



HYPOTHESIS TESTING THROUGH EXAMPLE

Flip Number

Result

Probability

1

Heads

0.50

2

Heads

0.25

3

Heads


I have a coin that you believe is fair to start.

To test if this coin is fair, you ask me to flip the coin repeatedly and record the results.



HYPOTHESIS TESTING THROUGH EXAMPLE

Flip Number

Result

Probability

1

Heads

0.50

2

Heads

0.25

3

Heads

0.125

Do you still think the coin is fair?


I have a coin that you believe is fair to start.

To test if this coin is fair, you ask me to flip the coin repeatedly and record the results.



HYPOTHESIS TESTING THROUGH EXAMPLE

Flip Number

Result

Probability

1

Heads

0.50

2

Heads

0.25

3

Heads

0.125

4

Heads

0.0625

Do you still think the coin is fair?


I have a coin that you believe is fair to start.

To test if this coin is fair, you ask me to flip the coin repeatedly and record the results.



HYPOTHESIS TESTING THROUGH EXAMPLE

Flip Number

Result

Probability

1

Heads

0.50

2

Heads

0.25

3

Heads

0.125

4

Heads

0.0625

5

Heads

0.03125

Do you still think the coin is fair?


I have a coin that you believe is fair to start.

To test if this coin is fair, you ask me to flip the coin repeatedly and record the results.






No longer believe the coin is fair.




HYPOTHESIS TESTING THROUGH EXAMPLE

Flip Number

Result

Probability

1

Heads

0.50

2

Heads

0.25

3

Heads

0.125

4

Heads

0.0625

5

Heads

0.03125


I have a coin that you believe is fair to start.

To test if this coin is fair, you ask me to flip the coin repeatedly and record the results.






No longer believe the coin is fair.




HYPOTHESIS TESTING THROUGH EXAMPLE

Flip Number

Result

P-value

1

Heads

0.50

2

Heads

0.25

3

Heads

0.125

4

Heads

0.0625

5

Heads

0.03125

NULL Hypothesis

Test Statistic

Decision on NULL Hypothesis


According to the CLT, sample means follow a Normal distribution as long as the sample size is big enough.


BIKE DATA EXAMPLE WITH MEANS


You believe the average daily number of total users is 4,000, but you want to know if there is more than that. You collect a sample of 731 days with an average daily number of total users at 4,504 with a standard deviation of 1,937.

BIKE DATA EXAMPLE WITH MEANS


You believe the average daily number of total users is 4,000, but you want to know if there is more than that. You collect a sample of 731 days with an average daily number of total users at 4,504 with a standard deviation of 1,937.

What is the probability you see this under the initial thought of 4,000 for an average?

BIKE DATA EXAMPLE WITH MEANS

 

 

 


What is the probability you see this under the initial thought of 4,000 for an average?

BIKE DATA EXAMPLE WITH MEANS

 

 

 


What is the probability you see this under the initial thought of 4,000 for an average?

BIKE DATA EXAMPLE WITH MEANS

 

 

 

 


What is the probability you see this under the initial thought of 4,000 for an average?  < 0.0001

BIKE DATA EXAMPLE WITH MEANS

 

 

 

 


You believe the average daily number of total users is 4,000, but you want to know if there is more than that. 

You collect a sample of 731 days with an average daily number of total users at 4,504 with a standard deviation of 1,937.

What is the probability you see this under the initial thought of 4,000 for an average?  < 0.0001!

Do you still believe your original hypothesis?


BIKE DATA EXAMPLE WITH MEANS


You believe the average daily number of total users is 4,000 (NULL Hypothesis), but you want to know if there is more than that. 

You collect a sample of 731 days with an average daily number of total users at 4,504 with a standard deviation of 1,937.  Test Statistic

What is the probability you see this under the initial thought of 4,000 for an average?  P-value

Do you still believe your original hypothesis?  Decision on NULL Hypothesis


BIKE DATA EXAMPLE WITH MEANS


A hypothesis test uses data to help evaluate an initial claim about a parameter from the population.

There are 4 main steps to hypothesis testing:

State the hypotheses

Test statistic

P-value

Decision on null hypothesis

SUMMARY


NULL AND ALTERNATIVE HYPOTHESIS

TESTING HYPOTHESES WITH DATA


 

HYPOTHESIS TESTING


It is not always obvious how the null and alternative hypotheses should be formulated.

The context of the situation is very important in determining how the hypotheses should be stated.

In some cases it is easier to identify the alternative hypothesis first!

Typically, the alternative is what we are trying to test and want to collect evidence for.


DEVELOPING NULL AND ALTERNATIVE


The null hypothesis is the status quo, or the initial claim about the data.

For example, the average daily number of total users is 4,000. 

 

 


The null hypothesis is the status quo, or the initial claim about the data.

For example, the average daily number of total users is 4,000. 



The null hypothesis is about the population parameter of interest, NOT sample statistics.

Parameters are unknown, while statistics are known.

 

 

 

 


 

 

 


 

 

 


 

NULL VS. ALTERNATIVE

 

 

 

 

 

 

One-Sided Tests

Two-Sided Test


 

SUMMARY


TEST STATISTIC

TESTING HYPOTHESES WITH DATA


The test statistic summarizes the amount of information provided in the sample.

Imagine this like evidence in a court case.

Test statistics have a common form:


TEST STATISTIC

 


The test statistic summarizes the amount of information provided in the sample.

Imagine this like evidence in a court case.

Test statistics have a common form:


TEST STATISTIC

 

Sample Information


The test statistic summarizes the amount of information provided in the sample.

Imagine this like evidence in a court case.

Test statistics have a common form:


TEST STATISTIC

 

Null Hypothesis Information


The test statistic summarizes the amount of information provided in the sample.

Imagine this like evidence in a court case.

Test statistics have a common form:


TEST STATISTIC

 

Estimated Variability from Sampling Distribution of Statistic


The test statistic summarizes the amount of information provided in the sample.

Sample means need the t-distribution because of the unknown values of the population standard deviation.


TEST STATISTIC FOR MEANS

 


The test statistic summarizes the amount of information provided in the sample.

Sample proportions use the Normal distribution.


TEST STATISTIC FOR PROPORTIONS

 


The test statistic summarizes the amount of information provided in the sample.

The test statistic calculation typically requires 3 pieces of information:

Statistic – information obtained from the sample.

Null value – information about the null hypothesis.

Standard error – measure of variability for the sampling distribution of the statistic.

SUMMARY


P-VALUE AND SIGNIFICANCE LEVEL

TESTING HYPOTHESES WITH DATA


Once the test statistic has been determined, we can calculate the probability that we got the information we did from our sample, assuming that the null hypothesis is true.

The p-value is the probability we got our sample, or a sample more extreme, under the null hypothesis.


P-VALUES


If the p-value is low, this implies that the sample we obtained from the population is extremely rare IF we assume that the null hypothesis is true.

This leads us to question the validity of the null hypothesis – rejecting the null hypothesis if the p-value is low enough.

How low is low enough?


SIGNIFICANCE LEVEL VS. P-VALUE


 

SIGNIFICANCE LEVEL VS. P-VALUE


 

SIGNIFICANCE LEVEL VS. P-VALUE


 

SIGNIFICANCE LEVEL VS. P-VALUE


 

 

 


 

 

 

 

P-value

 

Values are “far apart” according to p-value


 

 

 

 

P-value

 

Values are “close together” according to p-value


 

 

 

 

P-value

 

Values are “far apart” according to p-value


 

 

 

 

 

Values are “far apart” according to p-value

P-value/2

 

P-value/2

 


I have a coin that you believe is fair to start.

To test if this coin is fair, you ask me to flip the coin repeatedly and record the results.






No longer believe the coin is fair – but could it be?




HYPOTHESIS TESTING THROUGH EXAMPLE

Flip Number

Result

Probability

1

Heads

0.50

2

Heads

0.25

3

Heads

0.125

4

Heads

0.0625

5

Heads

0.03125


I have a coin that you believe is fair to start.

To test if this coin is fair, you ask me to flip the coin repeatedly and record the results.






No longer believe the coin is fair – but could it be?   YES!




HYPOTHESIS TESTING THROUGH EXAMPLE

Flip Number

Result

Probability

1

Heads

0.50

2

Heads

0.25

3

Heads

0.125

4

Heads

0.0625

5

Heads

0.03125


Defines the unlikely values of the sample statistic if the null hypothesis is true.

This area is typically called the rejection region of the sampling distribution.

Selected before the hypothesis test is even run!

Typical values are 0.01, 0.05, 0.10.


 


The p-value is the probability we got our sample, or a sample more extreme, under the null hypothesis.

If the p-value is low, this implies that the sample we obtained from the population is extremely rare IF we assume that the null hypothesis is true.

The significance level defines the unlikely values of the sample statistic if the null hypothesis is true.



SUMMARY


HYPOTHESIS TEST FOR MEANS

TESTING HYPOTHESES WITH DATA


You believe the average daily number of total users is 4,000, but you want to know if there is more than that so you can decide on orders for future bikes to be added. 

You collect a sample of 731 days with an average daily number of total users at 4,504 with a standard deviation of 1,937.

With a significance level of 0.05, conduct a hypothesis test on this claim.


BIKE DATA EXAMPLE FOR ONE-TAIL HYPOTHESIS TEST


 

BIKE DATA EXAMPLE FOR ONE-TAIL HYPOTHESIS TEST


 

BIKE DATA EXAMPLE FOR ONE-TAIL HYPOTHESIS TEST

 


 

BIKE DATA EXAMPLE FOR ONE-TAIL HYPOTHESIS TEST

 


FINDING P-VALUE

One-Tail

0.25

0.20

0.15

0.10

0.05

0.025

0.01

0.005

0.001

.0005

Two-Tail

0.50

0.40

0.30

0.20

0.10

0.05

0.02

0.01

0.002

0.001

.

.

.

.

.

.

.

.

.

.

.

90

0.677

0.846

1.042

1.291

1.662

1.987

2.368

2.632

3.183

3.402

100

0.677

0.845

1.042

1.290

1.660

1.984

2.364

2.626

3.174

3.390

250

0.675

0.843

1.039

1.285

1.651

1.969

2.341

2.596

3.123

3.330

500

0.675

0.842

1.038

1.283

1.648

1.965

2.334

2.586

3.107

3.310

1000

0.675

0.842

1.037

1.282

1.646

1.962

2.330

2.581

3.098

3.300

.

.

.

.

.

.

.

.

.

.

.


FINDING P-VALUE

One-Tail

0.25

0.20

0.15

0.10

0.05

0.025

0.01

0.005

0.001

.0005

Two-Tail

0.50

0.40

0.30

0.20

0.10

0.05

0.02

0.01

0.002

0.001

.

.

.

.

.

.

.

.

.

.

.

90

0.677

0.846

1.042

1.291

1.662

1.987

2.368

2.632

3.183

3.402

100

0.677

0.845

1.042

1.290

1.660

1.984

2.364

2.626

3.174

3.390

250

0.675

0.843

1.039

1.285

1.651

1.969

2.341

2.596

3.123

3.330

500

0.675

0.842

1.038

1.283

1.648

1.965

2.334

2.586

3.107

3.310

1000

0.675

0.842

1.037

1.282

1.646

1.962

2.330

2.581

3.098

3.300

.

.

.

.

.

.

.

.

.

.

.


FINDING P-VALUE

One-Tail

0.25

0.20

0.15

0.10

0.05

0.025

0.01

0.005

0.001

.0005

Two-Tail

0.50

0.40

0.30

0.20

0.10

0.05

0.02

0.01

0.002

0.001

.

.

.

.

.

.

.

.

.

.

.

90

0.677

0.846

1.042

1.291

1.662

1.987

2.368

2.632

3.183

3.402

100

0.677

0.845

1.042

1.290

1.660

1.984

2.364

2.626

3.174

3.390

250

0.675

0.843

1.039

1.285

1.651

1.969

2.341

2.596

3.123

3.330

500

0.675

0.842

1.038

1.283

1.648

1.965

2.334

2.586

3.107

3.310

1000

0.675

0.842

1.037

1.282

1.646

1.962

2.330

2.581

3.098

3.300

.

.

.

.

.

.

.

.

.

.

.


 

BIKE DATA EXAMPLE FOR ONE-TAIL HYPOTHESIS TEST

 


BIKE DATA EXAMPLE FOR ONE-TAIL HYPOTHESIS TEST

 

 

 

P-value < 0.0005

 


 

BIKE DATA EXAMPLE FOR ONE-TAIL HYPOTHESIS TEST

 


You believe the average daily number of total users is 4,000, but you want to know if there is more than that so you can decide on orders for future bikes to be added OR less than 4,000 so you can pull stock from the streets, so bikes don’t sit unused. 

You collect a sample of 731 days with an average daily number of total users at 4,504 with a standard deviation of 1,937.

With a significance level of 0.05, conduct a hypothesis test on this claim.


BIKE DATA EXAMPLE FOR TWO-TAIL HYPOTHESIS TEST


 

BIKE DATA EXAMPLE FOR ONE-TAIL HYPOTHESIS TEST

 


FINDING P-VALUE

One-Tail

0.25

0.20

0.15

0.10

0.05

0.025

0.01

0.005

0.001

.0005

Two-Tail

0.50

0.40

0.30

0.20

0.10

0.05

0.02

0.01

0.002

0.001

.

.

.

.

.

.

.

.

.

.

.

90

0.677

0.846

1.042

1.291

1.662

1.987

2.368

2.632

3.183

3.402

100

0.677

0.845

1.042

1.290

1.660

1.984

2.364

2.626

3.174

3.390

250

0.675

0.843

1.039

1.285

1.651

1.969

2.341

2.596

3.123

3.330

500

0.675

0.842

1.038

1.283

1.648

1.965

2.334

2.586

3.107

3.310

1000

0.675

0.842

1.037

1.282

1.646

1.962

2.330

2.581

3.098

3.300

.

.

.

.

.

.

.

.

.

.

.


FINDING P-VALUE

One-Tail

0.25

0.20

0.15

0.10

0.05

0.025

0.01

0.005

0.001

.0005

Two-Tail

0.50

0.40

0.30

0.20

0.10

0.05

0.02

0.01

0.002

0.001

.

.

.

.

.

.

.

.

.

.

.

90

0.677

0.846

1.042

1.291

1.662

1.987

2.368

2.632

3.183

3.402

100

0.677

0.845

1.042

1.290

1.660

1.984

2.364

2.626

3.174

3.390

250

0.675

0.843

1.039

1.285

1.651

1.969

2.341

2.596

3.123

3.330

500

0.675

0.842

1.038

1.283

1.648

1.965

2.334

2.586

3.107

3.310

1000

0.675

0.842

1.037

1.282

1.646

1.962

2.330

2.581

3.098

3.300

.

.

.

.

.

.

.

.

.

.

.


FINDING P-VALUE

One-Tail

0.25

0.20

0.15

0.10

0.05

0.025

0.01

0.005

0.001

.0005

Two-Tail

0.50

0.40

0.30

0.20

0.10

0.05

0.02

0.01

0.002

0.001

.

.

.

.

.

.

.

.

.

.

.

90

0.677

0.846

1.042

1.291

1.662

1.987

2.368

2.632

3.183

3.402

100

0.677

0.845

1.042

1.290

1.660

1.984

2.364

2.626

3.174

3.390

250

0.675

0.843

1.039

1.285

1.651

1.969

2.341

2.596

3.123

3.330

500

0.675

0.842

1.038

1.283

1.648

1.965

2.334

2.586

3.107

3.310

1000

0.675

0.842

1.037

1.282

1.646

1.962

2.330

2.581

3.098

3.300

.

.

.

.

.

.

.

.

.

.

.


 

BIKE DATA EXAMPLE FOR ONE-TAIL HYPOTHESIS TEST

 


BIKE DATA EXAMPLE FOR ONE-TAIL HYPOTHESIS TEST

 

P-value/2 < 0.0005

 

P-value/2 < 0.0005

 

 

 

 


 

BIKE DATA EXAMPLE FOR ONE-TAIL HYPOTHESIS TEST

 


ETHICS AROUND INFERENCE WITH DATA

TESTING HYPOTHESES WITH DATA


Hypothesis tests depend on sample data.

Therefore, hypothesis tests may be wrong!

There are two types of errors in hypothesis testing – Type I and Type II errors.


ERRORS IN HYPOTHESIS TESTS


TYPE I VS. TYPE II ERRORS

Correct

Type II 

Type I 

Correct

TRUTH

CHOICE


A Type I error is rejecting the null hypothesis when the null hypothesis was actually true.

In other words, you have a false rejection.

The probability of making a Type I error in a hypothesis test is called the significance level.

Most hypothesis tests are referred to as significance tests because they only control the Type I error.


TYPE I ERROR


A Type II error is accepting the null hypothesis when the null hypothesis was actually false.

In other words, you have falsely accepted.

The probability of NOT making a Type II error in a hypothesis test is called the power.

Difficult to control the Type II error.

Can only control for Type I or Type II at a time.


TYPE II ERROR


What if your sample of data happened to be drawn on data from only summer months with clear days?

Maybe the days would be estimated to have too many users.

This could lead to incorrect actions to be taken.



CAREFUL WITH INFERENCE


What if your sample of data happened to be drawn on data from only summer months with clear days?

Maybe the days would be estimated to have too many users.

This could lead to incorrect actions to be taken.


Hypothesis tests completely depend on the data they are built from.

Garbage in 🡪 Garbage out




CAREFUL WITH INFERENCE


What if your sample of data happened to be drawn on data from only summer months with clear days?

Maybe the days would be estimated to have too many users.

This could lead to incorrect actions to be taken.


Hypothesis tests completely depend on the data they are built from.

Garbage in 🡪 Garbage out


Hypothesis tests results reveal something, but not everything!


CAREFUL WITH INFERENCE


Hypothesis tests results reveal something, but not everything!

People sometimes forget the possibility of errors when making claims from a statistical test. 

For example:

“We know that more than 4,000 bikes per day are used on average.”



CAREFUL ABOUT JUSTIFICATION


Hypothesis tests results reveal something, but not everything!

People sometimes forget the possibility of errors when making claims from a statistical test. 

For example:

“We have strong evidence that more than 4,000 bikes per day are used on average.”



CAREFUL ABOUT JUSTIFICATION


Hypothesis tests results reveal something, but not everything!

People sometimes forget the possibility of errors when making claims from a statistical test. 

For example:

“We have strong evidence that more than 4,000 bikes per day are used on average.”


Remember the analogy of a court case 🡪 we incorrectly claim people are guilty sometimes. Careful about rushing to judgement!



CAREFUL ABOUT JUSTIFICATION


A Type I error is rejecting the null hypothesis when the null hypothesis was actually true.

A Type II error is accepting the null hypothesis when the null hypothesis was actually false.

Hypothesis tests completely depend on the data they are built from.

People sometimes forget the possibility of errors when making claims from a statistical test. 


SUMMARY


Остання зміна: понеділок 17 жовтня 2022 13:26 PM