Appendix 4—Example Set of "Laboratory" Problems ©2003, D.F. Parkhurst
E538—Lab 1 on Significance Testing—2001—D. Parkhurst
As you have seen, I believe that significance testing is seldom a useful tool for analyzing data, and is often downright misleading. Even so, significance tests are so widely used that anyone who either produces or uses scientific results should understand the "logic" that underlies them. Each of you may at times be required to perform these tests, and will certainly need at times to interpret them. These exercises are designed to give you practice in those tasks. (I do not claim that the tests here provide useful information for the decisions to which they apply, but then, such irrelevance seems common.)
For each of the exercises, you will learn about testing most effectively if you identify for yourself the null and alternative hypotheses, the test statistic, and the sampling distribution (reference distribution) that would be expected for the test statistic if the null hypothesis were true.
For at least some of these exercises,
it may save you time if you use the fact that for a random variable X and a particular value 
, 
.
1. At some water supply reservoirs, the US EPA imposes an upper limit of 20 CFU[1] (100 mL)-1 on the allowable concentration of fecal coliform bacteria (FCB) permitted to be drawn from the reservoir. This limit may be exceeded in no more than 10% of samples taken during any six-month period.
Suppose you work for a water supply that has a secondary reservoir ("Upper Lake") that can feed water into the primary reservoir ("Main Lake") at which the above regulation is in force. During a minor drought, your supervisor has to decide whether to release some water from Upper Lake into Main Lake. FCB concentrations have been checked biweekly for the last six weeks at Upper, and were above 20 in three of the twelve samples.
Your supervisor asks you to perform a NHST (null-hypothesis significance test), based on the hypotheses
versus 
,
where 
represents the true
proportion of samples from Upper Lake that would have FCB concentrations
greater than 20 CFU (100 mL)-1. Your tasks for this exercise are to:
A. Perform this hypothesis test. Show and describe your work. (Hints: the significance test for the tea-tasting lady serves as an example for the test to be performed here, and you may find the CDF.Binom function in SPSS useful.)
B. State explicitly whether or
not you would reject this null hypothesis at the 
level. Explain.
C. Describe briefly how the results from A and B would influence a recommendation that you might give your supervisor on whether to import Upper Lake water into Main Lake or not.
D. Suppose your supervisor had not specified the particular test above, but instead had simply asked you to “analyze the data and recommend whether or not to import the Upper Lake water.” Would you then (i) Perform this same significance test?, (ii) Perform some different analysis instead (specify)?, or both? Explain briefly.
E. If the test were
actually performed in the real situation, would it make more sense to use an 
value different from
5%? What would be the effects of
choosing a larger or smaller a value?
2. As manager of an air
monitoring program, you have to purchase the specialized, expensive filters for
your agency’s air samplers. Recently
you’ve found many defective filters, so you consider changing your supplier. The FiltoTech company claims to have only a
1% defect rate in these hard-to-manufacture filters, so you give them a try and
purchase an initial pack of 100 filters.
You plan to count the number of defective filters, and to test (at 
)
versus 
,
where 
is the true mean
number of defects per 100 pack that the company may supply. Technically, the counted number k is a binomial random variable here (with
a maximum of 100), but when a typical count is a small fraction of the maximum
number possible, the Poisson distribution provides an accurate and useful
approximation to the binomial. The
Poisson is often easier to work with mathematically than the binomial in
situations like these, and I recommend its use for this problem. You could calculate the necessary
probabilities from the formula given in the notes, or you could use the
CDF.Poisson function from SPSS. Here are your tasks:
A. Perform the suggested test, if the first 100-pack you receive contains three defective filters.
B. Explain whether or not H0 is rejected.
C. Explain how your result would influence whether or not you would continue to purchase filters from this new company.
D. Does significance testing actually provide helpful information in this situation?
E. Would some value of 
other than 5% be more
practical to use in this application?
3. In anticipation of our
consideration of reverse tests,
consider the tea-tasting lady again.
Suppose when we present her with ten randomly prepared cups, she
discriminates exactly five of them correctly.
Although some might think that this demonstrates that she is a pure
guesser (with 
) and has no discriminatory ability beyond that, that is
logically untrue, since she might be 80% right in the long run, but only
identify five of the first ten cups correctly.
If significance testing is
considered to be useful for deciding whether her
is greater than
0.5 (our original test), then it ought to be equally useful (or equally
useless, perhaps) for deciding whether her
were less than
any value that we might consider importantly greater than 0.5. Suppose, for example, that we would be
impressed if she really could distinguish 70% of cups given to her over the
long run. That would mean that for her,
. Let’s use her
result of 
correct out of ten
trials to test
versus 
.
(Note the “<” in 
.) Perform that test
(as usual with 
, for no particular reason), and write out your
interpretation of the result. You may
find the table of likelihoods that I provided you to be useful for this
test. You might also want to use the
SPSS CDF.Binom function to obtain necessary probabilities.
4. Although the EPA has not yet set binding upper limits on allowable concentrations of Giardia cysts in drinking water, suppose they propose to set such a limit at 1 cyst per 10 liters, i.e., at 0.1 cyst L-1. During a public comment period, two (imaginary) organizations -- the Association of Immuno-Suppressed Individuals (AISI), and the American Water Supply Association (AWSA) ¾ submit suggestions to use significance testing in place of the simple limit proposed by the EPA. Note that the AISI represents people who are particularly sensitive to Giardia and related diseases, while the AWSA represents firms (and municipal water supplies) whose water-treatment costs could rise if the new limit is put in place.
I. In particular, one of these groups suggests requiring five
separate ten-liter samples, counting the total number of cysts in those
combined fifty liters, and then testing H0:
versus H1:
(with 
being the true mean
number of cysts per 50 L of water).
Then, they suggest, a water supply would be in violation if that null
hypothesis were rejected in favor of the stated alternative. They ask for
to be set at 10%.
II. The other group also suggests requiring five ten-liter samples
and counting the total number of cysts in those fifty liters, but they suggest
testing H0:
versus H1:
(note the reversed direction of the < sign). This group suggests that a water supply
would be in violation unless the null hypothesis were rejected in favor of this
stated alternative. This group also asks
for
to be set at 10%.
Your tasks are to:
A. Identify which group would benefit from proposal I, and which would benefit from proposal II.
B. Noting that the null hypothesis is the same for both proposals, calculate the probability distribution for various numbers of cysts that might be found in fifty liters of water, if that null hypothesis were true. (You may limit your search to whatever range allows you to answer Parts C and D.)
C. Determine for what number or numbers of cysts (in fifty liters) the null hypothesis in Suggestion I would be rejected.
D. Repeat that determination for Suggestion II.