To check whether the conclusion of Analysis 2 depends on the assumptions of the t-test, an analyst runs a permutation test with 5000 permutations. The effect is the mean of time_A minus the mean of time_B. The analyst's code and output are shown below.

obs_diff <- times_long |>
  group_by(layout) |>
  summarise(mean_time = mean(time_s)) |>
  summarise(diff = mean_time[layout == "A"] - mean_time[layout == "B"]) |>
  pull(diff)
perm_diffs <- map_dbl(1:5000, \(i) {
  times_long |>
    mutate(layout_perm = sample(layout)) |>
    group_by(layout_perm) |>
    summarise(mean_time = mean(time_s)) |>
    summarise(diff = mean_time[layout_perm == "A"] - mean_time[layout_perm == "B"]) |>
    pull(diff)
})
(sum(abs(perm_diffs) >= abs(obs_diff)) + 1) / (5000 + 1)

> obs_diff
[1] 6.86
> sd(perm_diffs)
[1] 4.19
> (sum(abs(perm_diffs) >= abs(obs_diff)) + 1) / (5000 + 1)
[1] 0.10

The analyst concludes: “The permutation p-value is 0.10, so there is no clear evidence that the layouts differ. This disagrees with Analysis 2 (p = 0.0009), so the paired t-test must be unreliable.”

Identify the error in the analyst's approach, explain how it produces the output shown in Output 2.7, and describe how to correct it. Then comment on the analyst's conclusion.

Use Panel A of Output 2.6 to describe how the participants' times differ between the layouts. Then explain what Panel B shows and how you would report the result in words for the company's managers, including the size of the effect and its uncertainty.

Analyses 1 and 2 give the same estimated difference (6.86 seconds), but Analysis 1 has a much larger p-value and a wider interval. What explains this?

To build a bootstrap distribution for the mean log ratio, what should be resampled with replacement?

Output 2.5 gives a 95% percentile bootstrap interval of (0.078, 0.193) for the mean log ratio. Taking exponentials gives (1.08, 1.21). Which interpretation is best?

The company wants an estimate of the average difference in completion time, in seconds, between the layouts for the same users. Which analysis output should be used?

An analyst wants the proportion of surge days within each sensor flag group, so that the proportions in each group add to 1. The analyst runs the code below and reports: “Only 15% of flagged days are surge days, compared with 10% of Normal days, so the flag barely matters.” Explain what has gone wrong in the code and describe how it should be changed. Then comment on the analyst's conclusion.

days |>
  count(sensor_flag, surge) |>
  mutate(prop = n / sum(n))

# A tibble: 4 × 4
  sensor_flag surge     n  prop
  <chr>       <chr> <int> <dbl>
1 High        No       36  0.15
2 High        Yes      36  0.15
3 Normal      No      144   0.6
4 Normal      Yes      24   0.1

Which statement about the sensor's flags is supported by Output 1.3?

Output 1.2 reports a correlation of 0.975 between the sensor and reference readings. What can the council conclude from this alone?

An analyst writes:

“From Output 1.1, P(Surge | High) = 36/72 = 0.50, so the sensor network correctly identifies 50% of surge days. In other words it misses half of all surges. Because 0.50 is much bigger than the overall proportion of surge days, 0.25, flagged High days cause asthma surges.”

Evaluate, explain and critique this conclusion. For parts you agree with, explain why they are true. For parts you disagree with, explain how to correct them.