Algorithms Research & Evidence-Based Practice 4

Question 1

What is 'p-hacking' in algorithm research and why is it problematic?

Accepted Answer

Running many statistical tests and only reporting those that show significance, inflating false-positive rates

Answer

P-hacking exploits the fact that running many tests increases the chance of a false positive, producing misleading claims of significance.

Question 2

In cache-oblivious algorithm research, what experimental evidence is typically gathered to validate theoretical predictions?

Accepted Answer

Cache miss rates measured using profiling tools on inputs of varying sizes

Answer

Cache miss rates directly test whether the algorithm achieves the predicted low cache-miss behavior as problem size grows relative to cache size.

Question 3

What does 'effect size' measure in an algorithm performance comparison study?

Accepted Answer

The practical magnitude of the performance difference, independent of sample size

Answer

Effect size quantifies how large the performance difference is in practical terms, preventing a statistically significant but negligibly small difference from being over-interpreted.

Question 4

Why do some algorithm researchers prefer to report median runtime instead of mean runtime?

Accepted Answer

Median is robust to outliers caused by OS scheduling or garbage collection pauses

Answer

Rare but extreme outliers (e.g., OS interrupts, GC pauses) skew the mean upward, while the median reflects typical performance more accurately.

Question 5

What is the 'streetlight effect' pitfall in algorithm benchmarking?

Accepted Answer

Only evaluating inputs or scenarios that are convenient, missing hard or realistic cases

Answer

The streetlight effect refers to evaluating only easy or convenient inputs rather than the diverse and potentially harder inputs the algorithm will face in practice.

Algorithms Practice Test

Algorithms Practice Test

Algorithms Research & Evidence-Based Practice 4