Practice Test Geeks(MAC) Marketing Analytics Certification Practice Test
← All MAC Flashcard Decks

MAC A/B Testing & Experimentation Flashcards

6 cards from real MAC practice questions. Tap to flip, then mark Knew It or Still Learning — missed cards come back until you master them.

Read the first 6 MAC A/B Testing & Experimentation flashcards as text

  1. In a marketing A/B test, what does the 'control' represent?

    Answer: The original version against which variations are compared

    The control is the original, unchanged version of a page or element that serves as the baseline for comparing the performance of new variations.

  2. What does statistical significance indicate in the context of an A/B test?

    Answer: The observed difference between variants is unlikely to be due to chance

    Statistical significance indicates there is a sufficiently low probability that the measured difference between variants occurred by random chance.

  3. Why is it important to define a primary metric before launching an A/B test?

    Answer: To prevent 'p-hacking' by changing the success metric after seeing results

    Pre-defining the primary metric prevents cherry-picking results after the test, which inflates the false positive rate and leads to unreliable conclusions.

  4. What is the minimum recommended statistical confidence level commonly used for marketing A/B tests?

    Answer: 95%

    A 95% confidence level (p-value ≤ 0.05) is the widely accepted standard for declaring a marketing test result statistically significant.

  5. What is 'sample size' in the context of A/B testing, and why does it matter?

    Answer: The number of users in each group; too few users leads to unreliable results

    Sample size refers to the number of users in each test group; insufficient sample sizes produce unreliable results because small differences can appear significant by chance.

  6. What is 'novelty effect' and how can it distort A/B test results?

    Answer: Users clicking on a new variant simply because it is different, inflating its early performance

    Novelty effect occurs when users engage more with a new variant simply because it is unfamiliar, causing temporary inflated performance that fades over time.