MAC MAC A/B Testing & Experimentation 2

Question 1

What distinguishes a multivariate test (MVT) from a standard A/B test?

Accepted Answer

MVT tests multiple elements and their combinations simultaneously; A/B tests one variable at a time

Answer

Multivariate testing simultaneously tests multiple page elements and their combinations to identify which combination produces the best outcome.

Question 2

In A/B testing, what is a 'false positive' (Type I error)?

Accepted Answer

Concluding a variant is better when the difference is actually due to chance

Answer

A false positive (Type I error) occurs when you incorrectly conclude that a variant outperforms the control when the observed difference is actually random.

Question 3

What is 'traffic splitting' in the context of an A/B test?

Accepted Answer

Randomly assigning users to control or variation groups to ensure unbiased comparison

Answer

Traffic splitting randomly divides incoming users between test variants so that each group is representative and results can be fairly compared.

Question 4

Why should an A/B test ideally run for at least one full business cycle (e.g., one week)?

Accepted Answer

To account for day-of-week traffic and behavior patterns that could skew results

Answer

Running a test for at least a full week captures daily behavioral differences—such as weekday vs. weekend patterns—that could otherwise distort results.

Question 5

What is 'interaction effect' in multivariate testing?

Accepted Answer

The combined impact of two or more simultaneously changed elements that differs from their individual effects

Answer

An interaction effect occurs when the combined impact of changing two elements together is different from what would be predicted by looking at each element's effect individually.

(MAC) Marketing Analytics Certification Practice Test

(MAC) Marketing Analytics Certification Practice Test

MAC MAC A/B Testing & Experimentation 2