PCA Monitoring, Alerting & Troubleshooting 2 — Questions and Answers
Question 1: An Alertmanager route has `group_wait: 30s` and `group_interval: 5m`. A second alert fires in the same group 10 seconds after the first. When will the second alert be sent?
- Immediately, because it is a new alert
- Together with the first alert after the 30s group_wait expires (Correct answer)
- After its own 30s group_wait starts independently
- After the 5m group_interval elapses from the first notification
Correct answer: Together with the first alert after the 30s group_wait expires
Alerts in the same group are batched together; the second alert joins the existing group and is sent when the group_wait of 30s expires.
Question 2: Which Prometheus metric type is most appropriate for tracking the total number of HTTP requests served since process start?
- Gauge
- Histogram
- Counter (Correct answer)
- Summary
Correct answer: Counter
Counters are monotonically increasing values ideal for cumulative totals like request counts.
Question 3: A Prometheus alert has been in the `PENDING` state for longer than the configured `for` duration but has not transitioned to `FIRING`. What is the most likely cause?
- The alerting rule file has a syntax error
- The alert condition became false before the for-duration elapsed (Correct answer)
- Alertmanager is not reachable
- The rule evaluation interval is longer than the for-duration
Correct answer: The alert condition became false before the for-duration elapsed
If the alert condition resolves before the for-duration completes, the alert resets to inactive without ever firing.
Question 4: What does the `absent()` function return when the time series passed to it exists and has a value?
- 1
- 0
- An empty vector (Correct answer)
- The current value of the series
Correct answer: An empty vector
absent() returns an empty vector when the time series is present, and a vector with value 1 only when the series is absent.
Question 5: In Prometheus, what is the effect of setting `honor_labels: true` on a scrape configuration?
- Prometheus ignores all labels from the target
- Conflicting labels from the target override Prometheus-assigned labels (Correct answer)
- Prometheus-assigned labels always override target labels
- Labels are deduplicated by keeping only unique values
Correct answer: Conflicting labels from the target override Prometheus-assigned labels
With honor_labels: true, if the target exposes a label that conflicts with a Prometheus-set label, the target's label wins.
Question 6: You notice Prometheus is dropping samples with the error 'out of order'. What is the most likely root cause?
- Two scrape jobs are targeting the same endpoint
- Samples are arriving with timestamps older than the current TSDB head block (Correct answer)
- The retention period has been exceeded
- The target is returning duplicate label sets
Correct answer: Samples are arriving with timestamps older than the current TSDB head block
Prometheus TSDB requires samples to arrive in order; samples with timestamps behind the latest ingested timestamp for a series are rejected as out of order.
Question 7: Which Alertmanager feature allows sending a notification only once for a group of related alerts, rather than once per alert?
- Silences
- Inhibition rules
- Grouping (Correct answer)
- Receivers
Correct answer: Grouping
Grouping batches related alerts (e.g., same cluster or job) into a single notification to reduce alert fatigue.
PCA Monitoring, Alerting & Troubleshooting 2
An Alertmanager route has `group_wait: 30s` and `group_interval: 5m`.
A second alert fires in the same group 10 seconds after the first.
When will the second alert be sent?