SecAI+ Secure AI Development and MLSecOps 2 — Questions and Answers
Question 1: What is 'model stealing' in the context of AI security?
- Physically stealing hardware containing proprietary AI models
- Reconstructing a proprietary model by querying its API and training on the responses (Correct answer)
- Extracting raw training data from a model's parameters
- Copying model weights from public repositories without attribution
Correct answer: Reconstructing a proprietary model by querying its API and training on the responses
Model stealing involves querying a target model's API to collect input-output pairs, then training a surrogate model that functionally approximates the original without access to its weights or training data.
Question 2: Which security principle governs limiting model and service access to only what is necessary for each role?
- Defense in depth
- Least privilege (Correct answer)
- Zero trust network access
- Fail-safe defaults
Correct answer: Least privilege
The least privilege principle ensures models, users, and system components hold only the minimum access necessary, minimizing the blast radius if any component is compromised.
Question 3: What is the primary supply chain security concern when using open-source pre-trained models?
- They are always less accurate than proprietary alternatives
- They may contain backdoors, malicious code, or poisoned weights from the source (Correct answer)
- They cannot be fine-tuned for domain-specific security use cases
- They require significantly more computational resources than custom models
Correct answer: They may contain backdoors, malicious code, or poisoned weights from the source
Open-source pre-trained models may have been intentionally tampered with to include backdoors or poisoned behaviors, representing a supply chain risk analogous to malicious packages in traditional software.
Question 4: Which technique is most effective for detecting unauthorized modifications to a deployed ML model?
- Retraining the model with fresh data on a regular schedule
- Model integrity monitoring using cryptographic checksums of model artifacts (Correct answer)
- Increasing the model's training dataset size after deployment
- Using ensemble methods to average predictions across model versions
Correct answer: Model integrity monitoring using cryptographic checksums of model artifacts
Cryptographic checksums allow security teams to detect any unauthorized modification to model weights or configuration after deployment, similar to file integrity monitoring in traditional security.
Question 5: What is 'shadow mode' deployment in ML operations and why is it used?
- Deploying models on dark-web infrastructure for anonymous testing
- Running a new model in parallel with production without affecting live outputs (Correct answer)
- Training models exclusively on encrypted datasets for privacy
- Deploying models behind a VPN to restrict access during validation
Correct answer: Running a new model in parallel with production without affecting live outputs
Shadow mode runs a candidate model alongside the production model, processing identical inputs but not serving its outputs to users, enabling safe validation of security and accuracy before full rollout.
Question 6: Which logging practice is most critical for detecting adversarial attacks on deployed ML models?
- Logging only error messages to minimize storage consumption
- Capturing input features, predictions, and confidence scores for audit and anomaly detection (Correct answer)
- Storing all raw training data alongside every prediction log entry
- Logging model weight snapshots after each individual prediction
Correct answer: Capturing input features, predictions, and confidence scores for audit and anomaly detection
Logging input features, predictions, and confidence scores creates the data needed for anomaly detection, forensic analysis, and identification of adversarial inputs or unexpected model drift.
Question 7: What security risk does 'concept drift' introduce in deployed security ML models?
- The model becomes too accurate, causing false negatives in threat detection
- Performance degrades as distributions shift, potentially leading to incorrect security decisions (Correct answer)
- The model begins consuming disproportionate computational resources in production
- The model starts rejecting all legitimate inputs as anomalies uniformly
Correct answer: Performance degrades as distributions shift, potentially leading to incorrect security decisions
Concept drift degrades model performance as real-world data distributions diverge from training data, which in security contexts can cause dangerous misclassifications such as missed threats or excessive false positives.
SecAI+ Secure AI Development and MLSecOps 2
What is 'model stealing' in the context of AI security?