AI Ultimate AI Engineer 5 — Questions and Answers
Question 1: What is 'prompt injection' in the context of deployed LLM applications?
- Adding few-shot examples to improve response quality
- Malicious input that overrides system instructions to hijack model behavior (Correct answer)
- Compressing long prompts to fit within context limits
- A technique to inject domain knowledge without fine-tuning
Correct answer: Malicious input that overrides system instructions to hijack model behavior
Prompt injection attacks embed adversarial instructions in user input that override the developer's system prompt, causing the model to follow attacker directives.
Question 2: Which evaluation framework is specifically designed to assess LLM applications end-to-end using LLM-as-a-judge scoring?
- Pytest with mock fixtures
- RAGAS for RAG pipelines or LangSmith evals (Correct answer)
- Apache JMeter for load testing
- Weights & Biases sweeps for hyperparameter search
Correct answer: RAGAS for RAG pipelines or LangSmith evals
RAGAS and LangSmith provide LLM-as-judge metrics like faithfulness, answer relevancy, and context precision tailored to generative AI pipelines.
Question 3: In the context of AI engineering, what does 'guardrails' refer to?
- Hardware barriers protecting GPUs during data center installation
- Input/output validation layers that enforce safety and policy constraints around LLM calls (Correct answer)
- Rate limiting rules enforced by the model provider's API
- Gradient clipping applied during backpropagation
Correct answer: Input/output validation layers that enforce safety and policy constraints around LLM calls
Guardrails are programmatic checks (classifiers, rules, or another LLM) applied to inputs and outputs to detect and block policy violations.
Question 4: What is the primary benefit of using structured output (e.g., JSON mode) when calling an LLM in a production pipeline?
- It reduces the number of tokens generated significantly
- It guarantees the model's output conforms to a parseable schema, removing brittle regex parsing (Correct answer)
- It increases the model's factual accuracy on domain questions
- It bypasses rate limits imposed by the API provider
Correct answer: It guarantees the model's output conforms to a parseable schema, removing brittle regex parsing
Structured output constrains the model's decoding to valid JSON (or another schema), making downstream parsing reliable and eliminating format-related failures.
Question 5: Which strategy best handles a user query that exceeds an LLM's context window in a document QA system?
- Truncate the document from the beginning to fit
- Use hierarchical summarization or a retrieval step to select only relevant chunks (Correct answer)
- Switch to a model with lower resolution attention
- Increase the model's temperature to compress responses
Correct answer: Use hierarchical summarization or a retrieval step to select only relevant chunks
Hierarchical summarization or retrieval-based selection ensures the most relevant content fits in the window without losing critical information through naive truncation.
Question 6: When deploying an AI model with Kubernetes, what resource should be explicitly requested in the pod spec to enable GPU scheduling?
- cpu: '16000m' with memory limit
- nvidia.com/gpu resource limit in the container spec (Correct answer)
- A ConfigMap with CUDA environment variables
- A PersistentVolumeClaim for model weight storage
Correct answer: nvidia.com/gpu resource limit in the container spec
Kubernetes requires explicit resource requests like `nvidia.com/gpu: 1` in the container spec so the scheduler assigns GPU-equipped nodes and the NVIDIA device plugin allocates the device.
Question 7: What distinguishes a 'zero-shot' evaluation from a 'few-shot' evaluation when benchmarking LLMs?
- Zero-shot uses no system prompt; few-shot includes a system prompt
- Zero-shot provides no examples in the prompt; few-shot includes demonstration examples (Correct answer)
- Zero-shot evaluates on unseen model architectures; few-shot reuses trained checkpoints
- Zero-shot measures token throughput; few-shot measures output quality
Correct answer: Zero-shot provides no examples in the prompt; few-shot includes demonstration examples
In zero-shot evaluation the model receives only the task instruction with no examples, while few-shot evaluation prepends k demonstration input-output pairs to the prompt.
AI Ultimate AI Engineer 5
What is 'prompt injection' in the context of deployed LLM applications?