NLP Advanced Topics & Theory 4 — Questions and Answers
Question 1: What is the key idea behind Reinforcement Learning from Human Feedback (RLHF) in LLM alignment?
- Fine-tuning the model on synthetic data generated by a rule-based reward function
- Training a reward model on human preferences, then using RL to optimize the LLM against that reward (Correct answer)
- Replacing next-token prediction with binary yes/no feedback signals
- Using RLHF only during the pre-training phase to reduce harmful outputs
Correct answer: Training a reward model on human preferences, then using RL to optimize the LLM against that reward
RLHF trains a reward model on human comparisons of outputs, then uses PPO or similar RL algorithms to steer the LLM toward higher-reward responses.
Question 2: What does the term 'hallucination' mean in the context of large language models?
- The model generating extremely long outputs beyond its context window
- The model producing fluent but factually incorrect or fabricated content (Correct answer)
- Attention heads attending to tokens outside the input sequence
- Embedding dimensions collapsing to near-zero values during training
Correct answer: The model producing fluent but factually incorrect or fabricated content
Hallucination refers to confident, fluent model outputs that are factually wrong or entirely made up, a central reliability challenge for LLMs.
Question 3: Which NLP task involves identifying the opinion expressed about specific aspects of a product or service?
- Document classification
- Aspect-based sentiment analysis (ABSA) (Correct answer)
- Slot filling
- Relation extraction
Correct answer: Aspect-based sentiment analysis (ABSA)
ABSA extracts sentiment polarity at a fine-grained level for specific attributes (e.g., 'The battery life is great but the screen is dim').
Question 4: What is 'prompt injection' in the context of LLM security?
- Inserting malicious SQL into prompts to corrupt the model's database
- Crafting input that overrides system instructions to make the model perform unintended actions (Correct answer)
- Injecting random noise into embeddings to test model robustness
- Using special tokens to force the model into a specific decoding mode
Correct answer: Crafting input that overrides system instructions to make the model perform unintended actions
Prompt injection attacks embed adversarial instructions in user input that hijack the model's behavior by overriding or confusing its system prompt.
Question 5: What does 'sparse attention' aim to achieve compared to full (dense) self-attention?
- Reduce vocabulary size by attending only to common tokens
- Reduce quadratic complexity by restricting each token to attend to a subset of positions (Correct answer)
- Increase model depth without increasing parameter count
- Replace softmax with a sparse activation function in feed-forward layers
Correct answer: Reduce quadratic complexity by restricting each token to attend to a subset of positions
Sparse attention (e.g., Longformer, BigBird) limits each token to attending to local windows or global tokens, reducing O(n²) cost for long sequences.
Question 6: In dependency parsing, what does a 'head' word determine?
- The root of a constituency parse subtree
- The syntactically dominant word to which a dependent word is attached (Correct answer)
- The first token of a named entity span
- The token with the highest TF-IDF weight in a sentence
Correct answer: The syntactically dominant word to which a dependent word is attached
In dependency grammar, each dependent word has exactly one head, and the arc between them encodes a grammatical relation such as subject or object.
Question 7: What problem does byte-pair encoding (BPE) solve in NLP tokenization?
- It eliminates the need for lowercasing and punctuation removal
- It balances vocabulary size and out-of-vocabulary coverage by merging frequent character pairs into subwords (Correct answer)
- It removes stop words before feeding text to a neural network
- It maps every word to a fixed 300-dimensional vector
Correct answer: It balances vocabulary size and out-of-vocabulary coverage by merging frequent character pairs into subwords
BPE iteratively merges the most frequent adjacent byte or character pairs to build a subword vocabulary, handling rare and unseen words gracefully.
NLP Advanced Topics & Theory 4
What is the key idea behind Reinforcement Learning from Human Feedback (RLHF) in LLM alignment?