Machine Learning Natural Language Processing 3 — Questions and Answers
Question 1: Which pre-training objective does BERT use to learn bidirectional representations?
- Next sentence prediction only
- Causal language modeling
- Masked language modeling and next sentence prediction (Correct answer)
- Contrastive sentence loss
Correct answer: Masked language modeling and next sentence prediction
BERT is pre-trained with Masked Language Modeling (predicting masked tokens) and Next Sentence Prediction (classifying sentence pairs) to learn deep bidirectional context.
Question 2: What differentiates an autoregressive language model from a masked language model?
- Autoregressive models predict tokens left-to-right using only past context (Correct answer)
- Autoregressive models use bidirectional attention
- Masked models predict tokens left-to-right
- Autoregressive models require larger batch sizes
Correct answer: Autoregressive models predict tokens left-to-right using only past context
Autoregressive models like GPT generate tokens sequentially, conditioning each prediction on all previous tokens, while masked models like BERT see full bidirectional context.
Question 3: In dependency parsing, what does the 'head' of a dependency arc represent?
- The first token in the sentence
- The governing word that another word depends on (Correct answer)
- The root of the entire parse tree
- The subject of the main clause
Correct answer: The governing word that another word depends on
In a dependency parse, the head is the word that syntactically governs a dependent word, capturing grammatical relations like subject or object.
Question 4: Which technique is used to reduce the dimensionality of word co-occurrence matrices in traditional distributional semantics?
- Byte-Pair Encoding
- Singular Value Decomposition (SVD) (Correct answer)
- Beam search
- Viterbi algorithm
Correct answer: Singular Value Decomposition (SVD)
SVD factorizes the large sparse co-occurrence matrix into lower-dimensional dense word vectors, forming the basis of methods like LSA.
Question 5: What problem does beam search solve in sequence generation?
- It prevents overfitting during training
- It approximates the most probable output sequence by maintaining top-k candidates (Correct answer)
- It speeds up tokenization
- It handles padding in batched inputs
Correct answer: It approximates the most probable output sequence by maintaining top-k candidates
Beam search keeps the top-k (beam width) partial sequences at each step, balancing quality and computational cost versus greedy decoding.
Question 6: In NLP, what does a 'pipeline' approach mean compared to an 'end-to-end' approach?
- Pipeline models train all tasks simultaneously with one loss
- Pipeline models chain separate specialized components for each subtask (Correct answer)
- End-to-end models use more layers than pipeline models
- Pipeline models require pre-trained embeddings
Correct answer: Pipeline models chain separate specialized components for each subtask
A pipeline chains discrete components (tokenizer → POS tagger → parser) where each feeds into the next, while end-to-end models jointly learn all transformations.
Question 7: Which algorithm is classically used for sequence labeling tasks like POS tagging by finding the most likely hidden state sequence given observations?
- Beam search
- Viterbi algorithm (Correct answer)
- Expectation-Maximization
- CYK parsing
Correct answer: Viterbi algorithm
The Viterbi algorithm efficiently finds the most probable hidden state sequence in a Hidden Markov Model (HMM) using dynamic programming.
Machine Learning Natural Language Processing 3
Which pre-training objective does BERT use to learn bidirectional representations?