NLP Text Preprocessing 4

Question 1

What preprocessing challenge arises specifically when handling Chinese or Japanese text that doesn't exist with English?

Accepted Answer

Word boundary segmentation since words are not space-delimited

Answer

Chinese and Japanese text lacks spaces between words, so word segmentation algorithms are required before any word-level processing.

Question 2

What is the effect of applying log normalization to term frequency (TF) values?

Accepted Answer

It reduces the impact of very high frequency terms by compressing the scale

Answer

Log normalization (e.g., 1 + log(tf)) prevents very frequent terms from dominating by compressing large frequency differences.

Question 3

In preprocessing a legal or medical corpus, why might domain-specific stop words be added beyond the standard list?

Accepted Answer

Common domain terms like 'whereas' or 'patient' appear frequently but carry little discriminative value

Answer

High-frequency domain terms that appear in nearly every document (like 'patient' in medical text) behave like stop words and can be added to a custom list.

Question 4

What is a 'character n-gram' and why is it useful in text preprocessing?

Accepted Answer

A sequence of n consecutive characters, useful for handling misspellings and morphology

Answer

Character n-grams capture subword patterns, making models robust to typos, morphological variants, and out-of-vocabulary words.

Question 5

Which text normalization step would convert '5 kilometers' and '5 km' to a consistent representation?

Accepted Answer

Unit normalization or entity normalization

Answer

Unit normalization maps different representations of the same measurement (km, kilometers, kilometre) to a canonical form.

NLP Practice Test

NLP Practice Test

NLP Text Preprocessing 4