AI & ML5 min read

NLP & Text Processing

Process and classify text with Python NLP tools - tokenization, named entity recognition, text classification with transformers, and text similarity.

NLPspaCyNLTKText ClassificationTransformers

skill: NLP & Text Processing category: AI & ML tags: [NLP, spaCy, NLTK, Text Classification, Transformers] readingTime: 5

The pre-transformer NLP toolkit (TF-IDF, bag-of-words, word2vec) is still valuable for resource-constrained environments. For most production NLP tasks, Hugging Face transformers with a pretrained BERT or RoBERTa model achieve better results with less custom code - at the cost of compute.

Workflow

Preprocess text with spaCyRaw text needs cleaning, tokenization, and linguistic normalization before ML models can use it. spaCy provides industrial-strength NLP in a fast, Pythonic API.
Classify text with TF-IDF baselineTF-IDF + logistic regression is the fastest text classification baseline. It trains in seconds, is interpretable, and beats transformers on small datasets.
Fine-tune BERT for classificationBERT-based models achieve state-of-the-art on most NLP tasks with minimal task-specific architecture. Hugging Face Transformers makes fine-tuning 20 lines of code.
Generate text embeddings for searchSentence transformers produce fixed-size embeddings where similar texts have high cosine similarity - enabling semantic search, clustering, and deduplication.
Extract structured information with NER and regexNamed entity recognition extracts people, organizations, locations, and dates from text. Combining spaCy NER with regex patterns covers most information extraction needs.

Output Format

NLP: spaCy doc.ents for NER, TF-IDF + LogReg baseline, DistilBERT fine-tuning via Hugging Face Trainer, sentence-transformers + FAISS for semantic search, and Matcher for custom entity patterns.

nlp = spacy.load("en_core_web_sm"); doc = nlp(text); doc.ents
Pipeline([("tfidf", TfidfVectorizer(ngram_range=(1,2))), ("clf", LR())])
AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=3)
SentenceTransformer("all-MiniLM-L6-v2").encode(texts)
faiss.IndexFlatIP(dim); index.add(embeddings); index.search(query, k=5)

Examples

Input

Output

Edge Cases & Gotchas

Tokenizer max_length truncation - BERT tokenizes to max 512 tokens. Long documents get truncated. For long text: sliding window (stride=128, max_length=512), or extract first/last N sentences before tokenizing.
Language detection before NLP - Always detect language before applying language-specific pipelines. langdetect or fasttext-langdetect can classify language in milliseconds. Applying English NER to French text produces garbage.
Sentence transformers domain shift - all-MiniLM-L6-v2 is trained on web text. For medical, legal, or scientific text, use domain-specific models: pritamdeka/PubMedBERT-mnli-snli-scinli for scientific text.
Tokenization for product names - Product names with hyphens, slashes, or special chars (iPhone 14 Pro Max, GPT-4o) may be split unexpectedly. Test tokenizer output for your domain's vocabulary.

When to Break These Rules

Use regex instead of NLP for simple, well-structured pattern extraction (emails, phone numbers, zip codes) - no model needed.
Use OpenAI API instead of fine-tuning BERT when labeled training data is unavailable - zero-shot/few-shot GPT classification is often competitive with fine-tuned smaller models on <1000 samples.

Reference Files

https://spacy.io/usage
https://huggingface.co/docs/transformers/