More than 600 million people speak Hindi, and a huge share of them are online, posting the way people actually type when they're not writing for a textbook: mixing Hindi and English mid-sentence, switching scripts, dropping in slang, letting word order drift depending on mood. Linguists call this code-mixing. Most content moderation systems just call it noise they weren't built to handle.

That gap matters. A model trained mostly on English inherits a blind spot the moment it meets Hinglish, and when a moderation system misses real abuse — or wrongly flags an innocent post because it couldn't parse the sentence — someone on the other end absorbs that failure.

I wanted to test something concrete: does stacking a BiLSTM on top of BERT's embeddings actually improve toxicity detection on Hinglish text, the way the technique is supposed to? I went in expecting a clean win for the hybrid. What I got was more useful than that.

The setup

• Libraries: torch, transformers, scikit-learn, tqdm
• Backbone: bert-base-multilingual-cased (mBERT)
• Hardware: CPU — no GPU needed, though budget 15-20 minutes per run

The dataset is a 500-row sample, 250 toxic and 250 non-toxic, pulled from a larger 60,000-post Hindi social media corpus. Worth being upfront here: the full corpus had real label noise. A chunk of posts marked "toxic" turned out to be ordinary news or opinion text, nothing abusive about them at all. Rather than train on that, I filtered for clearly-labeled examples in each class and checked them by hand. That makes this a clean teaching example, not a benchmark on real-world noise — a distinction worth keeping in mind for everything below.

Three architectures, one question

Plain BERT. mBERT's pooled sentence representation goes straight into a linear classifier. No sequence modeling beyond what BERT already does internally.

class BertClassifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.bert = BertModel.from_pretrained('bert-base-multilingual-cased')
        self.classifier = nn.Linear(768, 1)

    def forward(self, input_ids, attention_mask):
        pooled = self.bert(input_ids=input_ids, attention_mask=attention_mask).pooler_output
        return torch.sigmoid(self.classifier(pooled)).squeeze()

BERT → BiLSTM. Instead of collapsing straight to one pooled vector, this feeds BERT's full token-level output — one embedding per token — into a bidirectional LSTM, which then handles the final classification.

class BertBiLSTM(nn.Module):
    def __init__(self):
        super().__init__()
        self.bert = BertModel.from_pretrained('bert-base-multilingual-cased')
        self.bilstm = nn.LSTM(768, 128, num_layers=2, bidirectional=True, batch_first=True)
        self.classifier = nn.Linear(256, 1)

    def forward(self, input_ids, attention_mask):
        bert_out = self.bert(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
        lstm_out, _ = self.bilstm(bert_out)
        output = lstm_out[:, -1]
        return torch.sigmoid(self.classifier(output)).squeeze()

The idea: Hindi's free word order means the word that tips a sentence from neutral into abusive can land near the start or the end. A bidirectional model should, in theory, track that better than a single squashed vector.

BiLSTM → BERT, frozen. BERT stays frozen entirely, and only its single pooled output — not the token sequence — gets passed through a BiLSTM.

class BiLSTMBert(nn.Module):
    def __init__(self):
        super().__init__()
        self.bert = BertModel.from_pretrained('bert-base-multilingual-cased')
        for param in self.bert.parameters():
            param.requires_grad = False
        self.bilstm = nn.LSTM(768, 128, num_layers=2, bidirectional=True, batch_first=True)
        self.classifier = nn.Linear(256, 1)

    def forward(self, input_ids, attention_mask):
        with torch.no_grad():
            pooled = self.bert(input_ids=input_ids, attention_mask=attention_mask).pooler_output
        lstm_in = pooled.unsqueeze(1)
        lstm_out, _ = self.bilstm(lstm_in)
        return torch.sigmoid(self.classifier(lstm_out[:, -1])).squeeze()

I threw this one in on purpose. It's almost guaranteed to underperform, and watching exactly how it fails tells you something.

What actually happened

I expected the middle row to win clearly over the top one. It didn't — plain BERT matched it, arguably edged it out. The frozen variant collapsed, exactly as expected.

Why the "obvious" improvement never showed up

This is the part most tutorials skip past.

The BiLSTM's real strength is tracking sequential context — how meaning shifts as a sentence unfolds — and that matters most when the text is ambiguous. My 500-row sample was filtered specifically to remove ambiguity: clear-cut toxic examples, clear-cut clean ones, nothing sitting in between. On text that unambiguous, mBERT's own attention mechanism is often already enough. There's no subtle sequential signal left for a BiLSTM to squeeze extra value out of.

My guess is the BiLSTM's advantage is real, just not visible here — it probably shows up on the harder, messier data: sarcasm, an insult buried in an otherwise casual sentence, a toxic-sounding word used in a way that isn't actually abusive. That's precisely the kind of ambiguity a small, clean sample is built to exclude.

The frozen variant's collapse is easier to explain. Freeze BERT, pass in only one already-pooled vector, and you've removed the sequence entirely — there's nothing left for a bidirectional LSTM to sweep forward and backward across. The architecture's whole premise depends on having a sequence of positions to work with. Take that away and weak results aren't surprising; they're confirmation the setup didn't hold together in the first place.

If you're building something similar, don't take "always add a BiLSTM" or "never bother" from this. Test on data that resembles your hardest real cases, not your cleanest ones. A model that looks identical to a simpler one on easy examples can pull ahead — or fall apart — once you feed it the sarcastic, code-mixed, ambiguous content that made the problem hard to begin with.

This is a small demonstration, not a production system. But if you're working on moderation for another low-resource, code-mixed language — Bengali, Tamil, Assamese, and plenty of others face the same gap — this three-way comparison is a fast way to find where the real signal actually sits, instead of assuming it.

Do you think the BiLSTM would pull ahead if I introduced more sarcastic or code-mixed examples? I'd love to hear where the "complexity ceiling" usually sits in your own NLP work.

Explore the full code & dataset on GitHub