Multimodal sarcasm detection
Sarcasm lives in delivery as much as wording, so the model listens as well as reads.
- Mean average precision
- ≈65%
Overview
Sarcasm is often carried by delivery rather than wording, which makes text-only detection structurally limited. This pipeline models acoustic and linguistic signal jointly to catch the cues a transcript drops.
Raw audio was converted into tonal and temporal features — pitch contours, loudness profiles, energy patterns — and paired with semantic embeddings from the dialogue transcripts. CNNs handle local feature extraction, RNNs handle temporal dependencies, across both modalities.
Challenges
A focused experiment converted audio into prosodic visual representations and trained a CNN ensemble on those alone, to test whether tonal variation carries sarcasm by itself.
It doesn't, reliably. Tone-only models degrade under neutral or low-variance speech — which is exactly the delivery deadpan sarcasm uses.
Decisions
The calls that shaped this project, the alternatives they beat, and what each one cost.
Test tone in isolation before trusting it in the fusion
over assume the audio branch contributes and move on
WhyConverting audio into prosodic visual representations and training a CNN ensemble on those alone answers a question the combined model can't: does tonal variation carry sarcasm by itself, or is it riding on the text?
Trade-offA separate experiment that produces no component for the final system.
Ensemble fusion of tonal and semantic cues
over a single model over concatenated features
WhyThe two modalities fail in different places — tone degrades on neutral, low-variance speech, and text misses sarcasm that reads literally. Keeping them separate until the aggregation step means one failing doesn't drag the other down.
Trade-offMore components to train and tune than a single fused model.
CNNs for local features, RNNs for temporal structure
over a text-only classifier over transcripts
WhyPitch contours, loudness, and energy only mean anything as trajectories over time. A model without a temporal component can see the features but not the delivery.
Trade-offSequence models need more data than MUStARD comfortably provides.
Outcomes
An ensemble fusion strategy aggregating tonal and semantic cues beat both unimodal baselines.
Highest mean average precision on MUStARD at roughly 65%, above baseline and single-modality approaches.