Why this problem
Mental health content online does not announce itself. Someone posts a meme of a dog sitting in a burning room saying "this is fine," and that is not small talk, it is a coping mechanism rendered in a format that makes the sentiment both easier to share and harder to process algorithmically.
There is a substantial body of work on detecting mental health signals in text, and a separate body of work on meme understanding. The gap sits between them, in classifying mental health memes at the symptom level. Not "this seems depressive" but specifically "this expresses a sleeping disorder" or "this expresses self-harm ideation." That granularity is what any downstream triage or flagging application would actually need.
We worked with two curated datasets. RESTORE covers depression as a multi-label problem, with categories derived from PHQ-9 including Feeling Down, Self-Harm, Eating Disorder, Sleeping Disorder, Low Self-Esteem, Concentration Problem, Lack of Interest and Lack of Energy. A single meme can express several of these at once. AxiOM covers anxiety as a single-label problem, with categories derived from GAD symptoms: Restlessness, Nervousness, Impending Doom, Difficulty Relaxing, Lack of Worry Control and Excessive Worry. Both datasets come from the reference work we built on, Figurative-cum-Commonsense Knowledge Infusion for Multimodal Mental Health Meme Classification (Mazhar et al., 2025), which introduced AxiOM alongside the M3H model and argued that commonsense reasoning and figurative understanding are necessary rather than optional for this task.
Both datasets are also noticeably imbalanced. On the depression side, Feeling Down appears 2,085 times in training while Lack of Energy appears 122 times, a ratio of roughly seventeen to one. The anxiety categories are far more even, ranging from 368 to 463 examples. That difference turns out to matter when reading the per-class results.
What the obvious approach gets wrong
The obvious approach is to pull the text out of the meme with OCR, embed it with BERT and classify. It works up to a point. Our OCR and BERT baseline, using EasyOCR for extraction, reached Macro F1 of 0.5527 on depression (weighted F1 0.6236) and Macro F1 of 0.4746 on anxiety (weighted F1 0.5548).
The problem is what the OCR text does not contain. A meme captioned "when someone asks if you're okay lol" over an image of someone visibly crying behind a forced smile communicates something the word tokens miss completely. The irony, the cultural shorthand and the visual contradiction all carry diagnostic signal, and a text encoder sees only the surface, which in this case states the opposite of what the meme means.
Adding global image features does not close that gap either. A classification head pooled over a whole image cannot separate the elements that make a meme legible, such as the facial expression, the caption placement and the reaction template being referenced. The semantics live in the relationship between the OCR text and particular visual regions, not in the image averaged into a single vector.
The core decision
We built three models rather than one, which was the right call in retrospect because it isolated where the gain actually came from.
- The baseline is OCR text and BERT, described above, and exists to establish how much of the task is solvable from surface text alone.
- The second model integrates three sources of information: OCR text, figurative reasoning, and visual content. Text is embedded with the BAAI/bge-m3 sentence transformer. Figurative reasoning is extracted using Qwen2.5-VL-7B-Instruct, prompted to produce cause-effect, metaphorical and emotional representations of each meme, and those are embedded through the same transformer. Visual features come from a modified ResNet-50. The three are concatenated into a 3,072-dimensional multimodal vector per meme, which is then used both for classification and, through retrieval, for providing similar training examples as context. This is the M3H pipeline extended with vision-enriched reasoning.
- A third architectural design, HyCore-M3Net, was proposed in the course report as a two-stream concept combining LXMERT cross-modal attention with a MentalBART encoder and retrieval augmentation. However, this architecture was a conceptual proposal and remains unimplemented and unmeasured with no reported experimental numbers in the source document. All reported experimental evaluations in the report and our findings reflect our implemented baseline and vision-enhanced fusion models.
The signal is not in the text and it is not in the image. It is in what the meme implies, and capturing that requires a model that reasons about figurative meaning before it classifies.
What worked, what didn't
The comparison we can report cleanly is between the baseline and the vision-enhanced model, and the gap is informative.
On anxiety, Macro F1 rose from 0.4746 to 0.6512, an improvement of roughly seventeen points, with weighted F1 moving from 0.5548 to 0.6510. On depression, Macro F1 rose more modestly from 0.5527 to 0.5822, with weighted F1 moving from 0.6236 to 0.6597.
That asymmetry is the most interesting thing in the results. Anxiety is single-label with reasonably balanced classes, and adding figurative reasoning and visual features improves it substantially. Depression is multi-label with a seventeen-to-one imbalance between its most and least frequent categories, and the same additions buy only three points of Macro F1. Macro F1 weights every class equally, so the rare depression categories are dragging the average down regardless of how much richer the representation gets. Better features do not fix a class you have barely any examples of.
The report's error analysis also points at a specific confusion in the anxiety set, between Impending Doom and Lack of Worry Control, which is not a modelling failure so much as a genuine overlap in how those two states get expressed in meme form. Two labels that different people would annotate differently are going to cap what any classifier can do.
The training curves are worth reading honestly. Training loss falls towards zero while validation loss climbs steadily from around the third or fourth epoch onward, across runs. The models memorise the training set, including the figurative reasoning traces attached to it. My own interpretation, which goes beyond what we measured, is that those traces are the constraint: Qwen produces plausible reasoning, but not consistently comparable reasoning, since two memes carrying nearly identical visual irony can receive quite different traces depending on image quality, prompt framing and which part of the cultural context the model happens to verbalise. Some traces translate the meme into symptom language precisely, while others describe the visual layout without engaging the implicit meaning at all. If that is right, architectural sophistication does not help, because it only memorises the inconsistencies faster.
What I'd do differently
I would audit and filter the figurative reasoning before it is used as a training feature. The pipeline currently generates a trace per meme and consumes it without any validation step. A lightweight quality filter, checking whether a trace engages the emotional subtext rather than merely describing the image, would plausibly do more for the results than any further change to the architecture. Consistency matters more than volume here.
I would also stop training much earlier. Validation divergence is visible and consistent by the fourth epoch in every run across both tasks, which means the useful checkpoints sit around epochs two and three. Running ten epochs was spending compute to overfit.
Finally, I would handle the depression class imbalance directly rather than hoping richer features would absorb it. Class-weighted loss, resampling or simply reporting per-class results alongside the Macro average would have made the depression numbers interpretable instead of merely low.
