The precise role of serotonin (5-HT) in psychotic disorders is not fully understood. However, a variety of preclinical and clinical data suggest that 5-HT is closely connected with both psychotic symptoms and the mechanism of newer atypical antipsychotic agents. The serotonin-2A receptor (5-HT2A) in particular seems to play a pivotal role. For example, certain psychotomimetic substances, such as psilocybin, appear to act as agonists at the 5-HT2A receptor. Conversely, 5-HT2A antagonists, such as ketanserin, may block the psychotomimetic effects of psilocybin. Thus, it is possible that some types of psychosis are related to 5-HT2A overactivity, whereas antagonism at this receptor produces antipsychotic effects.
While psilocybin and LSD-based models of psychosis may not be fully applicable to schizophrenia, the mechanisms of several atypical antipsychotics may involve antagonism at the 5-HT2 receptor. Indeed, this same receptor may be involved in some of the negative symptoms of schizophrenia, for which atypical agents seem especially useful. For example, Ceulemans et al. showed that setoperone, a 5-HT2 antagonist not in clinical use in the United States, led to improvement in emotional withdrawal, autistic behavior and dysphoria in patients with schizophrenia.
Of course, the issue of serotonin’s role in psychosis is complicated by the plethora of 5-HT receptor subtypes, and their varying effects on behavior. For example, some animal models of psychotic behavior suggest that the 5-HT2A and 5-HT2C receptors may have opposite actions. Futhermore, “pure” antagonists of the 5-HT2A receptor (such as M100907) may have different effects on, for example, locomotor activity in rodents than do “mixed” antagonists (5-HT2A+5-HT2C) such as ritanserin. Clinical studies of psychosis also suggest that 5-HT effects are very complex. Thus, selective serotonin reuptake inhibitors (e.g., fluoxetine [Prozac]) may ameliorate negative symptoms of schizophrenia in some patients, which seems somewhat paradoxical in view of the aforementioned 5-HT2 antagonist effects. Clearly, we have much more to learn about the role of 5-HT in psychosis, but there is little doubt that serotonin is a “major player.”