In the videotape, a 25-year-old Englishwoman describes herself as having “gone back to being six.” Like a child with a doll, she clutches an empty egg carton. She denies knowing her age or what day it is. She tells the interviewer that she wants to sit on her husband’s knee and be cuddled by him. The regression began, she said, when officials threatened to arrest her husband on a criminal charge. Years earlier, the young woman recalls, she suffered a “breakdown” when, as a nursing student, she was required to assist at a birth.

The woman, Patient E, was one of eight (five English and three American) who were videotaped as part of a 1971 study comparing the diagnostic habits of English and American psychiatrists. Eighty-five percent of the American psychiatrists diagnosed her with schizophrenia. Only 7 percent of their British counterparts agreed. Roughly three-fourths of British psychiatrists believed that Patient E was a hysteric, but only one American did. The other seven patients elicited striking diagnostic mismatches as well. The crisis of credibility deepened, with rising discontent about the failure of psychodynamic psychiatry, the dominant practice at the time, to help people with serious mental illness. At the same time, the development of medications for treating psychosis, mania, and depression was gaining momentum, and their impact needed to be studied systematically.

These upheavals spurred the American Psychiatric Association to overhaul its diagnostic system. In 1980, the Diagnostic and Statistical Manual of Mental Disorders, Third Edition, (DSM-III) was unveiled. (The first two editions, appearing in 1952 and 1968, had included psychodynamic formulations presumed to reflect patients’ unconscious internal conflicts.) The blockbuster manual comprised 265 disorders, each defined by a set of detailed criteria determined by a committee. Rightly hailed as a “diagnostic revolution,” the manual established a much-needed common language with which to discuss patients. Psychiatrist Peter Kramer, the author of Listening to Prozac, told me that he credits the DSM-III with establishing “groundbreaking diagnoses such as autism, by splitting it off from schizophrenia,” and also “obsessive-compulsive disorder, and anorexia.” Researchers, too, welcomed the DSM-III as a blueprint for uncovering the biological underpinnings of severe conditions such as schizophrenia, bipolar disorder, and major depression.

Forty-two years later, the DSM revolution has run its course. We still lack clinically useful diagnostic tests, such as signature patterns on a brain scan or an electroencephalograph. Nor can we identify precise genetic risk factors, deploy neurophysiology to describe a predictable course of illness within categories, or foretell with confidence which patient will respond to which medication. Over time, the much-celebrated reliability of the DSM-III has waned in some places. After criteria were emended in the DSM-5, disagreement increased among professionals about what constituted major depressive disorder, substance-abuse disorder, and schizophrenia. Even more important, psychiatrists cannot demonstrate that diagnoses correspond to something real in nature, that they are valid. Despite great hopes and millions of dollars in research grants, scientists have yet to map the disorders onto abnormal brain mechanisms or developmental processes, and it therefore makes little sense to orient biological research around them. Clinically, this means that patients often receive multiple diagnoses at once and medications that too often are only partly effective.

The manual is thus a useful framework for organizing clinical phenomena but not for explaining them. However, as Kramer says, “this is not to say it has no utility, not at all — we can derive information from the DSM.” This is true. For the purposes of pharmacological treatment, for example, it is important to distinguish major depression from the depressive phase of bipolar disorder (because antidepressants can “flip” a bipolar patient into mania); it is important to isolate catatonia from schizophrenia (because catatonia responds well to tranquilizers but schizophrenia does not); it is important to distinguish psychosis brought on by methamphetamine from the mania of bipolar illness (because the drug-abusing person will probably “clear” with a day or two of rest).

The problem with the DSM is the overall absence of sharp, natural boundaries between categories at the level of defining characteristics. The frustration this has engendered is not new. Among the manual’s prominent critics are the former heads of the DSM-III and DSM-IV task forces and two former directors of the National Institute of Mental Health. But now the criticisms have coalesced into a new conventional wisdom within the field that the DSM approach is fundamentally faulty and that fixing treatment for severe mental ailments depends on fixing the diagnostic system.

Can it be fixed? Advances won’t happen overnight, but researchers are making progress in defining neural mechanisms and biomarkers that predict responses to treatment. Consider, for example, the search for distinct and reliable subtypes of DSM-defined disorders that share the same underlying pathophysiology. Work by Stanford’s Amit Etkin aims to predict better responses to particular medications, which would spare doctors and patients the trial-and-error hunt for the most effective medication. Researchers are studying EEG patterns — also called brain signatures — to predict responses to the antidepressant sertraline (Zoloft) in depressed patients. These biomarkers are synthesized using machine learning to identify people who respond better to a drug than to a placebo. This form of precision psychiatry may be the best means of deploying existing medications and helping to develop new ones.

Other teams are examining subtypes of depression using brain imaging to define the relationship between clinical symptoms and the strength of connections (circuitry) between key parts of the brain. In one study by Andrew T. Drysdale, now at Columbia University, 80 percent of people with a “biotype” that was characterized by reduced connectivity between the frontal cortex and the amygdala (areas associated with fear and the assessment of emotion) responded to a new treatment called transcranial magnetic stimulation. For the other three biotypes in that study, the response rate to that intervention was below 50 percent.

Another approach analyzes subtypes of psychiatric conditions using specific symptom profiles. Taking one such condition — psychosis — Jeffrey P. Kahn, of Weill Cornell Medical Center, tells me that “we too often casually lump psychoses together as schizophrenia.” To refine the picture, Kahn and his colleague André Barceila Veras have classified five distinct psychoses linked to five patterns of anxiety and depression. When a person with one of these forms of anxiety or depression also has biological factors (such as reduced frontal-lobe function) that make him prone to disordered thinking, the result can be the hallucinations and delusions of psychosis. One subtype features obsessive-compulsive traits, another hearing voices, another having persecutory delusions, another delusional depression, and another mania with delusions. Each responds best to a combination of an antipsychotic with specific antidepressants, mood stabilizers, anti-convulsants, or tranquilizers.

These projects either are in the testing phase or need independent replication. While promising, none have yet made their way into the clinic. Of course, even perfected medications and devices won’t be enough. In his new book, Healing: Our Path from Mental Illness to Mental Health, Thomas Insel, M.D., a former director of the National Institute of Mental Health, confesses to having long “misunderstood the problem” of treating mental illness. “While we identified the neuro-anatomy of addiction, overdose deaths had increased threefold. While we mapped the genes for schizophrenia, people with the disease were still chronically unemployed and dying twenty years early.” Well-designed programs for recovery, he writes, should aim at “finding connection, sanctuary, and meaning.” They should acknowledge, too, that poverty and life stress play a major role in outcomes.

Against this backdrop, the American Psychiatric Association (APA) just published a “text revision” to the DSM-5 that includes a new disorder: prolonged grief disorder, or PGD. Among the criteria are emotional numbness, a powerful sense of disbelief that a loved one is dead, and other symptoms lasting at least a year after the loss. Greeting the debut of PGD were the same basic questions asked of every new entry since 1980. Isn’t the manual simply medicalizing natural emotion — in this case, mourning? If not, why couldn’t it fall under an existing category — in this case, major depressive disorder? Also, why draw bright lines between pathology and health when they exist on a continuum?

To be fair, the DSM has been candid about its limitations. In the introduction, the manual says, “There is no assumption that each category of mental disorder is a completely discrete entity with absolute boundaries dividing it from other mental disorders or from no mental disorder.” Yet despite this crucial caveat, having elaborate definitions of disorders enshrined in official nomenclature inevitably suggests to the public that the categories are “real” and sharply distinct from one another and from normalcy.

One might ask why taxonomy even matters if, practically speaking, a person in pain could benefit from professional help. The answer involves insurance companies: They use the DSM to guide reimbursement. Other institutions lean heavily on the DSM as well. Social-service agencies rely on it to assess disabilities, and courts turn to it to resolve questions of personal injury and criminal culpability. The FDA ties its approval of new medications to DSM diagnoses. Finally, the DSM is a major source of income for the APA itself.

These interests largely ensure that the DSM is not going away anytime soon. And so in several years, the DSM-6 will arrive. For practicing psychiatrists, it will be a nonevent: After all, we don’t greatly rely on the DSM to treat patients, since our medications mainly treat symptoms, not disorders. And the plethora of categories it contains affords us enough leeway to fit patients into some kind of diagnostic cubbyhole for billing purposes. Still, as we use the agreed-upon nomenclature, we should also help downsize the significance of the DSM as a definitive arbiter of mental health in the eyes of the public.

Researchers operate in a different sphere. Perhaps they need their own DSM, an online, agile, easy-to-update version that catalogues the new subtypes of illness for further verification as they emerge. Instead of beginning with categories based on symptom groupings (e.g., schizophrenia, bipolar disorder, and panic disorder) and working backward to their neurobiological origins, a “research DSM” would classify conditions using underlying biology or a regrouping of symptoms that bear on treatment outcomes.

“The DSM is clearly at a crossroads, but the path it should take has no roadmap,” writes sociologist Allan V. Horwitz, the author of DSM: A History of Psychiatry’s Bible. As for future revolutions, we would be wise to heed the late historian of medicine Gerald N. Grob, who wrote of the disenchantments of psychiatry that every generation since the 1800s has “insisted that the specialty stood on the threshold of fundamental breakthroughs that would revolutionize the ways in which mental disorders were understood and treated.”

To be fair, we psychiatrists now have many medications to offer, and sometimes they truly transform patients’ lives, or at least quell unbearable anguish. We have also learned a vast amount about the brain, but we’ve overpromised on how quickly we would be able to translate those neuroscientific insights into strides in patient care.

The truly revolutionary idea is to forgo expectation of radical transformation. Progress in science and therapeutics will come, but it will be incremental and spring from myriad sources. And it will confirm that suffering can never be captured by categories.