The Most Important Thing That Ever Happened: Big,Bad Data and the Doubling of Human Life Expectancy

The greatest demographic disaster ever to strike the city of New Orleans was not Hurricane Katrina in 2005 but the yellow fever epidemic of 1853. Like Katrina, the disaster was man-made, a product of natural forces meeting (and revealing) the political and economic realities of life on the ground. To truly appreciate the devastation, we would need to tour the homes of 10,000 men, women, and children who found themselves suddenly vomiting up “black bile of a most fetid and acrid nature.” Instead, I want to view the disaster from a perspective almost sadder: that of the city’s beleaguered Board of Health. ¹

Unlike most antebellum cities, New Orleans did have a Board of Health, but it was ill-equipped, ill-informed, and disrespected. In May, the Board reported the first case of yellow fever in the city, but the newspapers refused to cover the story as the situation was not yet dire. A month later, the situation was so dire that the newspapers refused to report the story lest panic dampen “the interests of trade.” “Madam Rumor has given rise to the fancied existence of yellow fever in this city to a very great extent,” reported the Picayune. “It is admitted that it has occurred in the city to the enormous extent of about four cases, a thing not unusual in any year past. But that it exists in a form or to an extent to produce alarm, except among old women, is most positively denied by all the most eminent physicians in the city.” Another month later, the city was in the grips of a total panic, and the newspapers reversed course, blaming the Board of Health and city government for failing to do anything about streets that “looked and smelt epidemical,” with gutters so filled with “carrion and carcasses and festering nastiness of every description” that they seemed to “exhale” “filthy compounds [and] abominable smells.” When a local physician dared advance the “strange and absurd theory” that the gutter garbage was not the source of the scourge, he was denounced as “a fool who deserved pity, or a knave who deserved the severest punishment.” And when a rare summer deluge washed out the gutters, the newspapers greeted it as “Providential interference.” “Could we only calculate upon as much rain once or twice every week of the summer!” exclaimed the Delta. “The weather is the strangest ever known in the city,” noted the Crescent. “It is regular April weather, almost in the middle of July!” ²

The “strange” weather was the problem; the heavy summer rains fed the pools of standing water that served as breeding ground for yellow fever’s carrier, the female Aedes aegypti mosquito. A fifth of the city died, and the blame went nowhere it belonged. Racists blamed the city’s “foreigners”—meaning Southern Europeans—for “bad hygiene.” Political reformers blamed the city’s grotesquely corrupt politicians. The corrupt politicians blamed the reformers for failing to pass infrastructure bills that would surely have helped (their own pocketbooks). And everyone blamed the only organization that had tried to make a difference—the Board of Health. “Will the proper officers please inform the public, who compose the ‘Board of Health?’” the Picayune asked sarcastically. “Does such a body exist?” It might as well not have. ³

Effective public health planning requires three things: an accurate assessment of the problems, the capacity to implement effective solutions, and the political will to act. In 1850, America lacked all three. Over the course of the next century, however, life expectancy in the United States (and across the globe) doubled. The story of that doubling—the most important thing that has ever happened—is typically told as a series of heroic medical and sanitary breakthroughs: the development of antisepsis, vaccines, antibiotics, clean milk, and water supplies. I want to make a parallel and complimentary case: the global doubling of human life expectancy was a triumph of bureaucracy and data.

* * *

Statistics are central to how a state “sees.” The words are etymologically related for a reason. A state that can “see” can do good or bad things. The most chilling scenes in Schindler’s List are not the retail killings of Amon Goeth but the stomach-churning revelations of a Nazi state committed to the wholesale modernization of murder. Desks await trains (that run on time), pencils are sharpened, and the paperwork is ready.

And yet Schindler made a list of his own because data is power.

As historian Patricia Cline Cohen has noted, the first half of the nineteenth century witnessed a dramatic expansion in quantification, numeracy, and data-gathering across the globe. The reasons were manifold. The rise of a market economy required and rewarded precision; common schools made the mass of men (and then women) mathematically literate; and the profit motive focused them on the bottom line. This rise of numeracy had its unsavory side. As Machiavelli famously noted “men forget more easily the death of their father than the loss of their patrimony,” and Americans were nothing if not sharpers. Melville wrote The Confidence Man in this period, and Poe wrote “Diddling Considered as an Exact Science.” The Americans’ most “prominent feature,” said Charles Dickens after his American tour, was a “love of ‘smart’ dealing, which gilds over many a swindle and gross breach of trust.” One glum bureaucrat put it more baldly: “It is more easy to satisfy mankind of the value of any other branch of statistics than that which relates to the numbers that die in any given period. In our practical, money-making country [we run] after those numbers which relate to the fluctuations of stocks, the valuation of exports and imports, the rates of taxation and the results in agriculture and manufactures, internal improvements and general commerce. [But] the life of man is of less importance than his larder and his cloth.” ⁴

Gradually and grudgingly, however, Americans applied their numerical talents to public health, and the results were astounding. In 1839, the American Statistical Association founded itself on the principle that data might “enhance human welfare.” Eleven years later, one of the Association’s founders, Lemuel Shattuck, published a sanitary survey of Massachusetts that became the blueprint for public health reforms across the country. Forty years after the lamentation about “larder and cloth,” US Surgeon General Joshua S. Billings could say with the flatness of fact that “[i]t is now…generally admitted that it is the duty of a State to protect its citizens against unnecessary disease and death.” This alone was a conceptual breakthrough. The Constitution of the United States mentions the “general welfare” of Americans exactly twice and never in the context of public health. As the Founders understood it, “general welfare” included “domestic tranquility” and “common defense” but not a common defense against disease. A hundred years later, Americans had come to expect that their government would, could, and should coordinate responses to biological threats as well as military ones. ⁵

How had this happened? Billings credited the data. Public health statistics, he said, had become “the eyes of the State Board of Health [and] without [them] the Board is like a blind man trying to put out a prairie fire.” Certainly it was still possible to blame epidemics on the poor or proselytize about differences between black-and-white biology, and for the next century, whole political platforms would be built on scapegoating and fearmongering. But for the first time, bad faith arguments had to be made in the face of good data. ⁶ OPEN IN VIEWER

Take the example of Florence Nightingale. She is remembered most (and most dismissively) as the “Lady with the Lamp,” making her tireless rounds of the sickbeds of the Crimean War. “This frail young woman,” noted one general, “embraced in her solicitude the sick of three armies.” She would be better understood, however, as a pioneering statistician, a wholesaler, not retailer, of medical care, who forced generals to see at scale. Preoccupied with inflicting and sustaining damage on the battlefield (a fixation that would continue through nightly news reporting on the killed in action among the Viet Cong), military personnel cannot always see that their primary enemy is microbial. In producing her famous rose diagram, displaying the “causes of mortality” sustained by her army, Nightingale made public health data actionable and compelling. An army is nothing if not a mobile city demanding sanitary solutions to the problems inevitably produced when we stir the biotic soup, bringing novel populations and diseases into close quarters. Nightingale was a pioneering nurse not because of her retail care but because of her wholesale commitment to data visualization in the service of city planning. ⁷

This revolution in data conception, collection, and popularization is particularly evident in the history of the US census. The census was created by Article I of the Constitution, which notes that “Representatives and direct Taxes shall be apportioned among the several States…according to their respective Numbers…. The actual Enumeration shall be made within three Years after [1787]…and within every subsequent Term of ten Years.” Strictly speaking, then, the sole constitutional function of the census is to count heads for purposes of apportioning representatives to Congress and the Electoral College. Even so, the need, value, and maturation of statistics, coupled with Americans’ pride and curiosity about their growing capacities, ensured that the census would be constantly expanded. The “mission creep” began immediately. In 1800, the census collected broad age ranges of the free white population because James Madison wanted to know how large an army we could field. In 1810, Jefferson suggested collecting Americans’ occupations, marital status, and countries of origin. And in 1850, the census bureau created an entirely new schedule—Schedule Three, the so-called Mortality Census—the first-ever public health survey in America.

To be sure, the survey was poorly conceived. Having located a head of household, census marshals were instructed to plow through Schedule One (demography) and Schedule Two (the “slave schedule”) before asking if anyone in the household had died in the previous year. There are myriad problems with this approach. Census marshals were generally seen as intrusive bureaucrats, the advance guard of taxmen; deputizing them to discuss intimate family matters was destined to fail. Certainly no American farmer of the period would have stood on their piazza with a government functionary and confessed to a murder or easily volunteered the medical conditions of his dependents—unwed daughters who had died in childbirth or enslaved men and women who had been beaten to death. And then there is the larger problem of an entire enterprise that depends on people ignorant of medicine to function as diagnosticians. As recorded in 1850, the leading cause of death in America was “unknown.” Behind these were “fever, not specified,” “accident, not specified,” and a ridiculous number of people who had apparently died of menstruation, herpes, masturbation, and a host of other unlikely scenarios. No wonder, then, that demographers have generally not wanted to touch the Mortality Census with a ten-foot slide rule.

Such data snobbery, however, has only made more difficult the job of pursuing what Steven Johnson has called the “sociology of error,” or I would call misepistemology—the history and science of how we come to “unknow” what we think we know. ⁸

Because here’s the truth: the Mortality Census was not perfectly bad data. The murder capital of America, according to the 1850 census, was California, followed quickly by Texas, Florida, Arkansas, Georgia, Kentucky, Louisiana, Alabama, Tennessee, Virginia, Mississippi, North Carolina, South Carolina, and Missouri. With a seven to one gender ratio of males to females, in a state where gold had just been discovered and law enforcement was minimal, California probably was a murder palace. And the fact that every state in the list thereafter was Southern? This couldn’t have been information that census superintendent James B. DeBow, himself a Southern nationalist, wanted to reveal, but it rings true, not least because murder was something anyone could diagnose, even the average American farmer. ⁹

More importantly, for the misepistemologist, no data are inherently bad. All data reveal something, if only the wretched history of how and why the data got collected in the first place and the more hopeful history of how it eventually got better. And the data did get (very gradually) better. During the Civil War, the 1860 census was calculated and released by census boys who might have been sent to the killing fields but were instead dragooned to make maps and tabulate data for the War Department. “No military expedition was ever based on sounder or surer data,” said Sherman as he embarked upon his March to the Sea. Armed with maps displaying county-level grain yields and enslaved populations, Sherman turned data into actionable military intelligence: “I had the Census statistics showing the produce of every county through which I designed to pass,” he told a friend. ¹⁰

During the 1870 census, public health statistics (and the real threats to humankind) again took a back seat to the bigger issue in census politics: the question of how many former slaves there actually were in the South. (This is an irony we don’t often think about, but the end of the Three-Fifths Compromise effectively increased the representation of the ex-Confederate states in Congress. If they could deny blacks the right to vote, white supremacists could achieve two-fifths more power.) For all these wretched politics, however, data have a way of pushing back and revealing the truth: what divides us politically can never be as strong as what unites us biologically; whatever enemies we find among ourselves, they are never so great a threat as the enemy that recognizes our common humanity, even when we can’t—the microbes that answer only to natural logic. And, so in the 1880s, at the nadir of American race relations, census personnel produced some of the nation’s first-ever attempts to visualize and map disease “hot spots” because the data had revealed that place, not race, was a better indicator of human health outcomes. In 1888, the Census Bureau sponsored a contest to see who could design the fastest data-tabulating machine, and the winner was a former census employee, Herman Hollerith, the founder of the American Tabulating Machine Company, later consolidated as IBM. However, grotesque its flaws and assumptions, the census had become the “Big Data” project of the nineteenth century, driving a data industry that got bigger better fast—ultimately producing the supercomputer. ¹¹

By 1900, the Mortality Census had been superseded by a new Bureau of Vital Statistics and the advent of a national, physician-overseen death registration system. The system was implemented unevenly, coming to South Carolina only in 1915 and Georgia in 1918. Even so, for the first time, Americans were recording what they were actually dying of and amazing developments followed. We put niacin in our bread so we would stop dying of pellagra. We put fluoride in the water so we’d actually have teeth beyond the age of forty. We put up screens and mosquito nets and eradicated malaria in the United States. From the ruins of the Malarial Control Division of World War II grew the Centers for Disease Control, which quickly created national programs focusing on venereal disease, tuberculosis, and mass immunization. In 1948, the United States took the lead in forming the World Health Organization, which by 1980 had slayed the dragon, eradicating smallpox, the greatest global killer of all humankind. For a short time, in certain circumstances and places, the nine most magical words in the English language were “I’m from the government and I’m here to help.”

* * *

In 1862, New Orleans fell to the Yankees. The occupying Union General, Benjamin “Beast” Butler, was reviled by city residents for his draconian rule. He muzzled the press. He had a man hanged for tearing down a US flag. He issued an order that any woman disrespecting Union soldiers would be treated as “a woman of the town, plying her avocation”—meaning a prostitute. But he also brought to the city new federal standards for sanitation. Under occupation, New Orleans was as healthy as it had ever been. Butler’s quarantine system was so effective that only two people died of yellow fever. After the war, New Orleans’ Reconstruction government wanted to extend Butler’s sanitary reforms by cleaning up the city’s low-rent slaughterhouses, which were washing “intestines and portions of putrefied animal matter” into the city water supply. The independent white butchers sued under the newly enacted Fourteenth Amendment, claiming that the City of New Orleans had denied them due process and equal protection in demanding that all livestock landing, penning, and slaughtering take place in one compliant facility. The Supreme Court found in their favor, and the butchers went back to dumping their pig parts wherever they pleased. ¹²

A doctor can only save you if you take the medicine. “I’m from the government and I’m here to help” only works if people want to be helped. In the past two decades, “deaths of despair” have risen dramatically; the United States is now losing hundreds of thousands of its citizens each year to alcoholism, drug abuse, opioid addiction, suicide, and other self-destructive behavior. American life expectancy has fallen four years in a row—a reversal not seen since 1918—and 2020s numbers are going to be even worse because of our botched response to COVID-19. At this point, we human beings have become the canaries in our own coalmine. Like bees, we are experiencing a sort “colony collapse” as we fail the simplest test in nature: acting in our own self-interest. ¹³

In his introduction to this special issue of the Journal of Planning History, Stephen Ramos rightly notes that urban planning emerged “as a reformist profession [charged with building] cities for industrial capital accumulation, while at the same time protect[ing] their publics from the social consequences of these new urban environments.” The concentration of industrial capital, in other words, produced a concentration of human beings, which in turn produced a concentration of disease vectors. Nothing could be a more elegant encapsulation of the boomerang effect of exploitation, which weakens the body politic and disturbs the environment, leading to diseases which fall heaviest on the exploited but effect the exploiters as well. Historians have long wondered why the rise of capitalism and the rise of a “humanitarian impulse” grew up together, but it is a nonmystery. Even as it enshrined self-interest, greed, and competition, capitalism required humanitarianism to contain the boomerang effect. The point of this article has been to make a related claim: capitalism’s demand for numeracy and data processing simultaneously allowed humanitarianism to become more precise and effective.

For 2 million years, the greatest threat to humankind was disease. Relatively yesterday, relatively overnight, we doubled the length of our lives—the most important thing that has ever happened—and we did it by following the numbers. What happens, however, if we stop believing in numbers? Caught in a cultural backwash of anti-masking, anti-vaxxing, and “reality wars,” America is experiencing an epistemic crisis. Having brought most major diseases to heel, we have weaponized a disease of our own making: ignorance.