Among his Columbia colleagues and students, Shaw-Hwa Lo is known as a brilliant statistician, a dedicated teacher and an innovative researcher. Lo, a Professor of Statistics, has taught at Columbia for nearly 30 years. He began his career as a mathematical statistician and his early contributions to the field were in the arcane-sounding areas of Asymptotics Theory, Survival Analysis, and Resampling Methods. But in the late 1990s, he expanded his reach by using statistical methods to understand the burgeoning field of genetics. Back then, advances in genome studies had increased the size of genetic data and identified genetic variants responsible for several diseases. How to predict which variants caused what diseases became a central question of the time. Lo merged statistics and molecular biology so as to quantify 100 millions of markers (SNPs) in DNA, helping ultimately to decode the human genome. He also built statistical models to identify genes associated with certain diseases such as inflammatory bowel disease, Gilles de la Tourette Syndrome, breast cancer and leukemia.  

Lo’s innovations in statistics has enhanced a number of fields such as molecular biology, transportation and medicine, and in recognition of his life-time achievements the New England Statistics Society (NESS) recently gave him the inaugural Chernoff Excellence in Statistics Award, its highest honor. The award recognizes a researcher who has made major contributions to the fields of statistics and data science.

“I’m honored to receive this award,” says Lo, “especially since it’s given in the name of Herman Chernoff, my colleague, mentor and long-time collaborator.”

In the early 2000s, working with another collaborator, Lo developed a statistical method called the I-Score that drastically improves the ability of statisticians to make accurate predictions in fields such as medicine, genetics, financial markets and electoral politics. He created the I-Score with Tian Zheng, his former student who is now Chair of the Statistics Department and Associate Director for Education at the Data Science Institute.  Zheng has collaborated with Lo for two decades and says she has benefited enormously from his guidance.

“I worked with Professor Lo from 1999-2002 for my doctoral studies and have been collaborating with him ever since,” adds Zheng. “I was lucky since he had just started working in statistical genetics and together we have worked on the I-Score, a highly predictive framework that enhanced the ability of statisticians to make better predictions.”

From Taiwan to California

Lo grew up in Taiwan, where as an undergraduate he studied math at National Taiwan University. A gifted student, he won a scholarship to study math at the University of California, Santa Barbara, where he first took statistics. He recalls trying to understand, without much success, a vexing mathematical problem. At the time, a math statistics professor at UC Berkeley (Lucien Le Cam) was the leading expert on the problem. Lo wrote to him for advice, and the two developed a rapport.  Lo later transferred to Berkeley to study with Le Cam, switching his major from math to statistics. After earning his doctorate, he was hired by Rutgers University as an assistant professor. He excelled there as an assistant professor and in 1986 was hired by Harvard. In 1990, he took a job at Columbia University, where he has taught ever since.

Here, he teaches statistics and probability while supervising doctoral and master’s students and the occasional undergraduate. Jonathan Auerbach, a doctoral candidate in statistics, describes Lo as a dedicated teacher and mentor– a brilliant statistician with an encyclopedic knowledge of the field. “He can anticipate problems that arise in research and communicate them expertly,” says Auerbach, “which makes him an ideal adviser and collaborator.”

Lo recreated the Statistics Department’s master’s program, adds Auerbach, and he meets with students often to advise them and introduce them to research. “Prof. Lo’s instruction is invaluable and many of his students have gone onto successful research careers.”

As a researcher, Lo has an uncanny ability to apply theoretical statistics to help solve real-world problems. And most recently he has been using his I-Score platform to combat a vexing transportation problem. Last year, Lo and Zheng partnered with the New York City Department of Transportation (DOT) on Vision Zero, a city-wide effort to end traffic deaths. The DOT collects data from accidents to analyze the factors that cause crashes. The data are complex – hundreds of variables come into play relating to road conditions and traffic – so DOT officials can use I-Score to predict where crashes are most likely to occur and what changes they can make to prevent them.

“This project concerns the safety of all the citizens of New York City,” says Lo. “We are developing a statistical method that will hopefully be of help to DOT. It’s a perfect example of how statistics can be applied to a serious social problem.”

Zheng says the DOT work illustrates one of Lo’s guiding principles, which he shares often with students.  “He always says not to work on problems that are caused by the limitations of today’s technology,” says Zheng, “but rather to work on challenging problems that cannot be solved by simple technological advancements.”

Xiao-Li Meng, a former student of Lo’s who is now  the Whipple V. N. Jones Professor of Statistics at Harvard, says Lo’s contributions to statistics are indicative of the breadth and depth of his thinking. Meng studied with Lo as a PhD student at Harvard and describes him as an inspirational teacher whose lectures he’s never forgotten.

“I can still recall,” says Meng, “how intrigued I was as a student when Professor Lo taught me how to think about Bootstrap–perhaps the most commonly used method for assessing uncertainty–through the lens of an innovative stochastic mapping perspective he had developed. His explanation immediately made Bootstrap much less mysterious to me.”

Lo’s work on estimating the number of new animal species, Meng adds, helped him understand another seeming statistical conundrum: How could anyone estimate the number of species that one has never seen, other than to say it’s zero?

“Professor Lo’s insights on quantifying species was fascinating,” says Meng, “and had the added benefit of making all my subsequent fishing trips much more educational, since I would go out thinking how many new kinds of fish would I catch today?”

Data Science Emerges

Over the course of his career, Lo has kept up with emerging technologies and he’s seen data science grow from a niche field to one that’s transforming the academy and the workplace. As a member of DSI, he’s a large proponent of data science and uses it, especially machine learning, to enhance his research. Whereas 15 years ago few statisticians ever heard of data science, he says, now it’s becoming an influential part of statistics, engineering, computer science and a growing number of fields in the arts and social sciences. Data scientists are in great demand, moreover, and command the highest starting salaries in today’s job market.

“Data Science is the newest and best method for making predictions about complex large-scale data,” says Lo. “When I was a graduate student, few ever heard of it. Yet look now at the tremendous impact it has had on so many fields.”

Papers, Chinese tea, and thoughts

On a recent afternoon, Lo welcomes a visitor warmly into his Columbia office – a mare’s nest of books, papers and manuscripts that carpet every surface of the room. Seated at his desk, he pecks away at a black laptop perched precariously atop a mound of papers. Stout textbooks and canisters of Chinese tea line his bookshelves. He loves strong black tea, and whenever his students visit China, they bring him back canisters of tea.

Lo dresses casually in jeans, t-shirt, and a black jacket. He’s energetic and fast-talking and is quick to laugh – the abandoned laughter of a person who knows what he was put on earth to do and is doing it happily. He lives on the Upper West Side with his wife, Vicky, and their Australian Shepherd. He likes to get up at 6 a.m. and go for runs in Riverside Park. Afterwards, he visits the Starbucks on Broadway for coffee. He brings with him a yellow writing pad on which he jots down the ideas that come to him unbidden in moments of reflection. “A number of  ideas for papers have come to me and I’ve applied them to a few of problems while sitting in cafes,” says Lo.

On his office wall, tacked to a corkboard, is a photo of two children playing in FAO Schwartz, the iconic toy store. They are his son and daughter, and they are both adults now. His daughter Adeline, is an assistant professor of  political science at the University of Wisconsin, Madison, and his son, Alexander, is a lead data scientist for a tech firm in Manhattan. Both graduated from Columbia College. His wife used to work in finance but left her job to raise her two children. “She told me then that raising two kids is a full-time job,” says Lo. They’ve been married, happily, for 35 years and in summer they like to visit America’s national parks.  For him, America has been a land of opportunity – the land of milk and honey -– and he’s quick to praise its virtues.

“I tell my students that here in America if you work hard and are lucky you’ll be successful,” he says. “I came here as a foreign student with nothing. And I’m teaching at Columbia now for nearly 30 years. I feel very lucky.”

Lo did what he advises his students to do. He worked hard and, with a stroke of good luck, became a distinguished professor at one of the most prestigious universities in the world.

Land of Opportunity: America

Every time opportunity knocked for him, Lo answered. He reached the pinnacle of higher education in working at Harvard and Columbia, and his winning the Chernoff Excellence in Statistics Award confirms his elite status in the field of statistics. Yet, despite his success, he remains humble.

At 96, Herman Chernoff – now Professor Emeritus of Applied Mathematics at MIT and of Statistics at Harvard University – still collaborates with Lo on statistical research. Chernoff first met Lo in 1987 when he was appointed as an associate professor at Harvard. The two had adjoining offices, and Chernoff enjoyed hearing about the progress of Lo’s research. They soon began collaborating and have worked together ever since. Chernoff says Lo is a brilliant statistician whose research has extended the field into areas such as genetics and molecular biology. And as chair of the Statistics Department at Columbia, adds Chernoff, Lo helped the department to flourish.

“Among his other capabilities, Professor Lo is a gourmet and whenever we get together he finds the best restaurants for us,” says Chernoff. “At first he confined his expertise to Chinese restaurants, then added international restaurants in New York, and now when he visits me in Boston he finds the best restaurants in the city for us – though I must admit this sometimes leads to parking problems. More seriously, though, I was touched when the New England Statistics Society told me they’d name their award for excellence after me. And I was doubly touched when I learned Professor Lo would be the first recipient of the award. For no one is more deserving of it.”

— Robert Florida

Photos by Defining Studios in Hartford, CT