Modern software systems don’t simply break—they unravel. When failure cuts across layers of code, infrastructure, and telemetry, failure doesn’t point to a single cause. Diagnosing what went wrong—and why—is one of the most expensive statistical challenges facing industry today.
Anish Agarwal, a Professor of Industrial Engineering and Operations Research at Columbia Engineering and a member of the Data Science Institute, is treating it as both a research problem and a startup opportunity. His company, Traversal, uses causal machine learning to pinpoint failure in large-scale software systems, applying the same analytical techniques he studies as a faculty member. Agarwal’s work sits squarely at the intersection of theory, engineering, and application, an increasing focus of the Data Science Institute.
“This problem, troubleshooting large-scale software systems, is the most complicated statistical workflow I’ve ever worked on,” says Anish Agarwal. “It’s petabytes of data. It’s hundreds of billions of dollars in direct costs to businesses. It’s real-world messiness. That’s where data scientists should be working.”
Co-founded with Columbia Engineering alum Ahmed Lone and staffed by several Columbia graduates, the company is now backed by legendary venture firms like Sequoia and Kleiner Perkins and already shortening incident response times at major companies like DigitalOcean, Eventbrite and a number of Fortune 100 companies, sometimes reducing hours of manual debugging to just minutes. But for Agarwal, Traversal also proves a larger point.
“There’s a lot of talk about academia versus industry. To me, they’re not opposites,” he says. “I think there’s a history of professors successfully bridging that gap. Startups are filled with uncertainty. So is research. To do both, you have to be comfortable in the unknown and find creative ways of creating a solution starting with very little.”
This perspective is shared across the Data Science Institute, which is Columbia’s central hub of data science as well as the home of Columbia AI, a university-wide initiative advancing interdisciplinary research and innovation in artificial intelligence.
“The true measure of our impact isn’t in new ideas alone, it’s also what our faculty do with them,” says Garud Iyengar, Avanessians Director of the Data Science Institute and Professor of Industrial Engineering and Operations Research at Columbia Engineering. “Anish and his colleagues at Traversal demonstrate how deep technical work can lead to real systems, real companies, and real change. That’s the kind of full-spectrum impact the Data Science Institute supports.”
Agarwal is one of a growing number of Columbia faculty exploring how to work more fluently across disciplines and sectors, publishing serious research while also building products that test it under real constraints. His background is academic: MIT PhD, causal inference researcher, member of Columbia’s Industrial Engineering and Operations Research and Computer Science departments. But his motivation is also deeply applied. Traversal didn’t emerge from a paper, but came from the same intellectual toolkit, applied to a problem that industry has been slow to solve.
Faculty like Agarwal show how Columbia’s research culture can extend beyond the lab into systems, startups, and applied breakthroughs that still rest on deep technical foundations. Traversal is one example of that. Others are coming.
Read more about Agarwal’s story at Columbia Engineering.