DSI Postdoc Emily Spratt Bridges the Divide Between Humanists and Technologists and Uses Machine Learning to Analyze Art

Emily L. Spratt is ardently curious and posseses an endless capacity to learn new things. She has studied a dozen languages and lived all over Europe, the Mediterranean, and even East Africa researching art and culture and did her doctoral studies in Byzantine and Renaissance art history at Princeton. Alongside her graduate work, she separately developed expertise in the applications of machine learning for the analysis of art and visual culture at large.

And now, as a Postdoctoral Research Fellow at the Data Science Institute, Spratt is calling upon her unique combination of skills to create a visionary research initiative that unites the arts and sciences. She is demonstrating how machine learning techniques can be applied as a methodology for the analysis of art and cultural heritage. In essence, Spratt is fusing the fields of art history, historic preservation, and computer vision science, yet most significantly she’s demonstrating the philosophical and ethical consequences of data science-oriented approaches to the construction of knowledge.

There’s often a divide between humanists and technologists, and few art historians take a deep interest in emerging technologies such as machine learning. Not Spratt. In her article “Dream Formulations and Deep Neural Networks: Humanistic Themes in the Iconology of the Machine-Learned Image,” she emphasizes the relevance of bringing a humanities framework to how we use AI for images. When asked about the difficulties involved in working in such disparate fields, she said:

“I don’t think about whether things are easy or hard, for me it’s just a matter of what questions need to be answered and what’s the best way to go about solving problems. I enjoy being a pioneer in emerging or neglected areas of research, as evidenced by my dissertation on the legacy of Byzantium in the early modern period, which required extensive field work in remote places. In the same vein, my work on art and AI is exploring a fundamentally new type of visual engagement with the world that is just beginning to be traversed.”

Machine Learning and Art

For her postdoctoral research project, she’s working on a book project called “The Ontology of Sight in the Age of AI,” which weaves together fields that are usually mutually exclusive: art, AI, and ethics and philosophy. At the core of this project, she is examining how machine learning is fundamentally changing society’s interactions with images. She’s also working with computer vision scientists to develop tools to enhance the study of art history. As a postdoctoral fellow she has two principle collaborators, one of whom ‒ Assistant Professor of Computer Science Carl Vondrick– specializes in computer vision and machine learning. They’ll work together to create predictive models of a massive dataset of images that Spratt collected as part of her research on Byzantine and Post-Byzantine icons. Jorge Otero-Pailos, her other collaborator, is an architect and director of the Historic Preservation at Columbia’s Graduate School of Architecture, Planning and Preservation. They intend to facilitate the use of machine learning to help preservationists track, study and conserve historic artifacts.

One of Spratt’s research goals is to demonstrate how developments in computer vision can be merged with traditional approaches to the study of art. In a paper titled “Computational Beauty: Aesthetic Judgment at the Intersection of Art and Science,” she discussed how 18th-century philosophies of aesthetics still influence contemporary theories on visual perception in the humanities. But she maintains that those theories, given advances in computer vision and neuroscience relating to perception of the visual world, require a major re-evaluation.

Spratt became interested in machine learning while working on her dissertation. Her article, “Toward a Definition of ‘Post-Byzantine’ Art: The Angleton Collection at the Princeton University Art Museum,” exemplifies the methodological issues inherent in her thesis subject.  For her research she conducted much fieldwork, traveling throughout the Eastern Mediterranean and the Balkans taking photographs of art and architecture. She assembled the images, many of them Byzantine icons, into a large database. Since icons are generally created in emulation of their prototypes, she says, one would assume that tracking their dissemination patterns would be easy to do, but this is not the case. Since her database encompasses many sets of visually similar images, Spratt recognized that making comparisons within the digital compendium of visual information was essential yet the tools were not there to easily visualize the patterns or to effectively go through the images. The question for her became how to sort, organize and analyze the multitude of images both with the discriminating eye of an art historian and with the aid of an automatic processing system. She therefore sought out answers on how to develop tools to help her, but was chagrined to learn how inadequate the resources were for managing large datasets of images. And that’s where machine learning came into play.

“When I learned about the burgeoning field of machine learning,” she says, “I immediately saw the potential for the development of these types of tools to transform not only art history but the way society at large engages with images. I saw that I could use machine-learning to visually analyze and organize my images and identify patterns of icon dissemination. More broadly, I realized AI would radically change the world we live in and would have a major effect on virtually every industry.” Prophetically, she made this observation back in 2012, when she boldly pivoted her research interests to also fully embrace AI developments.

Spratt has also been working on a new area of AI research called Generative Adversarial Networks (GANs), a class of machine-learning systems that can actually create new images. Thomas Fan, a software engineering at DSI and a core developer for scikit-learn, is one of her collaborators on this project at Columbia. Previously, Spratt curated “Unhuman: Art in the Age of AI,” which explored the use of a type of GANs system to make art in the first exhibition of its kind with deep-learning techniques, which took place in Los Angeles and Frankfurt.

The Ethics of Image Use, Accessibility, and Generation

Spratt also is an active participant in the area of ethics and AI. Recently, for instance, news reports have focused on the potential use of different subsets of AI such as GANs for unethical purposes – the ability say to create fake photos or videos of people ‒and Spratt is bringing the critical emphasis that this area of research necessitates for the ethical use and production of images.

“I am focused on ethical practices with image accessibility, use, and generation” she says, “and also making clear the claims of what ML [machine-learning] and GANs systems can and cannot do.”

In this vein, Spratt is working with several different foundations including The International Council on Monuments and Sites (ICOMOS) to develop guidelines for ethical image practices that will set policies in the cultural heritage management sector. She is also the ethics adviser for the Paris-based cultural foundation for the digital recording of monuments called Iconem. In the US, she is the ethics adviser for the AI Finance Institute of New York City. Additionally, she consults for The Frick Collection and Art Reference Library on the use of the newest technologies for art history. In the business world, she keeps the pulse of the vibrant AI startup community and actively sources and evaluates machine learning related companies for the Manhattan chapter of the Princeton Alumni Angels investor group. Spratt was also the strategic adviser for the art market company Artory, which she helped pivot into a blockchain-based company that catalogues art and object records through sales transactions, and has provided consulting services for many startup companies in the art-tech sector.

Spratt is both a polymath and a polyglot. She has, for instance, studied Modern Greek, French, Italian, Spanish, German, Czech, Russian, Kiswahili, and Ancient Greek, Latin, and Old Church Slavonic. Though born in New York City, she’s a world citizen who has lived in Greece, Italy, Germany, Cyprus, Bulgaria, and East Africa. Back when she was an undergraduate, she spent a year hiking the Sierra Nevada and sea kayaking in Baja, Mexico, and participated in expeditions all over the Serengeti in Tanzania, and in Kenya. In California and East Africa, she learned mountaineering, orienteering, rock climbing, and wilderness medicine. At one point she loved nature and the outdoors so much that she considered devoting her life to the environment. But sometime during her time in the hills of Africa she had an epiphany: She realized she was more interested in history and the cultures that surrounded her than the natural beauty that enveloped her.

“Although I deeply enjoyed hiking the mountains, forests, plains, and deserts that I traversed on those expeditions,” she says, “I preferred learning about the cultures we encountered along the way. Trying to learn Kiswahili and Maa was just as fascinating to me as retrieving glacier-core samples from the top of Mount Kenya.”

From Byzantium to the Renaissance to the New World of AI

So at Cornell, where she earned her B.A., she studied religious studies, the history of art, and psychology. She then completed an M.A. in Byzantine Art History at the University of California, Los Angeles, and an M.A. in Renaissance and Baroque Art History at Princeton, where she is about to defend her dissertation, titled “Byzantium Not Forgotten: Constructing the Artistic and Cultural Legacy of an Empire between East and West in the Early Modern Period.”

Spratt has also taught in the Department of Art History and the Program in Cultural Heritage and Preservation Studies at Rutgers. Before beginning at Princeton, she was the assistant to the director of the 21^st Byzantine Ephorate, which is the representing body of the Hellenic Ministry of Culture in Corfu, Greece. This was at the crucial time when Corfu Town was gaining recognition to become a UNESCO world heritage site.  Additionally, for her work at the Byzantine and Christian Museum of Athens, Spratt received a notable award from the Hellenic Ministry of Culture.

An authority on art, world heritage, and artificial intelligence, Spratt was the honorary guest editor for the magazine edition on computers and art published by the Association of Computing Machinery. She was also a fellow at The Frick Collection, where with her colleagues she organized a symposium titled “Searching Through Seeing: Optimizing Computer Vision Technology for the Arts,” for which she gave the keynote address.

Outside academia, she is a photographer, a songwriter, an environmentalist, and an animal rights advocate. Now, once again living in Manhattan, you might find her in Central Park walking her bichon frise named Pumba. She travels widely for her work and recently returned, for example, from the annual Wiki Conference in Stockholm and the International Committee of Architectural Photogrammetry (CIPA) conference in Avila, Spain. Nonetheless, she is always happy to come back to New York — and is delighted to be a Postdoctoral Research Fellow at Columbia’s Data Science Institute.

“I appreciate being in the vibrant community of the DSI and look forward to learning about the many exciting projects underway in the program,” she says. “I think New York City is an essential hub for AI development, both in academia and industry, and it’s necessary to have one foot in both worlds as the tempo and reach of tech innovation requires the flexibility to traverse multiple fields and sectors without hesitation, like a fast-tracked Harlem lindy hop set to a Big Band tune.”

— Robert Florida

Photos by John Palisay