Machine Learning at Macaulay Library

Powering an intelligent wildlife media archive.

Using millions of photos, audio recordings, and videos collected from citizen scientists around the world, we are building datasets and models to power tools like Merlin Bird ID. These datasets and models will power the future of engaging identification apps, scientific research, and biodiversity conservation.

Machine Learning visualization
Photo:
© Zhong Ying Koay
5 Mar 2017
Perak, Malaysia
Macaulay Library eBird Checklist

Machine Learning Blog

All articles
Allen's Hummingbird

From Sound to Images, Part 2: Spectrogram Image Processing.
By Benjamin Hoffman and Grant Van Horn
Picking up on where we left off in the previous post, we will now look at the various ways one can transform the spectrogram image prior to analysis by a convolutional neural network (CNN) and how these transformations affect model performance. Amplifying a hidden signal With the spectrogram image in hand, the next challenge is…
Read more From Sound to Images, Part 2: Spectrogram Image Processing.
Lesser Goldfinch

From Sound to Images, Part 1: A deep dive on spectrogram creation.
By Benjamin Hoffman and Grant Van Horn
In our first post, we described the idea of using a computer vision model to identify bird vocalizations. But how does a computer vision model “listen” to a sound? For Sound ID, we use the short-time Fourier transform (STFT) to convert the raw waveform (which tracks air pressure as a function of time) into an…
Read more From Sound to Images, Part 1: A deep dive on spectrogram creation.
Behind the Scenes of Sound ID in Merlin

Behind the Scenes of Sound ID in Merlin
By Benjamin Hoffman and Grant Van Horn
What is Sound ID? Today we announced one of our biggest breakthroughs—Sound ID, a new feature in the Merlin Bird ID app—and a major leap forward in sound identification and machine learning to date. Sound ID lets people use their phone to listen to the birds around them, and see live predictions of who’s singing. Currently,…
Read more Behind the Scenes of Sound ID in Merlin

Team

Grant Van Horn, PhD
Grant uses data in the Macaulay Library to prototype machine learning applications that can be utilized and deployed throughout the Cornell Lab of Ornithology. His research focuses on detection and identification of wildlife in images, audio, and video. His passion lies at the intersection of human-machine collaboration, where the collective strengths of humans and machines can be used to answer questions from data. Grant received his PhD from Caltech in 2018, advised by Dr. Pietro Perona.

Sam HeinrichSam Heinrich
Sam is an engineer who creates machine learning models including Photo and Sound ID from the vast Macaulay Library archives. His passion for birds began at a young age when his dad introduced him to birding, and has continued to grow since. He’s excited about using technology to help others learn about birds and to foster solutions to important conservation challenges.

Past team members

Benjamin Hoffman, PhD
Since receiving his PhD in mathematics from Cornell, Benjamin has focused on applications of machine learning to biology and conservation. On a given day, you might find him implementing custom layers for audio signal processing, or designing metrics for model robustness to environmental noise. He is excited to see how Merlin Sound ID can be used by conservation organizations, and how the techniques developed by the machine learning team may help answer biological questions.

Jess Sullivan
Jess is an avid birder and enthusiastic software developer. She is passionate about advancing the role of technology in conservation, and cherishes working alongside a wonderful and like-minded team!

 

Machine learning projects are run in close collaboration with staff from the Macaulay Library and eBird.

Datasets

NABirds

Partners

Visipedia