How the Merlin bird ID will evolve — and what it means for you

By Anders and Beverly Gyllenhaal

Second of two parts

Ian Owens noticed something odd while strolling the birding trails around the Cornell Lab of Ornithology in Upstate New York where he’s director. Some of the birders armed with the Merlin app that instantly identifies birdsongs spend more time staring down at their phones than looking up at the birds they’ve come to see.

The discovery bothered him as he thought about the role of this innovative service that’s meant to enhance the connection between people and birds. “The idea is that we should be absorbing nature,’’ Owens told us. “But are we getting in the way of it?’’

By every measure, the Merlin app has been a resounding success, gaining wide acceptance across the birding landscape with more than 15 million downloads at a pace that’s only accelerating. But the lab’s staff hopes this free tool will do more than spit out a list of species on birders’ smartphones.

They see Merlin as a guide for helping people tune in to birds and their songs. They hope the lab’s apps will serve to deputize birders in the valuable work of gathering data. And they’d like to think the experience will lead people to a heightened conservation ethic in ways that end up helping protect birds.

Each of these possibilities has its own set of challenges. “So we’re doing quite a lot of work behind the scenes,’’ said Owens, “trying to figure out what people ultimately want.’’

The first of our two posts on the Merlin project earlier this month explored how the app is reshaping the practice of birding. It has become as commonplace as binoculars for many, helped recruit millions of newcomers and is spreading a pastime long dominated by older people to more and more younger ones.

This post looks at how the Cornell Lab hopes to develop what has the potential to be a pivotal and dynamic force. Between Merlin and the companion eBird app that helps birders track their sightings, the Cornell Lab has already created the world’s most successful citizen science initiatives that enable Cornell to gather data on birds from around the world.

The question is how to make the most of these tools – for birders, for the lab and for birds.

How it works

In the coming weeks, when the Cornell Lab releases a new design for the Merlin app (currently being tested), the fresh look will emphasize simplicity. The cover page features three buttons that let people choose whether to use sound identification, photo ID or walk through a series of questions to zero in on the species they’re seeing. (Here’s a link to the lab’s Merlin user guide.)

The inner workings of Merlin, though, are anything but simple. Behind those three buttons sits a massive archive of data that uses machine learning to identify which of the more than 11,000 species is singing within earshot.

Each of those three buttons takes a different route to zero in on the bird species.

Sound ID is the most complex. When the app records a birdsong, it converts the sound into graphic form that looks like a smudge of shadows called a spectrograph on your phone. With the help of lightning-fast algorithms, Merlin then searches for a match in the lab’s collection of audio graphics.

Here’s where the challenge comes: Birds can have dozens and a few even hundreds of different songs, calls and chips. So the library needs as many versions of those sound graphics as it can get – good quality as well as poor quality recordings to match the vast variety of sounds it will pick up in the field.

“We need really good sound recordings and we also need cell phone recordings because you’re using Merlin on your cellphone,’’ said Alli Smith, the Merlin project coordinator. “If we only train Merlin on high quality recordings, it won’t be able to identify the low-quality recording you get on your phone.’’

The photo identification works in a similar way. In order to match photo of birds to their names, each species needs an archive of more than a hundred different photos, some sharp, some fuzzy, taken from every angle of the bird.

The half dozen Cornell Lab staffers devoted to Merlin spend their days perfecting this matching system, adding to the archives and creating the thousands of photo and audio clips. “It’s tedious work,’’ Smith said, especially the efforts to create the key graphic elements that capture a bird’s song.

Here’s a look at how staffers carve out the graphic segments that are used to identify birds.

The third button on Merlin asks users to answer a series of five questions about a bird’s size, color, location and whether it’s flying, foraging, on the ground or in a tree. With the help of the lab’s extensive data on what birds do and where, the answers help narrow down candidates to the likely bird.

Flow of Data

So the key to Merlin is the ability to collect millions of sound and audio files, which largely come from people using eBird or those willing to send photos and audio clips to Cornell. This means that the very people served by Merlin are critical to its success.

 “We couldn’t do this without the support of birders,’’ said Smith, the project coordinator. “It’s all built on the work of birders around the world.’’

The flow of this data has been steadily increasing with the growth of the Cornell Lab’s apps: Users have submitted more than 50 million images and just over 2 million audio files to date. Drew Weber, the Merlin project manager, has done some calculations about what these reams of data amount to. “One interesting stat is that in 2023, Merlin processed approximately a thousand years of audio,’’ he said.

The larger the archive, the more accurate the system becomes. The photo identification is correct 92 percent of the time, the lab has calculated. Sound ID accuracy varies with percentages close to that.

How much potential reach can Merlin expect in the future?  During busy stretches during fall and spring migration, up to 5 million people a month will be using Merlin. That’s just a fraction of the nearly 100 million people the U.S. Fish and Wildlife Service estimates watch birds in the U.S. — so Weber hopes to reach most of those.

 “That seems like a good target,’’ he said.

Although Merlin’s highest usage is in the U.S., it’s quickly spreading to the rest of North America, Europe, India and other parts of the world. This growth will require an enormous expansion of those archives in order to cover the globe’s species of well over 10 times the number of birds found in North America.

Both eBird and Merlin are gaining acceptance in many other countries, but it’s Merlin with its sound ID that is leading the way. Owens uses the example of Latin America with such vast numbers of species (about a third of the globe’s total number) that at first the lab thought it would take a long time to gather the data. But the participation among partners working to support the archives has surprised even the lab. “Our guess was it was going to be hard to do every species in Latin America,” he said. “But now we think we’ll be able to do it.”

Here’s a map that shows the spread of eBird usage across the world, which gives a sense of where the lab’s reach is greatest and where there’s room to grow:

Shaping the app

Owens sees plenty of ways to make sure people don’t stare so much at their apps that they miss seeing the birds themselves. Along with identifying the species, the future app may give users clues on nearby birds they have yet to see, enable users to share their bird sightings with friends, and also encourage people to support conservation.

Since eBird keeps a life list of the species for birders, Cornell will eventually be capable of letting people know when there’s a bird nearby you haven’t yet seen. “We’re working on that now,’’ said Owens. “We might say, there’s a Blue-winged Warbler singing that you might want to hear.’’

“People often ask about that: What do I do now that your app has made me love birds.’’

-Cornell Lab director Ian Owens

The app could also add social elements, enabling people to share the news when they see a bird they’re especially proud of. “I don’t think they want us to create a new TikTok,’’ he said. “But they do want to be able to say, ‘Hey, I saw this sparrow today.’ So it might allow some limited sharing.’’

Most importantly, Owens said, will be the ability for Merlin and eBird – or perhaps eventually some combination of the two – to encourage people to support bird conservation. That could mean everything from taking steps around the house to safeguard species to joining local groups working on behalf of birds.

“That’s one of the most important things we’re thinking of, how to connect people with some sort of action,’’ said Owens. “People often ask about that: What do I do now that your app has made me love birds.’’

There’s no shortage of ways to contribute (click here for our section on what you can do to help birds.) “The trick is going to be, allowing all these sorts of things to happen without Merlin becoming this huge lumbering app.’’