Synthesizing Birdsong with the RP2040

V. Hunter Adams (vha3@cornell.edu)

Introduction

In this laboratory exercise, you will synthesize one of the most common songs of the northern cardinal, a very pretty (and rather vocal) bird that's common in Ithaca. You can listen to this song in the first ten seconds of the recording linked here. Your goal will be to generate a synthesizer that is accurate enough to trick Cornell Ornithology's Merlin Bird ID App.


missing
Fig 1: A Northern Cardinal

Fig. 2 shows the spectrogram for the song that we'll be synthesizing (5730 students will also synthesize an additional song, described at the bottom of this webpage). Cardinals and many other songbirds produce almost pure frequency-modulated tones (i.e., they whistle!). As can be seen in the spectrogram below, the cardinal sweeps through frequencies from ~2kHz to ~7kHz. We'll assume that the dominant tones (the darkest lines on the spectrogram) are significantly louder than all other frequencies (the lighter lines). We'll only synthesize these loudest frequencies. The generated song sounds quite realistic under this assumption, as demonstrated in the embedded video farther down the webpage. Your synthesizer will be controlled via a keypad.


missing
Fig 2: Spectrogram for a cardinal song

There will be a number of interesting outcomes from this laboratory exercise. You'll gain familiarity with all of the key concepts listed below, but you may also notice that something strange occurs in your own brain! After gaining such an intimate understanding the cardinal's song, you may find that you notice birds singing after completing this lab! It's really nice when an engineering project changes the way that you see and appreciate the world around you, I hope that occurs for some of you.

If you'd like to take things farther, you can also synthesize songs for other, more challenging birds like the Baltimore oriole. (If you're lucky, you can spot these in Ithaca too. They're gorgeous and have a very distinctive song). If you're feeling really ambitious, do additive synthesis to generate a Bobolink (might make a good final project!).

Key concepts: Timers, timer interrupts, Direct Digital Synthesis, SPI communication, GPIO, threading/concurrency, digital-to-analog converters, fixed-point arithmetic, keypads, debouncing, curve fitting, numeric integration, code characterization

Demonstration

I've embedded a demo below of the system that you will build.

Required reading

Experience shows that students prefer these webpages short. For that reason, please find the reading and background materials on the webpages linked below. Please note that the information in these readings will be critical for completing the lab.

Course information

  • Policy page: This page describes the structure and format of the course, course expectations, and grading policies.
  • Course demo code: Familiarize yourself with the layout of this repository, and the examples that it contains.

Engineering background

  • Building demo code: This webpage describes how to download and build the course demo code on the laboratory computers. If you'd like to setup the build environment on your own machine, see here for Windows, here for Mac, and here for Linux.
  • C/C++ SDK architecture for RP2040: This webpage describes the organization of the C SDK.
  • Using the C SDK: This webpage provides a brief practical introduction to the C SDK by means of a simple example.
  • Audio examples of incremental complexity: In this lab, as in all labs, you are provided with demo code from which to start your assignment. This repository contains a sequence of example projects which incrementally build to the demo code from which you should start your assignment. The documentation for each of these examples is meant to be read in order.
  • Keypad: Your birdsong synthesizer will be cotrolled via a keypad. Please see this page for examples of and information about the keypad that we will be using, and clues about a debouncing state machine.
  • SPI Communication: The digital-to-analog converter that we are using is an SPI device. This webpage describes SPI communication. Understanding these concepts will make configuring the SPI channel on the RP2040 much easier.
  • RP2040 Datasheet: Sections on GPIO, SPI, Timer, and UART
  • C SDK hardware API: Same sections as above
  • DAC datasheet: Datasheet for the MCP4822 12-bit DAC that we'll be using in this lab.

Mathematical & algorithmic background

Weekly Checkpoints

Note that these checkpoints are cumulative. In week 2, for example, you must have also completed all of the requirements from week 1.

Week 0

  • Using this webpage as a guide, download and build the course demo code on a laboratory PC

Week 1

  • In this example code, direct digital synthesis is used to generate a pure sine wave "beep". Connect the DAC to the Pico per the #define statements at the top of the source code and the DAC datasheet, build the code, and load it onto the Pico. Please note that different manufacturers use different names for SPI signals! Here are the correct connections:

    3.3V(OUT) --> VDD
    GND --> VSS
    PIN_CS --> $\overline{\text{CS}}$
    PIN_SCK --> SCK
    PIN_MOSI --> SDI
    LDAC --> $\overline{\text{LDAC}}$

  • Confirm that the waveforms are being generated by the DAC with the oscilloscope, then use a 3.5 mm audio socket so that you can plug in speakers and listen to the generated beeps.

Week 2

  • Use the keypad webpage to connect the keypad to the Pico. Use the keypad example code to confirm that your connections are correct.
  • Integrate the keypad example code with the beep beep demo code from last week. Have the program produce a single beep each time you press a key (requires debouncing).
  • Modify the program such that it produces exactly one swoop sound each time you press the 1 key, and one chirp sound each time you press the 2 key. These sounds should be amplitude modulated so that there are no audio artifacts (clicks, pops, etc.).
  • The program should set a GPIO pin when it enters the synthesis ISR, and clear the GPIO pin when it leaves the synthesis ISR. Use the oscilloscope to measure the time that you are spending in the ISR. How can you make it faster? Why are some sounds faster than others?
  • Finishing a checkpoint does NOT mean you can leave lab early!

Week 3 (4760)

Write a C program that meets the following requirements . . .

  • The program has two modes: play mode and record mode. The system boots into play mode, in which it plays the swoop sound when the 1 key is pressed and the chirp sound when the 2 key is pressed. Each sound will play for 130ms, exactly once, independent of the duration of the button push. There should be no clicks, pops, or other audio artifacts.
  • By pressing a separate key on the keypad, or an external button, or toggling an external switch, the user places the device into record mode. In record mode, the device remembers the sequence of inputs that the user specifies. When the user transitions from record mode back to play mode, the device plays back the recorded sequence with the correct timing.
  • The program should set a GPIO pin when it enters the synthesis ISR, and clear the GPIO pin when it leaves the synthesis ISR. Use the oscilloscope to measure the time that you are spending in the ISR.

When you demonstrate the program to Hunter or a TA, you will be asked to play back a sequence of sounds which simulates the birdsong in Fig. 2. If you're able to trick the Merlin App, you will earn a handshake. You will also be asked to play back a random sequence of swoops/chirps/silences that Hunter or the TA uses the keypad to generate. At no time during the demo can you reset or reprogram the RP2040. Finishing a checkpoint does NOT mean you can leave lab early!

Week 3 (5730)

In addition to the above requirements for 4760, 5730 students should also . . .

  • Synthesize another Northern Cardinal song, shown in Fig. 3. You may deconstruct this song into 4 sound primitives, as illustrated in that figure's annotations. Fit these frequency sweeps to functions using similar analysis to that shown at the top of this document.
  • Each of these 4 sound primitives should be mapped to 4 additional keys on the keypad.
  • You may fit to any function that you like. 2 linear function and a parabolic or sine curve sound nice.

When you demonstrate the program to Hunter or a TA, you will be asked to play back a sequence of sounds which simulates the birdsongs in Fig. 2 and Fig. 3. If you're able to trick the Merlin App, you will earn a handshake. You will also be asked to play back a random sequence of swoops/chirps/silences that Hunter or the TA uses the keypad to generate. At no time during the demo can you reset or reprogram the RP2040. Finishing a checkpoint does NOT mean you can leave lab early!

missing
Fig 3: Spectrogram for an additional cardinal song (5730 students only)

Lab report

Your written lab report should include the sections mentioned in the policy page, and:

  • A scope display of a swoop/chirp birdcall showing rise, sustain, fall
  • A table which shows the time spent in the ISR for each of your sound primitives. Please comment on the differences among these times, any optimizations that you implemented to improve this speed, and any additional optimizations that you could have implemented with more time.
  • A spectrogram of the sound produced by your MCU. You can generate this anyway you like. You could plug the DAC output into the mic input of your laptop via an aux cable, and then generate a spectrogram in Python. You could alternatively use some software on the lab PC to generate a spectrogram (WaveForms). Or you can use the Merlin app.
  • A heavily commented listing of your code