CMA-ES: finding didgeridoo shapes with fewer simulations

DidgeLab designs didgeridoos by search. You describe what you want to hear, for example a drone on D and toots on F#, D and G. DidgeLab then tries thousands of bore shapes, simulates the sound of each one and keeps the ones that come closest. Every try costs one acoustical simulation, so the fewer tries an optimizer needs, the faster you get a good didgeridoo.

Until now DidgeLab had one optimizer, a genetic algorithm called Nuevolution. It now has a second one, CMA-ES. In our benchmark CMA-ES found the better didgeridoo in two thirds of all runs, and it reached the result of the genetic algorithm with 1.5 to 4.5 times fewer simulations.

How a shape is searched

In DidgeLab a bore shape is described by a list of numbers between 0 and 1, the genome. For the default Kigali shape these are 49 numbers: the length, the bell diameter, how the bore widens and small bumps along the way. A loss function says how far the sound of a shape is from what you want. The optimizer looks for the genome with the smallest loss.

The genetic algorithm works like breeding. It keeps a population of good shapes and creates new ones from them: it changes some numbers at random (mutation) or mixes the numbers of two parents (crossover). The best new shapes stay, the others are dropped. This works, but it is blind. The mutations always have the same size, and the algorithm never learns which changes tend to help.

What CMA-ES does differently

CMA-ES (Covariance Matrix Adaptation Evolution Strategy) was introduced by Nikolaus Hansen and Andreas Ostermeier in Completely Derandomized Self-Adaptation in Evolution Strategies (Evolutionary Computation, 2001) and is one of the most widely used methods for optimization problems without gradients. A readable introduction is Hansen’s tutorial The CMA Evolution Strategy: A Tutorial; the Wikipedia article gives a shorter overview.

CMA-ES does not keep a population of individual shapes. It keeps a cloud of possible shapes, a probability distribution around the shape it currently thinks is best. Every generation it does three things:

  1. It draws a handful of shapes from the cloud and simulates them.
  2. It moves the centre of the cloud towards the best of them.
  3. It changes the size and the form of the cloud. If successful steps keep pointing in the same direction, the cloud grows and stretches in that direction. If they start to jump back and forth, the cloud is close to the goal and shrinks.

CMA-ES on two genes: the cloud moves, stretches along the valley and narrows in

The picture shows this for a made-up problem with only two genes. The darker the blue, the lower the loss. The good shapes lie in a narrow diagonal valley: the two genes only give a good result if they change together. A genetic algorithm that changes one gene at a time keeps falling out of such a valley. CMA-ES learns the direction of the valley and stretches its cloud along it (middle). Near the goal it contracts and lands exactly on the best point (right).

Didgeridoos are full of such valleys. If you make the tube longer, all resonances go down. To keep the drone in tune, the bore has to become narrower or the bell wider at the same time. Hitting four resonances at once needs many genes to change in a coordinated way. This is the kind of problem CMA-ES was made for.

The experiment

We wrote a benchmark notebook that lets both optimizers solve the same tasks: optimizer_benchmark.ipynb. It combines

Every optimizer gets exactly 3000 simulations per run, and every combination runs with three different random seeds. That makes 144 runs and about 430,000 simulations. The notebook counts the simulations itself, so no optimizer gets a hidden advantage.

Results

CMA-ES had the best result in 33 of 48 comparisons. The small genetic algorithm won 9, the configuration of the web app 6.

Best loss against the number of simulations, for all targets (rows) and shapes (columns)

Each panel shows how the best loss found so far goes down with the number of simulations. Red is CMA-ES, blue and green are the two genetic algorithms. The line is the median over the three seeds, the shaded band the range. Lower is better, and the scale is logarithmic.

The second way to read the plots is from right to left: how many simulations does CMA-ES need to reach the final result of the better genetic algorithm? Wherever it gets there, it needs 650 to 1950 instead of 3000 simulations, that is 1.5 to 4.5 times fewer.

Two more observations:

The setting of the web app (50/400) did worst. With 3000 simulations it only gets through 60 generations, and managing 400 shapes per generation also costs time: 15.5 seconds per run, compared with 3.5 seconds for CMA-ES with the same number of simulations.

What the difference sounds like

The next picture shows the best bore of each optimizer for the task drone and toots on the Kigali shape, with its impedance spectrum. The dashed lines are the four target frequencies.

The best bores of the three optimizers and their impedance spectra

All three bores look similar and all three hit the targets roughly. The difference is in the details: added up over the four peaks (with the drone counted twice), the CMA-ES bore is off by about 1 cent, the bores of the genetic algorithms by about 5 cent.

Faster simulations, too

The benchmark profits from a second improvement. Each loss evaluation in DidgeLab has become about six times faster. The simulation now looks for the peaks on a coarse frequency grid and then measures each peak precisely, instead of simulating the whole spectrum finely, and the inner loop of the simulation computes fewer expensive math functions. On top of that the peak frequencies are now accurate to about 0.1 cent, where the old grid was off by up to 2.5 cent. Together with CMA-ES, reaching the same result now takes roughly 3 to 10 times less computing time than before.

How to use it

CMA-ES can be used wherever you used Nuevolution before:

import numpy as np
from didgelab import CMAES, CompositeTairuaLoss, FrequencyTuningLoss, KigaliShape

loss = CompositeTairuaLoss()
targets = np.array([73.4, 185.0, 293.7, 392.0])  # D2, F#3, D4, G4
loss.add_component("freq", FrequencyTuningLoss(np.log2(targets), np.full(4, -1.0), [2.0, 1.0, 1.0, 0.5]))

evo = CMAES(loss, KigaliShape(), num_generations=200, population_size=16, sigma0=0.3)
population = evo.evolve()
best_geo = population[0].genome2geo()

population_size is the number of shapes per generation, and sigma0 the initial size of the cloud. 16 and 0.3 are good defaults when you start from scratch. To improve a didgeridoo you already have, start from its genome with start_from_father=True and a small cloud, for example sigma0=0.05.

The web version, DidgeLab online, still uses the genetic algorithm. Given these results, switching it to CMA-ES is the obvious next step.

If you want to dig deeper or run the benchmark with your own targets and shapes, open the benchmark notebook. The mathematical details of CMA-ES are explained in Nikolaus Hansen’s tutorial The CMA Evolution Strategy.