PESTO: Real‑Time Pitch Estimation with Self‑Supervised Transposition‑Equivariant Objective

2.1 Pitch estimation

Monophonic pitch estimation has been a subject of interest for over 50 years (Noll, 1967). The earlier methods typically obtain a pitch curve by processing a candidate‑generating function such as cepstrum (Noll, 1967), autocorrelation function (Dubnowski et al., 1976), or average magnitude difference function (Ross et al., 1974). Other functions, such as the normalized cross‑correlation function (Boersma, 1993; Talkin, 1995) and the cumulative mean normalized difference function (de Cheveigné and Kawahara, 2002; Mauch and Dixon, 2014), have also been proposed. On the other hand, Camacho and Harris (2008) perform pitch estimation by predicting the pitch of the sawtooth waveform whose spectrum best matches the one of the input signal.

As in many other domains, these signal processing–based approaches have been recently outperformed by data‑driven ones. In particular, CREPE (Kim et al., 2018) is a deep neural network consisting of six convolutional blocks followed by a fully‑connected layer. Operating directly on waveform data, CREPE processes audio chunks of 64 ms (1024 samples at 16 kHz). Trained in a supervised manner on a large collection of music datasets, it achieves exceptional performance and has become a widely adopted solution for monophonic pitch estimation.

Building on CREPE, Singh et al. (2021) introduce dilated convolutions with residual connections to expand the receptive field, while Ardaillon and Roebel (2019) improve the architecture further and propose downsampling input signals to 8 kHz to reduce computational costs. Morrison et al. (2023) later proposed additional training strategies, such as incorporating layer normalization, increasing batch sizes, and making minor architectural changes to enhance performance.

These advancements have led to state‑of‑the‑art results using both music and speech datasets. However, the fundamental framework—modeling pitch estimation as a supervised classification problem—remains unchanged. This dependence on large quantities of annotated data is a significant drawback, as obtaining precise F0 annotations is time‑consuming and challenging. Furthermore, these methods often perform poorly when applied to data outside the training distribution (Morrison et al., 2023), limiting their applicability in diverse scenarios.

2.2 Self‑Supervised Learning with siamese networks

Self‑Supervised Learning (SSL) has emerged as a promising paradigm to address the challenges associated with gathering large quantities of annotated data, which is often tedious, error‑prone, and biased. SSL leverages the data itself to provide a supervision signal by training a neural network to solve a pretext task. One common pretext task involves training Siamese networks (Hadsell et al., 2006) to project data points into a latent space and minimize the distance between pairs of inputs that share semantic information. These positive samples (or views) can be artificially created by applying transforms to an input data point.

In other words, the goal of Siamese networks is to learn a mapping $f:\mathcal{X}\to \mathcal{Y}$ that is invariant to a set of transforms $\mathcal{T}:\mathcal{X}\to \mathcal{X}$; that is, for any data point $\mathbf{x}\in \mathcal{X}$ and any transform $t\in \mathcal{T}$, the following holds:

The set of transforms $\mathcal{T}$ is usually composed of semantic‑preserving data augmentations. For example, in the image domain, Chen et al. (2020) propose using transforms such as cropping, rotation, and color jittering, which drastically change the pixel values but preserve the content of the image itself. Later, Baevski et al. (2022) propose creating views by masking the input.

However, without additional constraints, the network might simply learn the trivial solution to Eq. (1), mapping all inputs to the same point. This phenomenon, called representation collapse, can be typically prevented by adding new loss terms computed over several data points. Successful approaches, typically requiring large batch sizes, include minimizing the similarity between negative samples through a contrastive loss (Chen et al., 2020) or regularizing over the batch statistics (Bardes et al., 2022; Wang and Isola, 2020; Zbontar et al., 2021).

Introducing asymmetry in the Siamese architecture can also be exploited to prevent collapse by changing the training dynamics, such as adding a predictor network after one branch while stopping gradients in the other (Chen and He, 2021; Grill et al., 2020). This approach relies solely on positive pairs and is less sensitive to batch size.

2.3 Equivariant Self‑Supervised Learning

Most SSL models aim to learn a mapping that is invariant to certain transformations. However, recent research has explored the idea of learning equivariant mappings instead.

Equivariance, in contrast to invariance, can be mathematically expressed as follows. Let $\mathcal{G}$ be a group, with group actions $t:\mathcal{X}\times \mathcal{G}\to \mathcal{X}$ and $t^{\prime }:\mathcal{Y}\times \mathcal{G}\to \mathcal{Y}$. A mapping $f:\mathcal{X}\to \mathcal{Y}$ is said to be equivariant with respect to $\mathcal{G}$ if, for any $x\in \mathcal{X}$ and $g\in \mathcal{G}$, the following condition holds:

In other words, $t$ and $t^{\prime }$ are transforms acting on the input and output spaces, respectively, with parameters from the group $\mathcal{G}$, and a mapping $f$ is equivariant with regard to $\mathcal{G}$ if transforming the input $\mathbf{x}$ by $t$ results in a corresponding transformation of the output $f(\mathbf{x})$ under $t^{\prime }$.

Note that invariance is a special case of equivariance, where $t^{\prime }(\cdot ,g)$ acts as the identity function for all $g\in \mathcal{G}$. However, in general, Eq. (2) does not have a trivial solution for $f$. As a result, Siamese networks trained to optimize an equivariance objective are not prone to collapse.

Equivariant representation learning has been explored as a means to navigate the latent space of (variational) autoencoders (Falorsi et al., 2018; Hinton et al., 2011). More recently, efforts have been made to incorporate equivariance into self‑supervised models, conditioning parts of the neural architecture on the parameters of the transforms applied (Dangovski et al., 2021; Devillers and Lefort, 2023). These approaches have shown particular promise in computer vision tasks that involve rotations (Garrido et al., 2023; Winter et al., 2022).

2.4 Self‑supervised learning for MIR

While originally developed for image representation learning, the aforementioned approaches have been successfully adapted to the audio domain (Anton et al., 2023; Niizumi et al., 2022; Saeed et al., 2021). Notably, contrastive pretraining has demonstrated impressive results for various MIR downstream tasks, such as auto‑tagging and genre classification (McCallum et al., 2022; Spijkervet and Burgoyne, 2021).

Few studies have attempted to train neural networks with equivariant properties to pitch and tempo shifts. SPICE (Gfeller et al., 2020) offers a method for pitch estimation without the need for annotated data. It creates pairs of views by pitch‑shifting an input CQT frame by a known number of semitones, $k_1$ and $k_2$. The model learns a mapping $f$ that projects a CQT frame to a scalar value, and it is trained to ensure that the difference between the scalar projections is proportional to the pitch difference, $k_2-k_1$. However, Gfeller et al. (2020) observe that this objective alone is insufficient for training, and thus the model requires an additional decoder and a reconstruction loss to prevent collapse. This approach has been extended to tempo (Gagneré et al., 2024; Morais et al., 2023) and key estimation (Kong et al., 2024).

Quinton (2022) applies a comparable approach to tempo estimation. In this case, an audio segment is time‑stretched by two factors, ${\alpha }_1$ and ${\alpha }_2$, to create a pair of segments $(x_1,x_2)$. A neural network then projects these segments to scalars $z_1$ and $z_2$, and the model is trained to minimize the following loss function:

This objective encourages the scalar projections to be proportional to the actual tempo of the song. Unlike SPICE, Quinton (2022) does not require a decoder, and the model avoids collapse by using only the scalar projection loss.

Another family of self‑supervised and unsupervised methods for pitch estimation rely on analysis‑by‑synthesis. Engel et al. (2020a) train a neural network to predict the parameters of a differentiable harmonic‑plus‑noise synthesizer that approximates the input audio, relying on parameter regression on synthetically generated signals. On a similar approach, Torres et al. (2024) suggest instead using an optimal transport‑inspired spectral loss function for training a fundamental frequency estimator for harmonic signals.

3.1 Frontend

In this section, we first describe the CQT, which serves as a specification for the VQT used in our experiments. We then introduce the VQT, which is a variant of the CQT, which smoothly decreases the Q factors of the analysis filters for low frequencies.

3.1.2 The variable‑Q transform

The VQT (Schörkhuber et al., 2014) differs from the CQT by adding a parameter $\gamma$, which smoothly decreases the Q factors of the analysis filters for low frequencies. For center frequency $f_k$, the length of the analysis window is given by

5

$w_k=\lceil \frac{Q{f}_s}{f_k+\frac{\gamma }{\zeta }}\rceil ,$

where $Q$ is the Q factor of the analysis filters; $f_s$ is the sampling rate; and $\zeta =2^{\frac{1}{12B}}-1$ is a constant that depends on $B$, the number of bins per semitone, following the implementations of Cheuk et al. (2020) and Schörkhuber et al. (2014). For $\gamma =0$, the VQT is equivalent to the CQT. As in the CQT, the center frequencies ${\{f_k\}}_{k=1}^K$ follow an exponential scaling. Figure 1 illustrates the analysis window size as a function of $\gamma$ and center frequency.

Figure 1

Since its time–domain analysis window for low frequencies is smaller than that of the CQT, the VQT can be computed more efficiently in terms of both time and memory. As we demonstrate empirically, smaller analysis windows that do not span several frames are also beneficial for the pitch estimation task.

3.1.3 Pitch‑shift in the VQT domain

Our technique to build pitch‑shifted views from a single VQT frame, originally proposed for the CQT by Gfeller et al. (2020), is depicted in Figure 2.

Figure 2

More precisely, from a full VQT frame $(x_1,\dots ,x_F)$, an integer $k_{\text{max}}\le F/2$, let $F^{\prime }=F-2k_{\text{max}}$. Then, given an integer $k$ sampled from $\left[-k_{\text{max}},k_{\text{max}}\right]$, one can easily compute frames with approximately the same timbre and a pitch shift of $k$ bins, i.e. $k/B$ semitones, by extracting

$\mathbf{x}=\left(x_{k_{\text{max}}},\dots ,x_{F-k_{\text{max}}}\right)\in {\mathbb{R}}^{F^{\prime }}$

and

${\mathbf{x}}^{(k)}=\left(x_{k_{\text{max}}+k},\dots ,x_{F-k_{\text{max}}+k}\right)\in {\mathbb{R}}^{F^{\prime }}$

from the original VQT frame $\mathbf{x}$. We set $k_{\text{max}}$ to $16$.

3.2 Training objective

The training procedure for our model takes inspiration from previous SSL methods based on Siamese networks. It is depicted in Figure 3.

Figure 3

Given a single VQT frame $\mathbf{x}$, we first crop the frame with a known offset and then translate it by a random but known number of bins $k$. Let ${\mathbf{x}}^{(k)}$ be the resulting pitch‑shifted frame. Then, we randomly apply a set of pitch‑preserving transformations to both $\mathbf{x}$ and ${\mathbf{x}}^{(k)}$ (white noise, random gain). We denote the transformed versions as $\tilde{\mathbf{x}}$ and ${\tilde{\mathbf{x}}}^{(k)}$, respectively. Next, we pass $\mathbf{x}$, $\tilde{\mathbf{x}}$, and ${\tilde{\mathbf{x}}}^{(k)}$ through the same neural network $f_{\theta }:{\mathbb{R}}^{F^{\prime }}\to {[0,1]}^K$, obtaining the pitch distributions $\mathbf{y}$, $\tilde{\mathbf{y}}$, and ${\tilde{\mathbf{y}}}^{(k)}$, respectively.

We know by construction that $\mathbf{x}$ and $\tilde{\mathbf{x}}$ have the same pitch. Therefore, we want their pitch distributions $\mathbf{y}$ and $\tilde{\mathbf{y}}$ to be equal, ensuring that $f_{\theta }$ is invariant to the transformations. Similarly, we know that the pitch difference between $\tilde{\mathbf{x}}$ and ${\tilde{\mathbf{x}}}^{(k)}$ is exactly $k$ bins. Thus, we want their pitch distributions to be equal but shifted by $k$ bins, ensuring that the network is equivariant to the pitch shift.

We train the network $f_{\theta }$ to minimize objectives that enforce these invariance and equivariance constraints.

3.3 Architecture

The architecture of our pitch estimator is inspired by Bittner et al. (2022) and Weiß and Peeters (2022). Each input VQT frame is processed independently through the following sequence: first, layer normalization (Ba et al., 2016) is applied, followed by a series of 1D convolutions along the log‑frequency dimension—three with skip‑connections (He et al., 2016) and four regular ones. The kernel size is set to $13B=39$ to span more than one octave (Bittner et al., 2022)—for $B=3$ bins per semitone. As in Weiß and Peeters (2022), we apply a non‑linear leaky‑ReLU with a slope of $0.3$ (Xu et al., 2015) and dropout with rate $0.2$ (Srivastava et al., 2014) between each convolutional layer. Importantly, the kernel size and padding of each of these layers are chosen so that the frequency dimension is never reduced. A translation of $k$ bins of a VQT frame, therefore, leads to a translation of $k$ bins of the output pitch distribution. The output is then flattened, fed to a final fully‑connected layer, and normalized by a softmax layer to become a probability distribution of the desired shape.

Note that all layers (convolutions,⁷ element‑wise non‑linearities, layer‑norm, and softmax), except the last final fully‑connected layer, preserve transpositions. To make the final fully‑connected layer also transposition‑equivariant, and in line with previous works on fully‑convolutional networks (Ardaillon and Roebel, 2019; Shelhamer et al., 2017), we propose to use Toeplitz fully‑connected layers. These consist of a standard linear layer without bias but whose weights matrix $A\in {\mathbb{R}}^{F^{\prime }\times K}$ is a Toeplitz matrix, i.e., each of its diagonals is constant.

In practice, this matrix $A$ is equivalent to a 1D‑convolution with kernel $(a_{-K+1},\dots ,a_{F^{\prime }-1})$ and padding $K-1$ and therefore preserves translations while having fewer parameters than an arbitrary fully‑connected layer.

3.4 Re‑centering pitch distributions

Recall that our model learns to predict pitches from VQT frames up to an additive constant and that the bins of the predictions do not correspond one‑to‑one to VQT log‑frequencies: during training, the bin associated with a given pitch is completely arbitrary and only depends on the initialization of the model.

In practice, due to random weight initialization, we sometimes observed that the performances of our model significantly drop when a low pitch value is initially mapped to a high bin of the pitch distribution (or the opposite) because the whole pitch distribution is concentrated in a few bins. This is particularly likely when training on datasets that span a wide range of frequencies (such as MDB‑stem‑synth). While this phenomenon is far from the norm, it can limit the usability of our model in real‑world scenarios.

To prevent that, we set our output probability distribution to cover more than $10$ octaves ($K=384$ bins with a resolution of $B=3$ bins per semitone), making collapse unlikely for most real‑world data. In addition, we explicitly force the median of all the predictions to remain roughly in the center of the pitch distribution during training. If it deviates too much, we apply a circular shift to the kernel $(a_{m-1},\dots ,a_{-n+1})$ of the final Toeplitz fully‑connected layer, which moves the predicted pitch distributions accordingly.⁸ This simple trick enables us to rectify bad initialization during training with minimal overhead.

3.5 Pitch decoding

4.1 Streamable VQT

By design, our model processes individual VQT frames independently, which makes it inherently suitable for real‑time applications. However, given an audio stream $\mathbf{s}=(s_1,s_2,\dots )$, computing a VQT frame centered around $s_i$ requires frames from $s_{i-w}$ to $s_{i+w}$, where $w$ is the largest VQT window size. This non‑causal operation is incompatible with real‑time processing, as it requires future audio samples.

We build upon nnAudio (Cheuk et al., 2020), which implements audio time‑frequency transforms as PyTorch modules using convolutional layers.⁹ In nnAudio's time–domain implementation, the transforms basis kernels are precomputed and stored as frozen model parameters. While nnAudio computes the CQT by convolving the input audio with CQT kernels followed by normalization, we modify this approach by replacing the CQT kernels with VQT kernels. However, the convolution kernel lengths still exceed the typical buffer size of 5–20 ms needed for real‑time applications (see Figure 1). To address this, we replace standard convolutional layers with cached convolutions, allowing our model to support streamed inputs by storing previous audio chunks in an internal circular buffer (Caillon and Esling, 2022).

4.2 Lag minimization

Our model predicts the pitch of the center of the VQT frames given as input. In addition to the compute time $\tau$ of a forward pass of our model (typically less than 10 ms), this implies a theoretical lag equal to half of the VQT's kernel size between when a buffer is acquired and when the pitch of a frame centered on this buffer is returned:

where $w$ represents the VQT largest kernel size and $f_s$ is the sampling rate (see Figure 4a). Given that $w$ is typically proportional to $f_s$ (see Eq. (5)), the main limiting factor is the window size.

Figure 4

For the CQT, the kernel size $w$ is particularly large at low frequencies (e.g., $w=131072$ samples, or $2.7$ seconds, at 48 kHz for an A0). By using the VQT, this issue is alleviated by reducing $w$ drastically, with the added bonus of enhancing model performance (see Section 7.1). For instance, by selecting $\gamma =7$, we have a maximum kernel size of $w=8192$ samples at 48 kHz (171 ms). Even with such improvements, a lag of around 70 ms remains perceptible for most applications, such as real‑time resynthesis.

Increasing $\gamma$ could further decrease $w$ (see Figure 1), but, beyond a certain point, this also compromises model accuracy, as shown in Section 7.1.

To minimize this lag, we propose a buffer refilling technique that artificially creates windows centered closer to the current buffer. By shifting the input buffer to the center of the VQT frame, and filling the right side of the window (i.e., the non‑causal region) by repeating samples from the end of the buffer, we are able to predict the pitch of the current audio chunk with minimal latency. For a causal audio stream $\mathbf{s}=(s_1,\dots ,s_t)$, the frame is computed from a modified chunk:

where $h$ is the hop/buffer size and $m\in [0,\frac{1}{2}]$ is the refill factor, with $m=0$ corresponding to no refilling at all (default behavior) and $m=\frac{1}{2}$ corresponding to the maximum possible refilling.

While it is common to reflect samples when repeating data as padding at analysis window boundaries, buffer refilling keeps the repeated samples in their original order. This is because reflection in the middle of the window would cause destructive interference between the imaginary components of the CQT kernel, which is conjugate symmetric under reversal.

As illustrated in Figure 4b, this technique enables us to construct windows centered on the most recent audio buffer, leading to:

In particular, the total lag can be reduced to $h/2f_s+\tau$ using maximum refilling ($m=0.5$), making the system extremely reactive.

To reduce the overall latency of our model, we also aim to reduce the compute time $\tau$ of our model itself. We improve the original implementation of nnAudio's VQT by computing both real and imaginary kernels with a single convolution layer, and further reduce compute time with Just‑In‑Time (JIT) compilation.

4.3 Frequency–domain toeplitz layer

Further, we propose a faster implementation of the final Toeplitz fully‑connected layer of our architecture (see Section 3.3). This layer, necessary for PESTO's translation equivariance, can be further optimized to reduce runtime complexity. Naïve implementation of the layer as a 1D convolution incurs a cost of $\mathcal{O}((m+n-1)\cdot n)$. This can be reduced to $\mathcal{O}(m\cdot n)$ through optimizations such as implicit padding, as implemented in optimized backends such as oneDNN and cuDNN, or even by directly realizing the $m\times n$ Toeplitz matrix. However, given the size of both the activation vector and implicit kernel, time complexity can be further reduced by performing convolution as multiplication in the frequency domain. This allows the Toeplitz layer to be applied in log‑linear time and eliminates redundancy in the model's forward pass.

5.1 Implementation details

Unless specified otherwise, VQT computations are performed with $f_{\text{min}}=27.5$ Hz, which is the frequency of the lowest key of the piano (A0), $B=3$ bins per semitones and at most $F=99B$ log‑frequency bins, which corresponds to the maximal number of bins respecting the Nyquist frequency for a $16$‑kHz signal. The maximal pitch‑shift between frames is $k_{\text{max}}=16$ bins, i.e., slightly more than five semitones. We set the VQT parameter $\gamma =7$ for our main experiments and use a resolution of $B=3$ bins per semitone.

For our equivariance loss, we fix $\alpha =2^{\frac{1}{12B}}$. The exponential moving average rate for the loss weighting is $\eta =0.999$. White noise and random gain are applied as pitch‑preserving augmentations with a probability of $0.7$ to the cropped VQT frames. Standard deviation values for the white noise are drawn from a uniform distribution between $0.1$ and $2$, and gain values are drawn between $-6$ and $3$ dB. For training, we use a batch size of $256$ and the Adam optimizer (Kingma and Ba, 2015) with a learning rate of $10^{-4}$ and default parameters. The model is trained for $50$ epochs using a cosine annealing learning rate scheduler. We refer to our code for precise configurations and hyperparameters. With our architecture being very lightweight, training requires only 845 MB of GPU memory and can be performed on a single GTX 1080Ti.

5.2 Datasets

Three datasets are considered for this study:

1. MIR‑1K (Hsu and Jang, 2010) contains $1000$ tracks (about $2$ h) of amateur singing of Chinese pop songs, with separate vocal and background music tracks provided. The isolated vocals partition is used for most experiments, apart from the one described in Section 6.7.

2. MDB‑stem‑synth (Salamon et al., 2017) contains 15.56 h of re‑synthesized monophonic music played by 25 different instruments. Perfect fundamental frequency annotations are available.

3. PTDB (Pirker et al., 2011) contains 4718 speech recordings from English speakers’ speech and corresponding laryngograph recordings for a total of $9.6$ h.

The datasets vary in size, pitch range, and granularity for the provided ground‑truth pitch annotations. Pitch annotations are provided with a hop size of 20 ms for MIR‑1K, 2.904 ms ($128$/$44100$) for MDB‑stem‑synth, and 10 ms for PTDB. We opt not to resample the annotations, as our model is not bound to a fixed hop size or sample rate. During training, we can take advantage of the original granularity of the annotations, while, on inference, we can choose the hop size that best fits the application. This is another advantage of using SSL with time‑frequency frontends compared to using supervised learning and raw waveform training: the sample rate and hop size can be changed on the fly for inference.

As noted in previous work (Morrison et al., 2023), a crucial problem in neural pitch estimation is overfitting to the pitch characteristics in the training data. Therefore, we conduct separate model training and evaluation on all datasets and examine generalization performance through cross‑evaluation. Another practical issue is the potential misalignment of pitch annotations, which can be difficult to detect and potentially worsen the performance of a model trained on these. For instance, on PTDB, pitch annotations are provided at time steps with a $5$‑ms offset (e.g., $5$, $15$, or $25$ ms for a hop size of $10$ ms) compared to MIR‑1K/MDB‑stem‑synth, where annotations are aligned with multiples of the hop size (e.g., $0$ ms, $10$ ms, $20$ ms). Indeed, we observed that not accounting for this offset negatively impacts the evaluation metrics. Since our model does not use annotations at training time, it is not affected by this issue, and potential misalignments can be accounted for without retraining.

5.3 Baselines

We compare our model to state‑of‑the‑art SSL and supervised neural pitch estimators:

• CREPE (Kim et al., 2018) is a supervised model trained on MDB‑stem‑synth (Salamon et al., 2017) and MIR‑1K (Hsu and Jang, 2010), as well as Bach10 (Duan et al., 2010), RWC‑Synth (Mauch and Dixon, 2014), MedleyDB (Bittner et al., 2014), and NSynth (Engel et al., 2017). We use the default pretrained model provided in the official repository and deactivate Viterbi smoothing for fair comparisons.¹⁰

• PENN (Morrison et al., 2023) is a supervised model which improves the FCNF0 architecture. We do not retrain PENN, but we reproduce the authors' results on MDB‑stem‑synth and PTDB with the publicly available FCNF0++ model,¹¹ which was trained on the training partitions of both datasets.

• SPICE (Gfeller et al., 2020) is an SSL model trained to minimize the pitch difference (up to the known shift) between two cropped CQT frames. SPICE additionally reconstructs the input frame using a decoder and employs a reconstruction regularization.

• DDSP‑inv (Engel et al., 2020a) performs self‑supervised pitch estimation by using analysis‑by‑synthesis and estimating the parameters of a harmonics plus noise synthesizer, with pretraining on synthetic data.

5.4 Evaluation metrics

Our evaluation procedure is standard and common to previous works (Gfeller et al., 2020; Kim et al., 2018; Morrison et al., 2023). In practice, estimated and reference frequencies are converted to fractional semitones using the mapping:

which maps the A4 (440 Hz) to $69$ (which is the MIDI standard) and the other pitches accordingly.

For any voiced frame, the pitch error $e$ between the estimated fundamental frequency $\hat{f}$ and the ground‑truth $f$ is therefore:

which exactly corresponds to the (fractional) number of semitones between the prediction and ground‑truth pitch. From this error, we compute the standard metrics introduced by Poliner et al. (2007):

• Raw pitch accuracy (RPA), which measures the proportion of voiced frames for which the pitch error is less than half a semitone ($50$ cents).

• Raw chroma accuracy (RCA), which measures the proportion of voiced frames for which $|e(\hat{f},f)$ $\text{mod}12|\le 0.5$, thus accepting octave errors.

In addition, we also report the mean pitch error (MnE) and median pitch error (MdE) in semitones over all voiced frames.

6.1 Comparison with self‑supervised methods

PESTO significantly advances the state‑of‑the‑art in self‑supervised pitch estimation, achieving $97.7\%$ and $97.0\%$ RPA on MIR‑1K and MDB‑stem‑synth respectively, outperforming previous self‑supervised approaches SPICE and DDSP‑inv by a large margin (Table 1).

6.2 Comparison with supervised methods

Table 2 shows experimental results comparing PESTO with supervised methods. Despite having only $130$k parameters—$170$ times fewer than CREPE's $22.2$ M—our model exceeds its performance on MIR‑1K ($97.7\%$ vs. $97.5\%$ RPA) and PTDB ($89.7\%$ vs. $87.1\%$ RPA), while being comparable on MDB ($97\%$ vs. $97.3\%$). While PENN achieves superior results when trained on matched domains ($99.6\%$ RPA on MDB, $95.1\%$ RPA on PTDB), PESTO maintains competitive performance without requiring supervision.

6.3 Cross‑dataset evaluation

Table 2 shows detailed results for cross‑dataset evaluation.

Music $\to$ Speech: PESTO models trained on music data outperform CREPE and PENN (when trained on music) on speech datasets. PENN, when trained only on MDB, achieves $63.2$ RPA on PTDB, while PESTO achieves $88.3$ RPA. CREPE, trained on many more music datasets, achieves $87.1$ RPA on PTDB.

Speech $\to$ Music: When trained on PTDB (speech), PESTO still achieves $96.3\%$ RPA on MDB (music), while PENN's performance drops to $51.6\%$ RPA.

Music/Speech $\to$ Singing Voice: PESTO consistently outperforms PENN on MIR‑1K across training conditions. When trained on both MDB and PTDB, PENN achieves $90.6\%$ RPA, while PESTO achieves $94.6\%$ RPA or higher in all cross‑evaluations for MIR‑1K.

Overall, our results corroborate the findings that supervised models are heavily influenced by the training pitch distribution (Morrison et al., 2023) and show that PESTO is more robust to out‑of‑distribution data. Although PESTO training and evaluating on the same dataset yield the best results, the cross‑domain performance gap remains considerably smaller than when using supervised baselines. Notably, the MdE metrics remain stable across training sets, suggesting PESTO captures fundamental pitch characteristics, while dataset‑specific training primarily improves robustness to challenging frames.

6.4 Multi‑dataset training

When evaluating on a given dataset, we observe that models trained solely on that dataset generally have better performance than models trained on combined datasets. MDB is a notable exception, where comprehensive training yields the best results, though the difference is marginal ($97.1\%$ RPA instead of $97.0\%$). This effect may be attributed to MDB's larger size dominating the combined dataset distribution.

On the contrary, PENN performs better (on PTDB) or equally (on MDB) when trained on multiple datasets, and it clearly outperforms PESTO when the training and test distributions overlap.

6.5 Inference speed

We evaluated the real‑time capabilities of our PESTO model, as well as PENN (Morrison et al., 2023) and YIN (de Cheveigné and Kawahara, 2002) as baselines, without Viterbi decoding. The real‑time factor (RTF), defined as the ratio of processing time to audio input duration, was measured across CPU (Intel Core i9‑12900H 2.9GHz) and GPU (NVIDIA RTX A2000 8GB) platforms. Using 5‑s audio segments at 16 kHz with a 10‑ms hopsize, we performed 100 measurements for each configuration. For PENN GPU inference, we used a batch size of 2048 frames using the publicly available API. All reported timings in Table 3 include the inference pipeline from loaded audio input to pitch estimation (or pitch candidates for YIN) output.

PESTO outperforms all other models in RTF by a significant margin. PENN achieves real‑time performance (in domain) but requires roughly five times more processing time than PESTO on CPU and three times more on GPU, while using parallel computation of frames with a batch size of $2048$. Finally, PESTO is approximately $60\%$ faster than the YIN baseline on CPU and 18 times faster when ran on a GPU.

6.6 Effect of buffer refilling

To determine the effect of the lossy buffer refilling technique (Section 4.2) on PESTO's performance, we retrained the model for refill factors $m\in \left\{0.1,0.2,\dots ,0.5\right\}$ on the MIR‑1k dataset. RPA and RCA metrics are reported on the test dataset in Table 4. We observe a minimal degradation in performance, with an RPA of $97.4\%$ at the maximum possible refill factor ($m=0.5$), compared to the standard PESTO model, which achieves $97.7\%$ ($m=0.0$). This suggests that PESTO is relatively robust to the removal of future context via buffer refilling and is thus suitable for deployment in real‑time and latency‑critical scenarios.

For comparison, we also try replacing the refilled portion of the analysis window with zeros. For $m\le 0.4$, this appears to perform equivalently to buffer refilling, but, at the maximum refill factor of $m=0.5$, we find that the zero‑filling technique suffers degraded performance.

6.7 Robustness to background music

In real‑world scenarios, we often do not have access to clean signals. In this section, we evaluate to what extent the predictions of our model are robust to the presence of background music in the signals. To do so, we use the MIR‑1k dataset, for which we have access to the separated vocals and background music, which allows testing various signal‑to‑noise (here, vocals‑to‑background) ratios.

We indicate the results in Table 5. As foreseen, the RPA of PESTO when trained only on clean vocals (row $\beta =0$) considerably drops: from $97.2$% to $46.8$%.

To improve the robustness to background music, we propose to modify the training pipeline slightly. Instead of the data augmentations described in Section 5.1, we create the augmented view $\tilde{\mathbf{x}}$ of our original vocals signal ${\mathbf{x}}_{\text{vocals}}$ by mixing it (in the complex‑CQT domain) with its corresponding background track ${\mathbf{x}}_{\text{background}}$:

with $\beta \in \mathbb{R}$.¹² The model is thus trained to ignore the background music to make its predictions.

We investigate different strategies for sampling $\beta$, whose respective results are depicted on Table 5.

We observe that using background music as data augmentation consistently improves performance regardless of the sampling strategy. Notably, sampling from a distribution instead of using a fixed value yields better results. This improvement arises because the model learns to handle varying signal‑to‑noise ratios and because the background level differs between the input $\tilde{\mathbf{x}}$ and its pitch‑shifted counterpart ${\tilde{\mathbf{x}}}^{(k)}$. Consequently, not only does the invariance loss ${\mathcal{L}}_{\text{inv}}$ help the model ignore the background, but the other losses also contribute. Sampling $\beta$ from the uniform distribution $\mathcal{U}(0,\frac{1}{2})$ generally leads to the best performance, except when vocals and background are equally mixed ($\text{SNR}=0\text{dB}$), where $\beta \sim \mathcal{N}(0,1)$ appears optimal, achieving $82.5$% RPA. Interestingly, incorporating background music during training not only enhances performance on noisy signals but also improves results on clean ones. Our best model achieves $98.3$% RPA on the clean test set of MIR‑1K, surpassing previous data‑augmentation strategies (see Table 2). This demonstrates that, when background music is available, it can significantly benefit single pitch estimators.

Finally, PESTO outperforms both supervised (CREPE, PENN) and self‑supervised baselines (SPICE), apart from $\text{SNR}=0\text{dB}$, where CREPE performs better. However, the drop in performance between $\text{SNR}=10\text{dB}$ and $\text{SNR}=0\text{dB}$ is higher for PESTO (approximately $15$%) than that for SPICE or CREPE (about $10$%), which suggests there is still room for improvement.

7.1 Impact of the input frontend

We vary the $\gamma$ parameter of the VQT between $0$ (CQT) and $15$ and train models for each dataset. Figure 5 shows the RPA and RCA results for cross‑evaluation for different values of $\gamma$. Each reported value is the mean of the top three scores achieved out of five runs with different random seeds, for a given training dataset and $\gamma$ value.

Figure 5

Our experiments demonstrate that the VQT as an input to the model significantly enhances pitch‑estimation accuracy compared to the CQT. As it uses smaller kernel sizes at lower frequencies ($\gamma >0$) than the CQT, the VQT achieves superior time–frequency alignment with the ground‑truth pitch contours. As an example, we illustrate in Figure 6 a comparison of spectrogram frames for the CQT ($\gamma =0$) and the VQT ($\gamma =7$). The VQT produces sharper temporal boundaries and more precise frequency localization than the CQT.

Figure 6

While increasing $\gamma$ improves performance across all datasets, this benefit plateaus and eventually deteriorates when frequency resolution is compromised. This trade‑off is particularly pronounced in the speech dataset, which has rapid phonetic transitions compared to musical signals yet still requires sufficient spectral detail for accurate pitch tracking. Interestingly, the optimal $\gamma$ value is not the same for all datasets, highlighting the fact that there is a domain‑dependent tradeoff between low‑frequency resolution and time resolution. Values in the range $\sim [7-10]$, however, provide the best results with relatively low variation for all datasets.

In theory, under supervised training, one could leverage the temporal context to compensate for misalignments or learn the optimal time–frequency representation for pitch estimation. Our results suggest, however, that a small adjustment to the CQT already provides a more flexible and efficient solution that generalizes across diverse acoustic domains while allowing for SSL training without the need for labeled data. Furthermore, as Section 4 details, the use of the VQT significantly improves the computation speed of the model, which is critical for real‑time.

7.2 Design choices

In this section, we examine the impact of various design choices on the performance of our model. The results for training on MIR‑1K are shown in Table 6a. First, we observe that the final Toeplitz fully‑connected layer, which ensures the architecture is transposition‑preserving, is essential for the model to learn effectively. In contrast, data augmentations are not necessary for training but usually enhance performance.

The influence of the different loss functions is more complex to interpret. When using only ${\mathcal{L}}_{\text{SCE}}$, the model does not learn at all, whereas ${\mathcal{L}}_{\text{equiv}}$ alone yields good in‑domain performance. However, combining ${\mathcal{L}}_{\text{equiv}}$ with the other two losses helps the model generalize to out‑of‑domain distributions. Interestingly, ${\mathcal{L}}_{\text{inv}}$ alone produces decent performance, indicating that learning to be insensitive to white noise and gain changes enables the model to focus on the fundamental frequency of the signal, though the accuracy is lower than when ${\mathcal{L}}_{\text{equiv}}$ is included.

Surprisingly, discarding ${\mathcal{L}}_{\text{equiv}}$ and using only the other two losses achieves the best out‑of‑distribution performance. This counterintuitive result prompted further evaluation of ${\mathcal{L}}_{\text{equiv}}$ when training on MDB‑stem‑synth (Table 6b) and PTDB (Table 6c). These experiments, however, did not reveal a consistent pattern: ${\mathcal{L}}_{\text{equiv}}$ appears beneficial for training on PTDB but slightly detrimental for MDB‑stem‑synth.

This suggests that the design of our loss functions and the weighting strategy (based on gradient norms; see Section 3.2.4) may benefit from further refinement.