A Lightweight Two‑Branch Architecture for Multi‑Instrument Transcription via Note‑Level Contrastive Clustering

2.1 AMT fundamentals

The majority of existing AMT approaches employ deep neural networks to transform audio inputs into piano roll– like representations, characterized as two‑dimensional matrices indexed by time frames and note pitches. The process generally comprises: 1) time–frequency feature extraction, 2) frame‑level probability estimation with neural networks, and 3) note generation through binarization or specialized output networks.

Stage 1 is typically implemented with classical signal‑processing transforms. The short‑time Fourier transform (STFT) yields linear‑spaced frequency bins, whereas mel‑scale transform and constant‑Q transform (CQT) provide logarithmic spacing that better matches musical pitch perception. However, these interpretable transforms may not always be optimal, leading to explorations of learnable encodings (Luo and Mesgarani, 2018; Ravanelli and Bengio, 2018; Zeghidour et al., 2021) and hybrid methods (Rouard et al., 2023; Su and Yang, 2015). The CQT is readily enriched into a harmonic CQT (HCQT) (Bittner et al., 2017) by vertically shifting the spectrogram according to the bin offsets of successive harmonics and concatenating the shifted copies along an additional axis (Balhar and Hajič, 2019). This prior‑driven expansion allows small convolutional kernels to focus efficiently on musically relevant frequency components.

Stage 2 produces a two‑dimensional posteriogram that encodes the probability of pitch activation at each time–frequency bin. Convolutional layers leverage the spectro‑temporal grid to capture local structures (Bittner et al., 2022; Wu et al., 2019), while temporal‑sequence models extend the receptive field along time, modeling long‑range dependencies (Hawthorne et al., 2018). To translate frame‑wise posteriors into discrete notes, in stage 3, most systems rely on onset detection (i.e., the initial frame of each note) and then link successive active frames within an onset‑defined segment to form complete note events (Bittner et al., 2022; Hawthorne et al., 2018; Wu et al., 2020, 2024).

2.2 Timbre‑Separated transcription

Multi‑instrument timbre‑separated transcription goes beyond traditional AMT by extracting instrument‑specific notes from polyphonic mixtures. Tamer et al. (2023b) demonstrated that simply cascading source separation followed by transcription leads to suboptimal results due to error propagation. Therefore, it is necessary to combine transcription and separation in a mutually reinforcing manner.

Innovative methods have emerged for this challenge. Gardner et al. (2022) introduced a sequence‑to‑sequence AMT model that outputs multiple instrument tracks with explicit instrument assignments, achieving high accuracy but relying heavily on diverse multi‑timbre data. Multitask learning frameworks like Timbre‑Trap (Cwitkowitz et al., 2024) and Cerberus (Manilow et al., 2020) show that joint optimization can lead to synergistic effects. Wu et al. (2020) proposed a self‑attention–based instance segmentation approach that performs joint note detection and instrument classification, achieving timbre‑separated transcription for a closed set of trained instruments.

For handling unseen timbres, researchers have explored different strategies. The zero‑shot learning method proposed by Lin et al. (2021) employs a query‑based mechanism: under contrastive supervision, it learns to encode a clean reference example into a timbre embedding that generalizes to unseen instruments. This embedding is then used to sequentially modulate the U‑Net layers, enabling zero‑shot separation and transcription of audio sources with novel timbres not encountered during training. The work by Tanaka et al. (2020) pioneered the application of deep clustering networks to timbre‑separated transcription. Their approach develops a generalizable timbre encoder and enables separation with a non‑fixed number of classes through clustering, showing how these tasks can complement each other when properly combined.

2.3 Deep clustering methodology

Deep clustering serves as a fundamental approach addressing two core challenges in source separation: the permutation problem and speaker‑independent separation. The methodology operates by encoding each time‑frequency bin into a high‑dimensional feature, then assigning labels through clustering algorithms to generate separation masks.

The pioneering work of Hershey et al. (2016) first applied deep clustering to separate unknown speakers. This was later extended by Luo et al. (2018), who introduced learnable attractors as speaker‑specific reference points in an embedding space that cluster time–frequency bins, enabling end‑to‑end separation of a variable number of sources. The adaptation to music transcription was first achieved by Tanaka et al. (2020), but their method uses transcription mainly to aid audio separation, lacks dedicated note‑creation postprocessing, and omits key implementation details. More fundamentally, these studies share two critical limitations: clustering at the frame level produces fragmented results, and the hard‑assignment strategy cannot handle overlapping situations. In contrast, our clustering postprocess is tailored for music transcription, aggregating frame embeddings into coherent note events with minimal computational overhead.

3.1 Problem configuration

Let $X={\sum }_{i=1}^K{x}_i\in {\mathbb{R}}^l$ be the mixed audio signal of $K$ instrument classes (timbres), where $l=22.05\text{\hspace{0.17em}[kHz]}\times \text{Time [sec]}$ is the signal length and $x_i$ is the single timbre audio corresponding to the frame activation $y_{Fi}\in {[0,1]}^{N\times T}$ and onset activation $y_{Oi}\in {[0,1]}^{N\times T}$. Since we are aiming for the pitch range $C_1\sim B_7$, we set $N=7\times 12=84$. The goal of timbre‑agnostic transcription is to map $X$ to ${\sum }_{i=1}^K{y}_{Fi}$ and ${\sum }_{i=1}^K{y}_{Oi}$, while the goal of timbre‑separated transcription is to obtain ${\{y_{Fi}\}}_{i=1}^K$. First, we apply the CQT to obtain $Q\in {\mathbb{C}}^{F\times T}$, where $T$ is the number of frames and $F$ is the number of frequency points analyzed. We extended the CQT analysis to eight octaves with three bins per semitone, ensuring each note has at least two harmonics and yielding $F=8\times 12\times 3=288$ bins. Our proposed model outputs the timbre‑agnostic transcription results $Y_F={\sum }_{i=1}^K{\hat{y}}_{Fi}$ and $Y_O={\sum }_{i=1}^K{\hat{y}}_{Oi}$ and the timbre‑sensitive representation $V\in {\mathbb{R}}^{D\times N\times T}$. Finally, $V$ is clustered to assign labels to $Y_F$, separating $\{{\hat{y}}_{F1},{\hat{y}}_{F2},\dots ,{\hat{y}}_{FK}\}$ from $Y_F$.

3.2 Overall architecture

As shown in Figure 1, our timbre‑separated transcription model comprises two parallel branches: timbre‑agnostic transcription and timbre encoding, both taking the HCQT as input. We also design a postprocessing pipeline tailored to this task.

Figure 1

The timbre‑agnostic transcription branch builds upon BasicPitch (Bittner et al., 2022), a lightweight fully convolutional network that demonstrates strong performance using only local contextual information. We identify several limitations and introduce targeted optimizations, yielding a more compact architecture with comparable accuracy.

The timbre‑encoding branch adopts a similar fully convolutional design. To incorporate global context, we employ InstanceNorm and GroupNorm for adaptive normalization across the entire input. While convolutional architectures are more deployment‑friendly on low‑resource devices than recurrent neural network (RNN)‑based approaches commonly used in deep clustering (Hershey et al., 2016; Tanaka et al., 2020), they may lack long‑range modeling capacity; alternative designs are discussed in the experiments.

3.3 EnergyNorm for spectral normalization

Conventional two‑dimensional normalization schemes often involve subtractive operations and compute statistics directly over the time–frequency plane. These practices alter the relative amplitude relationships among harmonic components within the same analysis frame, thereby undermining the acoustic basis of timbre perception.

We, therefore, introduce an interpretable normalization approach based on frame‑wise energy. Leveraging Parseval’s theorem and assuming a Gaussian time‑domain signal, we standardize the sample variance of frame energy to unity, yielding a chi‑squared–distributed amplitude envelope. Given a CQT spectrum $Q\in {\mathbb{C}}^{F\times T}$, the process is described as:

This approach reduces the normalization complexity from two dimensions to one, significantly lowering computational costs. Furthermore, to address the limitations of strictly nonnegative features in neural network learning, we apply a logarithmic transformation followed by an affine transformation to enhance representational capacity. The final normalized representation is defined as:

where $k$ and $b$ are learnable scalars.

3.4 Dilated convolution for harmonic context

In BasicPitch, a large convolutional kernel spanning 39 frequency bins (one octave plus one note) is explicitly mentioned as effective for mitigating octave errors. However, the reference implementation applies padding to both ends, effectively limiting the receptive field to half an octave above and below, rather than the intended full range above or below. Given the explicit spatial correspondence between input and output representations, this limitation prevents the model from directly attending to information at harmonic intervals. To address this issue, we adopted a more robust strategy by extending the receptive field to cover two octaves in both directions. We implemented this using a dilated convolution with a frequency‑axis kernel size of 25 and a dilation factor of 3. This configuration ensures comprehensive harmonic coverage while simultaneously reducing the parameter count.

3.5 Focal loss for class imbalance

BasicPitch employs a class‑balanced cross‑entropy loss to address the sparsity of onsets, with negative and positive class weights set at 0.05 and 0.95, respectively. However, our experiments revealed that this weighting scheme exacerbates class imbalance. Figure 2 presents our controlled experimental results, demonstrating that assigning a larger weight to the positive class of onsets leads to severe false positives, indicating that sparse classes should be assigned smaller weights.

Figure 2

We directly used Focal Loss (Lin et al., 2017) in Eqn. (5) with $\gamma =1$, setting the positive class weights for notes and onsets at 0.2 and 0.06, respectively.

3.6 InfoNCE for contrastive cluster formation

Hershey et al. (2016) and Tanaka et al. (2020) use the following deep clustering objective:

where $Z\in {\mathbb{R}}^{M\times K}$ denotes the one‑hot class labels for active bins ($K$: number of classes, $M$: number of bins with ground‑truth activation) and $U^T\in {\mathbb{R}}^{D\times M}$ is the timbre encodings at the corresponding locations extracted from $V\in {\mathbb{R}}^{D\times N\times T}$. The $m$‑th column ${\mathbf{u}}_{\mathbf{m}}\in {\mathbb{R}}^D$ of $U^T$ represents the timbre encoding of the $m$‑th active time–frequency bin. Defining $\hat{Z}=Z{Z}^T\in {\mathbb{R}}^{M\times M}$ yields:

This loss function benefits from low spatial complexity and high computational efficiency. However, it is at least three orders of magnitude larger than the AMT loss (frame and onset transcription with Focal Loss) and converges slowly, which makes joint optimization more difficult when a backbone is shared. Furthermore, this objective enforces orthogonality between encodings of different classes. We argue that strict orthogonality is an overly stringent constraint; for clustering objectives, antiparallel alignment yields larger interclass distances than orthogonality. Consequently, we adopt the contrastive InfoNCE loss (Oord et al., 2018) to facilitate cluster formation:

where $y_m$ denotes the ground‑truth class label of ${\mathbf{u}}_m$. Empirically setting the temperature parameter $\tau =0.15$ aligns the magnitude of the clustering loss with the AMT loss and synchronizes their convergence rates. Crucially, in scenarios involving a single timbre, the InfoNCE loss naturally vanishes. In contrast, the MSE‑based loss continues to force feature vectors toward specific directions even in the absence of contrasting classes, causing the mean output to deviate significantly from the origin and resulting in potential instability.

3.7 Postprocessing for timbre‑separated transcription

Existing deep clustering–based source‑separation methods are predominantly designed for spectrogram reconstruction. In the field of music transcription, however, there is a lack of discussion on methods for creating specific notes from clustering results.

Tanaka et al. (2020) proposed a frame‑level transcription method that predicts a binary mask by assigning an additional cluster to ‘silence bins’ and performing K‑means clustering over the entire time–frequency space. However, this approach suffers from several critical limitations:

• The absence of onset information hinders accurate note creation from masks.

• Frame‑level separation is prone to fragmenting notes into scattered pieces.

• The large number of bins makes clustering algorithms very slow.

• Most clustering algorithms require specifying the number of clusters, and the hard‑assignment strategy cannot handle overlapping situations, where instruments play the same pitch simultaneously.

Benefiting from the dual‑branch architecture, the separation process can be conducted at the note level. We first obtain notes using BasicPitch’s note decoding method, then weight and sum the encodings of each frame in the note to get its timbre encoding, which is subsequently clustered via spectral clustering. This method aggregates time–frequency bins of the same note, greatly reducing the number of samples for clustering and mitigating note fragmentation. For clustering, we construct an affinity matrix by exponentiating cosine similarities between note embeddings, matching the formulation in our clustering loss (Eqn. 6). Figure 3 illustrates the results of frame‑level and note‑level transcription, with the latter nearly perfectly reconstructing compared to the former’s fragmented notes.

Figure 3

The last issue remains unresolved, as our attempted iterative matching‑filter approach suffers from category merging and is limited to frame‑level processing; thus, we do not discuss it further.

4.1 Experimental setup

4.1.1 Datasets

Table 1 summarizes the datasets used in our experiments. We train primarily on MusicNet (Thickstun et al., 2017) but replace its DTW‑aligned annotations, which are known to be inaccurate, with the refined labels from MusicNetEM (Maman and Bermano, 2022). All audio is resampled to 22,050 Hz and split into non‑overlapping 900‑frame (i.e., 10.4‑s) clips (frame size: 256 samples, matching the CQT hop length). Tracks sharing the same instrument are mixed to form a single timbral class per piece.

Table 1

Datasets used in experiments. ‘Our:AMT’ and ‘Our:Sep’ are our synthetic datasets created to examine whether human composition and real recordings are indispensable. The latter’s timbres are grouped into 10 classes; when training, cross‑class mixtures yield single‑sample multi‑timbre pieces. Bold datasets are used for training.

Dataset	Dur.	Songs	Instr.	K/Song
MusicNet	34 h	330	11	1–8
BACH10	334 s	10	4	4
PHENICX	637 s	4	10	8–10
URMP	1.3 h	44	14	2–4
Our:AMT	24 h	8316	33	1
Our:Sep	836 s	120	34	1

For evaluation, we use three real‑world polyphonic datasets with solo tracks: BACH10 (Duan et al., 2010) (fixed quartet of violin, bassoon, clarinet, and saxophone), URMP (Li et al., 2019), and PHENICX (Miron et al., 2016), with the latter featuring complex orchestral mixtures with an average of 9.5 instrument classes per piece.

Motivated by Slakh2100 (Manilow et al., 2019), a large‑scale, fully synthesized dataset, we question the necessity of human‑composed, real‑recorded data. We, therefore, generate synthetic audio via a simple algorithm that randomly places notes across our 84‑note pitch range and adds harmonically overlapping chord tones with probability. MIDI sequences are rendered with FluidSynth, applying randomization in dynamics, pitch tuning, and articulation to improve realism. Each sample uses a single timbre; an example appears in Figure 4.

Figure 4

For the timbre‑agnostic model, we synthesize 252 clips with 900 frames each from 33 General MIDI programs, yielding $\sim$24.1 hours of training data. For timbre‑separated transcription, we define 10 categories of acoustically similar instruments, each contributing 12 clips with 600 frames each, totaling $\sim$13.9 minutes of base material. During training, clips are dynamically mixed by blind addition across categories with random gain scaling; all possible combinations are exhaustively enumerated across epochs, greatly increasing data diversity.

4.2 Implementation details

4.3 Overall performance comparison

Table 2 presents the results for timbre‑agnostic transcription. Comparing the rows ‘Ours,’ ‘${\text{BP}}^{\text{fl}}$,’ and ‘${\text{OF}}^{\text{fl}}$’ (all trained with focal loss) and the additional rows ‘MN,’ ‘${\text{BP}}_{\text{MN}}^{\text{fl}}$,’ and ‘${\text{OF}}_{\text{MN}}^{\text{fl}}$’ (trained on MusicNet), we observe that our model achieves performance on par with BasicPitch using significantly fewer parameters. Moreover, our CNN‑based architecture substantially outperforms the RNN‑based Onsets&Frames, confirming the efficiency of our design.

A similar trend appears in timbre‑separated transcription in Table 3: our method (‘Ours’) markedly surpasses the RNN‑based baseline of Tanaka et al. (2020) (‘Tanaka’). Notably, during training, both RNN baselines exhibit an earlier rise in validation loss, indicating overfitting to the training set. Given their much larger parameter counts, this suggests limited generalization under data constraints and further underscores the robustness, sample efficiency, and trainability of our lightweight architecture.

4.4 Ablation study on core components

EnergyNorm Comparing rows ‘Ours’ and ‘noLog’ in Table 2, we find that nonnegative features indeed constrain the model’s expressive capacity. We further compare against BatchNorm (row ‘BN’), which is used in BasicPitch. While ‘BN’ slightly outperforms our method on BACH10, it underperforms significantly on the other two test sets. Inspired by Esaki et al. (2024), we hypothesize that our synthetic data distribution aligns more closely with BACH10, and because BatchNorm stores learned statistics for inference, it suffers from poor generalization to more divergent domains, highlighting the advantage of our normalization strategy in cross‑dataset robustness.

Dilated convolution Replacing BasicPitch’s big kernel with our frequency‑dilated kernel (rows ‘Ours’ and ‘Conv39’) yields slightly better performance despite fewer parameters, confirming the benefit of an expanded receptive field. Moreover, non‑dilated variants consistently require higher thresholds (not shown), likely because they cannot directly model octave‑spanning context, leading to stronger harmonic ghosts (false positives) that must be suppressed post‑hoc.

Focal loss While the sophisticated postprocessing in BasicPitch may attenuate the undesirable impact of the blurred onset predictions produced by its original loss (Figure 2), such mitigation is inherently limited. Table 2 reveals that models trained with focal loss (‘Ours’, ‘${\text{BP}}^{\text{fl}}$’, ‘${\text{OF}}^{\text{fl}}$’) consistently achieve significantly higher $F_N$ than their counterparts without it, demonstrating that accurate onset estimation remains crucial for high‑quality note creation.

InfoNCE loss Comparing rows ‘Ours’ and ‘MSE’ in Table 3, which correspond to $L_{\text{InfoNCE}}$ and $L_{\text{affinity}}$, respectively, shows that $L_{\text{InfoNCE}}$ consistently and significantly outperforms $L_{\text{affinity}}$.

Postprocessing As shown in Table 3, $F_{FS}$ is almost always lower than $F_S$, indicating that our postprocessing method is not only faster but also yields better performance. This is also illustrated in the first row of Figure 5, where note‑level aggregation yields more separable embeddings, significantly improving clustering robustness.

Figure 5

4.5 Exploratory and negative results analysis

4.6 Efficiency

Our model is simple and lightweight enough to run directly in a web browser. On an Intel^® Ultra 7 255H CPU using Microsoft Edge, timbre‑agnostic transcription of a 301‑s duet takes 17.1 s, compared to 39.1 s for BasicPitch. The postprocessing step requires 78 ms. For timbre‑separated transcription, our model inference takes 38.8 s, followed by 704 ms for clustering‑based postprocessing. In contrast, RNN‑based baselines suffer from ONNX export issues and have prohibitively large parameter counts, rendering them impractical for real‑world deployment.