From random weights to a voice

Training a model from random weights makes every intermediate representation feel consequential. With speech, the chain is especially tangible: a sequence of characters becomes a pattern of frequencies, then a pressure wave you can hear.

RESONANCE is my small laboratory for that chain. The acoustic model and vocoder were trained from random initialization on LJ Speech. The public exhibit preserves six generated phrases, a training figure, and a view of the research instrument. Together they make it possible to inspect the project at several levels: what went in, what the network represented, and what came out.

The most useful question is where the system learned something useful and where the pipeline still needs work. Speech has an unusually unforgiving final interface. A sound can be plausible while a word is wrong. A spectral loss can improve while timing remains awkward. Listening and measurement have to stay connected.

From characters to a sound field

The first stage maps a character sequence to an 80-band mel spectrogram. Think of that representation as a time-by-frequency description of the sound: each frame summarizes how much energy lies in each of 80 frequency bands. It is a useful intermediate target because it exposes acoustic structure without asking the text model to directly specify every waveform sample.

The vocoder turns those predicted features into a waveform. This separation gives the experiment a diagnostic advantage. We can ask how well the acoustic model predicts features, how well the vocoder reconstructs sound from real features, and how well the two work together on features the acoustic model actually produces.

This general family of learned speech synthesis has substantial prior work. Tacotron, for example, demonstrated character-conditioned speech synthesis trained from random initialization. That is methodological context for this project, rather than a claim that RESONANCE reproduces its architecture or reported quality. [1]

Listen to the checkpoint

The following clips are saved outputs from acoustic checkpoint step 17,500 and vocoder checkpoint step 20,000. Playback uses the existing files. No live model inference happens in this article.

These examples establish a concrete artifact to discuss. They are selected short generations, so they leave many questions open: longer passages, uncommon names, repeated clauses, punctuation, and unfamiliar word combinations. A short gallery is a starting point for evaluation. It does not supply a population-level quality score.

Read the fine print on a fast training result

The saved figure reports 12,873 accepted clips, batch size four, and fp32 training. It records 10,000 parallel updates in 5.8 minutes, alongside a separate 13.7-minute timing preparation stage and a teacher checkpoint. Those details matter when interpreting the headline. The result describes a particular training stage with prepared information available to it.

The figure also reports a known-duration validation mel L1 of 0.792 and duration mean absolute error of 1.89 frames per character. The first asks how close the predicted features are under a timing condition. The second concerns timing prediction itself. Collapsing them into one statement about speech quality would erase an important distinction in what the model was asked to do.

The plotted text-only sample lasts 6.32 seconds against a 7.03-second reference. That discrepancy is useful evidence about duration, but it does not by itself tell us whether individual words were correctly pronounced or whether listeners found the result natural. Those questions require different measurements.

The intermediate representation is a debugging interface

Suppose an output sounds poor. There are several possible causes: an incorrect text interpretation, an acoustic sequence with the wrong duration, spectral features that miss important detail, or a vocoder that cannot reconstruct its input accurately. An end-to-end audio sample alone cannot localize those errors.

A useful diagnostic sequence holds parts of the pipeline fixed. Feed the vocoder real mel features from a recording. Then feed it predicted features for the same text. Compare the acoustic prediction with the reference using a consistent display scale. Finally, inspect the full text-to-waveform result. The differences help identify which stage deserves the next unit of effort.

The same principle applies far beyond speech. Whenever a system has several learned stages, an interpretable boundary can become a place to measure, substitute, and isolate failures. That boundary has architectural value even if a fully joint model is eventually the better production choice.

What would make the next evaluation stronger?

I would first freeze a prompt set before selecting the next checkpoint. It should span short and long sentences, awkward consonant sequences, numbers, proper names, repeated words, and punctuation changes. The evaluation should preserve every output, including failures, so that the gallery cannot quietly become a selection of the easiest cases.

Then I would separate intelligibility, timing, and perceptual quality. Human transcription can reveal missing or substituted words. Duration and alignment diagnostics can identify instability. Blinded listening comparisons can test preferences without telling listeners which checkpoint is supposed to be better. Latency and training cost belong in the report too, with preparation and upstream stages counted explicitly.

The long-term idea of a controllable vocal instrument is much broader than this experiment. The existing model and gallery provide a narrower foundation: a trained text-to-acoustics-to-waveform pipeline with artifacts that can be heard and examined. That is a useful place from which to ask the next precise question.