Accepted at TMLR / Research project

Human-aligned
evaluation for
speech role-play.

A voice can say the right words and still miss the role. Speech-DRAME studies the emotion, character, and context behind a spoken performance.

Read the paper Explore the code
Speech-DRAME
Character.Voice.Context.
Jiatong Shi1,2, Jionghao Han2, Yichen Lu2, Santiago Pascual2, Pengfei Wu2, Chenye Cui2, Shinji Watanabe1, Chao Weng2, Cong Zhou2
1 Carnegie Mellon University   2 Anuttacon

01 The framework

One performance.
Two perspectives.

DRAME-EvalBench pairs bilingual human judgments with complementary evaluation protocols. DRAME-Eval learns to estimate those judgments from speech and textual context.

Does the voice
fit the role?

Begin with a role and scene. Generate speech, then ask human raters whether it sounds natural, clear, and appropriate for the character.

  1. 01Role & sceneSet the character
  2. 02Generated speechPerform the role
  3. 03Human judgmentsEvaluate the fit

What is scored Audio quality, human likeness, and appropriateness, with a separate content-pass check.

8,272Archetype records

15,414Realism records

EN + ZHBilingual human annotation

Experimental corpus counts. The public release is limited to the authorized subset; released counts are still being finalized.

02 The findings

Alignment has
its limits.

DRAME-Eval aligns strongly with human judgments on the benchmark distribution. Authentic human acting tells a different story.

Archetype

0.629

Benchmark distribution
English + Mandarin pooled

Realism

0.602

Benchmark distribution
English + Mandarin pooled

Authentic human acting

−0.142

1,084 clean held-out recordings
Mandarin only

The synthetic-to-authentic gap

Within-speaker correlation on authentic speech is only +0.029, compared with matched human single-rater agreement of +0.34. Strong benchmark alignment does not establish reliable discrimination of acting differences among human performances.

Values are mean Pearson correlations under masking for unavailable labels. Pooled bilingual correlations can reflect between-language effects; see the paper for per-language results and uncertainty. The authentic-speech pool contains 1,080 adaptation and 1,084 held-out test records; the entire pool is not a test set.

03 Explore & use

The work,
in the open.

Evaluation code on GitHub

Two public Hugging Face pages, two distribution formats. The planned releases share authorized record IDs and experimental splits.

Data release in preparation. Both dataset pages currently contain introductory information only. Benchmark data are not yet uploaded.

01 / Feature edition

Anuttacon

Metadata + mel features

Authorized benchmark metadata, mel-spectrogram features, and reconstruction instructions.
02 / With playback files

ftshijt

Metadata + mel + reconstructions

The same authorized records and features, plus permitted pre-rendered vocoder reconstructions.

Approved dataset contributions will use CC BY-NC 4.0, with applicable source notices retained. Releases are limited to granted sources, records, and representations. Neither includes source waveforms or extra development data. Vocoder reconstructions are model-generated rather than original recordings. The code uses Apache 2.0.

04 Reference

Cite the work.

@misc{shi_speech_drame,
  title = {Speech-DRAME: Human-Aligned Evaluation for Speech Role-Play},
  author = {Shi, Jiatong and Han, Jionghao and Lu, Yichen and
            Pascual, Santiago and Wu, Pengfei and Cui, Chenye and
            Watanabe, Shinji and Weng, Chao and Zhou, Cong},
  note = {Accepted at Transactions on Machine Learning Research},
  url = {https://openreview.net/forum?id=m9IP9ECrk2}
}

Citation details will be updated with the final publication record.