X-axis: time (in s). Y-axis: perceived pitch (in semitones, ST); dotted grid lines are 2 ST apart.
The vertical dotted lines show segmentation boundaries.
Click on the picture to see a full size, high resolution version in GIF file format.
Large rich
The large format adds the calibration of X and Y axes (in ST, relative to 1 Hz).
Rich format adds intensity (thin green line) and F0 (thick blue line) on a ST scale, for validating the results.
Automatic segmentation
This prosogram uses automatic segmentation based on intensity of band-pass filtered signal.
Since the early days of intonation research, automatic transcription of the intonation
in speech corpora has been on the wish list of many a researcher in
phonetics, linguistics, and discourse analysis.
Most phoneticians use the fundamental frequency (F0) to represent pitch contours in speech. F0 is an acoustic parameter;
it provides useful information about the acoustic properties of the speech signal.
But it certainly is not the most accurate representation of the
intonation contour as it is perceived by human listeners.
In the seventies pitch contour stylization was introduced as a way to simplify
the F0 curve to those aspects which are relevant for speech communication.
The approach originates from work by J. 't Hart, R. Collier, and A. Cohen at IPO,
Eindhoven, and was further improved by D. Hermes in the '80 and '90.
Other types of stylization have been proposed, such as the
Momel system
by D. Hirst, R. Espesser from Aix-en-Provence.
However, most of these stylization approaches are based on statistical or mathematical
properties of the F0 data and mostly ignore the facts of pitch perception.
It is well known that the auditory perception of pitch variations depends
on many factors other than F0 variation itself.
In 1995 a stylization based on the simulation of tonal perception
was proposed by Ch. d'Alessandro & P. Mertens
(Mertens & d'Alessandro, 1995,
d'Alessandro & Mertens, 1995).
The purpose of this stylization is to provide a curve which approximates the
auditory image in the listener's mind.
This tonal perception model was validated in listening experiments using stimuli
resynthesized using the stylized contour
(Mertens et al, 1997).
This same approach may be used to obtain a transcription
of intonation.
This requires a segmentation of the speech signal into syllable-sized units,
that are motivated by phonetic, acoustic or perceptual properties.
The prosogram may be used in conjunction with various types of segmentation:
an automatic segmentation into local peaks in the intensity of the band-pass filtered speech signal;
a (manual) segmentation into vowels, stored in an phonetic alignment file (Praat's TextGrid files);
a (manual) segmentation into syllables;
a (manual) segmentation into syllable rimes;
any segmentation provided by an external program.
The stylization is applied to the F0 curve of those segmented units, which are approximations of the syllabic nuclei.
Obtain a segmentation into units of the types indicated above.
Select the relevant units (e.g. vowels, syllables).
Select the voiced portion of these units, that has sufficient intensity/loudness
(using difference thresholds relative to the local peak).
Stylize the F0 of the selected time intervals.
Determine pitch range used in speech fragment.
Plot stylized pitch and some annotation tiers (text, phonetic transcription).
Use a musical (semitone) scale and add calibration lines
at every 2 ST for easy interpretation of pitch intervals.
The system is implemented as a Praat script.
Praat is a tool for acoustic and phonetic research,
written by Paul Boersma and David Weenink, of the Institute of Phonetic Sciences in Amsterdam.
The choice of Praat is motivated by the fact that
it is powerful, user-friendly, programmable, freely available,
running on many platforms, and actively maintained.
automatic segmentation The automatic segmentation mode of Prosogram does not require a preliminary segmentation
into sounds or syllables.
This kind of segmentation is based on acoustic parameters
or on an estimation of perceived attributes of the signal.
The current version of the prosogram uses a segmentation based on loudness,
computed from the cochleagram.
It does not identify the underlying speech sounds.
manual segmentation It can be obtained using Praat and will be stored in a "TextGrid" file.
semi-automatic segmentation Alternatively the segmentation can be obtained semi-automatically,
using automatic alignement with some phonetic transcription of the speech signal.
The alignment will be based either on automatic speech recognition (ASR), or on
speech synthesis.
The phonetic transcription can be obtained either manually, or using
grapheme-to-phoneme conversion and natural language processing (NLP).
Jean-Philippe Goldman provides a phonetic alignment system
Easy Align, based on ASR and grapheme-to-phoneme conversion.
A small corpus of spoken French was processed
to illustrate the results obtained with the transcription tool.
The corpus consists of about 4 minutes of an interview between Fayard and
Benoîte Groult broadcasted on Radio de la Suisse Romande.
Some F0 variations are clearly perceived as rises or falls; others go unnoticed
unless after repeated listening; still others are simply not perceived at all.
Indeed, tonal perception depends upon several factors.
The auditory threshold for pitch variation, known as the glissando
threshold G. It depends on the amplitude (extent) and the duration of the
F0 variation. Since the work of J. 't Hart, it is usually expressed in ST/s (semitones per second).
In hearing experiments using short stimuli, either pure tones or speech-like signals,
with repeated presentations,
a glissando threshold G = 0.16/T^2 was measured.
Changes in the spectral properties of the signal tend to function as boundaries
(House, 1990), breaking up a voiced continuum into a sequence of syllabic nuclei.
Changes in signal amplitude tend to function as boundaries.
The presence of a pause following the F0 variation lowers the threshold for the
perception of that variation (House, 1995).
A change in slope is perceived provided it is sufficiently large.
This is called the differential glissando threshold.
Our approach to stylization takes into account
the segmentation into syllabic (or vocalic) nuclei, due to spectral and amplitude changes,
the glissando threshold,
the differential glissando threshold.
The two latter thresholds are model parameters, which can be adjusted.
The stylization will show the effect on the estimated perceived pitch contour.
This is shown in the next sample, which compares the F0 curve and two
stylization variants: the first with G=0.16/T^2,
the second with G=0.32/T^2, i.e. a glissando threshold twice as high.
The (intravocalic) pitch movements found on "les", "chefs" and "gieux", in the case of G=0.16/T^2,
no longer appear in the stylization with G=0.32/T^2.
In normal conversation, utterances are heard only once.
Given the continuous flow of speech, the listener has no time to reflect on the
auditory properties of the signal.
How then should the glissando parameter be chosen in order to obtain
a correct representation of pitch perception in continuous speech ?
The intonation of Groult corpus has been transcribed by ear by two
expert transcribers.
This auditory transcription was compared with various stylizations using
different settings of the stylization parameters.
The setting with G=0.32/T^2 better matches the performance of the human transcribers.
Mertens, Piet (2004)
Un outil pour la transcription de la prosodie dans les corpus oraux.
Traitement Automatique des langues 45 (2), 109-130.
PDF
Mertens, Piet (2004)
The Prosogram : Semi-Automatic Transcription of Prosody based on a Tonal Perception Model.
in
B. Bel & I. Marlien (eds.)
Proceedings of Speech Prosody 2004, Nara (Japan), 23-26 March. (ISBN 2-9518233-1-2)
PDF
Mertens, Piet (2004)
Le prosogramme : une transcription semi-automatique de la prosodie.
Cahiers de l'Institut de Linguistique de Louvain 30, 1-3, 7-25
Other references
Alessandro, C. d'; Mertens, P. (1995)
Automatic pitch contour stylization using a model of tonal perception.
Computer Speech and Language 9(3), 257-288.
Campione, Estelle & Véronis, Jean (2001)
Etiquetage prosodique semi-automatique des corpus oraux.
Actes TALN 2001, 123-132
Hart, J. 't (1974) Discriminability of the size of pitch movements in speech.
I.P.O. Annual Progress Report 9, 56-63.
Hart, J. 't (1976) Psychoacoustic backgrounds of pitch contour stylisation.
IPO-APR 11, 11-19.
Hart, J. 't (1979a)
Explorations in automatic stylization of F0 curves.
IPO-APR 14, 61-65.
Hart, J. 't (1981)
Differential sensivity to pitch distance, particularly in speech.
JASA 69(3), 811-821
Hart, J. 't; Collier, R. & Cohen, A. (1990)
A perceptual study of intonation.
Cambridge Studies in Speech Science and Communication.
Cambrigde: Cambridge Univ. Press, 227 pp.
Hermes, D. (2006)
Stylization of pitch contours
in:
Sudhoff, Stefan; Lenertová, Denisa; Meyer, Roland; Pappert, Sandra; Augurzky, Petra; Mleinek, Ina; Richter, Nicole; Schließer, Johannes (eds) (2006)
Methods in Empirical Prosody Research. De Gruyter.
pp. 29-61.
Hermes, D.J. & Gestel, J.C. van (1991)
The frequency scale of speech intonation.
JASA 90(1), 97-102
Hirst, D. and Espesser, R. (1993)
Automatic Modelling of Fundamental Frequency Using a Quadratic Spline Function.
Travaux de l'Institut de Phonétique d'Aix-en-Provence, 15, 75-85, 1993.
House, David (1990) Tonal Perception in Speech. Lund: Lund Univ. Press.
House, David (1995)
The influence of silence on perceiving the preceding tonal contour.
Proc. Int. Congr. Phonetic Sciences 13, vol. 1, 122-125 (Stockholm 1995)
Mertens, P. (1987a) L'intonation du français. De la description linguistique à la reconnaissance automatique.
Unpublished Ph.D. (University of Leuven, Belgium), 2 vol., pp. 317 + 90.
Mertens, P. (1987b) Automatic segmentation of speech into syllables.
Proceedings of the European Conference on Speech Technology. Edinburgh.
Laver & Jack (eds) (1987), vol. II, 9-12.
Mertens, P. (1989) Automatic recognition of intonation in French and Dutch.
Eurospeech 89, vol 1, 46-50 (Paris, September 1989, J.P. Tubach & J.J. Mariani (editors))
Mertens, P. & Alessandro, Ch. d' (1995)
Pitch contour stylization using a tonal perception model.
Proc. Int. Congr. Phonetic Sciences 13, 4, 228-231 (Stockholm 1995)
Mertens, P.; Beaugendre, F. & Alessandro, Ch. d' (1997)
Comparing approaches to pitch contour stylization for speech synthesis.
in Santen, J.P.H. van; Sproat, Richard W.; Olive, Joseph P. & Hirschberg, Julia (eds) (1997) Progress in Speech Synthesis. p. 347-363. N.Y.: Springer Verlag
Rietveld, A.C.M. (1984)
Syllaben, klemtonen en de automatische detectie van beklemtoonde syllaben in het Nederlands.
Ph. D., University of Nijmegen, 262pp.
Rossi, M. (1971a)
Le seuil de glissando ou seuil de perception des variations tonales pour la parole.
Phonetica 23, 1-33
Rossi, M. (1978a)
La perception des glissandos descendants dans les contours prosodiques.
Phonetica 35(1), 11-40
Rossi, M. (1978b)
The perception of non-repetitive intensity glides on vowels.
Journal of Phonetics 6(1), 9-18
Rossi, M. (1978c)
Interactions of intensity glides and frequency glissandos.
Language & Speech 21, 384-396
Rossi, M. (1979)
Les configurations et l'interaction des pentes de F0 et de I.
PICPS 9(1), 246(A)
Rossi, M.; Di Cristo, A.; Hirst, D.; Martin, Ph. & Nishinuma, Y. (1981)
L'intonation. De l'acoustique à la sémantique.
Paris: Klincksieck, 364 pp.
Spaai, G.W.G.; Storm, A.; Derksen, A.S.; Hermes, D.J. & Gigi E.F. (1993)
An Intonation Meter for teaching intonation to profoundly deaf persons.
IPO MAnuscript no. 968.
Applications
Patel, A.D.; Iversen, J.R.; & Rosenberg, J.C. (2006)
Comparing the rhythm and melody of speech and music: The case of British English and French.
Journal of the Acoustical Society of America, 119:3034-3047.
