Prosogram is a tool for the analysis and transcription of pitch variations in speech.
Its stylization simulates the auditory perception of pitch by the listener.
A key element in tonal perception is the segmentation of speech into syllable-sized elements, resulting from changes in the spectrum (sound timbre) and intensity.
The tool also provides measurements of prosodic features for individual syllables (such a duration, pitch, pitch movement direction and size), as well as prosodic properties of longer stretches of speech (such as speech rate, proportion of silent pauses, pitch range, and pitch trajectory).
The tool can easily interact with other software tools.
It is used as the first step in automatic phonological transcription of intonation, the detection of sentence stress and intonation boundaries.
The first illustration shows a light Prosogram with the stylization (black lines) and the pitch range (red horizontal lines indicating top, median and bottom).
The annotations of sounds, syllables and words are provided by the corpus.
Wide, light, with pitch range
The next illustration shows a rich Prosogram, which adds the parameters of F0 (blue line), intensity (green line), and voicing, as well as the segmentation, and the calibration of X and Y axes (in ST relative to 1 Hz, and in Hz).
The vertical dotted lines correspond to the segmentation boundaries in the annotation.
The third illustration shows a light Prosogram, in a more compact size.
The next figure shows a Prosogram using automatic segmentation into syllable-sized units.
Thy magenta curve shows the intensity of the band-pass filtered speech signal, on which this segmentation is based.
The last figure shows the screen of the interactive Prosogram. Here the user can interactively browse the speech signal and its stylization, play back parts (syllables, words...), and resynthesize the signal with the stylized pitch.
(The tonal annotation in tier "polytonia" is obtained using the Polytonia script, which is not part of Prosogram.)
Many phoneticians use the fundamental frequency (F0) curve to represent pitch contours in speech.
F0 is an acoustic parameter; it provides useful information about the acoustic properties of the speech signal.
But it certainly is not the most accurate representation of the intonation contour as it is perceived by human listeners.
In the seventies, pitch contour stylization was introduced as a way to simplify the F0 curve to those aspects which are potentially relevant for speech communication.
The approach originates from work by J. 't Hart, R. Collier, and A. Cohen at the I.P.O. (Institute for Perception Research) in Eindhoven, and was further improved by D. Hermes in the '80 and '90.
Other types of stylization have been proposed, such as the
by D. Hirst, R. Espesser (1993) from Aix-en-Provence.
However, most of these stylization approaches are based on statistical or mathematical properties of the F0 data and ignore the facts of pitch perception.
It is well known that the auditory perception of pitch variations depends on many factors other than F0 variation itself.
In 1995 a stylization based on the simulation of tonal perception was proposed by Ch. d'Alessandro & P. Mertens
(Mertens & d'Alessandro, 1995,
d'Alessandro & Mertens, 1995).
The purpose of this stylization is to provide a representation which approximates the image in the listener's auditory memory.
This tonal perception model was validated in listening experiments using stimuli resynthesized using the stylized contour (Mertens et al, 1997).
This approach may be used to obtain a low-level transcription of pitch level and pitch movement and.
It requires a segmentation of the speech signal into syllable-sized units, motivated by phonetic, acoustic or perceptual properties.
The Prosogram uses various types of segmentation:
an automatic segmentation into local peaks of intensity (both that of the band-pass filtered speech signal and that of the full band signal);
a (manual) segmentation into vowels or phones (phonetic alignment) stored in an annotation file (Praat's TextGrid file);
a (manual) segmentation into syllables;
a (manual) segmentation into syllable rhymes;
a segmentation provided by an external program.
The stylization is applied to the F0 curve of those segmented units, which are approximations of the more sonorous part of the syllable.
Calculate acoustic the parameters: F0, intensity, intensity of band-pass filtered speech, voicing (V/UV).
Obtain a segmentation into units of the type indicated above.
Select the relevant unit (e.g. vowel, rhyme, syllable, voiced portions).
Select the voiced portion of these units, that has sufficiently high intensity (using difference thresholds relative to the local peak).
Stylize the F0 of the selected time intervals.
Detect silent pauses.
Determine the pitch range used by the speaker.
Plot the result: parameters, stylized pitch, segmentation, and some annotation tiers (text, phonetic transcription, etc.).
The system is implemented as a Praat script.
Praat is a tool for acoustic and phonetic research,
written by Paul Boersma and David Weenink, of the Institute of Phonetic Sciences in Amsterdam.
The choice of Praat is motivated by the fact that it is powerful, user-friendly, programmable, freely available, running on many platforms, and actively maintained.
A suitable segmentation can be obtained in various ways.
Automatic segmentation The automatic segmentation mode of Prosogram does not require a preliminary segmentation into sounds or syllables. It is based on the acoustic signal only.
The automatic segmentation in Prosogram detects the local peaks in the intensity of the band-pass filtered speech signal, and adjusts their boundaries using the full band intensity, voicing and F0 discontinuities.
(An earlier implementation uses the loudness, computed from the cochleagram. But this parameter introduces a smoothing which masks syllable boundaries at sonorous segments, such as nasals and glides.)
The automatic segmentation does not identify the speech sounds.
Manual segmentation It can be made interactively using Praat and will be stored in a TextGrid file.
Semi-automatic segmentation using automatic phonetic alignment The segmentation can be obtained semi-automatically, using automatic alignement between a phonetic transcription of the utterance and the speech signal.
This alignment will be based either on automatic speech recognition (ASR), or on speech synthesis.
The phonetic transcription which is used in the alignment procedure can be obtained either manually, or using grapheme-to-phoneme conversion and natural language processing (NLP). In the latter case, an orthographic transcription of the words in the utterance is required.
Several tools for automatic alignment are available.
A small corpus of spoken French was processed to illustrate the results obtained with the transcription tool.
The corpus consists of about 4 minutes of an interview between Fayard and Benoîte Groult broadcasted on Radio de la Suisse Romande.
Some F0 variations are clearly perceived as rises or falls; others go unnoticed
unless after repeated listening; still others are simply not perceived at all.
Indeed, tonal perception depends upon several factors.
The auditory threshold for pitch variation, or glissando threshold G, specifies the minimal pitch interval required for a pitch variation of a given duration to be perceived as a changing pitch, rather than as a level tone. It depends on the size and duration of the F0 variation. Since the work of J. 't Hart, it is usually expressed in ST/s (semitones per second).
In hearing experiments using short stimuli, either pure tones or speech-like signals,
with repeated presentations, a threshold G = 0.16/T^2 was measured.
Major changes in the spectral properties of the signal tend to function as boundaries (House, 1990), breaking up a voiced continuum into a sequence of smaller parts corresponding to syllabic nuclei.
Changes in signal amplitude tend to function as boundaries.
The presence of a pause following the F0 variation lowers the threshold for the perception of that variation (House, 1995).
A change in slope is perceived provided it is sufficiently large. This is called the differential glissando threshold DG.
Our approach to stylization takes into account
the segmentation into syllabic nuclei (high intensity region within the rhyme), due to spectral and amplitude changes,
the glissando threshold,
the effect of pause presence on the glissando threshold,
the differential glissando threshold,
the minimal duration for a plateau in a complex pitch movement.
The stylization shows the effect of a change of the model parameters on the estimated perceived pitch contour.
This is shown in the next sample, which compares the F0 curve and two stylization variants: the first with G=0.16/T^2, the second with G=0.32/T^2, i.e. a glissando threshold twice as high.
The (intravocalic) pitch movements found on "chefs" and "gieux", in the case of G=0.16/T^2, no longer appear in the stylization with G=0.32/T^2.
In speech communication, utterances are heard only once.
The listener has no time to reflect on the auditory properties of the signal. This differs from the situation of a hearing experiment, where a stimulus is usually repeated several times ans separated by silent pauses.
How then should the glissando parameter be chosen in order to obtain a correct representation of pitch perception in continuous speech?
By using (TD-PSOLA) resynthesized utterances in combination with stylizations for alternative parameter settings and presenting them to listeners together with a resynthesis of the original utterance, one can determine the glissando threshold for which listeners are unable to distinguish the stylized pitch contour from the original one.
The setting with G=0.32/T^2 matches the performance of the listeners in continuous speech.
To take into account the impact of a silent pause on the perception of the preceding pitch movement, the glissando threshold may be adjusted dynamically, depending on the presence of a pause.
The stylization by Prosogram has been used for automatic transcription of pitch contours and intonation.
A first type, called Polytonia (Mertens, 2014), indicates the pitch level and pitch movement of each syllable. Pitch levels are determined on the basis either of the speaker's pitch range and pitch intervals in the local context of a syllable ad within the syllable.
A second type identifies positions in prosodic structure, such as stressed syllable, pre-stress syllable, and prosodic boundary, and reinterprets Polytonia's pitch levels and movements in terms of such positions.
This approach is called ToPPos (for Tones on Prosodic Positions) (Mertens, to appear).
Mertens, Piet (to appear)
The Prosogram model for pitch stylization and its applications in intonation transcription.
in Barnes, J.A. and Shattuck-Hufnagel, S. (eds) Prosodic Theory and Practice. Cambridge, MA: MIT Press.
Mertens, Piet (2014)
Polytonia: a system for the automatic transcription of tonal aspects in speech corpora.
Journal of Speech Sciences 4 (2), 17-57.
Mertens, Piet (2004)
Un outil pour la transcription de la prosodie dans les corpus oraux.
Traitement Automatique des langues 45 (2), 109-130.
Mertens, Piet (2004)
The Prosogram : Semi-Automatic Transcription of Prosody based on a Tonal Perception Model.
B. Bel & I. Marlien (eds.)
Proceedings of Speech Prosody 2004, Nara (Japan), 23-26 March. (ISBN 2-9518233-1-2)
Mertens, Piet (2004)
Le prosogramme : une transcription semi-automatique de la prosodie.
Cahiers de l'Institut de Linguistique de Louvain 30, 1-3, 7-25
Alessandro, C. d'; Mertens, P. (1995)
Automatic pitch contour stylization using a model of tonal perception.
Computer Speech and Language 9(3), 257-288.
Campione, Estelle & Véronis, Jean (2001)
Etiquetage prosodique semi-automatique des corpus oraux.
Actes TALN 2001, 123-132
Hart, J. 't (1974) Discriminability of the size of pitch movements in speech.
I.P.O. Annual Progress Report 9, 56-63.
Hart, J. 't (1976) Psychoacoustic backgrounds of pitch contour stylisation.
IPO-APR 11, 11-19.
Hart, J. 't (1979a)
Explorations in automatic stylization of F0 curves.
IPO-APR 14, 61-65.
Hart, J. 't (1981)
Differential sensivity to pitch distance, particularly in speech.
JASA 69(3), 811-821
Hart, J. 't; Collier, R. & Cohen, A. (1990)
A perceptual study of intonation.
Cambridge Studies in Speech Science and Communication.
Cambrigde: Cambridge Univ. Press, 227 pp.
Hermes, D. (2006)
Stylization of pitch contours
Sudhoff, Stefan; Lenertová, Denisa; Meyer, Roland; Pappert, Sandra; Augurzky, Petra; Mleinek, Ina; Richter, Nicole; Schließer, Johannes (eds) (2006)
Methods in Empirical Prosody Research. De Gruyter.
Hermes, D.J. & Gestel, J.C. van (1991)
The frequency scale of speech intonation.
JASA 90(1), 97-102
Hirst, D. and Espesser, R. (1993)
Automatic Modelling of Fundamental Frequency Using a Quadratic Spline Function.
Travaux de l'Institut de Phonétique d'Aix-en-Provence, 15, 75-85, 1993.
House, David (1990) Tonal Perception in Speech. Lund: Lund Univ. Press.
House, David (1995)
The influence of silence on perceiving the preceding tonal contour.
Proc. Int. Congr. Phonetic Sciences 13, vol. 1, 122-125 (Stockholm 1995)
Mertens, P. (1987a) L'intonation du français. De la description linguistique à la reconnaissance automatique.
Unpublished Ph.D. (University of Leuven, Belgium), 2 vol., pp. 317 + 90.
Mertens, P. (1987b) Automatic segmentation of speech into syllables.
Proceedings of the European Conference on Speech Technology. Edinburgh.
Laver & Jack (eds) (1987), vol. II, 9-12.
Mertens, P. (1989) Automatic recognition of intonation in French and Dutch.
Eurospeech 89, vol 1, 46-50 (Paris, September 1989, J.P. Tubach & J.J. Mariani (editors))
Mertens, Piet (2013)
Automatic labelling of pitch levels and pitch movements in speech corpora
in Bigi, Brigitte & Hirst, Daniel (2013) Proceedings TRASP 2013, Tools and Resources for the Analysis of Speech Prosody. (Aix-en-Provence, August 30, 2013), pp. 42-46. ISBN 978-2-7466-6443-2.
Mertens, P. & Alessandro, Ch. d' (1995)
Pitch contour stylization using a tonal perception model.
Proc. Int. Congr. Phonetic Sciences 13, 4, 228-231 (Stockholm 1995)
Mertens, P.; Beaugendre, F. & Alessandro, Ch. d' (1997)
Comparing approaches to pitch contour stylization for speech synthesis.
in Santen, J.P.H. van; Sproat, Richard W.; Olive, Joseph P. & Hirschberg, Julia (eds) (1997) Progress in Speech Synthesis. p. 347-363. N.Y.: Springer Verlag
Rietveld, A.C.M. (1984)
Syllaben, klemtonen en de automatische detectie van beklemtoonde syllaben in het Nederlands.
Ph. D., University of Nijmegen, 262pp.
Rossi, M. (1971a)
Le seuil de glissando ou seuil de perception des variations tonales pour la parole.
Phonetica 23, 1-33
Rossi, M. (1978a)
La perception des glissandos descendants dans les contours prosodiques.
Phonetica 35(1), 11-40
Rossi, M. (1978b)
The perception of non-repetitive intensity glides on vowels.
Journal of Phonetics 6(1), 9-18
Rossi, M. (1978c)
Interactions of intensity glides and frequency glissandos.
Language & Speech 21, 384-396
Rossi, M. (1979)
Les configurations et l'interaction des pentes de F0 et de I.
PICPS 9(1), 246(A)
Rossi, M.; Di Cristo, A.; Hirst, D.; Martin, Ph. & Nishinuma, Y. (1981)
L'intonation. De l'acoustique à la sémantique.
Paris: Klincksieck, 364 pp.
Spaai, G.W.G.; Storm, A.; Derksen, A.S.; Hermes, D.J. & Gigi E.F. (1993)
An Intonation Meter for teaching intonation to profoundly deaf persons.
IPO MAnuscript no. 968.
Patel, A.D.; Iversen, J.R.; & Rosenberg, J.C. (2006)
Comparing the rhythm and melody of speech and music: The case of British English and French.
Journal of the Acoustical Society of America, 119:3034-3047.
Page created: 2002-06-20. Last updated: 2016-07-06.