Dr. Cong Zhang

Selected publications

1. remote

   Investigating differences in lab-quality and remote recording methods with dynamic acoustic measures

   Cong Zhang, Kathleen Jepson, and Yu-Ying Chuang

   Laboratory Phonology, 2024

   Abs Bib PDF

   Increasingly, phonetic research utilizes data collected from participants who record themselves on readily available devices. Though such recordings are convenient, their suitability for acoustic analysis remains an open question, especially regarding how the individual methods affect acoustic measures over time. We used Quantile Generalized Additive Mixed Models (QGAMMs) to analyze measures of F0, intensity, and the first and second formants, comparing files recorded using a laboratory-standard recording method (Zoom H6 Recorder with an external microphone), to three remote recording methods, (1) the Awesome Voice Recorder application on a smartphone (AVR), (2) the Zoom meeting application with default settings (Zoom-default), and (3) the Zoom meeting application with the "Turn on Original Sound" setting (Zoom-raw). A linear temporal alignment issue was observed for the Zoom methods over the course of the long, recording session files. However, the difference was not significant for utterance-length files. F0 was reliably measured using all methods. Intensity and formants presented non-linear differences across methods that could not be corrected for simply. Overall, the AVR files were most similar to the H6’s, and so AVR is deemed to be a more reliable recording method than either Zoom-default or Zoom-raw.

   @article{zhang2024investigating,
     title = {Investigating differences in lab-quality and remote recording methods with dynamic acoustic measures},
     author = {Zhang, Cong and Jepson, Kathleen and Chuang, Yu-Ying},
     year = {2024},
     journal = {Laboratory Phonology}
   }

2. AMPPS

   Multidimensional signals and analytic flexibility: Estimating degrees of freedom in human speech analyses

   Ste Coretta, Joseph V Casillas,  ..., Cong Zhang,  ..., and Timo B Roettger

   Advances in Methods and Practices in Psychological Sciences, 2023

   Abs Bib HTML PDF

   Recent empirical studies have highlighted the large degree of analytic flexibility in data analysis which can lead to substantially different conclusions based on the same data set. Thus, researchers have expressed their concerns that these researcher degrees of freedom might facilitate bias and can lead to claims that do not stand the test of time. Even greater flexibility is to be expected in fields in which the primary data lend themselves to a variety of possible operationalizations. The multidimensional, temporally extended nature of speech constitutes an ideal testing ground for assessing the variability in analytic approaches, which derives not only from aspects of statistical modeling, but also from decisions regarding the quantification of the measured behavior. In the present study, we gave the same speech production data set to 46 teams of researchers and asked them to answer the same research question, resulting in substantial variability in reported effect sizes and their interpretation. Using Bayesian meta-analytic tools, we further find little to no evidence that the observed variability can be explained by analysts’ prior beliefs, expertise or the perceived quality of their analyses. In light of this idiosyncratic variability, we recommend that researchers more transparently share details of their analysis, strengthen the link between theoretical construct and quantitative system and calibrate their (un)certainty in their conclusions.

   @article{coretta2023multidimensional,
     title = {Multidimensional signals and analytic flexibility: Estimating degrees of freedom in human speech analyses},
     author = {Coretta, Ste and Casillas, Joseph V and ... and Zhang, Cong and ... and Roettger, Timo B},
     journal = {Advances in Methods and Practices in Psychological Sciences},
     year = {2023}
   }

3. ICPhS

   Language redundancy effects on F0: A preliminary controlled study

   Cong Zhang, Catherine Lai, Ricardo Souza, Alice Turk, and Tina Bögel

   In Proceeding of 20th International Congress of Phonetic Sciences, 2023

   Abs Bib HTML PDF Presentation

   Previous research suggests that words with a high level of language redundancy (i.e. recognition likelihood from familiarity and predictability based on syntactic, pragmatic, and semantic factors) have reduced acoustic salience, such as shorter duration and reduced vowels. The Smooth Signal Redundancy Hypothesis proposes that acoustic salience is controlled via prosodic structure, and makes the prediction that parameters such as fundamental frequency should also be affected by language redundancy. This study investigates the relationship of F0 with lexical frequency, together with bigram (verb-adjective or adjective-noun) frequency and the ratio between these two bigram frequencies. Results from a carefully controlled experiment with quadruplets of minimal pairs suggests that language redundancy can affect fundamental frequency in English.

   @inproceedings{zhang2023language-redundancy,
     title = {Language redundancy effects on F0: A preliminary controlled study},
     author = {Zhang, Cong and Lai, Catherine and Napoleão de Souza, Ricardo and Turk, Alice and Bögel, Tina},
     booktitle = {Proceeding of 20th International Congress of Phonetic Sciences},
     year = {2023}
   }

4. gamification

   Collecting Big Data Through Citizen Science: Gamification and Game-based Approaches to Data Collection in Applied Linguistics

   Yoolim Kim, Vita V Kogan, and Cong Zhang

   Applied Linguistics, 2023

   Abs Bib HTML PDF

   Gamification of behavioral experiments has been applied successfully to research in a number of disciplines, including linguistics. We believe that these methods have been underutilized in applied linguistics, in particular second-language acquisition research. The incorporation of games and gaming elements (gamification) in behavioral experiments has been shown to mitigate many of the practical constraints characteristic of lab settings, such as limited recruitment or only achieving small-scale data. However, such constraints are no longer an issue with gamified and game-based experiments, and as a result, data collection can occur remotely with greater ease and on a much wider scale, yielding data that are ecologically valid and robust. These methods enable the collection of data that are comparable in quality to the data collected in more traditional settings while engaging far more diverse participants with different language backgrounds that are more representative of the greater population. We highlight three successful applications of using games and gamification with applied linguistic experiments to illustrate the effectiveness of such approaches in a greater effort to invite other applied linguists to do the same.

   @article{kim2023collecting,
     author = {Kim, Yoolim and Kogan, Vita V and Zhang, Cong},
     title = {{Collecting Big Data Through Citizen Science: Gamification and Game-based Approaches to Data Collection in Applied Linguistics}},
     journal = {Applied Linguistics},
     pages = {amad039},
     year = {2023},
     issn = {0142-6001},
     doi = {10.1093/applin/amad039}
   }

5. EMNLP Findings

   Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddings

   Jian Zhu, Zuoyu Tian, Yadong Liu, Cong Zhang, and Chia-wen Lo

   Findings of Empirical Methods in Natural Language Processing, 2022

   Abs Bib HTML PDF Code

   Inducing semantic representations directly from speech signals is a highly challenging task but has many useful applications in speech mining and spoken language understanding. This study tackles the unsupervised learning of semantic representations for spoken utterances. Through converting speech signals into hidden units generated from acoustic unit discovery, we propose WavEmbed, a multimodal sequential autoencoder that predicts hidden units from a dense representation of speech. Secondly, we also propose S-HuBERT to induce meaning through knowledge distillation, in which a sentence embedding model is first trained on hidden units and passes its knowledge to a speech encoder through contrastive learning. The best performing model achieves a moderate correlation (0.5 0.6) with human judgments, without relying on any labels or transcriptions. Furthermore, these models can also be easily extended to leverage textual transcriptions of speech to learn much better speech embeddings that are strongly correlated with human annotations. Our proposed methods are applicable to the development of purely data-driven systems for speech mining, indexing and search.

   @article{zhu2022bootstrapping,
     title = {Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddings},
     author = {Zhu, Jian and Tian, Zuoyu and Liu, Yadong and Zhang, Cong and Lo, Chia-wen},
     year = {2022},
     journal = {Findings of Empirical Methods in Natural Language Processing}
   }

6. rhythm

   Task effect on L2 rhythm production by Cantonese learners of Portuguese

   Yuqi Sun, and Cong Zhang

   DELTA: Documentação de Estudos em Lingüística Teórica e Aplicada, 2022

   Abs Bib HTML PDF Code

   This study examines L2 Portuguese speech produced by eight native Cantonese speakers from Macao, China. The aims of this study are to investigate (1) whether the speech rhythm in L2 Portuguese is more source-like (more similar to Cantonese) or more target-like (more similar to Portuguese), and (2) whether L2 speech rhythm differs across three different tasks: a reading task, a retelling task, and an interpreting task. Seven rhythm metrics, i.e., %V, ΔC, ΔV, VarcoC, VarcoV, rPVI_C, and nPVI_V, were adopted for comparison and investigation. The results showed that L2 Portuguese rhythm produced by Cantonese speakers differed from L1 Portuguese speakers’ rhythm. R-deletion and vowel epenthesis were the reasons for the variabilities and instabilities of L2 Portuguese production by Cantonese learners, as they affect the duration and the number of vowel intervals and consonantal intervals. Moreover, in Cantonese learners’ L2 Portuguese production, the semi-spontaneous tasks (retelling and interpreting) presented a significant difference from the reading task. The driving force for such a difference was the cognitive load behind the tasks.

   @article{sun2022task,
     author = {Sun, Yuqi and Zhang, Cong},
     year = {2022},
     title = {Task effect on L2 rhythm production by Cantonese learners of Portuguese},
     volume = {38},
     issn = {0102-4450},
     doi = {10.1590/1678-460X202258943},
     number = {3},
     journal = {DELTA: Documentação de Estudos em Lingüística Teórica e Aplicada},
     publisher = {Pontifícia Universidade Católica de São Paulo - PUC-SP},
     pages = {202258943}
   }

7. Interspeech

   ByT5 model for massively multilingual grapheme-to-phoneme conversion

   Jian Zhu*, Cong Zhang*, and David Jurgens [* equal contribution]

   In Interspeech 2022, 2022

   Abs Bib PDF Code

   In this study, we tackle massively multilingual grapheme-to-phoneme conversion through implementing G2P models based on ByT5. We have curated a G2P dataset from various sources that covers around 100 languages and trained large-scale multilingual G2P models based on ByT5. We found that ByT5 operating on byte-level inputs significantly outperformed the token-based mT5 model in terms of multilingual G2P. Pairwise comparison with monolingual models in these languages suggests that multilingual ByT5 models generally lower the phone error rate by jointly learning from a variety of languages. The pretrained model can further benefit low resource G2P through zero-shot prediction on unseen languages or provides pretrained weights for finetuning, which helps the model converge to a lower phone error rate than randomly initialized weights. To facilitate future research on multilingual G2P, we make available our code and pretrained multilingual G2P models at: https://github.com/lingjzhu/CharsiuG2P.

   @inproceedings{zhu2022byt5-g2p,
     title = {ByT5 model for massively multilingual grapheme-to-phoneme conversion},
     author = {{Jian Zhu*, Cong Zhang*} and {[* equal contribution]}, David Jurgens},
     year = {2022},
     booktitle = {Interspeech 2022}
   }

8. Interspeech

   Synchronising Speech Segments with Musical Beats in Mandarin and English Singing

   Cong Zhang, and Jian Zhu

   In Interspeech 2021, 2021

   Abs Bib HTML PDF Code

   Generating synthesised singing voice with models trained on speech data has many advantages due to the models’ flexibility and controllability. However, since the information about the temporal relationship between segments and beats are lacking in speech training data, the synthesised singing may sound off-beat at times. Therefore, the availability of the information on the temporal relationship between speech segments and music beats is crucial. The current study investigated the segment-beat synchronisation in singing data, with hypotheses formed based on the linguistics theories of P-centre and sonority hierarchy. A Mandarin corpus and an English corpus of professional singing data were manually annotated and analysed. The results showed that the presence of musical beats was more dependent on segment duration than sonority. However, the sonority hierarchy and the P-centre theory were highly related to the location of beats. Mandarin and English demonstrated cross-linguistic variations despite exhibiting common patterns.

   @inproceedings{zhang2021synchronising,
     author = {Zhang, Cong and Zhu, Jian},
     title = {{Synchronising Speech Segments with Musical Beats in Mandarin and English Singing}},
     year = {2021},
     booktitle = {Interspeech 2021},
     pages = {1199--1203},
     doi = {10.21437/Interspeech.2021-1841}
   }

9. JASA

   Comparing acoustic analyses of speech data collected remotely

   Cong Zhang, Kathleen Jepson, Georg Lohfink, and Amalia Arvaniti

   The Journal of the Acoustical Society of America, 2021

   Abs Bib HTML PDF

   Face-to-face speech data collection has been next to impossible globally due to COVID-19 restrictions. To address this problem, simultaneous recordings of three repetitions of the cardinal vowels were made using a Zoom H6 Handy Recorder with external microphone (henceforth H6) and compared with two alternatives accessible to potential participants at home: the Zoom meeting application (henceforth Zoom) and two lossless mobile phone applications (Awesome Voice Recorder, and Recorder; henceforth Phone). F0 was tracked accurately by all devices; however, for formant analysis (F1, F2, F3) Phone performed better than Zoom, i.e. more similarly to H6, though data extraction method (VoiceSauce, Praat) also resulted in differences. In addition, Zoom recordings exhibited unexpected drops in intensity. The results suggest that lossless format phone recordings present a viable option for at least some phonetic studies.

   @article{zhang2021comparing,
     author = {Zhang, Cong and Jepson, Kathleen and Lohfink, Georg and Arvaniti, Amalia},
     year = {2021},
     title = {{Comparing acoustic analyses of speech data collected remotely}},
     doi = {10.1121/10.0005132},
     issn = {0001-4966},
     journal = {The Journal of the Acoustical Society of America},
     number = {6},
     pages = {3910--3916},
     publisher = {Acoustical Society of America},
     volume = {149}
   }

10. Speech Prosody

    Segment Duration and Proportion in Mandarin Singing

    Cong Zhang, and Xinrong Wang

    In Proc. Speech Prosody 2020, 2020

    Abs Bib HTML PDF Code Presentation

    Speech-based singing synthesis has various merits while it also has unsolved issues. One of the most noticeable issues is the segment duration and proportion in synthesised singing, which is caused by the difference in the short syllables in speech and the lengthened syllables in singing. This study therefore investigates how syllables are lengthened in Mandarin singing data. A total of 20 songs from the MIREX singing corpus were segmented and analysed. The results showed that (1) the segment proportions in Mandarin syllables are different in speech and in singing; (2) the lengthening is influenced more by the slots in the syllable structure than by the types of segments; (3) in the syllable structure of CGVX in Mandarin, the nuclear V lengthens the most and X follows. The durations of C and G also increase but their proportions in a syllable decrease.

    @inproceedings{zhang2020segment-sing,
      presentation = {https://osf.io/ybdup?view_only=c87fe156d1874ffba8a16cc363b225af},
      author = {Zhang, Cong and Wang, Xinrong},
      title = {{Segment Duration and Proportion in Mandarin Singing}},
      year = {2020},
      booktitle = {Proc. Speech Prosody 2020},
      pages = {596--600},
      doi = {10.21437/SpeechProsody.2020-122}
    }

11. tone-intonation

    Stacking and Unstacking Prosodies : The Production and Perception of Sentence Prosody in a Tonal Language

    Cong Zhang

    In Proceeding of 19th International Congress of Phonetic Sciences, 2019

    Abs Bib PDF

    Teasing apart lexical prosody and sentence prosody has been one of the most difficult tasks in the study of intonational tunes in tonal languages. Are different prosodic manifestations stacked, or are they an integrated whole? With evidence from production and perception data of the intonational yes/no question tune in Tianjin Mandarin at sentence level, this paper proposes that (1) lexical tonal alterations (a.k.a tone sandhi) are lexical-level prosody and do not belong to sentence-level tune; (2) pitch accents induced by information structure are “intra-tune” features, which are such sentence-level prosody features that do not cause sentence type change. Despite being sentence-level prosody features, they are not a part of the tune for intonational yes/no question.

    @inproceedings{zhang2019stacking,
      author = {Zhang, Cong},
      booktitle = {Proceeding of 19th International Congress of Phonetic Sciences},
      year = {2019},
      title = {{Stacking and Unstacking Prosodies : The Production and Perception of Sentence Prosody in a Tonal Language}},
      keywords = {intonation,tianjin mandarin,tone}
    }

12. Speech Prosody

    Chanted Call Tune in Tianjin Mandarin: Disyllabic Calls

    Cong Zhang

    In 9th International Conference on Speech Prosody 2018, 2018

    Abs Bib HTML PDF

    This paper examines the chanted call tune in Tianjin Mandarin in order to investigate the possibilities of intonational components, i.e. pitch accents, boundary tones, etc., in a tonal language. Six native Tianjin speakers’ production of disyllabic names and kinship terms were recorded. The speech materials were composed of a set of left-prominent disyllabic names and a set of right-prominent disyllabic names. The results show that there is a L% boundary tone at the end of the intonational phrase, regardless of the lexical tones. Different from the IntQ data, the L% boundary tone is phonetically manifested and overrode the lexical tone contours. A H* pitch accent was found to be associated with the H of each lexical tone. Lengthening was also found in the CC tune. The CC tune in Tianjin Mandarin can be represented as follows: [[H*]sustained]higher register + L%.

    @inproceedings{zhang2018chanted,
      author = {Zhang, Cong},
      title = {{Chanted Call Tune in Tianjin Mandarin: Disyllabic Calls}},
      year = {2018},
      booktitle = {9th International Conference on Speech Prosody 2018},
      doi = {10.21437/SpeechProsody.2018-106},
      pages = {522--526},
      publisher = {ISCA}
    }