Investigating the Effects of Valence, Arousal, Concreteness, and Humor on Words Unique to Singapore English

Introduction to Singapore English

Singapore English is the dialect of English spoken by individuals who reside in Singapore, a country in Southeast Asia which was formerly a British colony (see Bolton et al. (2020), for an overview of Asian Englishes and how they are influenced and shaped by British colonization in the region). Sociolinguists who study Singapore English recognize that it is comprised of two varieties that sit at two ends of a continuum in a diglossic context (Gupta, 1994): Singapore Standard English (SSE) and Singapore Colloquial English (SCE), also commonly known as “Singlish” in popular culture. Standard Singapore English is the “high” or prestige variety used in written and formal contexts, and is not particularly different from other dominant varieties of English such as British English and North American English. On the other hand, Singapore Colloquial English is the “low” variety commonly used in informal contexts (Cavallaro et al., 2020; Ching, 2020; Tan, 2017), and is generally perceived as a “broken”, improper form of English that potentially threatens the country’s economic progress and global standing (Rubdy, 2001).

In this paper, we were particularly interested in the colloquial form of Singapore English. This is because SCE consists of several lexical items and lexical borrowings from other languages that do not necessarily have an equivalent form in SSE. An example is the word paiseh, which is derived from the Chinese dialect Hokkien, and refers to feeling embarrassed and is also used in place of “excuse me” or “sorry” (as an interjection). It is also noteworthy that given the creole status of Singapore English these concepts are initially and predominantly used in spoken communication among various ethnic groups residing in Singapore. With the advent of new technologies such as email, text messaging, the Internet, and social media, many of these spoken forms acquired written forms and are still commonly used among speakers of Singapore English in both online and face-to-face communication (Gonzales et al., 2023). Although there has been previous efforts leveraging distributional semantics and machine learning approaches to infer sentiment of SCE vocabulary (Ho et al., 2018; Lo & Lim, 2018), validated lexical norms and psycholinguistic repositories of Singapore Colloquial English vocabulary developed using behavioral data collected from users of the languages themselves are lacking and not particularly well established.

Limited linguistic coverage in language research

It is well established that research in the language and cognitive sciences is dominated by the study of English speakers (Blasi et al., 2022), and in particular, by speakers of dominant, prestigious dialects of English, such as Standard North American English and British English (Levisen, 2019). When we narrow our focus on the languages that are represented in lexical-semantic databases or psycholinguistic megastudies, the same pattern of dominance by a handful of languages is also observed. Siew (2024a) conducted a brief analysis of languages represented in linguistic databases listed in the Linguistic Annotated Bibliography (Buchanan et al., 2019) and found that approximately 53% of all databases were in English, with only 8 other languages (Spanish, German, French, Chinese, Italian, Dutch, Portuguese, Japanese) comprising almost a third of all databases. Moreover, a large majority of databases involved participants who spoke American or British English. A notable exception is a prominent megastudy of auditory lexical decision which was developed with data from Singapore English speakers (Goh et al., 2020); however, all the words in their database consisted of words found in standard dialects of English and do not include any of the items that we study in this paper.

Despite greater awareness of the limited linguistic coverage in language research, it is striking that the situation has not improved much based on recent bibliometric reviews (Kidd & Garcia, 2022; Singh et al., 2023; Thalmayer et al., 2021). We hope that conducting the research detailed in the present paper can provide a small but meaningful contribution to psycholinguistic research, and showcase how studying colloquial language can offer new opportunities and raise interesting research questions for psycholinguistics.

Lexical-semantic and affective norms: Valence, arousal, concreteness, and humor

In the present paper, we developed lexical-semantic and affective norms of valence, arousal, concreteness, and humor. Although there are many possible norms that could have been established (e.g., sensorimotor ratings or socialness), we decided to focus on these four as an initial starting point and because these are considered to be highly central and important norms, as evidenced by the extensive literature where these norms have been established for thousands of words and concepts in English (i.e., high lexicon coverage; e.g., (Brysbaert, Warriner, et al., 2014; Engelthaler & Hills, 2018; Warriner et al., 2013)) and in languages other than English (e.g., valence and arousal norms for Spanish (Stadthagen-Gonzalez et al., 2017), concreteness norms for Dutch (Brysbaert, Stevens, et al., 2014)). Below we briefly discuss what these norms are and their relevance for psycholinguistic studies.

Valence and arousal are affective ratings of the emotions elicited by a particular word or concept (Osgood et al., 1957). Valence refers to the pleasantness of the emotion elicited, ranging from highly pleasant (e.g., joke and kitten) to unpleasant (e.g., jail and kill). Arousal refers to the degree of arousal invoked, ranging from highly arousing (e.g., sex and thrill) to unexciting (e.g., statue and bored). The development of these affective norms support research on the contributions of emotional features to memory and lexical processing (Kousta et al., 2009; Kuperman et al., 2014) and for sentiment analyses of written text (Fatima et al., 2021; Mohammad & Turney, 2013; Teixeira et al., 2021). The psycholinguistic literature generally reports a facilitatory effect for positive words (Haro et al., 2024) or a facilitatory effect for emotionality, where words that are positively and negatively valenced experience a processing advantage relative to neutral words (Kousta et al., 2009). As for arousal, the general pattern is that highly arousing words are recognized less slowly (Kuperman et al., 2014), in line with the automatic vigilance model of emotion (Pratto & John, 1991), where highly arousing and negatively-valenced stimuli automatically capture attentional resources which slows processing.

Concreteness refers to the extent to which the concept represented by a word (or short phrase) refers to a perceptible entity (Brysbaert, Warriner, et al., 2014; Paivio, 2013). In accordance with Paivio’s Dual Coding Theory, highly concrete words are easier to remember than more abstract words because they activate both perceptual and verbal codes in memory, whereas abstract words only activate a verbal code (Paivio, 2013). An alterative explanation for the concreteness effect is offered by the Context Availability Theory (Schwanenflugel et al., 1988), where concrete words benefit from being readily associated with rich contextual information relative to abstract words. In this study, words that receive a higher concreteness rating refers to an entity that exists in the real world and elicits immediate experiences of that concept through one’s senses. Words that receive a lower concreteness rating are considered “abstract” and refers to an entity that cannot be directly experienced through one’s senses. Concreteness is another important construct for psycholinguistic and memory research, as evident from studies that have shown how concreteness/abstractness influences various cognitive and lexical process such as word recognition (Goh et al., 2016), word learning (Groot & Keijzer, 2000), humor perception (Siew et al., 2022; Westbury & Hollis, 2019), and historical language change and evolution (Hills & Adelman, 2015; Y. Li et al., 2024). Specifically, the facilitative effects of concreteness is well-established within the domains of lexical processing and word recognition, where concrete words are responded to more quickly and accurately as compared to abstract words (Bottini et al., 2022; Goh et al., 2016).

Finally, humor norms capture the extent to which a word elicited feelings of amusement or humorous thought (Engelthaler & Hills, 2018). Examples of humorous English words include booty and waddle, and examples of the least humorous words include nightmare and pain. To the best of our knowledge, although there has been some interest in quantifying the humor of nonwords (Westbury et al., 2016) and of word pairs (Siew et al., 2022; Westbury & Hollis, 2019), there is only one existing database of word humor norms in English (Engelthaler & Hills, 2018). In that paper, the authors found that word humor was associated with several other lexical-semantic and affective variables such as concreteness and valence, but was most strongly anti-correlated with word frequency, such that humorous words tend to be more rare in language. Word humor was also positively correlated with lexical decision RTs obtained from the English Lexicon Project, such that humorous words were more slowly responded to in visual lexical decision. Although humor is a somewhat more recent addition to the realm of psycholinguistic norms, we decided to collect humor ratings for our items because research has demonstrated interesting connections between the use of colloquial language to elicit humor (Joser et al., 2023; Ruiz-Gurillo, 2021) and we anticipate that the collection of humor norms for SCE could enable future investigations into the socio-pragmatic effects of colloquial language use in a Singaporean context.

Finally, another key aspect of the current paper is to also collect visual lexical decision data for SCE words, so that we can evaluate the ability of these lexical-semantic and affective norms in explaining lexical processing performance. The visual lexical decision task is a classic psycholinguistic task that is commonly used to explore the influence of various lexical variables on lexical retrieval. Here, we were interested to see if valence, arousal, concreteness, and humor would show similar patterns and effects in the recognition of SCE lexical items as reported in the prior literature. Based on the discussion above, we expected to find facilitative effects for SCE words of higher concreteness, positive valence, low arousal, and low humor.

Why do we need to develop lexical norms for Singapore English?

As mentioned earlier, Singapore English is one of the many dialects of English that exist worldwide. Given that English serves as a lexifier for SCE, it is unsurprising to find a large overlap of vocabulary among Singapore English and other English dialects. However, unpublished data from our lab suggests that there are approximately 4,000 words, phrases, and concepts that are unique to Singapore. We would like to emphasize that the present investigation is focused on lexical items unique to SCE and thus not found in the vocabulary of other major English dialects. Specifically, among the list of 283 items that we collected lexical-semantic and affective measures for in this paper, only 9 of these items were found in the English Lexicon Project, a megastudy of North American English (Balota et al., 2007) and only 12 were found in the British Lexicon Project, a megastudy of British English (Van Heuven et al., 2014). Therefore, lexical-semantic and affective information and behavioral data was only available for a small proportion of the items in our current study. This also necessitated the need to collect new visual lexical decision data (Part 3) for validation, as opposed to simply re-analyzing pre-existing megastudy datasets.

Leveraging Large Language Models for Developing Lexical Norms

There has been recent interest among psycholinguists to harness the capabilities of Large Language Models (LLMs) to develop lexical-semantic and affective norms for psycholinguistic research at a massive scale (Trott, 2024). Some recent papers have suggested that this is a promising avenue, showing that LLMs are able to generate valid, high-quality lexical-semantic and affective norms such as familiarity, valence, arousal, and concreteness, for English words and multi-word expressions (Martínez, Molero, et al., 2024) and Spanish words (Martínez, Conde, et al., 2024) at a fraction of the cost and time that is usually invested in human-based data collection. The results for Spanish are particularly interesting as they suggest that even if the LLM is not predominantly trained on the target language, it is still able to produce reasonable estimates (Martínez, Conde, et al., 2024). Hence, a secondary goal of this paper to explore the potential of LLMs to provide lexical-semantic and affective estimates of concepts unique to a minority English dialect, Singapore English, and to compare its explanatory power in a behavioral task to traditionally obtained norms. If AI-generated norms are comparable to human ratings and produce expected behavioral results when used as predictors, perhaps LLMs could be leveraged to produce lexical norms for under-resourced languages and dialects in future work.

Outline of paper

We first describe the development of lexical-semantic and affective norms for a subset of items in the SCE vocabulary. In Part 1 we show that participants are able to provide fairly reliable ratings for these items, and Part 2 describes the process by which the same ratings were obtained from GPT-4. We then report the results of a visual lexical decision (VLD) study to assess if these lexical-semantic and affective measures have any measurable impact on lexical processing in Part 3. We demonstrated that some of these measures are able to account for additional variance in VLD performance beyond traditional measures such as word length and frequency.

Method

Results

Data exclusion followed a two-tiered approach. In the first step, we excluded participants who self-reported that English was not their native/first language, that they had a prior history of speech or language disorders, or that they were significantly distracted while completing the task. We also excluded participants who did not consider themselves current residents of Singapore (e.g., exchange students), or had resided in Singapore for less than 10 years. Out of the initial 360 participants, 324 participants remained after applying these criteria. In the second step, we excluded participants who provided low-quality data as determined by lack of variability in responses (i.e., the same rating was provided to all words; n = 4), poor correlation of ratings (less than 0.1) with the average ratings of the other raters (n = 20), or poor lexical knowledge of a high proportion of the items presented (n = 3). Specifically, participants who gave the same response to all items, or whose own ratings correlated poorly with the overall mean ratings, or who indicated that they did not know more than 60% of the items presented, were excluded from data analysis. These additional data cleaning steps are typically done in other studies that developed lexical-semantic word norms (Brysbaert, Warriner, et al., 2014).

Then, trials where the participants indicated that they did not know the word or phrase were also removed. Using the remaining trials, valence, arousal, concreteness, and humor norms were computed for each item using cumulative link mixed effect modeling to obtain latent norms (Taylor et al., 2022). Although item norms are commonly computed by taking the average of the rating for each word, Taylor et al. (2022) showed that the use of cumulative link mixed-effects models (CLMM) to compute item norms has several advantages, including taking into account the ordinal nature of Likert ratings (instead of treating ratings as continuous), and being less influenced by response biases of the participants. Specifically, the CLMM models Likert scale ratings as ordinal dependent variables which are mapped onto ordered regions of a normally distributed latent distribution, and since these CLMMs are fitted with item random effects, the extracted random effects of the model represents estimate of the latent mean for each item (i.e., the item’s latent norm on a particular dimension) (Taylor et al., 2022).

Although lexical norms are analyzed using the CLMM approach in this manuscript, traditionally computed item norms are available in the OSF for interested readers to review. The trial-level data was submitted to a R Shiny application developed by Taylor et al. (2022) (see: https://github.com/JackEdTaylor/shinynorms) to compute latent word norms using cumulative link mixed-effects models with a probit link function. All data (trial-level rating data, lexical norms, stimuli) and analysis scripts can be found at https://osf.io/bjxfa.

Table 1 shows examples of items that received the highest and lowest ratings on each of the four measures, and Table 2 shows a correlation table of the four measures collected for 283 items. Figure 1 shows the distributions for each of the four norms, and Table 3 presents basic descriptive information for their distributions.

Figure 1

Method

Our approach closely followed that of Martinez and colleagues (Martínez, Conde, et al., 2024; Martínez, Molero, et al., 2024). To query GPT-4 about various lexical-semantic and affective features of the SCE word or concept, we provided it with two prompts that was adapted from Martínez, Molero, et al. (2024). We used the MacBehaviour R package (Duan et al., 2024) to automate the interactions with GPT-4 (gpt-4o-2024-11-20). Although there are many other LLMs that could have been queried, we focused on GPT-4 since it was previously shown to provide the best results (Martínez, Conde, et al., 2024; Martínez, Molero, et al., 2024). We provide an example of the specific prompt used for valence below (a complete list of prompts used can be found in the OSF):

To explore the possibility that we might get better results by “nudging” the model into a Singapore English context, we also queried the model with a second slightly modified prompt. An example for valence shown below, with differences from the previous prompt highlighted in bold:

The temperature parameter in LLMs influences the randomness of generated responses. In this case, the temperature of the model was set to 0, so that the same results (i.e., least random) are obtained each time a word is presented to the model. We also extracted the estimated probabilities of the 5 most likely answers (via the “log_probs” function). For instance, if the word shiok received a probability of .6 (rounded off) for rating 7 (high valence), .3 for rating 6, .05 for rating 8, and .025 for ratings 5 and 9, one could calculate two values: the raw rating with the highest probability (i.e., 7) and a more precise, weighted value found by multiplying the ratings with their probabilities. For shiok this would give a value of .6 × 7 + .4 × 6 + .05 × 8 + .025 × 5 + .025 × 9 = 7.35. Finally, the prompt was repeated anew for each item to prevent instruction dilution.

To summarize, for each of the 283 SCE concepts, we obtained both a raw and weighted numerical rating for valence, arousal, concreteness, and humor from GPT-4. This was repeated twice, the first set of ratings was obtained from a general prompt with no reference to Singapore English, and the second set of ratings was obtained from a specific prompt with an explicit reference to Singapore English.

Results

To evaluate the quality of the LLM-generated ratings, we correlated the four numeric ratings obtained from GPT-4 (raw/weighted rating × general/specific prompt) with the human-generated ratings described in Part 1. Table 4 shows the Pearson correlations among these measures. All correlations were statistically significant at the p < .01 level.

Figure 2 shows the distributions for each of the four norms generated by GPT and the same norms provided by human raters, to provide a visual comparison. Table 5 presents basic descriptive information of the LLM distributions. A 2-sample Kolmogorov-Smirnov non-parametric test was used to determine whether human-generated and LLM-generated distributions differed significantly across each of the four norms. The results indicated that the human-generated and LLM-generated distributions were significantly different; valence: D = 0.29, arousal: D = 0.47, concreteness: D = 0.41, humor: D = 0.40, all ps < .001.

Figure 2

There are a few interesting results in the correlation analyses. First, correlations of human ratings and GPT ratings obtained with a general prompt were smaller than the correlations with GPT ratings obtained with a specific prompt, across all four measures. This points to the importance of carefully designing prompts to focus the model’s output in a specific context, in this case, Singapore English. Second, correlations were slightly higher for the weighted ratings as compared to the raw ratings, in accordance with previous research (Martínez, Molero, et al., 2024). This indicates that obtaining a weighted value based on estimated probabilities can be one way of improving the quality of LLM-generated ratings, as compared to simply taking the first answer (i.e., raw rating). Third, correlations were highest for valence, followed by concreteness, arousal, and humor. This also appears to be consistent with (Martínez, Conde, et al., 2024; Martínez, Molero, et al., 2024) who found the highest correlations for valence than arousal in both Spanish and multi-word expressions in English. It appears that LLM performance is quite poor for humor. Finally, the best performing measure (weighted ratings + specific prompt) produced estimates that correlated well with human ratings, albeit these values were on the lower end of the range of correlations that have been reported in the literature. In the next section, we will evaluate the performance of these LLM-generated measures to account for human performance in a visual lexical decision task.

Method

Results

For the RT data, errors and nonword trials were first excluded. Trials with latencies that were 2.5 SDs above or below each participant’s mean RT were considered outliers and excluded from analysis. This resulted in 3.1% of the data being removed.

Using the lme4 R package, a linear mixed effects (LME) model was used to predict RTs and a generalized linear mixed effects (GLM) model was used to predict accuracy from the lexical decision data (Bates et al., 2014). Both RT and Accuracy models included the following predictors: (a) random intercept effects of participants and items, (b) fixed effects of interest (valence, arousal, concreteness, humor), and (c) fixed effects for control variables (number of letters, word frequency, orthographic neighborhood size, and mean bigram frequency). All predictor variables were standardized. The Satterthwaite’s method as implemented in the lmerTest R package (Kuznetsova et al., 2017) was used to obtain the degrees of freedom and p-values for all the linear mixed effects models reported in this paper.

A two-step hierarchical approach was used. Number of letters, word frequency, orthographic neighborhood size, mean bigram frequency were entered into the LME model in Step 1 (base model). Valence, arousal, concreteness, and humor were entered into the LME model in Step 2 (base+norms model), in addition to the previously entered variables. Partitioning the analysis into two steps was done to determine if the lexical-semantic and affective measures collected in the previous section accounted for additional variance over conventional lexical variables.

LLM-Generated Norms

Valence, arousal, concreteness, and humor ratings obtained from GPT-4 were entered into the LME model in Step 2 (base+norms model) as described above, specifically, the best performing LLM measure (weighted ratings + specific prompt). This analysis would enable us to see if the LLM-generated norms would be as predictive of human linguistic performance as the human-generated norms as described above. In addition, we evaluate the Akaike information criterion (AIC) to provide a comparison across non-nested models. Generally, models that have a lower AIC score are preferred as the AIC represents the relative amount of information loss that occurred when fitting the model to the data (Wagenmakers & Farrell, 2004). Table 9 shows the results of the RT and Accuracy models where the ratings were obtained from humans were used as predictors, as well as the corresponding models where the ratings were obtained from GPT-4 were used as predictors.

Using LLMs to provide psycholinguistic norms for under-resourced languages

This paper also explored the possibility of using LLMs to obtain lexical-semantic and affective ratings for concepts unique to Singapore English. LLM-generated ratings obtained from GPT-4 were compared to human ratings. Results indicated that while the correlations between LLM ratings and human ratings were positive and statistically significant, there was quite a lot of variance in the LLM performance. Specifically, correlations were much higher when the prompt given included a reference to Singapore English, indicating that a higher level of prompt engineering may be necessary to “nudge” LLMs into an appropriate context prior to generating ratings for under-resourced and under-represented languages and dialects. Correlations were also highest for valence, and lowest for humor, which further indicate that LLMs may provide better or worse results depending on the type of lexical information that is being queried. Despite the significant correlations, it is worth highlighting that the overall distributions of the ratings were sigificantly different across human raters and the LLM (see Figure 2).

Given the previous work showed that GPT-4 was able to generate valid ratings for Spanish, a language that the model was not specifically trained on (Z. Li et al., 2024; Martínez, Conde, et al., 2024), the results for Singapore English could be considered somewhat disappointing in comparison. However, this might not be too surprising because even if one does not have access to the raw contents of the training data, one could still reasonably assume that Spanish text, while not forming the majority of the LLM’s training data, is several times larger than the amount of English text produced by native speakers of Singapore English. Indeed, computational experiments have suggested strong correlations between the LLM’s performance in various languages and the proportion of those languages in its pre-training corpus (Z. Li et al., 2024). Our results may serve as a gentle warning that LLMs are not a panacea for psycholinguistic research, particularly when trying use them to develop lexical norms for minority languages and dialects. It is important to develop and validate LLM-generated norms in close conjunction with data that is ideally newly obtained from humans to guard against data leakage and contamination (Trott, 2024).

Finally, we compared the ability of LLM-generated norms to account for human performance on the VLDT. As the results in Part 3 showed, the model that used human ratings as predictors out-performed the model that used GPT ratings as predictors on the AIC model performance measure. In addition, when GPT ratings were used as predictors instead of the human ratings, the overall patterns were different from the original model. Specifically, the human model showed consistent effects of valence and arousal across both RT and accuracy, whereas the only consistent effect across RT and accuracy in the GPT model was humor, which was not originally significant in the human model. This is a somewhat surprising finding given that the correlations between the human and GPT ratings were much higher for valence (r = 0.78) and arousal (r = 0.59) than for humor (r = 0.39). While it is difficult to provide any convincing explanation for this discrepancy, these results provide an initial argument that LLM-generated norms may need to be held to a higher standard. Beyond establishing a high correlation with human generated norms or annotations (typically done in previous studies, e.g., (Martínez, Molero, et al., 2024; Trott, 2024)), researchers should also assess if LLM-generated norms are able to account for known behavioral effects in psycholinguistic experiments to a better extent than human norms, and discrepancies in the behavioral patterns and overall distribution patterns such as those reported here could provide important clues into potential limitations of LLM-generated norms. After all, a key goal of developing lexical-semantic and affective norms is to enable a better characterization of empirical data in psycholinguistics–this is why researchers continue to develop better and bigger word frequency databases (Brysbaert & New, 2009) or propose new variables such as prevalence or word knowledge (Keuleers et al., 2015) based on their improved performance at accounting for variance in large-scale behavioral datasets.

Limitations and future directions

A clear limitation of this research is that we have only collected lexical-norms for a small subset of the SCE vocabulary, which pales in comparison to the sizes of modern psycholinguistic databases (consisting of at least several thousand items). However, because lexical-semantic and affective measures have never been developed for concepts unique to SCE, a pragmatic approach that we have decided on was to first obtain norms for a subset of well-known items to determine if meaningful ratings could be obtained. Our results clearly show that speakers of Singapore English converge on their ratings of these concepts, and that these ratings have implications for lexical processing. The current results provide us with some confidence that it is a worthwhile research pursuit to persist with our data collection efforts for the rest of our database. They also motivate us to potentially extend data collection to other types of norms, such as prevalence, which provides an index of word knowledge across the population (Brysbaert et al., 2019) and a potentially important complement to word frequency information, which may be less relevant for colloquial lexical items that are more often spoken and less often written, and sensorimotor norms, which provides information about perceptual and action strength and can enhance our understanding of the sensorimotor basis of semantic representations of Singapore English concepts (Lynott et al., 2020).

Another limitation of this research is that only basic demographic varibles were collected from the participants. However, future work could collect additional information such as personality or sociolinguistic attitudes towards Singapore English, to explore how individual differences may impact elicitation of ratings on various lexical-semantic and affective dimensions, and in particular for word humor ratings which may be especially liable to well-established individual differences in humor styles (R. A. Martin et al., 2003). It is also worth mentioning that Chinese participants were over-represented in our rating sample (~89% of participants for which we had ethnicity information on), which is higher than the national proportion of 75% Chinese ethnicity (with 14% Malay, 9% Indian, and the remainder as Others). This could have an impact on the rating results depending on the item’s etymology and relative differences in usage patterns of specific items across ethnicities. Future studies should look into this issue in more detail.

In conclusion, we hope that this paper inspires researchers to devote some effort toward developing psycholinguistic norms for informal, colloquial languages (such as creoles) and other less studied varieties of English, and motivates researchers of Singapore English to exploit this lexical-semantic and affective database for their own research.