You’re listening to “ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models,” by Aparna Elangovan and colleagues. Published in 2024. Abstract. In this position paper, we argue that human evaluation of generative large language models (LLMs) should be a multidisciplinary undertak- ing that draws upon insights from disciplines such as user experience research and human behavioral psychology to ensure that the ex- perimental design and results are reliable. The conclusions from these evaluations, thus, must consider factors such as usability, aesthetics, and cognitive biases. We highlight how cog- nitive biases can conflate fluent information and truthfulness, and how cognitive uncertainty affects the reliability of rating scores such as Likert. Furthermore, the evaluation should dif- ferentiate the capabilities and weaknesses of increasingly powerful large language models – which requires effective test sets. The scala- bility of human evaluation is also crucial to wider adoption. Hence, to design an effec- tive human evaluation system in the age of generative NLP, we propose the ConSiDERS- The-Human evaluation framework consisting of 6 pillars – Consistency, Scoring Critera, Differentiating, User Experience, Responsible, and Scalability. 1 Introduction. Generative tasks in natural language processing (NLP) have to rely on human evaluation, as the current set of automated metrics does not correlate well with human judgment. Human evaluation tends to be expensive and difficult to repeat or reproduce. Even more importantly, an all-too-common scenario tends to be that the eval-uation method is fundamentally misaligned with the problem statement. In the age of generative large language mod-els (LLMs) with increasing capabilities that can generate fluent content to even fool humans, ensuring that the human evaluation is set up appropriately to measure the right aspects and reach the right conclusions is crucial. In this position paper, first, we argue that to design and interpret the results of human evalu-ation accurately, the evaluation pipeline needs to be human-centric in the age of generative AI, ac-counting for human evaluators and their cognitive biases. The field of user experience (UX) takes into account the emotional states of a user, a.k.a. how a user feels. It is a well-known fact in UX that users tend to be heavily influenced by aesthetic aspects, while actual function or usability aspects take a sec-ond place when users perceive a system as useful, leading to the notion “what is beautiful is useful”. Aesthetics also extends to language. Factors such as fluency can affect the evaluation, outweighing the actual content or sub-stance. Studies in human-computer interface (HCI), cognitive, and social psychology have demonstrated that processing fluency – the ease with which information is perceived and pro-cessed in the human mind – has a positive effect on evaluation. Current state-of-the-art (SOTA) LLMs tend to be quite fluent and produce content that is easy to read and understand, and as a result, users can conflate fluency and use-fulness. Therefore, we need to closely examine our human evaluation before reaching conclusions such as – The LLM can perform function simi-lar to or better than a trained professional. NLP evaluation procedures, therefore, at the very least must delineate style vs. substance. Secondly, the effectiveness of the test set in measuring the capabilities of a model is critical, as ineffective test sets cannot adequately evaluate these models, a common theme that has surfaced in many leader-boards and public data sets. Hence, in this position paper, we make the fol-lowing contributions: 1) We propose a framework, a structure for orga-. nizing and contextualizing human evaluation, that can be customized and adapted to specific contexts. Our proposed framework – the ConSiDERS-The-Human evaluation framework – has 6 pillars: The 6 pillars of ConSiDERS-The-Human Evaluation Framework: (See Checklist in Appendix A.1 to follow.) • Consistency of human evaluation: The find-ings of human evaluation must be reliable and generalizable. • Scoring Criteria: The scoring criteria must include both general purpose criteria such as readability, as well as be tailored to fit the goal of the target tasks or domains. • Differentiating: The evaluation test sets must be able to differentiate the various capabilities as well as the weaknesses of generative LLMs. • User Experience: The evaluation must take into account user experience, including their emotions & cognitive biases, when designing experiments and interpreting results. • Responsible: The evaluation needs to ac-count for responsible AI including as-pects such as bias, safety, robustness, and privacy capabilities of the model. • Scalability: Human evaluation must be scal-able for pragmatic widespread adoption. 2) We make the case for why UX and the psy-. chology of cognitive biases should be at the fore-front of human evaluation. In the last 20 years, less than 7% of the papers (only 16 papers) with “human” and “eval” in their title available in ACL Anthology mention user experience-related key-words in either the title or the abstract (see query in Appendix A.2.6). 3) We highlight how neglecting the role of cog-. nitive biases in human evaluation can lead to in-correct inclusions from the study. We, therefore, provide specific recommendations to mitigate the effects of common cognitive biases. We also pro-vide tips to troubleshoot and improve consistency issues in human evaluation. In the rest of this paper, we introduce the neces-sary background concepts in Section 2 and explore each of the 6 pillars in detail in Section 3. 2 Background concepts 2.1 Usability Usability, according to ISO 9241–11:2018, is de-fined as – “The extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use”. Usability testing includes the following five elements or the 5 Es of usability. 1) Easy to learn: This aims to address questions such as a) How easy is it for users to complete basic tasks the first time they use the system? b) When users return to the design after a period of not using it, how well is the user able to recollect how to use the system? 2) Efficiency: How quickly can experienced users accomplish tasks? 3) Effective: How completely and accurately the work or experience is completed or goals reached? 4) Error tolerant: How many errors do users make, how critical are these errors, and how easily can they recover from the errors? 5) Engaging / Satisfaction: How much does the user like using the system? These 5Es are crucial when designing human evaluation solutions to ob-tain reliable evaluation results. 2.2 UX and HCI User experience (UX), according to ISO 9241-110:2010 is defined as – “a person’s perceptions and responses that result from the use and/or antic-ipated use of a product, system or service”. UX, thus, expands beyond the con-cepts of the 5Es of usability to take into account the broader emotions experienced by the users when using the system. In other words, UX takes into account usability as well as the users’ feelings as to how products “dazzle their senses, touch their hearts and stimulate their minds”. HCI is the UX when humans interact with computer systems, including user interfaces and how information is presented on the digital screen. Commercial LLMs facing end users, such as Chat-GPT, have dazzled the minds of their users. Users’ emotion, including perceived usability or useful-ness, tends to be heavily influenced by aesthetics. Aesthetics influences the user heavily initially or in one-off tests, but over time aesthetics plays a much lesser role and us-ability becomes crucial. The field of UX, therefore, attempts to disambiguate the perceptions that users form as a result of aesthetics, and the need to measure actual function as defined by the 5Es of usability, whilst embracing user emotions. 2.2.1 Measuring UX: Perception vs. Performance UX feedback can be qualitative such as user in-terviews or quantitative metrics. Quantitative UX metrics include performance-based metrics such as time to complete a task, the errors the users en-counter, and perception-based self-reported met-rics through rating scale and preferences. Likert scale is a type of attitude scale, a special case of a rating scale, that measures the degree to which a person agrees/disagrees with a given statement. In NLG, performance-based metrics designed to measure the impact of the end system are considered extrin-sic evaluations, while intrinsic evaluations attempt to evaluate properties of the NLG text (van der Lee et al., 2019). Measuring aspects such as fluency are intrinsic evaluations, usually measured through a rating scale or preference tests, where the evaluator is asked which model’s output they prefer. Rating scales and preference tests are based on user per-ception, and therefore subject to cognitive biases. 2.3 Cognitive biases Cognitive biases are systematic errors in human judgment or aspects that drive irrational behavior. This is usually a result of relying on heuristics to make a decision. There are many types of cognitive bi-ases, the sources can be broadly categorized into a) too much information b) lack of information or lack of understanding or meaning associated with the information c) need to act or make judgments fast d) information that is remembered or recalled. For example, when a user is presented with a long list (too much information) during information retrieval, to quickly filter out information, the user would simply click on the first link due to position bias. There are several studies on the effects of cog-nitive biases on information search and retrieval and crowd-sourcing. For instance, White (2013) finds that evaluation of search and retrieval systems is impacted by con-firmation bias – people’s unconscious tendency to prefer information that confirms their beliefs and disregard evidence that refutes it. There are over 180 different types of cognitive biases identified, resulting from a range of factors from how ques-tions are framed to prior beliefs that attempt to explain the heuristics humans use to make decisions. These heuristics also impact how humans evaluate LLMs. 3 ConSiDERS-The-Human framework 3.1 User Experience Cognitive biases play a key role in how humans judge or rate the system. Despite this, there is little reporting of the influence of these biases in NLG tasks with human evaluation. In this section, we make the case as to why UX and the psychology of cognitive biases are crucial components of human evaluation in NLP. Since UX and the psychology of cognitive biases are entire fields on their own, it is impossible to cover all the details in this paper. We highlight the significant impact of cognitive biases on human evaluation of NLG tasks that can lead to incorrect conclusions. Processing fluency – the ease with which infor-mation is processed by the human mind – affects factors such as perceived truthfulness and usefulness of statements. These lessons from psychology also apply to NLG human evaluation, where the human evaluation strategy needs to isolate the effects of linguistic fluency vs. aspects such as factual correctness. As presented in the introduction section, the notion “what is beauti-ful is useful” also extends to language, where information that is presented in an easy-to-process manner can be perceived as true. To be clear, we are not calling for linguistic fluency and coherence to be trivialized. On the contrary, we highlight how powerful its influence is on hu-man judgment and evaluation. In the following section 3.1.1, we provide a few tips on how to iso-late such effects in NLG evaluation. 3.1.1 Recommendations to mitigate common cognitive biases in NLG evaluation 1. Cognitive uncertainty in user feedback includ-. ing rating schemes: Explicit user feedback such as 1-5 rating scales, and preference-based tests are inherently subject to cognitive uncertainty, there-fore the same user can change their rating on the same item when asked again at a later point in time, even within a few minutes after the initial rat-ing. Uncertainty in user feed-back is a well-known problem in recommendation systems, where a user’s rating is considered to be noisy. Jasberg and Sizov (2020) demonstrate the scale of the problem where 65% of the users change their rating even within short intervals between re-rated items. Intra-user rating consistency tends to be higher when the ratings are extreme, e.g., very good or very bad, whereas the in-between ratings tend to have lower consistency. Hence, cognitive un-certainty is more likely to affect model evaluations where the outputs are neither very good nor very bad, where the ratings fall in the mid-point, e.g., 3 on a rating scale of 1 to 5. Similar problems can also surface in preference tests, where users choose “A is marginally better than B” when they don’t have strong opinions. Mitigation: Rating denoising algorithms in rec-ommendation systems obtain multiple ratings from the same user and attempt to keep only some of the ratings. For instance, Amatriain et al. (2009b) pro-pose to keep only those intra-user ratings whose difference is less than a predefined threshold and choose the mildest rating (most neutral rating) as the final rating for that user. The intuition here is that the mildest rating is not likely to affect the items recommended to the user. NLP human eval-uation can potentially leverage such denoising al-gorithms. Some studies have reported that binary preference-based evaluation has better consistency compared to rating. 2. Conflate fluency with attributes such as. truthfulness: In psychology, the subjective ease with which the mind processes information is more likely to be judged as true. This subjective ease with which the mind processes information can be due to factors including how information is presented, how frequently it is repeated and cues of familiarity, such as native speakers can seem truer than those with a foreign accent. Thus, humans rely on shortcuts and draw inferences on aspects such as truthfulness from feelings. Conflating fluency with truthfulness is a result of a cognitive bias called the halo ef-fect. The halo effect is the influence of a global evaluation on evaluations of individual attributes. This particularly af- fects NLP scoring criteria such as factual complete-ness, salience, and truthfulness, where these indi-vidual attributes can be impacted by the overall global attribute – linguistically fluent easy to pro-cess information. Therefore, the reliability of rating schemes particularly affects tasks that require rig-orous detailed inspection, such as truthfulness or factual completeness. This cognitive bias has been largely ignored when using rating scales including Likert to evaluate the factual correctness of LLMs. As a result, experimenters can inadvertently con-clude that LLMs are as factually comprehensive in performing a given function as a trained profes-sional like a doctor. We find that 9 out of the 19 papers we sampled from top-tier medical journals use the perception-based Likert scale to evaluate factual correctness and completeness (details in Appendix A.4), indicating how widespread these practices are. Mitigation: Tasks such as fact-checking and fac-tual completeness that require inspecting individual traits must be ideally split into atomic facts as de-tailed in Section 3.2 to isolate the impact of fluency and ease of information processing vs. facts. 3. Over-reliance on initial information: An-. choring bias or over-reliance on an initial piece of information affects how models are scored using preference or rating tests. For instance, when performing preference tests, if the model presented first on the screen is always the same, then the ini-tial perception of evaluators can have a significant impact on the rest of the evaluation. Mitigation: It is important to shuffle the display order so that the order, such as the sequence of hu-man evaluation tasks, doesn’t give away the model. In preference tests, within each task where the out-puts of say 2 models (e.g., model A vs. B) are compared, the underlying model representing A and B must also be randomly shuffled so that A does not always refer to the same model. 4. Perception vs. Performance Most self-reported feedback such as user ratings tends to be based on human perception. While perception-based metrics represent how a user feels are necessary to measure subjec-tive aspects such as readability, they do not capture functional performance-based metrics such as effi-ciency. Studies have shown that users can dislike a system that performs well or like a system that does not perform well. For instance, in an experiment where participants are asked to choose between one-level and two-level menus for sorting categories, participants preferred the two level menu, even though during actual use the one level menu was much faster and less error-prone. This demonstrates the discrepancy between preferences reported vs. measurements of efficiency. In the context of LLM evaluation, consider a hy-pothetical application scenario where an LLM gen-erated output is used to automatically draft email re-sponses. In this scenario, how much a user prefers the output of a model versus the reality of how use-ful that model output is in boosting productivity (time spent drafting emails) need not be correlated. The main challenge with conducting usability stud-ies is that the software system needs to be built to study how it impacts an end user’s productivity. In addition, confounding factors such as poorly de-signed UI can result in reducing productivity and these aspects need to be taken into account when designing and analyzing usability studies to un-derstand the impact of an LLM generated output. Liebling et al. (2022) highlight a similar problem in evaluating large-scale machine translation, where the end user’s experience can be quite different from simply evaluating the model output. Mitigation: NLP tasks that are meant to assist end users, must eventually conduct usability studies to capture performance metrics such as efficiency. 3.2 Consistency of human evaluation Reproducibility of experiments in science is a widespread challenge and has even led to the term “replication crisis” being coined. Hu-man evaluation is no exception to this challenge, where less than 5% of human evaluations are repeat-able. Despite the challenges, con-sistency cannot be ignored, as poor reproducibility can point to core design problems. Broadly speaking, non-random or systematic inconsistencies primarily arise due to 5 main de-sign flaws 1) ill-defined evaluation guidelines pro-vided to the annotators 2) high complexity task 3) evaluators who are not well qualified or suited to the task 4) small number of evaluators and/or evalu-ation set size 5) rating scales such as the Likert. We specifically need to be able to differentiate between random and non-random inconsistencies, as human evaluators are subject to decision errors/outcomes depending on their cognitive state. Random errors can neither be predicted nor controlled. Thus, understanding the role of non-random variations due to system design is key to improving the consistency of evaluation. 1) Ill-defined or complex evaluation guide-. lines: Ill-defined guidelines are often ambiguous, incomplete, do not address boundary cases, and do not provide adequate examples. To illustrate this point, Pradhan et al. (2022) use the example of a seemingly simple task “Is there a dog in this image?” where the authors point out how even this simple task can elicit several clar-ification questions such as “Does the dog need to be a real animal?”, “What if the dog is only partially visible in the image?” and “What about a wolf?”. The authors further suggest a 3-stage workflow to improve annotation guidelines: Stage 1 involves workers identifying ambiguous samples; Stage 2 involves labeling a few ambiguous examples to add as clarifying examples in the instructions; and Stage 3 involves workers performing the actual annotation using the revised guidelines with the clarifying examples. Overly long annotation guide-lines might even require training the annotators, and hence annotators with task-relevant experience tend to be referred. Wu and Quinn (2017) find that using simple vocabulary and logical ordering of instructions can improve the guideline quality. Improving guidelines alone may not be sufficient, as enhancing the user inter-face design can help reduce cognitive load of the annotator, which in-turn can improve the accuracy of annotation tasks. A simple way to identify deficiencies in guide-lines is to have “experts” independently evaluate a set of task items using the guideline and compute the IRA score using an appropriate metric such as a Kappa score. Low IRA can indicate potential problems with the annotation guidelines, in which case revising the guidelines iteratively can lead to reasonable agreement. The 5Es of usability testing criteria listed in Sec-tion 2.1 is an important strategy to follow when designing human evaluation solutions, including aspects such as how quickly human evaluators can complete their tasks while minimizing errors, how easily the annotation guidelines can be followed and memorized. For example, an overly detailed hard-to-remember evaluation guideline can simply result in poor usability affecting ease of learning, efficiency, and error tolerance resulting in poor inter-rater agreement and/or very slow evaluation turnaround times. 2) High task complexity: Tasks that involve. high cognitive load for the evaluator, can lead to lower agreement. High cognitive load can even simply involve asking the evalua-tor to assign a rating of 1-5. One way of mitigating this is to simplify the task. Simplification is key to obtaining consistent results. An example of task simplification to evaluate the factual completeness or saliency of a model-generated summary is to break a long text into hier-archical units of facts using pro-tocols such as Pyramid. Breaking a reference summary into atomic facts allows the evaluators to verify if the fact is present, and the total number of facts captured in the summary can be summed up automatically to compute the overall score. Thus, the fact-level re-call score is likely to yield much more consistent and reliable results compared to a grading scheme, such as the Likert scale, which would involve ask-ing the evaluator to rate the completeness, framing the problem as “How complete do you think the summary is on a scale of 1-5”. A flip side to break-ing long text into atomic units is that it can lead to loss of information when measuring certain types of criteria, e.g., qualitative aspects such as coher-ence cannot be evaluated using atomic facts. 3) Ill-suited evaluators: Ill-trained annotators. can also be a source of inconsistency, usually a scenario encountered when using crowdsourced workers to evaluate specialized tasks. Annotators who demonstrate poor attention can be identified using a set of attention check questions. Similarly, ill-trained or ill-qualified workers can be identified using an “exam set”, a test set for which answers are known. Lower IRA can also be a result of variation in the skills or qualifications of the evaluators. 4) Small number of evaluators and/or test set:. Low number of participants, or the sample size of the evaluators, is one of the key contributors to poor reproducibility. Using a larger pool of evaluators can mitigate experimenter bias introduced when using very small groups. A small sample size of the test set is another source that introduces replication problems, A caveat here is that a large group of evaluators and a large test set is necessary but not sufficient to ensure that the study is experiment bias-free, as selection bias can result in a large size that is not representative of the target population. 5) Rating scales such as the Likert: As dis-. cussed in Section 3.1.1, self-reported user feedback using rating schemes is inherently noisy and, there-fore, unreliable. Despite 50% of human evalua-tions relying on Likert scale (van der Lee et al., 2021), there has been little investigation into the controversies surrounding it in NLP. These include Likert’s consistency issues, aggrega-tion & interpretation of scores and the methods to com-pute IRA that the NLP community needs to research further. 3.2.1 IRA: Importance and caveats IRA or inter-rater agreement in human evaluation measures how well two or more evaluators agree on the scores or preferences they assign independently. While it is important to measure IRA to detect problems in the design, especially given that only 18% of the papers using human evaluation report IRA, there are several caveats called out on the use of IRA metrics in the medical community which we will be discussing below. Gisev et al. (2013) guide when to use which IRA measure, e.g., Krippendorff’s-α vs. Cohen’s-κ, depending on experiment design factors such as the number of annotators and the type of variable (e.g., ordinal, nominal, etc). Despite such high-level guidance, which IRA measures to use and how to interpret it is contentious (ten Hove et al., 2018; McHugh, 2012). Using an inappropriate metric, such as Fleiss-κ for interval data, is common in NLG. Furthermore, even when an appropriate class of IRA metric is used, depending on the IRA chosen, the scores can range from poor to almost perfect (ten Hove et al., 2018). An intuitive measure of inter-rater agreement is using percent agreement. However, the main criti-cism was that this does not take chance agreement into account. The question to ask when using IRA metrics, such as a kappa statistic is why and when does chance agreement matter? Some tasks genuinely have a class imbalance, e.g., span annotation tasks for named entity recognition, and unmarked spans for the majority class, which would lead to inflated percentage agreement. Another reason is the assumption that some annotators might be making random guesses when they don’t know the answer, and that the majority of the raters may NOT be making deliber-ate choices. Hence, various IRA measures of Kappa (e.g., Cohen’s-κ) or Alpha (e.g., Krippendorff’s-α), estimate the observed chance agreement or disagreement empirically when com-puting IRA. However, if we consider a group of conscientious raters, does chance agreement mat-ter for rating model outputs? For instance, if a model is a high performer and the most common rating is a 4, should disagreeing on a minority rat-ing such as a 1 vs. 2 drastically reduce the IRA? We further demonstrate, using a toy example in Appendix A.3, how disagreement on minority la-bels can substantially reduce IRA measured using a Krippendorff’s-α while percentage agreement barely changes. What is considered as “low” IRA can vary from task to task, as complex tasks or difficult samples tend to have low IRA. This is crucially important for interpreting IRA and is par-ticularly relevant for NLG evaluation. NLG tasks have typically reported relatively low IRA, e.g., average Krippendorff’s-α of 0.62, the standard interpretation is that experi-ments with scores less than 0.67 must be deemed unreliable. For instance, rating “How good is the generated story?” is more likely to have a lower IRA compared to “Is 99 the largest 2-digit number”. Hence, while it is mandatory to measure IRA, it is important to ensure that the scores are interpreted in the context of the task. Performing detailed anal-ysis including computing IRA using multiple met-rics such as baseline percentage agreement and vi-sualizing the item-wise agreement scores can help analyze the results. Researchers have also called for further investigation to understand the useful-ness of a measure for a given problem, demonstrat-ing the challenges in selecting an appropriate IRA measure (ten Hove et al., 2018). Scoring criteria 3.3 Scoring criteria refers to “What aspects to score". The 4 common scoring criteria covered in NLP liter-ature are (a) Linguistic Fluency - the quality of sin-gle sentence (b) Coherence - overall flow or read-ability (c) Relevance - importance of content (d) Factuality - factual correctness. Also note that the nomencla-ture used to indicate a given criterion need not be consistent, e.g., as fluency vs. naturalness, across various studies (van der Lee et al., 2019). Eval-uation criteria also need to be customized across NLP tasks, and this is necessary as (a) the crite-ria will vary between domains and tasks (b) the generated text almost always needs to be evaluated against multiple criteria. For instance, Freitag et al. (2022) pro-pose the use of Multidimensional quality metrics (MQM) for evaluating machine translation. MQM is a generic framework for evaluating translation quality and provides a catalog of over 100 issues or error types organized in a hierarchy that evaluators can check for. This hierarchical categorization of errors enables granular as well as coarse-grained analysis of the quality of translation, and can be adapted for NLG evaluations. In addition to the exhaustive evaluation catego-rization provided in the MQM framework, respon-sible AI (RAI) must be factored into human evalua-tion. Sun et al. (2024) propose 6 categories for RAI evaluation – truthfulness, safety, fairness, robust-ness, privacy, and machine ethics. Domain-specific customization and extensions also form an inte-gral part of evaluation. For instance, evaluating the effectiveness of an LLM for a domain such as legal should include additional criteria such as case analysis and charge damages calculation. See conceptual view in Table 1. 3.4 Differentiating “To differentiate is to identify the differences be-tween things” (vocabulary.com). The test sets used in evaluation must be able to differentiate between the various capabilities as well as the weaknesses of generative LLMs. For instance, when the test sets are relatively easy, most models can achieve high scores and seem very capable. Conversely, when the test sets are very difficult, models might achieve low scores, making the models seem in-effective. Therefore, tests that are ineffective in differentiating capabilities can lead to (a) incorrect conclusions about the capabilities of a model, (b) poor calibration or ranking of various models, or (c) inability to identify any improvements or degra-dation between model versions, despite spending significant resources retraining new models. Hence, constructing effective tests that can differentiate model capabilities is crucial, otherwise evaluation simply results in wasted effort. Traditionally, models have been evaluated using public datasets and benchmark such as GLUE and Super-GLUE. These evaluations are not without their share of problems as models can exploit weaknesses in the official test sets re-lying on shortcuts or spurious correlation – such as length of the input – to predict the target label achieving high performance yet non-generalizable beyond the official test set. In addition to these problems, evaluating the cur-rent generation of SOTA LLMs poses further chal-lenges. 1) LLMs are trained on billions of tokens available on the internet, and the training data used is rarely well documented. Hence, these LLMs may have consumed public benchmark test sets as part of their training data. 2) The models are generative, producing natural language output unlike a model trained on a classification task, making it difficult to automate evaluation and thus are far more reliant on human evaluation. 3) The LLMs are capable of following natural language instructions to solve a vast variety of tasks, hence they need to be evalu-ated on a range of instructions provided as prompts that emulate end-user use cases. Emulating end-user use cases is crucial, as there are fundamental differences in the tasks in NLP benchmarks vs. the kind of questions or problems users can ask an LLM. Firstly, end users tend to be very creative and prompt the LLM for all kinds of queries. This diversity is particularly important in safety critical applications such as Medicine, as evaluation blindspots can potentially lead to harm-ful consequences. Secondly, the same message or question can be framed (prompted) in many ways. Hence, test scenarios need to consider the vari-ous natural ways in which users write their inten-tions when interacting with a LLM. If these aspects are not taken account when constructing a test set, solely relying on traditional NLP tasks such as sen-timent analysis can result in traditional SOTA mod-els outperforming ChatGPT (Koco ́n et al., 2023), when clearly this is not reflective of the end-user experience. Hence, we argue that the test cases used for evaluating LLMs need to represent end user scenarios, in addition to standard NLP tasks. Benchmarks such as Big-Bench (bench authors, 2023) take a step towards this direction by enabling GitHub contributors to add new tasks to the bench-mark. Furthermore, the same prompts may not be effective against all LLMs. Hence, prompts might often have to be customized for individual models, making benchmarking non-trivial. In light of these new challenges, curating ef-fective test sets is a critical problem that the NLP community must tackle. Benchmarks such as HELM to evaluate LLMs use multiple public datasets across various tasks such as Question Answering (QA) and sentiment analysis and measure aspects beyond accuracy to include toxicity and bias. While these are steps in the right direction, the effectiveness of public test sets such as IMDB dataset for sentiment analysis or XSum for summarization in measuring the capabili-ties of SOTA LLMs can be limited for the reasons discussed earlier. These datasets simply do not suf-ficiently represent end-user use cases. XSum also has hallucinated content in over 75% of its gold summaries, demonstrating further weaknesses in the test sets themselves. Robustness testing is also a key aspect of eval-uation, as models can be vulnerable to basic per-turbations such as capitalization, white spaces and prompt formatting and need not understand basic linguistic concepts such as negation. While efforts such as DynaBench to curate progressively harder test sets and behavior testing of models are promising approaches to curate effective test sets, curation still needs more research to make significant progress to target end-user use cases to evaluate LLMs. Responsible 3.5 In human evaluation, the Responsible pillar has to consider two aspects, a) Is the model behavior responsible? b) Do the human evaluators introduce bias to the evaluation results? While there is no for-mal definition of Responsible AI, at the very least it entails fairness, safety, truthfulness, and privacy. Searching papers with “respon-sible” in their title results in 20 papers, expanding the search to include terms such as “bias” and “pri-vacy” results in ≈1000 or 1% of the papers in ACL Anthology, while none of the human evaluation papers mention responsible AI related keywords in their title or abstract (query in Appendix A.2.8), a telltale sign that more work is needed. 1. Is the model behavior responsible? An-. swering this question requires evaluating the model for bias, its ability to withstand privacy attacks, truthfulness, and whether the responses are safe. Bias is “prejudice in favor of or against one thing, person, or group”. Bias affects groups by gender, race, culture, religion, geography, and disability. Thus, the tests sent for human evaluation need to specifically cater to these cases and report the performance of various subgroups to ensure that the LLM behaves responsibly. This requires that the tests have metadata curated such as race to be able to report across these segments. Generative LLMs are susceptible to leaking pri-vate details such as person identifiable information (PII) from the training data, including when sub-ject to privacy attacks. One method to mitigate such pri-vacy leaks is to obfuscate PII information such as names and locations from training data using tech-niques such as Pseudonymization – recognizing privacy-sensitive information and replacing them with realistic substitute and differential-privacy-based approaches that add noise to the input. Safety en-sures that the models do not generate content that harms a person’s physical safety or mental health. Red teams, a group of people authorized to imitate an adver-sary’s attack or exploitation capabilities, are used to evaluate the robustness, safety, and privacy of LLMs. 2. Do the human evaluators introduce bias to. the evaluation results? As discussed previously, when a substantial portion of evaluation relies on human perception, the diversity of the human eval-uators plays a key role in how representative the results are of the wider population. Despite this, less than 3% of papers report demographic informa-tion about their evaluators (van der Lee et al., 2021). Having a small group of evaluators or even when the size is large, aspects such as selection bias can result in biased results. Hence, we call for human evaluation to consider the evaluator demographic to mitigate bias effects in the evaluation. Scalability 3.6 Human evaluation is expensive, yet large volumes of test cases are necessary to differentiate model capabilities and ensure consistency. Hence, opti-mizations to reduce cost and time is crucial for wider adoption. Automating parts of human eval-uation can reduce cost. For instance, automation might be potentially helpful to shortlist a set of candidates for human evaluation. While the short-comings of n-gram-based automated evaluations using metrics like Rouge are well stud-ied, approaches such as us-ing LLMs to evaluate LLMs need further exploration. Firstly, the effectiveness of LLM-based evaluation is measured using its correlation with human evalu-ation, hence the human evaluation procedures need to be strengthened first to draw comprehensive and robust conclusions, creating a chicken-and-egg problem. Secondly, LLMs to evaluate LLMs should take into account the differences between perception-based metrics and evaluations that rely on facts, as discussed in Section 3.1.1. Studies that attempt to reduce the turn-around times of human evaluation itself are limited, less than 50 papers mention “cost” or “scale” with hu-man eval in the title (see Appendix A.2.7 for search query) in the last 20 years. Levinboim et al. (2021) report using coarse-grained caption (as opposed to fine-grained) annotations from crowdsourced users to be able to scale, a method that can be adopted for human evaluation. Huang et al. (2023) propose to identify effective test samples to reduce the cost of human evaluation in conversation systems. Design-ing a UI that enables the human annotators to work efficiently (one of the Es Efficiency in usability in section 3.1) can reduce time. Given the limited amount of work in this area of scalability of human evaluation, we call for further research. 4 Conclusion & Moving forward We presented the ConSiDERS-The-Human evalua-tion framework to keep up with the increasing capa-bilities of LLMs. We highlight the effects of human emotions and cognitive biases on evaluation, given how commonly the perception-based metrics are used to evaluate aspects such as truthfulness. We, hence, encourage researchers in NLP to collaborate with their counterparts in UX, HCI & psychology to ensure that the evaluation measures the right things the right way and the results are interpreted accurately. We also call for further research in crit-ical areas – including curating effective test sets, scalability of human evaluation, and responsible AI components such as privacy, bias, robustness, and safety considerations, when evaluating increasingly powerful & ubiquitous generative LLMs. 5 Limitations Human evaluation is a challenge, especially given the increasing capabilities of SOTA LLMs. Firstly, LLMs have many potential applications and effec-tively evaluating each application or domain might need customization. Our main aim in this paper is to provide a generic framework extensible for specific domains or applications. Effectively cus-tomizing for individual cases might require trial and error. Secondly, perception, whilst important, cannot solely dictate how the quality of LLMs is measured. Humans use heuristics to make deci-sions, and evaluation has to cater to these heuris-tics. While there are over 180 cognitive biases, in our paper we only highlight a few that can impact evaluation, we specify this limitation in Section 2.3 as well. Acknowledgements. We would like to thank our colleagues Seonwoo Min and Han-Chin Shing for reviewing this paper and providing feedback. 1. Were expert annotators independently able to follow the annotation guidelines and achieve higher inter-annotator agree-. ment? 2. Is there a mechanism for evaluators to report issues such as ambiguity in the guideline? High task complexity 3. Can you simplify the task by breaking down the task into easier tasks? 4. Are you asking multiple questions in a single task? If yes, split each question into separate tasks so that a) the answer to. one question doesn’t bias another one, b) the annotators feel positively motivated that they can complete a given task fast. Ill-suited evaluators 5. Is there a qualification exam for evaluators? 6. Do you insert exam and attention check quality control metrics during evaluation to identify the quality of individual. annotators as part of each batch of evaluation? 7. Does your task need fresh annotators? For instance, if your task is to measure how good the LLM’s instructions are, you. need to make sure that the annotators are new and not used to how to complete the task. Small number of evaluators and/or test set 8. Is it possible to include more evaluators? 9. Is it possible to increase the size of the test set? Rating scales such as the Likert 10. Are you trying to measure perception or non-perception-based aspects such as truthfulness? If you are measuring non-. perception-based metrics, avoid the Likert scoring, and modify the task to collect many more objective metrics such as extracting facts and verifying each fact. Inter-rater agreement 11. Which primary IRA measure did you use? 12. Does the primary measure take into account how the evaluation is designed, such as the aspects defined by Gisev et al. (2013)? 13. Do you expect your task to be inherently unbalanced? If not, do you report baseline percentage agreement to verify if the. observed chance estimation lowers the overall IRA? 14. Is the item-wise IRA higher for some items and not the others? The ones with lower IRA might be pointing to higher. complexity tasks. Report on the distribution of items IRA to troubleshoot the problem. 15. Are the qualifications of the human evaluators similar? If not, the disparity in the evaluators’ skills can lead to lower IRA. 16. Do you continuously measure IRA for each evaluation? If yes, did you observe a sudden change in IRA? It might be due. to changes in annotators (e.g., adding new annotators) or changes in guidelines or tasks. New guidelines and tasks take a few iterations to settle. Scoring Criteria 1. Is the model evaluated on typical dimensions: Fluency, Coherence, Relevance, Factuality? 2. What multi-dimensional domain-specific criteria were the model evaluated on? 3. What are the responsible AI criteria the model was evaluated on? Differentiating 1. How many test cases (test examples) did you use to evaluate each of the criteria? 2. What were the end user use cases, e.g., legal document summarization, were tested? If so, report the end user test cases. their corresponding number of tests, and the scores per user case per criteria. 3. Do you suspect that some of the test cases may have already been used during LLM training? If not sure, answer not sure. If yes, where possible, report the percentage of test cases that may have been impacted. 4. Did you run robustness tests to evaluate model weaknesses? What were the scenarios covered, e.g., semantic preserving. perturbations - such as sensitivity to white spaces, synonyms, etc? User Experience 1. Were rating denoising algorithms applied on the rating-based metrics, such as the Likert, to account for the cognitive. uncertainty of the human evaluators? 2. Did you split the content into atomic facts for criteria, such as factual completeness and truthfulness, that require inspecting. individual traits of the model-generated text? If so, detail the criteria. 3. Is the Likert scale used appropriately to measure perception-based aspects such as readability? 4. During human evaluation, were examples shuffled properly so that the evaluator cannot tell which model generated which. example? 5. Did you test the actual usability of the model’s output, in the context of the end-user application (aka extrinsic evaluation)? If so, briefly describe the details. Responsible 1. Was safety testing performed? If yes, how many test cases were used and what were the test scenarios? 2. Was privacy testing performed? If yes, how many test cases were used and what were the test scenarios? 3. Was the model tested for bias? If yes, what subgroups such as gender or disability was the model evaluated for? 4. How many evaluators were used? What was the diversity of evaluators or several evaluators grouped by aspects such as. education qualifications, race, religion, gender, age groups, and nationalities? Scalability 1. Were parts of the human evaluation automated? If yes, which aspects did you attempt to automate, and what types of. automated metrics were used? 2. If you used LLM-based automation, please provide the details of the LLM and prompts. 3. Did you optimize the usability aspects for the annotators to reduce annotation time? If so, provide a summary of how this. was achieved. A.2 Meta-analysis of papers available in ACL Anthology A.2.1 Papers with “human” and “eval” in either abstract or title We search for abstracts or titles containing keywords “human” and “eval”, with ≈3900 papers, where 900 of those published in 2023 as detailed in Figure 1. (title|abstract=human and title|abstract=eval) A.2.2 Papers with “human” and “eval” in the title only We search for titles containing keywords “human” and “eval”, this results in around 238 papers as shown in Figure 2, reducing the number from 3900 when we include abstracts as seen above. ((title=human and title=eval)) A.2.3 Papers with phrase “human eval” in either abstract or title We search for abstracts or titles containing phrases human evaluation, and human - evaluation. Around 1300 papers were published. A.2.4 Papers with keywords “human” and “eval” in title/abstract and usability keywords in either abstract or title We search for titles or abstracts that mention human evaluation and contain usability keywords (usability, hci, user experience, and human computer). This results in 172 papers. The problem with this query is that it is too noisy as a result of looking for “human” and “eval” in either the title or abstract. For instance, while results contain some human evaluation, the paper’s primary focus is not on the design of human evaluation itself. Code Listing 1: Jabref search query for human eval and usablity ) A.2.5 Papers with phrase “human eval” in either abstract/title and usability keywords Code Listing 2: Jabref search query for phrase human eval in title and usablity in title or abstract ( ) ) A.2.6 Papers with keywords “human” and “eval” in title and usability keywords Out of 238 paper keywords human and eval in the title “(title=human and title=eval)", only 16 mention usability-related keywords in the title or abstract ACL. Code Listing 3: Jabref search query for phrase human in title and usablity in title or abstract ) A.2.7 Paper that attempt to scale or optimise human evaluation Code Listing 4: Jabref search query for phrase human in title and cost / scale in title or abstract ( ) Code Listing 5: Jabref search query for auto eval human in title ((title=auto and title=eval) ) A.2.8 Paper that attempt to mention responsible AI related terms in their title Code Listing 6: Jabref search RAI terms in title (((title=responsib) or (title=fair) or (title=truth) or (title=trust) or (title=privacy) or (title=bias) or (title=safe)) and (not (title=inductive and title=bias))) Code Listing 7: Jabref search Human eval in title and RAI in asbtract (not (abstract|title=inductive and abstract|title=bias))) A.3 Krippendorff’s Toy example In this example, we demonstrate with 6 items, 6 raters where each item is rated exactly by 3 raters how simply changing 1 label (row 4, col 2) the Krippendorff’s-α drops from 0.7 to 0.24, while percentage agreement only drops from 94.4 to 88.9. In scenario 3, we again just change 1 label, where the % agreement remains the same at 88%, but Krippendorff’s-α drops from 0.24 to -0.06. import numpy as np import krippendorff from collections import Counter def computeagreement(strreliabilitydata): reliabilitydata = [] for coder in strreliabilitydata: reliabilitydata. append ([ np.nan if l == "" else int (l) for l in coder.split ]) computekrippendorff(reliabilitydata) computepercentageagreement(reliabilitydata) def computekrippendorff(reliabilitydata): agreement = krippendorff.alpha(reliabilitydata = reliabilitydata, levelofmeasurement = "nominal") print (np.matrix(reliabilitydata)) print ("Krippendorff 's alpha for nominal metric: ", round (agreement, 2)) def computepercentageagreement(reliabilitydata): Percentage agreement, only for binary values. itemwisedata = np. array (reliabilitydata).T itemwiseagreement = [] print (" Percentage agreement ") for item in itemwisedata: rowlabels = [ l for l in item if not np.isnan(l) ] highestfrequency = Counter(rowlabels).mostcommon [ 0 ][ 1 ] itempercentage = round (100 highestfrequency / len (rowlabels), 2) itemwiseagreement. append (itempercentage) print (rowlabels, itempercentage) print ("Percentage agreement:", np. mean (itemwiseagreement)) print print (" Scenario 2: Change just 1 label, results in much lower alpha )) print print (" Scenario 3: Same % agreement as Scenario 2, but much lower alpha This produces the following output [[ 1. 0. nan nan nan nan] [ 1. nan 1. 1. 1. nan] [ 1. 0. 1. 1. 1. nan] [nan 0. 1. 1. nan 1.] [nan nan nan nan 1. 1.] [nan nan nan nan nan 0.]] Krippendorff 's alpha for nominal metric: 0.7 ") computeagreement(strreliabilitydata = ( "1 0 ", coder A. "1 1 1 1 ", coder B. "1 0 1 1 1 ", coder C. " 1 1 1 1", coder D. " 1 1", coder E. " 0", coder F. ") computeagreement(strreliabilitydata = ( "1 1 ", coder A. "1 1 1 1 ", coder B. "1 0 1 1 1 ", coder C. " 1 1 1 1", coder D. " 1 1", coder E. " 0", coder F. )) Scenario 1: Baseline Percentage agreement [1.0, 1.0, 1.0] 100.0 [0.0, 0.0, 0.0] 100.0 [1.0, 1.0, 1.0] 100.0 [1.0, 1.0, 1.0] 100.0 [1.0, 1.0, 1.0] 100.0 0. nan nan nan nan]. [ 1. nan 1. 1. 1. nan] [ 1. 0. 1. 1. 1. nan] [nan 1. 1. 1. nan 1.] [nan nan nan nan 1. 1.] [nan nan nan nan nan 0.]] Krippendorff 's alpha for nominal metric: 0.24 1. nan nan nan nan]. [ 1. nan 1. 1. 1. nan] [ 1. 0. 1. 1. 1. nan] [nan 1. 1. 1. nan 1.] [nan nan nan nan 1. 1.] [nan nan nan nan nan 0.]] Krippendorff 's alpha for nominal metric: -0.06 A.4 Papers in medical journals that use the Likert scale for evaluating ChatGPT We searched medical journals as follows and manually selected 19 (not cherry-picking) papers that used human evaluation the Likert scale to assess ChatGPT. We find that the Likert scale was used to evaluate factual completeness, saliency correctness in 9/19 papers, 4/19 of papers use the Likert scale appropriately to measure user perception and for the rest of 6/19 we were not sure of what the criteria meant, for details see Table 3. • Nature & sub-journals: Performed search within the nature website using keywords ⟨gpt medical. likert⟩ • Lancet: Performed search within the Lancet website using keywords ⟨gpt likert⟩. • JMIR: Searched for ⟨gpt likert, jmir⟩ on Google Scholar and used the first 2 pages of results to filter relavant context. Scenario 2: Change just 1 label, results in much lower alpha [[ 1. Percentage agreement [1.0, 1.0, 1.0] 100.0 [0.0, 0.0, 1.0] 66.67 [1.0, 1.0, 1.0] 100.0 [1.0, 1.0, 1.0] 100.0 [1.0, 1.0, 1.0] 100.0 [1.0, 1.0, 0.0] 66.67 Percentage agreement: 88.89 Scenario 3: Same % agreement as Scenario 2, but much lower alpha [[ 1. Percentage agreement [1.0, 1.0, 1.0] 100.0 [1.0, 0.0, 1.0] 66.67 [1.0, 1.0, 1.0] 100.0 [1.0, 1.0, 1.0] 100.0 [1.0, 1.0, 1.0] 100.0 [1.0, 1.0, 0.0] 66.67 Percentage agreement: 88.89