Published Research & Projects

Burrus, J., Naemi, B., & Mattern, K., & Roberts, R. D. (2017). Building better students: Preparation for the workforce. Cambridge, MA: Oxford University Press.

What is Inside?

Led by a team of experts this edited, multidisciplinary book discusses a variety of issues surrounding workforce readiness in the 21st century. It includes the latest research, practice, and policy on what has emerged as a particularly fertile field. By featuring such topics as how to measure and define workforce readiness; how to prepare students for the workforce; and bridging the gap between college and the working world, this volume is a necessary contribution to today’s “skills gap” literature. Indeed, it appears increasingly timely as society works to not only secure our own economic futures, but our children’s futures, as well.

Lipnevich, A. A., Preckel, F., & Roberts, R. D. (Eds.) (2016). Psychosocial skills and school systems in the 21st Century: Theory, research, and practice. New York: Springer.

What is Inside?

This book provides a comprehensive overview and in-depth analysis of research on social and emotional learning (SEL) skills, examining both theory and areas of application. It discusses students’ SEL skills both as components of academic success and desired educational outcomes in grades K through 12. The book describes an organizing framework for SEL skills and examines a range of specific constructs that includes achievement, motivation, self-efficacy, creativity, emotional intelligence, resilience, and the need for cognition. In addition, it reviews specific school-based interventions and examines issues that concern the malleability of SEL skills. It addresses issues relating to the integration of SEL skills into school curriculum as well as large-scale assessment policies.

Matthews, G., Zeidner, M., & Roberts, R. D. (2011). Emotional intelligence 101. New York: Springer.

What is Inside?

This book discusses one of the most pivotal new psychological concepts of the twenty-first century -- emotional intelligence (EI) -- where the lead authors have been at the forefront for the past two decades.

Ziegler, M., MacCann, C., & Roberts, R. D. (Eds.) (2011). New perspectives on faking in personality assessment. New York: Oxford University Press.

What is Inside?

This edited book showcases a diverse range of opinions from leading international experts in the field of personality and social and emotional learning skills assessment. It offers new practical and theoretical insights into the prevention and correction of faking and other response biases that would otherwise pose serious problems to the assessment of noncognitive constructs.

Zeidner, M., Matthews, G., & Roberts, R. D. (2009). What we know about emotional intelligence: How it affects learning, work, relationships, and our mental health. Cambridge, MA: MIT Press.

What is Inside?

Emotional intelligence—the ability to perceive, regulate, and communicate emotions, to understand emotions in ourselves and others—has been the subject of best-selling books, magazine cover stories, and countless media mentions. It has been touted as a solution for problems ranging from relationship issues to the inadequacies of local schools. But the media hype has far outpaced the scientific research on emotional intelligence. In What We Know about Emotional Intelligence, three experts who are actively involved in research into EI (one of whom is a co-founder of RAD Science, the other an academic affiliate) offer a state-of-the-art account of EI in theory and practice. They tell us what we know about EI based not on anecdote or wishful thinking but on science. This authored book -- which was awarded a Professional Award for Scholarly Excellence (PROSE) in Biomedicine and Neuroscience by the Association of American Publishers -- looks at current knowledge about EI with the goal of translating it into practical recommendations in work, school, social, and psychological contexts.

Kyllonen, P. C., Roberts, R. D., & Stankov, L. (Eds.) (2008). Extending intelligence: Enhancement and new constructs. New York: Routledge.

What is Inside?

This volume presents research from a variety of perspectives on the enhancement of human skills, both cognitive and noncognitive. It is organized around five themes – enhancement via instruction; enhancement via development (over the life cycle); enhancement over time; enhancement via new constructs; and new directions in enhancement.

Matthews, G., Zeidner, M., & Roberts, R. D. (Eds.) (2007). The science of emotional intelligence: Knowns and unknowns. Series in Affective Science. New York: Oxford University Press.

What is Inside?

This book is an edited collection of papers by internationally renowned experts in areas relevant to emotional intelligence, social emotional learning, and related constructs. It provides a scientific definition and taxonomic model of emotional intelligence, explores the state of the art in assessment and application, and make recommendations for future research.

Schulze, R., Freund, P. A., & Roberts, R. D. (Eds.) (2005). Emotionalen Intelligenz: Ein Internationales Handbuch. Göttingen, Germany: Hogrefe.

What is Inside?

(From the cover) Emotionale Intelligenz (EI) wird als die Fähigkeit verstanden, Emotionen in Bezug auf sich selbst und andere Personen wahrzunehmen, auszudrücken, zu verstehen und zu regulieren. Die EI ist derzeit eines der bedeutenden psychologischen Konstrukte, dem einerseits sowohl von Experten in der Wissenschaft als auch in der breiten Öffentlichkeit Aufmerksamkeit geschenkt wird und das andererseits Gegenstand kontrovers geführter wissenschaftlicher Diskussion ist. In diesem herausgegebenen Band präsentieren führende Experten ihre Perspektiven zu Theorien der EI, Ansätze zur Messung sowie theoretische Konzepte und Untersuchungsergebnisse zu Determinanten und Konsequenzen im organisationspsychologischen, pädagogischen und klinischen Kontext. Darüber hinaus werden viel versprechende Konzeptualisierungen und bedeutende empirische Befunde aus der empirischen EI-Forschung dargestellt, konzeptuelle und empirische Probleme reflektiert und eine kritische Bestandsaufnahme des wissenschaftlichen Status Quo der Emotionalen Intelligenz geliefert. Ein Kommentarkapitel, in dem die verschiedenen Beiträge integriert werden, unterstützt den Leser abschließend bei der Bewertung der vorgestellten Konzepte und Befunde.

Schulze, R., & Roberts, R. D. (Eds.) (2005). Emotional intelligence: An international handbook. Cambridge, MA: Hogrefe & Huber Publishers.

What is Inside?

This book, co-edited by RAD Science personnel and one its key academic affiliates, brings together leading experts from around the world to present their perspectives on the status of emotional intelligence (EI), social and emotional learning, and related constructs. It covers theories of EI and assessment approaches in depth, as well as theoretical concepts and research findings on the antecedents and consequences of EI in occupational, educational, and clinical settings. The contributions provide an overview of the empirical evidence that supports (as well as contradicts) many common assumptions about EI and its relation to other forms of intelligence. The book thus reflects the diverse approaches to finding solutions for the still unresolved conceptual and empirical problems, and offers a critical appraisal of the current status of EI.

Matthews, G., Zeidner, M., & Roberts, R. D. (2003). Emotional intelligence: Science and myth. Boston, MA: MIT Press.

What is Inside?

This book (which was awarded an Honorable mention in the “The Outstanding Professional and Scholarly Titles of 2002” by the Association of American Publishers) offers a comprehensive critical review of EI. It examines current thinking on the nature, components, determinants, and consequences of EI, and evaluates the state of the art in EI theory, research, assessment, and applications. It highlights the extent to which empirical evidence supports EI as a valid construct and debunks some of the more extravagant claims that appear in the popular media. Finally, it examines the potential use of EI to guide practical interventions in various clinical, occupational, and educational settings.

Ackerman, P. L., Kyllonen, P. C., & Roberts, R. D. (1999). (Eds.) Learning and individual differences: Process, trait, and content determinants. Washington, DC: American Psychological Association.

What is Inside?

Researchers from the United States and seven other countries present leading-edge research and theory concerning the topic of learning and individual differences, which continues to be a vibrant area for multidisciplinary investigation. While cutting across traditional boundaries in psychology, researchers provide incisive historical reviews and integrative analyses of critical research.

Roberts, R. D., Martin, J., & Olaru, G. (2015). A Rosetta Stone for Noncognitive Skills Understanding, Assessing, and Enhancing Noncognitive Skills in Primary and Secondary Education. Asia Society and ProExam: New York.

What is Inside?

This authored paper, sponsored by the Asia Society, distils research showing that social and emotional learning (SEL) skills are as important for academic performance as are cognitive abilities, and that these traits positively predict performance, behavior, and satisfaction in work life and at home. In the context of recent studies showing that personality can change over the lifespan, research suggests that noncognitive factors could and should play a more pivotal role in educational policy and practice than hitherto realized. The paper concludes with a call to action for more valid assessments of SEL; an activity that RAD Science has been fully engaged in throughout 2018.

Kyllonen, P.C., Lipnevich, A. A., Burrus, J., & Roberts, R. D. (2014). Personality, motivation, and college readiness: A prospectus for assessment and development. ETS Research Report Series, 1, 1-48.

What is Inside?

This review article concerns how noncognitive constructs—personality and motivation—can be assessed and developed to increase students' readiness for college. We propose a general framework to account for personality and motivational differences between students. We review numerous studies showing that personality and motivational factors are related to educational outcomes, from early childhood to adulthood. We discuss various methods for assessing noncognitive factors, ranging from self‐assessments to performance tests. We consider data showing that personality and motivation change over time and find that particular interventions have proven successful in changing particular personality facets, leading to increased achievement. In a final section we propose a strategy for implementing a comprehensive psychosocial skills assessment in middle and high school, which would include setting proficiency standards and providing remedial instruction.

Petway II, K. T., Rikoon, S. H., Brenneman, M. W., Burrus, J., & Roberts, R. D. (2016). Development of the Mission Skills Assessment and evidence of its reliability and internal structure. ETS Research Report Series, 1, 1-15.

What is Inside?

This technical report details the psychometric properties of the early ancestor of both ACT TesseraTM and SELF+eTM: The Mission Skills Assessment (MSA). The MSA is an online assessment that targets 6 noncognitive constructs: creativity, curiosity, ethics, resilience, teamwork, and time management. Each construct is measured by means of a student self‐report scale, a student alternative scale (e.g., situational judgment test), and a teacher report scale. Use of the MSA provides schools with the opportunity to examine and monitor development of noncognitive skills in their students from Grade 6 to Grade 8. The use of the MSA has facilitated changes to curricula and more targeted instruction. The MSA scales exhibited meaningful relationships with standardized test scores and absenteeism, suggesting that these constructs play an important role in student behaviors and academic performance. This report presents reliability and factor analysis evidence for the student self‐report and teacher report scales of the MSA to highlight its capacity to measure the 6 constructs.

Burrus, J., Jackson, T., Holtzman, S., Roberts, R. D., & Mandigo, T. (2013). Examining the efficacy of a time management intervention for high school students. Educational Testing Service Research Report No: RR-13-25. Princeton, NJ: Educational Testing Service.

What is Inside?

It is one thing measuring SEL skills, it is another to develop interventions that show a meaningful effect. This report focuses on one such attempt, focusing specifically on Work Ethic/Conscientiousness as it might be assessed by SELF+e. This report presents the results of 2 quasiexperimental studies conducted to examine the efficacy of a new time management intervention designed for high school students. In both studies, there was no difference between the treatment and control groups in improvement in self‐reported time management skills as a result of the intervention. However, the treatment group reported significantly greater improvement than the control group for secondary outcomes such as stress (Studies 1 and 2), anxiety (Studies 1 and 2), depression (Study 1), and knowledge of time management strategies (Study 1). Additionally, advisor ratings of student time management skills were higher for the treatment than for the control group in Study 2. Implications and suggestions for improving the intervention are discussed.

Zhuang, X., MacCann, C., Wang, L., Liu, L., & Roberts, R. D. (2008). Development and validity evidence supporting a teamwork and collaboration assessment for high school students. Educational Testing Service Research Report No: RR-08-50. Princeton, NJ: ETS.

What is Inside?

Various policy papers and research studies assert that teamwork is one of the most important SEL skills for students to learn if they are to become meaningful contributors to the 21st century workforce. However, outside of organizational psychology and adult populations, few reliable assessments of this construct exist, with suitable validity evidence scant or nonexistent. To redress this imbalance, teamwork assessments for high school students were developed using multiple methods: self‐report ratings, situational judgment testing, and teacher reports. Exploratory factor, confirmatory factor, and latent class analyses were used to determine the structure of the scales. Measures showed reasonable reliability and promising validity evidence, relating to each other and to academic achievement, while remaining relatively independent from personality. The advantages and disadvantages of each methodology and the potential applications for identification and intervention, selection, and evaluation of training programs are discussed. This report also serves as an archival document for the teamwork and collaboration assessments that have been developed previously for high school students.

Burrus, J., & Roberts, R. D. (2012). Dropping-out of High School: Prevalence, risk factors, and remediation strategies. R&D Connections, 18, Princeton, NJ: Educational Testing Service.

What is Inside?

The article discusses the high school dropping out phenomena in the United States. The authors estimate that the number of 16- to 24-year-old dropouts in the United States may exceed 4 million at any one time. They discuss the social and economic consequences for the affected individuals and for society at large. They state that it is possible to identify the great majority of students on track to drop out and the schools most likely to produce dropouts. Research indicates that the dropout problem can be ameliorated, assuming that there is a political will to do it. One such solution involves paying greater attention to measures of social and emotional learning indicators and interventions than has been given in the past.

Burrus, J., Elliott, D., Brenneman, M., Markle, R., Carney, L., Moore, G., Betancourt, A., Jackson, T., Robbins, S., Kyllonen, P. C., & Roberts, R. D. (2013). Toward a comprehensive understanding of student persistence and goal attainment: “Putting and keeping you on track”. Educational Testing Service Research Report No: RR-13-14. Princeton, NJ: Educational Testing Service.

What is Inside?

This report involves a new framework for retention, specifically targeting the college rather than the K12 market. Despite near universal acceptance in the value of higher education for individuals and society, college persistence rates in 4-year and community colleges are low. Only 57% of students who began college at a 4-year institution in 2001 had completed a bachelor’s degree by 2007, and only 28% of community college students who started school in 2005 had completed a degree 4 years later (National Center for Education Statistics, 2011). To address this problem, this report identified 3 goals. The first was to review the extant literature on persistence in higher education. The second was to develop a working model of persistence informed by our literature review. This resulted in a model centered on 3 basic categories of variables: those that put you on track towards persistence, those that push you off track, and those that keep you on track. The final goal was to outline a research agenda to develop student-centered assessments informed by our model, and we conclude with a discussion of this agenda.

Naemi, B., Burrus, J., Ventura, M., Kyllonen, P. C., & Roberts, R. D. (2012). Building a case to develop noncognitive assessment products and services targeting workforce readiness at ETS. Research Memorandum: ETS RM-12-23.

What is Inside?

This paper aims to establish the case for conducting research on the readiness of individuals for the workforce with specific emphasis on noncognitive indicators of readiness. The report includes a conceptual framework that encompasses noncognitive constructs and measures, as well as a review of the literature highlighting the importance of noncognitive predictors in education and the workforce.

Gertner, A., Zaromb, F., Schneider, R., Roberts, R. D., & Matthews, G. (2016). The Assessment of Biases in Cognition. MITRE Technical Report. McLean, Virginia: The MITRE Corporation.

What is Inside?

(From the Executive Summary) The Assessment of Biases in Cognition (ABC) is a new standardized assessment of biases in judgment and decision-making behavior that was developed by The MITRE Corporation and the Educational Testing Service (ETS) for the Intelligence Advanced Research Projects Activity (IARPA) Sirius Program. The purpose of the IARPA Sirius Program is to create serious video games designed to train intelligence analysts to improve their explicit knowledge of, and ability to recognize, six well-known cognitive biases and to significantly mitigate the influence of those biases on behavior as a result of this training. The six biases are: (1) confirmation bias (CB), (2) fundamental attribution error (FAE), (3) bias blind spot (BBS), (4) anchoring bias (ANC), (5) representativeness bias (REP), and (6) projection bias (PRO). The first version of the ABC (ABC-1) was developed for the first phase of the Sirius Program to assess mitigation of CB, FAE, and BBS. The second version of the ABC (ABC-2) was developed for use in second phase of the Sirius Program to assess mitigation of ANC, REP, and PRO.

The ABC-1 and the ABC-2 each include one recognition and discrimination (RD) scale and three behavioral elicitation (BE) scales, one for each bias. The RD scales consist primarily of multiple-choice items and are intended to assess declarative knowledge of the biases. The BE scales consist of a variety of innovative tasks intended to evaluate test-takers’ procedural knowledge regarding how to avoid committing the targeted biases in judgment and decision-making tasks specifically designed to give test-takers opportunities to commit those biases. Each version of the ABC is administered online using a customized test delivery platform developed by the MITRE Corporation and takes approximately 45 to 60 minutes to complete. The ABC-1 and ABC-2 both include three equated test forms. This facilitated evaluation of bias mitigation training outcomes by making it possible to compare test-takers’ performance on one form post-training with their pre-training performance on an alternate, equated ABC test form. This report summarizes the (1) test development process, (2) research conducted during the development and validity evaluation of the ABC, (3) validity argument for the ABC, and (4) suggestions for future research.

Jackson, T., Burrus, J., Bassett, K., & Roberts, R. D. (2011). Teacher leadership: Towards an assessment framework. Educational Testing Service Research Report No: RR-10-24. Princeton, NJ: Educational Testing Service.

What is Inside?

This report, co-authored by two of RAD Science’s co-founders, examines various definitions and frameworks that have been used to justify the emergence of a new category for the teacher professional: teacher leader. The emergence of this new professional category may lead to greater retention levels, and improved knowledge management and transfer within the teaching profession. Various key dimensions of this profession are examined, allowing us to highlight some key personal skills that would appear requisite for the teacher leader. An agenda for developing and validating assessments of teacher leadership is then proposed. It is argued that these assessments have the potential of legitimizing research within this field, as well as providing the opportunity to better understand what it takes to become a successful professional in this new domain of teaching practice.

MacCann, C., & Roberts, R. D. (2010). Development of a student health assessment system: Health knowledge, attitudes, and behaviors in middle-schoolers. Educational Testing Service Research Report No: RR-10-04. Princeton, NJ: Educational Testing Service.

What is Inside?

This report chronicles the development of assessments of nutrition and exercise knowledge, attitudes, and behavior that were administered to 383 eighth‐graders. Evidence for the validity of assessment scores was evaluated with five findings. First, parent‐ and self‐reported behaviors were similar and congruent for healthy eating and exercising but not for sedentary behaviors or unhealthy eating. Second, the theory of planned behavior structural model was confirmed for exercise and nutrition attitudes: Attitudes predicted exercising (R2=.34), healthy eating (R2=.21), sedentary behaviors (R2=.11), and unhealthy eating (R2=.09). Third, gender differences replicated previous research: Girls have better diets and nutrition attitudes whereas boys exercise more frequently. The assessments distinguished normal‐weight from obese individuals and showed incremental validity in predicting school grades. The paper concludes with a discussion of the assessments' potential for informing interventions.

David Taylor Design NJ Website Design Company