Formative assessment: quick surprise quizzes online in class in mathematics higher education
ABSTRACT. In literature, moments of Active Learning in which all students are working effectively committed are considered to be an added value. Even better if, at the same time, they provide a formative assessment opportunity in which students receive immediate feedback. Thus, in a Calculus curricular unit, 31 online surprise quizzes were administered in Moodle to 112 students, distributed across nearly all classes. Students can repeat the quizzes as many times as necessary until they obtain the correct answer, without grade deductions. In an anonymous online survey, almost every student indicated that the quizzes were useful. Although they were not mandatory, the participation rate was very high. Many mentioned they were more attentive due to the quizzes, finding them useful for feedback, for understanding the level they were reaching, and for learning new things. Teachers also confirm that it is a pedagogical strategy worth maintaining.
Professionalising teachers with a digital formative assessment tool: a case study of the SMART tests
ABSTRACT. SMART, a digital formative assessment tool for mathematics education, aids teachers by offering short online diagnostic checks and detailed analyses of response patterns. It identifies student misconceptions and provides teachers with valuable insights on students’ level of understanding. This should have the potential to not only influence teaching and student understanding but also the teachers’ professionalisation. This paper explores the potential impact of using SMART on five mathematics teachers' PCK in elementary algebra. Initial findings indicate the tool's effectiveness in enhancing teachers’ knowledge of student thinking and typical mistakes.
Leveraging formative assessment to develop students’ mathematical reasoning through images and student generated language
ABSTRACT. A sixth-grade teacher who taught at a highly diverse school in the United States participated in professional development activities during the 2022-23 school year to learn about and implement an instructional framework designed for use during problem-solving lessons. The framework is intended to guide teachers to simultaneously attend to developing their students’ mathematical reasoning and learning of the mathematics register. In a case study undertaken to examine teachers’ use of the framework, an example emerged of how formative assessment can be leveraged to support the development of students’ mathematical reasoning through student generated language and images. This study contributes to the research literature by demonstrating how the use of formative assessment can promote the use of language to “carry” a mathematical concept in the sense that language can help students create a mental image of that concept.
Designing with the Teaching for Robust Understanding framework: indicators for the activation and realization of formative assessment strategies
ABSTRACT. The study explores the implementation of formative assessment strategies in the context of algebraic thinking and argumentation within a teaching experiment. Teaching for Robust Understanding framework and theoretical references guide the task design. Specific indicators for two formative assessment strategies are developed, tailored on the learning goals, and examples of their instances (activation and realization by teachers and students) are provided. Future work will extend this analysis to other strategies and assess its applicability in other learning sequences.
Evaluating the reliability of a framework for mathematical activities using generalizability theory
ABSTRACT. This study investigates the measurement reliability of the Framework for Mathematical Activities (FfMA), developed to assess the quality of activity scripts and their implementations. Utilizing Generalizability Theory, the measurement reliability of scores obtained from the FfMA tool was determined. Data were collected based on a descriptive survey model. In this context, four activity scripts and classroom implementation videos of these scripts were requested from each of 20 middle school mathematics teachers. The data were scored independently by three raters using the FfMA tool. The scores obtained from the raters were analyzed using the EDUG 6.1e program. Findings indicate that the measurement reliability of the FfMA tool is considered reliable with a value of 0.78, and it does not fall below 0.70 even in scenarios with the minimum number of raters (2) in D studies. These coefficients suggest that the FfMA tool consistently measures the quality of mathematical activities.
ABSTRACT. Comparative judgement approaches to assigning grades to students’ work have received interest from mathematics education researchers over recent decades. These approaches involve assessors deciding which of two presented pieces of work is ‘better’, and the decisions are then converted into scores. Several objections have been raised to using comparative judgement for summative assessment and in this theoretical paper I respond to objections that such approaches are ‘not valid’. These include objections that there is a lack of evidence supporting validity, and that researchers assert comparative judgement is ‘intrinsically valid’ in ways that are incomplete and inconsistent. I argue that most validity objections are addressed by published evidence, and that the validity of applying comparative judgement to mathematics education assessments is a special case.
Contextuality of Application Tasks in Large-Scale Summative Assess-ments at Lower Competence Levels for Lower Secondary Education
ABSTRACT. This article investigates the item pool of measurements of mathematical competencies in students at or below the lowest proficiency level through a qualitative analysis of Germany's 2018 IQB Trends in Students Achievement. Contrary to conventional views, it emphasizes a nuanced understanding of the difficulty of application tasks in large-scale assessments, with a focus on contextualization. Introduc-ing a novel task pool designed for special educational needs students, the study addresses previous limitations in accurately assessing their performance. The qualitative analysis reveals improved au-thenticity and relevance in the new tasks, particularly in private contexts. The findings underscore the importance of refining authenticity and relevance criteria for application tasks at lower proficiency levels, providing valuable insights for inclusive education contexts.
Development and validation of a three-dimensional framework for classification of authentic tasks in mathematics
ABSTRACT. Global trends in mathematics education place increasing importance on problem-solving skills in authentic contexts. The range of mathematical problems with authentic content varies greatly, and a framework is needed to classify them. In this paper, the authors propose a three-dimensional framework for the classification of authentic tasks in mathematics. These dimensions, which combine the complexity of the task but can at the same time be analyzed separately, are (1) complexity of the mathematical model, (2) context of the given problem, and (3) strategic complexity connecting the problem and the mathematical model. To validate the framework, the levels of complexity for 12 tasks are determined and students' performance on these tasks is compared. This framework is a valuable tool for designing both learning and assessment tasks.
Assessment by skills of an interdisciplinary project in which mathematics is the common thread.
ABSTRACT. In this contribution we propose the design of an interdisciplinary project named “Dark ages?”. It is a Project Based Learning (PBL) and outdoor learning, in which Digital and manipulative tools, virtual and real experiences, in the classroom and in the real world, are intertwined. A formative and summative evaluation proposal is presented, and the first results analysed. Some results of a survey on how students experienced the project are also presented.
Open-ended items in digital formative assessments: Decision Trees as (AI-compatible) approach to reliably code students’ understanding?
ABSTRACT. Open-ended items, in which students draw images, explain meanings or argue, allow them to express their own mental representations of situations and make it possible to grasp even fragile concepts in nuances and details. However, those answers-types are rarely found in digital formative assessment, also because they are often difficult to evaluate. This paper reports on the integration of open-ended items into the digital formative assessments of the Mastering Math – Online-Check and exemplifies for an item on conceptual understanding of multiplication how current approaches of category-based scoring could be optimized by using decision trees to rate features of responses. In preparation for the integration of an automatic pre-coding by an artificial intelligence, an exploratory study is presented on the functioning of prompt-based classification of students’ answers by ChatGPT.
The potential of AI: generating answers for multiple-choice questions using ChatGPT
ABSTRACT. Computer-based multiple-choice tests reduce teachers' workloads and enable students to receive immediate feedback on their performance, which is why they are widely used in the education system. To create a high-quality multiple-choice test, the author has to develop plausible distractors to be included among the provided answers. Therefore, creating a high-quality test is time-consuming. The purpose of this paper is to determine the suitability of ChatGPT for generating answer options for arithmetic and textual multiple-choice questions using Estonian and English prompts as examples, based on typical mistakes made by students and their similarities and differences with the answer options created by a human expert. This paper uses ChatGPT3.5 to test if it is possible to generate answers for multiple-choice questions that are based on given parameters. The results show that it is possible to generate answer options for multiple-choice questions using ChatGPT.
Secondary mathematics teachers’ experiences of using ChatGPT to design probability and statistics assessment items
ABSTRACT. The purpose of this study was to explore secondary mathematics teachers’ experiences of using ChatGPT to design assessment items. For this purpose, we analyzed 22 secondary mathematics teachers’ conversations with ChatGPT and their survey responses in terms of overall experiences with ChatGPT, acceptance of ChatGPT, affordances and challenges of using ChatGPT. The results show that most teachers did not specify the purpose of assessment and only one teacher mentioned about the mathematical error in ChatGPT’s response. The teachers employed a wide range of follow-up questions in responding to the ChatGPT’s suggestions. The survey results showed that the teachers’ responses to the intention to use, the ease of use, and the usefulness of ChatGPT were polarized. The teachers perceived that ChatGPT provide affordances such as efficiency but identified limitations such as potential mathematical errors, ethical issues, and security of assessment.
ABSTRACT. This study concerns problem-posing as a means of assessment in upper-secondary mathematics education. The open character of problem-posing as a task, allows students to show their creativity but makes it difficult to control the focus on the learning goals. Problem-posing can be structured by adding an initial problem to the prompt. We aim to investigate how this form of structuring affects the resulting problems and the extent to which they reveal students' thinking and knowledge with respect to learning goals. In line with previous research on assessment through problem-posing by Kwek and by Mishra and Iyer, we classify the complexity of the problems. Additionally, we analyze whether the problems address the learning goals and are solvable. The main outcome is that structuring the problem-posing prompt is more suitable for assessment since the resulting problems align better with the learning goals and reveal more of the qualities and misunderstandings of the students.
Students’ feedback process using automated post-submission report in their modeling process
ABSTRACT. In this study, we explored a pair of students’ interactions with the automated post-submission report to learn how they use this tool to deepen their own modeling process. The students worked on a modeling activity consisting of four example-eliciting tasks in which they were asked to construct position-over-time graphs. The findings show that by interacting with the post-submission report, new sequences of modeling competencies use were identified, which enables fostering the student’s interpreting and validating the mathematical results.
Analytical readings’ method as a tool to assess the word problem-solving process involving rational numbers
ABSTRACT. Word problem solving in mathematics is essential, and the processes that students follow to solve them may vary. Consequently, teachers face the challenge of interpreting these strategies, while students sometimes have difficulties explaining their processes. The analytical readings’ method is presented as a resource for the assessment of the problem-solving processes, in addition to helping students develop skills in this area. In this context, the purpose of this study is to document reflection, especially among undergraduate mathematics students with an interest in teaching, on analytical readings as a tool to assess problem solving. Reflections highlight that the analytical readings’ method allow assesses students' comprehension of the problem, their performed operations, the reasoning behind them, providing a comprehensive view of the thinking process.
Effects of a teacher professional development program in formative assessment on teachers’ conceptions of feedback and assessment and their self-reported feedback practices
ABSTRACT. Formative assessment has been shown to have the power to improve student achievement. Therefore, many professional initiatives (PD initiatives) have been carried out to support teachers to developing their formative assessment practice. However, accomplishing such practices have been proven difficult. Among factors that are important for outcomes of PDs are teachers’ beliefs and conceptions. This study examines the effects of a PD in formative assessment on teachers’ conceptions of assessment and feedback, and self-reported feedback practices. These variables were measured through a surves in the beginning and at the end of the PD, and differences between the intervention and a control group were examined at both time points using factor analytic methods. For the intervention group, significant positive differences were found in both the means of important conceptions and in the strength of relationships, while this was not the case for the control group.
From automatic diagnosis to lesson plannings: how teachers use elements of a digital formative assessment tool
ABSTRACT. As the effectiveness of formative assessment depends on the form of implementation in school lessons, we investigate in this study which elements of the digital formative assessment tool SMART teachers use for which purposes. In an exemplary qualitative analysis of two teacher interviews, we found that teachers use the given teaching suggestions and materials for designing their upcoming lessons. Furthermore, they use the test items and didactical information about (mis)conceptions to professionalize themselves by gaining deep insights into students’ thinking.
Mathematics teacher educators' microteaching implementations: selection of settings and approaches to assessment and feedback for prospective classroom teachers
ABSTRACT. This study examines the microteaching implementations within mathematics method courses provided by mathematics teacher educators (MTEs) to equip prospective classroom teacher with mathematics teaching competencies. Surveying 65 MTEs across 71 universities offering teacher preparation programs at undergraduate level in Turkey, the research focuses on three pivotal aspects: topic selection, preferred settings, and evaluation and feedback approaches in microteaching. Findings highlight a significant inclination towards educator-led topic selection and a preference for peer-based settings for microteaching sessions. A majority of respondents also indicate customizing their own evaluation rubrics, demonstrating a tailored approach to assessment and feedback. The study further finds an inclination towards revealing evaluation criteria to candidates before teaching, facilitating a transparent feedback process that is supplemented by peer evaluations. We share the implications of these practices on formative assessment and feedback, essential for the development of teacher candidates, aligning with the broader educational goals of fostering comprehensive mathematical instruction capabilities.
Large language models as formative assessment and feedback tools? – A systematic report
ABSTRACT. This paper discusses the use of large language models (LLMs) for formative assessment and feedback in mathematics education. First, a brief introduction to the research on LLMs in mathematics education is given. Subsequently, the LLM ChatGPT 4.0 is systematically evaluated with regard to the aspects 1) Input and localization, 2) Assessment-quality, 3) Content and form of feedback, and 4) Adaptivity and receiver of feedback. It is shown that ChatGPT has the potential to provide meaningful feedback on mathematical work, but that its use is associated with a number of challenges.
Examining ChatGPT responses to TPCK assessment items
ABSTRACT. The development and, thus, the objective and valid assessment of the skill of selecting digital learning material (dLM) is important for pre-service teachers. In this paper, we compare the responses of ChatGPT 3.5 with responses from pre-service teachers to items for assessing this skill to gain insights into the capabilities of ChatGPT and the longevity of the items for its assessment. The results reveal that, for one, ChatGPT 3.5, so far, cannot analyze dynamic dLMs, and second, it does not make a decision on the use of the dLM but provides predominantly TPK but appropriate reasoning for using or not using it. ChatGPT TPK responses are comparable with pre-service teachers’ responses, and further studies are required to understand its impact fully. Still, the presented results support the projected effects of ChatGPT on assessments in teacher education and the evaluation of the skill of selecting dLM.
GeoGebra pop-up notification as formative feedback for learning computational thinking in mathematics lessons
ABSTRACT. Computational thinking (CT) is a fundamental skill for everyone that is relevant to the 21st-century skills. Initiatives have been carried out to enhance CT outside computer science (CS) courses. In this study, we integrated CT into mathematics lessons utilizing GeoGebra. Our lessons’ development was guided by the educational design research approach (EDR). When we piloted our lessons with junior high school students, we found that GeoGebra could provide a pop-up notification as formative feedback to support students’ debugging skills. This pop-up notification could act as a negotiated-style interruption and an immediate-style interruption. The first interruption is when the students get a pending message about the errors so that they will know it later. Meanwhile, the latter informs students to revise the errors immediately. In this paper, we will discuss how these interruptions as a formative assessment could be a means of support for students while learning CT+Maths lessons.
Assessment of students’ learning mathematics with technology using video-based activities in an online course
ABSTRACT. Using technology for mathematical learning, but also for assessing students’ mathematical learning has proven to enhance, support and impact mathematics education in innovative, yet challenging ways. One of the goals of the online asynchronous master’s module we designed is to prepare postgraduate students (who are either prospective or practicing mathematics teachers) for assessing school students’ mathematical learning when using digital technologies. Fostering postgraduate students’ noticing and interpreting skills when analysing and assessing mathematical learning while a digital tool is used, has become a key priority for the design of our module’s activities. This paper presents: (a) our current research study for investigating how best to support postgraduate students develop skills for assessing mathematical learning when using digital technologies; and (b) an innovative video-based activity that addresses this developmental need.
CLASSIFICATION tasks as basis for assessment – Family resemblance as principle for choosing mathematical objects
ABSTRACT. Formative assessment is aimed at supporting learning processes. Classification tasks can be used as a basis for formative assessment, particularly for evaluating students’ understanding of definitions and object properties. Here, we present six design principles for choosing mathematical objects in such tasks—across grade-levels and across the curriculum—using literature-based ideas about categories of objects, family resemblance, hierarchical structures of categories, and intuitiveness. We demonstrate how we used these principles for designing a digital classification task, and discuss further areas in which these design principles should be considered.
Unravelling (Mis)conceptions about Algebraic Letters: Exploring Response Patterns in the 'Meaning of Letters' SMART-test using Latent Class Analysis
ABSTRACT. Identifying typical hurdles, common errors and misconceptions in a certain domain is crucial to deepen our understanding of students’ learning. In this paper, we explore response patterns of 2051 German grade 7 and 8 students shown in the SMART-test “Meaning of Letters”, designed to assess (mis)conceptions regarding variables, more precisely, the letter-as-object misconception. Using Latent Class Analysis, we were able to identify six response patterns. These patterns are described and thoroughly analysed. They urge us to think more deeply about the interplay between (mis)conceptions and contexts and can help build valid assessment tools to diagnose students’ current understanding.
