ABSTRACT. Workshop 1: An Introduction to R 

Jonathan Weeks and Szu-Fu Chao

Must be preregistered. With the increased use of R for research and operational analyses, there is a growing need for individuals who are familiar with the application. This session will be conducted as a highly-interactive lecture combined with hands-on data analysis in two parts. No prior knowledge about R is assumed. Participants will be expected to bring their own laptop (PC, Mac, Linux).

Part 1 – 8:00-9:30 The first part of the session will be an introduction to R, including the use of basic R commands, accessing help and documentation, importing data, descriptive statistics, significance tests, and the creation of plots.

Part 2 – 9:30-11:00 The second part of the session will focus on more intermediate topics in R, including data manipulation, the use of loops, installing and working with packages, and writing your own functions. An overview of relevant packages for common statistical and psychometric analyses will be provided, and the use of RStudio will be discussed. Examples will be presented via a step-by-step approach.

ABSTRACT. Workshop 2: Designing Conversational Items Using Spoken Dialog Technology

Vikram Ramanarayanan, David Suendermann-Oeft, and Keelan Evanini

Must be preregistered. This workshop will introduce participants to the basics of designing conversational items for using spoken and multimodal dialog technology. The workshop will assume no prior knowledge of dialog technology, and will demonstrate the use of open-source software tools in building conversational items. The last half of the workshop will be specifically dedicated to a hands-on item building session, where participants will have a chance to design and deploy their own dialog item from scratch on a cloud-based dialog platform. Participants are encouraged to bring their own laptops (with Windows or Mac operating systems installed; Linux not supported). Additional software installation instructions will be sent prior to the workshop. 

Part I: State of the art (8 AM – 9:05 AM)

8:00 – 8:05: Welcome and Workshop Overview (Vikram)

8:05 – 8:25: Introduction to Spoken Dialog Technology (Vikram/DSO)

8:25 – 8:40: Designing a Library of Conversational Items (Keelan)

8:40 – 8:50: Pragmatics and Feedback for Language Learning (Veronika/Keelan)

8:50 – 9:05: Bleeding Edge Topics – Scoring, Accessibility, Psychometrics (Vikram)

Part II: Hands-on Tutorial Workshop (9:05 AM – 11 AM)

9:05 – 9:15: Brief Intro to Goals of the Session and Intro to OpenVXML (Vikram)

9:15 – 10:00: Step-by-step Tutorial in Designing a Simple Conversational Item in OpenVXML (led by Eugene, with the assistance of Vikram, Keelan and DSO)

10:00 – 10:55: Hands-on Development and Deployment of Custom Applications (All)

10:55 – 11:00: Concluding Remarks (Vikram)

ABSTRACT. Citizens in the 21st century must be able to engage in reading to learn from multiple sources in academic, professional, and personal life. Doing so requires specialized reading, critical thinking, and communicating practices. For the past 7 years, in the context of Project READI, (one of six projects funded by the IES Reading for Understanding Network initiative) I have been immersed in the texts and discourses of three disciplines – science, history, and literary analysis. In this project, we iteratively designed, implemented, studied, and revised materials and instructional practices and routines. Our work aimed to support adolescent students in building the requisite knowledge, strategies, and dispositions to engage in evidence-based argument from multiple information sources - and to do so in ways that respected the epistemology of each discipline, especially its goals and values. In this talk, I present the theoretical framework that has guided this work (Goldman, et al., 2016), and provide specific classroom-based cases of talk, texts, and talk about texts. These cases reflect the central role of discourse in learning and illustrate learning the discourse of a discipline. To conclude I consider important, unanswered questions for research in this area.

ABSTRACT. Conversation-Based Assessment (CBA) is a new approach to measurement which leverages automated natural language discourse as a means to adaptively elicit information from students. This approach has been tested within a sample of 632 middle school students and results indicate the conversational items allowed, on average, 25% of students to provide a more complete response, and thus improve their score on particular cognitive skills (e.g., science reasoning).

ABSTRACT. This study investigated whether the pedagogical agent formality impacted summary writing and impressions in AutoTutor. Participants (N=164) were randomly assigned into three conditions in which agents spoke formally, informally, or the mixed discourse. Results showed that participants achieved significant learning gains when they interacted with both agents who spoke formally or informally. A tendency showed that participants preferred the agent with the formal discourse rather than the informal discourse. Implications for instruction are discussed.

ABSTRACT. AutoTutor is a computer simulation of a human tutor that uses artificial intelligence to hold a conversation with students in natural language. In this paper we discuss some of the natural language processing techniques used in AutoTutor and successor systems to i) understand student input and ii) manage the dialogue as a human tutor might.

ABSTRACT. Over more than a decade, AutoTutor has been implemented in various ways across multiple content domains. In two experiments, we tested whether a serious game that incorporates AutoTutor would contribute to learning. In Experiment 1, it was found that the game increased both shallow and deep learning equally. Experiment 2 compared learning from different modules in the game which differ on pedagogy. Results suggest that direct instruction had the largest impact on learning.

ABSTRACT. Much work has been done towards computational modeling of collaborative processes in the past decade. However, while multiple successes of computationalization have been reported in the literature, we explore the extent to which evidence points to an intertwining between dimensions that calls into questions assessments that can be made based on the observation of codes as they are applied to collaborative discourse. Implications for analysis of rhetorical structure of collaborative discourse will be discussed.

ABSTRACT. This paper presents the current state in constructing a comprehensive, social-cognitive model of instructional communication. The central aim of the model is to integrate research regarding the dual function of communication in instruction: communicating content as well as negotiating the social relationships between learner and instructor. I will review research along a triangle of instructors, learners, and content, concentrating on their interrelations and especially the communication behaviors of instructors and learners.

ABSTRACT. This paper identifies the dialectical tension between a teacher’s rhetorical communication (clear instruction that promotes learning outcomes) and relational communication (appropriate and respectful interaction with students). Nowhere in the classroom is this juxtaposition more prominent than in the giving of instructional feedback, when skillful teachers employ face-attentive messages to render critical assessment more acceptable and accessible to students. During this symposium, this appropriateness-effectiveness dialectic will be explored with a view toward developing greater instructional competence.

ABSTRACT. Uncertainty terms (e.g., possible, some) are words whose use is sometimes motivated by face-work, a motivation that can create ambiguity in terms of how they are interpreted. I will discuss research examining how the interpersonal process of face-work, and variables that affect it (e.g., power), influence the interpretation and use of these words, as well as the consequences this effect has for the communication of uncertainty and the interpretation of the meaning of self-report items.

ABSTRACT. The authors examine whether incorporating paraphrasing can help automated-scoring features from previous work that rely only on verbatim quotation of source texts when scoring source-based writing. Straightforward incorporation of automatically-generated paraphrases yielded no improvements in scoring performance. A post-hoc corpus analysis of words used by the test-takers revealed that the register and specificity of the automatically-generated paraphrases are confounding factors and need to be controlled in future experiments.

ABSTRACT. Using data collected from 761 middle school students in the US, we examined test-takers’ composition strategies under timed assessment condition by using action sequences and variables extracted from keystrokes. Comparisons were made for essays of different score levels and submitted by the two gender groups, males and females. The findings of this study have implications for gaining deeper understanding of observed group differences and for designing interventions to close the achievement gaps among population groups.

ABSTRACT. In this study, we evaluate students’ composition processes in a timed-essay assessment context. Using a data set containing about 550 middle-school students, we develop threshold that characterizes burst optimal to an individual’s writing style, and further cluster students’ writing profiles into three groups. The resulting personalized burst length correlates with essay quality considerably greater than existing features. The three clusters of students differ significantly in their essay quality and writing processes.

ABSTRACT. We explored the extent that comprehension strategies (i.e., paraphrasing, bridging, elaboration) support coherence building. Computational tools were used to analyze think-alouds for strategies and levels of noun and verb cohesion between adjacent protocols. Bridging and elaboration scores were positively correlated with cohesion for both nouns (entities) and verbs (events/actions), but paraphrasing was correlated only with noun cohesion. The results offer insights into comprehension processes and the impact of reinstating entities and events while processing text.

ABSTRACT. This study focused on students’ essay writing processes using keystroke logging data from a writing task for middle-school students. We investigated the performance gaps of ELL with Former and Non-ELL students in various ways. Results showed that the lower performance of ELL students on both language and content aspects of their essay writing is largely explained by their background knowledge task. Subgroup comparisons on over 200 writing-process features revealed further information about ELL-students writing processes.

ABSTRACT. The current experiments examined the conditions that influence the reinstatement of antecedents. Experiment 1 demonstrated that readers made anaphoric inferences even when there was no direct anaphor and no demand sentence. Experiments 2 and 3 demonstrated that processing difficulty associated with anomalous anaphors is influenced by both the relatedness of the anaphor and the extent to which the reader waits for activation, integration, and validation processes to operate before moving on in the text.

ABSTRACT. In two studies, we compared the performance of sighted and visually-impaired children with sighted children on different inference tasks. The results indicate that VI children show a deficit in making global coherence inferences compared to both age-matched and comprehension-age matched sighted children, whereas their ability to remember literal information and to make local cohesion inferences is not impaired. Furthermore, VI children had particular problems making inferences about spatial information (but not emotional or temporal information).

ABSTRACT. The purpose of this study is to analyze the role of working memory and sustained attention in the comprehension of narratives in 5 and 6-year-old children. Results showed that age has an effect on the comprehension of general information and the ability to generate inferences, but this effect is mediated, in part, by a child's ability to sustain attention on the narration and to temporarily store the information received while listening to it.

ABSTRACT. Inference making is fundamental to the construction of a coherent mental model of a text. We examined inference making from preschool (N=420) to third grade to determine how vocabulary and working memory, two key predictors of inference making, influence its development. Vocabulary was a critical determinant of inferencing both within and across time; we did not find the same relationship with working memory. The results highlight the critical role of knowledge in successful text comprehension.

ABSTRACT. Do readers simulate the cognitive processes of story characters and remember what they remember? To investigate this question, participants read narratives that described a story character who was motivated to remember a list of words. We then measured readers’ memory for the same word lists. Across two experiments we demonstrate that story characters’ memory processes influence the memory of readers.

ABSTRACT. This study compared participants’ memory for sources asserting distinct compared to congruent information, as well as the information associated with each source and confidence in those memories. Compared to semantically distinct texts, when reading semantically congruent texts, readers showed better claim and evidence memory but more source confusions. Readers rated confidence higher on congruent claims and sources compared to distinct. Results suggest that congruent texts promote content integration at the expense of source attention.

ABSTRACT. Using a spoken narrative task of the Pear Story film (n=20), we investigated the production of Vietnamese referential forms (i.e. null pronouns, overt pronouns, NPs) in both subject and object positions. While the division between null and overt pronouns in Vietnamese is not as clear as standardly assumed, we found evidence for subjecthood effect and more importantly, parallelism effect, highlighting the importance of examining the referent’s grammatical roles in both the current and preceding utterances.

ABSTRACT. An EFA of a verbal analog of the Standardized Wisconsin Card Sorting Test and the Swanson Operation Span Task in conjunction with a study of expository text recall, revealed a moderate correlation of .23 between the two measures. None of the subscores of the VWCST correlated with the Swanson OST and none of the subscores of the Swanson OST correlated with the VWCST, indicating that these two Tests predominantly measure different aspects of executive processing.

ABSTRACT. A new operational measure for assessing cognitive flexibility, called the Verbal Wisconsin Card Sorting Task(VWCST)is proposed. The effects of Verbal Working Memory Span(VWMS)and VWCST on recall of scientific expository texts with varying numbers of topic shifts, demonstrated that participants low in VWCST and moderate in VWMS had more difficulty than participants high in VWCST and high in VWMS in recalling propositions from texts that had more topic shifts.

ABSTRACT. Sarcasm appreciation requires perspective taking. Children, adolescents, and adults were presented with sarcasm in three parties present conditions: private evaluation, public evaluation, and gossip. Participants interpreted speaker attitude and humor from the perspective of addressee and bystander. Children showed no influence of interpretive perspective or parties present. Adolescents and adults shifted interpretations but differed in perceptions of gossip. The ability to modulate appreciation of sarcasm by interpretive perspective and parties present develops in adulthood.

ABSTRACT. While the protagonist comprises an essential element of readers’ representations, readers do not read from the perspective of the protagonist. However, under certain conditions readers will comprehend from the protagonist’s perspective. The first three experiments demonstrated that readers can adopt the perspective of the protagonist when explicitly instructed and when reading from the first person point of view. Experiment 4 established that taking the protagonist’s perspective is due to strategic processing, not simply increased attention.

ABSTRACT. Traditional cloze tests may not measure comprehension processes beyond the single sentence level (Shanahan et al., 1982). In contrast, a deep cloze test has gaps that are strategically chosen to assess specific aspects of comprehension at discourse level. A study of 83 adult students found that a new deep cloze test contributed unique variance to reading comprehension measured with a conventional test even after control for sentence-level cloze and vocabulary.

ABSTRACT. We investigated relations between undergraduates’ moral convictions, plausibility, attitudes, and knowledge about climate change, as well as differences in the variables based on moral stance. Moral convictions, plausibility, and attitudes were significantly related, and also, significant predictors of post reading knowledge. We also found that only attitudes changed after reading a refutation text. These results suggest the importance of cognitive evaluations in learning about important and controversial topics, such as climate change.

ABSTRACT. Aspect is fundamental to expressing time and sequence of events in narratives. In second language acquisition literature little is known about how aspect is comprehended. This study manipulates aspect in narratives and compares comprehension of Spanish speakers to English and Arabic speakers. English and Spanish have morphosyntactic aspect, Arabic does not. Predicted results include no effect in Spanish speakers’ comprehension of manipulations compared to English or Arabic speakers. Data will be presented at the conference.

ABSTRACT. Is there something unique about the way in which U.S. Presidential candidates speak that can potentially influence a voter’s behavior? Significant linguistic differences were discovered between political parties. Additionally, linguistic differences between D.J. Trump and H.R Clinton are discussed. These findings suggest that candidates need to be cognizant of what they say, how it is being said, and to whom it is being said.

ABSTRACT. We employed a mixed-methods approach to investigate arguments for and against vaccination. Arguments were extracted from popular web sources that discussed vaccination specifically from mothers’ point of view and were analyzed along both epistemic and affective dimensions. Quantitative and qualitative findings were integrated to provide a robust interpretation of the arguments collected and provide insight into how the side of the debate for which the argument is made may influence what is said and why.

ABSTRACT. Reader/text matching algorithms have been proposed as a way to help teachers and learners select reading materials that are challenging, yet not so challenging as to cause frustration or reduce motivation. Recent analyses have suggested that many existing algorithms rely on assumptions and approximations that may contribute to a narrowing of the targeted reading construct. An alternative IRT-based matching algorithm is proposed, and evidence of improved construct representation is presented.

ABSTRACT. Natural language processing (NLP) and machine learning (ML) were used to predict human ratings of literary text comprehension. An ensemble classifier that used unigrams (single content words), elaborative (new) n-grams, and linguistic features in combination was the most accurate in identifying idea units as paraphrase, text-based inference, or interpretive inference (accuracy 0.61 to 0.95, F=0.23 to 0.79). The findings indicate that combining NLP and ML offers an effective means of automating literary text comprehension assessment.

ABSTRACT. Using a corpus of 4575 questions, we developed an NLP algorithm assessing question quality to inform feedback on questions generated by students within iSTART (an intelligent tutoring system that teaches reading strategies). The questions were coded using a four-level taxonomy. NLP indices were calculated for each question and machine learning was used to predict question quality. NLP indices related to lexical sophistication modestly predicted question type. Accuracies improved when predicting two levels (shallow versus deep).

ABSTRACT. This study explores constructing dialogues in English conversations on the varieties of English,conducted among/between American students (AS) and Korean students (KS) in the U.S. with reference to AS’s and KS’s language awareness and attitudes toward World Englishes. It confirms that AS and KS utilize various types of constructed dialogues and discourse strategies in expressing language awareness and attitudes toward World Englishes, in order to achieve various interactional goals.

ABSTRACT. This study develops a second language (L2) writing proficiency model using 480 test-takers’ responses to source-based and independent writing tasks via structural equation modeling. Findings indicated that in a latent model, 71.78% of the variance in L2 Writing Proficiency was explained by L2 Linguistic Ability (51.98%) and L2 Discourse Ability (19.80%). It was also found that the latent model was generalizable across writing prompts (with the exception of lexical features), gender, and learning contexts.

ABSTRACT. The authors examined how limited cognitive resources affect inferential processing for unheralded pronoun resolution in a second language (L2). Participants read stories containing either an ordinary pronoun or an unheralded pronoun. In an intermediate L2, the unheralded pronouns were processed slower than the ordinary pronouns and the associated information to the referent appeared to be activated through active memory search. In contrast, advanced L2 readers demonstrated a similar processing pattern to that of L1 readers.

ABSTRACT. The purpose of this study is to replicate and extend the seductive effect of decorative pictures in expository text comprehension to an e-learning environment. One hundred and twenty four undergraduate student completed WM tests, and an e-learning course that presented two texts (with and without images), followed by comprehension questions. Scores on comprehension were significantly lower for the texts with decorative images, and varied positive and linearly with WM scores.

ABSTRACT. We analyzed keystroke logs as a way of characterizing the processes in essay composition. Low-level timing data were modeled, which were thought to reflect processes associated with keyboarding skills and composition fluency. Heavy-tailed probability distributions were found to fit to individual students’ data reasonably well; estimated parameters were found to be more robust than total writing time or scores across prompts for the same writing purpose; and they were associated with human essay scores.

ABSTRACT. We analyzed how skilled and less-skilled comprehenders engaged in feedback processing in task-oriented reading. Students read two texts, and answered multiple-choice questions with the text available on the computer, while also thinking-aloud in one of the texts. We found individual differences only on the percentage of verbalizations after question answering failure vs. success. The focus of attention when processing feedback, as well as metacognitive and cognitive processes were very similar between both groups of students.

ABSTRACT. This study examined participants’ engagement in inquiry with multiple texts for purposes of creating an explanatory model. Given the challenges commonly experienced, participants in a second condition were provided with a graphic organizer designed to support multiple aspects of this process. Results suggest that participants in the graphic organizer condition engaged in significantly more elaborative processing of relevant information from the texts and performed significantly better on measures of model quality and learning.

ABSTRACT. People often utilize the inaccurate information they read, even when they should know better. We investigated whether exposure to far-fetched alien conspiracies, and beliefs about their viability, might influence people’s use of inaccurate information. Participants were generally influenced by the falsehoods they read regardless of whether they previously watched a conspiracy or control video. But people’s beliefs about conspiracies, and their predilections towards careful reasoning, helped distinguish the likelihood of their being influenced by inaccuracies.

ABSTRACT. This research studies the automated scoring and feedback in supporting students’ construction of written scientific arguments in an online earth science curriculum module. The constructed responses were evaluated by an automated scoring engine, then scores and automated feedback were provided in real-time to students to guide their revisions of the answers. We analyse the log data of student activities and score details to study students’ response to and impact of the automated scores and feedback.

ABSTRACT. Using an experimental design, this study investigates the causal effects of a teacher PD program that intends to improve middle school student writing. Both final scores and writing-process indicators extracted from keystroke logs were examined as outcome variables using a school random effect model. The study found null effect on essay scores but significant effect on a writing process indicator, local and word level editing, although the effect was significant only for high SES schools.

ABSTRACT. This research examined how certain textual features vary across an entire book. Excerpts from Harry Potter were processed through TextEvaluatorTM to examine changes on several metrics. The results indicate substantial variation in overall text complexity, ranging from first to eleventh grade; in subdimensions of complexity (e.g., syntactic complexity, level of argumentation); and from passage to passage sequentially. The implications for classroom use of text analytic tools with full length books are discussed.

ABSTRACT. We present the results of three norming experiments designed to produce metaphors and target words for use in studies of semantic access. In Experiment 1, we collected familiarity ratings for 264 metaphors. In Experiment 2, we developed target words related to the figurative or literal meaning of 132 of the metaphors. In Experiment 3, we verified that the target words are related to the matched metaphors and are unrelated to matched, neutral literal sentences.

ABSTRACT. Participants read texts that primed an inference associated with a positive (Experiment 1) or negative (Experiment 2) emotion. Using a divided visual-field paradigm, participants performed a lexical decision task for related target words presented to either the left visual field-right hemisphere or right visual field-left hemisphere. Results showed significant priming in both hemispheres only for negative emotion inferences. This pattern suggests that readers are faster to infer negative emotions from a text than positive emotions.

ABSTRACT. Adults with and without autism spectrum disorder (ASD) show empathetic responses toward similar others. However, it remains unclear whether they show empathetic responses and helping motivation toward similar others. Twenty-two ASD adults and 20 typically developing (TD) adults rated empathy and helping motivation after reading 24 stories. The results showed that TD adults had empathetic responses and helping motivation toward TD characters. However, ASD adults showed empathy to ASD characters, but not helping motivation.

ABSTRACT. This study examined discourse data associated with co-construction of knowledge that individuals displayed while engaging in a collaborative simulation-based task, and how those co-construction processes related to performance outcomes. We further explored how individual personality traits may relate to co-construction processes. Significant relationships between personality and co-construction processes did not emerge. However, performance outcomes were impacted by co-construction; individuals who displayed collaborative behaviors performed better relative to those who engaged in non-collaborative behaviors.

ABSTRACT. This study investigated the effect of reflective-writing and knowledge of research-methods on students' beliefs about knowledge and learning, epistemological beliefs (EB). Students -- from research-methods and cognitive-processes class -- took the EB-survey three times (first-day/baseline, pre, and post reflective-writing) during a semester. Only research methods students showed a change in their beliefs after a reflective writing task. This study identified the importance of students’ knowledge about research methods in potentially shaping their beliefs about knowledge.

ABSTRACT. Reading times and humor ratings for 80 headlines (40 satirical, 40 non-satirical) were gathered from 76 participants. Results demonstrated no significant reading time differences between satirical and non-satirical headlines. However, results reported that a higher age and higher perceptions of humor caused significantly slower satire processing, while higher levels of prior knowledge resulted in significantly higher perceptions of satirical humor. These results further emphasize the importance that age and prior knowledge have on satire comprehension.

ABSTRACT. Virtual internships are online simulations of professional. Prior work used classifiers trained on participant data to automatically assess notebook entries from these environments. However, when teachers create new internships using available authoring tools, no such data exists. We evaluate a method for generating classifiers using specifications provided by teachers during their authoring process instead of participant data. Our method produces some classifiers that perform well on unseen data, but requires further refinement.

ABSTRACT. In this study, we examined the conceptual consequences of conversational audience design. In a Wizard-of-Oz paradigm, participants described novel images for a computer dialogue application or human partner. Subsequently, they independently sorted the images into discrete groups. When talking to the computer partner, participants’ descriptions focused mostly on geometric features. Moreover, participants’ post-dialogue sorts were more shape-based following human-computer interaction. Attending to objects for communicative purposes affects how speakers construe objects for themselves.