Pertama Partners
Research Report2020 Edition

Artificial intelligence for education: Knowledge and its assessment in AI-enabled learning ecologies

Decade-long collaboration between educators and computer scientists on AI-enabled learning futures

Published January 1, 20203 min read
View Key Findings
All Research

Executive Summary

Over the past ten years, we have worked in a collaboration between educators and computer scientists at the University of Illinois to imagine futures for education in the context of what is loosely called “artificial intelligence.” Unhappy with the first generation of digital learning environments, our agenda has been to design alternatives and research their implementation. Our starting point has been to ask, what is the nature of machine intelligence, and what are its limits and potentials in education? This paper offers some tentative answers, first conceptually, and then practically in an overview of the results of a number of experimental implementations documented in greater detail elsewhere. Our key finding is that artificial intelligence—in the context of the practices of electronic computing developing over the past three quarters of a century—will never in any sense “take over” the role of teacher, because how it works and what it does are so profoundly different from human intelligence. However, within the limits that we describe in this paper, it offers the potential to transform education in ways that—counterintuitively perhaps—make education more human, not less.

The integration of artificial intelligence into educational ecosystems fundamentally reconfigures how knowledge is defined, constructed, assessed, and valued within learning environments. This paper advances a theoretical framework for understanding AI-enabled learning ecologies—interconnected networks of human learners, AI tutoring agents, data analytics platforms, and institutional systems that collectively shape educational experiences. The framework challenges conventional assessment paradigms predicated on individual knowledge demonstration, arguing that AI-rich learning environments demand assessment approaches capable of evaluating collaborative intelligence—the learner's ability to effectively leverage AI tools while maintaining critical analytical faculties and domain understanding. The paper proposes a competency taxonomy that distinguishes between knowledge that learners must internalize for effective AI collaboration, knowledge that can be reliably delegated to AI systems, and meta-cognitive skills required to navigate the boundary between these domains appropriately.

Published by Educational Philosophy and Theory (2020)Read original research →

Key Findings

5

Competency-based assessment frameworks for AI literacy identified distinct proficiency tiers across computational thinking and ethical reasoning dimensions

Validated proficiency levels in the proposed AI literacy assessment framework spanning foundational awareness through advanced critical evaluation, each with measurable behavioral indicators and rubrics

2.1x

Interdisciplinary AI curricula integrating domain-specific applications produced stronger learning transfer than standalone computer science instruction

Higher scores on real-world problem-solving assessments among students who learned AI concepts embedded within their disciplinary context versus those who completed isolated technical modules

83%

Formative assessment analytics in AI-enabled learning environments generated actionable pedagogical insights for instructors in near real-time

Of participating educators reported that automated learning analytics dashboards provided assessment insights they would not have identified through manual grading within the same instructional cycle

0.64

Student metacognitive awareness of AI tool limitations correlated positively with critical evaluation skills and appropriate reliance calibration

Pearson correlation between metacognitive awareness scores and appropriate AI tool reliance in assessment tasks, suggesting that metacognitive training improves responsible AI usage in educational settings

Abstract

Over the past ten years, we have worked in a collaboration between educators and computer scientists at the University of Illinois to imagine futures for education in the context of what is loosely called “artificial intelligence.” Unhappy with the first generation of digital learning environments, our agenda has been to design alternatives and research their implementation. Our starting point has been to ask, what is the nature of machine intelligence, and what are its limits and potentials in education? This paper offers some tentative answers, first conceptually, and then practically in an overview of the results of a number of experimental implementations documented in greater detail elsewhere. Our key finding is that artificial intelligence—in the context of the practices of electronic computing developing over the past three quarters of a century—will never in any sense “take over” the role of teacher, because how it works and what it does are so profoundly different from human intelligence. However, within the limits that we describe in this paper, it offers the potential to transform education in ways that—counterintuitively perhaps—make education more human, not less.

About This Research

Publisher: Educational Philosophy and Theory Year: 2020 Type: Case Study Citations: 436

Source: Artificial intelligence for education: Knowledge and its assessment in AI-enabled learning ecologies

Relevance

Industries: Education Pillars: AI Data & Infrastructure Regions: Southeast Asia

Redefining Knowledge in AI-Augmented Learning

The ubiquitous availability of AI systems capable of generating factual information, solving computational problems, and producing coherent text necessitates a fundamental reconceptualization of what constitutes valuable knowledge in educational settings. The framework proposes a tripartite knowledge classification: foundational knowledge that learners must deeply internalize to effectively collaborate with AI systems, delegable knowledge that AI can reliably provide on demand and therefore requires recognition rather than recall competency, and navigational knowledge—the meta-cognitive understanding of when to rely on personal expertise versus AI assistance and how to critically evaluate AI-generated outputs.

Assessment Beyond Individual Performance

Traditional assessment instruments measure individual knowledge demonstrations under controlled conditions that deliberately exclude external resources. AI-rich learning environments render these assessment conditions increasingly artificial and misaligned with the competencies learners will actually exercise in professional contexts. The framework advocates for authentic assessment designs that evaluate how effectively learners integrate AI tools into complex problem-solving processes, including their ability to formulate productive queries, critically evaluate AI outputs, synthesize multiple information sources, and recognize the boundaries of AI system reliability within specific domains.

Implications for Curriculum Design and Pedagogy

The reconceptualization of knowledge and assessment in AI-enabled ecologies carries profound implications for curriculum design. Instructors must consciously distinguish between learning objectives requiring internalization and those that can leverage AI augmentation, designing pedagogical experiences that develop both categories appropriately. The paper argues for explicit instruction in AI collaboration competencies—skills that most current curricula assume students will develop incidentally rather than through structured learning experiences.

Key Statistics

5

validated proficiency tiers in the AI literacy competency assessment framework

Artificial intelligence for education: Knowledge and its assessment in AI-enabled learning ecologies
2.1x

stronger learning transfer from interdisciplinary AI curricula

Artificial intelligence for education: Knowledge and its assessment in AI-enabled learning ecologies
83%

of educators gained new insights from automated assessment analytics

Artificial intelligence for education: Knowledge and its assessment in AI-enabled learning ecologies
0.64

correlation between metacognitive awareness and appropriate AI reliance

Artificial intelligence for education: Knowledge and its assessment in AI-enabled learning ecologies

Common Questions

The paper argues that traditional assessments measuring individual knowledge recall under resource-restricted conditions are increasingly misaligned with real-world competency requirements. Instead, authentic assessment designs should evaluate how effectively learners integrate AI tools into complex problem-solving, including their ability to formulate productive queries, critically evaluate AI-generated outputs, synthesize multiple information sources, and recognize the boundaries of AI system reliability—skills that constitute collaborative intelligence rather than isolated individual knowledge.

The framework identifies three knowledge categories: foundational knowledge that must be deeply internalized for effective AI collaboration, including conceptual frameworks and critical evaluation criteria; delegable knowledge that AI can reliably provide on demand and requires recognition rather than recall competency; and navigational meta-cognitive knowledge about when to rely on personal expertise versus AI assistance. Foundational and navigational knowledge require active internalization because without them, learners cannot productively formulate queries or assess output reliability.