Explore

Challenges & Risks

Adopting AI in education isn't without significant challenges. Academic integrity, data privacy, algorithmic bias, and the digital divide require thoughtful, proactive strategies.

Academic integrity is perhaps the most immediate challenge. With generative AI capable of producing essays, solving problems, and even generating code, traditional assessment methods face an existential crisis. Institutions must evolve beyond detection-based approaches toward assessment redesign that emphasizes process, critical thinking, and authentic demonstration of learning.

Data privacy and security concerns are paramount when dealing with student data, especially for minors. AI systems require vast amounts of data to function effectively, creating tension between personalization and privacy. Regulations like FERPA, COPPA, and GDPR provide frameworks, but implementation in the rapidly evolving AI landscape remains complex.

Algorithmic bias poses serious equity concerns. AI systems trained on biased data can perpetuate and amplify existing inequalities in education — from biased grading algorithms to recommendation systems that reinforce stereotypes. Additionally, the digital divide means AI's benefits may disproportionately flow to already-privileged communities, widening rather than closing achievement gaps.

The challenge nobody lists: effort avoidance

Integrity, privacy and bias are the three risks every list names, and they belong on it. The one that gets left off is quieter and, we think, larger: a technology that removes difficulty will be used to remove difficulty, including the difficulty that was doing the work. Nothing is violated. No policy is broken. The learner simply arrives at the end of the task without having been changed by it.

This is not a discipline problem and it does not have a discipline solution. It is a design problem: if a task can be completed without understanding, it will be, and it was probably a weak task before AI existed. See No Shortcuts to Mastery for why the repetitions cannot be skipped, and Intelligence Is a Muscle for what the two outcomes look like side by side.

Integrity is an assessment-design question

Detection tools are unreliable, and their failures are not evenly distributed — writing by second-language learners is flagged disproportionately, which converts an integrity policy into an equity problem. The durable response is to change what is being asked, not to escalate surveillance: work that is defended in person, drafted visibly, built on the learner's own material, or assessed on the process rather than the artefact.

Our Assessment & Evaluation work is largely this, and the Ethics Simulator lets a staff team rehearse the specific decisions — a suspected case, a parent challenge, a disputed flag — before one lands on someone's desk at 4pm on a Friday.

Privacy: the questions to ask before signing

Student data, particularly minors' data, carries obligations that most classroom-tool procurement was never designed to check. Three questions settle most of it: does anything entered here train a model; where is it stored and under whose jurisdiction; and can it be deleted on request, in fact and not just in policy. A vendor who cannot answer all three in writing has answered them.

Where a legal framework is involved, the obligations are usually more specific than general guidance suggests. Our Dominican Republic educator training on the Penal Code is the fully worked example: the actual articles, what they require of a school, and what they do not — built with a public audit report of its own review process.

Bias, and the drift you cannot see

A model reproduces the patterns in what it was trained on, and in education those patterns decide who gets flagged as at risk, whose writing reads as proficient, and whose name is spelled correctly. A related failure is subtler and affects everyone equally: models produce confident, well-formatted, entirely fabricated specifics — citations, statistics, named studies — and they survive review because they look exactly like the real thing.

We have been caught by this ourselves and published the account rather than quietly fixing it: The Infographic I Didn't Publish. The working rule on this site is that a claim is cited, or it is stated directionally, or it does not run.

And a challenge worth putting in proportion

Writing, the printing press, the pocket calculator, the ballpoint pen, television, the internet, Wikipedia and the smartphone were each, at the time, going to destroy learning. The pattern is documented in The Recurring Panic. Naming it is not a way of dismissing the risks above — it is a way of keeping the conversation on the risks that are actually specific to this technology, which are quite enough to be getting on with.