Spotting AI-generated code in a student’s submission isn’t a one-tool problem. After running Codequiry’s combined plagiarism and AI detection across three semesters of intermediate Java assignments, our department landed on a repeatable, three-tier scanning model that catches far more than any single detector ever could. The core insight: AI detection, similarity analysis, and instructor-led evidence review each notice different things, and only together do they form a defensible signal. Here’s the framework we use, the thresholds we settled on, and the false-positive patterns we learned to expect.
Why Traditional Plagiarism Detectors Miss AI-Generated Code
Standard similarity checkers—MOSS, JPlag, and the token‑based engines inside most LMS integrations—assume that cheating leaves a trail of matched code between two or more human authors. When a student prompts an LLM instead, the submission is often unique within the class corpus. There is no peer match. A submission that generates clean, idiosyncratic-looking code from a ChatGPT prompt will sail straight through a MOSS scan with a 0% match score. We confirmed this in a controlled experiment: 18 of 20 ChatGPT‑generated Python functions for a common CS1 list‑manipulation problem returned no similarity flag above 8% against a 200‑submission corpus. The only two that triggered a flag did so because the prompt had been copied verbatim by another student using the same AI template—a collusion‑plus‑AI edge case we will revisit in Tier 2.
AI-generated code also tends to avoid the tell‑tale copy‑paste artifacts that web‑scraping detectors look for. A submission cloned from Stack Overflow will contain the exact same comment “// function to reverse a linked list” that appears on a public post; a GPT‑4o‑produced solution will almost certainly not. The scanning approach must therefore separate the question “is this from a known online source?” from the question “does this display the statistical fingerprints of an LLM?”
The Three-Tier Scanning Model
The framework we teach to new teaching assistants now runs every submission through three sequential gates. At each gate, a score or a set of flags determines whether the submission advances to instructor review or is cleared as likely human‑authored.
Tier 1 — AI-Generated Code Detection
This is the first filter because it is the most sensitive to the patterns unique to LLM‑written source. Modern detectors like Codequiry’s AI code engine, GPTZero, and Turnitin’s authorship detector look beyond surface text and analyze structural signals. The metrics that matter in source code are different from those in prose:
- Comment density and placement. LLMs frequently insert line‑by‑line explanatory comments that real students in a hurry rarely write. A 2024 manual review of 300 CS1 assignments at our institution found that submissions flagged by Codequiry’s AI scanner contained an average of 1.8 lines of comment per 10 lines of code, compared with 0.3 in unflagged submissions.
- Variable naming consistency. Human programmers drift between naming conventions (camelCase, snake_case, single‑letter loop indices). LLM‑produced code is unnaturally consistent within a single file, often sticking to a single style with minimal deviation.
- Control‑flow predictability. Language models favor certain idiomatic patterns—list comprehensions in Python, stream pipelines in Java—even when a simpler loop would be more natural for a novice. Detectors measure the deviation from the “idiomatic density” seen in typical student work for a given course level.
- Repetition and template‑lock. When students tweak AI output, they tend to leave large blocks untouched and modify only the obvious variable names or output strings. A scanner trained on entropy change between adjacent lines can flag this “frozen core” pattern.
“AI detectors for code are not judging whether the code is good enough to be human. They are measuring how much it resembles the distribution of code that thousands of real students in that course wrote in prior semesters. That distributional check is the statistical backbone.” —Dr. Elena Torres, CS Education Researcher, University of Texas at Austin
Tools vary substantially in how they present this signal. Codequiry provides a per‑submission AI likelihood score (0–100) together with a sentence‑level highlight map, which is critical for Tier 3 evidence gathering. GPTZero offers similar scoring but with a coarser breakdown. Turnitin’s AI indicator gives a percentage of AI‑generated text but can miss code‑specific patterns embedded inside mixed prose/code submissions. For courses where students submit .java or .py files directly, a code‑native engine is essential. For courses that accept PDF reports containing embedded snippets, a hybrid tool that scans both prose and code blocks reduces the gap.
Threshold tuning is where the human judgment enters the scanning pipeline. After running a semester’s worth of ground‑truth data (72 known‑AI and 140 known‑human Python submissions collected with informed consent), we settled on a two‑band system:
| Codequiry AI Score | Action |
|---|---|
| 0–40 | Cleared; no further AI review |
| 41–70 | Flagged for Tier 2 similarity cross‑check; instructor reviews if similarity is also elevated |
| 71–100 | Routed directly to Tier 3 manual review regardless of similarity |
These bands keep the false‑positive rate under 2% in our context; your own calibration will shift depending on course level, language, and assignment design. The crucial practice is to never use a single numeric threshold as a verdict—the score is a filter, not a final decision.
Tier 2 — Code Similarity and Collusion Analysis
Once an AI‑detection flag is raised—or even on all submissions, if you have the TA bandwidth—the submission runs through a similarity engine that checks two distinct sources: other submissions in the same course and a database of known AI‑generated template code that the platform has accumulated.
The first check catches the scenario where multiple students submit independently‑generated AI code that the LLM happened to render almost identically. This occurs more often than you might expect. In our experiment, the prompt “Write a Java program that reads a CSV file and calculates the average of column 3” produced code with 78% token‑level overlap across five separate ChatGPT‑4o queries when temperature was set to 0.1. When students use the same popular prompt (or copy it from a Discord server), the output varies far less than you would guess. A similarity check against the class cohort surfaces these clusters even when the AI detector score is moderate.
The second check—against a database of canonical AI‑generated outputs—helps identify submissions that are not similar to each other but are abnormally similar to known LLM responses. Codequiry maintains a growing corpus of millions of AI‑generated code samples across 12 languages, and a submission that shows >60% similarity to one of these canonical samples is a strong corroborating signal.
This tier also serves as a fairness mechanism. If a student’s code flags for AI detection but shows zero peer similarity and zero AI‑template similarity, we downgrade the suspicion level. It might be a false positive, or a student whose natural style inadvertently matches statistical signatures. The combination of orthogonal signals—AI likelihood score and similarity evidence—produces a far more defensible case than either alone.
Tier 3 — Instructor Review and Evidence Packaging
The scanning framework pushes the hardest cases to a human reviewer who sees:
- The AI likelihood score and the highlighted regions that contributed most to the score.
- Any matched similar submissions or AI‑template matches, with side‑by‑side diffs.
- The submission’s full version history if the course uses a version‑control system. A student who writes genuine code tends to commit small incremental changes over hours or days. A submission that appears fully‑formed in a single commit, with no intermediate debugging prints or mistakes, adds contextual doubt.
- Metadata such as time‑on‑task from the LMS, the number of compile attempts, and the student’s performance trend across prior assignments.
The goal is not to generate a guilty verdict—it is to prepare a set of observations that can be shown to the student in a non‑accusatory conversation. We phrase it as: “Your submission contains patterns that match both AI‑generated code and a template seen in a public repository. Can you walk me through how you approached the assignment?” Many students confess at this point; the few who do not are asked to do a live coding exercise of similar scope. The scanning data provides the factual basis for the conversation without preempting it.
Selecting the Right Detection Engine for Your Stack
Not all AI code scanners are built alike, and the best choice depends heavily on the programming languages in your curriculum and your institutional integration requirements. Here is the decision matrix we used when evaluating three leading platforms.
Language coverage. Codequiry’s AI detector covers Python, Java, C++, JavaScript, C#, and TypeScript with tuned models for each—critical for a department that runs intro Python, Java in data structures, and C++ in systems. Turnitin’s code‑aware detection works best on Python and JavaScript, with limited support for compiled languages. GPTZero is stronger on prose than code, missing many code‑specific signals.
Integration depth. Codequiry’s API allowed us to embed the scanner directly into our auto‑grader pipeline so that every submission on Gradescope receives an AI score column in the gradebook without any extra upload step. Turnitin integrates natively with many LMSs, but the AI indicator appears inside the general report without code‑specific highlights. If your workflow requires a separate upload step, expect TA compliance to drop 30–40% after the first two weeks.
Evidence quality. For an academic integrity board, a simple “76% AI” number is rarely enough. We needed line‑level highlight maps showing which code blocks triggered the signal, and cohort similarity comparisons. Codequiry’s summary report includes all three tiers—AI score, similarity matches, and web‑source checks—on a single page, which reduced our board review time from 25 minutes per case to roughly 9.
False‑positive transparency. A detector that never tells you its false‑positive rate is impossible to calibrate. We required a vendor that publishes validation results and allows administrators to adjust sensitivity per assignment. Codequiry’s institution dashboard lets you set course‑level thresholds; GPTZero exposes only a single global sensitivity slider. In a senior‑level graphics course that explicitly teaches design patterns, we raised the threshold to account for the fact that clean, pattern‑heavy code looks AI‑like even when it is student‑authored.
Integrating the Framework into a Semester Workflow Without Burning Out TAs
A scanning framework can easily collapse under its own weight if it adds more than about 90 seconds of overhead per submission for a 200‑student course. Our implementation uses automation to keep the load manageable:
- Automatic scan on submission. The auto‑grader script calls Codequiry’s API immediately after the student uploads. By the time the TA opens the grading queue, the AI Score column is already populated.
- Scheduled Tier 2 batch check. After the assignment deadline, a nightly cron job runs peer‑similarity and AI‑template similarity on the full cohort. The job produces a sorted list of flag clusters, with the most suspicious submissions at the top—the TA never has to scan all 200 one‑by‑one.
- Canned evidence reports. When a TA marks a submission for instructor review, the platform generates a single‑page PDF evidence packet that can be emailed directly to the student or attached to an academic integrity case. No assembly of screenshots or diffs required.
This pipeline processed a spring 2024 data‑structures section of 214 students with a total TA overhead of just under 2 hours per assignment—significantly less than the 5 to 6 hours we previously spent chasing random tips from graders who “thought a submission looked weird.”
False Positives, Fairness, and the Limits of Statistical Detection
No AI‑generated code detector has a zero false‑positive rate. The very nature of probabilistic models means that a small fraction of genuinely human submissions will score in the high‑AI range, especially under these conditions:
- Students who follow strict style guides. A student who consistently uses Google’s Java style guide will produce code that looks more uniform and LLM‑like. We saw this with two students who were prior industry programmers returning for a master’s degree; their code was flagged at 75 and 82, yet a one‑on‑one walkthrough confirmed original authorship.
- Auto‑generated boilerplate. IDEs that insert getters, setters, and toString() methods create blocks of code that are indistinguishable from AI‑generated template code. Our solution: strip IDE‑generated boilerplate before scanning by running a pre‑processing step that removes methods annotated with @Generated or those that match a known IDE‑template fingerprint.
- Short assignments. On submissions under 40 lines of logic, the statistical signal weakens. We assign a lower weight to AI scores on very short assignments and rely more on Tiers 2 and 3.
To mitigate fairness concerns, we make the scanning policy transparent in the syllabus, stating: “Your code may be screened by statistical detection tools that recognize patterns common in LLM‑generated output. A high detection score is not a finding of misconduct; it triggers a human review and, when warranted, a conversation with you.” Most students accept this in the same way they accept MOSS scanning for similarity—it is part of the ecosystem.
Where the Framework Breaks Down—and How We Are Extending It
The current three‑tier approach works well for fully AI‑generated submissions and for lightly‑edited AI code. It struggles with heavily refactored output where a student has renamed every variable, swapped control structures, and injected deliberate bugs that they later “fix” in a second commit to create a plausible version history. This adversary‑aware behavior is still rare—we saw it in only about 2% of flagged submissions—but it will grow.
Our next phase adds behavioral biometrics: keystroke‑log data from our in‑browser coding environment (we use a Code‑in‑Place‑style interface in exams) and time‑series analysis of edit bursts. Early results from a 2025 pilot show that AI‑assisted editing produces characteristic gap‑then‑burst commit patterns that differ starkly from the continuous typing of a student working through the problem. That data layer, when combined with the existing scanning tiers, raises the bar considerably.
Frequently Asked Questions
How accurate are AI code detectors for student assignments?
In controlled tests on Python and Java assignments at the CS1–CS2 level, tuned detectors achieve 92–95% accuracy with false‑positive rates between 1 and 3%. Accuracy drops on shorter submissions, heavily templated code, and advanced coursework where idiomatic patterns are expected. No detector should be used as a sole source of truth.
Can students evade AI detection by running the output through a paraphrasing tool?
Code paraphrasing tools that rename variables and reorder statements can lower a detector’s confidence score, but the structural bones—comment density, control‑flow idioms, and semantic redundancy—often persist enough to keep the score above 40. Combined with peer‑similarity checks, obfuscated AI code becomes easier to flag because it stands out as unusual relative to the class norm.
What is the best way to present AI evidence to a student?
Show, don’t accuse. Provide the student with a brief report that highlights the specific code regions contributing to the AI score, any matched similar submissions, and the context of their version history. Ask open‑ended questions about their process rather than stating “this was generated by ChatGPT.” A calm evidence‑forward conversation resolves the vast majority of cases without escalating to a formal hearing.
Does Codequiry’s AI detection work alongside its plagiarism checker?
Yes. The platform runs both engines on every submission and presents a unified report. The AI score, similarity matches, and web‑source hits appear in one interface, which is exactly the layered scanning structure the three‑tier model requires out of the box.