The Second-User Test

In my work with G7–G12 students at Force Academy, a research initiative in Chongqing, I sometimes take a project that appears to be finished and give another student responsibility for using, checking, or modifying it. The handoff is a test of understanding: Can the work still make sense when its creator is no longer there to translate it? The original author knows what every variable means, why a particular decision was made, which tool was used, and which parts can safely be changed. The second student has only what the project makes visible. Generative AI makes the handoff more revealing because it lowers the cost of producing a convincing first version. Students can generate code, analysis, prose, visualizations, and prototypes far more quickly than they could a few years ago. In a classroom, however, a project is useful only insofar as it shows what the student understands and can defend. AI can supply missing code, smooth over incomplete reasoning, or turn a rough idea into fluent prose, so polish is no longer strong evidence of understanding by itself. A concrete example came from a student paper on AI and labor economics, Verification Responsibility in AI-Applicable Occupations. The paper asked what remains economically important when AI makes candidate outputs cheap but a person still has to check those outputs, decide whether they are safe to use, and remain responsible for accepting them. The lead student developed the conceptual model and most of the analytical workflow. A second student was given a narrower job: rerun the main analyses independently, record the versions of the source data, document how occupations in one dataset were linked to occupations in another, and compare the reported tables and figures with the underlying outputs. What looked like procedural verification quickly became a substantive review of the study. The first problem was reproducibility. The analysis ran on the original machine, but too much of the workflow depended on choices that lived in one student’s memory. To reproduce it elsewhere, the team had to record the data transformations, write down the rules for linking occupational datasets, and keep intermediate results so another person could see how the final numbers had been produced rather than simply rerunning a script and trusting the output. The more consequential problem was in the measurement design. An early version of the study used some of the same autonomy and leadership survey items in two different measures: verification responsibility and adaptive capacity. The software had calculated the relationship between those measures correctly, but part of the apparent relationship could have been created by the fact that the two measures shared some of the same inputs. In other words, the code was doing what it had been asked to do; the question was whether the quantities being compared had been defined cleanly enough. The team rebuilt the measures so they no longer shared survey items, removed ambiguous items, and revised the workflow so source versions, cross-dataset links, exclusions, intermediate files, and reported results could be traced. Later versions added alternative specifications and robustness checks. What began as “please rerun this” ended up forcing a redesign of the study. The handoff changed how I read student work. A second user does not merely test whether the files have been packaged neatly; they ask the awkward questions the creator may have stopped noticing. Why is a particular variable there? Which conclusion depends on an assumption that was never made explicit? In this project, those questions reached back into the measurement design. The educational value of the handoff comes from the fact that schoolwork and production software have different ends. Professional developers learn a great deal after software leaves their hands because they have to deliver it, explain it, support it, maintain it, and respond when someone uses it in an unexpected way. Student projects usually disappear when a course ends, and that is fine; the artifact is not the goal. What matters is importing some of the learning that comes from delivery and maintenance: explaining decisions, supporting someone else’s use, and revising the work when private context is no longer available. At Force Academy, the exact task depends on the project. For a computational model, another student may change a parameter or replace one rule and then ask what should remain stable. For an AI-supported workflow, the job may be to move it to a different model or environment. For a data analysis, the second student may try to reconstruct one result from the recorded source files and transformations. When that transfer fails, I want the original student to decide what the other person actually needed to know and what should be changed in the project, rather than simply standing beside them and explaining it. I use the word “handles” for the visible points of entry that make that possible. A handle gives another person something concrete to inspect or change without first receiving a private briefing. It might be a clearly named parameter, an explicit assumption, a test that records an earlier failure, a replaceable component, a retained intermediate dataset, or a prompt whose important context is written down instead of buried in a private chat history. Handles also make disagreement more productive. An exposed assumption can be challenged; a component with a clear boundary can be modified or replaced without rebuilding the whole project; and a numerical result linked to its source data and transformations can be checked rather than accepted on authority. Outside school, AI systems expose the same weakness. A prompt, workflow, skill, or agent configuration may look portable because it can be copied, while the conditions that make it work remain elsewhere: tool access, file structure, model behavior, private memory, naming conventions, or small decisions the creator never needed to document. Moving the artifact often reveals how much of the working system was never inside the artifact in the first place. A first successful run is therefore weak evidence. It tells me that a particular combination of person, model, tools, data, and context produced an output once. A stronger test is whether another person can reconstruct what matters, distinguish assumptions from implementation details, and make a change without destroying the logic of the work. Tim O’Reilly’s idea of an architecture of participation gives me the broader frame. A technical artifact becomes more useful when other people can enter it without negotiating privately with its creator. In software, that usually means interfaces, conventions, documentation, modularity, and replaceable components. In data analysis, the same principle shows up as recorded source versions, visible transformations, retained intermediate results, explicit assumptions, and enough analytical logic for someone else to rerun or challenge the work. For education, the milestone I care about is whether the student understands the work well enough to make some of that understanding available to someone else. A second student who can use, question, and extend the project gives me evidence that a polished first run cannot. AI expands what one person can build, including what a teenager can attempt without a large technical team. The expanded scope is a real gain, but it also makes successful generation easier to confuse with understanding. When I want to know what a student really understands, I now learn more from the first serious handoff than from the first successful run.