A doctoral student asks an AI assistant for R code to merge survey responses with exam scores and run a regression. The code runs cleanly and the output table looks great. Three weeks later she discovers the merge duplicated every student with two exam records: the sample grew from 412 to 538 and every standard error shrank. No warning appeared. Code that runs and code that is correct are different things.
Generative AI assistants are powerful coding tools, especially for researchers without programming backgrounds. But they shift the risk from “I can't write code” to “I don't know what this code is doing”.
What is safe and what is risky
| Task | Risk | Why |
|---|---|---|
| Explaining an error message | Low | You can check immediately by rerunning |
| Writing plotting code | Low | Mistakes are usually visible |
| Interpreting an output table | Moderate | Prone to over-interpretation or confusing parameters |
| Cleaning, merging, recoding data | High | Errors are silent |
| Choosing a statistical method | High | Plausible suggestions may violate assumptions of your design |
| Uploading real data for the AI to “analyse” | Very high | Personal data risk, and intermediate steps are invisible |
The most common silent errors
- Merging on the wrong key: duplicating rows or dropping unmatched ones without notice.
- Default missing-data handling: many functions drop any row with a missing value, so each model quietly uses a different sample.
- Reverse-coded items: not reversed, or reversed twice.
- Wrong variable types: a numerically coded categorical variable treated as continuous.
- An unexpected reference category: leading you to interpret coefficients backwards.
- Invented packages or functions: usually obvious because the code fails, but occasionally an AI suggests a name close to a real package, and installing an unfamiliar package is a security risk.
Describe the data instead of uploading it
You rarely need to share real data to get good code. Instead:
- Describe the structure: variable names, types, meanings, codings, how missing values are marked.
- Create a few rows of fake data with the same structure, or ask the AI to generate them, to test the code.
- State the design clearly: repeated measures, students nested in classes and schools, or independent samples.
- Run the code on your own machine, on the real data.
This keeps participant data where your consent form almost certainly requires it to stay, and forces you to understand your data properly.
A verification routine after every step
- Count rows before and after every cleaning, merging and filtering step. Write it down: “after merge: 412 rows, matching the number of students”.
- Inspect a few specific rows before and after recoding and compare them by eye.
- Test on a case with a known answer: run the code on fake data where you know the correct result, such as two groups differing by exactly 5 points.
- Cross-check with another tool: compute a simple descriptive statistic by hand or in different software.
- Read every line and ask the AI to explain anything you don't understand. If you still don't understand it, don't use it.
The last rule matters most. You must be able to explain every line of your analysis to an examiner or reviewer. “The AI wrote it that way” is not an answer.
Choosing methods: ask to learn, not to decide
AI is useful for mapping options: “For repeated measures at three time points on the same students, which methods are commonly used and what does each assume?” It is less reliable when asked “which method should I use?”, because it does not know what you have not told it about your data. Once you have the options, read a trustworthy source on the one you choose, and consult your supervisor or a statistician if you can.
P-hacking at industrial scale
When writing code takes seconds, trying twenty models is as easy as trying one. That opens the door to p-hacking without any intent. Write your analysis plan first, keep a log of every model you run, and label exploratory analyses honestly.
Reporting in your methods section
Many journals require or encourage disclosure of AI use in analysis. A sample sentence: “Analysis code was written in R 4.4 with assistance from a generative AI tool for initial code drafting; all code was reviewed, rerun and verified on simulated data by the authors. The code is available at …”. Sharing your code is the best way to make AI use something readers do not have to guess about.
Checklist before results go into the paper
- The sample size in every table matches the number of participants after exclusions.
- The direction of reverse-coded scales has been checked.
- Reference categories and coefficient units are confirmed.
- You can explain every line of code.
- A clean rerun reproduces every number in the manuscript.
Your next step: take the most recent AI-written code you rely on, add a row count after each processing step and run it again. Any number that surprises you is where to look closely before you submit.
Câu hỏi thường gặp
Can I use AI to write the analysis code for my thesis?
Usually yes, if your institution allows it, but you must understand and be able to explain every line, verify the results and disclose AI use where required.
Should I upload my survey data file to an AI assistant?
Not if it contains real participant information. Describe the data structure or use fake data to get the code, then run it locally.
How do I know AI-generated code is correct?
Count rows after each step, inspect rows before and after processing, test on fake data with a known answer and cross-check a few statistics with another tool.
Can AI choose the right statistical test for me?
It is better not to rely on it. AI can list options and their assumptions, but the decision should rest on your design and be checked against textbooks or an expert.
Do I need to mention AI in the methods section?
Many journals require or encourage it. A sentence stating what the tool was used for and how you verified the code is enough, ideally with a link to the shared code.
