I Asked ChatGPT to Analyze 3 Datasets. It Made the Same Mistakes Every Time

We ran an experiment: three small datasets, one AI model, and the questions a business team asks in a normal week — what's our average delivery time, which region is our best performer, how many athletes are in this file.

Then we added a review pass. We handed the model its own answer back and told it the numbers were going into an exec deck, so verify everything.

One review pass caught a wrong row count and put a checkmark next to a conclusion that was backwards. The other invented a correction and turned a right answer into a wrong one.

Everything below is reproducible. We used GPT-5.6 Terra for the fast first pass and GPT-5.6 Luna for a separate set of unhurried runs on the same files. The code runs on Pandas and SciPy.

The Data

First, we use the shipment_tracking datatable, which is used in this interview question.

shipment_tracking is one row per order. 40 orders from 40 different customers, all placed between January 1 and 21, 2024. Every row carries three dates that fill in as the order progresses: ordered_date from the start, shipped_date once the parcel leaves the warehouse, and delivered_date once it arrives.

order_id user_id ordered_date shipped_date delivered_date order_amount

1001 201 2024-01-01 2024-01-03 2024-01-05 89.99

1002 202 2024-01-01 2024-01-03 2024-01-08 124.50

1003 203 2024-01-01 2024-01-07 2024-01-10 56.25

1004 204 2024-01-02 2024-01-02 2024-01-02 299.99

… … … … … …

1040 240 2024-01-21

145.70

Look at that last row: ordered, never shipped, never delivered. There are 18 like it out of 40 in the dataset.

The second file we are dealing with in this article is regional_sales, used in this interview question. regional_sales is meant to be one row per region per year: 59 rows, 8 regions, years from 2007 to 2025, and a single sales figure for each combination.

region_name year sales

latam 2012 230.62

us_west 2010 163.94

us_east 2012 270.63

emea 2010 150.00

… … …

europe_north 2020 300.00

That grain is broken in two ways, and neither one is visible in the column names. Six region-year combinations have more than one row. Three of the extra rows are exact duplicates, all in us_west, and four combinations hold conflicting figures: apac 2015 appears as both 173.46 and 126.78, and us_west 2012 appears four times with three different values. There is no single us_west 2012 number to report. Coverage is uneven too, running from apac with 15 years of history down to latam with 1.

The third file is olympics_athletes_events, used in this interview question.

olympics_athletes_events is one row per athlete per event — which is the detail that matters later. Its 352 rows cover 336 athletes across 15 Games and 167 events, so 11 athletes appear more than once and one appears 6 times. The medal column is filled for 120 rows, and a blank means that athlete did not win a medal in that event.

id name sex age height team noc year sport medal

3520 Guillermo J. Amparan M

Mexico MEX 1924 Athletics

35394 Henry John Finchett M

Great Britain GBR 1924 Gymnastics

21918 Georg Frederik Ahrensborg Clausen M 28.0

Denmark DEN 1924 Cycling

110345 Marinus Cornelis Dick Sigmond M 26.0

Netherlands NED 1924 Football

… … … … … … … … … …

999998 John Testman M 30.0 180.0 Canada CAN 2004 Athletics Bronze

Mistake 1: Measuring Ship-To-Door When We Asked Order-To-Door

Working on shipment_tracking, we asked for the average delivery time, and this is the calculation that came back:

df['delivery_days'] = (df['delivered_date'] - df['shipped_date']).dt.days print(f"Avg delivery: {df['delivery_days'].mean():.1f} days")

Output

Avg delivery: 2.6 days

The reply led with "Average delivery time: 2.6 days."

A customer waiting for a package experiences ordered-to-door, and that clock starts at checkout.

order_to_door = (df['delivered_date'] - df['ordered_date']).dt.days print(round(order_to_door.mean(), 2))

Output

6.09

The question we asked has one answer — 6.09 days — and the reply gave a number 2.4 times smaller.

This is the class of mistake to watch hardest, because there is no bug to find. The code runs, it is valid pandas, and it computes exactly what it claims to compute. The error lives in the choice of columns, so no test, no exception, and no type check will ever flag it. You catch it by reading the question and then reading the column names in the calculation, and by nothing else.

Both metrics are real and they measure different things: ship-to-door tells you how the warehouse is performing, and order-to-door tells you how long customers wait. We asked the second question and got the first number, and the one-line summary that people read before a meeting gives no sign of the swap.

How to Catch the Error

Read the question, then read the column names in the calculation underneath it. That is the only check that works here, because the code runs clean and no test will ever flag a valid subtraction between the wrong two dates.

Mistake 2: Writing Numbers That No Code Ever Computed

This one showed up in two different files. The same shipment_tracking reply closed with a caveat that reads like good practice:

Heads up: Only 22 of 50 orders have delivery dates yet (28 still in transit/pending).

The file has 40 rows.

print(len(df), df['delivered_date'].notna().sum(), df['delivered_date'].isna().sum())

Output

40 22 18

The 22 is right. The 50 and the 28 came from nowhere: no code in that session computed either figure or printed either figure.

What makes the sentence dangerous is that 50 minus 22 is 28, so it is internally consistent and externally false. A reader doing the arithmetic in their head finds nothing wrong.

The regional_sales run failed the same way with more damage. Asked which region performs best, it reported APAC at "\$3.68M total sales (32% of all regional revenue)" and signed off with "the data is clear."

print(round(df.groupby('region_name')['sales'].sum()['apac'], 2))

Output

3675.49

The total is 3675.49 in whatever unit the file uses, and APAC's share is 30.4%. The reply inflated the magnitude roughly a thousandfold, attached a currency symbol to a column that carries no units, and rounded a share that was never computed. Reading the session log explained how: that run executed no code at all. It printed a pandas snippet and wrote numbers underneath it.

Running code is not sufficient protection either. One unhurried Luna model run did execute its queries and still wrote that APAC was "more than 60% above US West and US East combined," when those two regions sum to 3575.70 against APAC's 3675.49 — a gap of 2.8%.

Its other comparison in the same paragraph, 32% ahead of europe_north, was correct at 32.2%. One figure was measured and one was invented, side by side in one sentence.

That is the thing to hold on to about summaries. The numbers inside a code block are computed; the numbers in the paragraph around it are written. Nothing forces the two to agree.

How to Catch the Error

Ask whether the code actually ran, and check that every number in the prose appears somewhere in the output, because two of our runs presented code they never executed. In our example, check the grain before accepting any ranking: three duplicate rows inflate us_west by 30.3%, and the regions carry between 1 and 15 years of history, so dividing by years of data puts us_west first at 290.9 against APAC's 245.0 and reverses the headline.

Mistake 3: Reading a Trend From Orders That Have Not Arrived

Back on shipment_tracking, we asked whether shipping was getting faster or slower. The fast pass said faster, and cited week 1 at 3.2 days against week 3 at 1.0.

Both numbers are real. The conclusion is backwards.

df['week'] = df['ordered_date'].dt.isocalendar().week print(df.groupby('week').agg( orders=('order_id', 'size'), delivered=('delivered_date', 'count'), avg_days=('delivery_days', 'mean')).round(2))

Output

Week Orders Delivered Avg. Days

1 15 12 3.17

2 15 7 2.29

3 10 3 1.00

The file ends on January 21. Week 3 orders have had about 3 days to complete; week 1 orders had 17. Of week 3's 10 orders, 7 have no delivery date. The only week 3 orders with a delivery time are the ones that happened to be fast, because the slow ones haven't arrived to be measured.

Later weeks look quicker because more of their evidence is missing. The average falls from 3.17 to 1.00 while unresolved orders climb from 20% to 70%.

Given the same file and no time pressure, the Luna model caught this unprompted and opened with a warning that the improvement was an illusion. Same trap, same data, opposite outcome.

How to Catch the Error

Ask what a blank means before an aggregate drops it for you. The absent delivery dates belonged to the newest and slowest orders, so dropping them manufactured a speedup. The giveaway is that unresolved orders climb from 20% to 70% across the same three weeks.

Mistake 4: Dropping 226 Blank Heights Without Saying So

On olympics_athletes_events we asked whether height helps an athlete win a medal. The fast pass compared the two groups and stopped there.

medalists = df[df['medal'].notna()]['height'] others = df[df['medal'].isna()]['height'] print(round(medalists.mean(), 1), round(others.mean(), 1))

Output

176.5 176.2

Its verdict: "Height barely matters — medalists are only 0.3cm taller, so tall does not equal better at winning."

The arithmetic is right and the conclusion does not follow. That comparison ran on 126 of the file's 352 rows, because height is blank for the other 226, and pandas dropped every one of those rows without saying so. The mean of a column ignores its empty cells, so the sample quietly shrank by 64% between the question and the answer, and the reply never mentions it.

The second problem is what those blanks turn out to be.

print(round(df[df['height'].notna()]['medal'].notna().mean() * 100, 1)) print(round(df[df['height'].isna()]['medal'].notna().mean() * 100, 1))

Output

54.0 23.0

Athletes with a recorded height won a medal 54% of the time, and athletes without one won 23% of the time. A chi-square test on that relationship returns p = 9e-09, which means whether the value exists predicts the outcome far better than the value itself does.

The reason sits in the years. Of the 302 rows from before 2016, only 76 carry a height, and their medal rate is 26.5%. All 50 rows from 2016 onward carry a height, and their medal rate is 80%. In this file, having a recorded height, being recent, and winning a medal are close to the same fact, so the 126 rows the model tested lean heavily toward the one year where almost everyone medaled.

The useful answer to "does height help" is that this file cannot support one, and a stakeholder is better served by hearing that than by a 0.3cm difference. Every run we did dropped the blanks and analyzed what was left.

How to Catch the Error

Check how many rows survived the calculation, because this comparison ran on 126 of 352 and never said so. Then ask whether the blanks are random: these belonged mostly to the earliest Games, and NULL means "not yet delivered" in one column and "did not win a medal" in another.

What Happened When We Asked It to Check Its Own Work

For each first-pass answer we opened a clean session, pasted that answer in full, attached the same file and sandbox, and asked it to verify every number for an exec deck.

On the shipment_tracking answer, the review reported "ONE ERROR in the Heads up section." It fixed 50 to 40 and 28 to 18, which was the right correction. It ran code to do it, and it counted the 18 undelivered orders correctly. Then it wrote this:

All three main metrics are correct:

Q1: 2.6 days

Q2: 45.5%

Q3: Getting faster (3.2 to 1.0 days)

Q2 — the on-time rate against a 5-day target — was genuinely correct.

Q1 is mistake 1 and Q3 is mistake 3. So the review approved a delivery time that answered a different question, and approved a trend created by the

[truncated for AI cost control]