A column swap in DOSM’s own open data

While building the youth unemployment piece, two columns in DOSM’s published Parquet file turned out to be swapped relative to their own metadata. Here’s how we caught it.
methodology
data quality
malaysia
Author

the haddock

Published

August 21, 2026

While preparing the youth unemployment vs. headline unemployment analysis, one column in DOSM’s lfs_month_youth dataset didn’t behave the way its own documentation said it should. This is a short note on what we found and how we checked it, since the same data is publicly available and others may hit the same issue.

What we saw

DOSM’s metadata for lfs_month_youth defines u_rate_15_24 as the unemployment rate for the 15–24 age group, which is a percentage, so it should sit somewhere between 0 and 100. Instead, the raw values looked like this:

date u_rate_15_24
2026-03-01 390.9
2026-04-01 394.7
2026-05-01 396.0
2026-06-01 400.8

A rate can’t be 390. That does not seem like a rounding quirk or an outlier. Rather it appears to be a value from an entirely different scale.

Ruling out our own mistake first

Before assuming DOSM’s file was wrong, we checked whether this was something we’d caused. The most likely candidate being duplicate column names, which can make pandas silently return the wrong data when selecting by label. Columns: [‘date’, ‘unemployed_15_24’, ‘u_rate_15_24’, ‘unemployed_15_30’, ‘u_rate_15_30’] Duplicate column names? False

No duplicates. Whatever was happening was already present in the file as downloaded.

Cross-checking against a published anchor

Rather than guess at what the columns actually contained, we checked two specific months against DOSM’s own published figures, reported via independent news coverage of the Labour Force Report:

  • March 2026: unemployment for ages 15–24 was reported at 10.2%, or 290,000 people. For ages 15–30, 6.2%, or 390,900 people.
  • May 2026: 15–24 unemployment was reported at 10.2%, or 291,600 people. For 15–30, 6.3%, or 396,000 people.

Lining these up against the raw file’s four numeric columns for the same months showed a clear pattern:

Column (as labelled) March 2026 value What it actually matches
unemployed_15_24 290.0 Correct — matches the 290,000 figure
u_rate_15_24 390.9 Actually the 15–30 count (390,900)
unemployed_15_30 10.2 Actually the 15–24 rate (10.2%)
u_rate_15_30 6.2 Correct — matches the 6.2% figure

The two outer columns were labelled correctly. The two inner columns — u_rate_15_24 and unemployed_15_30 — appear to be swapped relative to their metadata description, consistently, across every recent month we checked.

The fix

We corrected the swap during preprocessing, remapping the mislabelled columns before computing any ratios:

# DOSM's lfs_month_youth.parquet has a column-swap bug as of this
# snapshot (confirmed against published DOSM figures, Aug 2026): the
# data under "u_rate_15_24" and "unemployed_15_30" are swapped.
youth_slim = pd.DataFrame({
    "date": youth_raw["date"],
    "u_rate_15_24": youth_raw["unemployed_15_30"],  # mislabeled in source
    "u_rate_15_30": youth_raw["u_rate_15_30"],       # correctly labeled
})

After the fix, the youth unemployment rate for recent months read 10.1–10.3% — consistent with the published figures above.

Why this is worth writing down

Official metadata describing a column isn’t a guarantee the underlying file matches it. This particular error didn’t announce itself with a crash or a missing value — the numbers were present, plausible-looking as some kind of figure, just not the figure the column claimed to hold. It only surfaced because a downstream ratio came out obviously wrong, and it only got resolved by checking against an independent, published anchor rather than trusting either the file or our own assumptions about it.

We haven’t checked whether this affects other DOSM datasets as of the date of this post, or whether it’s already been corrected upstream by the time you’re reading this. If you’re working with lfs_month_youth yourself, it’s worth verifying against a recent month’s published figures before trusting the columns as labelled.

Sources: Department of Statistics Malaysia, Monthly Youth Unemployment (OpenDOSM, CC BY 4.0). Published unemployment figures cross-checked via The Rakyat Post and Human Resources Online coverage of DOSM’s March and May 2026 Labour Force Reports.