Validating Synthetic Trajectories Against Real Mobility Metrics

Four distributions decide whether a synthetic mobility set is usable: jump length, waiting time, radius of gyration, and the visitation frequency across cells. All four must match the real population, and matching all four is compatible with having memorised individual traces — so the membership test runs alongside, never instead.

Core Calculation # permalink

Each check compares a synthetic distribution $Q$ against the real one $P$, and the right statistic differs by variable type.

Jump length and waiting time are heavy-tailed, so a mean comparison is meaningless — the mean is dominated by the tail and unstable in both samples. Compare the exponents. Fitting $P(\Delta r) \sim \Delta r^{-(1+\beta)}$ over the body of the distribution, the check is

Radius of gyration is a per-agent scalar, so compare the whole distribution with a two-sample Kolmogorov–Smirnov statistic:

Visitation frequency is a vector over cells, so compare it as a distribution over the tessellation. Use the Earth Mover’s Distance rather than a per-cell correlation, because moving trips between adjacent cells matters far less than moving them across the city and correlation cannot tell the difference.

The reason all four are needed is that they fail independently. A model can reproduce trip lengths perfectly while placing every trip in the wrong part of the city; another can get the spatial distribution right and generate agents who never go home.

Worked numeric example # permalink

A 200 000-agent synthetic set generated from a 50 000-user real cohort:

Two failures with different meanings. The waiting-time exponent is a utility failure: the synthetic agents move on too quickly, so any dwell-based analysis built on this set is wrong. The nearest-real distance is a privacy failure: some synthetic points sit on top of real ones, which means the model memorised rather than generalised, and no amount of distributional fidelity excuses it.

A set that passed all four utility checks and failed the fifth would still be unpublishable. That asymmetry is the reason the membership test is not filed under “validation” alongside the others.

Python Implementation # permalink

from __future__ import annotations

import numpy as np
from scipy.spatial import cKDTree
from scipy.stats import ks_2samp, wasserstein_distance


def fit_tail_exponent(values: np.ndarray, lo_pct: float = 10.0,
                      hi_pct: float = 90.0) -> float:
    """Slope of log-log survival over the body — robust where a mean is not.

    Heavy-tailed mobility variables have unstable sample means, so comparing
    means between real and synthetic sets says almost nothing. The exponent is
    the parameter the generative model actually claims to reproduce.
    """
    v = np.asarray(values, dtype=float)
    v = v[(v > np.percentile(v, lo_pct)) & (v < np.percentile(v, hi_pct))]
    v = np.sort(v)
    surv = 1.0 - np.arange(len(v)) / len(v)
    ok = surv > 0
    slope, _ = np.polyfit(np.log(v[ok]), np.log(surv[ok]), 1)
    return float(-slope - 1.0)


def validate_synthetic(real: dict, syn: dict, cell_size_m: float,
                       exp_tol: float = 0.05, ks_tol: float = 0.05,
                       emd_tol: float = 0.05) -> dict:
    """Four utility checks plus the membership check that can veto them all.

    Args:
        real, syn: dicts with `jumps_m`, `waits_s`, `rg_m`, `visits` (per-cell
            frequency vector, same cell order) and `xy` (N,2 projected points).
        cell_size_m: tessellation resolution — the floor a nearest-neighbour
            distance must clear to be evidence of generalisation.
    """
    b_r, b_s = fit_tail_exponent(real["jumps_m"]), fit_tail_exponent(syn["jumps_m"])
    a_r, a_s = fit_tail_exponent(real["waits_s"]), fit_tail_exponent(syn["waits_s"])
    ks = ks_2samp(real["rg_m"], syn["rg_m"]).statistic
    support = np.arange(len(real["visits"]), dtype=float)
    emd = wasserstein_distance(support, support,
                               real["visits"], syn["visits"]) / max(len(support) - 1, 1)

    # Membership: how close does the nearest synthetic point get to a real one?
    dist, _ = cKDTree(syn["xy"]).query(real["xy"], k=1)
    leak = float(np.mean(dist < 1.0))

    utility = {
        "jump_exponent_diff": abs(b_r - b_s),
        "wait_exponent_diff": abs(a_r - a_s),
        "rg_ks_statistic": float(ks),
        "visitation_emd": float(emd),
    }
    passes = {
        "jump_ok": utility["jump_exponent_diff"] <= exp_tol,
        "wait_ok": utility["wait_exponent_diff"] <= exp_tol,
        "rg_ok": utility["rg_ks_statistic"] <= ks_tol,
        "visitation_ok": utility["visitation_emd"] <= emd_tol,
    }
    return {
        **utility, **passes,
        "min_nearest_real_m": float(dist.min()),
        "leak_fraction": leak,
        # A membership failure is a veto: it is not traded off against utility.
        "publishable": bool(leak == 0.0 and dist.min() > cell_size_m),
    }

Verification # permalink

The function above is the verification, and the field to read first is publishable — it depends only on the membership result, not on the utility scores. That ordering is deliberate: a reviewer scanning a report should not be able to reach a favourable overall impression from four green utility rows while the membership row is red.

Two further checks are worth running on any set that passes.

Stratified utility. Report the four metrics per activity stratum — light, medium and heavy travellers — rather than pooled. A generator fitted to the bulk of the population routinely gets the heavy-traveller tail wrong, and pooled statistics hide it because heavy travellers are a small share of agents and a large share of trips.

Rare-transition audit. Count the synthetic transitions whose real support came from a single individual. If the contributor floor was applied correctly this is zero by construction; if it is not zero, the floor was applied after fitting rather than before.

Edge Cases & Adjustments # permalink

• Comparing means on heavy tails. The sample mean of a power-law variable with $\beta < 1$ does not converge, so two draws from the same distribution can differ by an order of magnitude. Always compare exponents or full distributions, never means.
• CRS in the distance checks. jumps_m, rg_m and the nearest-neighbour query are all metres, so the points must be projected. scikit-mobility works in WGS84 degrees by default, which silently makes every distance figure meaningless — the trap described under synthetic mobility data generation.
• The nearest-neighbour floor. A synthetic point should not be closer to a real one than the tessellation resolution, because below that the model has resolved a position it never had access to. Comparing against a fixed metre threshold instead lets a coarse-grid model pass trivially.
• Sample size asymmetry. Generating 200 000 agents from 50 000 real ones makes a nearest-neighbour collision more likely by chance alone. Report the collision rate against a permuted baseline — real points against a shuffled real set — so the figure is interpretable.
• Utility metrics that were fitted. A generator explicitly fitted to reproduce the radius-of-gyration distribution will reproduce it, and that check then measures the fit rather than validating it. Hold out a portion of the real cohort and validate against the held-out part.

FAQ # permalink

If all four distributions match, is the data anonymous?

No. Distributional fidelity is a statement about aggregates and says nothing about whether individual traces were reproduced. A model that memorised a thousand real traces and generated the rest will match every aggregate. The membership test is the only one of these checks that addresses the question.

Which threshold should the exponents use?

0.05 is a working value for a cohort of tens of thousands; the right number depends on the confidence interval of the fit, which narrows with sample size. Fit the exponent on bootstrap resamples of the real cohort and set the threshold at roughly twice that interval’s width.

Why Earth Mover’s Distance for visitation rather than correlation?

Correlation treats every cell as an independent coordinate, so moving a thousand trips from one side of the city to the other scores the same as moving them next door. EMD charges for distance moved, which is what a spatial reader cares about — the reasoning set out under comparing Earth Mover’s Distance for masked distributions.

Does a differentially private generator still need these checks?

It needs the utility checks — a DP fit can be useless as easily as any other. It does not need the membership test as a privacy gate, because the guarantee already bounds that. Running it anyway is a cheap implementation check on whether the DP fit was actually applied.

Related # permalink

• Synthetic Mobility Data Generation — the models these metrics validate
• Generating Synthetic GPS Traces with Markov Models — the smoothing and contributor floor that prevent memorisation
• Comparing Earth Mover’s Distance for Masked Distributions — the spatial distance statistic used above
• Utility Preservation Metrics for Masked Maps — the equivalent framework for masked rather than synthetic releases

← Back to Synthetic Mobility Data Generation