“Accurate” can mean identifying the correct country, landing within 25 kilometres, or matching the exact street. Those outcomes are not interchangeable. A product can achieve a respectable country score while still producing unusable coordinates, and a few continent-scale errors can hide behind a friendly average.
Recent benchmarks such as IMAGEO-Bench and EarthWhere evaluate visual geolocation across globally distributed images and multiple spatial scales. Their most important lesson for product users is not one headline percentage: performance changes with scene type, region, available clues and the precision demanded.
The four measurements that should appear together
| Metric | Question answered | Main weakness |
|---|---|---|
| Country accuracy | Was the country correct? | Hides city and coordinate error |
| Median distance error | How far is a typical prediction from truth? | Does not show the worst tail |
| Threshold accuracy | What share falls within 1, 25, 200 or 750 km? | Depends on thresholds chosen |
| Calibration | Do high-confidence answers succeed more often? | Needs enough held-out examples |
Distance should be calculated as great-circle distance, not a flat-map subtraction. Report the median and several percentiles. The mean alone is fragile: one wrong-continent prediction can dominate many good city-level results.
Why a balanced dataset matters more than a large-looking score
A test containing famous monuments mostly measures landmark recognition. A real set should separate recognisable landmarks, ordinary urban streets, rural roads, natural landscapes and interiors. Record region, resolution, visible text and metadata availability before running the model.
Freeze the dataset and policy before testing: product version, date, image preprocessing, prompt, retry count and how multiple candidates are scored. If failed images are quietly removed or the best of five attempts is counted, the reported number no longer describes a normal user experience.
Geographic bias is an accuracy problem
Benchmark research reports stronger results for distinctive urban, outdoor and street-level scenes and weaker results in visually ambiguous or under-represented regions. Break results down geographically rather than publishing a global average that may be driven by Western Europe and North America.
Also distinguish model uncertainty from dataset ambiguity. A forest road with no local identifier may genuinely support only a climate zone. Penalising a careful regional answer less than an invented coordinate creates a test that rewards honesty.
Run metadata and visual tests separately
If a file contains GPS, a system may appear perfect without understanding the scene. Create at least two tracks: original-file performance and visual-only performance after location metadata is removed. A third track can test screenshots and compressed social-media copies, which better represent many real uploads.
Test confidence rather than trusting its wording
Sort predictions into confidence bands and calculate success in each band. If “90% confident” results are correct only half the time, the confidence is not calibrated. When no numeric score exists, use candidate spread: alternatives clustered within one city indicate a different uncertainty level from alternatives on three continents.
Publish failure slices
- Correct country, wrong city.
- Correct city, wrong side of the metropolitan area.
- Brand or destination text mistaken for camera location.
- Architecture overgeneralised to a famous nearby city.
- High-confidence wrong-continent result.
- Exact-looking pin generated from region-level evidence.
Show representative failures and the clues that probably caused them. This turns an accuracy report into guidance: users learn when to trust a lead and what to verify.
A minimum publishable benchmark report
Include dataset source and licence, sample counts per category, ground-truth method, model version, test date, preprocessing, retries, country accuracy, median and 90th-percentile distance, threshold accuracy, calibration plot or bands, regional breakdown and all exclusions. Release anonymised raw results when licensing and privacy permit.
Sources
Put the method into practice
When you have a photo to investigate, start with our AI photo location finder and treat its result as a lead to verify, not automatic proof.