GELOS

Geospatial Exploration of Latent Observation Space

Denys Godwin

Title and author block of the paper 'No One Knows the State of the Art in Geospatial Foundation Models' by Corley et al.

Benchmarks rank model performance on ground truth GELOS complements this by exploring the frozen embedding space to see where models differ, and why

Clark University seal NASA logo

3 datasets

Land Cover scene classification · this talk

Fire change detection

Crop temporal dynamics

GELOS Land Cover

Class separation for global categorical land cover

  • 78,585 pure land cover chips
  • GSD 960 m × 960 m
  • Sentinel-1 RTC: 4 scenes / year
  • Sentinel-2 L2A: 4 scenes / year*
  • Landsat Collection 2: 4 scenes / year*
  • DEM: 1 annual scene

* Cloud-free

1,267 chips 992 chips 822 chips 791 chips 752 chips 738 chips 733 chips 729 chips 719 chips 708 chips 679 chips 678 chips 661 chips 642 chips 637 chips 633 chips 600 chips 593 chips 588 chips 588 chips 583 chips 551 chips 545 chips 515 chips 513 chips 513 chips 513 chips 496 chips 491 chips 486 chips 480 chips 478 chips 477 chips 475 chips 472 chips 471 chips 464 chips 462 chips 462 chips 453 chips 450 chips 441 chips 439 chips 437 chips 434 chips 426 chips 420 chips 419 chips 418 chips 412 chips 410 chips 407 chips 404 chips 404 chips 402 chips 402 chips 402 chips 401 chips 401 chips 401 chips 401 chips 401 chips 401 chips 400 chips 398 chips 398 chips 396 chips 395 chips 393 chips 393 chips 393 chips 390 chips 385 chips 385 chips 382 chips 380 chips 379 chips 376 chips 374 chips 371 chips 369 chips 369 chips 367 chips 365 chips 365 chips 364 chips 364 chips 360 chips 359 chips 357 chips 357 chips 354 chips 354 chips 352 chips 344 chips 342 chips 335 chips 326 chips 323 chips 320 chips 319 chips 318 chips 317 chips 305 chips 305 chips 304 chips 303 chips 303 chips 298 chips 291 chips 285 chips 284 chips 283 chips 281 chips 281 chips 280 chips 279 chips 271 chips 269 chips 268 chips 267 chips 263 chips 261 chips 242 chips 241 chips 241 chips 241 chips 238 chips 235 chips 233 chips 228 chips 219 chips 218 chips 218 chips 217 chips 216 chips 216 chips 213 chips 206 chips 201 chips 196 chips 196 chips 195 chips 195 chips 190 chips 185 chips 184 chips 183 chips 183 chips 178 chips 178 chips 177 chips 175 chips 173 chips 173 chips 173 chips 171 chips 167 chips 165 chips 164 chips 163 chips 163 chips 162 chips 162 chips 161 chips 156 chips 155 chips 154 chips 151 chips 148 chips 140 chips 140 chips 139 chips 135 chips 133 chips 133 chips 133 chips 132 chips 129 chips 128 chips 128 chips 127 chips 126 chips 124 chips 121 chips 121 chips 120 chips 117 chips 116 chips 116 chips 116 chips 116 chips 116 chips 115 chips 114 chips 114 chips 112 chips 112 chips 110 chips 109 chips 106 chips 105 chips 103 chips 101 chips 100 chips 100 chips 100 chips 99 chips 99 chips 97 chips 97 chips 96 chips 95 chips 94 chips 93 chips 93 chips 89 chips 89 chips 89 chips 87 chips 87 chips 87 chips 87 chips 86 chips 85 chips 85 chips 84 chips 82 chips 81 chips 81 chips 80 chips 80 chips 80 chips 79 chips 79 chips 78 chips 78 chips 78 chips 78 chips 78 chips 78 chips 77 chips 76 chips 74 chips 74 chips 73 chips 73 chips 73 chips 73 chips 72 chips 70 chips 70 chips 69 chips 68 chips 67 chips 67 chips 67 chips 67 chips 66 chips 66 chips 66 chips 65 chips 65 chips 64 chips 64 chips 63 chips 63 chips 60 chips 60 chips 59 chips 59 chips 58 chips 58 chips 58 chips 57 chips 56 chips 56 chips 55 chips 55 chips 55 chips 55 chips 55 chips 54 chips 54 chips 54 chips 52 chips 52 chips 52 chips 51 chips 51 chips 51 chips 50 chips 50 chips 50 chips 50 chips 49 chips 49 chips 49 chips 48 chips 47 chips 47 chips 47 chips 45 chips 45 chips 45 chips 44 chips 43 chips 43 chips 43 chips 42 chips 41 chips 41 chips 41 chips 41 chips 41 chips 41 chips 40 chips 39 chips 39 chips 39 chips 38 chips 38 chips 37 chips 36 chips 36 chips 36 chips 36 chips 36 chips 34 chips 33 chips 33 chips 33 chips 32 chips 32 chips 32 chips 32 chips 32 chips 31 chips 31 chips 31 chips 30 chips 30 chips 30 chips 30 chips 29 chips 28 chips 28 chips 28 chips 27 chips 27 chips 27 chips 27 chips 27 chips 27 chips 27 chips 27 chips 27 chips 26 chips 26 chips 26 chips 26 chips 25 chips 25 chips 25 chips 25 chips 25 chips 25 chips 25 chips 25 chips 24 chips 24 chips 24 chips 24 chips 24 chips 24 chips 24 chips 24 chips 23 chips 23 chips 22 chips 22 chips 22 chips 22 chips 21 chips 21 chips 21 chips 21 chips 21 chips 21 chips 20 chips 20 chips 19 chips 19 chips 19 chips 19 chips 19 chips 19 chips 19 chips 18 chips 18 chips 18 chips 18 chips 18 chips 18 chips 17 chips 17 chips 17 chips 17 chips 17 chips 17 chips 17 chips 16 chips 16 chips 16 chips 16 chips 16 chips 16 chips 15 chips 15 chips 15 chips 15 chips 15 chips 15 chips 15 chips 15 chips 14 chips 14 chips 14 chips 14 chips 14 chips 14 chips 14 chips 14 chips 13 chips 13 chips 13 chips 13 chips 13 chips 13 chips 13 chips 13 chips 13 chips 13 chips 13 chips 13 chips 12 chips 12 chips 12 chips 12 chips 12 chips 11 chips 11 chips 11 chips 11 chips 10 chips 10 chips 10 chips 10 chips 10 chips 10 chips 10 chips 10 chips 10 chips 10 chips 9 chips 9 chips 9 chips 9 chips 9 chips 9 chips 9 chips 9 chips 9 chips 8 chips 8 chips 8 chips 8 chips 8 chips 8 chips 8 chips 8 chips 8 chips 8 chips 8 chips 8 chips 8 chips 8 chips 8 chips 8 chips 7 chips 7 chips 7 chips 7 chips 7 chips 6 chips 6 chips 6 chips 6 chips 6 chips 6 chips 6 chips 6 chips 6 chips 6 chips 6 chips 6 chips 5 chips 5 chips 5 chips 5 chips 5 chips 5 chips 5 chips 5 chips 5 chips 5 chips 4 chips 4 chips 4 chips 4 chips 4 chips 4 chips 4 chips 4 chips 4 chips 4 chips 4 chips 4 chips 4 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 3 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 2 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 1 chips 78,585 chips · dot area ∝ chips within 1°
Chips per land cover class Rangeland: 35,468 chips Rangeland 35,468 Crops: 17,623 chips Crops 17,623 Trees: 11,806 chips Trees 11,806 Bare ground: 6,396 chips Bare ground 6,396 Water: 4,434 chips Water 4,434 Built area: 2,858 chips Built area 2,858

The config

STAC and xarray parameters are controlled here

  • 78,585 pure land cover chips
  • GSD 960 m × 960 m
  • Sentinel-1 RTC: 4 scenes / year
  • Sentinel-2 L2A: 4 scenes / year*
  • Landsat Collection 2: 4 scenes / year*
  • DEM: 1 annual scene

* Cloud-free

Why 960 m chips?

160 m 320 m 640 m 960 m Sentinel-2 10 m 16 px patch = 160 m Landsat 30 m 16 px patch = 480 m 16 px → 1 patch 5.33 px → 0.33 patches CANDIDATE CHIP 160 m 32 px → 2 patches 10.67 px → 0.67 patches CANDIDATE CHIP 320 m 64 px → 4 patches 21.33 px → 1.33 patches CANDIDATE CHIP 640 m 96 px → 6 patches 32 px → 2 patches CANDIDATE CHIP 960 m 960 m = 2 × lcm(160, 480)

Embedding extraction configurations

Jan–Mar Apr–Jun Jul–Sep Oct–Dec S2 L2A S1 RTC Landsat DEM chip 044914 · 96 × 96 px at 10 m 6 × 6 patches of 16 px PRETRAINED ENCODER Terramind V1 Large Jan–Mar time 0 Apr–Jun time 1 Jul–Sep time 2 Oct–Dec time 3 144 tokens = 4 time steps × 36 patches 14 14 14 14 13 14 15 16 exp010_terramind_v1_large.yaml · embedding_extraction_strategies Center Patch × 4 Time Stepsall_steps_of_middle_patch time start: 0 · stop: null · step: 1 patch start: 14 · stop: 15 · step: 1 → 4 tokens 4 Central Patches, Apr–Junfour_patches_from_april_to_june time start: 1 · stop: 2 · step: 1 patch start: 13 · stop: 17 · step: 1 → 4 tokens

+ baseline comparison to raw pixels

t-SNE

What does each model’s latent space look like?

How GELOS works

STACSTACxarrayxarrayDaskDask
Zarrcloud ZarrSource Cooperativesource.coop
Prithvi EO V2 600M: tsne_scatter_2d

Across models

STACSTACxarrayxarrayDaskDask
Zarrcloud ZarrSource Cooperativesource.coop
Prithvi EO V2 600M: tsne_scatter_2dTerramind V1 Large S2+S1: tsne_scatter_2dOlmoEarth V1.2 Base S2+S1: tsne_scatter_2dAlphaEarth Annual (Center Patch): tsne_scatter_2dSpectral Bands (S2+S1): tsne_scatter_2d

kNN purity · kNN GSD

Do models organize latent space by land cover, or by location?

Across model families

STACSTACxarrayxarrayDaskDask
Zarrcloud ZarrSource Cooperativesource.coop
KNN Purity — 2: Across Families (Prithvi 600M, OlmoEarth v1.2 Base, Terramind Large, AlphaEarth): knn_purity_plot

Across model families: location

STACSTACxarrayxarrayDaskDask
Zarrcloud ZarrSource Cooperativesource.coop
Spatial kNN (GSD & Latitude Difference) — 2: Across Families (Prithvi 600M, OlmoEarth v1.2 Base, Terramind Large, AlphaEarth): knn_gsd_plot

From t-SNE to the dashboard

STACSTACxarrayxarrayDaskDask
Zarrcloud ZarrSource Cooperativesource.coop
Prithvi EO V2 300M: tsne_scatter_2d
PMTilesPMTilesAmazon S3Amazon S3

Select a sample to visualize it clarkcga.github.io/gelos-app

See the sample in any sensor’s imagery clarkcga.github.io/gelos-app

Box-select a cluster, then switch models clarkcga.github.io/gelos-app

Box-select a cluster, then switch models clarkcga.github.io/gelos-app

Select a cluster on the map… clarkcga.github.io/gelos-app

… and find it in every latent space clarkcga.github.io/gelos-app

Visualize nearest neighbors in multiple models clarkcga.github.io/gelos-app

Thank you

Try the app

clarkcga.github.io/gelos-app

Denys Godwin · Rufai Balogun · Eli Simonson · Yao-Ting Yao

Sam Khallaghi · Sujit Roy · Rahul Ramachandran · Hamed Alemohammad

Clark University seal NASA logo