iSimLoc
A visual localization system that uses simulated reference images to recognize aerial locations across changes in lighting, orientation, and altitude.
Contributed real-world flight data, simulated reference imagery, and GAN/VPR model experiments to the team’s localization system.
- Technologies
- Unreal Engine 4
- Microsoft AirSim
- DJI Mavic 2 Pro
- GoPro Max omnidirectional imagery
- GAN-based domain transfer
- visual place recognition
- spherical convolutions
- hierarchical localization
Recognizing aerial places
The same aerial location can look very different as lighting, camera orientation, and flight altitude change. At CMU’s AirLab, I contributed to iSimLoc, a collaborative research project that combines real-to-sim domain transfer, viewpoint-invariant place recognition, and coarse-to-fine global localization. The system uses simulated and overhead reference imagery to recognize locations without requiring prior flights through every target environment.
Field data, simulation, and models
Collecting real-world flight data
I piloted a DJI Mavic 2 Pro carrying a GoPro Max omnidirectional camera to collect real-world data around the CMU campus. These flights captured the same locations under different lighting conditions and viewpoints for training and evaluation.
Simulation and model experimentation
I built the simulated campus environment in Unreal Engine 4 and Microsoft AirSim, starting from a captured 3D LiDAR-image composite mesh created by my master’s advisor, Dr. Ji Zhang. I developed the simulated-image collection pipeline to generate reference views corresponding to real flight trajectories, and trained and modified both GAN-based domain-transfer models and VPR networks. Peng Yin also made substantial contributions to the GANs and localization algorithms; my work connected field data collection, simulation, and model experimentation within the broader team effort.
Global localization results
The team’s complete system achieved 88.7% successful global localization on the CMU Campus dataset and 83.8% on the Large Terrain dataset at a reported 1.5 seconds per query in the linked preprint. Success meant a match within 20 m on campus or 40 m on the terrain dataset. These results measure global re-localization; continuous navigation would additionally require visual odometry and finer image alignment.