Building a Mixed-Reality Tour Guide with Android XR, the Geospatial API, and Gemini

Posted by Coco Fatus, UX Designer, Alon Hetzroni, UX Engineer, Azin Mehrnoosh, Product Manager Android XR At this year's Google I/O , we announced an update for spatial experiences: the Geospatial API is now available as a preview in ARCore for Jetpack XR . * To explore what the Geospatial API could unlock, our team built a demo: the XR Geospatial Tour. Imagine walking into a new city, putting on a pair of wired XR glasses (like the upcoming XREAL Project Aura), and instantly having a knowledgeable, local guide showing you around.

You don't need to stare down at a 2D map—instead, 3D models gently guide your path, and an intelligent voice tells you about the historical landmarks right in front of you. We combined the Geospatial APIs , Gemini API using Firebase AI Logic , Google Maps Grounding , and Jetpack XR SDK to create a hands-free, immersive walking tour experience. *Disclaimer: Video and Tour Guide application are for demonstration purposes only.

Some sequences have been shortened. Any hardware depicted may be under development; final product details may differ. Let’s walk through the implementation details and show how we tied these APIs together to build a world-scale spatial experience.

1. Pinpointing the User with ARCore Geospatial API (VPS) Enhance your navigation experience on XR by combining the power of GPS with the precision of VPS. The accuracy and precise orientation that comes with VPS allows 3D waypoints to align with the physical world.

This is why the Geospatial API on Android XR can help you build custom experiences. By using advanced computer vision, VPS tries to provide a GeospatialPose (including latitude, longitude, and heading) that is more accurate than GPS. longitude}") } Because the entire experience relies on this accuracy, we monitor the horizontalAccuracy and orientationYawAccuracy until they meet our thresholds.

If the user is indoors or in an unrecognized area, we prompt them to "walk to an outdoor public space and look around". 2. Crafting the Itinerary with Gemini API & Google Maps Grounding Once we have a location, we use the Gemini API using Firebase AI Logic to prompt the Gemini model to act as a local tour guide.

longitude}. , historical, food, nature). ") Large Language Models are great at generating rich descriptions, but they can sometimes hallucinate exact latitude/longitude coordinates.

To solve this, we used Google Maps Grounding to ground the AI. 3. 5 TTS To make the tour guide feel truly present, we implemented dynamic voiceovers.

5-flash-tts model, we can configure our model generation config to natively return audio data instead of just text! inlineData 4. Bringing it to Life in 3D with Jetpack XR The final piece of the puzzle is rendering this data in the user's field of view.