Working closely with MLB, Nexus Studios developed MLB Next, a first-of-its-kind mobile app powered by T-Mobile 5G, available on iOS and Android for fans at T-Mobile Park in Seattle. The app streams real-time ball and player positions from MLB’s StatCast directly to your device and makes the action viewable in a 3D model of the stadium and overlaid in AR through your phone’s camera. Additionally, a 3D Strike Zone visualization gives a close-up view of every pitch, while a replay feature allows fans to rewatch the action of previous plays. Audio streaming of Mixhalo feeds augments the experience with live commentary in both English and Spanish. Full player stats data is of course available as well, rounding out this feature-rich next-generation companion app.

To create a digital twin of the stadium for use in AR mode, Nexus worked with the team at Immersal to map the stadium and localize viewers using the phone’s built-in camera. The result of merging location-based AR and Live-Data feels like magic: fans can point their camera at the pitcher’s mound and immediately see the names of players floating above their heads. Player names can be tapped to pull up player cards to read stats and learn more.

Along with ultra-low-latency data pipe and snappy and laser-accurate localization, the app boasts a brisk and responsive UX that builds upon MLB’s existing range of innovative products, packing nuance and detail into every inch of screen. Connected to MLB’s powerful and data-rich backend, the hi-tech experience feels natural and provides an easy way to engage with baseball’s myriad stats. Crucially, the AR experience for HRD and ASG evolves MLB’s signature visual design style into the real world. This meant uncovering solutions to the challenges of viewing stats, figures, and other digital content in a 3D environment – essentially evolving MLB’s iconic presence off the screen and into reality.

Through advanced data processing techniques, we achieved real-time synchronization between the physical game and its digital AR counterpart. To create this dynamic live-data augmented system for fans, engineers at Nexus Studios needed to work closely with the data team at MLB to ensure sub-second latency from pitch to pixel. We built a custom backend infrastructure and meticulously optimized the real-time data streaming pipeline to support 50k simultaneous users. The pipeline accesses live data captured by Hawk-Eye Innovations technology and processed by StatCast and delivers it to fans in less than 500ms after it happens on the field. This means we can keep up with the most high-velocity gameplay to provide a seamless and immersive experience for MLB fans.

For example, for the Home Run Derby event, we can wow fans with an explosion of hit trails that arc across the sky, visualizing the power and prowess of MLB’s biggest hitters. This ‘hit spray’ visualization is like an x-ray superpower for fans who want to feel the mighty impact of the cumulative action on the field. For All-Star Game, we designed a comprehensive digital experience that sits on top of the on-field action, capturing player and ball locations in real-time and serving fans a deeper look into the dynamics of the game. It is designed to be a 9-inning companion for fans, an experience that can be easily jumped into and out of throughout the event.

“MLB is known for its robust line of thoughtfully crafted products for fans, and we were thrilled to be brought in as trusted partners to deliver on their innovative vision, leveraging T-Mobile 5G. Under the hood, MLB Next includes a state-of-the-art integration of real-time player and ball tracking with precise AR camera localization delivered over a high speed network. For fans, it marks the beginning of a new phase of their engagement, where data is visible, interactive, and responsive in a way that provides a deeper connection to the game. This high watermark is only possible because of the unique ecosystem of partnerships and technologies that enabled it – including Immersal VPS, Hawkeye Innovations and Mixhalo Audio streaming – and the exceptionally talented team at Nexus Studios.”

– Pablo Colapinto, Head of Immersive at Nexus Studios.