The consumer AR market today remains largely in the confines of the mobile AR space, offering gaming and shopping experiences. Anything more immersive and captivating for consumers requires fully functioning, lightweight, comfortable AR glasses enabled by AR maps. This will begin the era of location-based AR experiences and the virtual ads market. Hardware and mapping requirements are the major bottlenecks to consumer AR that we will analyze in this blog.
Hardware limitations are the most common challenges discussed in the AR community
The currently best-performing AR headset, the Microsoft Hololens 2, weighs more than 500 g. All available analogs are also much greater than the 80 g target that is ideal for consumer AR glasses. Widespread adoption of AR requires major innovations in materials and hardware manufacturing to miniaturize available AR glasses components. This will enable a transition from low-volume enterprise market applications toward the consumer mass market, with applications affecting nearly every element of lives and businesses. AR optics, light engines, cameras, and sensors are the components with the greatest potential for innovations.
In a recent report on hardware innovations in AR/VR, we analyze the maturity and innovation potential of holographic and diffractive waveguides and reflectors. This report also compares display approaches and discusses timelines for microOLED, microLED, and laser beam scanning systems. Additionally, we discuss how IR sensing and metalenses will improve spatial perception and world tracking in AR applications. While it will take five to seven years before we see widespread availability of light, comfortable, aesthetic, affordable, and fully immersive AR glasses, consumers will be able to engage in augmented experiences with limited immersion using smart glasses, which will appear in the next two to three years and allow for truly hands-free location-based AR applications.
However, to enable such experiences and to overlay the real and virtual worlds, a digital copy of the physical space must be created, along with algorithms for precise localization and mapping. Satellite-based navigation provides several-meter accuracy and no angular information for stationary objects, let alone users' direction of sight, which is insufficient for the location-based AR applications requiring 20 cm to 50 cm precision and data on the direction of the user's gaze. AR Maps should allow for differentiating adjacent locations, such as which brand is in which location, in densely populated retail environments.
Visual positioning systems (VPS) offer the precision and directionality that AR maps require. The mapping mechanism is like that of panoramic pictures, where multiple and overlapping images of the real world along with supplemental GPS data are stitched together to construct a 3D representation. Actual positioning is then done by matching a picture to an existing map in a point cloud or other data formats. Among many location-based use cases and advertisement opportunities, one can picture a scenario where a personalized promotional price pops up virtually as a consumer approaches a store, and this is where distinguishing between co-located competing brands and the underlying need for triangulation precision below 50 cm with gaze directionality are paramount.
Both Google and Apple spent billions of dollars on general street view maps (dubbed 2.5D maps) using special cameras and sensors. As much as these maps provide a competitive edge in present navigation and monetization services, the rasterization is not sufficient for location-based AR applications, though it allows for rudimentary VPS and experiences with limited immersion suited for smartphone AR applications.
On the other hand, many companies continue to perfect their algorithms for mapping and VPS, either using specialty systems or processing consumer-grade images from multiple sources in crowdsourced mapping efforts, but lack the data volumes to comprehensively and swiftly cover most popular locations. Immersal, Mapillary, Atlatec, Civil Maps, Pixel8, ARWay, Mapbox, NavVis, Matterport, and many others are developing proprietary algorithms to compete with tech giants. In the mapping technology space, many players face the same challenge: cost and means of data collection.
There are several strategies businesses pursue in applying mapping algorithms to build 3D world representation:
- Dedicated services: to map areas of interest using specialty equipment and trained personnel, such as those used by Google and Apple. This method provides high-quality data but is extremely expensive both for original data collection and for periodic updates. It also takes considerable time to cover limited areas. The majority of self-driving car/mapping companies also fall in this category. It works well for automotive navigation; however, location-based AR applications will largely focus on pedestrian and walking areas.
- Crowdsourcing by volunteers: The idea is similar to what was behind the huge success of the OpenStreetMap project (an open-source equivalent of Google and Apple maps but without graphical details). The data may come from smart dash cameras like Nexar that provide free cloud storage for videos and use them for mapping but cover only automotive roads. Another notable example is Tesla, which generates constantly updated HD maps and refines self-driving algorithms from its car fleet driven by consumers. On the other hand, Mapillary and Pixel8 are equipment-agnostic and process data from any camera device, encouraging users to cover broader areas. While this method may have spotty coverage or require multiple data entries for sufficient quality, the no-cost acquisition model or very broad areas with continuous imagery updates are highly advantageous.
- Rewarded crowdsourcing: Pokemon Go by Niantic is the most famous example here, in which many users play a game hunting for virtual objects while simultaneously recording their journey, which Niantic uses for mapping. While the data acquisition by the players comes at no direct cost, game development involves substantial expenditures. This approach is still cheaper than using dedicated hardware and services. However, the big advantage is that the company can steer large crowds to the desired location (including indoors, where the company uses algorithms from 6D.ai that Niantic acquired recently) to quickly generate abundant data and subsequently map targeted regions, including remote or otherwise unpopular places. Local and global pandemic restrictions strongly impede this approach.
Because the virtual ad space is such a lucrative opportunity (contrary to real maps with defined real estate), tech giants invest heavily in 3D mapping and explore various mapping approaches. Apple and Google have relied on their own mapping fleet. Facebook invested in crowdsourcing mapping startup Mappilary, acquired Scape Technologies, and also launched Project Aria, where volunteers will be wearing glasses with built-in cameras and other sensors to map the world in 3D, among many other research objectives.
Snapchat's future also heavily depends on the company's strategy in AR; the social platform does not often come across as an AR hardware developer in the public media, but compared with the more known AR contenders, the company might be the best-positioned to lead AR customer engagement. The company has a growing share of tech-savvy users interested in AR (more than 75% of the U.S. population aged 13 to 34 years old uses the platform). Snapchat offers a wide range of AR applications (AR lenses) and has mapped Carnaby Street in London and other prominent locations to demonstrate its location-based AR experiences – a way for users to engage in global mapping efforts. The social platform is building future generations of AR glasses and acquired StreetCred in hopes of scaling up crowdsourced AR mapping.
Whereas roadmaps for consumer AR hardware are relatively clear, the emergence of AR maps is less so. In the coming year, we will see video passthrough AR systems like Lynx, Varjo, and Apple headsets offering hands-free consumer AR experiences. In two to three years, more examples of smart glasses for extended and all-day-long use will become available before fully immersive AR glasses come to market in five to seven years.
The time is now for materials and electronics manufacturers to take advantage of a very large opportunity, as AR will replace a major share of personal computers and mobile functions. However, the fully immersive market is also gated by the availability of AR maps to unlock location-based experiences. It is very likely that robust and comprehensive AR maps will come from crowdsourcing, and while it is not clear how to speed up such a massive undertaking and consumer participation, the consequences will affect more than just AR applications (think of autonomous robotics in public spaces, including indoors). Given the broad scope of impact, those that can unlock the miniaturization of AR hardware and those that solve the challenge of AR mapping will be well-positioned to capitalize on this massive trend.