A decade ago, it wouldn’t have been possible to create an AR demo for your smartphone in a day or two. What’s more we were able to do this for both iOS and Android using AR for web. We used Google’s open-sourced AR on the Web, available for ARCore (Android) and ARKit (iOS) to create cross-platform AR experience for users who downloaded our custom web browser. Below is a quick overview of the ARKit and ARCore features that we leveraged to build a quick Christmas demo project.
The doors that ARKit and ARCore opened for mobile developers
The above video is Google showing Android users what they could potentially see through their smartphones powered by ARCore in the coming years. While game developers have been able to develop AR experiences for years with the powerful Unreal or Unity game engines, web developers previously did not have similar tools to develop AR experiences for the web. Even mobile developers had to put together libraries or custom-build components to develop an app with AR features for smartphones.
With the launch of Apple’s ARKit and Google’s ARCore in 2017, developers could create more interactive experiences for mobile apps because the frameworks packaged together the essential features for AR that used to come separately:
- motion tracking
- light estimation and
- environmental understanding
— Made With ARKit (@madewithARKit) June 30, 2017
— Google (@Google) August 29, 2017
These kits in short help developers create AR content by tracking information on the environment and positioning 3D objects and animations in the foreground in relation to the real world environment viewed through a smarphone camera.
Both Apple and Google’s AR libraries can integrate easily to game engines and other libraries, but are not yet available for all mobile devices. ARKit can be used with Metal and SceneKit, but only iPad and iPhone models with with Apple A9 processors can support ARKit. The beta version of ARCore is only available on Pixel and Samsung S8 at the time of writing.
How AR has developed in the past 10 years
As mentioned above, augmented reality depends on 3 main components: a camera to capture information on the environment, motion tracking for objects in the camera view, a set of 3D objects that can interact with the environment in the camera, and a rendered view.
Back in 2007, these 4 components were usually separately, making a full-featured AR application much more technically challenging and required developers to be knowledgeable about all the components. In addition, the hardware couldn’t support an interactive experience. The 1st generation iPhone was just introduced, which means that most people would be using a webcam attached to a desktop, which isn’t easy to move or interactive with.
Fast-forward 10 years, and we’ve arrived at a mobile-first era with a selection of open-source AR-related libraries to choose from. The mobile experience has created the opportunity to film and render. Nowadays, users can interact directly with what the camera sees on their mobile interface and interact with objects by touching the screen, creating a more seamless experience.
However, ARKit and ARCore are native mobile experiences and are not available for the web directly. Developers would have to create an iOS and Android version for demo projects and prototypes. By leveraging WebAR, we can create an AR experience that users of any smartphone can see from a web browser.
Using AR for Web for small projectsSo what is AR for web? It is simply another variation of tools that run augmented reality on web browsers. Our team worked with three.ar.js, developed by Google, for tracking and three.js for 3D objects. You can see the diagram below to see the layers that go into an AR experience.
On the native layer, ARKit and ARCore help developers do many key processes to create an AR experience including AR markers, surface tracking, geo tracking, face tracking, color tracking, object tracking, and ambient light tracking. This forms the foundation for processing the data needed from the environment so that the developer can focus on creating the 3D objects that will interact with it.
Creating the 3D objects is usually the domain of artists and people who work with 3D animations. However, now lean teams can use Google’s Poly project to get template objects that can be customized. We used Adobe Fuse CC to turn our dancer into Ben, Oursky’s CEO.
After the 3D object is created, we need to load the objects we want in specific scenarios. In our case, we needed to give “Dancing Ben” a flat surface to share his dance movers on. With Web for AR, displaying the object on a designated type of mapped surface becomes as simple as a line of code as shown below:
With AR for web, we were able to focus mostly on modifying a template 3D object and choosing where it would be suitable to dance on. We didn’t have to get bogged down with important details such as fine tuning the accuracy of tracking. The result, our Christmas dancing Ben, was rendered from a compatible phone, like the iPhone X, on our custom browser.
Other great opportunities with AR
Augmented reality has limitless potential in what objects and information can interact with what our smartphones are capturing on camera and displaying on screen. AR’s market potential is just starting to be realized with the improvements in smartphone processors and cameras and open-source developer tools. Brands can engage customers in ways beyond a single sales point. They can create helpful product previews, informative tools and guides, and entertaining brand experiences. Still want more examples? Check out this quick iOS 11 video:
Below are some example applications:
- Interactive stickers for video chats and messages, such as Snapchat, Instagram, Google Hangouts,
- Previewing items in a space like Ikea Place
- Maintenance simulations such as Hyundai’s Virtual Guide that labels parts or mapping out infrastructure below street surface
- Daily tools like a camera tape measure
- Training simulators for everything from cooking to flight control
- Medicine, such as creating an image the brain in 3D on top of a person’s head for surgery
- Navigation and sightseeing, with visualizing routes with Fitness AR by Strava, or interactive cityguides and travel apps
- Interactive advertising with integrated information based on location by companies like Layar
- Interactive gaming experiences such as Pokemon Go
AR for the web is great for cross-platform prototyping and MVP development to test your products or projects. While our demo project still required a custom browser, we predict that within a year or two, AR for Web will make sharing your prototype as easy as sending a link to your web app. If you need some inspiration for your AR app, check out our post on an Augmented reality store for iOS!
If you found this piece helpful, give us some claps or follow the Oursky Publication for more startup/entrepreneurship/project management/app dev/design hacks! 👏
😻 At Oursky we’re all about helping brands and entrepreneurs make their ideas happen. Get in touch if you’re looking for a partner to help build your next digital product.