About The Author

Preston So is a product architect and strategist, digital experience futurist, innovation lead, developer advocate, three-time SXSW speaker, and author of … More about Preston ↬

Email Newsletter

Weekly tips on front-end & UX .
Trusted by 200,000+ folks.

In this overview of WebXR technologies and the Babylon.js framework, we’ll embark on a journey through the past, present, and future of mixed reality both on the web and on immersive headsets. We’ll inspect the underpinnings of WebXR and the most important aspects of the WebXR Device API before turning our attention to Babylon.js, a framework for building immersive applications in JavaScript for web, mobile, and headset. This tutorial is geared toward web and JavaScript developers who build web applications but are looking to dip their toes into immersive experiences.

Immersive experiences, especially those governed by mixed reality (XR), which encompasses both augmented and virtual reality, are quickly gaining new attention among developers and architects interested in reaching users and customers in novel ways. For many years, the lack of adoption of mixed reality experiences came down to hardware — too expensive and unwieldy — and software — too complex and finicky to use.

But the Coronavirus pandemic may be scrambling all of those old calculations by encouraging the sorts of experiences that are mostly limited to the gaming world, which is seeing immense growth in playing time during the current crisis. The math behind three-dimensional spaces can also present barriers to developers, but fortunately, you need only a little vector geometry and matrix math to succeed with XR experiences, not a college course in linear algebra and multivariate calculus.

Though browser support for WebXR is widening, building immersive experiences into browsers or headsets can be complicated due to shifting specifications and APIs as well as rapidly evolving frameworks and best practices. But incorporating immersion into your next web application can also introduce new dimensionality and richness to your user experience — all without the need to learn a new programming language.

More after jump! Continue reading below ↓

What Is WebXR?

Simply put, WebXR is the grouping of standards responsible for supporting rendered three-dimensional scenes in virtual and augmented reality, both experiential realms known together as mixed reality (XR). Virtual reality (VR), which presents a fully immersive world whose physical elements are entirely drawn by a device, differs considerably from augmented reality (AR), which instead superimposes graphical elements onto real-world surroundings.

WebXR-compatible devices run the gamut from immersive 3D headsets with built-in motion and orientation tracking and names like Vive, Oculus, and Hololens to eyeglasses with graphics placed over real-world images and smartphones that display the world — and additional elements — on their native cameras.

The WebXR Specification And Browser Support

The WebXR Device API is the primary conduit by which developers can interact with immersive headsets, AR eyeglasses, and AR-enabled smartphones. It includes capabilities for developers to discover compatible output devices, render a three-dimensional scene to the device at the correct frame rate, mirror the output to a two-dimensional display (such as a 2D web browser), and create vectors that capture the movements of input controls.

Currently a working draft, the WebXR specification is a combination of the foregoing WebVR API, which was designed solely for virtual reality use cases, and the all-new WebXR Augmented Reality Module, which remains highly experimental. WebVR, formerly the predominant and recommended approach for virtual reality experiences, is now superseded by WebXR, and many frameworks and libraries offer migration strategies between WebVR and the newer WebXR specification.

Though WebXR is now witnessing adoption across the industry, browser support remains spotty, and it’s not yet guaranteed that a mixed reality application built according to the WebXR specification will work off the shelf in production.

Chrome 79, Edge 79, Chrome for Android 79, and Samsung Internet 11.2 all offer full WebXR support. But for unsupported browsers like Firefox, Internet Explorer, Opera, Safari, or certain mobile browsers (Android webview, Firefox for Android, Opera for Android, and Safari on iOS), there’s a WebXR Polyfill available thanks to WebXR community members that implements the WebXR Device API in JavaScript so that developers can write applications according to the latest state of the specification. On Firefox for web and Firefox for Android, you can enable the experimental feature flag by navigating to about:config and setting dom.vr.webxr.enabled to true in your browser’s advanced settings.

Installing the WebXR API Emulator on Chrome or Firefox on a personal computer will introduce additional tools to help you with debugging and testing.

The WebXR Device API depends on WebGL (Web Graphics Library), the rendering engine that supports three-dimensional graphics, and therefore employs many WebGL concepts when it performs the rendering, lighting, and texturing needed for a scene. Though the deepest reaches of WebGL are well beyond the scope of this article, those already familiar with WebGL will benefit from existing expertise.

Several open-source JavaScript frameworks are available to interact with WebGL and WebXR, namely Three.js and Babylon.js . A-Frame , a browser-based, markup-focused approach to WebXR, is built on top of Three.js. In this tutorial, we spotlight Babylon.js, which has gained attention as of late due to its large API surface area and relative stability. But these aren’t like the JavaScript libraries and frameworks we use to build two-dimensional web applications; instead, they play in the sandbox of three-dimensional spaces.

Field Of View (FOV) And Degrees of Freedom (DoF)

In this article, we’ll focus on building a simple immersive experience with limited input and a static object, which means our need for deep knowledge about WebGL is minimal. But there are critical WebXR concepts outside of WebGL that are fundamental not to three-dimensional graphics themselves but how to interact with three-dimensional spaces. Because WebXR is rooted in a viewer’s experience, everything revolves around the immersive headset or viewport the user faces.

All headsets and smartphones have a camera serving as the user’s viewport into an immersive experience. Every camera has a certain field of view (FOV) that encompasses the extent of the viewer’s surroundings that is visible at any given time in a device. A single human eye has an FOV of 135º, while two human eyes, with overlapping FOVs, have a combined FOV of 220º wide. According to MDN, most headsets range between 90º to 150º in their field of view.

The virtual or augmented world seen through the camera’s field of view can be adjusted through movement , which occurs along degrees of freedom whenever a device is shifted in certain ways while the user remains stationary. Rotational movement occurs along three degrees of freedom (3DoF), which is a baseline for most basic immersive headsets:

  • Pitch is movement incurred by looking up and down. In pitch, the user’s head pivots on the x-axis, which extends horizontally across the viewport.
  • Yaw is movement incurred by looking left and right. In yaw, the user’s head pivots on the y-axis, which extends vertically across the viewport.
  • Roll is movement incurred by tilting left and right. In roll, the user’s head pivots on the z-axis, which extends forward into the viewport and into the horizon.
A diagram showing three degrees of freedom in rotational movement, with yaw, pitch, and roll.
Rotational movement occurs along three degrees of freedom (3DoF): pitch (pivoting on the x-axis), yaw (pivoting on the y-axis), and roll (pivoting on the z-axis). ( Large preview )

Though three degrees of freedom are sufficient for simpler immersive experiences, users typically wish to move through space rather than merely changing their perspective on it. For this, we need six degrees of freedom (6DoF), whose latter three degrees define translational movement through space — forward and reverse, left and right, up and down — to pitch, yaw, and roll. In short, 6DoF includes not just pivoting along the x-, y-, and z-axes but also moving along them. Due to the frequent requirement of external sensors to detect translational movement, only high-end headsets support all six degrees.

A diagram showing six degrees of freedom in rotational and translational movement.
Rotational and translational movement occur along six degrees of freedom (6DoF). ( Large preview )

WebXR Session Modes

With WebXR superseding the foregoing WebVR specification, it now provides one API as a single source of truth for both augmented and virtual reality. Each WebXR application begins by kicking off a session , which represents an immersive experience in progress. For virtual reality, WebXR makes available two session modes: inline , which deposits a rendered scene into a browser document, and immersive-vr , which depends on a headset. For augmented reality, because rendering is only possible into smartphone cameras and transparent glasses or goggles instead of browsers, immersive-ar is the only available mode.

Because many of us don’t have an immersive headset handy at home, and because the WebXR Augmented Reality Module remains in heavy development, we’ll focus our attention on an immersive VR experience that can be rendered into a browser canvas.

Setting A Scene With WebXR And Babylon.js

In this section, we’ll learn how to create and render a WebXR scene with Babylon.js, the surroundings of our environment and the setting of our experience, before turning any attention to actions like user input or motion. Babylon.js is a free and open-source web rendering engine based on WebGL that includes support for WebXR and cross-platform applications in the form of Babylon Native. Babylon.js offers a slew of additional features, including a low-code Node Material Editor for shader creation, and deep integration with WebXR features like session and input management. The Babylon.js website also provides playground and sandbox environments.

Though selecting between Babylon.js and Three.js comes down to developer preference, Three.js focuses on extensibility over comprehensiveness, with a host of interchangeable modules that add supplemental functionality. Babylon.js, meanwhile, provides a more full-featured suite that may prove overkill for smaller projects but does offer the needed surface area for many implementations.

Introducing Babylon.js

Though you can handle all of the interactions with the WebXR Device API yourself, Babylon.js provides an optional Default Experience Helper that can set up and shut down sessions on your behalf. The Default WebXR experience helper also includes input controls and other features, as well as a rudimentary HTML button to enter the immersive experience. To experiment with the default experience helper, let’s write an HTML page that provides a canvas for XR display and serves the Babylon.js source from a CDN. You can find this HTML page at the GitHub repository for this tutorial on the main branch.

Open index.html in your preferred code editor and in a browser. For this first portion of the tutorial, we’ll inspect the file instead of adding code. In a WebXR-enabled browser like Chrome or Firefox (with the WebXR feature flag enabled in the case of Firefox), you’ll see a canvas containing the initial Babylon.js playground — a “Hello World” of sorts — and you can drag your mouse on the screen to reorient yourself. The screenshot below depicts this initial state.

A screenshot showing the initial state of the Babylon.js default playground.
The initial state of the Babylon.js default playground. ( Large preview )

First, we’ll embed the latest versions of Babylon.js from the Babylon CDN as well as other helpful dependencies. We’ll also add some styles for our scene canvas element in , which is where our immersive experience will render.

						Babylon WebXR Demo

Now it’s time for our Babylon.js implementation. Inside a

If you run the code in the main branch or any of the other repository branches on a WebXR-compliant browser or device, you’ll see our completed scene. As a next step, try adding an animation to see the animation callback at work.

Next Steps: Supporting And Managing User Input

It’s one thing to establish a virtual or augmented world for viewers, but it’s another to implement user interactions that allow viewers to engage richly with your scene. WebXR includes two types of input: targeting (specifying a single point in space, such as through eye-tracking, tapping, or moving a cursor) and actions (involving both selection, like tapping a button, and squeezes, which are actions like pulling a trigger or squeezing a controller).

Because input can be mediated through a variety of input sources — touchscreens, motion-sensing controllers, grip pads, voice commands, and many other mechanisms — WebXR has no opinion about the types of input your application supports, beyond intelligent defaults. But because of the colossal surface area exposed by all input sources, especially in Babylon.js , it would take another full article in its own right to capture and respond to all manner of eye movements, joystick motions, gamepad moves, haptic glove squeezes, keyboard and mouse inputs , and other forms of input still over the horizon.

Debugging, Extending, And Bundling Babylon.js

Once you’ve completed the implementation of your WebXR application, it’s time to debug and test your code, extend it as desired for other rendering mechanisms and game physics engines, and to bundle it as a production-ready file. For a variety of use cases, Babylon.js has a rich ecosystem of debugging tools, rendering mechanisms, and even physics engines (and the ability to integrate your own ) for realistic interactions between objects.

Debugging Babylon.js With The Inspector

Beyond the browser plugins available for WebXR emulation, Babylon.js also makes available an inspector for debugging (built in React). Unlike tools like Jest, because Babylon.js lacks an official command-line interface (CLI), debugging takes place directly in the code. To add the inspector to our Babylon.js application, we can add an additional external script to the embedded scripts in our :


Then, just before we finish creating our scene, let’s indicate to Babylon.js that we want to render the scene in debug mode by adding the line scene.debugLayer.show() just before our return statement:

	// Initialize XR experience with default experience helper. const xrHelper = await scene.createDefaultXRExperienceAsync(); if (!xrHelper.baseExperience) { // XR support is unavailable. console.log('WebXR support is unavailable'); } else { // XR support is available; proceed. scene.debugLayer.show(); return scene; }

The next time you load your Babylon.js application in a browser, you’ll see a “Scene Explorer” to navigate rendered objects and an “Inspector” to view and adjust properties of all entities Babylon.js is aware of. The screenshot below shows how our application now looks with debug mode enabled, and branch debugging-1 reflects this state in the tutorial code.

A screenshot showing Babylon.js’s debug mode in the browser, with the Babylon.js inspector enabled.
Babylon.js’s debug mode in the browser, with the Babylon.js inspector enabled. ( Large preview )

The Babylon.js documentation offers both comprehensive information about loading and using the inspector and a series of videos about inspection and debugging.

Integrating And Bundling Babylon.js With Other JavaScript

Although over the course of this tutorial, we’ve used a script embedded directly into the HTML containing our canvas, you may wish to execute the script as an external file or to leverage an application framework like React or Ionic. Because Babylon.js makes all of its packages available on NPM, you can use NPM or Yarn to fetch Babylon.js as a dependency.

	# Add ES6 version of Babylon.js as dependency using NPM. $ npm install @babylonjs/core # Add ES6 version of Babylon.js as dependency using Yarn. $ yarn add @babylonjs/core # Add non-ES6 version of Babylon.js as dependency using NPM. $ npm install babylonjs

Documentation is available on the Babylon.js website for integrations of Babylon.js with React (including react-babylonjs , a React renderer for Babylon.js) and Ionic (a cross-platform framework). In the wild, Julien Noble has also written an experimental guide to leveraging Babylon.js in React Native’s web renderer.

For front-end performance reasons, you may also consider introducing a server-side rendering mechanism for the Babylon.js applications you build. Babylon.js offers a headless engine known as NullEngine , which replaces Babylon.js’ default Engine instance and can be used in Node.js or server-side environments where WebGL is absent. There are certain limitations, as you’ll need to implement a replacement for browser APIs like XMLHttpRequest in Node.js server frameworks like Express.

Meanwhile, on the client side, generating a lightweight client bundle that can be parsed quickly by a browser is a common best practice. While you can use Babylon.js’ CDN to download a minified version of the core Babylon.js library, you may also wish to combine Babylon.js and your Babylon.js implementation with other scripts like React by using a bundler such as Webpack. Leveraging Webpack allows you to use Babylon.js modularly with ES6 and TypeScript and to output client bundles representing the full scope of your JavaScript.

Immersing Yourself In WebXR

The road ahead for WebXR is bright if not fully formed. As people continue to seek more immersive and escapist experiences that enfold us completely in a virtual or augmented world, WebXR and Babylon.js adoption will only accelerate.

In these early days, as browser support solidifies and developer experiences mature, the promise of WebXR and rendering engines like Babylon.js can’t be understated. In this tutorial, we’ve only had a glimpse of the potential of immersive experiences on the web, but you can see all of our code on GitHub .

That said, it’s essential to remember that mixed reality and immersive experiences in WebXR can present problems for certain users. After all, virtual reality is, for all intents and purposes, a gambit to trick the viewer’s eyes and brain into perceiving objects that aren’t actually there. Many people experience virtual reality sickness , a dangerous illness with symptoms of disorientation, discomfort, and nausea. Physical objects that aren’t visible in virtual reality headsets can also pose hazards for users of immersive experiences. And perhaps most importantly, many immersive experiences are inaccessible for users with cognitive and physical disabilities such as blindness and vertigo-associated disorders.

Just as immersive experiences still remain out of reach for many users, whether due to lack of access to an immersive headset or WebXR-enabled browser or because of disabilities that stymie the user experience, mixed reality also remains a bit of a mystery for developers due to shifting sands in specifications and frameworks alike. Nonetheless, given immersive media waits just around the corner for digital marketing, we’ll see a new scene get the spotlight and take shape very soon — all puns very much intended.


3D Graphics and WebGL

WebXR Device API


Smashing Editorial (sh, ra, yk, il)