ShaderGraph in visionOS

This article provides an overview, sources of information, and sample code for visionOS’s ShaderGraph.

What’s a Shader

ShaderGraph is a way of representing a shader. The Shader is a program that performs shading calculations in the rendering process of a 3D scene. It is mainly executed on GPUs.

Material represented using ShaderGraph is called ShaderGraph Material. For the sake of brevity, this article will simply refer to ShaderGraph even when it includes ShaderGraph Material.

Fig.1 What’s a Shader

Shaders in RealityKit

RealityKit provides two types of Shaders: GeometryModifier (commonly called Vertex Shader in 3D processing), which calculates the position of the vertices of the 3D model in the shading calculation, and SurfaceShader (commonly called Fragment Shader), which calculates the color of each pixel based on the surface material of the 3D model, considering lighting and camera position.

Fig. 2 Shaders in RealityKit

ShaderGraph in visionOS

In RealityKit for macOS/iOS/iPadOS, Shaders are written using MSL (Metal Shading Language), a programming language based on C++ 14.

This has been changed in the visionOS RealityKit to use a graphical node editing tool called ShaderGraph, which is currently not available for creating shaders using MSL. This node editing tool is provided as part of the Reality Composer Pro functionality, so you need to use Reality Composer Pro to create Shaders for visionOS. Each node used in node editing conforms to the Material X standard specifications. Interoperability of tools compliant with the Material X standard is expected to be improved in the future.

Fig. 3 Shaders in iOS/visionOS

Resources

A variety of resources are available to help you learn the newly introduced ShaderGraph.

For more information on ShaderGraph Node Editor, please refer to the WWDC23 video and Apple Documentation.

• Session: Explore materials in Reality Composer Pro
  (https://developer.apple.com/videos/play/wwdc2023/10202/)
• Apple Documentation: Designing RealityKit content with Reality Composer Pro
  (https://developer.apple.com/documentation/visionOS/designing-realitykit-content-with-reality-composer-pro)

For more information about Material X, refer to the specification. I have published a list including RealityKit’s own nodes. There are about 140 types of nodes.

• MaterialX Spec v1.38 (https://materialx.org/)
• My Note on GitHub: MaterialX Nodes in RealityKit
  (https://github.com/ynagatomo/evolution-Metal-ARKit-RealityKit-sheet)

Fig. 4 Resources

Example

Here is an example of a ShaderGraph Material that applies a textured image to the inside of a sphere. This material displays an animation in which an all-sky image appears from bottom to top and top to bottom as time passes.

Fig. 5 Example

ShaderGraph Material is created using Reality Composer Pro. Material is added to the scene and edited in the ShaderGraph Editor. The usage of each node can be found in the MaterialX Specification. Some node parameters can be changed dynamically from Swift Code. To do so, Promote the corresponding node (blue node in the figure).

Fig. 6 Example

Here is an example of modifying a ShaderGraph node parameter from Swift Code, loading the ShaderGraph Material and setting the value to the parameter of the promoted node using the setParameter(name:value:) method. In this sample, the name of the texture image is set.

Fig. 7 Example

This is a brief introduction to the new ShaderGraph Material introduced in visionOS. At present, only basic function nodes are provided. Future development is expected.

Even a simple ShaderGraph material can create interesting visual effects. This is the same ShaderGraph material on a robot.

The sample code for this article is available on GitHub. Please refer to it if you like.

• GitHub: ynagatomo/360ViewVO