Hey, curious minds! Today, we’re diving into a super cool topic called foveated rendering. It might sound like a big, fancy word, but don’t worry, we’ll break it down together. It’s part of the secret sauce for Apple Vision Pro and several other headset makers. Foveated rendering is a special trick that helps make those experiences even cooler, without making the spatial computer or VR/AR headset work too hard. Let’s explore what this is all about!

How Our Eyes Work

Before we talk about foveated rendering, let’s chat about our eyes. When we look at something, we don’t see everything around us in super clear detail. There’s a spot where what we’re looking right at is very clear and detailed, and this spot is called the “fovea.” The area around the fovea isn’t as clear, but we don’t usually notice because our brains are smart and fill in the blanks! The image below comes from this article at Science Direct, and does a great job illustrating how our field of view works.

The Magic of Foveated Rendering

Foveated rendering uses the magic of how our eyes work to make games and VR/AR experiences better. That is, if the headset has cameras or sensors that are able to do what’s called “eye tracking.” Because the headset can identify what you are looking at, it can make just that part super clear and detailed. It works just like the fovea in our eyes. The areas on the screen that you’re not looking directly at aren’t made with as much detail, which is okay because we don’t notice that part as much. This post from last summer by Osma Ahvenlampi is a great deep dive on that.

Apple’s implementation of foveated rendering gets a superpower in the form of the R1 chip. This custom designed processor is designed to reduce latency, which means very little lag. You look at something, and Vision Pro is aware. It’s that simple.

Why Foveated Rendering is Awesome

Saves Power

Imagine if you had to clean your entire room, but your mom said you only had to make sure the middle of the room was super clean and just tidy up the edges. You’d save a lot of energy, right? Foveated rendering does something similar for headsets and spatial computing. By only focusing on making the part you’re looking at super detailed, it saves a lot of computer power. University of Illinois researchers estimated it could save 40% energy consumption. This means your games can look amazing without needing a super powerful computer. In Apple’s case, this means they are able to combine both a powerful computer and spectacular graphics in the same product, without the need to be connected to a separate computer.

Foveated Rendering Makes Games Look Better

Games (and everything else) look even more real and cool. Because it saves energy on the parts you’re not looking at closely, it can use that energy to make the parts you are looking at super detailed and awesome.

Helps VR Feel Real

In VR, feeling like you’re really in the game is super important. Foveated rendering helps make this happen by making sure wherever you look, things look sharp and real, just like in the real world.

How Foveated Rendering Works

Foveated rendering is like having a smart assistant who knows exactly where you’re looking. Eye trackers in the headset tells the software assistant where your eyes are looking. This assistant quickly cleans up and makes everything look perfect right where your eyes are focused, but relaxes a bit on the parts you’re not looking at. It uses special eye-tracking technology to know where your fovea (the super clear part of your vision) is pointing, and then it adjusts the display in real-time.

Cool Facts About Foveated Rendering

• It’s a big deal in because less computing power and less energy needed helps make headsets lighter and last longer without needing to be charged.
• Scientists and computer experts are always finding new ways to make foveated rendering even better, so games and immersive experiences will keep getting cooler and more realistic.
• Though Apple Vision Pro launched in February 2024, the site Patently Apple has found patent records from 2021 detailing their work on foveated rendering.

In Conclusion

Foveated rendering is a super smart way to make games and virtual reality experiences look amazing without needing the most powerful gadgets. It’s all about understanding how our eyes see the world and using that knowledge to make our digital worlds feel just as real. Next time you use a spatial computer or a VR/AR headset with eye tracking, think about how foveated rendering is working behind the scenes to make your adventure as awesome as possible.

Thanks for learning about foveated rendering with me today. Keep being curious, and who knows? Maybe you’ll help invent the next big thing in spatial computing one day!

Latest Posts

• Uncomfortable Creatures
• From Convenience to Concern: How Data Mining Threatens Your Privacy
• Eye Tracking: A Peek Into Where You Look
• Tunnel Vision VR360 Test
• What Is Foveated Rendering?