Apple’s mysterious fisheye projection

If you've read my first post aboutSpatial Video, the second aboutEncoding Spatial Video, or if you've used mycommand-line tool, you may recall a mention of Apple's mysterious fisheye" projection format. Mysterious because they've documented aCMProjectionType.fisheyeenumeration with no elaboration, they stream their immersive Apple TV+ videos in this format, yet they've provided no method to produce or playback third-party content using this projection type.

Additionally, the format is undocumented, they haven't responded to anopen question on the Apple Discussion Forumsasking for more detail, and they didn't cover it in their WWDC23 sessions. As someone who has experience in this area - and a relentless curiosity - I've spent time digging-in to Apple's fisheye projection format, and this post shares what I've learned.

Mike Swanson

There is just so much cool technology crammed into the Vision Pro, from the crazy displays down to, apparently, the encoding format for spatial video. Too bad Apple seems to have forgotten that a technology is not a product, as even the most ardent Apple supporterts - like John Gruber, or the hosts of ATP - have stated their Vision Pro devices are lying unused, collecting dust, just months after launch.