Monday, December 14, 2020

The Tales of Vulkan on iOS: Simple mistakes

    At the end of last week, I had thought that I would be done with Vulkan for good. Turns out a simple mistake I made back when I ported GWindow got me stuck on Vulkan. To understand what got me stuck, we need to first learn about UIKit's UIViews and CALayers. A NSView, which macOS uses for content in windows, can optionally be what is called "layer-backed" and CALayers can be added to the View to get special rendering effects, like gradients, fades, and animation. Metal subclasses CALayer into CAMetalLayer to get their GPU accelerated rendering, and this is true for macOS and iOS. Where they diverge is how you actually get a CAMetalLayer onto the views. On macOS you can just add it at runtime as long as you are on the UIThread. iOS has more restrictions with this but we will get to that later. It is critical to know that to make a Vulkan surface for iOS you need to pass a UIView that has a CAMetalLayer attached or a CAMetalLayer instance itself.

    UIViews cannot be not layer-backed like NSViews can. This means that UIViews need to have their Layer defined under the "layerClass" method when you need to be different than the default CALayer. Now when I got Vulkan rendering before, I did just this, but I soon found that we have an issue with that. Like Ozzie, I ran into the duplicate symbols issue when trying to link to the Vulkan Surface file. This had not happened when I had the UIView subclass in a different file. This usually wouldn't be such a big deal, just use the Objective-C runtime library to get around the file-per-implementation rule. Altough this could be done, I could not find a way for Vulkan to detect the Metal support I was implementing at runtime. I even tried changing the methods of the meta-class of the subclassed UIView with no luck. Vulkan just would not recognize the Metal support.

    This is where I had started to panic. I had effectively 5 days until my presentation and I did not have Vulkan rendering anymore. I started to look for alternatives to passing the UIView to Vulkan and found that you could just pass the CAMetalLayer instance itself. I tried casting the CALayer that was attached to the runtime view, but even that would recognize as not Metal compatible by Vulkan. I had run out of ideas and looked for a alternative to subclassing UIView altogether. It was lucky that I started to do this, since I found a subclass of UIView that (presumably) was not created using the Objective-C Runtime Library. This class was MTKView, which stands for Metal Kit View. You can create an instance of this class and it will have a CAMetalLayer attached by default. This saved my skin, as it made the Vulkan surface creation method happy. But there was one other problem, the view would only render the top left quadrant of the screen. I thought that it was back to the lab with this so I went back.
    
    I tried adding a sublayer to the default UIView and rendering with that, but that didn't work and made the problem worse, even though it did render. Finally I had to call for some help. I first got Ozzie in a call, to see if I was doing anything wrong with the Objective-C runtime, but it turns out I made the classes perfectly (somehow) and it really was Vulkan not recognizing the View. Then I called Lari, the Graphics man himself. He instantly recognized the issue as a viewport scaling issue. We got to work messing with the scaling options with the sublayer implementation. That ended up not working too well, so we went back to the MTKView implementation. Here is where we made a big discovery. iOS uses an odd method of getting pixel data. They use a width and height of pixels, but it is a little different with retina-enabled devices. They also use a "Scale Factor" property. This is different per-device, usually the larger the device, the larger the scaling factor. The iPhone 8 has a scaling factor of 2 while an iPhone 6s+ has a scaling factor of 3. Older phones like the iPhone 3 and 4 have a factor of 1. 

    With all of this information we finally came to the conclusion that the View and Layers all had the correct scaling and resolution applied. But there was one object that did not, the Vulkan Swapchain. I did not mention that neither Vulkan, nor Metal had any errors when rendering. This is because the Vulkan swapchain was half the resolution it should have been. By simply doubling the width and height of the swapchain, we fixed the top left corner rendering issue. The problem was actually in GWindow all along. It did not account for the scaling of retina devices. Once I resolved that mistake the Vulkan Surface worked like a charm. 

References:

UIKit

MetalKit

MoltenVK

UIView

MTKView

NSView

CALayer

CAMetalLayer

Objective-C Runtime Library

Monday, December 7, 2020

Vulkan on iOS part 2

 After being able to link Vulkan libraries in our Xcode iOS project, we are finally able to write Vulkan code for iOS. To use Vulkan with iOS, we need to give Vulkan the pixel buffer area for the screen. Usually this is done through something like HWND for DirectX, but with iOS we cannot get that low to a raw window pointer. So Vulkan asks for a UIView pointer, but not just any UIView. The one that you give to Vulkan must have a backing layer if CAMetalLayer or something that derives from CAMetalLayer. In MacOS this is an easy thing to achieve, just grab the NSWindow and say that it's backing layer is Metal, but in iOS, it gets a bit messy.

 iOS UIviews are layer-backed, meaning that they are given a layer upon creation and you absolutely cannot change it after (unless you write a custom View controller). This creates a conflict for Gateware. We want the user to be able to create a UIView themselves if they wish and still be able to use Vulkan, but we cannot change what they have specified. It is possible to run 2 controllers at the same time, or even just 2 views on the same controller. This solution has its drawbacks too. Since Gateware is usually used for games, this extra view or controller would be unecessarily using system resources. There must be a solution where the user can create a window but have Gateware put Vulkan into it.

 This is where I had an idea, if we cannot change the current view, can we change the current view controller? The answer was simple, as long as you present the view once you are done swapping them, the newly created view would be the one that is visible, and on top. So I had decided to create a Gateware View and View controller that had support for Metal. Next was just to detect whether or not the users View was Metal capable and overwrite it if not. This allows easy access for people wanting to develop Vulkan for their games and a level of control for people that already have a game looking to use Gateware.

  I am glad to say that the effort paid off. After fiddling around with some of the desktop Unit Tests, (and some incorrect file IO pathing) I was able to run them on iOS Simulator and an iPhone 7 running iOS 13. It feels nice to finally be done with Vulkan and to move on to Audio.

 References:

MoltenVK 

https://github.com/KhronosGroup/MoltenVK

UIKit UIViewController 

 https://developer.apple.com/documentation/uikit/uiviewcontroller

UIKit UIView 

 https://developer.apple.com/documentation/uikit/uiview

Metal CAMetalLayer 

https://developer.apple.com/documentation/quartzcore/cametallayer