The Tales of Vulkan on iOS: Simple mistakes

    At the end of last week, I had thought that I would be done with Vulkan for good. Turns out a simple mistake I made back when I ported GWindow got me stuck on Vulkan. To understand what got me stuck, we need to first learn about UIKit's UIViews and CALayers. A NSView, which macOS uses for content in windows, can optionally be what is called "layer-backed" and CALayers can be added to the View to get special rendering effects, like gradients, fades, and animation. Metal subclasses CALayer into CAMetalLayer to get their GPU accelerated rendering, and this is true for macOS and iOS. Where they diverge is how you actually get a CAMetalLayer onto the views. On macOS you can just add it at runtime as long as you are on the UIThread. iOS has more restrictions with this but we will get to that later. It is critical to know that to make a Vulkan surface for iOS you need to pass a UIView that has a CAMetalLayer attached or a CAMetalLayer instance itself.

    UIViews cannot be not layer-backed like NSViews can. This means that UIViews need to have their Layer defined under the "layerClass" method when you need to be different than the default CALayer. Now when I got Vulkan rendering before, I did just this, but I soon found that we have an issue with that. Like Ozzie, I ran into the duplicate symbols issue when trying to link to the Vulkan Surface file. This had not happened when I had the UIView subclass in a different file. This usually wouldn't be such a big deal, just use the Objective-C runtime library to get around the file-per-implementation rule. Altough this could be done, I could not find a way for Vulkan to detect the Metal support I was implementing at runtime. I even tried changing the methods of the meta-class of the subclassed UIView with no luck. Vulkan just would not recognize the Metal support.

    This is where I had started to panic. I had effectively 5 days until my presentation and I did not have Vulkan rendering anymore. I started to look for alternatives to passing the UIView to Vulkan and found that you could just pass the CAMetalLayer instance itself. I tried casting the CALayer that was attached to the runtime view, but even that would recognize as not Metal compatible by Vulkan. I had run out of ideas and looked for a alternative to subclassing UIView altogether. It was lucky that I started to do this, since I found a subclass of UIView that (presumably) was not created using the Objective-C Runtime Library. This class was MTKView, which stands for Metal Kit View. You can create an instance of this class and it will have a CAMetalLayer attached by default. This saved my skin, as it made the Vulkan surface creation method happy. But there was one other problem, the view would only render the top left quadrant of the screen. I thought that it was back to the lab with this so I went back.

    

    I tried adding a sublayer to the default UIView and rendering with that, but that didn't work and made the problem worse, even though it did render. Finally I had to call for some help. I first got Ozzie in a call, to see if I was doing anything wrong with the Objective-C runtime, but it turns out I made the classes perfectly (somehow) and it really was Vulkan not recognizing the View. Then I called Lari, the Graphics man himself. He instantly recognized the issue as a viewport scaling issue. We got to work messing with the scaling options with the sublayer implementation. That ended up not working too well, so we went back to the MTKView implementation. Here is where we made a big discovery. iOS uses an odd method of getting pixel data. They use a width and height of pixels, but it is a little different with retina-enabled devices. They also use a "Scale Factor" property. This is different per-device, usually the larger the device, the larger the scaling factor. The iPhone 8 has a scaling factor of 2 while an iPhone 6s+ has a scaling factor of 3. Older phones like the iPhone 3 and 4 have a factor of 1. 

    With all of this information we finally came to the conclusion that the View and Layers all had the correct scaling and resolution applied. But there was one object that did not, the Vulkan Swapchain. I did not mention that neither Vulkan, nor Metal had any errors when rendering. This is because the Vulkan swapchain was half the resolution it should have been. By simply doubling the width and height of the swapchain, we fixed the top left corner rendering issue. The problem was actually in GWindow all along. It did not account for the scaling of retina devices. Once I resolved that mistake the Vulkan Surface worked like a charm. 

References:

UIKit

https://developer.apple.com/documentation/uikit?language=objc

MetalKit

https://developer.apple.com/documentation/metalkit/

MoltenVK

https://github.com/KhronosGroup/MoltenVK

UIView

https://developer.apple.com/documentation/uikit/uiview?language=objc

MTKView

https://developer.apple.com/documentation/metalkit/mtkview

NSView

https://developer.apple.com/documentation/appkit/nsview

CALayer

https://developer.apple.com/documentation/quartzcore/calayer?language=objc

CAMetalLayer

https://developer.apple.com/documentation/quartzcore/cametallayer

Duplicate Symbols Issue, By Ozzie
https://fsgateware.blogspot.com/2020/07/duplicate-symbols.html

Objective-C Runtime Library

https://developer.apple.com/documentation/objectivec/objective-c_runtime?language=objc

Vulkan on iOS part 2

 After being able to link Vulkan libraries in our Xcode iOS project, we are finally able to write Vulkan code for iOS. To use Vulkan with iOS, we need to give Vulkan the pixel buffer area for the screen. Usually this is done through something like HWND for DirectX, but with iOS we cannot get that low to a raw window pointer. So Vulkan asks for a UIView pointer, but not just any UIView. The one that you give to Vulkan must have a backing layer if CAMetalLayer or something that derives from CAMetalLayer. In MacOS this is an easy thing to achieve, just grab the NSWindow and say that it's backing layer is Metal, but in iOS, it gets a bit messy.

 iOS UIviews are layer-backed, meaning that they are given a layer upon creation and you absolutely cannot change it after (unless you write a custom View controller). This creates a conflict for Gateware. We want the user to be able to create a UIView themselves if they wish and still be able to use Vulkan, but we cannot change what they have specified. It is possible to run 2 controllers at the same time, or even just 2 views on the same controller. This solution has its drawbacks too. Since Gateware is usually used for games, this extra view or controller would be unecessarily using system resources. There must be a solution where the user can create a window but have Gateware put Vulkan into it.

 This is where I had an idea, if we cannot change the current view, can we change the current view controller? The answer was simple, as long as you present the view once you are done swapping them, the newly created view would be the one that is visible, and on top. So I had decided to create a Gateware View and View controller that had support for Metal. Next was just to detect whether or not the users View was Metal capable and overwrite it if not. This allows easy access for people wanting to develop Vulkan for their games and a level of control for people that already have a game looking to use Gateware.

  I am glad to say that the effort paid off. After fiddling around with some of the desktop Unit Tests, (and some incorrect file IO pathing) I was able to run them on iOS Simulator and an iPhone 7 running iOS 13. It feels nice to finally be done with Vulkan and to move on to Audio.

 References:

MoltenVK 

https://github.com/KhronosGroup/MoltenVK

UIKit UIViewController 

 https://developer.apple.com/documentation/uikit/uiviewcontroller

UIKit UIView 

 https://developer.apple.com/documentation/uikit/uiview

Metal CAMetalLayer 

https://developer.apple.com/documentation/quartzcore/cametallayer

Getting 3D Graphics on modern iOS without Metal

    Apple deprecated OpenGL in 2018 with iOS 12. Apple recommends using their graphics API, Metal. The issue with this is that not all cross-platform developers have the resources to port their game to Metal, instead choosing to use a portable API like OpenGL. Gateware likes to support as many platforms as possible, and although we plan to support Metal in the future, I don't have the time to design and implement a Metal Surface in the time I will be working on Gateware. What options does an iOS developer have for 3D Graphics? Well, there are 3 options, Metal, which is not viable at the moment, OpenGL, which isn't viable either, since I would have to have apps run exclusively on iOS 11.4 or earlier. The only option left is running Vulkan. If you didn't know, you absolutely can run Vulkan on iOS, now this isn't like other devices, as iOS doesn't officially support Vulkan like they support Metal. You can run Vulkan code through a translation layer. MoltenVK acts as this translation layer. MoltenVK takes Vulkan code and translates it to Metal code, so to the iOS device, it sees native Metal code.
    This solution of using Vulkan is good in my case, since we already have a Vulkan Library that I can port, and the process of porting is not all that complex since Vulkan was made to be portable. Most of the code should be able to be ported copy-and-paste style. This will save a lot of time since I only have a few more weeks of working on Gateware, and I would like to have 3D graphics for my presentation. Now getting MoltenVK onto a cross-platform project isn't as straight-forward as I would like, but it isn't too complex. We use Vulkan for macOS in much the same way so my first thought was to copy the process to link to MoltenVK on macOS to the iOS target, but this is unfortunately not possible. Since iOS doesn't have support for Vulkan, you cannot just place Vulkan code in an app and expect it to work. You need to also package MoltenVK in your app bundle for the Vulkan code to work properly. 

    The next step was to get MoltenVK for iOS. I had thought that you could reuse the macOS files for this, but that was incorrect. LunarG provides macOS files for MoltenVK, but I needed iOS files, so I went to the Github repository to build the library myself. This turned out to be a big waste of time. Although I did build the surprisingly easy to build the library, it takes quite a long time, around 10 hours on my old Macbook for both iOS and iOS Simulator. On top of long build times, it turns out that the files that LunarG provides actually contain the .dylib and .xcframework files for iOS and iOS Simulator. This revelation made me realize I spent 3 days building a library that I already had built 3 days earlier.
    Now that I had the files that I needed, I could get to properly getting MoltenVK into the iOS app bundle. This is where I am running into issues. For macOS, we are looking into the default location for libraries and linking to a .dylib file. Although possible with an iOS project, this isn't ideal since there is no easy way to ensure that the user's iOS device has MoltenVK installed. As such, I need to link another way. This is where the lovely documentation for MoltenVK shines. They have 2 ways to link that would work. I could link to the .dylib file and copy it to the app bundle and change 8 paths in build settings for that target, or I could copy the .xcframework file and change a single build setting. I opted for the .xcframework route for simplicity. From there I could run Vulkan code. 
    At the moment I am working on making a CMake script that automates this process for future developers on Gateware working on iOS. I am having a bit of trouble but that will be it for this post. 

References:

MoltenVK Github Repo

https://github.com/KhronosGroup/MoltenVK

LunarG MoltenVK Download

https://vulkan.lunarg.com/doc/sdk/1.1.130.0/mac/via.html

NSBundle or the C++ way?

 While developing for GFile, I found that you could use both C++ and NSBundle for accessing data from the app bundle for iOS. I wanted to know which way as better for the architecture we are using. As such I searched for which way was more widely used, as well as best practices for iOS, and finally, what would fit into how Gateware works.

The first thing I noted from this research was that Apple recommends that you use NSBundle for accessing files in the app bundle. Another thing going for using NSBundle is the ability to not need the file path to the file that you want, you can also find your file using it's filename, assuming that there are not multiple files with the same name. These benefits were appealing when deciding which FileIO system to use. There were also drawbacks to this approach. Namely a memory management one. NSBundle can dynamically allocate memory for you if it finds that it needs more memory for searching the bundle for your assets. Another drawbacks to consider, although it doesn't have to do with NSBundle itself, is that Gateware already has an interface that accepts filepathing as the main form of traversal through a file system, not searching.

There are other options to consider though. C and C++ style FileIO is also possible on iOS, albeit not recommended by Apple themselves. Because of this I could make minimal tweaks to the existing codebase and reuse most of the Mac implementation for iOS. While this is the most simple solution it comes with its own issues. Firstly Apple does not recommend using app bundles in this way. Next the C and C++ ways may not be available in future releases of iOS, as some employees have hinted at in developer forums. 

In the end I decided to go with the C++ way of FileIO for iOS. That way I could spend more time in other libraries, and so I wouldn't have to change something that mostly already worked. I had to change a setting in XCode for how it creates the bundle to avoid a crash using the C++ way, but other than that the code worked. This solution ended up being good for more than time. It also helped gain some precious mobile memory and kept Gateware away from using search-based FileIO and kept me from writing code based on whether or not a file was in the bundle.

Finally "Fixing" Suspend and Resume

     Since last time, I have ported over DX12 to UWP and the suspend/resume issue has been "resolved". The reason that is in quotation will be explained shortly.

The Problem:

    Turns out I found a slight bug/feature in Gateware's GEventReceiver interface. If a GEventGenerator is pushing events from multiple threads, events can be missed. That was the issue I was receiving for my suspend/resume. My rendering loop was not resuming because I was not getting the event to my receiver as GWindow in UWP pushes from 2 different threads. 

The Solution:

    The solution was to just use GEventQueue instead of GEventReceiver. With a queue, there are no missed events. The queue is popped from in a while loop until the queue is empty. While this is a fix for my problem, this uncovered a slight issue with GEventReceiver which Lari is currently looking into fixing. As of writing this, GEventReceiver only stores one message at a time and in non-blocking, which is safe from a threading perspective but not ideal if a user is expecting every message to go through, like I was.

    Attached gif is me suspending and resuming my demo scene, I am using an event queue to capture events from GWindow to know when I am suspended or not. I use this to flip a bool to stop the rendering loop.

     

BlueScreen of Death

 Not that blue screen. 

BlueScreen is one of the templates that we maintain for users of Gateware to learn from or build off of. It's a cross platform implementation of an OpenGL renderer. It uses Gateware libraries to create a window, and then render a colored triangle inside it. It runs on Linux and Windows. It's also supposed to change color when you resize the window. For some reason, on Linux only, the resize event wasn't being received properly and so the lambda passed into the create function never gets hit.

I found this issue when updating and testing the 1.2 release candidate on all the templates on all platforms and so I was assigned to fix it. I spent most of last Thursday, Friday, Monday, and Tuesday only trying to solve this problem. 

First of all, the main hurdle was trying to debug using Codelite on Linux. For some reason, if you build the project in CMake, you'll have no debugging unless you specify that you want it. To solve this issue, I had to copy over the the LinuxSetup script from the devops folder in the main Gateware dev repo and modify it to work in a different directory.

Once that was sorted, I got to work. I set breakpoints everywhere I could think of that might be useful. No luck. I put assert statements everywhere I could think of. No luck. I added print statements for debug information. No luck. 

Countless hours of research into X window and X11 on Linux, with nothing to show for it. Wednesday comes around and we have a release team meeting (that I missed, unfortunately). Ozzie and Lari worked on it and Lari came up with a very clever way to debug the problem. He put print statement in the lambda, then put a return at the beginning of the Create() function, rand the program, then moved the return down a few lines and tested again. They knew they found the line causing the problem once it stopped printing from the lambda.

Apparently, the issue was in X11. OpenGL was overriding settings that GWindow originally sets up and some of those settings had to do with whether the event got received. 

Definitely not an obvious fix.

The iOS App Bundle

 Last week I have been tasked with porting GFile to iOS. As part of file IO, GFile needs to be able to get the content out of the App Bundle that ships with the app executable. The end user would place all of their assets into this Bundle and it would be accessable for reading only from that Bundle when the app is released on the app store.

I ran into a bit of an issue with this, I did not know how to get GFile to read the information from the App Bundle. So I read up on Bundle documentation and Apple suggests using the NSBundle class to grab the info. At first I had thought to use the Apple recommended method, so I looked into using some kind of search method to see if the current directory was in the bundle itself. After seeing this was a difficult task, I looked into using C++ functions to read the data. I found that you can use plain C functions to read the data from the bundle, assuming that you can get the path to the bundle. I tested this with the C++ equivalent code, and it worked nearly-perfectly. I had an exception thrown when you tried to access a file that did not exist in the bundle due to a change made in iOS 13. 

I then researched on this change in iOS 13 and it came down to a setting in XCode that was not set in CMake which would usually be set in a new iOS project. I set the setting and the code works like intended. I will be moving onto GLog, which should be a fairly simple library to port, and then onto GWindow in hopes to get some graphics running in the future.