Getting 3D Graphics on modern iOS without Metal

    Apple deprecated OpenGL in 2018 with iOS 12. Apple recommends using their graphics API, Metal. The issue with this is that not all cross-platform developers have the resources to port their game to Metal, instead choosing to use a portable API like OpenGL. Gateware likes to support as many platforms as possible, and although we plan to support Metal in the future, I don't have the time to design and implement a Metal Surface in the time I will be working on Gateware. What options does an iOS developer have for 3D Graphics? Well, there are 3 options, Metal, which is not viable at the moment, OpenGL, which isn't viable either, since I would have to have apps run exclusively on iOS 11.4 or earlier. The only option left is running Vulkan. If you didn't know, you absolutely can run Vulkan on iOS, now this isn't like other devices, as iOS doesn't officially support Vulkan like they support Metal. You can run Vulkan code through a translation layer. MoltenVK acts as this translation layer. MoltenVK takes Vulkan code and translates it to Metal code, so to the iOS device, it sees native Metal code.
    This solution of using Vulkan is good in my case, since we already have a Vulkan Library that I can port, and the process of porting is not all that complex since Vulkan was made to be portable. Most of the code should be able to be ported copy-and-paste style. This will save a lot of time since I only have a few more weeks of working on Gateware, and I would like to have 3D graphics for my presentation. Now getting MoltenVK onto a cross-platform project isn't as straight-forward as I would like, but it isn't too complex. We use Vulkan for macOS in much the same way so my first thought was to copy the process to link to MoltenVK on macOS to the iOS target, but this is unfortunately not possible. Since iOS doesn't have support for Vulkan, you cannot just place Vulkan code in an app and expect it to work. You need to also package MoltenVK in your app bundle for the Vulkan code to work properly. 

    The next step was to get MoltenVK for iOS. I had thought that you could reuse the macOS files for this, but that was incorrect. LunarG provides macOS files for MoltenVK, but I needed iOS files, so I went to the Github repository to build the library myself. This turned out to be a big waste of time. Although I did build the surprisingly easy to build the library, it takes quite a long time, around 10 hours on my old Macbook for both iOS and iOS Simulator. On top of long build times, it turns out that the files that LunarG provides actually contain the .dylib and .xcframework files for iOS and iOS Simulator. This revelation made me realize I spent 3 days building a library that I already had built 3 days earlier.
    Now that I had the files that I needed, I could get to properly getting MoltenVK into the iOS app bundle. This is where I am running into issues. For macOS, we are looking into the default location for libraries and linking to a .dylib file. Although possible with an iOS project, this isn't ideal since there is no easy way to ensure that the user's iOS device has MoltenVK installed. As such, I need to link another way. This is where the lovely documentation for MoltenVK shines. They have 2 ways to link that would work. I could link to the .dylib file and copy it to the app bundle and change 8 paths in build settings for that target, or I could copy the .xcframework file and change a single build setting. I opted for the .xcframework route for simplicity. From there I could run Vulkan code. 
    At the moment I am working on making a CMake script that automates this process for future developers on Gateware working on iOS. I am having a bit of trouble but that will be it for this post. 

References:

MoltenVK Github Repo

https://github.com/KhronosGroup/MoltenVK

LunarG MoltenVK Download

https://vulkan.lunarg.com/doc/sdk/1.1.130.0/mac/via.html

NSBundle or the C++ way?

 While developing for GFile, I found that you could use both C++ and NSBundle for accessing data from the app bundle for iOS. I wanted to know which way as better for the architecture we are using. As such I searched for which way was more widely used, as well as best practices for iOS, and finally, what would fit into how Gateware works.

The first thing I noted from this research was that Apple recommends that you use NSBundle for accessing files in the app bundle. Another thing going for using NSBundle is the ability to not need the file path to the file that you want, you can also find your file using it's filename, assuming that there are not multiple files with the same name. These benefits were appealing when deciding which FileIO system to use. There were also drawbacks to this approach. Namely a memory management one. NSBundle can dynamically allocate memory for you if it finds that it needs more memory for searching the bundle for your assets. Another drawbacks to consider, although it doesn't have to do with NSBundle itself, is that Gateware already has an interface that accepts filepathing as the main form of traversal through a file system, not searching.

There are other options to consider though. C and C++ style FileIO is also possible on iOS, albeit not recommended by Apple themselves. Because of this I could make minimal tweaks to the existing codebase and reuse most of the Mac implementation for iOS. While this is the most simple solution it comes with its own issues. Firstly Apple does not recommend using app bundles in this way. Next the C and C++ ways may not be available in future releases of iOS, as some employees have hinted at in developer forums. 

In the end I decided to go with the C++ way of FileIO for iOS. That way I could spend more time in other libraries, and so I wouldn't have to change something that mostly already worked. I had to change a setting in XCode for how it creates the bundle to avoid a crash using the C++ way, but other than that the code worked. This solution ended up being good for more than time. It also helped gain some precious mobile memory and kept Gateware away from using search-based FileIO and kept me from writing code based on whether or not a file was in the bundle.

Finally "Fixing" Suspend and Resume

     Since last time, I have ported over DX12 to UWP and the suspend/resume issue has been "resolved". The reason that is in quotation will be explained shortly.

The Problem:

    Turns out I found a slight bug/feature in Gateware's GEventReceiver interface. If a GEventGenerator is pushing events from multiple threads, events can be missed. That was the issue I was receiving for my suspend/resume. My rendering loop was not resuming because I was not getting the event to my receiver as GWindow in UWP pushes from 2 different threads. 

The Solution:

    The solution was to just use GEventQueue instead of GEventReceiver. With a queue, there are no missed events. The queue is popped from in a while loop until the queue is empty. While this is a fix for my problem, this uncovered a slight issue with GEventReceiver which Lari is currently looking into fixing. As of writing this, GEventReceiver only stores one message at a time and in non-blocking, which is safe from a threading perspective but not ideal if a user is expecting every message to go through, like I was.

    Attached gif is me suspending and resuming my demo scene, I am using an event queue to capture events from GWindow to know when I am suspended or not. I use this to flip a bool to stop the rendering loop.

     

BlueScreen of Death

 Not that blue screen. 

BlueScreen is one of the templates that we maintain for users of Gateware to learn from or build off of. It's a cross platform implementation of an OpenGL renderer. It uses Gateware libraries to create a window, and then render a colored triangle inside it. It runs on Linux and Windows. It's also supposed to change color when you resize the window. For some reason, on Linux only, the resize event wasn't being received properly and so the lambda passed into the create function never gets hit.

I found this issue when updating and testing the 1.2 release candidate on all the templates on all platforms and so I was assigned to fix it. I spent most of last Thursday, Friday, Monday, and Tuesday only trying to solve this problem. 

First of all, the main hurdle was trying to debug using Codelite on Linux. For some reason, if you build the project in CMake, you'll have no debugging unless you specify that you want it. To solve this issue, I had to copy over the the LinuxSetup script from the devops folder in the main Gateware dev repo and modify it to work in a different directory.

Once that was sorted, I got to work. I set breakpoints everywhere I could think of that might be useful. No luck. I put assert statements everywhere I could think of. No luck. I added print statements for debug information. No luck. 

Countless hours of research into X window and X11 on Linux, with nothing to show for it. Wednesday comes around and we have a release team meeting (that I missed, unfortunately). Ozzie and Lari worked on it and Lari came up with a very clever way to debug the problem. He put print statement in the lambda, then put a return at the beginning of the Create() function, rand the program, then moved the return down a few lines and tested again. They knew they found the line causing the problem once it stopped printing from the lambda.

Apparently, the issue was in X11. OpenGL was overriding settings that GWindow originally sets up and some of those settings had to do with whether the event got received. 

Definitely not an obvious fix.

The iOS App Bundle

 Last week I have been tasked with porting GFile to iOS. As part of file IO, GFile needs to be able to get the content out of the App Bundle that ships with the app executable. The end user would place all of their assets into this Bundle and it would be accessable for reading only from that Bundle when the app is released on the app store.

I ran into a bit of an issue with this, I did not know how to get GFile to read the information from the App Bundle. So I read up on Bundle documentation and Apple suggests using the NSBundle class to grab the info. At first I had thought to use the Apple recommended method, so I looked into using some kind of search method to see if the current directory was in the bundle itself. After seeing this was a difficult task, I looked into using C++ functions to read the data. I found that you can use plain C functions to read the data from the bundle, assuming that you can get the path to the bundle. I tested this with the C++ equivalent code, and it worked nearly-perfectly. I had an exception thrown when you tried to access a file that did not exist in the bundle due to a change made in iOS 13. 

I then researched on this change in iOS 13 and it came down to a setting in XCode that was not set in CMake which would usually be set in a new iOS project. I set the setting and the code works like intended. I will be moving onto GLog, which should be a fairly simple library to port, and then onto GWindow in hopes to get some graphics running in the future.

 

 

GController Woes

    Since last time, I have finished the DirectX 11 port. It works great, and I even ported over a previous Gateware project I had from a few months ago with very little effort. I had to get rid of all the input detection I had as it was using functions that just aren't available on UWP. I honestly surprised myself with how quickly it came together. All the old Gateware calls I originally wrote 6-7 months ago for a different platform worked on UWP thanks to the work I have done thus far. With that out of the way, on to GController I went.

Problem:

    The initial plan was to just get the Xinput library implemented for Win32 to work. From the research that was done, it should have been possible to use that code as is and just have it work. It would compile just fine, but it would not sense any input being sent in. As hard as I tried it just would not work for me. Time is running short before graduation so I can't allow myself to get stuck on any one thing for too long. I still have a minor issue with suspend and resume I need to address.

Solution:

    The solution to this problem turned out to just be rewriting a good portion of the library using WinRT APIs. This took two days to get working fully, my whole weekend. But now that I have Xbox One controllers working on UWP, getting general controller support will very easy. Getting general controller support on UWP working could also bring it to the Win32 platform as the APIs can also be used there. That would allow Win32 to finally have PS4 controller support.

To Do:

• Fix the Suspend/Resume issue.
• Get general controller support.
• Merge development branch into App development branch to get up to date fixes for libraries.
• DirectX 12

I Broke Vulkan

 So now that I've finished cleaning up all the warnings on Windows, Linux, and Mac, we're getting pretty close to a new version release. Part of that process is merging branches back into the main development branch. I merged my warnings branch back in and set to work testing and updating the templates to the new version.

The first project I opened was RedScreen, our Vulkan template. It built and rand fine but then if you tried to resize the window or maximize it, this happened:

Good ol' Vulkan puking up errors that nobody can understand. 

Basically, what it's saying is that the swap chain needs to be recreated from scratch every time you resize the frame buffer (the window), but it wasn't happening.

Testing with the previous version of Gateware.h showed that it worked fine there, so clearly somebody screwed things up recently. Lari initially thought he broke it with a fix that he added earlier in the day, but I tested the commit right before his, and the problem persisted. 

So now I'm searching through commit after commit, trying to find all the instances where the Vulkan core is modified. And lo and behold!

Culprit located. In my very first commit to the gateware repo, I found this line with an unused return value. I seem to have brain farted, because instead of just removing the assignment, I removed the entire line. We need GetSurfaceData to actually set the proper variables when the window/surface is resized. This problem was there for over a month, it just wasn't caught until now because we don't have unit tests that test if the Vulkan surface is successfully re-sizable. 

So I added GetSurfaceData back in, tested RedScreen again and...

All is well in the Vulkan template again.

Apple's Uncanny Entry Point

     The week before last I had encountered a problem, one that I believe very few developers, if any, have tried to solve. To understand the problem we must first understand the goal. First Gateware code is meant to be compatible with 3 (currently) platforms, Windows, Linux, and macOS. Over the last few months, Chase and I have been porting Gateware to less desktop based environments, Windows UWP for him, and iOS for me. There is a subtle difference between desktop and app ecosystems that most developers don't think of off the top of their heads; entry points. Entry points serve as the beginning of any program, and most (if not all) programs need an entry point to start out. Windows (Win32), Linux, and macOS all use some form of

 int main(int argc, char** argv); 

for an entry point, but things get a little funky with app based programs.

    UWP and, if you configure it correctly,  Win32 take a different function as an entry point

 int WINAPI wWinMain(...);  

and while this may be convenient for developers of those platforms to get the extra information from the parameters of this function, it makes the life of a cross-platform developer much more difficult. To account for this change in entry, we made GApp, a class that would define whatever entry point was required on a platform, and call the usual main that the user would write. This keeps the code simple as the users of Gateware would only have to write one main for every platform. For UWP this was pretty simple, you take main as a parameter to a global constructor and run it from there, making sure to define WinMain at the same time. For iOS, this is a different story.  

    iOS does traditional main, so there's no issue right? Well there is a caveat with that. To get the storyboard to render, one must call the

 int UIApplicationMain(int argc, char * _Nullable *argv, NSString *principalClassName, NSString *delegateClassName);  

function. Well then just call this function in GApp and we're good right? Nope. Not only must this function be called on the main operating thread, it must also be called in main, or in any function that main calls (assuming that it isn't on another thread). Well, this is an issue, how can we call this function and run the code that the user supplies? You cannot multithread this function on the same thread, as it overrides regular main and as such it will never run the regular main code. And you cannot dispatch this function, as dispatch will favor UIApplicationMain over traditional main and traditional main will be put on the back burner.

    So is that it? Is there any way to have main run on the same thread put asynchronous to UIApplicationMain? I do not know. But there is an unexpected solution. With UWP, GApp defines the main entry point and calls the user-defined main later, so why can't we do the same with iOS? So we defined the traditional main entry point, and later in the file, did this

 #define main Gateware_main  

and it worked. As with the UWP GApp, the iOS GApp constructor takes the main function as a parameter and calls it later, but this time the main function that the user writes is not called, as it is now recognized by the compiler as "Gateware_main" and as such doesn't run it. Instead the compiler will see the GApp main as the entry point and run that. So we can have both functions. The GApp main handles everything for the iOS side, and then later calls the users main so that their functionality still works. This works as UIApplicationMain does require to be on the main thread and called by main, so that condition is checked, and there are none for the users main, other than it is expected to be the start of the program, so we run it as soon as the app is done launching.

     

3 Months with an Undiscovered Bug

     Since my last blog post, GWindow has finally been finished. Well maybe not finished, but it is in a very good place so that I can start porting other important libraries so someone could begin to make a game for Xbox. This was a decision that had to be made. I can't spend the rest of my last month on this project getting one library to 100% when other ones are still sitting 0. With GWindow mostly out of the way, I began to work on the DirectX 11 port. This is where I found the bug that has been there for 3 months and I only now discovered it.

The Problem:

    When I went to use the macro IID_PPV_ARGS, I kept receiving the error that it was undefined, when I know for a fact that it is. This led me down the rabbit hole of trying to figure out why this was the way it was. After a little research and cross referencing other C++/WinRT projects, I came to find out that C++/WinRT overrides a function that the macro is representing. This override is in base.h, which is generated at runtime with  C++/WinRT. Turns out that this whole time, my UWP Gateware project was never generating any of the header files it needs for the language projection from C++ to WinRT APIs. How was it even running then you ask, well it was using the WinRT header files included in the Windows 10 SDK, which are from the first iteration of C++/WinRT and that base.h did not have that override function I needed for the macro.

The Solution:

    Lari had the idea to check the the .sln and .vcxproj files to see if anything in those differ from what was in the projects that I know work. Eureka! There was a discrepancy in the .vcxproj file. It did not include references to C++/WinRT while the working projects did. I needed to come up with a way to get those in there automatically without having me and future devs go in there manually and fix this. What I ended up doing was creating a PowerShell script that takes in the file after CMake compiles it and edits it. I tried my best to write it in a way that if more files are put into the project that my script will not need to be rewritten. Here is that script: 

     This script adds back in the references to the .vcxproj file and allows the project to generate the the header files needed. In the end, the macro worked again and I can now continue my DirectX 11 port. Speaking of the DirectX port it is coming along great so far, minus a few threading issues. I'll save that for next week when its done.