Month 2-Week 1 of Gateware: Finally Moving on to UWP

    Mac is finally behind me (for now), and UWP and Windows lie ahead! The first task that stood before me was bringing the UWP branch that Chase Richards and Lam Truong had worked on and merging the main branch into it to begin the updating process. This, of course, led to several conflicts that needed to be sorted out. Luckily, TotoiseGit has a really nice GUI that's good for sorting out conflicts and letting you edit the files outside of the conflicting areas as well. So after getting all of those conflicts sorted out, the Win32 version, of course, still wouldn't compile. But it turned out that all that was needed was changing some file pathing since UWP required some different folder structuring. And within the first day and a half, I had the Desktop version of Gateware compiling and passing all of the tests.

    But the Win32 version is only half the battle; now it was on to the UWP. And surprisingly, it wasn't that bad. The only thing that really gave issues was the UWP implementation of GFile, but to fix that, I really just had to copy-paste some code from the Win32 implementation of GFile. So I was actually able to get the UWP compiling and passing all the tests in the same day I was able to finish the Win32 implementation. Man, only two days in, and I'm flying through this! This UWP stuff is gonna be a breeze!

    That's what I thought at the end of the second day; now, here I am at the end of day 5, and, on the surface, nothing has happened since then. First, allow me to set the stage as to what is halting progress.

    As I mentioned before, UWP requires some very different folder structuring since it needs one solution for Win32 and another for the UWP. Now for a human being with an ever-changing brain, this is no problem. But for a machine that is running off of a specific set of commands, looking for a specific set of folders and files, this can be a huge issue. And it just so happens that the runners that we use to test the different implementations of Gateware are that kind of machine. So, until we can find a way to get those runners to recognize this different folder structure, the UWP branch cannot pass the pipeline tests and, therefore, cannot be merged into the main branch.

    But this is only the first problem. The second problem has to do with the Windows SDK versions (just as a heads up, I am in Windows 11 throughout this entire process). So to develop a UWP application, you need to download the UWP development tool through Visual Studio. When you install the UWP tool, it also installs Windows 11 SDK 10.0.22000.0, which is an SDK for Windows 11. If you try to delete this SDK, it will also delete the UWP tool, no matter what. So there is no way to have the UWP tool, without this version of the Windows SDK. Also, an additional piece of information to keep in mind; you cannot use a Windows 11 SDK while running Windows 10.

    Well this is all cool and all, but what does this have to do with Gateware? Well, when CMake is run, it has to create a UWP project, and a UWP project requires a Windows SDK version. But CMake can't just give it a random version, so it finds the latest version that you have installed and uses that in the project settings. And now, you suddenly have a UWP project that is targeting Windows 11, even if you're on Windows 10. Now, if you're a human being with an ever-changing brain, you can simply just go into the project properties and change the platform version to a Windows 10 SDK that you have installed, granted that would get very annoying since you'd have to do that every time you ran CMake. But, once again, our runners are not humans being with an ever-changing brain and can't just change the platform version as they wish.

    So Lari and I began trying to find a way to get CMake to find the Windows SDK version you had installed that was both older than the current OS version you were running and the newest out of that whole bunch. But after two whole days, we could not find a strong and reliable way to do it without creating more work for the user. So we came to the final solution. Just use Windows 11. While it will still be possible to use Gateware UWP on Windows 10, there will be hoops that will be needed to be jumped through, so it will be recommended to just use Windows 11. As I am typing this right now, I am using a freshly installed Windows 11, and we are currently looking into updating the Windows runner to Windows 11. 

Week 4 of Gateware: End of the First Month

     Well, here we are at the end of the month. Much learned, but still far more to learn. The problem we were having with the MacOS crashing last week was that the CMake code we were using to link the Vulkan SDK was outdated and since then, CMake has built-in support for linking with newer Vulkan SDK's, so we just needed to update that bit of code to get it working. So after that was done, it was finally compiling, but it was now crashing on the VulkanSurface tests.

    So in the newer Vukan SDK's for MacOS, they force the extension "VK_KHR_PORTABIITY_subset" to be enabled. This extension allows for applications to exist on non-conformant platforms and since MoltenVK isn't fully conformant, this extension needs to be enabled. And just for those who don't know (like me up until two weeks ago), MoltenVK is needed so that Vulkan can can map to Apple's Metal graphics framework. To opt-in to this extension, you need to modify the VkInstanceCreateInfo by adding the VK_INSTANCE_CREATE_ENUMERATE_PROTABILITY_BIT_KHR bit to the flag parameters and add the VK_KHR_PORTABILITY_ENUMERATE_EXTENSION_NAME extension to the ppEnabledExtensionNames array along with incrementing the enabledExtensionCount variable. Links and a screenshot of this information will be posted below.
Link: https://vulkan.lunarg.com/doc/sdk/1.3.243.0/mac/getting_started.html

While this is a very simple thing to implement in an ordinary application, Gateware is no ordinary application and took a bit more finessing. 

    GVulkanSurface uses a file called GVulkanSurface_core to implement the majority of the VulkanSurface implementation and thus makes the OS-specific files much smaller. But since our VkInstanceCreateInfo is in the core file, we can't just make all those changes there since Linux and Windows are comformant platforms and would cause another crashing error. So instead, after the original VkInstanceCreateInfo is created and setup, we have a new function called "PlatformOverrideInstanceCreate," which allows us to pass in the VkInstanceCreateInfo, along with some of its original variables to be cleaned up later, and then modify them all in the function to opt-in to the extension. We then of course had to include dummy versions of this function into the Linux and Windows version of VulkanSurface so that the function can still be called on other Operating Sytems.

    Since this extension only exists on newer versions of Vulkan, we aren't able to get Gateware to compile on older SDK's with the new Mac implementation. With this said, we will be dropping support for Vulkan SDK's on MacOS older than 1.3.216.

    Since we have been doing a lot of updating on the Mac side of things, we are also trying to update our Mac testing machines to the latest MacOS that they can support, which is Monterey. However, this has caused a previously discovered bug to pop up again. GWindow causes crashing on Monterey (and likely other newer MacOS's as well). While we aren't entirely certain what the problem is, we believe it has something to do with multi-threading. I wasn't really able to discover anything new with this issue, just tried to understand what was already found and made. I would like to return and try to fix this problem later on before I graduate, but we'll see when we get there. For now, this is the link to the issue in GitLab. https://gitlab.com/gateware-development/gsource/-/issues/229.

Week 3 of Gateware: MacOS

     For the past 5 days, I have been dealing with the donor MacBook. Originally, I was using it to fix a compiler issue involving GController unit tests, but in the process of doing so, I found another issue where GSource doesn't seem to compile using the latest Vulkan SDK version. After some research, we believe that the issue is that the OS on the donor MacBook is simply too outdated and we are hoping that updating the OS will solve the issue. So I just had to update the OS and the problem should be fixed. Simple, right? Of course not. Since the MacBook also had Linux and Windows operating systems on it, there wasn't enough space to install the update. So I had to factory reset the Mac and re-install the OS, but for whatever reason, it couldn't download the installer and install the OS. So I had to get access to the second donor MacBook, download a new MacOS (Monterey), transfer it to a USB drive, convert that USB drive to a bootable installer, plug it into the original MacBook, and then I can finally install a newer OS. And I am happy to say that it did successfully install. ☺

    Now with the new OS installed, I can start re-installing all the necessary software to compile and run GSource and everything was moving smoothly until I got to XCode. To use XCode, you need to sign into your Apple ID account, which we have. However, two-factor authentication has been set up for the account and we currently have no way around it. I'm hoping to soon find who set up the two-factor authentication so we can get rid of it, but if we can't find that person, things might get a little more complicated.

    It might be asking too much, but I'm hoping that once we've got this Vulkan compiler issues sorted out, I'll be able to figure out the other bug fixes fairly quickly so that I can get started on the UWP tasks and just have to worry about the Windows development environment. But as everything in the development world, that probably won't be the case.

Week 2 of Gateware: Research

    My big plan for Gateware or the next several months that I was going to be working on was sort of a two-parter. Firstly, I was going to complete the work of previous developers to get Gateware running as a UWP application so that it could run on additional Microsoft devices such as the Xbox, Windows Mixed Reality Headset, Hololens, and so on. Secondly, I wanted to convert the UWP code over to WinUI 3 to future-proof Gateware, as it seemed that Microsoft was dropping support for UWP and replacing it with WinUI 3. The problems arose with this second part, as the previous statement only seems to be partly true.

     From what I can tell, when people are talking about "UWP" in various articles and websites, they seem to be talking about two different things. The first thing they could be talking about is the UWP application framework. This is the overall structure of a project that makes a UWP app a UWP app. The second thing they could be talking about is UWP/XAML. This is a library/language used to create UI elements inside of a UWP application. Up until WinUI 3 came out, UWP/XAML was the only way to create UI elements in UWP apps, and that was really the only use that UWP/XAML had. 

    But now, with the introduction of WinUI 3, Microsoft seems to be trying to replace UWP/XAML (not the UWP application framework) with WinUI 3. But since people have always just used "UWP" to refer to both parts of the UWP ecosystem, this now seems to have created some confusion. Or at least it has really confused me. 

    The main thing to be learned from this is that WinUI 3 only brings about new libraries and API's for creating UI elements, which is something that Gateware does not have/need any time soon. Since Gateware uses graphics API's to draw to the screen, Gateware users can just create UI elements using those API's. Because of this, WinUI 3 has to use being in the current Gateware. I can see it possibly having some use cases in the future, as Gateware has more features added, but until then, there is not really anything to be done.

    Next Wednesday (April 19, 2023), Microsoft will be holding their monthly WinUI 3 Community Call where they has a QnA towards the end. I plan on attending this Community call to try and learn more about WinUI 3 and UWP, just to confirm that WinUI 3 holds no prospects for Gateware in present time.

    In the meantime, I will be working on various fixes that involves Linux and MacOS to not just get more Gateware experience, but to also get some experience in working with those other OS's. I do still plan on completing the UWP work to get Gateware running on Xbox, but that will be after I have confirmed that there is nothing to be done concerning WinUI 3.

Colby Peck - Gateware Week 18: Wrapping Up

The pipeline document is finished! It took quite a bit longer than I had hoped it would, but looking at it now, it’s a 27-page in-depth user’s manual for a CI/CD pipeline that spans four repos. I probably should have known it would take longer… In any case, any future CI/CD developers that join the Gateware team will have a thorough and useful reference guide on how the pipeline works.

That wraps up my penultimate week as a student member of the Gateware team. Next week is probably going to be spent primarily on writing up my postmortem. Right when you think you’ve escaped the documentation, there’s more to be had! If I manage some spare time in next week, I’ll probably be cracking open my mac to try and see if I can’t make any more progress on the ghost window issue.

Colby Peck - Gateware Week 17: Documentating!

The biggest remaining task for me is writing up a document that explains the pipeline in detail - most pertinently, how and why we’ve used the solutions we have. The short answer is mostly along the lines of “Well, there is an easier way to do it, but it’s a gitlab premium feature.” We ended up using tokens, masked pipeline variables, and curl requests instead of pipeline secrets and multi-project pipelines because the latter two are only available to gitlab premium. 

There are a few sticking points in the pipeline’s design - it necessarily makes some assumptions about the names of certain folders and branches in Gateware’s various repos, for example. If we ever rename the default branch of GCompiler or GTemplates from ‘master’ to the newer standard ‘main,’ the pipeline will break if it isn’t updated. And it may not break in an obvious way.

Even for the things that are more common and banal as pipelines come, it’s good to have detailed documentation available for reference. I’ve got two more weeks on this project, and I want to make sure that I leave it such that it’s easy for future developers to pick up where I’ve left off. I’m very satisfied with the pipeline that I’ve made, but if the requirements change in the future and the pipeline can’t change to meet the new requirements, then I haven’t done my job right.

I’m not going to say that I’m happy with the rate of progress on writing this document - I’m not. I’ve mostly finished the references for GSource and GCompiler’s pipelines, but I still need to write up some sections describing Gateware’s pipeline, GTemplates’ pipeline, the single-header compiler, and our handling of tokens. I was honestly hoping to have the document done by now, but writing is very difficult for me to focus on, even for relatively short periods of time. And I’ve had some non-gateware responsibilities come and bite me from behind this week (turns out there’s some paperwork and administrata you’re supposed to do when you’re near graduation - who'd've thunk?) On the bright side, the sections that are finished are highly detailed and (hopefully) easily navigable. If you’re working on the pipeline in the future and you come to a part that you don’t understand, it should be easy to find the section of this document that covers that part.

Colby Peck - Gateware Week 16: GTemplates Gets a Pipeline Too!

You get a pipeline! And you get a pipeline! You ALL get a pipeline!!!

My big task this week was to set up a pipeline for GTemplates. If you aren’t aware, GTemplates is a repo that contains a few template projects that use Gateware to make a few different windows. We have one for each of our graphics APIs (DirectX11, DirectX12, OpenGL, Vulkan, and GBlitter for 2D), as well as one for a console application. The pipeline’s job is simple: whenever a new version of Gateware is released, grab that version of Gateware and compile all of the templates with it. 

The first step of this task was to make a script that finds and makes all of the cmake projects in the repo. Well, to make three scripts - one for windows, one for mac, and one for linux. That part was actually not so bad. I got the scripts made and working with little issue. They aren’t set up to work with cmake projects that contain subdirectories (that is, subdirectories that contain their own cmakelist.txt files that are included in a root cmakelists.txt), but none of the templates have those, so it’s not a problem for now. It’s something I’ll be fixing soon, hopefully.

After that, I just needed to make a compile job for each of the templates. Well, a job for each template for each platform it’s supposed to run on. Well, two for each template for each platform - one debug, and one release.

The GTemplates Pipline

As you can see, it actually added up to quite a few. Fortunately, the jobs are all similar enough that I was able to copy/paste them with only minor changes between them:

The Windows compile job for BlueScreen

The Windows compile job for GreenScreen

And, as always, when the pipeline was added it shook out a few bugs. Some of the templates were failing to compile on linux because Gateware was expecting C++ 17 but not getting it. I fixed that by adding it to their respective cmake files (they all got a “set CMAKE_CXX_STANDARD 17” and a “set CMAKE_STANDARD_REQUIRED ON”), and that fixed the issue. Apparently, the source code is supposed to be checking for that already, but clearly isn’t (at least not on linux). More work for the future!

Aside from the two development tasks I’ve mentioned, I’m also working on writing fairly extensive documentation on the gitlab pipeline. It’s a bit of a monster! It’s responsible for testing, compiling, re-testing, and releasing Gateware automatically (as well as testing the templates now), and it’s spread across four repos. I’m doing what I can to try and make future CI/CD work on Gateware easier, but writing documentation (and blog posts) is actually fairly difficult for me to focus on! I often find it easier to read and write code than english…