Colby Peck - Gateware Week 18: Wrapping Up

The pipeline document is finished! It took quite a bit longer than I had hoped it would, but looking at it now, it’s a 27-page in-depth user’s manual for a CI/CD pipeline that spans four repos. I probably should have known it would take longer… In any case, any future CI/CD developers that join the Gateware team will have a thorough and useful reference guide on how the pipeline works.

That wraps up my penultimate week as a student member of the Gateware team. Next week is probably going to be spent primarily on writing up my postmortem. Right when you think you’ve escaped the documentation, there’s more to be had! If I manage some spare time in next week, I’ll probably be cracking open my mac to try and see if I can’t make any more progress on the ghost window issue.

Colby Peck - Gateware Week 17: Documentating!

The biggest remaining task for me is writing up a document that explains the pipeline in detail - most pertinently, how and why we’ve used the solutions we have. The short answer is mostly along the lines of “Well, there is an easier way to do it, but it’s a gitlab premium feature.” We ended up using tokens, masked pipeline variables, and curl requests instead of pipeline secrets and multi-project pipelines because the latter two are only available to gitlab premium. 

There are a few sticking points in the pipeline’s design - it necessarily makes some assumptions about the names of certain folders and branches in Gateware’s various repos, for example. If we ever rename the default branch of GCompiler or GTemplates from ‘master’ to the newer standard ‘main,’ the pipeline will break if it isn’t updated. And it may not break in an obvious way.

Even for the things that are more common and banal as pipelines come, it’s good to have detailed documentation available for reference. I’ve got two more weeks on this project, and I want to make sure that I leave it such that it’s easy for future developers to pick up where I’ve left off. I’m very satisfied with the pipeline that I’ve made, but if the requirements change in the future and the pipeline can’t change to meet the new requirements, then I haven’t done my job right.

I’m not going to say that I’m happy with the rate of progress on writing this document - I’m not. I’ve mostly finished the references for GSource and GCompiler’s pipelines, but I still need to write up some sections describing Gateware’s pipeline, GTemplates’ pipeline, the single-header compiler, and our handling of tokens. I was honestly hoping to have the document done by now, but writing is very difficult for me to focus on, even for relatively short periods of time. And I’ve had some non-gateware responsibilities come and bite me from behind this week (turns out there’s some paperwork and administrata you’re supposed to do when you’re near graduation - who'd've thunk?) On the bright side, the sections that are finished are highly detailed and (hopefully) easily navigable. If you’re working on the pipeline in the future and you come to a part that you don’t understand, it should be easy to find the section of this document that covers that part.

Colby Peck - Gateware Week 16: GTemplates Gets a Pipeline Too!

You get a pipeline! And you get a pipeline! You ALL get a pipeline!!!

My big task this week was to set up a pipeline for GTemplates. If you aren’t aware, GTemplates is a repo that contains a few template projects that use Gateware to make a few different windows. We have one for each of our graphics APIs (DirectX11, DirectX12, OpenGL, Vulkan, and GBlitter for 2D), as well as one for a console application. The pipeline’s job is simple: whenever a new version of Gateware is released, grab that version of Gateware and compile all of the templates with it. 

The first step of this task was to make a script that finds and makes all of the cmake projects in the repo. Well, to make three scripts - one for windows, one for mac, and one for linux. That part was actually not so bad. I got the scripts made and working with little issue. They aren’t set up to work with cmake projects that contain subdirectories (that is, subdirectories that contain their own cmakelist.txt files that are included in a root cmakelists.txt), but none of the templates have those, so it’s not a problem for now. It’s something I’ll be fixing soon, hopefully.

After that, I just needed to make a compile job for each of the templates. Well, a job for each template for each platform it’s supposed to run on. Well, two for each template for each platform - one debug, and one release.

The GTemplates Pipline

As you can see, it actually added up to quite a few. Fortunately, the jobs are all similar enough that I was able to copy/paste them with only minor changes between them:

The Windows compile job for BlueScreen

The Windows compile job for GreenScreen

And, as always, when the pipeline was added it shook out a few bugs. Some of the templates were failing to compile on linux because Gateware was expecting C++ 17 but not getting it. I fixed that by adding it to their respective cmake files (they all got a “set CMAKE_CXX_STANDARD 17” and a “set CMAKE_STANDARD_REQUIRED ON”), and that fixed the issue. Apparently, the source code is supposed to be checking for that already, but clearly isn’t (at least not on linux). More work for the future!

Aside from the two development tasks I’ve mentioned, I’m also working on writing fairly extensive documentation on the gitlab pipeline. It’s a bit of a monster! It’s responsible for testing, compiling, re-testing, and releasing Gateware automatically (as well as testing the templates now), and it’s spread across four repos. I’m doing what I can to try and make future CI/CD work on Gateware easier, but writing documentation (and blog posts) is actually fairly difficult for me to focus on! I often find it easier to read and write code than english…