Friday, August 19, 2022

Colby Peck - Gateware Week 18: Wrapping Up

The pipeline document is finished! It took quite a bit longer than I had hoped it would, but looking at it now, it’s a 27-page in-depth user’s manual for a CI/CD pipeline that spans four repos. I probably should have known it would take longer… In any case, any future CI/CD developers that join the Gateware team will have a thorough and useful reference guide on how the pipeline works.

That wraps up my penultimate week as a student member of the Gateware team. Next week is probably going to be spent primarily on writing up my postmortem. Right when you think you’ve escaped the documentation, there’s more to be had! If I manage some spare time in next week, I’ll probably be cracking open my mac to try and see if I can’t make any more progress on the ghost window issue.

Friday, August 12, 2022

Colby Peck - Gateware Week 17: Documentating!

The biggest remaining task for me is writing up a document that explains the pipeline in detail - most pertinently, how and why we’ve used the solutions we have. The short answer is mostly along the lines of “Well, there is an easier way to do it, but it’s a gitlab premium feature.” We ended up using tokens, masked pipeline variables, and curl requests instead of pipeline secrets and multi-project pipelines because the latter two are only available to gitlab premium. 

There are a few sticking points in the pipeline’s design - it necessarily makes some assumptions about the names of certain folders and branches in Gateware’s various repos, for example. If we ever rename the default branch of GCompiler or GTemplates from ‘master’ to the newer standard ‘main,’ the pipeline will break if it isn’t updated. And it may not break in an obvious way.

Even for the things that are more common and banal as pipelines come, it’s good to have detailed documentation available for reference. I’ve got two more weeks on this project, and I want to make sure that I leave it such that it’s easy for future developers to pick up where I’ve left off. I’m very satisfied with the pipeline that I’ve made, but if the requirements change in the future and the pipeline can’t change to meet the new requirements, then I haven’t done my job right.

I’m not going to say that I’m happy with the rate of progress on writing this document - I’m not. I’ve mostly finished the references for GSource and GCompiler’s pipelines, but I still need to write up some sections describing Gateware’s pipeline, GTemplates’ pipeline, the single-header compiler, and our handling of tokens. I was honestly hoping to have the document done by now, but writing is very difficult for me to focus on, even for relatively short periods of time. And I’ve had some non-gateware responsibilities come and bite me from behind this week (turns out there’s some paperwork and administrata you’re supposed to do when you’re near graduation - who'd've thunk?) On the bright side, the sections that are finished are highly detailed and (hopefully) easily navigable. If you’re working on the pipeline in the future and you come to a part that you don’t understand, it should be easy to find the section of this document that covers that part.

Saturday, August 6, 2022

Colby Peck - Gateware Week 16: GTemplates Gets a Pipeline Too!

You get a pipeline! And you get a pipeline! You ALL get a pipeline!!!

My big task this week was to set up a pipeline for GTemplates. If you aren’t aware, GTemplates is a repo that contains a few template projects that use Gateware to make a few different windows. We have one for each of our graphics APIs (DirectX11, DirectX12, OpenGL, Vulkan, and GBlitter for 2D), as well as one for a console application. The pipeline’s job is simple: whenever a new version of Gateware is released, grab that version of Gateware and compile all of the templates with it. 

The first step of this task was to make a script that finds and makes all of the cmake projects in the repo. Well, to make three scripts - one for windows, one for mac, and one for linux. That part was actually not so bad. I got the scripts made and working with little issue. They aren’t set up to work with cmake projects that contain subdirectories (that is, subdirectories that contain their own cmakelist.txt files that are included in a root cmakelists.txt), but none of the templates have those, so it’s not a problem for now. It’s something I’ll be fixing soon, hopefully.

After that, I just needed to make a compile job for each of the templates. Well, a job for each template for each platform it’s supposed to run on. Well, two for each template for each platform - one debug, and one release.

The GTemplates Pipline

As you can see, it actually added up to quite a few. Fortunately, the jobs are all similar enough that I was able to copy/paste them with only minor changes between them:

The Windows compile job for BlueScreen

The Windows compile job for GreenScreen

And, as always, when the pipeline was added it shook out a few bugs. Some of the templates were failing to compile on linux because Gateware was expecting C++ 17 but not getting it. I fixed that by adding it to their respective cmake files (they all got a “set CMAKE_CXX_STANDARD 17” and a “set CMAKE_STANDARD_REQUIRED ON”), and that fixed the issue. Apparently, the source code is supposed to be checking for that already, but clearly isn’t (at least not on linux). More work for the future!

Aside from the two development tasks I’ve mentioned, I’m also working on writing fairly extensive documentation on the gitlab pipeline. It’s a bit of a monster! It’s responsible for testing, compiling, re-testing, and releasing Gateware automatically (as well as testing the templates now), and it’s spread across four repos. I’m doing what I can to try and make future CI/CD work on Gateware easier, but writing documentation (and blog posts) is actually fairly difficult for me to focus on! I often find it easier to read and write code than english…

Thursday, July 21, 2022

Colby Peck - Gateware Week 15: Beaten by Apple

This week was mostly spent trying to fix GWindow’s mac implementation. I wasn’t able to figure out a full fix, but I advanced the debugging process a fair bit. I patched the exception that was causing the tests to fail and crash, but a new issue arose: ghost windows. The tests were creating grey, empty windows that weren’t closing until the UT  project finished. I narrowed down the issue to how we’re handling fullscreen windows - whenever a fullscreen window is created or a window is reconfigured to become fullscreen, it leaves behind a ghost window. 

Some of these windows seem to be created from a timing issue - if you create a fullscreen window but delete it before it has the chance to finish the fullscreen animation, it spawns a ghost window. This can be fixed by delaying the deletion until the window is finished animating. But other times, even if the window is given ample time to finish the animation it still spawns a ghost window. I met with Lari yesterday to pair-debug the issue, and we decided that it would be a better use of the month I have left before graduating to work on the GTemplates pipeline instead - this mac issue is going to require a deeper understanding than I can get quickly, and we want to capitalize on the fact that I’ve already become familiar with Gateware’s CI/CD process.

So I did what I could to leave the GWindow mac problem in a better state than I found it - I made sure to document our findings on the Gitlab issue card and I set up the branch to clearly recreate the problem. Hopefully whomever this task is assigned to in the future will have an easier go of it than I have.

Friday, July 15, 2022

Colby Peck - Gateware Week 14: Documentation and GMatrix wrap-up

I took this week a bit slow, finishing up the pipeline refactor took a bit out of me. So far, I’ve spent this week mostly on wrapping up the refactor of the GMatrix tests. I finished those up yesterday with a small fix to the source code (MakeRelative only works with invertible matrices, but it was returning success and garbage values if given an non-invertible matrix; it now returns failure). That branch is now ready to be merged into master. I spent this morning writing and updating documentation for the pipeline: HOWTOBUILD.txt and HOWTORELEASE.txt got updated, and I added a HOWTOUSETHEHEADERLIST.txt just for good measure. HOWTORELEASE.txt now contains information on what changes might unexpectedly break the newly automated release process (essentially, moving or changing directories or some specific files will break the release process if the pipeline isn’t updated).

My next big task is going to be getting Gateware to work on the latest mac OS. My last post wasn’t quite right; Gateware is compiling, but GWindow is failing some tests. It seems to be a timing issue related to GWindow’s thread handling. 

Ah, yes, multithreading. My favorite.

Lari let me know that he’s going to be taking a vacation in August (I can’t blame him, that man works hard). So I’m going to be taking advantage of his presence to help me troubleshoot this mac stuff while I still can (misery loves company). After he leaves, I’m going to be working on GTemplate’s automated pipeline. I’ve spent six weeks working on CI/CD stuff at this point, so I’m pretty comfortable doing it without Lari’s direct supervision.

Thursday, July 7, 2022

Colby Peck - Gateware Week 13: A Light at the end of the Pipeline


The pipeline refactor is officially finished!

This last week has been very rewarding for me; I properly started work on this pipeline refactor 7 weeks ago, and yesterday Lari and I finally finished it with some repo renaming. The repo once called Gateware is now called GSource, as its job is to host Gateware’s source code. The single header compiler (and single header test project) has been moved out to a repo called GCompiler. The repo we temporarily named GRelease has been renamed to Gateware, and it now houses Gateware’s single header release as well as the relevant documentation, licenses, etc. 

Let me give you a tour of the new pipeline, starting at the inception of an addition to Gateware and ending with a new release (from the perspective of a Gateware developer):

You’ve come up with (or been given) an idea for an addition to Gateware; maybe it’s a bug fix, maybe it’s a new library or an addition to an existing one. You make a branch off of Gateware’s master branch. Every time you push a new change to your branch, the repo will automatically run your code through all of the unit tests on each platform. You can, of course, also run the tests locally. What’s important is that when you go to make a merge request, to merge your changes into Gateware’s master branch, all of the unit tests must pass before the merge request can be approved. You finish up your changes and make a merge request; all of the tests pass and your request is approved. What happens nest is the part I’ve been working on for seven weeks: after passing all of the tests on the master branch, GSource’s pipeline will then tell GCompiler’s pipeline to run. GCompiler clones GSource (using a shallow clone, if you’re curious), then compiles Gateware’s source code into a single header and runs all of the unit tests on that single header (it’s easier than you think to compile a broken single header from working source code). If all of those tests pass, it will then generate a fresh batch of documentation using Doxygen. After the documentation is generated, it will then take the new single header and documentation (as well as some license files and such) and push them to the Gateware repo, tagging the commit with an auto-generated version number. Finally, Gateware’s pipeline uses the version number from the tag to make a new release of Gateware.

In summary, the process of releasing Gateware has been fully automated! Once a merge request is approved, no more action is needed. Developers never have to worry about the single header, and Lari doesn’t have to manually make a new release.

There remains one small hiccup, though. The single header compiler is pretty good at its job, but it still isn’t set up to figure out dependencies and order the headers accordingly. I set it up to accept an external text file to sort the headers by, but developers will still have to manually edit this file and make sure the headers are ordered correctly. Ideally, the compiler would do that automatically (that’s one of the reasons we made compilers in the first place, after all), but that’s a big task that we’ve put off for now. We have a working manual solution, and tasks are more pressing. Did you know that Gateware doesn't compile on the latest version of mac OS? I do. I know it all too well…

Tuesday, June 28, 2022

Colby Peck - Gateware Week 12: The End is in Sight!

After five weeks of work, it’s finally looking like the pipeline refactor is nearly done! I’m currently having a few hiccups in generating and pushing the doxygen documentation, but the pipeline is currently compiling a single header of Gateware from the development repo, running all of our unit tests on it, and pushing that single header to our release repo! The pipeline is also pushing licenses, a readme, and a version.txt file, and will soon be pushing freshly-generated doxygen documentation with an up-to-date version number. Once that’s done, all that’s left is to set up a remote trigger for GCompiler’s pipeline and to call it from the development repo’s pipeline any time a branch is merged into master. 

This is what the GCompiler pipeline looks like. As you can see, it's working! What you can't see is that the documentation isn't quite generating right. Oh well, progress is progress.

I’m very excited to finish up this task at long last! Once it’s all wrapped up, Gateware developers will never have to worry about the single header generation (or doxygen for that matter) ever again! Well, until someone else comes along and decides they want to improve the pipeline, of course. 

...And, of course, I should mention that the windows runner is still having issues with git authentication. I'm still not sure why. It looked like the issues were fixed when I switched the pipeline to use a shallow clone of the development repo instead of having it as a submodule, but they popped back up as soon as it tried wo do a clone of the GRelease repo using a token. It works when the pipeline runs on my machine, but not Lari's. That still needs investigation.