Colby Peck - Gateware Week 18: Wrapping Up

The pipeline document is finished! It took quite a bit longer than I had hoped it would, but looking at it now, it’s a 27-page in-depth user’s manual for a CI/CD pipeline that spans four repos. I probably should have known it would take longer… In any case, any future CI/CD developers that join the Gateware team will have a thorough and useful reference guide on how the pipeline works.

That wraps up my penultimate week as a student member of the Gateware team. Next week is probably going to be spent primarily on writing up my postmortem. Right when you think you’ve escaped the documentation, there’s more to be had! If I manage some spare time in next week, I’ll probably be cracking open my mac to try and see if I can’t make any more progress on the ghost window issue.

Colby Peck - Gateware Week 17: Documentating!

The biggest remaining task for me is writing up a document that explains the pipeline in detail - most pertinently, how and why we’ve used the solutions we have. The short answer is mostly along the lines of “Well, there is an easier way to do it, but it’s a gitlab premium feature.” We ended up using tokens, masked pipeline variables, and curl requests instead of pipeline secrets and multi-project pipelines because the latter two are only available to gitlab premium. 

There are a few sticking points in the pipeline’s design - it necessarily makes some assumptions about the names of certain folders and branches in Gateware’s various repos, for example. If we ever rename the default branch of GCompiler or GTemplates from ‘master’ to the newer standard ‘main,’ the pipeline will break if it isn’t updated. And it may not break in an obvious way.

Even for the things that are more common and banal as pipelines come, it’s good to have detailed documentation available for reference. I’ve got two more weeks on this project, and I want to make sure that I leave it such that it’s easy for future developers to pick up where I’ve left off. I’m very satisfied with the pipeline that I’ve made, but if the requirements change in the future and the pipeline can’t change to meet the new requirements, then I haven’t done my job right.

I’m not going to say that I’m happy with the rate of progress on writing this document - I’m not. I’ve mostly finished the references for GSource and GCompiler’s pipelines, but I still need to write up some sections describing Gateware’s pipeline, GTemplates’ pipeline, the single-header compiler, and our handling of tokens. I was honestly hoping to have the document done by now, but writing is very difficult for me to focus on, even for relatively short periods of time. And I’ve had some non-gateware responsibilities come and bite me from behind this week (turns out there’s some paperwork and administrata you’re supposed to do when you’re near graduation - who'd've thunk?) On the bright side, the sections that are finished are highly detailed and (hopefully) easily navigable. If you’re working on the pipeline in the future and you come to a part that you don’t understand, it should be easy to find the section of this document that covers that part.

Colby Peck - Gateware Week 16: GTemplates Gets a Pipeline Too!

You get a pipeline! And you get a pipeline! You ALL get a pipeline!!!

My big task this week was to set up a pipeline for GTemplates. If you aren’t aware, GTemplates is a repo that contains a few template projects that use Gateware to make a few different windows. We have one for each of our graphics APIs (DirectX11, DirectX12, OpenGL, Vulkan, and GBlitter for 2D), as well as one for a console application. The pipeline’s job is simple: whenever a new version of Gateware is released, grab that version of Gateware and compile all of the templates with it. 

The first step of this task was to make a script that finds and makes all of the cmake projects in the repo. Well, to make three scripts - one for windows, one for mac, and one for linux. That part was actually not so bad. I got the scripts made and working with little issue. They aren’t set up to work with cmake projects that contain subdirectories (that is, subdirectories that contain their own cmakelist.txt files that are included in a root cmakelists.txt), but none of the templates have those, so it’s not a problem for now. It’s something I’ll be fixing soon, hopefully.

After that, I just needed to make a compile job for each of the templates. Well, a job for each template for each platform it’s supposed to run on. Well, two for each template for each platform - one debug, and one release.

The GTemplates Pipline

As you can see, it actually added up to quite a few. Fortunately, the jobs are all similar enough that I was able to copy/paste them with only minor changes between them:

The Windows compile job for BlueScreen

The Windows compile job for GreenScreen

And, as always, when the pipeline was added it shook out a few bugs. Some of the templates were failing to compile on linux because Gateware was expecting C++ 17 but not getting it. I fixed that by adding it to their respective cmake files (they all got a “set CMAKE_CXX_STANDARD 17” and a “set CMAKE_STANDARD_REQUIRED ON”), and that fixed the issue. Apparently, the source code is supposed to be checking for that already, but clearly isn’t (at least not on linux). More work for the future!

Aside from the two development tasks I’ve mentioned, I’m also working on writing fairly extensive documentation on the gitlab pipeline. It’s a bit of a monster! It’s responsible for testing, compiling, re-testing, and releasing Gateware automatically (as well as testing the templates now), and it’s spread across four repos. I’m doing what I can to try and make future CI/CD work on Gateware easier, but writing documentation (and blog posts) is actually fairly difficult for me to focus on! I often find it easier to read and write code than english…

Colby Peck - Gateware Week 15: Beaten by Apple

This week was mostly spent trying to fix GWindow’s mac implementation. I wasn’t able to figure out a full fix, but I advanced the debugging process a fair bit. I patched the exception that was causing the tests to fail and crash, but a new issue arose: ghost windows. The tests were creating grey, empty windows that weren’t closing until the UT  project finished. I narrowed down the issue to how we’re handling fullscreen windows - whenever a fullscreen window is created or a window is reconfigured to become fullscreen, it leaves behind a ghost window. 

Some of these windows seem to be created from a timing issue - if you create a fullscreen window but delete it before it has the chance to finish the fullscreen animation, it spawns a ghost window. This can be fixed by delaying the deletion until the window is finished animating. But other times, even if the window is given ample time to finish the animation it still spawns a ghost window. I met with Lari yesterday to pair-debug the issue, and we decided that it would be a better use of the month I have left before graduating to work on the GTemplates pipeline instead - this mac issue is going to require a deeper understanding than I can get quickly, and we want to capitalize on the fact that I’ve already become familiar with Gateware’s CI/CD process.

So I did what I could to leave the GWindow mac problem in a better state than I found it - I made sure to document our findings on the Gitlab issue card and I set up the branch to clearly recreate the problem. Hopefully whomever this task is assigned to in the future will have an easier go of it than I have.

Colby Peck - Gateware Week 14: Documentation and GMatrix wrap-up

I took this week a bit slow, finishing up the pipeline refactor took a bit out of me. So far, I’ve spent this week mostly on wrapping up the refactor of the GMatrix tests. I finished those up yesterday with a small fix to the source code (MakeRelative only works with invertible matrices, but it was returning success and garbage values if given an non-invertible matrix; it now returns failure). That branch is now ready to be merged into master. I spent this morning writing and updating documentation for the pipeline: HOWTOBUILD.txt and HOWTORELEASE.txt got updated, and I added a HOWTOUSETHEHEADERLIST.txt just for good measure. HOWTORELEASE.txt now contains information on what changes might unexpectedly break the newly automated release process (essentially, moving or changing directories or some specific files will break the release process if the pipeline isn’t updated).

My next big task is going to be getting Gateware to work on the latest mac OS. My last post wasn’t quite right; Gateware is compiling, but GWindow is failing some tests. It seems to be a timing issue related to GWindow’s thread handling. 

Ah, yes, multithreading. My favorite.

Lari let me know that he’s going to be taking a vacation in August (I can’t blame him, that man works hard). So I’m going to be taking advantage of his presence to help me troubleshoot this mac stuff while I still can (misery loves company). After he leaves, I’m going to be working on GTemplate’s automated pipeline. I’ve spent six weeks working on CI/CD stuff at this point, so I’m pretty comfortable doing it without Lari’s direct supervision.

Colby Peck - Gateware Week 13: A Light at the end of the Pipeline

 

The pipeline refactor is officially finished!

This last week has been very rewarding for me; I properly started work on this pipeline refactor 7 weeks ago, and yesterday Lari and I finally finished it with some repo renaming. The repo once called Gateware is now called GSource, as its job is to host Gateware’s source code. The single header compiler (and single header test project) has been moved out to a repo called GCompiler. The repo we temporarily named GRelease has been renamed to Gateware, and it now houses Gateware’s single header release as well as the relevant documentation, licenses, etc. 

Let me give you a tour of the new pipeline, starting at the inception of an addition to Gateware and ending with a new release (from the perspective of a Gateware developer):

You’ve come up with (or been given) an idea for an addition to Gateware; maybe it’s a bug fix, maybe it’s a new library or an addition to an existing one. You make a branch off of Gateware’s master branch. Every time you push a new change to your branch, the repo will automatically run your code through all of the unit tests on each platform. You can, of course, also run the tests locally. What’s important is that when you go to make a merge request, to merge your changes into Gateware’s master branch, all of the unit tests must pass before the merge request can be approved. You finish up your changes and make a merge request; all of the tests pass and your request is approved. What happens nest is the part I’ve been working on for seven weeks: after passing all of the tests on the master branch, GSource’s pipeline will then tell GCompiler’s pipeline to run. GCompiler clones GSource (using a shallow clone, if you’re curious), then compiles Gateware’s source code into a single header and runs all of the unit tests on that single header (it’s easier than you think to compile a broken single header from working source code). If all of those tests pass, it will then generate a fresh batch of documentation using Doxygen. After the documentation is generated, it will then take the new single header and documentation (as well as some license files and such) and push them to the Gateware repo, tagging the commit with an auto-generated version number. Finally, Gateware’s pipeline uses the version number from the tag to make a new release of Gateware.

In summary, the process of releasing Gateware has been fully automated! Once a merge request is approved, no more action is needed. Developers never have to worry about the single header, and Lari doesn’t have to manually make a new release.

There remains one small hiccup, though. The single header compiler is pretty good at its job, but it still isn’t set up to figure out dependencies and order the headers accordingly. I set it up to accept an external text file to sort the headers by, but developers will still have to manually edit this file and make sure the headers are ordered correctly. Ideally, the compiler would do that automatically (that’s one of the reasons we made compilers in the first place, after all), but that’s a big task that we’ve put off for now. We have a working manual solution, and tasks are more pressing. Did you know that Gateware doesn't compile on the latest version of mac OS? I do. I know it all too well…

Colby Peck - Gateware Week 12: The End is in Sight!

After five weeks of work, it’s finally looking like the pipeline refactor is nearly done! I’m currently having a few hiccups in generating and pushing the doxygen documentation, but the pipeline is currently compiling a single header of Gateware from the development repo, running all of our unit tests on it, and pushing that single header to our release repo! The pipeline is also pushing licenses, a readme, and a version.txt file, and will soon be pushing freshly-generated doxygen documentation with an up-to-date version number. Once that’s done, all that’s left is to set up a remote trigger for GCompiler’s pipeline and to call it from the development repo’s pipeline any time a branch is merged into master. 

This is what the GCompiler pipeline looks like. As you can see, it's working! What you can't see is that the documentation isn't quite generating right. Oh well, progress is progress.

I’m very excited to finish up this task at long last! Once it’s all wrapped up, Gateware developers will never have to worry about the single header generation (or doxygen for that matter) ever again! Well, until someone else comes along and decides they want to improve the pipeline, of course. 

...And, of course, I should mention that the windows runner is still having issues with git authentication. I'm still not sure why. It looked like the issues were fixed when I switched the pipeline to use a shallow clone of the development repo instead of having it as a submodule, but they popped back up as soon as it tried wo do a clone of the GRelease repo using a token. It works when the pipeline runs on my machine, but not Lari's. That still needs investigation.

Post Mortem June 2022

 It's certainly been a very interesting 4 months working on the Gateware project, as this is my final blog post on the day of the Full Sail post mortem presentation. This week I was mostly writing all the final documentation for all the tasks I had completed over my time here on the project, as well as providing notes to the two unfinished tasks I still had leftover before the code freeze the day before post mortem. Those two issues that were left over are the GWindow Race condition on Mac, and GVulkanSurface's multiple renderpass feature. 

Looking back, I find it's crazy that in the course of four months I went from GFile up to starting the research needed to add a new feature to Vulkan, as well as all the other things I got my hands on when working with the Gateware Team. Learning the new OS's, IDE's, and how to use new Git clients really was interesting when learning them. I have found that Mac may be a bit of a handful from a native windows user, and Linux can be a bit picky with some code that works on Windows and Mac, but it gave me learning experience I cannot understate the benefits of.

To all my teammates who worked with me during my time on the Gateware project, thank you for your help and support!

Colby Peck - Gateware Week 11: Pipeline Progress

This week opened with me fixing a bunch of pathing issues related to moving the compiler and single-header test projects out into their own repos. It turns out some of the unit tests include relative paths that need to be preserved (they point at various resources used for testing). No big deal, the GCompiler pipeline now builds the executables to a location with the same relative path as the standard unit tests, and passes in the proper relative path (which is actually a rather roundabout and unintuitive process). But it works now! 

GCompiler's current pipeline

Additionally, the pipeline now has a release stage! Right now, it just generates the doxygen, but once Lari and I decide on a solution for pushing from pipeline, this stage will also be pushing an updated single header release (as well as documentation) to the release repo. On that note,  I made a personal, private repo (a ‘burner’ repo, if you will) so I could safely experiment with tokens and repo access without Lari needing to grant me admin privileges over any of Gateware’s repos (and also so if I leaked credentials or access, the compromised repo could just be deleted with little to no loss of work). And I did it!

Woah, is that a gitlab job committing and pushing? Yes. Yes it is.

That is, I successfully got a pipeline to push to its own repo (the repo that launched the pipeline). Next up on the docket is getting a pipeline to push to a separate repo. Once I’ve figured that out and implemented it into the GCompiler pipeline (and removed all of the single-header stuff from the development repo), the pipeline refactor is essentially finished!

…is what I would say if  the windows runner wasn’t having git issues. For whatever reason, it doesn’t seem to have permission to get the development repo as a submodule, and I have no idea why. It was working, then it just wasn’t! I’m hoping it’s something simple like a credentials issue, but until it gets solved I’m stuck running the GCompiler windows pipeline on my laptop. Not a big issue at the moment, but in the long run, if the pipeline doesn’t work on Lari’s runners then it doesn’t work. 

Second to last Week of Final Project

I've reached a point where I may very well be working on the last set of tasks before my graduation. This week, I did research into why our Macbook had very strange behaviors after installing the new MacOS. In its current form, GWindow has a race condition that I have yet to find the source of beyond a function call to "nextEventMatchingMask" whose apparent use is to perform an event clean up on GWindow for MacOS.

Another odd behavior I encountered while working with the mac for the first half of the week was the Mac's output window logging an error during GAudio's test, but passing the unit test as normal. This doesn't seem to have any harm upfront, but certainly does make the output a bit messy when running the unit test suite on MacOS Monetery.

And Finally, after spending a few days with the Mac and XCode, I decided to take a breather and recollect as I had been surfing through apple documentation for most of the time spent while trying to crack the race condition on Monterey. While I stepped back, I went back to GVulkanSurface to begin implementing the foundation for a new feature: Multiple Render Passes. In its default state, GVulkan surface only has reference to a single render pass, which makes tasks such as rendering UI very difficult, since many sources on the topic of UI and Vulkan point in the direction of using multiple passes to render out World geometry and the UI. At the current moment, I have added a factory of pairs to hold the Renderpasses and user specified flags for rendering.

m_VkRenderPasses is the container that will hold all the information for renderpasses in a future version of Gateware. By Default, GVulkanSurface uses index 0 upon creation and users can expand the container by using addRenderPass(), a planned function that will allow users to simply create a new renderpass and add it to the container.

It's such a bizarre feeling to know there's just one more week before my graduation. I'm happy to have the experiences I gained through Gateware, and am excited for Full Sail's Post-Mortem presentations next Friday!

Colby Peck - Gateware Week 10: The Compiler is Moving Out!

 This week, I finally got together with Lari and we figured out the next step in the pipeline refactor. The single header compiler (and any other release-related stuff) are getting moved out of the development repo! We renamed the old GTools repo to GCompiler and I started work on moving the single header compilation and testing over to it. The GCompiler repo has the development repo as a submodule, and it pulls latest from master any time it runs.

It's going very well.

All joking aside though, the rate of progress on this pipeline is actually very satisfactory. The pipeline is already compiling and testing a single header from latest source. Sure, the tests are failing, and I don’t know why  they’re failing, but seeing as I only got two days of work into it (I met with Lari on Wednesday), I’d say it’s going at a good pace.

Factory deployment and venturing back into Macbook work

     My work on Factory finally reached a stopping point this week, as the work had been officially merged into master! Factory was an interesting project to work with, as it also had many challenges I had to overcome and study while fixing it. Adding it to GBlitter was quite simple, though I did run into a challenge of adding a piece of ld_vector functionality and debugging the functionality to make it work with GBlitter entirely.

    After getting the changes made to Gblitter, I transitioned to one of our Macbook test laptops. Updating the machine to monterey and getting Gateware to compile with the new updates wasn't a very simple task, as I personally am a Windows user. But, with the help of my teammate we were able to get gateware to compile until we ran into an issue inside of GFile with would cause the compilation of the single header to fail. The reason it wasn't able to complete the task was due to the lack of a declaration for <stdlib.h>, which had the definition for a function that had previously been defined in another spot. After deleting the build on the mac and recompiling, Gateware's unit tests would run. However, I then discovered that GWindow has a tendency to fail during the tests inside of an Objective-C file that handles closing the window. Investigation will need to be done on this but I feel it won't be too stressful while I'm already halfway through my final month of Full Sail

Colby Peck - Gateware Week 9: Pipeline Refactor Candidate 1

I finished up the first big step of the pipeline refactor this week! The new single header testing project is now solely responsible for generating, testing, and updating Gateware’s single header. All dependencies and calls to the single header or its compiler were removed from the unit test and dummy main projects, and the pipeline is in a fully functional format on the pipeline refactor branch. If it somehow turns out that further progress on the pipeline refactor can’t be made, we have a state we can revert to and merge into master that still improves the current pipeline (a quick-save, if you will). 

This is what the current stage of the pipeline refactor looks like on gitlab. I know, I know, the linux debug test is failing. There's a timing issue with a multithreading test. Lari's working on it on his own branch, so it's been deemed 'not my problem' for now.

This setup would require Gateware developers to manually update the single header using the single header test project, which is a big downside. Further along in the refactor, the handling of the single header will hopefully be fully automated, but that’s not something that can be implemented until Lari and I figure out what solution we want to use for automatically pushing to a repo. 

While we’re figuring out what’s next in regards to the pipeline refactor (and while I’ve been waiting for the pipelines to run), I’ve been working on refactoring and expanding the unit tests for GVector. I started with a refactor pass that changed all of the existing unit tests to use a lambda and the vector comparison macro I wrote. That made the tests both more unified and more easily extensible. Then, I started expanding the coverage of the unit tests, one by one. The Magnitude and Normalize tests got a bit of special treatment; I guess I was just extra 'in the zone' when I got to them. I haven't finished expanding all of the tests yet, but I'm getting there.

Alright, now I'm going to talk in-depth about some linear algebra for a second here. If you're not interested in that, then this is your final chance to escape. 

...Still here? Cool. 

So! If we define a 'homogenous' vector as a vector where all components either have equal magnitude (positive or negative) or are 0, then there are 64 possible homogenous 4-component vectors that can be composed with a given magnitude. We can leverage this fact to make a batch of 64 vectors that will test positive values, negative values, and 0s in all possible locations and combinations.

 This is a screenshot of the one-component and two-component homogenous vectors being tested in the magnitude test. The three-component and four-component homogenous vectors are also generated and tested. When you're composing a homogenous vector and intend for it to have a given magnitude, each component must equal (intended magnitude / (sqrt(number of non-zero components))).

 I then used this batch of 64 vectors to make some test functions for the Magnitude and Normal tests, then called those functions using some for loops to test a lot of different vectors.

Magnitude test loop

Normalize test loops

As a result, there are now a lot more assertions being done by the vector test.

The math tests previously had about 6,000 assertions, now they have over 180,000

Beginning of the Fourth Month

     This week marks the first week of my fourth month on the Gateware project, and it's a wild feeling thinking about my upcoming graduation while still working away at this project. Last month encompassed my research into gtl::factory, a data structure that keeps contiguous memory by shuffling its data members in an array, but maintaining order of the items using a router array. This week I only had to add two more functions to the .cpp file to ensure GBlitter's transition from ld_vector to factory could be as smooth as possible. And I did have some success with the first few tests of the transition! Only five errors popped up initially, and all of them were removed within the same day as I created some functions to fill in the ld_vector related gaps that Factory didn't have. The new functions check the validity of the key passed into the router, and will return false if the key assigned isn't a valid range in the data array. Other than the new function, GBlitter only needed one line of code adjusted, a call to set_valid, which got replaced with a call to remove. Remove is slightly slower than ld_vector's set_valid, but that's due to the fact that memory is being manipulated, where as set_valid only marks a section of memory as invalid and safe to be over written. 

    Another task I had my hands on this week was investigating why one of our Macbook testing laptops wouldn't compile our code, and the main issue was caused by the recent install of the new OS. Unfortunately, the test machine itself is rather old and doesn't have lots of disk space left after being upgraded to Monterey, the newest version of Mac OS at this time. Installing all the programs also was a struggle, as disk space was quite the premium when attempting installing all the necessary programs needed for running Gateware on mac OS.

    The last thing I began to dive into is more Vulkan research. During month two I had created a function that allowed users to access the depth buffer image for the purpose of clearing. The good news I heard was that the function works as expected, but unfortunately GVulkanSurface's current structure does not allow the user to call the function Vulkan uses to clear the depth buffer. I plan to find a solution in the coming days, so another Vulkan issue is sure to be interesting.

Colby Peck - Gateware Week 8: Incremental Progress

To be perfectly honest, my 8th week on the Gateware team was much slower than I would have liked. Some outside life stuff came up which threw a bit of a wrench into my productivity. Still, I was able to make progress on the pipeline refactor. 

The big important thing that happened this week was a long discussion between myself and Lari of what exactly we wanted the end product to look like. When I say long, I mean it; I think that talk was multiple hours. The end goal we landed on was having the single header live in its own ‘release’ repo, alongside the doxygen documentation and any other user-facing documents. Any time a merge request into master is approved, the pipeline would then run all of the unit tests on source. If they pass, it will generate and test a single header. If those tests pass as well, it will automatically push a new single header to the repo, as well as generate and push fresh doxygen documentation.

I made a diagram of the current build pipeline for the discussion (it’s run three times; once on mac, once on windows, and once on linux).

In the meeting, we discovered that the next big hurdle to surmount in moving towards this pipeline is navigating the security concerns that come with trying to push changes from inside a publicly-accessible .yaml file. That’s still currently in the research phase. 

In the meantime, I made the single-header test project update the local single header any time it runs and all tests pass. This allowed me to clean up the existing unit test project and dummy main test project (they no longer generate or use a single header). As a result, our pipeline now only runs the single header compiler twice instead of six times. I’d love to cut that down to once, but that will require a bit more work. Presently, the single header test project is built twice; once in debug mode and once in release. I could probably make one of those builds reliant upon the other (at least in the pipeline), but that introduces some issues to manually updating the single header (which is what will have to be done if the pipeline cannot be fully automated).

gtl::factory is operational

gtl::factory has definitely been an interesting structure to handle, but the experience I've gained during my month of research has been well worth it. I got to update a structure, create unit tests for it, stress test, and optimize the structure for use within Gateware. Factory was compared against std::vector, std::list, and gtl::ld_vector for speed. When running the tests I found that my first attempt on iterators wasn't as optimal as it could be, and I ended up doing research on how to make them operate similarly to std::vector. Most of the work that I had to add to factory was iterator related, since most of the other functions had existed previously, though any memory allocation related functions had to be refactored so they could accommodate resizable structures like strings. Other issues I found with factory were also crossplatform related, as I would find mac didn't exactly play nice with any mistakes that Visual Studio would ignore, though the issues being highlighted helped me get a better footing on where to fix my work on factory.

    All in all, it's been a very interesting month to say the least, a bit of a different take compared to my previous months where I had seen more graphics and math related tasks. This research project that I ended up undertaking to make sure factory was up and operating with ease was great! Now I continue my journey with Gateware into my fourth and final month of Capstone.

Colby Peck - Gateware Week 7: Linux Fixes (and Breaks)

Week seven working on Gateware has been slower and more frustrating than the last six. My first task was to fix LookAtRH (it wasn’t behaving correctly). That was, thankfully, an easy fix (forwards = eye - at in a right-handed system, not at - eye). Following that, I got right to work refactoring the single-header compiler to use a hard-coded list to order the individual headers (I also snuck in some refactoring that should make the future full refactor easier). I actually tend to find refactoring code to be kind of meditative; it can be very zen. I finished that refactor up around the middle of the week, and it was time to get back to the pipeline! I got to work setting up the single-header test project to compile a single header to a temporary location; simple, right? Well, yeah actually. I set up the cmake script to contain a pre-build step for the test project that compiles the temporary single header, and got it working locally within the hour. But once I pushed that to the repo, the linux build pipeline broke. The remainder of my week was spent digging around the linux runner and the linux pipeline, trying to track down what could possibly be causing the issue (reader, I have yet to find it). The project builds and runs just fine on the linux runner when I build it on its own, but the pipeline refuses to work at the moment. SO! Seeing as I’m stumped on the pipeline at the moment, I’ve shifted my focus onto other tasks until I can get some assistance from Lari. Currently, I’m digging into getting the yaml script to copy the temporary header into the final header position iff the unit tests all pass, and following that I’ll be making a flowchart for Gateware’s pipeline!

gtl::factory reaches a testing phase!

     After three weeks of working at gtl::factory, I'm proud to see it finally reach a stable enough state to be tested against other containers such as gtl::ld_vector. I began writing the tests earlier this week, and it compares the speed of four containers: vector, list, factory, and ld_vector.

     The tests time each container on their performance when using push_back, removal functions such as remove or erase, iteration through the container, random removal from the container, and storing large objects. When writing the test for the large object, I discovered some extra minor bugs that hadn't been spotted in the first two weeks of research, but resolved them in a short amount of time. The biggest issue that factory had was resetting the highest_key during remove. Remove should not change the key unless the programmer calls for a remove from the end of the array, or uses pop_back. The key being adjusted caused issues with loop-based removing items from the container. Another issue I came across was the router table missing keys, since it was using capacity instead of highest_key. the router array may not be the same size as the data array but that's the intention so every key is retained when using the factory.

    the test on a large image (2048x2048) is shown in this screenshot. the test adds 40 instances of the image to the containers, iterates, removes 10 random spots of the array, then iterates through the smaller container after removals finish. The speed is recorded using a simple implementation of std::chrono, which I did revisit when writing the test!

    All-in-all, another good week working on the factory, and I'm pleased to see more progress! Soon I plan to implement factory inside of GBlitter for an optimization feature.

Colby Peck - Gateware Week 6: A New Pipeline

 My sixth week here on the Gateware team started out with finalizing a refactor of our float-comparison macros, as well as cleaning up some rather opaquely-worded doxygen documentation. When I first started on the task of patching a small hole in the GMatrix library, I was not expecting to get a task that would balloon out into a full-week ordeal, but such is the way of things I suppose. 

With all of that finally finished, I was ready to move on to my next big task: a pipeline refactor. Lari and I went over the desired final state of the pipeline on Monday. We made several tasks on gitlab defining incremental deployable changes that would ultimately take us to that desired final pipeline (#214-#219). TLDR, we want to set up an automated release process that will: 

• Contain a new ‘release’ branch
• Generate a new single header and run all unit tests on it whenever changes are pushed, then push that single header to the repo if all tests pass
• Generate and push new doxygen documentation any time a new valid single header is generated
• Update the release branch any time a new valid single header is generated

I got started on these tasks on Tuesday. I managed to make a project that tests the current single header and partially integrate it into the pipeline much faster than I had anticipated; it only took about a day. Of course, as soon as I did, I ran headfirst into a bug. It turns out that the mac and linux DUMMY_MAIN projects weren’t testing the single header like they were supposed to, and the single-header compiles into a broken state on Linux. As in, projects that include a single-header generated on Linux will not compile. So, Gateware’s Linux release is broken. It has been for some time, but we didn’t know until now. 

Even if this bug wasn’t more important than refactoring the pipeline, it prevents me from knowing if the pipeline is working on Linux. So I’m diving into the compiler once again. The issue is that the files in the Core directory are ordered differently on Linux than they are on Windows/Mac, and the compiler currently expects them to be ordered the way they are on Windows/Mac. My current plan is to add a patch to the current compiler (hard-coding a specified order for Core), and to create a task for a more comprehensive compiler refactor (to be picked up by myself or someone else in the future). The compiler is currently very brittle; it’s ordering the directories manually and the files within directories are ordered by the number of dependencies they have. It is assumed on good faith that writing the files with more dependencies first will result in a good single header (this is already not the case with Core). A more robust compiler would actually build and traverse a dependency hierarchy, but that’s a much more comprehensive refactor. I don’t like kicking the can down the road, but I can only work on one thing at a time. So for now, patching and putting off the refactor is probably the way to go.

Another week of gtl::factory unit tests!

 While working on gtl::factory I came across multiple edge cases that caused the task to take slightly longer than expected, but I did manage to finish them during the week. Strings and vectors are quite an odd edge case to handle, but wasn't too harsh. The first issue came from the [] operator returning a copy instead of a reference, which made calling functions such as .push_back() or .append() to modify a copy instead of the data itself. Other operators also had memory leaking issues, as the old arrays weren't being dealt with before copying new data into the new array. Most of these edge cases were only an issue with resizable objects, but the process of debugging the code and removing all the leaks proved to be a bit of a challenge, but nothing too heavy beyond chasing down memory addresses.

This was the last instance of a memory leak I fixed before I checked out for the week. the original array was never being deallocated and led to issues with memory leaking.

Colby Peck - Gateware Week 5

I’m just now finishing up the fifth week of my time here on the Gateware team. I ended up accidentally joining a month earlier than I was supposed to (bit of a long story), so this is the end of my first week as a proper Gateware dev where I was able to dedicate my full attention to the project. This first post of mine is going to be a bit lengthy, since it covers 5 weeks of work; I hope you’ll forgive me that, dear reader.

My first task was to add some new functionality to the single header compiler; the ability to specify files to be prepended or appended to the single header. My onboarding process was a bit delayed, so I wasn’t able to make or commit any changes for my first few days on the project. I spent that time reading through the entire single-header compiler and writing down - in plain english - what all the code was doing in a separate document. This ended up being more useful than I first anticipated. When I was later fully onboarded, my first act was to refactor the code I was about to change - to make it more mutable - and I had just written a guide to the whole thing. All told, the refactor and feature addition itself only took about a day’s work once I sat down and really got to it, but it was only so straightforward because I fully comprehended the code before I began.

After the relatively painless work on the compiler, I was due for a task that would have me pulling my hair out at moments - such is only fair. The task in question was to suppress and fix the warnings generated when Gateware is compiled. A simple task to understand, but completing it took a lot out of me. Since Gateware is multi-platform, it’s internally compiled on Windows, Linux, and Mac as part of the automated testing pipeline. This meant that I had three different compilers to work with. Not a huge issue on its own; it’s not hard to find the appropriate preprocessor directives to suppress specific warnings. Then, all I had to do was make sure those preprocessor directives got parsed by the compiler before it compiles Gateware. Not hard for the single-header version: I just got all of the directives into some compiler-specific #ifdef blocks and used the new prepend functionality I had just added (as well as adding some blocks that unsuppressed the warnings to the end with the append functionality). But suppressing the warnings on our pipeline was a much trickier process - I’m feeling exhausted just recalling how much digging around it took. After several days of trying various configurations and placements of preprocessor directives (none of which worked), I figured out how to suppress warnings with the command-line arguments used in our cmake scripts. That did the trick.

Following the warning suppression, I was tasked with some work in the GMatrix library. LookAtLH needed a right-handed variant and some of the projection-matrix-construction functions had gotten mirrored versions, but hadn’t gotten their unit tests. Making the new function was fairly simple, but when I opened up the GMatrix unit tests, I found that all of the tests only tested a single case. So… In addition to adding the new tests, I expanded the coverage of nearly all of the existing ones. Every test got a refactor pass as well. 

I made a real-life transform gizmo out of LEGO to help visualize the various matrix functions and how they were supposed to behave. It was delightful and very helpful.

When I added some new cases for the RotationGlobalY test, I butted up against an edge case in Gateware’s floating-point comparison. Now, float comparison is a bit of a rabbit hole that I won’t take you down; suffice it to say that comparing floats can get messy. Anyways, I ended up refactoring Gateware’s float comparison macros.

Old comparison macros, now deprecated

New comparison macros

They're more-or-less in their final form now, pending discussion and review with Lari. We might need a macro that allows for a hybrid test with two different margins (one absolute, one relative).

Refactoring & Legacy

You may notice that in each of my tasks, I’ve refactored some legacy code. As I’ve gone through with the refactors, I’ve been thinking about the nature of refactoring code in the context of long-term group projects like Gateware. If you’d indulge me, I’d like to lay out my thoughts regarding the matter in the next few paragraphs.

The goal of refactoring is to make code more readable. But why is this important? Because in order to change code, you must comprehend it. The less readable the code, the harder it is to comprehend, and therefore the harder it is to change. Hard-to-read code is hard-to-change code; rigid code. And codebase rigidity makes change (and therefore progress) more difficult. So it’s vitally important for a project’s long-term health that the codebase be made as mutable as possible, and good readability is the chief factor of good mutability. If I know how something works, I can change it. If I have no idea what’s going on, trying to add changes will break things in unexpected ways.

However, refactoring legacy code feels kind of disrespectful. It feels like reorganizing someone else’s workspace when they aren’t looking. My discomfort has some roots in imposter syndrome: “Who am I to say that my factoring of this code is cleaner or better than the one that’s already here?” It feels somewhat judgmental too. “I, the arbiter of clean code, have judged thy code messy.” To address these feelings, let me tell you a story.

When I went home for Christmas last year, I ended up doing some work in my uncle’s wood shop. At one point, he had me use his belt sander (yay, power tools!) Now, this belt sander was a bit of a mess: it still had an old, tattered belt on it and the casing was full of sawdust. When I saw this, I didn’t think less of my uncle for it. I didn’t blame him for leaving his sander in a messy state at the end of a long workday. But I did disassemble and clean the sander before I used it, and I cleaned it again once I was finished.

An ideal codebase gets perpetually cleaner and easier to change. In order to make that happen, all that’s needed is for each programmer to leave whatever code they work on in a slightly cleaner state than they find it (or at least not to make it messier). Conversely, if we do not allow ourselves to clean up the code that we’re working on (via refactoring), the codebase can only get messier. So, “Who am I to say that my factoring is better than the one that’s already here?”  Well, I’m the guy working on this code. For the moment, this shared workspace is my responsibility, and I’ll keep it as tidy as I can. And as for “I, the arbiter of clean code, have judged thy code messy.” …That’s a very draconian way of putting it, but it contains some truth. In order to clean something, you must acknowledge that it’s messy. We must be honest with ourselves if we want to keep our codebase clean. The important distinction is that calling someone’s code messy isn’t equivalent to calling them a bad programmer. The natural state of code is a barely working mess. Any programmer with any amount of experience has learned this firsthand. I write messy code too! But when I am fortunate enough to have the time and energy to clean my code, I do. And when I work with legacy code that’s messy, I clean it. My goal with this practice is to leave everything I touch just a bit better than I found it - and hopefully make the lives of Gateware programmers (myself included) easier.