Tuesday, June 28, 2022

Colby Peck - Gateware Week 12: The End is in Sight!

After five weeks of work, it’s finally looking like the pipeline refactor is nearly done! I’m currently having a few hiccups in generating and pushing the doxygen documentation, but the pipeline is currently compiling a single header of Gateware from the development repo, running all of our unit tests on it, and pushing that single header to our release repo! The pipeline is also pushing licenses, a readme, and a version.txt file, and will soon be pushing freshly-generated doxygen documentation with an up-to-date version number. Once that’s done, all that’s left is to set up a remote trigger for GCompiler’s pipeline and to call it from the development repo’s pipeline any time a branch is merged into master. 

This is what the GCompiler pipeline looks like. As you can see, it's working! What you can't see is that the documentation isn't quite generating right. Oh well, progress is progress.


I’m very excited to finish up this task at long last! Once it’s all wrapped up, Gateware developers will never have to worry about the single header generation (or doxygen for that matter) ever again! Well, until someone else comes along and decides they want to improve the pipeline, of course. 


...And, of course, I should mention that the windows runner is still having issues with git authentication. I'm still not sure why. It looked like the issues were fixed when I switched the pipeline to use a shallow clone of the development repo instead of having it as a submodule, but they popped back up as soon as it tried wo do a clone of the GRelease repo using a token. It works when the pipeline runs on my machine, but not Lari's. That still needs investigation.

Friday, June 24, 2022

Post Mortem June 2022

 It's certainly been a very interesting 4 months working on the Gateware project, as this is my final blog post on the day of the Full Sail post mortem presentation. This week I was mostly writing all the final documentation for all the tasks I had completed over my time here on the project, as well as providing notes to the two unfinished tasks I still had leftover before the code freeze the day before post mortem. Those two issues that were left over are the GWindow Race condition on Mac, and GVulkanSurface's multiple renderpass feature. 

Looking back, I find it's crazy that in the course of four months I went from GFile up to starting the research needed to add a new feature to Vulkan, as well as all the other things I got my hands on when working with the Gateware Team. Learning the new OS's, IDE's, and how to use new Git clients really was interesting when learning them. I have found that Mac may be a bit of a handful from a native windows user, and Linux can be a bit picky with some code that works on Windows and Mac, but it gave me learning experience I cannot understate the benefits of.

To all my teammates who worked with me during my time on the Gateware project, thank you for your help and support!

Wednesday, June 22, 2022

Colby Peck - Gateware Week 11: Pipeline Progress

This week opened with me fixing a bunch of pathing issues related to moving the compiler and single-header test projects out into their own repos. It turns out some of the unit tests include relative paths that need to be preserved (they point at various resources used for testing). No big deal, the GCompiler pipeline now builds the executables to a location with the same relative path as the standard unit tests, and passes in the proper relative path (which is actually a rather roundabout and unintuitive process). But it works now! 

GCompiler's current pipeline



Additionally, the pipeline now has a release stage! Right now, it just generates the doxygen, but once Lari and I decide on a solution for pushing from pipeline, this stage will also be pushing an updated single header release (as well as documentation) to the release repo. On that note,  I made a personal, private repo (a ‘burner’ repo, if you will) so I could safely experiment with tokens and repo access without Lari needing to grant me admin privileges over any of Gateware’s repos (and also so if I leaked credentials or access, the compromised repo could just be deleted with little to no loss of work). And I did it!


Woah, is that a gitlab job committing and pushing? Yes. Yes it is.



That is, I successfully got a pipeline to push to its own repo (the repo that launched the pipeline). Next up on the docket is getting a pipeline to push to a separate repo. Once I’ve figured that out and implemented it into the GCompiler pipeline (and removed all of the single-header stuff from the development repo), the pipeline refactor is essentially finished!


…is what I would say if  the windows runner wasn’t having git issues. For whatever reason, it doesn’t seem to have permission to get the development repo as a submodule, and I have no idea why. It was working, then it just wasn’t! I’m hoping it’s something simple like a credentials issue, but until it gets solved I’m stuck running the GCompiler windows pipeline on my laptop. Not a big issue at the moment, but in the long run, if the pipeline doesn’t work on Lari’s runners then it doesn’t work. 


Friday, June 17, 2022

Second to last Week of Final Project

I've reached a point where I may very well be working on the last set of tasks before my graduation. This week, I did research into why our Macbook had very strange behaviors after installing the new MacOS. In its current form, GWindow has a race condition that I have yet to find the source of beyond a function call to "nextEventMatchingMask" whose apparent use is to perform an event clean up on GWindow for MacOS.
Another odd behavior I encountered while working with the mac for the first half of the week was the Mac's output window logging an error during GAudio's test, but passing the unit test as normal. This doesn't seem to have any harm upfront, but certainly does make the output a bit messy when running the unit test suite on MacOS Monetery.

And Finally, after spending a few days with the Mac and XCode, I decided to take a breather and recollect as I had been surfing through apple documentation for most of the time spent while trying to crack the race condition on Monterey. While I stepped back, I went back to GVulkanSurface to begin implementing the foundation for a new feature: Multiple Render Passes. In its default state, GVulkan surface only has reference to a single render pass, which makes tasks such as rendering UI very difficult, since many sources on the topic of UI and Vulkan point in the direction of using multiple passes to render out World geometry and the UI. At the current moment, I have added a factory of pairs to hold the Renderpasses and user specified flags for rendering.


m_VkRenderPasses is the container that will hold all the information for renderpasses in a future version of Gateware. By Default, GVulkanSurface uses index 0 upon creation and users can expand the container by using addRenderPass(), a planned function that will allow users to simply create a new renderpass and add it to the container.

It's such a bizarre feeling to know there's just one more week before my graduation. I'm happy to have the experiences I gained through Gateware, and am excited for Full Sail's Post-Mortem presentations next Friday!


Monday, June 13, 2022

Colby Peck - Gateware Week 10: The Compiler is Moving Out!

 This week, I finally got together with Lari and we figured out the next step in the pipeline refactor. The single header compiler (and any other release-related stuff) are getting moved out of the development repo! We renamed the old GTools repo to GCompiler and I started work on moving the single header compilation and testing over to it. The GCompiler repo has the development repo as a submodule, and it pulls latest from master any time it runs.


It's going very well.



All joking aside though, the rate of progress on this pipeline is actually very satisfactory. The pipeline is already compiling and testing a single header from latest source. Sure, the tests are failing, and I don’t know why  they’re failing, but seeing as I only got two days of work into it (I met with Lari on Wednesday), I’d say it’s going at a good pace.

Friday, June 10, 2022

Factory deployment and venturing back into Macbook work

     My work on Factory finally reached a stopping point this week, as the work had been officially merged into master! Factory was an interesting project to work with, as it also had many challenges I had to overcome and study while fixing it. Adding it to GBlitter was quite simple, though I did run into a challenge of adding a piece of ld_vector functionality and debugging the functionality to make it work with GBlitter entirely.

    After getting the changes made to Gblitter, I transitioned to one of our Macbook test laptops. Updating the machine to monterey and getting Gateware to compile with the new updates wasn't a very simple task, as I personally am a Windows user. But, with the help of my teammate we were able to get gateware to compile until we ran into an issue inside of GFile with would cause the compilation of the single header to fail. The reason it wasn't able to complete the task was due to the lack of a declaration for <stdlib.h>, which had the definition for a function that had previously been defined in another spot. After deleting the build on the mac and recompiling, Gateware's unit tests would run. However, I then discovered that GWindow has a tendency to fail during the tests inside of an Objective-C file that handles closing the window. Investigation will need to be done on this but I feel it won't be too stressful while I'm already halfway through my final month of Full Sail

Tuesday, June 7, 2022

Colby Peck - Gateware Week 9: Pipeline Refactor Candidate 1

I finished up the first big step of the pipeline refactor this week! The new single header testing project is now solely responsible for generating, testing, and updating Gateware’s single header. All dependencies and calls to the single header or its compiler were removed from the unit test and dummy main projects, and the pipeline is in a fully functional format on the pipeline refactor branch. If it somehow turns out that further progress on the pipeline refactor can’t be made, we have a state we can revert to and merge into master that still improves the current pipeline (a quick-save, if you will). 

This is what the current stage of the pipeline refactor looks like on gitlab. I know, I know, the linux debug test is failing. There's a timing issue with a multithreading test. Lari's working on it on his own branch, so it's been deemed 'not my problem' for now.


This setup
would require Gateware developers to manually update the single header using the single header test project, which is a big downside. Further along in the refactor, the handling of the single header will hopefully be fully automated, but that’s not something that can be implemented until Lari and I figure out what solution we want to use for automatically pushing to a repo. 


While we’re figuring out what’s next in regards to the pipeline refactor (and while I’ve been waiting for the pipelines to run), I’ve been working on refactoring and expanding the unit tests for GVector. I started with a refactor pass that changed all of the existing unit tests to use a lambda and the vector comparison macro I wrote. That made the tests both more unified and more easily extensible. Then, I started expanding the coverage of the unit tests, one by one. The Magnitude and Normalize tests got a bit of special treatment; I guess I was just extra 'in the zone' when I got to them. I haven't finished expanding all of the tests yet, but I'm getting there.

Alright, now I'm going to talk in-depth about some linear algebra for a second here. If you're not interested in that, then this is your final chance to escape. 

...Still here? Cool. 
So! If we define a 'homogenous' vector as a vector where all components either have equal magnitude (positive or negative) or are 0, then there are 64 possible homogenous 4-component vectors that can be composed with a given magnitude. We can leverage this fact to make a batch of 64 vectors that will test positive values, negative values, and 0s in all possible locations and combinations.

 This is a screenshot of the one-component and two-component homogenous vectors being tested in the magnitude test. The three-component and four-component homogenous vectors are also generated and tested. When you're composing a homogenous vector and intend for it to have a given magnitude, each component must equal (intended magnitude / (sqrt(number of non-zero components))).

 I then used this batch of 64 vectors to make some test functions for the Magnitude and Normal tests, then called those functions using some for loops to test a lot of different vectors.

Magnitude test loop


Normalize test loops

As a result, there are now a lot more assertions being done by the vector test.

The math tests previously had about 6,000 assertions, now they have over 180,000




Friday, June 3, 2022

Beginning of the Fourth Month

     This week marks the first week of my fourth month on the Gateware project, and it's a wild feeling thinking about my upcoming graduation while still working away at this project. Last month encompassed my research into gtl::factory, a data structure that keeps contiguous memory by shuffling its data members in an array, but maintaining order of the items using a router array. This week I only had to add two more functions to the .cpp file to ensure GBlitter's transition from ld_vector to factory could be as smooth as possible. And I did have some success with the first few tests of the transition! Only five errors popped up initially, and all of them were removed within the same day as I created some functions to fill in the ld_vector related gaps that Factory didn't have. The new functions check the validity of the key passed into the router, and will return false if the key assigned isn't a valid range in the data array. Other than the new function, GBlitter only needed one line of code adjusted, a call to set_valid, which got replaced with a call to remove. Remove is slightly slower than ld_vector's set_valid, but that's due to the fact that memory is being manipulated, where as set_valid only marks a section of memory as invalid and safe to be over written. 

    Another task I had my hands on this week was investigating why one of our Macbook testing laptops wouldn't compile our code, and the main issue was caused by the recent install of the new OS. Unfortunately, the test machine itself is rather old and doesn't have lots of disk space left after being upgraded to Monterey, the newest version of Mac OS at this time. Installing all the programs also was a struggle, as disk space was quite the premium when attempting installing all the necessary programs needed for running Gateware on mac OS.

    The last thing I began to dive into is more Vulkan research. During month two I had created a function that allowed users to access the depth buffer image for the purpose of clearing. The good news I heard was that the function works as expected, but unfortunately GVulkanSurface's current structure does not allow the user to call the function Vulkan uses to clear the depth buffer. I plan to find a solution in the coming days, so another Vulkan issue is sure to be interesting.

Wednesday, June 1, 2022

Colby Peck - Gateware Week 8: Incremental Progress

To be perfectly honest, my 8th week on the Gateware team was much slower than I would have liked. Some outside life stuff came up which threw a bit of a wrench into my productivity. Still, I was able to make progress on the pipeline refactor. 

The big important thing that happened this week was a long discussion between myself and Lari of what exactly we wanted the end product to look like. When I say long, I mean it; I think that talk was multiple hours. The end goal we landed on was having the single header live in its own ‘release’ repo, alongside the doxygen documentation and any other user-facing documents. Any time a merge request into master is approved, the pipeline would then run all of the unit tests on source. If they pass, it will generate and test a single header. If those tests pass as well, it will automatically push a new single header to the repo, as well as generate and push fresh doxygen documentation.


I made a diagram of the current build pipeline for the discussion (it’s run three times; once on mac, once on windows, and once on linux).



In the meeting, we discovered that the next big hurdle to surmount in moving towards this pipeline is navigating the security concerns that come with trying to push changes from inside a publicly-accessible .yaml file. That’s still currently in the research phase. 


In the meantime, I made the single-header test project update the local single header any time it runs and all tests pass. This allowed me to clean up the existing unit test project and dummy main test project (they no longer generate or use a single header). As a result, our pipeline now only runs the single header compiler twice instead of six times. I’d love to cut that down to once, but that will require a bit more work. Presently, the single header test project is built twice; once in debug mode and once in release. I could probably make one of those builds reliant upon the other (at least in the pipeline), but that introduces some issues to manually updating the single header (which is what will have to be done if the pipeline cannot be fully automated).