gtl::factory is operational

gtl::factory has definitely been an interesting structure to handle, but the experience I've gained during my month of research has been well worth it. I got to update a structure, create unit tests for it, stress test, and optimize the structure for use within Gateware. Factory was compared against std::vector, std::list, and gtl::ld_vector for speed. When running the tests I found that my first attempt on iterators wasn't as optimal as it could be, and I ended up doing research on how to make them operate similarly to std::vector. Most of the work that I had to add to factory was iterator related, since most of the other functions had existed previously, though any memory allocation related functions had to be refactored so they could accommodate resizable structures like strings. Other issues I found with factory were also crossplatform related, as I would find mac didn't exactly play nice with any mistakes that Visual Studio would ignore, though the issues being highlighted helped me get a better footing on where to fix my work on factory.

    All in all, it's been a very interesting month to say the least, a bit of a different take compared to my previous months where I had seen more graphics and math related tasks. This research project that I ended up undertaking to make sure factory was up and operating with ease was great! Now I continue my journey with Gateware into my fourth and final month of Capstone.

Colby Peck - Gateware Week 7: Linux Fixes (and Breaks)

Week seven working on Gateware has been slower and more frustrating than the last six. My first task was to fix LookAtRH (it wasn’t behaving correctly). That was, thankfully, an easy fix (forwards = eye - at in a right-handed system, not at - eye). Following that, I got right to work refactoring the single-header compiler to use a hard-coded list to order the individual headers (I also snuck in some refactoring that should make the future full refactor easier). I actually tend to find refactoring code to be kind of meditative; it can be very zen. I finished that refactor up around the middle of the week, and it was time to get back to the pipeline! I got to work setting up the single-header test project to compile a single header to a temporary location; simple, right? Well, yeah actually. I set up the cmake script to contain a pre-build step for the test project that compiles the temporary single header, and got it working locally within the hour. But once I pushed that to the repo, the linux build pipeline broke. The remainder of my week was spent digging around the linux runner and the linux pipeline, trying to track down what could possibly be causing the issue (reader, I have yet to find it). The project builds and runs just fine on the linux runner when I build it on its own, but the pipeline refuses to work at the moment. SO! Seeing as I’m stumped on the pipeline at the moment, I’ve shifted my focus onto other tasks until I can get some assistance from Lari. Currently, I’m digging into getting the yaml script to copy the temporary header into the final header position iff the unit tests all pass, and following that I’ll be making a flowchart for Gateware’s pipeline!

gtl::factory reaches a testing phase!

     After three weeks of working at gtl::factory, I'm proud to see it finally reach a stable enough state to be tested against other containers such as gtl::ld_vector. I began writing the tests earlier this week, and it compares the speed of four containers: vector, list, factory, and ld_vector.

     The tests time each container on their performance when using push_back, removal functions such as remove or erase, iteration through the container, random removal from the container, and storing large objects. When writing the test for the large object, I discovered some extra minor bugs that hadn't been spotted in the first two weeks of research, but resolved them in a short amount of time. The biggest issue that factory had was resetting the highest_key during remove. Remove should not change the key unless the programmer calls for a remove from the end of the array, or uses pop_back. The key being adjusted caused issues with loop-based removing items from the container. Another issue I came across was the router table missing keys, since it was using capacity instead of highest_key. the router array may not be the same size as the data array but that's the intention so every key is retained when using the factory.

    the test on a large image (2048x2048) is shown in this screenshot. the test adds 40 instances of the image to the containers, iterates, removes 10 random spots of the array, then iterates through the smaller container after removals finish. The speed is recorded using a simple implementation of std::chrono, which I did revisit when writing the test!

    All-in-all, another good week working on the factory, and I'm pleased to see more progress! Soon I plan to implement factory inside of GBlitter for an optimization feature.

Colby Peck - Gateware Week 6: A New Pipeline

 My sixth week here on the Gateware team started out with finalizing a refactor of our float-comparison macros, as well as cleaning up some rather opaquely-worded doxygen documentation. When I first started on the task of patching a small hole in the GMatrix library, I was not expecting to get a task that would balloon out into a full-week ordeal, but such is the way of things I suppose. 

With all of that finally finished, I was ready to move on to my next big task: a pipeline refactor. Lari and I went over the desired final state of the pipeline on Monday. We made several tasks on gitlab defining incremental deployable changes that would ultimately take us to that desired final pipeline (#214-#219). TLDR, we want to set up an automated release process that will: 

• Contain a new ‘release’ branch
• Generate a new single header and run all unit tests on it whenever changes are pushed, then push that single header to the repo if all tests pass
• Generate and push new doxygen documentation any time a new valid single header is generated
• Update the release branch any time a new valid single header is generated

I got started on these tasks on Tuesday. I managed to make a project that tests the current single header and partially integrate it into the pipeline much faster than I had anticipated; it only took about a day. Of course, as soon as I did, I ran headfirst into a bug. It turns out that the mac and linux DUMMY_MAIN projects weren’t testing the single header like they were supposed to, and the single-header compiles into a broken state on Linux. As in, projects that include a single-header generated on Linux will not compile. So, Gateware’s Linux release is broken. It has been for some time, but we didn’t know until now. 

Even if this bug wasn’t more important than refactoring the pipeline, it prevents me from knowing if the pipeline is working on Linux. So I’m diving into the compiler once again. The issue is that the files in the Core directory are ordered differently on Linux than they are on Windows/Mac, and the compiler currently expects them to be ordered the way they are on Windows/Mac. My current plan is to add a patch to the current compiler (hard-coding a specified order for Core), and to create a task for a more comprehensive compiler refactor (to be picked up by myself or someone else in the future). The compiler is currently very brittle; it’s ordering the directories manually and the files within directories are ordered by the number of dependencies they have. It is assumed on good faith that writing the files with more dependencies first will result in a good single header (this is already not the case with Core). A more robust compiler would actually build and traverse a dependency hierarchy, but that’s a much more comprehensive refactor. I don’t like kicking the can down the road, but I can only work on one thing at a time. So for now, patching and putting off the refactor is probably the way to go.

Another week of gtl::factory unit tests!

 While working on gtl::factory I came across multiple edge cases that caused the task to take slightly longer than expected, but I did manage to finish them during the week. Strings and vectors are quite an odd edge case to handle, but wasn't too harsh. The first issue came from the [] operator returning a copy instead of a reference, which made calling functions such as .push_back() or .append() to modify a copy instead of the data itself. Other operators also had memory leaking issues, as the old arrays weren't being dealt with before copying new data into the new array. Most of these edge cases were only an issue with resizable objects, but the process of debugging the code and removing all the leaks proved to be a bit of a challenge, but nothing too heavy beyond chasing down memory addresses.

This was the last instance of a memory leak I fixed before I checked out for the week. the original array was never being deallocated and led to issues with memory leaking.

Colby Peck - Gateware Week 5

I’m just now finishing up the fifth week of my time here on the Gateware team. I ended up accidentally joining a month earlier than I was supposed to (bit of a long story), so this is the end of my first week as a proper Gateware dev where I was able to dedicate my full attention to the project. This first post of mine is going to be a bit lengthy, since it covers 5 weeks of work; I hope you’ll forgive me that, dear reader.

My first task was to add some new functionality to the single header compiler; the ability to specify files to be prepended or appended to the single header. My onboarding process was a bit delayed, so I wasn’t able to make or commit any changes for my first few days on the project. I spent that time reading through the entire single-header compiler and writing down - in plain english - what all the code was doing in a separate document. This ended up being more useful than I first anticipated. When I was later fully onboarded, my first act was to refactor the code I was about to change - to make it more mutable - and I had just written a guide to the whole thing. All told, the refactor and feature addition itself only took about a day’s work once I sat down and really got to it, but it was only so straightforward because I fully comprehended the code before I began.

After the relatively painless work on the compiler, I was due for a task that would have me pulling my hair out at moments - such is only fair. The task in question was to suppress and fix the warnings generated when Gateware is compiled. A simple task to understand, but completing it took a lot out of me. Since Gateware is multi-platform, it’s internally compiled on Windows, Linux, and Mac as part of the automated testing pipeline. This meant that I had three different compilers to work with. Not a huge issue on its own; it’s not hard to find the appropriate preprocessor directives to suppress specific warnings. Then, all I had to do was make sure those preprocessor directives got parsed by the compiler before it compiles Gateware. Not hard for the single-header version: I just got all of the directives into some compiler-specific #ifdef blocks and used the new prepend functionality I had just added (as well as adding some blocks that unsuppressed the warnings to the end with the append functionality). But suppressing the warnings on our pipeline was a much trickier process - I’m feeling exhausted just recalling how much digging around it took. After several days of trying various configurations and placements of preprocessor directives (none of which worked), I figured out how to suppress warnings with the command-line arguments used in our cmake scripts. That did the trick.

Following the warning suppression, I was tasked with some work in the GMatrix library. LookAtLH needed a right-handed variant and some of the projection-matrix-construction functions had gotten mirrored versions, but hadn’t gotten their unit tests. Making the new function was fairly simple, but when I opened up the GMatrix unit tests, I found that all of the tests only tested a single case. So… In addition to adding the new tests, I expanded the coverage of nearly all of the existing ones. Every test got a refactor pass as well. 

I made a real-life transform gizmo out of LEGO to help visualize the various matrix functions and how they were supposed to behave. It was delightful and very helpful.

When I added some new cases for the RotationGlobalY test, I butted up against an edge case in Gateware’s floating-point comparison. Now, float comparison is a bit of a rabbit hole that I won’t take you down; suffice it to say that comparing floats can get messy. Anyways, I ended up refactoring Gateware’s float comparison macros.

Old comparison macros, now deprecated

New comparison macros

They're more-or-less in their final form now, pending discussion and review with Lari. We might need a macro that allows for a hybrid test with two different margins (one absolute, one relative).

Refactoring & Legacy

You may notice that in each of my tasks, I’ve refactored some legacy code. As I’ve gone through with the refactors, I’ve been thinking about the nature of refactoring code in the context of long-term group projects like Gateware. If you’d indulge me, I’d like to lay out my thoughts regarding the matter in the next few paragraphs.

The goal of refactoring is to make code more readable. But why is this important? Because in order to change code, you must comprehend it. The less readable the code, the harder it is to comprehend, and therefore the harder it is to change. Hard-to-read code is hard-to-change code; rigid code. And codebase rigidity makes change (and therefore progress) more difficult. So it’s vitally important for a project’s long-term health that the codebase be made as mutable as possible, and good readability is the chief factor of good mutability. If I know how something works, I can change it. If I have no idea what’s going on, trying to add changes will break things in unexpected ways.

However, refactoring legacy code feels kind of disrespectful. It feels like reorganizing someone else’s workspace when they aren’t looking. My discomfort has some roots in imposter syndrome: “Who am I to say that my factoring of this code is cleaner or better than the one that’s already here?” It feels somewhat judgmental too. “I, the arbiter of clean code, have judged thy code messy.” To address these feelings, let me tell you a story.

When I went home for Christmas last year, I ended up doing some work in my uncle’s wood shop. At one point, he had me use his belt sander (yay, power tools!) Now, this belt sander was a bit of a mess: it still had an old, tattered belt on it and the casing was full of sawdust. When I saw this, I didn’t think less of my uncle for it. I didn’t blame him for leaving his sander in a messy state at the end of a long workday. But I did disassemble and clean the sander before I used it, and I cleaned it again once I was finished.

An ideal codebase gets perpetually cleaner and easier to change. In order to make that happen, all that’s needed is for each programmer to leave whatever code they work on in a slightly cleaner state than they find it (or at least not to make it messier). Conversely, if we do not allow ourselves to clean up the code that we’re working on (via refactoring), the codebase can only get messier. So, “Who am I to say that my factoring is better than the one that’s already here?”  Well, I’m the guy working on this code. For the moment, this shared workspace is my responsibility, and I’ll keep it as tidy as I can. And as for “I, the arbiter of clean code, have judged thy code messy.” …That’s a very draconian way of putting it, but it contains some truth. In order to clean something, you must acknowledge that it’s messy. We must be honest with ourselves if we want to keep our codebase clean. The important distinction is that calling someone’s code messy isn’t equivalent to calling them a bad programmer. The natural state of code is a barely working mess. Any programmer with any amount of experience has learned this firsthand. I write messy code too! But when I am fortunate enough to have the time and energy to clean my code, I do. And when I work with legacy code that’s messy, I clean it. My goal with this practice is to leave everything I touch just a bit better than I found it - and hopefully make the lives of Gateware programmers (myself included) easier.

Creating a unit test from the ground up!

 This week I got to venture into creating unit tests, but instead of modifying an exist test, I had to create the test itself! My task for this week involved finishing an old custom data structure called factory (scoped in as gtl::factory) 

The first section of the unit test checking for the default state of a factory

The data structure was pretty simple, though the iterator class was the only part of the factory that was left unfinished. I did have some struggle getting the pointers to cooperate, as the Data array internally was not in the scope of the iterator. This was solved by making the constructor for the iterator take in the data array, and creating a function for the end user to create an iterator simply by calling the ".iterator()" function.

This week was a pretty simple week of studying up on a data structure. Next week I intend to implement the rest of the unit tests, and add the factory to the GBlitter class for optimizations!