Friday, December 20, 2019

Last GCollision Update (of the year)

In the last update, we have decided to priority batch implementing functions based on what most Gateware users would require. Since that decision, I have confirmed about 5 functions passing their tests. I also had to review which functions would be delayed in order to implement the pre-requisites for them. An additional change is modifying the Separation Distance between two shapes which calculates the distance between two shapes unless intersecting ( in which it would return 0). In many of the function implementations that I explored, it became pretty apparent that there was a lot of value to be had from the squared distance between two shapes thus saving n amount of square roots per call. This would be up to the caller to decide the amount of precision they required from the distance. In order to offer more clarity as well, the name is going to become SqDistance. I also implemented OBB vs OBB which is probably one of the costlier functions in the library unlike sphere to sphere and AABB to AABB which are nearly identical in terms of performance. Anyway, I am going to leave you to have a wonderful winter break and will be posting next year!

The possible results from some of the GCollisions functions
When you rapidly instantiated AABBs with a quick mouse and drag option

GAudio, GAudio and GAudio (and a little bit of GAudio3D)

Month: 2, Week: 4

I was working almost exclusively on GAudio and GAudio3D for the whole month. The huge breakthrough was achieved on all 3 platforms and majority of memory leaks are resolved (no GAudio related memory leaks on Windows and Mac; Linux has some minor memory leaks caused by pulseaudio mainloop functions that I could not find a resolution to). GAudio and GAudio3D now pretty stable and ready for a new Gateware release.

My branch that was based on Jacob's branch was merged into Alpha today pushing Gateware version to 7 due to addition of new library. A lot of work still remains on GAudio and GAudio3D (including some major one like multi-channel issues on Mac and Linux) which I will continue working on in January.

Linux has some very deep stability issues which I cannot reproduce (possibly caused by some internal race conditions but happens rarely) and the volume setting issue is quite complicated one to solve. Pulseaudio doesn't support per-stream volume setting, so I would need to load additional modules to make virtual sound sinks for every single sound/music to mimic that functionality (unless I find a cleaner solution).

Mac has issues with 5.1 matrix to play our surround system from GAudio3D. That would be one of the main goals for me next month. Also I would need to update GAudio/GAudio3D interfaces and rewrite some of the outdated Doxygen comments. A lot of general refactoring can still be done on GAudio systems (external thread-safety and stream/loading functions especially have to be addressed).

Thursday, December 19, 2019

Where am I?

Over the course of this week I've been mostly investigating the final memory leaks on GWindow in Linux. There are only two left but they're from the same source and both the same amount of bytes. 40 bytes, that is. I jumped firmly over the garden wall in my search to unravel these and as I continue I'm feeling like I'm getting more and more lost instead of closer to the answer.

Valgrind gives me a stack trace of the leaks and it reports each bit of memory as advancing from unitTestMain all the way to the very first call of OpenWindow in the GWindow unitTests, enters the function, and disappears.

The very last place the memory can be seen is hitting the XInitThreads function from X11, so I decided to start trying to unravel that knot.

The first day of my search I exhausted a pretty high number of resources to tell me what XInitThreads does, and ended up further into the woods. Both descriptions I could find had the same unhelpful, copy-pasted description.

The second day of my search I exhausted even more resources looking for a definition of the function, only to find nothing by lunchtime. I spent the rest of the day trying and hoping to find something else in the code that would be leaking the memory instead and got further into the woods.

The third day Alex gave me a source for the X11 repo and I downloaded it to start searching for the source code of XInitThreads and after scouring the entire thing the only thing I found was, guess what, more trees.

Today I'm back to trying to drag out the source code from anywhere on the internet. My hope has been gone for a while, this is an awful problem to try to fix. If nothing comes up by the end of the day I may just hard eject from the situation and try to start something new once the break is over. Two blocks of 40 bytes that only leak once isn't the worst memory leak ever, and I'm certain there's something more important I could be doing with all the time I've wasted trying every combination of words to ask Linux for help.

CMake the last C Bender

CMake existed in harmony with Windows, but everything changed when I built to Linux and Mac. The errors were everywhere, it was chaos! Mac wanted c++14 but we're building for c++11, Linux didn't like how CMake was telling it what to do so it got CMake dir errors. With these problems, a new quest begins. A quest for answers, a quest of research.

Overall, CMake isn't so bad to get the hang of. There's a lot it can do and I believe that is where the learning curve comes in to play, but if you know exactly what you want it to do and you understand the dependencies of your project, CMake is not as daunting as it may seem.

Once the errors on Linux and Apple are resolved and the porting process starts to pick up speed, I'll hopefully be able to get started with the CI/CD configuration, and hopefully it makes everybody's life better.

CMake makes C

Week two month two, CMake. I spent several days running through a lot of CMake documentation. When I finally got to testing some of my new found knowledge I only had to write a few lines to get things up and running. Conclusion, CMake is awesome and so simple that it can get confusing easy when trying to debug.

Tuesday, December 17, 2019

Six days of work to display a cat photo

It took me less than three full days to get a working proof of concept for Linux. I thought that was a long time to get something working. I'm going on day seven of experimenting on Mac, and I still don't have a raw pixel array drawing to the screen. I might make a breakthrough in one more day, or it might take another week. With the wild inconsistency in the search results I've found, I can't really say for sure.

I made some adjustments to my Linux proof of concept to better simulate actual usage, since rendering to the pixel array and drawing to the screen will happen on separate threads, and as a result, drawing to the screen doesn't have to wait on rendering to finish. I removed the continual re-clearing of the pixel array and replaced it with just a memcpy call, and the result was a jump from ~300 fps to ~1300 fps on VM. I wasn't able to test this change on actual hardware due to technical problems with the shared Linux machines, but I expect to get performance similar to the existing Windows raster surface.

I decided to focus on NSImage over CGImage for drawing on Mac because the recommendations I've found for drawing images aren't definitive one way or the other, but from my own testing I've found NSImage to be easier to get close to a working state. More specifically, I can get an NSImage object created at all, but I haven't been able to get a CGImage to work at all.

Kai showed me how to change some build settings in Xcode near the end of the day today and thanks to that, I was finally able to get at least something to draw to the screen, but given that it was a JPEG photo of a cat that was loaded from a file, it's still a far cry from real-time BLIT drawing of raw pixel data. I'm getting close, though. Or at least I think I am. If I can find the reason that an image loaded from a file will draw but an image created at runtime won't, I should be able to finally finish dancing the Mac BLIT Mambo without too much trouble, but that's a pretty big 'if'. Progress has been slow on this task, but I am still making progress.

Working executable for the file concatenation tool for all platforms

Platform specific executables are made, for windows i used Visual Studio to build exe, for linux i use CodeBlocks to build exe, for Mac i used XCod to build exe. Each executable works as intended, all produces same result. I will began working on GWindow porting to the new architecture now. My obstacle right now is understand how inheritance for implementation classes work with the new archietecture

Friday, December 13, 2019

Apple seems to not want people to develop for Apple

I knew when I started working on Gateware that there was going to be a steep learning curve to go from Windows-only development to cross-platform, but I didn't expect to have nearly as many issues with it as I've had. I've spent an entire week trying to get a testing window running on Mac to be able to test options for blitting, but it took four days straight to get so much as an empty window running without halting progress through the program.

Getting a test running in Linux was a hurdle, but doing the same for Mac is turning out to be more like trying to jump over a skyscraper. Before I could even start trying to write any code, I had to learn a new language, since Mac uses Objective-C for the most part instead of C++. Official documentation is sparse and next to useless for understanding anything but syntax, and other sources like StackOverflow depend entirely on user responses to questions having accurate, relevant, and up-to-date information, which is almost never the case. It's been hard to even narrow down options to test because of the large number of conflicting recommendations I've found. Most of the code examples provided by people are either too old and use things that are now deprecated or are for Swift, which Gateware doesn't use. It's as if Apple changes everything all the time to make sure nobody quite knows how to write software for Mac outside of Apple.

Something I've proven countless times is that no matter how hard a problem is, the most important part of solving it is persistence. If you refuse to give up and stop looking for more ways to solve a problem, you'll eventually find something that works. My goal is to have a functional proof of concept for both Linux and Mac by the end of next week.

Debug like a champ!

A major part of writing a library is having proper debug tools. Most developers spend the majority of their development debugging or implementing safety that prevents bugs from happening in the first place. This is doubly triply so for the collision detection library. Just looking at numbers is not going to yield that great of a benefit. The computations quickly become out of control with no telling where the data is going bad. Even trivial calculations on a large scale can make anyone's mind go numb. What can you do to debug in a timely manner and also check for edge and corner cases?

A debug renderer. No one wants to just look at numbers changing. That's boring. A debug renderer to draw the data in a way that's more intuitive is useful. It goes beyond useful. It saves your time and your life. To me, it's almost as important as the unit tests and should be growing beside it. I started creating the debug renderer at the beginning of the month and have it printing unit tests for me. I can see the negative and positive test cases and then snapshot the values into my unit tests and move on to bigger things like implementing the functions to pass the unit tests I just created.


GAudio on Linux

Month:2, Week: 3

For the whole week I've been working on Linux side of GAudio. Probably the most complicated platform to grasp in terms of both understanding the base audio api (pulseaudio in case of Linux) and debugging with Valgrind. There were a lot of threading issues and hard pathing issues involved with Linux side, but I fixed most of them.

I can achieve 100% stability running the library (GSound) from Codeblocks, but Valgrind still manages to fail sometimes (it slows down the process tremendously and also running on VM, so this might be the issue). There is still some memory getting hold on by pulseaudio and I have not figured out a solution to it. It is pretty consistent and does not bleed any extra over the runtime, so for now it is safe to say that majority of memory leaks were resolved.

The other issue I just found out is inability to change volume at all which caused all of the sounds play in mono when I attempted to play surround/stereo sound. I am going to explore the documentation more, even though the callback is being called and the volume values are correct, the volume on actual output does not change.

Next week I am going to apply GSound fixes to GMusic and I will explore the volume setting issue and will attempt to achieve multi-channel support on Linux before the holidays.

Porting is happening


With us moving to Gitlab we have officially started Porting. We cranked out the Math libraries and unit tests with relative ease an it all compiles and works (at least on Windows) so that's exciting. With Caio working on cmake we should be able to test on the other platforms relatively soon. This week we also started porting GWindow which is our first real port and is giving us some slight problems. So far the porting itself isn't too difficult but the tedium of it adds up quickly. With us enforcing alot harsher coding standards since this will all be 1 header file its mainly scoping in names and cleaning up alot of stuff.

The Gwindow mambo

So I started this week with the intention of getting Linux's GWindow event system working correctly. Originally it was responding to just a couple of events and then going away, and not responding at all to movement or maximize events. I spent the first several days researching the problem before I started testing and it took me learning that everything was caused by an early loop exit to fix the problem.
After getting the event system to not exit early I fixed the move and maximize conditions and then learned that my fix had re-caused a mass of memory leaks I fixed earlier. I spent the next day creating a new escape method with an atomic bool to flag closing (thanks for the help Lari, it worked great) and that put me back at my starting leak count. Then with Alex's help I snagged his common.cpp leak fix and brought the count down even further and now I only struggle with two remaining blocks leaked.
Overall this week has been pretty great, it started off real rough but I finally feel like I'm helping people in my position and its brought my motivation up well. Anyway, I'm looking forward to another good week of gateware work.

Thursday, December 12, 2019

File concatenation tool made big progress

The file concatenation tool made a huge progress yesterday. I was able to parse all interfaces and its implementations into one single header, currently this is just a working prototype, it is big in file size. My next step in to implement filtering to remove files that has already been included.

Here is a code that preprocesses each file (i.e. replacing hpp with its content):

void DotHFile::Preprocess()
{
for (size_t i = 0; i < includeFiles.size(); ++i)
includeFiles[i]->Preprocess();
unsigned int index = 0;
std::string newRawData = rawData;
for (auto iter = std::sregex_iterator(rawData.begin(), rawData.end(), rgexInclude); iter != std::sregex_iterator(); ++iter)
{
std::smatch match = *iter;
std::string lineString = match.str(0);
std::regex regex(lineString);
newRawData = std::regex_replace(newRawData, regex, includeFiles[index++]->GetRawData());
}
rawData.clear();
rawData = newRawData;
}

Monday, December 9, 2019

Commence the Porting

After a month of research and getting used to the Gateware environment, I've finally been tasked to start working in the new architecture. My first job has been to write unit tests for some of the existing libraries for the new system, I chose to go with GThreadShared. This library is a wrapper for mutex and SRWLocks which allow the user to lock and unlock either read or write critical sections. The first couple tests written are just battery tests to make sure inheritance is working as expected, the second set of tests will be testing the functionality of the locking mechanisms. If all goes well over the next few days, I might be honored with the task to write the unit tests for multi-threading libraries that the Gateware developers will be using for their libraries.

Friday, December 6, 2019

When a project becomes a hydra you have to know how to trim its heads (GCollision Unit Test Phase)

Writing a library for Gateware can be a task that may seem vast and steep. Previously, I had discussed the monstrous interface I created for the Gateware GCollision library and now I will discuss the challenge with it going into the next stage of creating this library which is: dummy unit tests. The GCollision currently has about 1950 functions, but a lot of them are the same functionality overloaded. I imagine about a fifth of that or less are somewhat unique functions. It's a bit hard to tell at the moment because even then it's quite overwhelming, but no matter how trivial some of these maybe they still require unit tests. With each unit test possibly requiring multiple positive and negative tests, you're now looking at a hydra of a project. So, where do we go from here?

I shared my update with the lead software architect, Lari, and he was easily able to suggest the solution to ensure this library will be complete within time. Prioritize completing the unit tests and functions that will most likely be the most used by the users. I know it seems obvious, but I had researched this for a month prior to this step and every function and geometrical representation that I chose had become one of my babies. I didn't want to abandon any of them. Lari had less attachment to them and his suggestion is more realistic. I'm also more healthier now that I can work on batches of the library instead of spending who knows how many man-hours writing unit tests before I even get a function going. Below is a table of the first priority batch which is the test if two objects are touching or not from the shapes: point to OBB.


Thursday, December 5, 2019

GAudio/GAudio3D on Mac (Objective-C memory management)

Month: 2, Week: 2

This month I jumped straight into debugging and identifying issues with GAudio and GAudio3D libraries on Linux and Mac (I fixed Windows side last month). My main task for the last couple of days was attempting to port Windows fix directly to other platforms. And this is not as easy as it sounds.


I started with Linux and made some changes, but I could not gain much information due to a steep learning curve on Valgrind (and CMake). I explored both Linux and Mac code bases and decided to focus on Mac first, as I had access to Mac only in the office.


I never worked on Mac before, so XCode, Mac UI and Objective-C were foreign to me but it was not too bad. I implemented my Windows fixes, and while it fixed the issues on C++ side, it did not fix any of the memory leaks that were caused by Objective-C objects. I found a solution but it was a dangerous one. Later that day I came across a very clean and detailed post about Objective-C memory management which made me realize that my solution is very error-prone. I rewrote the cleanup code, refactored some functions and now GAudio/GAudio3D libraries on Mac are not leaking memory and stable.


"So now the question becomes "how long can I safely use the object before the autorelease pool is drained?" In Cocoa, the pool is drained after every NSEvent is sent. For example, if the user clicks the mouse twice then the pool will be drained in between the first and the second click. This is why it is safe to use an object temporarily, but it is not safe to keep an object unless you own it. If you don't retain your ivar and the user moves her mouse, suddenly your ivar is gone and you're probably going to crash very shortly."

Objective-C Memory Management post: https://www.tomdalling.com/blog/cocoa/an-in-depth-look-at-manual-memory-management-in-objective-c/

Now I am moving back to Linux with a better understanding of GAudio, Valgrind and CMake. It should be pretty a straight forward fix, unless I run into multi-threading problems with Pulse audio (Linux), like I did with XAudio2 on Windows.









Starting the Port


This month we get to start the porting job and I'm excited about it. Learning how to make unit tests has been a fun adventure since the last time I wrote some unit tests was in PG2. Not alot has really happened I'm just super excited to start the meat of the porting process

Linux is strange

Over the last week I've been digging deep into GWindow on Linux to find and correct memory leaks that were pretty wound into it. Plus it all being on Linux meant some strange complpications came into place, such as spending a day proving that the library was the source of a leak only for me to be unable to replicate the error the very next day. All that combined with the sheer quantity of memory leaks made it difficult to trace case by case instances and so I struggled.

As I continued over the week I found myself getting more and more accustomed to the memory leak tool valgrind and reading it got easier. I learned new functionality for it that allowed me to trace problems more specifically and started getting a more solid idea of just how GWindow's functionality is implemented and where the inconsistent pieces were. It turned out that Linux's event system used a different heirarchy than Windows or Mac and so threads were being launched but never checked on again. From that point I was able to start cleaning up small leaks and on the same path found a way to stop the vast majority of Linux's GWindow leaks. In my solution with only GWindow enabled there were still two leaks left to handle, one not relating to GWindow, but I have a new assignment now so they'll wait another little bit yet.


Unit Tests for new GatewareX interfaces

I begun writing unit tests for GEvent interfaces: GEvent, GEventGenerator, GEventReceiver, and GEventQueue using catch2 which is an open source header-only framework for creating unit tests. Here are some code:
TEST_CASE("GEventGenerator core method test battery")
{
GW::GReturn gr;
GW::CORE::GInterface emptyInterface;

unsigned int listenerCount = 0;

GW::CORE::GEventGenerator eventGenerator;
SECTION("Testing Empty proxy method calls")
{
REQUIRE(eventGenerator.Observers(listenerCount) == GW::GReturn::EMPTY_PROXY);
REQUIRE(eventGenerator.Push(GW::GEvent()) == GW::GReturn::EMPTY_PROXY);
REQUIRE(eventGenerator.Register(emptyInterface, nullptr) == GW::GReturn::EMPTY_PROXY);
}

SECTION("Testing Creation/Destruction method calls")
{
REQUIRE(eventGenerator.Create() == GW::GReturn::SUCCESS);
REQUIRE(eventGenerator);
eventGenerator = nullptr;
REQUIRE(!eventGenerator);
REQUIRE(eventGenerator.Create() == GW::GReturn::SUCCESS);
}

// Now we can begin checking valid proxy method calls
REQUIRE(eventGenerator.Create() == GW::GReturn::SUCCESS);
SECTION("Testing Valid proxy method calls")
{
REQUIRE(+eventGenerator.Observers(listenerCount));
REQUIRE(+eventGenerator.Push(GW::GEvent()));
CHECK(+eventGenerator.Register(emptyInterface, nullptr));
REQUIRE(+emptyInterface.Create());
}
}

So far I've been pretty successful on writing new unit tests, the biggest thing here is that debugging through the unit tests gives me more understanding of the new event system. Which I will have to tell Chris about so he can document on it.

Monday, December 2, 2019

Hello

My name is Jadon Lindburg and I will be working on 2D rendering.