Designing a Multiplatform Touch Interface Part 1

     With more people using mobile devices and touchscreen laptops, developers need to interface with the devices' touchscreen API. Gateware should be doing the same thing if it hopes to support mobile devices. One of the big issues with designing a touch interface is the different ways that devices store the touch information. A good rule is that an interface should provide as much functionality to the user that they should need to use.

    When designing this interface one should consider what major platforms support touch and which are most popular. iOS and Android devices are likely the most common touch devices with touch as a primary method of input. Windows and macOS also support touch input, but they are secondary input methods, as most Windows and macOS devices do not have touch input. Other notable devices are Nintendo Switch, Smart Watches, and embedded devices. As such, I propose this touch interface mainly for iOS and Android, but with expansion options for those other devices. 

    All of these different platforms have different information sent to the developer when the screen is touched. I have created a struct that supports the main information that most developers would need.

enum class GTouchState : unsigned int {
	TOUCH_BEGIN			= 0,	// Finger just hit the screen.
	TOUCH_MOVED			= 1,	// Finger moved from last position.
	TOUCH_END			= 2,	// Finger lifted off the screen.
	TOUCH_STATIONARY	= 3,	// Finger is on screen but hasn't moved.
	TOUCH_CANCELLED		= 4,	// System stopped tracking the touch.
	TOUCH_COUNT,
};
 
struct GTouchData {
	// Current location of the touch in screen coordinates
	float x, y;					
	// Change in location in screen coordinates
	float deltaX, deltaY;		
	// First recorded location of the touch in screen coordinates
	float initialX, inititalY;	
	// Change in time between the current position and delta
	float deltaTime;			
	// Pressure of the touch
	float pressure;			
	// Number of consecutive touches in the same general area
	unsigned int tapCount;		
	// What touch is this? Always less than the number of total touches on the screen
	unsigned int index;			
	// What state is this touch in?
	GTouchState state;			
};

iOS and Android support more information, but most of the information not included here is for stylus support, with information on how the stylus is aligned with specific axes. Most Gateware users won't need this information, as such I have omitted them to give some simplicity to this interface.

    Another part of the API is common support for touch gestures. Many mobile users are used to common touch controls to get around their mobile device. Swipe up to go down on a page, pinch to zoom out, and two finger spin for rotate, along with others are just some of the gestures that this API should support for a better experience for the user. As such here are the common gestures that most platforms recognize.

Single Tap

Double Tap

Rotation (Two finger)

Swipe

Pan (Two Finger)

Long Press

Pinch

    Here is the end for now. Expect another part to this blog once I have figured out what interface would be best for Gateware.

References:

    Touch data: 

        https://developer.apple.com/documentation/uikit/uitouch?language=objc 

        https://developer.android.com/reference/android/view/MotionEvent.PointerCoords

    Gesture recognizers:

        https://developer.apple.com/documentation/uikit/uigesturerecognizer?language=objc

        https://developer.android.com/reference/android/view/GestureDetector

 

     

Porting the Audio Libraries to UWP

The Task:

    Getting CMake to properly work with Gateware Win32 and UWP projects was no simple task but since it was finally done, I was able to start porting Gateware to UWP in earnest. I was given the choice of what to start with, I chose the audio libraries as it seemed like an (almost) one to one port was in the cards. I was right, but it didn't come without its issues.

The Problem:

    My initial evaluation of the audio libraries had me changing just a few functions here and there that were not available for UWP apps. This assumption did turn out to be correct but with some caveats. I based all of the UWP implementation of the GAudio libraries off of the Win32 ones as they are all build using XAudio2 which is compatible with UWP. After changing some key functions that were not compatible, ex. CreateFile() to CreateFile2(), I got everything compiling. Success? No, not really. When I would then run the Unit Tests, the program would call abort() from somewhere.

The Solution:

    The first thing to do was to find where abort() was being called from. Using the debugger I was able to track it down to the CreateFile2(). I was very surprised to find this as the documentation for the function says it is compatible with Windows Store apps. After reading through it again more thoroughly, I found this "When called from a Windows Store app, CreateFile2 is simplified. You can open only files or directories inside the ApplicationData.LocalFolder or Package.InstalledLocation directories." Knowing that I have not done anything with file I/O yet with any of the libraries, I knew pretty quickly this was the issue.

    Getting to the installed location of the app is done pretty easily with WinRT calls so that was not the hard part. Getting the Unit Test Resources, such as the audio files, in the installed location is what stumped me for the longest time. This is easily done in Visual Studios by just including it in the project and setting it to "content" in the properties but this needs to be done before a dev even opens up the project. Back to CMake I went as I already had the Asset folder being put in as content for the UWP app, so I figured it would not be any different for another folder. I was right in that thinking, it just took a lot of trial and error to get it just right.

    With that out of the way, I was able to get the sound files from the installed location and everything compiled and ran perfectly.

   As of right now, all audio unit tests pass.

The Extents of Controller Support

During testing last week, I had one of the manual Unit Tests fail for GController on the Mac. It was a test of the down button input on a controller, and no matter how much I pressed the button or reran the test, the input wasn't being detected. I found this very strange because I had tested it not too long before, and the test passed without issue. The difference between the time before and last week is that I was using a different controller.

Different controllers, different codes

The problem turned out to be a difference in input codes for the D-Pad between the two controllers. The generic controller I initially tested with used codes of 0 - 360 for the D-Pad. The PS4 controller used codes of 0 - 8. This discrepancy was easy to fix by adding support for both ranges of input codes. However, it got me wondering about the rest of the controller input.

Unit Test upgrade

The current Unit Test sampled a small portion of controller input, only conducting 11 input checks. On Monday, I made modifications to support the testing of each input supported by GController, totaling 40 input checks. I also changed the way the test behaved to speed up the ability to get through the test. The changes to the test were necessary to find out if there were more issues with controller inputs.

How a Unit Test is written matters

The first thing I discovered with the new Unit Test is a range problem with the trigger inputs. On Windows, the trigger inputs gave a value of 0 - 1. On Mac and Linux, the trigger inputs gave a value of 0 - 5. This difference was never caught during the original Unit Test because it looked for a value greater than 0 to satisfy the test's trigger is pressed state check. In the new test, the value must be exactly 1. The actual values both my controllers reported was 0 - 255. Initially, I thought dividing by 255 to normalize the value would be sufficient. I eventually discovered that I had one other thing to consider, how the device is connected to the computer.

Different connections, different values

When connected wirelessly, the trigger input of the PS4 controller gave values of 0 - 255. When connected via USB, the trigger input of the PS4 controller gave values of 0 - 315.

 

Thankfully, IOKit has a function for getting the maximum value of the trigger. Using that, we can futureproof the normalizing of the trigger input values so we will always have a range 0 -1 regardless of the controller or connection type. A solution like this is needed for the Mac with the controllers I have tested so far. It also may be necessary on Windows and Linux implementations as different controllers are tested. At this time, a solution like that isn't implemented on those platforms.

Different connections, different identities

Linux is sensitive to connectivity as well. The PS4 controller input mappings are different from the generic controller. When the PS4 controller is connected via Bluetooth, Linux identifies it as a Wireless Controller. Before, GController would use generic mappings for a controller with this identification. With the wrong mappings, the square and triangle buttons on the PS4 controller are swapped. I changed the code so that mappings for PS4 controllers are used when a Wireless Controller is identified. This may be an issue if other brands of wireless controllers use different input mappings, but I don't have any other controller to test that theory.

It's not over yet

Next week, I'll be able to test an official Xbox controller with GController. The library was designed with this type of controller, so I don't expect any issues. However, I wonder if there will be a difference depending on the connection type used. We'll see.

Sidenote

I tried using the steam controller on Linux to test GController. When connected via USB, some of the inputs worked, some were unresponsive, and some caused the Unit Tests to crash. I tried installing the driver for it by installing Steam and connecting it via Bluetooth. Unfortunately, that ended up crashing Linux and causing issues with X Server. I managed to get X Server working again, but it ended up changing my entire Linux GUI environment. If anyone tries this, beware.

MacOS High Sierra: Security going against the developer

The first major roadblock I faced was getting Vulkan to compile, install, and run on MacOS so that I could test with Gateware. This Vulkan install would make developing on iOS much easier for the graphics libraries, as much of the same code could be used with MacOS. The main problem was security, or rather user permissions. As of MacOS High Sierra, the user no longer has permission to write to the "/usr/local" directory. Vulkan uses a Python script to install to your local machine, and this script assumes a few things, one, that you already have a couple of folders made in the "/usr/local" directory, and that those directories can be written to by a user.

The solution that I ended up using was not a very clean one. I had run the script many times, looking out for the error messages, and changed my folder structure and user permissions until I got the installer to install Vulkan. First the installer wanted a folder at the directory "/usr/local/include" to be created, but there is a problem, the average user cannot make folders in the "/usr/local/" directory. So I had to change the permissions to allow myself to change the folder structure there. I ended up creating the folder and when I ran the Vulkan installer again, I found that it wanted more folders to be created, first the "include" folder, then the "library" folder, then the "bin" folder, and so on. These errors would only appear one at a time, after I created each folder. So the process went like so: Run the installer and see what folder needed to be created next, then create the folder, and repeat for every folder that the installer will make in the "/usr/local/" directory. After all of the folders were created, then the installer would give errors such as "/usr/local/library/ directory permission denied" and I had to change the permission level of all of the folders that I just created. Eventually I had every folder created and Vulkan would then finally install properly. All that was left was to build and run Gateware and get the Unit Tests working on Mac. 

Overall this experience has taught me a lot about the permission system on Unix based machines. I did not know much about it until this experience requiring me to go and research many commands to do what I needed to do. Getting Vulkan on MacOS should enable me to develop graphics libraries on iOS much easier, as I wouldn't have to write a new graphics library from scratch. Now I should be able to reuse at least some of the code from the MacOS implementation of Vulkan to get things going.

CMake and C++/WinRT Pt. 2

     Another slow week but not due to lack of work. I've been stuck in CMake again for the last week as it is very crucial to have it working correctly before we start porting Gateware over to UWP. Last week had me getting CMake to compile a C++/WinRT project in a Gateware test environment. While I was successful, it was not without it's faults.

The Problem:

    The first problem that became clear was that the library that Gateware uses for unit tests, Catch2, was not giving output for the tests it was running. UWP works fundamentally different from anything Gateware has seen. UWP is not able to launch windows dynamically from a console application like all current supported platforms of Gateware. This means that a console is not present, therefore no console output.

    This led to the discovery of another problem. The first thing that was done after discovering that there was no output was to run through the code of Catch2 to see where it was outputting the test data and see what alternatives were available. That is when it was discovered that I could not debug in this project. It was possible in other C++/WinRT projects that Visual Studio provided, but not in the one CMake made.

 The Solution:

    While the debugging problem was discovered 2nd, it was important to solve it first so that the output problem can be found and fixed. After many hours of Googling and going over Stack Overflow forums, it turned out to be a one line fix. It was one linker debug flag that needed to be set by CMake, DEBUG:FASTLINK. 

    With that out of the way I was able to dig into Catch2 to find where it was outputting the test information. I narrowed it down to an if statement checking a config data structure that is never set in the current implementation of our tests. By changing a string in that config structure to "%debug", it used used OutputDebugString() instead of cout. OutputDebugString() displays a string in the debug output window in Visual Studios. With the test session now using the config struct, it started outputting test output data to the debug output window for devs to read. This now allows us to know what libraries are going to need ported and which ones are going to work as is.

To Do:

    I am still currently trying to get the UWP unit tests and the Win32 unit tests in the same solution via CMake. This has proven to be very difficult as the current version of CMake does not support different target platforms in the same Visual Studios solution.

Ghost Windows & PS4 Controller Input

Gateware is going through its last round of bug fixes before version 1.2a is released. The hope was to have the release ready at the start of this past week, but a few new bugs and several failed manual tests set us back a bit. These issues have the potential to cause grief to any developer that uses Gateware. Having them fixed will ensure developers have a consistent, hassle-free experience with Gateware.

New bugs

During all this testing, I found several new issues:

• On Mac, GWindow tests sometimes cause a closed window's outline to remain on the screen. 
• On Mac, GController does not detect the down button of a PS4 controller. 
• On Linux, GController doesn't respond to Xbox controller connection events. 
• On Windows, GInput tests do not pass the scroll-wheel checks. 

It wasn't immediately clear to me how to fix these problems. Other minor issues came up during the tests, but they were straightforward enough to fix on the spot.

Ghost windows

The shadow left behind by a closed window looks ghost-like to me. It would also stick around or "haunt" the desktop until all Unit Tests had finished.

I found the Create() and Reconfigure() Unit Test I made last week was the only place the problem occurred. I narrowed it down further to it happening only when the window is reconfigured to or from fullscreen. I'm not certain what the root cause of the issue is. However, if the Unit Tests are slowed down, thereby creating more time between window transitions, each window closes without leaving behind a shadowy remnant. Slowing down the code causing the transition to and from fullscreen fixed half of the problem. Waiting for the window to finish resizing before styling it, fixed the other half. I'm not satisfied with the solution as it is difficult to follow. There must be a better way to handle window style changes on the Mac, but I haven't found it yet. For now, the current solution works.

Mouse move crashes

Just as I was running my last round of Unit Tests to verify my ghost window fix, I found another bug from something I worked on in the past. Towards the end of the Unit Tests, a window is created for GBufferedInput. If your mouse happens to move within that window while it is on screen, the program crashes with an EXC_BAD_ACCESS error. The error seemed similar to the EXC_BAD_ACCESS errors I was getting last week with GWindow. The error occurred when the window was deallocated before an overridden NSResponder function like windowWillClosed() had been called. In this case, the window was being deallocated before mouseMoved() had been called. Luckily, this was an easy fix; stop mouse events before closing the window.

PS4 directional-pad partially supported

I would not have noticed this bug if the Mac driver for my wired Xbox controller was still working. Thankfully, connecting the PS4 controller to Mac with Bluetooth is easy to do, requiring no additional driver downloads. I know the D-Pad on the Xbox controller is detected fine because I tested it with these Unit Tests a few weeks ago. After the PS4 controller failed the down button test, I modified the Unit Test to check the other directional buttons. I discovered the up button was the only D-Pad button that was detected. GController_mac.hpp has a hefty amount of code checking the input of the controller. Thankfully, all the code I needed to modify was in a big switch statement laid out sensibly enough that I could make modifications easily. After outputting the raw controller D-Pad values, I added them to the directional code already in place. Afterward, the D-Pad passed its Unit Test.

More to go

Even though the D-Pad problem is fixed, there might be more issues with PS4 input. Updating the current controller Unit Test to test more controller input values would verify controller input is detected correctly. There is also the controller connection issue on Linux and the GInput scroll wheel detection on Windows that I haven't gotten to yet. After all that is done, Gateware users will finally get access to version 1.2a and all its glory.

CMake and C++/WinRT

    Not a long blog post this time as not a lot has happened in the beginning week of month 2, as I was still finishing up and incorporating feedback into my final draft of my research paper while also packing up and flying back to Florida. That being said, I was still given a task to complete.

The Problem    

    CMake is an open-source, cross-platform family of tools to build, test, and package software (CMake, n.d.). Because of CMake's cross-platform capabilities, it is used for the distribution of Gateware. That way, whatever platform an end-user is on, a project file is compiled and made correctly for them to use and develop on. My task for this week was to make a CMakeLists.txt, the file that CMake uses to set up the IDE project files, that will make a C++/WinRT project. The problem that made itself evident very quickly was the very little documentation available on how to do that task.

The Solution

    After many hours of scouring over the internet, I found a reply to a GitHub issue post that claimed to have done was I was searching for (Lindfors, 2017). They linked to a CMakeList.txt file that they had made, and it did appear to have everything that would be needed for a C++ UWP Visual Studios project. The only thing that appeared to be missing was the C++/WinRT NuGet package that is required to link to the WinRT APIs.

    Taking that post as inspiration and something to bounce off of, I made my own version of it using the code from the C++/WinRT Core Application template that Microsoft provides. This worked great, the only issue was that the NuGet package for C++/WinRT was not installed, just as I suspected. Because it wasn't installed the project did not compile, but it would once I manually installed the package from the NuGet Manager. While this worked, this creates more work for the end-user. The less work the end-user has to get their code running the better, so I set out to fix this issue. 

     This led me to finding out that NuGet has a restore feature built right into Visual Studios. All I needed to do was make sure to include a packages.config file in the Visual Studio solution that CMake compiles. That file lets Visual Studio know what NuGet packages are installed. It checks that file every time the end-user builds the solution and re-installs any missing packages if need be. Including that file let Visual Studios install the C++/WinRT NuGet package without doing so manually, therefore fixing that problem.

References

CMake. (n.d.). CMake. Retrieved September 8, 2020, from cmake.org

Lindfors, J. (2017). Leverage CMake to generate project using cppwinrt. Retrieved on September 2, 2020 from https://github.com/microsoft/cppwinrt/pull/126/files

GWindowStyle Flag Issue

Background

While working on the WINDOWEDLOCKED flag, a feature to lock a window's size, I found an issue on Linux when creating a GWindow using either the WINDOWBORDERLESS or FULLSCREENBORDERLESS flags. When calling Create() with one of these flags, the program generated a "Bad Match" error and crashed. At the time, I added the issue to GitLab and finished up with the feature I was working on originally. Last week, I came back to the issue and found that the problem was more widespread than just the two flags.

A problem on all platforms

GWindowTest.hpp was missing a Unit Test that called Create() with each existing style flag, so I made one. After running this test on each platform, I discovered the results of using some of the flags varied. While the program didn't crash on Windows, using either of the fullscreen flags did not produce a fullscreen window. Similar problems were on the Mac. After creating a spreadsheet to track all the flags with issues, I considered there might be even more problems when using the second function that uses style flags, ReconfigureWindow(). The table below shows the results of calling Create() with each flag and if it is visually correct on each platform.

Style flag issues when calling Create()

More problems

ReconfigureWindow() can be called anytime after Create() to change the style of a GWindow. I saw a potential issue with using different combinations of calling Create() and then ReconfigureWindow(). My theory was that creating a window with one flag may affect how a window is reconfigured with another flag. To ensure all windows restyle correctly, I made another Unit Test for all the possible combinations of calling Create() and then ReconfigureWindow(). There are six flags tested against each other, 36 tests total. Running the test on each platform revealed more issues. Each table below is for one of the three different platforms Gateware runs on. The left column is the flag used when calling Create(). The top row is the flag used to call ReconfigureWindow() after calling Create(). The tables below show which combination of flags produce visually correct results on each platform.

Windows working flags calling Create() then ReconfigWindow()

Linux working flags calling Create() then ReconfigWindow()

Mac working flags calling Create() then ReconfigWindow()

Windows and Linux

Fixing the issues on Windows and Linux didn't take much time. I already had familiarity with those GWindow implementations from working on the WINDOWEDLOCKED and SetIcon() features. The tests on these operating systems are running on recent computers. As a result, the turn around time of making changes and testing happens quickly. The Mac was a more involved process. 

Mac

On the Mac, the style flag issues with calling Create() had straightforward solutions. However, fixing the problems calling ReconfigureWindow() on the Mac was not as easy as the other platforms. The Mac I am using is from 2012, running the latest version of macOS. As a result, it is a significantly slow computer to develop on. Building and running Gateware takes anywhere from 1 to 10 minutes. On this platform, the biggest issue I faced was with all the animations whenever a window changes its size or position. I have found that making changes to a mid animation window produces strange results, especially to or from fullscreen. 

For example, a fullscreen window cannot immediately minimize. The window must be brought out of fullscreen before it can minimize. If the window is transitioning out of fullscreen when minimize is called, something strange happens; the window will appear in the dock as if it had minimized; however, the animation of the window transitioning from fullscreen will continue to a windowed state that is visible on the desktop. The minimized version of the window will come out of the dock and merge with the desktop window.

There is a way to stop animations. However, this only causes more strange behavior. On the Mac, fullscreen windows exist in their own space beside the desktop. With animations turned off, bringing the window out of fullscreen and minimizing it will work as expected, but the space it existed in will remain. This behavior is likely because when animations are off, Mac developers are expected to handle specific actions that are normally performed for you. I haven't found a way to delete empty spaces, and it appears macOS developers have no way to do so at this time. Running through all of the Unit Tests results in multiple unoccupied spaces that aren't deleted until the program stops.

The solution I came up with to minimize a fullscreen window is to leave animations on, wait for the window to exit fullscreen completely, and then minimize the window from there. This solution works nicely. Unfortunately, I have to return SUCCESS from ReconfigureWindow() even though the window hasn't been told to minimize yet. However, I don't see any other way to go about it.

Most of the flag issues on the Mac were simple fixes that could be addressed with general solutions. The rest required more specific solutions to handle each particular case. The last eight flag situations I worked on took twice the amount of time to work on than the other 41. As ReconfigureWindow() began styling windows more visually correct, I started getting errors from other Unit Tests, adding to the existing list of issues.

Even more problems on the Mac

Calling Maximize() immediately followed by Minimize() causes the program to crash with an EXC_BAD_ACCESS error. I believe this might be a threading issue. The error doesn't always happen, which is a trait of threading issues. I think this issue is caused when trying to change a windows state while it is in transition to another. I've found that this can be avoided by waiting for the window to maximize fully before calling Minimize(). 

Another new Mac issue is the debug message "not in fullscreen state." This is being reported whenever I try to toggle out fullscreen while the window is transitioning to fullscreen. I haven't found a way to stop the transition or force it to go back. Overriding animationResizeTime() - a method of NSWindow - so that it returns 0 helps to some degree. Although I haven't noticed any differences in the window's transition animation timing, I receive less debug messages. After testing it further, this might be causing windows to stick around long after their Unit Test is over, so I won't be using this.

One last Mac issue is with IsFullscreen(). It reports when a window is fullscreen. However, it can report a window as fullscreen even though it is still transitioning to fullscreen. This misreporting is potentially an issue if a program needs to be sure the window is fullscreen before moving on. This is something that I discovered while looking into the problems I mentioned above. The current IsFullScreen() implementation checks if the window has a fullscreen flag set, which is done when toggleFullScreen() is initially called. I tried modifying this to also compare the window size to the size of the screen, but this is not reliable either. The window can be the size of the screen and still be in its transition animation. The best option might be returning a boolean value that changes based on fullscreen events. This solution would change the function's behavior from returning true when the window is on its way to fullscreen, to returning true when it is fullscreen. However, this would require further testing because the Mac implementation of GWindow will behave differently than how it has been. I also don't know if any other Gateware libraries use IsFullscreen(). If so, they might be affected as well by the changes in behavior I described.

The Mac solution

Waiting for each transition and style change to complete fully avoids any issues. I accomplished this by transitioning the window to a neutral state before moving on to its final state. This only needs to be done when the window is fullscreen or minimized. When that is the case, the window is transitioned into a windowed state (the neutral state) first and then brought into its final state. This behavior is not how the Windows and Linux implementations of GWindow work, but with the Mac this is the cleanest way to it. Especially with the Unit Tests that open, close, and resize windows at blazing fast speeds. 

Conclusion

I may have said it before, but Unit Tests are very helpful in uncovering issues with implementation. Catching these problems now means developers who use Gateware won't experience them later. Although I struggled more than I would have liked to on the Mac issues, I gained valuable insight into how NSWindow works and how to work with it.

The Beginning of a Journey

 The iOS Revolution

    Hello Gateware. I am Jacob Morales, and I will be on Gateware over the next few months. I just started as a developer for Gateware just two days ago. At this time, I have gotten Gateware built and running on two machines, one running Windows and one running MacOS. My goal is to port Gateware over to iOS, so that it could be brought to iPhone, iPod, and iPad, so that potential customers could use Gateware on those platforms. I still need to do more research on getting Gateware onto iOS. There is a lot still unknown, so I will get back to my research.