In the inevitable run-up to Xamarin Forms tumbling slowly off of a cliff face our attentions turn towards alternative technologies to usher us into a new golden age. I’ve seen a good number of developers transition to things such as the classic Flutter/Dart combination. As far as Microsoft offerings, the next evolution for cross-platform application development is .NET MAUI. As we get to grips with this, one of the questions to recently emerge is ‘what options do we have for automated testing?’. The answer appears to be…not a great deal (yet), which is to be expected at this stage in the game.
There is still a good deal of debate as to how this will ultimately pan out; with the future of ‘Xamarin.UITest‘ test suites in the balance (it isn’t all that clear what will happen). However, there are a few (very early) conceptual pieces starting to appear, which look fantastic. I’m going to explore one of these just for kicks!
NOTE: This isn’t authoritative information; more just a drive-by on what I can piece together over a couple of hours (and a few coffees). Also, mobile app development is not a primary wheel house of mine, so it should be interesting! Grab a ☕ and settle in!
Maui.UITesting
This Maui.UITesting repository is an incredibly interesting place to start. It makes use of a ‘.NET Interactive Workbook‘ via the ‘Polyglot Notesbooks‘ VS Code extension. I have long been a fan of the C# Interactive Window in Visual Studio and love any kind of workbook-style flow; enabling you to execute sections of code and see results directly, in the editor.
The included ‘.NET MAUI UI Testing – HackWeek‘ video gives a taste of what is ahead.
Let’s Get Forking!
To get started, I forked the Maui.UITesting repository and cloned this locally, just to experiment. The structure of the solution is as follows:
In addition to the physical driver source code, and useful extension methods, the solution also includes a sample application. Some of this has certainly moved on from the information you’ll see in the YouTube video.
To build this solution and to use Polyglot Notebooks, the .NET 7 SDK needed to be installed:
As shown, Visual Studio Code does allow you to install the relevant SDK after installation of the Polyglot Notebooks extension. I cheated here and opted to get my hands on it by updating Visual Studio 2022 (it’s been a few months, so a version update was on the cards anyway).
An Aside – Polyglot Notebooks
You’ll want to get your mitts on Polyglot Notebooks (not only for sampling UI testing but just because interactive/workbook-style antics are plain fun!). This can be installed from the Extensions Marketplace in Visual Studio Code (as noted above, this is in preview and the .NET 7 SDK is required):
To quickly illustrate how this works, via the command palette, we can create a new Jupyter Notebook:
Then, in a very unimaginative way, I constructed a mixture of inline C# snippets interlaced with some markdown comments (for flavour) that generate a list of integers, inspect the data, reverse the elements and then re-inspect the data:
You are free to re-run steps, execute code from the target step and all steps above/below it or re-order statements with ease. This stuff is simply brilliant. Building shared, collaborative notebooks just for conceptual work is a massive boon in my eyes – I’d love to see this used more. I’ve seen this idea used effectively using Azure Data Studio, with SQL notebooks.
The interactive UI test samples are using a preview file format (‘.dib‘), with an ongoing discussion happening here about the future of this. If you are after a much better example of utilising notebooks, Andrew Lock’s brilliant post will do a stellar job:
Exploring .NET interactive notebooks with VS Code
Build Errors
There were a couple of hurdles as far as compilation goes, but otherwise pretty plain sailing:
C:\Program Files\dotnet\sdk\7.0.102\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Publish.targets(299,5): error NETSDK1094: Unable to optimize assemblies for performance: a valid runtime package was not found. Either set the PublishReadyToRun property to false, or use a supported runtime identifier when publishing. When targeting .NET 6 or higher, make sure to restore packages with the PublishReadyToRun property set to true
The ‘dotnet restore‘ can be retrofitted with the parameter mentioned, so I went ahead and ran the following command:
dotnet publish -p:PublishReadyToRun=true
Following this guidance, I was merrily on my way: Publishing a ReadyToRun Project Guidance.
From a compilation standpoint, we were now up and running. It should also be noted that the driver is using a preview of the ‘Microsoft.WinAppDriver.Appium.WebDriver 1.0.1-Preview‘ package:
warning NU1701: Package 'Microsoft.WinAppDriver.Appium.WebDriver 1.0.1-Preview' was restored using '.NETFramework,Version=v4.6.1, .NETFramework,Version=v4.6.2, .NETFramework,Version=v4.7, .NETFramework,Version=v4.7.1, .NETFramework,Version=v4.7.2, .NETFramework,Version=v4.8, .NETFramework,Version=v4.8.1' instead of the project target framework 'net6.0'.
The idea for me here was to rush headlong, bull-in-a-china-shop fashion, to make progress (but worth pointing out that we’re in ‘fluid’ territory, so your mileage may vary if you decide to pick this up).
Grabbing a Local NuGet Package
A one-shot viewing of the YouTube video illustrated that I should just be able to grab and trial this using a locally generated NuGet package. The assembly of choice looked to be ‘Redth.Microsoft.Maui.Automation.Interactive‘, so within Visual Studio I used the standard pack command:
dotnet pack
Grabbing the package from the relevant location on disk (‘Maui.UITesting\src\Interactive\bin\Debug‘) I knocked up a folder on the C drive called ‘LocalPackages‘ as a staging area for testing (dropping the generated ‘nupkg‘ file in this location, which was named ‘Redth.Microsoft.Maui.Automation.Interactive.1.0.0.nupkg‘).
.NET Workbook UI Test Sample
To get underway, I generated a new file called ‘sample-ui-test.dib‘ in my staging ‘LocalPackages‘ folder on the C drive. I then opened this file in Visual Studio Code. The first commands deal with bringing the relevant NuGet packages into scope for the session; which is where I hit an initial snag. I ended up, at the time of writing, having to include a couple of extra packages (which were obtained from the relevant ‘bin\Debug‘ folders on disk):
#i "C:\LocalPackages\" #r "nuget:Redth.Microsoft.Maui.Automation" #r "nuget:Redth.Microsoft.Maui.Automation.Driver" #r "nuget:Redth.Microsoft.Maui.Automation.Interactive"
So far, so good! A further command is noted here (‘#!uitest‘), which I didn’t inspect but may circle back to at a future point to see what is available.
With this in place, we now need to construct a driver to execute UI tests. I am running the Android variant, so you just need to steer the ‘AppFilename‘ method to the signed apk file, in the ‘bin/Debug/net7.0-android‘ folder, as in my case. You also need to specify a device (I’m using an installed Android emulator at the specified API version):
var driver = new AppDriverBuilder()
.AppFilename(@"C:\Users\lgrin\source\repos\Maui.UITesting\samples\SampleMauiApp\bin\Debug\net7.0-android\com.companyname.samplemauiapp-Signed.apk")
.Device("pixel_5_-_api_33")
.Build();
I couldn’t quite remember all of the Android Virtual Devices I had installed, not being a super seasoned mobile app dev. The ‘-list-avds‘ command can get you a valid identifier:
Executing this will start the emulator, and we are finally getting to the juicy bit! Right on cue, I was an idiot – when executing the tests (using ‘await driver.Start()‘) you may encounter a further error:
await driver.Start();
This turned out to just to be because I hadn’t referenced the signed variant of the apk file when calling the ‘AppFilename‘ method; so one to watch out for.
Executing ‘await driver.Start()‘ should start the target application within the emulator.
It’s possible to take a screenshot during test execution, storing this with a designated file name. So, upfront, I executed this snippet to create a value for a variable called ‘screenShotFilename‘ using a formatted string from ‘DateTime.Now‘ (which I can use in the subsequent step where the tests physically execute):
string screenShotFilename = $"login_fields_completed_{DateTime.Now.ToString("ddMMyyy_HHmmss")}.jpg";
screenShotFilename
Lastly, with the application running, we use the driver to interact with elements; whether that be completing text fields or tapping on buttons, for example:
await driver.AutomationId("entryUsername").First().InputText("xamarin");
await driver.AutomationId("entryPassword").First().InputText("1234");
await driver.Screenshot(@$"C:\LocalPackages\{screenShotFilename}");
await driver.By().Type("Button").ContainsText("Login").Tap();
await driver.AutomationId("entryUsername").NoElements();
The basics are there for finding elements by facets (such as a ‘button’ that contains the text ‘Login’) or just by a designated ‘AutomationId‘. The ‘InputText‘ and ‘Tap‘ extension methods here perform the actual interaction step on the elements, once discovered. The ‘Screenshot‘ does what it says on the tin, generating a screenshot of the UI at the current stage of the test. Simple assertions can be made, to ensure functionality has triggered as expected, using methods such as ‘NoElements‘ (for example, we are expecting to not discover the ‘entryUsername‘ textbox after a successful login).
You can see an execution example in action below (executing each step in turn):
MAUI Blazor?
Here, I think things get slightly more complicated. From my birdseye view, circling the problem at speed; the key thing here is the abstraction of views behind a ‘BlazorWebView‘. Tests, as you would expect, won’t (from my first ‘stab’ at it) be able to discover/interact with web views hosted in this manner, without manual intervention. This seems like a natural challenge to consider and overcome:
Getting access to the web view, programmatically (in test code), maybe a possibility to explore (to get to elements under the hood to directly interact with)…
Playwright/Xappium
Both are very interesting in their own right. Playwright has been used in a sample project that includes Web, Mobile and Mobile Blazor samples, but the testing capabilities themselves are directed at web-based applications.
Xappium again did catch my attention, but the information available at this time is very light. It also seems to be geared at the more native experience, rather than the Blazor-hosting flavour.
Relevant links:
bUnit
From a Blazor component-level test standpoint, bUnit looks like a great contender (and perhaps this kind of ‘integration’ test will just become more preferred). It’s potent stuff, allowing you to verify constructed HTML and trigger event handlers (and even inject services behind the scenes), quite compelling. The opening gambit even alludes to (and challenges) typical browser-based UI tests.
bUnit browser-based UI/component test comparison quote:
bUnit builds on top of existing unit testing frameworks such as xUnit, NUnit, and MSTest, which run the Blazor components tests in just the same way as any normal unit test. bUnit runs a test in milliseconds, compared to browser-based UI tests which usually take seconds to run.
Relevant links:
Discussions
This key discussion, which has been ongoing for over 2 years, provides some insight into the general fears around the switch up to MAUI and the future of automated testing. From a native perspective, will existing work invested into Xamarin.UITest be lost? It seems unclear. This of course extends to test execution via the App Center.
Switching to MAUI Blazor suggests to me that a more in-depth rethink is needed (e.g. finding different tools, hunting for support, going web-only for automated tests, pivoting to component-based tests…for me, further research is required). The only question I could find on StackOverflow noted bUnit and Playwright (reframing automation from a web perspective).
Conclusions (at this time)
The future of automated UI testing for MAUI applications seems, at this stage, quite unclear. From a native app viewpoint, the validity of existing banks of Xamarin.UITest code may not be directly portable (I’m working with a team currently experimenting with this). There is hope as far as other approaches/platforms.
With MAUI Blazor, it looks as if the use of a BlazorWebView (if developers decide to pivot in this direction) will require a re-think/different approach to automation. The hunt is on for a good resolution on this front; if anyone reading this has any thoughts please comment below, it’d be greatly appreciated. If this gets less muddy I may do a follow-up to highlight a golden path!
Until the next time, happy coding out there! 💻
bearandhammer 