Reducing the impact of Chrome updates

[Piedone]

We noticed that Chrome updates, what come automatically (and without us being able to influence them in any way) with GitHub Actions runner image updates, make our UI tests unreliable. This is the most recent example: Lombiq/Open-Source-Orchard-Core-Extensions#991. Visual verification tests are another example where random (font) rendering changes can break them even if we have generally suitable tolerances. But other random breakage happens routinely too.

Do you have any suggestion to make this situation better? Thanks in advance!

[YevgeniyShunevych]

You can pin the version of Chrome you want to use on CI and locally. And probably upgrade that version once per few months.

I would start with trying Selenium Manager for that. Now it also supports browser setup in addition to drivers setup. For Chrome it will download Chrome for Testing (CfT) of the version you specify. For basic configuration all you need is just specify the BrowserVersion property of ChromeOptions.

ChromeOptions options = new ChromeOptions();
options.BrowserVersion = "132";

In such case remove Atata functionality of driver setup, like AutoSetUpDriverToUse. Let Selenium Manager set up browser and driver automatically.

Alternatively, you can take a look at Selenium Grid to manage browsers, quite a powerful tool. There are Selenium Grid Docker images. Also there are paid third-party services like BrowserStack, Sauce Labs, etc.

[Piedone]

Ah, brilliant, I didn't know about that property, awesome, thank you!

I've opened Lombiq/UI-Testing-Toolbox#478 about this.

[Piedone]

What's unclear to me is, when would we still use Atata.WebDriverSetup? When we want to use the currently installed browser, whatever its version?

[YevgeniyShunevych]

Basically, with a rise of Selenium Manager, Atata.WebDriverSetup package becomes less needed. WebDriver's Selenium Manager sets up driver automatically if you don't specify explicitly a path to driver in environment variable (e.g. "ChromeDriver") the way Atata.WebDriverSetup does.

So if you want to pin a browser to a specific version, I think you can just remove Atata.WebDriverSetup package dependency, and rely completely on built-in WebDriver Selenium Manager. Atata.WebDriverSetup just doesn't have a browser setup functionality.

If you setup auto version of driver corresponding to an installed browser, like you did before, I would say that Atata.WebDriverSetup is still a good choice. You control how and when you want to perform a setup, for example I do it in global setup method. With Selenium Manager you are a bit limited in options, at least now. Also Atata.WebDriverSetup is a bit faster in driver setup. I also don't like the implementation approach of Selenium Manager. Basically, WebDriver executes an extra process "selenium-manager.exe" (you can find 3 OS-dependent executables in folder bin/Debug/net8.0/selenium-manager), which does the actual browser/driver management job. While Atata.WebDriverSetup is a pure ordinary library.

[Piedone]

I see, thank you.