Originally posted on Plio’s Blog

Plio’s priority has always been building a simple and easy-to-use platform. The team works super hard to ensure any new functionality is properly reviewed, tested, and comes with its own set of unit tests. Rigorous testing cycles that sometimes have taken several rounds of code review before a feature gets released.

When things had started settling down, a random question popped up. It was a powerful one:

How do we ensure the system works as a whole, without breaking, when an actual user loads it up? How do we guarantee all the individual components work well when clubbed together? Is there a way to test as a genuine user and set up an automated testing for how they would interact with Plio?

End-to-End Testing

End-to-end testing (E2E testing) refers to an automated method that involves testing an application’s workflow from beginning to end. It basically aims to replicate real user scenarios so that the system can be validated for the integration of components and data consistency across the entire workflow.

To think how a real-user scenario is replicated, here’s an example of a typical E2E test case and the steps involved in it:

1. Open a browser.
2. Go to the login page.
3. Enter login credentials.
4. Navigate to the home page.
5. Perform an action, like typing into the search box or clicking on the “Create Plio” button.
6. Compare the action output with the expected output.

That’s exactly how a normal user will interact with the platform!

Selecting an E2E tool

Among several available End-to-End testing tools like Selenium and Cypress, we decided to go with TestCafe. Here’s why:

1. Selenium is a beast when it’s come to E2E testing. It has been in existence even before E2E was a real thing. However, it comes with a bunch of pre-requisites and a lengthy installation process. We wanted something fairly simple and easy to get started.
2. Cypress is the most popular and adopted tool for E2E testing in modern applications. With comprehensive documentation and a strong community backing up, it’s been evolving like many JavaScript frameworks these days. There was one issue though, it doesn’t support testing in Safari (yet). Safari is a crucial browser for us as it’s used by a significant percentage of Plio creators.

We wanted an E2E tool that can overcome these constraints. Something that has the best of both worlds. That was TestCafe.

Writing Test Cases

Writing test cases was fairly straightforward. TestCafe delivered exactly what it said. Implementing browser actions and adding assertions rules were quick and fast.

Here’s a pretty straightforward test case that:

1. Navigates to login page
2. Sets locale to English
3. Verifies visibility of elements on the login page

Continuous Integration

Once we were done with writing the first few test cases and running them locally, it was time to add them to our CI process. Meaning that running test cases should not be manual at all. We set up the Continuous Integration script, a YAML file that utilises GitHub Actions, and runs the following instructions whenever a Pull Request is created or modified:

1. Set up an Ubuntu instance.
2. Install Plio frontend and backend applications via Docker.
3. Install TestCafe.
4. Run the E2E test suite.

BrowserStack

While the CI script ran just as expected, there was one limitation with the setup. The Ubuntu instance where TestCafe was running, comes only with a handful of browsers. This means the output from the tests was only valid for a fixed version of Chrome or Firefox running on an Ubuntu 20.04 LTS instance. It’s rare a Plio user will be having the same setup.

This essentially meant that the setup is still incomplete and probably not even covering 1% of our user base. We still can’t be sure how a teacher is creating plios on her Windows 10 Edge browser or what a 12-year old student on Android UC Browser sees in fairly smaller screen size.

To overcome this limitation, we picked up BrowserStack. It’s a platform that provides literally hundreds of browsers and Operating Systems (including mobiles) where one can run their E2E tests. TestCafe integrates very smoothly with BrowserStack. An added bonus was that BrowserStack offers its premium services for free to open source projects!

After setting up BrowserStack, here’s what our Continuous Integration steps looked like:

1. Set up an Ubuntu instance.
2. Install Plio frontend and backend applications via Docker.
3. Install TestCafe.
4. Select Operating Systems and browsers available on BrowserStack
   • Window 10 with Chrome, Firefox & Edge.
   • OSX Big Sur with Chrome, Firefox & Safari.
   • OSX Catalina with Chrome, Firefox & Safari.
   • Run the test cases in every browser + OS combination

Here’s the full GitHub workflow YAML script.

A recording of an automated E2E test running on Firefox + OSX Catalina on BrowserStack:

Final words

Martin Fowler, the co-author of the Agile Manifesto (which is the foundation of modern software development practices), defined E2E testing as follows:

End-to-end tests give you the biggest confidence when you need to decide if your software is working or not.

With more E2E tests on our to-do list, we look forward to Plio becoming a more robust application with:

• Fewer bugs
• Fewer breaking changes
• A faster release cycle, and
• A more meaningful impact.