Originally posted on Plio’s Blog
Plio’s priority has always been building a simple and easy-to-use platform. The team works super hard to ensure any new functionality is properly reviewed, tested, and comes with its own set of unit tests. Rigorous testing cycles that sometimes have taken several rounds of code review before a feature gets released.
When things had started settling down, a random question popped up. It was a powerful one:
How do we ensure the system works as a whole, without breaking, when an actual user loads it up? How do we guarantee all the individual components work well when clubbed together? Is there a way to test as a genuine user and set up an automated testing for how they would interact with Plio?
End-to-end testing (E2E testing) refers to an automated method that involves testing an application’s workflow from beginning to end. It basically aims to replicate real user scenarios so that the system can be validated for the integration of components and data consistency across the entire workflow.
To think how a real-user scenario is replicated, here’s an example of a typical E2E test case and the steps involved in it:
That’s exactly how a normal user will interact with the platform!
Among several available End-to-End testing tools like Selenium and Cypress, we decided to go with TestCafe. Here’s why:
We wanted an E2E tool that can overcome these constraints. Something that has the best of both worlds. That was TestCafe.
Writing test cases was fairly straightforward. TestCafe delivered exactly what it said. Implementing browser actions and adding assertions rules were quick and fast.
Here’s a pretty straightforward test case that:
Once we were done with writing the first few test cases and running them locally, it was time to add them to our CI process. Meaning that running test cases should not be manual at all. We set up the Continuous Integration script, a YAML file that utilises GitHub Actions, and runs the following instructions whenever a Pull Request is created or modified:
While the CI script ran just as expected, there was one limitation with the setup. The Ubuntu instance where TestCafe was running, comes only with a handful of browsers. This means the output from the tests was only valid for a fixed version of Chrome or Firefox running on an Ubuntu 20.04 LTS instance. It’s rare a Plio user will be having the same setup.
This essentially meant that the setup is still incomplete and probably not even covering 1% of our user base. We still can’t be sure how a teacher is creating plios on her Windows 10 Edge browser or what a 12-year old student on Android UC Browser sees in fairly smaller screen size.
To overcome this limitation, we picked up BrowserStack. It’s a platform that provides literally hundreds of browsers and Operating Systems (including mobiles) where one can run their E2E tests. TestCafe integrates very smoothly with BrowserStack. An added bonus was that BrowserStack offers its premium services for free to open source projects!
After setting up BrowserStack, here’s what our Continuous Integration steps looked like:
Here’s the full GitHub workflow YAML script.
A recording of an automated E2E test running on Firefox + OSX Catalina on BrowserStack:
Martin Fowler, the co-author of the Agile Manifesto (which is the foundation of modern software development practices), defined E2E testing as follows:
End-to-end tests give you the biggest confidence when you need to decide if your software is working or not.
With more E2E tests on our to-do list, we look forward to Plio becoming a more robust application with: