We worked together for 5+ years at a tech start-up that deployed multiple times a week. After every deployment, a bunch of our developers would manually smoke test the application by running through all of the critical user experiences. This manual testing was expensive in terms of our time. To speed up the tests’ run time, we dedicated developer resources to writing and managing Selenium scripts. That was expensive at “compile time” due to authoring and maintenance. At a high-level, we believe that the problem with automated end-to-end testing comes down to two things: tests are too hard to create, and they take too much time to maintain. These are the two issues we are trying to solve with Reflect.
Reflect lets you create end-to-end tests just by using your web application, and then executes that test definition whenever you want: on a schedule, via API trigger or simply on-demand. It emails you whenever the test fails and provides a video and the browser logs of the execution.
One knee-jerk reaction we’re well aware of: record-and-playback testing tools, where the user creates a test automatically by interacting with their web application, have traditionally not worked very well. We’re taking a new approach by loading the site-under-test inside of a VM in the cloud rather than rely on a locally installed browser extension. This eliminates a class of recording errors due to existing cookies, proxies or other extensions introducing state that the test executor is not aware of, and unifies the test creation environment with the test execution environment. By completely controlling the test environment we can also expose a better UX for certain actions. For example, to do visual testing you just click-and-drag over the element(s) you want to test. For recording file uploads we intercept the upload request in the VM, prompt you to upload a file from your local file system, and then store that file in the cloud and inject it into the running test. If you want to add additional steps to an existing test, we’ll put you back into the recording experience and fast-forward you to that point in the test, where again all you need to do is use your site and we’ll add those actions to your existing test. Controlling the environment also allows us to reduce the problem space by blocking actions which you typically wouldn’t want to test, but which are hard to replicate and thus could lead to failed recordings (e.g. changing browser dimensions mid-recording). As an added bonus, our approach requires no installation whatsoever!
We capture nearly every browser action from hovers to file uploads, and drag-and-drops to iframes, while building a repeatable, machine-executable test definition. We support variables for dynamic inputs, and test composition so your test suite is DRY. The API provides flexible integration with your CI/CD out of the box and supports creating tests in prod and running them in staging on the fly. You don’t need to use a separate test grid, as all Reflect tests run on our own infrastructure. Parallel execution of your tests is a two click config change and we don’t charge you extra for it.
Some technical details that folks might find interesting:
- For every action you take we’ll generate multiple selectors targeting the element you interacted with. We wrote a custom algorithm that generates a diverse set of selectors (so that if you delete a class in the future your tests won’t break), and ranks them by specificity (i.e. [data-test-id] > img[alt=”foo”] > #bar > .baz).
- To detect certain actions we have to track DOM mutations across async boundaries. So for example we can detect if a hover ended up mutating an element you clicked on and thus should be captured as a test step, even if the hover occurred within a requestAnimationFrame, XHR/fetch callback, setTimeout/setInterval, etc.
- We detect and ignore auto-genned classes from libraries like Styled Components. We use a heuristic to do this so it’s not perfect, but this approach allows us to generate higher quality selectors than if we didn’t ignore them.
- One feature in beta that we’re really excited about: For React apps we have the ability to target React component names as if they were DOM elements (e.g. if you click on a button you might get a selector like “<NotificationPopupMenu> button”). We think this is the best solution for the auto-genned classes problem described in the bullet above, as selectors containing component names should be very stable.
We tried to make Reflect as easy as possible to get started with - we have a free tier (no credit card required) and there’s nothing to install. Our paid plans start at $99 and you pay primarily for test execution time. Thanks and we look forward to your feedback!
by Andrey
Azimov