Meet SafeTest – a new approach to frontend testing
In this article, we'll talk about SafeTest, a revolutionary library that offers a fresh take on end-to-end (E2E) testing for user interface (UI) web applications.
Problems with traditional UI testing
Traditionally, user interface testing has been done through unit or integration testing (also sometimes called E2E testing). However, each of these methods involves a trade-off: you have to choose between control over the test setup and control over the test driver.
For example, when using a unit testing solution such as react-testing-library, you retain full control over what should be displayed and how the underlying services and imports should behave. However, you lose the ability to interact with the actual page, which can lead to a lot of painful moments:
- Problems interacting with complex UI elements such as
components. - Inability to test CORS setup or GraphQL calls.
- Poor visibility of z-index issues affecting button clickability.
- Complex and unintuitive test development and debugging.
Conversely, using integration testing tools (such as Cypress or Playwright) provides control over the page, but sacrifices the ability to instrument the application initialization code. These tools work by remotely controlling the browser and interacting with the page. This approach has its drawbacks:
- Difficulty in making calls to alternate API endpoints without implementing custom network layer API rewrite rules.
- Inability to make assertions to spies/mocks or execute code within the application.
- Testing things like dark mode requires hitting the theme switch or knowing the localStorage mechanism to override.
- Inability to test application segments. For example, if a component becomes visible only after clicking a button and waiting for a 60-second timer to count down, the test would have to complete these steps, which would take at least a minute.
In order to cope with these shortcomings of the company
And
offer solutions like E2E Component Testing. Although these tools attempt to overcome the shortcomings of traditional integration testing methods, they have their own architectural limitations. They run a dev server with initialization code to load the desired component, which limits their ability to work with complex enterprise applications that may have OAuth or a complex build pipeline. Moreover, updating TypeScript may break tests until the Cypress/Playwright team updates their runner.
SafeTest
SafeTest aims to solve these problems with a new approach to UI testing. The main idea is that at the application loading stage, you should have
, which implements hooks to run tests. Note that this way of working does not have a noticeable impact on the normal use of your application – SafeTest uses lazy loading to dynamically load tests only when they are executed (in the README example, the tests are not in the production bundle at all). Playwright can then be used to run regular tests, which helps achieve the ideal browser control we want for our tests.
This approach also opens up some interesting possibilities:
- Deeply link to a specific test without having to run a test server.
- Two-way communication between the browser and the test context.
- Access to all DX features that come with Playwright (except those that come with playwright/test).
- Video recording of tests, viewing traces and a page pause function to try out different selectors/page actions.
Testing examples using SafeTest
SafeTest will look familiar to anyone who has done UI testing before, as it leverages the best parts of existing solutions. Here is an example of how you can test the entire application:
import { describe, it, expect } from 'safetest/jest';
import { render } from 'safetest/react';
describe('my app', () => {
it('loads the main page', async () => {
const { page } = await render();
await expect(page.getByText('Welcome to the app')).toBeVisible();
expect(await page.screenshot()).toMatchImageSnapshot();
});
});
You can also easily test a specific component:
import { describe, it, expect, browserMock } from 'safetest/jest';
import { render } from 'safetest/react';
describe('Header component', () => {
it('has a normal mode', async () => {
const { page } = await render(<Header />);
await expect(page.getByText('Admin')).not.toBeVisible();
});
it('has an admin mode', async () => {
const { page } = await render(<Header admin={true} />);
await expect(page.getByText('Admin')).toBeVisible();
});
it('calls the logout handler when signing out', async () => {
const spy = browserMock.fn();
const { page } = await render(<Header handleLogout={spy} />);
await page.getByText('logout').click();
expect(await spy).toHaveBeenCalledWith();
});
});Using Overrides
SafeTest uses React Context to provide the ability to override values during testing. As an example, let's say we have a fetchPeople function used in a component:
import { useAsync } from 'react-use';
import { fetchPerson } from './api/person';
export const People: React.FC = () => {
const { data: people, loading, error } = useAsync(fetchPeople);
if (loading) return <Loader />;
if (error) return <ErrorPage error={error} />;
return <Table data={data} rows=[...] />;
}We can modify the People component to use the override:
import { fetchPerson } from './api/person';
+import { createOverride } from 'safetest/react';
+const FetchPerson = createOverride(fetchPerson);
export const People: React.FC = () => {
+ const fetchPeople = FetchPerson.useValue();
const { data: people, loading, error } = useAsync(fetchPeople);
if (loading) return <Loader />;
if (error) return <ErrorPage error={error} />;
return <Table data={data} rows=[...] />;
}Now the test can override the response for this call:
const pending = new Promise(r => { /* Do nothing */ });
const resolved = [{name: 'Foo', age: 23], {name: 'Bar', age: 32]}];
const error = new Error('Whoops');
describe('People', () => {
it('has a loading state', async () => {
const { page } = await render(
<FetchPerson.Override with={() => () => pending}>
<People />
</FetchPerson.Override>
);
await expect(page.getByText('Loading')).toBeVisible();
});
it('has a loaded state', async () => {
const { page } = await render(
<FetchPerson.Override with={() => async () => resolved}>
<People />
</FetchPerson.Override>
);
await expect(page.getByText('User: Foo, name: 23')).toBeVisible();
});
it('has an error state', async () => {
const { page } = await render(
<FetchPerson.Override with={() => async () => { throw error }}>
<People />
</FetchPerson.Override>
);
await expect(page.getByText('Error getting users: "Whoops"')).toBeVisible();
});
});The render function also accepts a function that will be passed to the original application component, allowing you to inject any desired elements anywhere in the application:
it('has a people loaded state', async () => {
const { page } = await render(app =>
<FetchPerson.Override with={() => async () => resolved}>
{app}
</FetchPerson.Override>
);
await expect(page.getByText('User: Foo, name: 23')).toBeVisible();
});With the help of overrides we can write complex test cases. For example, to ensure that a service method that combines API requests from
/foo
,
/bar
And
/baz
, has a proper retry mechanism only for failed API requests and still displays the return value correctly. So, if /bar takes 3 attempts to resolve, the method will make a total of 5 API calls.
Overrides are not limited to just API calls (as you can use page.route), we can also override application-level specific values, such as feature flags or changing a static value:
+const UseFlags = createOverride(useFlags);
export const Admin = () => {
+ const useFlags = UseFlags.useValue();
const { isAdmin } = useFlags();
if (!isAdmin) return <div>Permission error</div>;
// ...
}
+const Language = createOverride(navigator.language);
export const LanguageChanger = () => {
- const language = navigator.language;
+ const language = Language.useValue();
return <div>Current language is { language } </div>;
}
describe('Admin', () => {
it('works with admin flag', async () => {
const { page } = await render(
<UseIsAdmin.Override with={oldHook => {
const oldFlags = oldHook();
return { ...oldFlags, isAdmin: true };
}}>
<MyComponent />
</UseIsAdmin.Override>
);
await expect(page.getByText('Permission error')).not.toBeVisible();
});
});
describe('Language', () => {
it('displays', async () => {
const { page } = await render(
<Language.Override with={old => 'abc'}>
<MyComponent />
</Language.Override>
);
await expect(page.getByText('Current language is abc')).toBeVisible();
});
});Overrides are a powerful feature of SafeTest, and the examples given here only scratch the surface. For more information and examples please contact
V
.
Reporting
SafeTest comes out of the box with powerful reporting capabilities such as automatic video replay linking, Playwright trace viewing, and even
directly to the finished tested component. IN
SafeTest repository has links to everything
as well as reports.
SafeTest in a corporate environment
Many large corporations require some form of authentication to use the application. Typically, going to localhost:3000 causes the page to take forever to load. You need to go to a different port, such as localhost:8000, which has a proxy to verify and/or inject authentication credentials into basic service calls. This limitation is one of the main reasons why the Cypress/Playwright component tests are not suitable for use on Netflix.
However, there is usually a service that can generate test users whose credentials can be used to log into and interact with the application. This allows you to create a lightweight wrapper around SafeTest to automatically generate and accept this test user. For example, here's how we do it at Netflix:
import { setup } from 'safetest/setup';
import { createTestUser, addCookies } from 'netflix-test-helper';
type Setup = Parameters<typeof setup>[0] & {
extraUserOptions?: UserOptions;
};
export const setupNetflix = (options: Setup) => {
setup({
...options,
hooks: { beforeNavigate: [async page => addCookies(page)] },
});
beforeAll(async () => {
createTestUser(options.extraUserOptions)
});
};Once configured, we simply import the above package instead of using safetest/setup.
Beyond React
While this post was focused on how SafeTest works with React, it is not limited to React. SafeTest also works with Vue, Svelte, Angular and can even run on NextJS or Gatsby. It also works using Jest or Vitest, depending on which test runner you started the code generation on.
demonstrates how to use SafeTest with various combinations of tools, and we encourage more examples to be added.
At its core, SafeTest is an intelligent combination of a test runner, a UI library, and a run in the browser. While Netflix most commonly uses Jest/React/Playwright, adapters for other options can be easily added.
Conclusion
SafeTest is a powerful testing framework used by Netflix. It makes it easy to create tests and provides comprehensive reports on when and how failures occurred, as well as links to play videos or manually run test steps to understand what exactly broke. We're looking forward to seeing how it will change UI testing.
