Codeless Visual Testing
TestingBot provides codeless visual testing, allowing you to perform scheduled visual checks against any website, without writing any code.
Simply add a URL to the webpage you'd like to test, pick the browsers you want to test on and indicate how frequently the test should run.
TestingBot will open the webpage on the browsers you indicated and perform a visual pixel-by-pixel check.
When TestingBot detects a mismatch, you will be alerted.
Setup
Please follow these steps to create your first codeless visual test:
- Click the Visual Testing menu item in the member area
- Choose Schedule a visual test
- Enter the URL of the webpage you want to visually test
Add many pages at once
To test more than a handful of pages, choose Add many pages on the list of scheduled visual tests. You can create a visual test for every page of a sitemap, or for a list of URLs:
- Sitemap URL: a sitemap or a sitemap index, optionally gzipped. An index is followed one level deep, up to 20 child sitemaps.
- URLs, one per line: paste the pages you want to test.
TestingBot reads up to 500 pages and shows them grouped, so you can pick the ones you want. Pages you already test are marked Already tested. Pages that are not public http or https URLs are left out.
One import creates at most 200 tests. All of them get the browsers, schedule and threshold you choose on the same page. Their first runs are spread over a period of time, so a large import does not take all of your parallel sessions at once.
Select Browsers
You'll need to select one or more browsers before you can start the visual test. How many browser configurations you can add to one test depends on your plan; the test's page shows how many are left.
You'll be able to choose between various platforms and browsers. Each browser runs on the latest version that is available to the general public.
For every browser configuration, you can specify a screen resolution and a geographical location.
Available platforms
- Windows 7
- Windows 8
- Windows 8.1
- Windows 10
- Windows 11
- macOS Tahoe
- macOS Sequoia
- macOS Sonoma
- macOS Ventura
- macOS Monterey
- iOS Simulators
- Android Emulators
Available browsers
- Chrome + Mobile Chrome
- Firefox
- Safari + Mobile Safari
- Microsoft Edge
- Internet Explorer
Steps before the screenshot
Some pages only show what you want to test after an action: a cookie banner has to be accepted, a tab opened, or you need to log in first. Under Before the screenshot on the test's settings page, you can add up to 20 steps. They run in order once the page has loaded, before the screenshot is taken.
| Step | What it does |
|---|---|
| Set a cookie | Sets a cookie (name, value and optionally domain and path), then loads the test's URL again. Several cookies in a row share one reload. The domain must be the host of the test's URL or a parent domain of it. |
| Click | Clicks the element that matches a CSS selector. |
| Type text | Types text into the element that matches a CSS selector, replacing what was there. Tick Secret for a password. |
| Wait for an element | Waits until an element that matches a CSS selector is visible, for 1 to 15 seconds. |
| Wait | Waits for a fixed time, up to 10,000 ms. |
| Go to a URL | Loads another page. It must be on the same host as the test's URL. |
| Scroll to | Scrolls the element that matches a CSS selector into view. |
| Hover | Moves the mouse over the element that matches a CSS selector. Desktop browsers only: on mobile browsers this step is skipped. |
Click, type, scroll and hover steps wait up to 15 seconds for their element to become visible. All steps of a run together may take 120 seconds.
If a step cannot be done, the run stops: nothing is compared, and a half set-up page never becomes the baseline. The run is marked as failed with a message such as Step 3: `#login` not found after 15 s, and the settings page shows the last failure per browser until the steps pass again (for up to a week).
Passwords and cookie values
Cookie values and text typed in a step marked Secret are stored encrypted and are never shown on the settings page again. To keep a stored secret, leave its field empty when you save. Before the screenshot is taken, the text of secret fields is made invisible, so it does not end up in the screenshot.
Changing steps resets the baselines
When you change what the steps do (add, remove or reorder steps, or change a selector, URL, wait or non-secret text), the test's baselines are reset: the next run of each browser becomes its new baseline. Changing only a password or cookie value keeps the baselines.
Password-protected sites (HTTP Basic Auth)
A staging or pre-launch site often asks for a username and password in a browser prompt before it shows anything. Open Password-protected site (HTTP Basic Auth) under Page on the test's settings page and enter the username and password. Every run then opens the page already logged in, and so do Go to a URL steps on the same host.
- The password is stored encrypted and never shown again. To keep it when you change other settings, leave the password field empty.
- The login is only sent to the host of the test's URL. A CI run against a preview URL on another host is opened without it.
- Setting or removing the login, or changing the username, starts new baselines. A new password alone keeps them.
- It works in Chrome, Microsoft Edge, Firefox and Chrome on Android. Safari on macOS, iPhone and iPad does not accept a login from a test, so while one is set those browsers are not run and the test's settings page names them.
A login form on the page itself (a username field and a button) is not HTTP Basic Auth: use steps for that.
Schedule Visual Tests
TestingBot allows you to specify a schedule when visual tests should run. You have various options to choose from:
- Run once on a specific date and time
- Run a visual test daily, at a specific time
- Run the test on a weekly basis, on a specific number of days and a specific hour
- Repeat the test every x minutes, on specific days and between specific hours
- Use a Cron Format to specify when the test should run
Alerting
When a visual test fails, you can choose to be alerted immediately.
Email Alert
One or more e-mail addresses can be added to the alerting section of each test. As soon as a visual test fails, you'll be alerted through e-mail.
The e-mail message will contain more details about what failed. The current screenshot will be shown, together with the baseline screenshot and a diff image (highlighting the differences between the baseline and the screenshot).
SMS Alert
You have the option to add your phone number, we will send you a text message when a visual test fails.
Custom message
Email and SMS alerts can carry a message of your own. Two placeholders are filled in when the alert is sent:
-
%test_url%: the URL of the visual test. -
%browser%: the browser and operating system of the run that failed.
For example: The %test_url% page changed in %browser%.
Webhook Alert
Add a webhook URL to have TestingBot send an HTTP POST with a JSON body to your own service whenever a browser of the test fails. The URL must be a public http or https address. The body looks like this:
{
"success": false,
"test_id": 123,
"test_url": "https://example.com/pricing",
"session_id": "<session id of the run>",
"visual_run_id": 456,
"pixel_mismatch": 3.25,
"threshold": 1.0,
"browser": {
"name": "Chrome",
"version": "<version>",
"os": "<operating system>"
},
"details_url": "https://testingbot.com/members/visual/<visual id>/runs/456"
}
-
pixel_mismatchandthresholdare percentages of the screenshot's pixels. -
details_urlopens the run, where you can approve or reject it. - The request is sent once, with a timeout of 25 seconds. It is not retried and redirects are not followed.
- The placeholders of a custom message do not apply to webhooks.
- The request is not signed. If your endpoint must only accept requests from TestingBot, include a hard-to-guess token in the webhook URL and check it on your side.
On a free account, alerts for visual tests are sent at most once every 24 hours.