Running tests/suites

We have moved

This is a legacy version of Copado Robotic Testing help. This page will not be updated anymore.

See https://docs.copado.com/ for up-to-date documentation.

You can start test run in Copado Robotic Testing in three ways:

  • Manually by clicking the run button

  • Scheduling them with scheduler

  • Triggering runs from external systems using API key

Running tests manually

To Run test cases manually, just select the suite in Copado Robotic Testing UI and click Run button [4]. Prior to starting run, you can select if test runs will be recorded as a video file [1] and if test run will be streamed live [2]. You can also select if the run is considered "regression" test run or "development test run" [3].

suites run manual

Recording videos and streaming

You can record a video of:

  • All test cases

  • Only failed test cases

  • None (turns recording off)

Recorded videos can be found after run ends from "Test case summary" section:

2902

When streaming is enabled, a separate browser window will pop-up and display the test execution live:

suites run streaming
Recording videos is currently available for cloud robots running web or mobile test automation.
Streaming run is currently available for cloud robots running web test automation.

Regression and Development runs

You can select if run is considered Regression run (default) or Development run.

Reression runs are "normal" runs and all results from these runs are available for metrics and dashboards. Development runs are meant to be used when testing new changes to test cases themselves; you use Development run while validating that the test case works correctly. Development runs are not counted towards metric calculations or show up in metrics.

Development runs are only available for newer Kubernetes based projects.

Scheduling test runs

To schedule test runs to be run automatically, for example every day at 12:00, you can use Copado Robotic Testing’s built-in scheduler.

Open the built-in scheduler

Click the test suite name you want to schedule to be run. Then click "Scheduler" button under test suite name:

suites scheduler icon

Using the scheduler

Select frequency [1] (daily, every x minutes, monthly etc.). and time [2] and [3] of test run using the scheduler dialog. Note that additional hours and minutes fields [2,3] depend on frequency selection. For example, if you select "Monthly" as frequency, then your additional selections will be date and time on every month.

Optionally, you can also directly give the schedule using the Crontab format [4].

You can record videos [5] of test runs with the exact same rules as with manual runs.

When you click Save [6], the schedule will be applied. With Delete [7] button you can remove existing schedule.

suites scheduler dialog

Setting up notifications

You can also setup Copado Robotic Testing to send notifications after test run. Notifications can be sent as an email or as a Slack message.

Click Notifications button

Click the test suite name you want to schedule to be run. Then click "Notifications" button under test suite name:

suites notifications icon

Add a new email notification

Click the (+) icon in order to add a new notification:

suites notifications add

Select Email as Type.

Select which trigger to use for your notification (test execution starts, test execution finalizes and there are failures etc.)

Fill in the recipients who will receive the email notification using Add users/groups -button.

suites notifications email

Add a new Slack notification

Click the (+) icon in order to add a new notification:

suites notifications add

Select Slack as Type.

Select which trigger to use for your notification (test execution starts, test execution finalizes and there are failures etc.)

Fill in the Slack Webhook URL and generate Slack application to your organizations Slack instance. More information about this can be found from Slack’s documentation.

suites notifications slack

Triggering runs from external sources

Triggering runs using API key is meant to be used, if you want to hook the test robot and suite into your Continuous Integration pipeline or other external scripts. Once Copado Robotic Testing’s api is called with generated API key, it will trigger the run for suite corresponding to API key and return results. Note: you can use suite specific API Key or your Personal Access Key.

Create a Personal Access Key

Personal access keys can be used when making REST API calls to Copado Robotic Testing. Most common use case is using API to trigger runs from other systems or CI pipeline. Personal access keys can be used instead of test suite specific API Keys or with pre-build solutions like GitHub Action for Copado Robotic Testing.

  1. To create a new personal access key, navigate to your profile.

  2. Click New Key button under Personal access keys

New Key button
  1. Name your key and set the expiration in days (optional)

New Key Dialog
  1. Click Save button.

  2. Your new personal access key has been generated. A dialog will open displaying the generated key.

Generated Key
You can copy the key to clipboard by clicking it.
  1. The new key can also be seen in Personal access keys -list

Personal keys list
Using Personal Access Key is preferred over suite specific API Key (described next)

Optional: Create an API key for a suite

To get started, select the suite you want to create an API key to. You can do this by clicking the suites name.

After the suite is selected, click the "API options" icon:

suites running api key icon

The next screen allows you to Generate and API key for the suite, or remove/revoke one if you had already created an API key earlier:

suites running api key generate
suites running api key created
You need to have Editor rights to robot in order to be able to generate API key.
At this phase, remember to copy the generated API key. Store it securely, since it provides access for running your test suite. You will need to include the API key to your script calling the Copado Robotic Testing API.

Use the Personal Access Key in scripts

Use Personal Access Key to trigger any test suite run. You can do this in basically any language that can call REST APIs, and most of our customers have their own favorite language.

We will provide an bash example script here, since it’s easy to use this in linux based continuous integration setups.

You will need to provide the following information to API:

  • CRT_API_URL

  • CRT_ACCESS_KEY

    • Your Personal Access Key to authorize making API calls

  • inputParameters (optional)

    • Additional parameters/variables for your run (see the example below)

#!/bin/bash
PROJECT_ID=[YOUR_PROJECT_ID_HERE]
SUITE_ID=[YOUR_SUITE_ID_HERE]
CRT_API_URL="https://api.robotic.copado.com/pace/v4/projects/${PROJECT_ID}/jobs/${SUITE_ID}/builds"
CRT_ACCESS_KEY=[YOUR_PERSONAL_ACCESS_KEY_HERE]
# Start the build
BUILD=$(curl -sS -H 'X-Authorization: '"${CRT_ACCESS_KEY}"'' -d '{"inputParameters": [{"key": "BROWSER", "value": "firefox"}]}' -H "Content-Type: application/json" -X POST ${CRT_API_URL})
echo "${BUILD}"
BUILD_ID=$(echo "${BUILD}" | grep -Po '"id":\K[0-9]+')
if [ -z "${BUILD_ID}" ]; then
  exit 1
fi
echo -n "Executing tests "
STATUS='"executing"'
# Poll every 10 seconds until the build is finished
while [ "${STATUS}" == '"executing"' ]; do
  sleep 10
  RESULTS=$(curl -sS -H 'X-Authorization: '"${CRT_ACCESS_KEY}"'' ${CRT_API_URL}/${BUILD_ID})
  STATUS=$(echo "${RESULTS}" | grep -Po '"status": *\K"[^"]*"' | head -1)
  echo -n "."
done
echo " done!"
FAILURES=$(echo ${RESULTS} | grep -Po '"failures":\K[0-9]+')
LOG_REPORT_URL=$(echo "${RESULTS}" | grep -Po '"logReportUrl": *\K"[^"]*"')
echo "Report URL: ${LOG_REPORT_URL}"
If you are using our EU instance, change url to api.eu-robotic.copado.com

Use the suite specific API key in scripts

To trigger runs using API, use the generated API key in your scripts. You can do this in basically any language that can call REST APIs, and most of our customers have their own favorite language.

We will provide an bash example script here, since it’s easy to use this in linux based continuous integration setups.

The preferred method for triggerin runs via API is using is to use Personal Access Key

You will need to provide the following information to API:

#!/bin/bash
CRT_API_URL=https://pace.qentinel.com/pace/v4/builds
CRT_API_KEY=[YOUR_API_KEY_HERE]
# Start the build
BUILD=$(curl -sS -d '{"key": "'${CRT_API_KEY}'", "inputParameters": [{"key": "BROWSER", "value": "firefox"}]}' -H "Content-Type: application/json" -X POST ${CRT_API_URL})
echo "${BUILD}"
BUILD_ID=$(echo "${BUILD}" | grep -Po '"id":\K[0-9]+')
if [ -z "${BUILD_ID}" ]; then
  exit 1
fi
echo -n "Executing tests "
STATUS='"executing"'
# Poll every 10 seconds until the build is finished
while [ "${STATUS}" == '"executing"' ]; do
  sleep 10
  RESULTS=$(curl -sS ${CRT_API_URL}/${BUILD_ID}?key=${CRT_API_KEY})
  STATUS=$(echo "${RESULTS}" | grep -Po '"status": *\K"[^"]*"' | head -1)
  echo -n "."
done
echo " done!"
FAILURES=$(echo ${RESULTS} | grep -Po '"failures":\K[0-9]+')
LOG_REPORT_URL=$(echo "${RESULTS}" | grep -Po '"logReportUrl": *\K"[^"]*"')
echo "Report URL: ${LOG_REPORT_URL}"