Configuration

Overview

There are 3 main areas which have to be configured. These are:

  • Mobile device

  • Environment provider, and

  • Test session or test runtime

These can be configured directly to the test scripts i.e. Robot files or, in addition to use more flexible configuration file in YAML format. The latter approach is preferred on the new test suites.

Typically a term capability means the same as configurable attribute or variable. For example, mobile device can have os_version, platform or deviceName. Many of these capabilities come from Appium and environment providers like BrowserStack. Also QMobile has own configurable attributes like default_timeout or case_sensitive.

Configuration Layers

There are different ways how configurable attributes can be introduced and how their values can be changed. There are 5 layers of configuration points:

Layer I — hard-coded

Default attributes in Python code

Layer II — pre-test

Configuration file in YAML format

Layer III — pre-test

Variables section in test (or robot) files

Layer IV — pre-test

Variables given in command-line for robot command e.g. robot -v default_timeout:30

Layer V — run-time

SetConfig and ResetConfig PaceWords as well as Suite Setup, Test Setup and Set [Global, Suite, Test] Variable keywords

Configurable attributes are key-value pairs. When the value of already introduced attribute (i.e. has the same key) is changed at the higher layer, then that will overwrite the current value. Layers II-IV are evaluated before test execution and Layer V during the test execution.

The values of resulted configuration are updated to both QMobile configuration and global Robot-Framework variables. In other words, one can access them by using GetConfig PaceWord or normal Get Variable Value or Get Variables RF keywords. In addition, collected capabilities are stored as Python dictionary.

Example of getting default_timeout in both ways:

*** Test Cases ***
Get Default Timeout
    # PaceWord
    ${timeout}=     GetConfig   default_timeout
    ${endpoint}=    GetConfig   capabilities.remote_url

    # RF keyword
    ${timeout}=     Get Variable Value   ${default_timeout}
    ${endpoint}=    Get Variable Value   ${capabilities['remote_url']}

Configuration File

The idea of a configuration file is to provide flexibility, re-usability and ease the maintainability of the test scripts. This is achieved by providing following functionalities and properties:

  • Uses human readable YAML format

  • Defines configurable attributes

  • Defines logical sections for defaults, providers, devices, sessions and an optional sub-section capabilities

  • Value substitution with other attributes makes it possible to build more complex configuration attributes

  • Test scripts can be written without device or provider specific configuration attributes

Example of a configuration file (simple_config.yaml) for local device testing on Android:

defaults:
  provider: local
  device: my_android
  session: test1
  EXECDIR: ~
providers:
  - provider: local
    host: localhost
    port: 4723
    appium_url: http://${host}:${port}/wd/hub
    capabilities:
      remote_url: $appium_url
devices:
  - device: my_android
    capabilities:
      udid: ~
      platform: android
      automationName: UIAutomator2
sessions:
    - session: test1
      capabilities:
          app: $EXECDIR/resources/app/ApiDemos-debug.apk

The location of this file is typically under resources directory of your test project, but it can be anywhere as long as robot command has access to that location and a permission to read the file. Used configuration file can be defined statically in test script when importing QMobile library:

Import   QMobile   configs=${CURDIR}${/}resources${/}another_config.yaml

Or, as a command-line parameter when starting the test execution:

robot -v configs:resources/simple_config.yaml text.robot
Configuration file given as command-line parameter overrides possible configuration file defined in library import statement (i.e. in test script).

Configuration Attribute Definition

In order to introduce a configuration attribute it has to be defined at least in one of the sections in configuration file. It is also possible to use pre-defined Robot Framework variables like $EXECDIR when it is defined in the file — typically in defaults section. As a minimum the value has to be set to ~ (tilde) which means None (or null).

It is good to know, that if attribute value hasn’t been set as part of configuration processing then that attribute is removed from the final and effective configuration.

Sections

The purpose of the sections is to group configuration attributes logically. The final and effective configuration will be the merged configuration. The sections are layered similarly as the configuration points described earlier. The order of section processing is as follows:

defaults

Default attributes with values or custom attributes which doesn’t fit to any other section

providers

Attributes and capabilities related to providers (or environments) i.e. local, BrowserStack or BitBar

devices

Attributes and capabilities related to DUTs i.e. device under test (udid)

sessions

Attributes and capabilities related to test session like SUT i.e. software under test (app)

In addition, there is an optional sub-section for above sections excluding defaults:

capabilities

All attributes defined in this sub-section are merged to one and combined capabilities. The order of merging is the same as above. In other words capabilities defined in providers are overwritten by providers and then by session if the key is the same.

Please note that only one item from providers, devices and sessions each can be selected for a test run. It is possible to define default item for each section under defaults (see example configuration file).

Attribute Value Substitution

Value substitution allows easier re-use of configuration attributes. It also makes it possible to build more complex configurations. One can substitute the value of an attribute with the value of another attribute by using the key in special format. Following attribute value types and substitution formats are supported:

# YAML configuration attribute types
int_key: 1234
float_key: 0.3456
str_key1: This is a string
str_key2: '1231234124124124'            <-- Type of this is also a string
bool_key: true
text: ~                                 <-- Attribute introduced with
number: ~                                   None (or null) value

# Substitution formats
integer: $int_key                       --> 1234
float: $float_key                       --> 0.3456
text1: $str_key1                        --> This is a string
text2: $text1 with substitution!        --> This is a string with substitution!
text3_in_between: BEGIN${str_key2}END   --> BEGIN1231234124124124END

# Substitution formats in Robot command-line with --variable (or -v)
-v text:"$str_key2 is a string"         --> 1231234124124124 is a string
-v number:$int_key                      --> 1234
-v number:${int_key}                    --> 1234
If substitution value is as part of other value i.e. is a sub-string, then substituted value format ${value} must be used like in text3_in_between in above example

The resolution algorithm of attribute substitutions uses several passes in order to solve all substitutions. But, if attribute value cannot be fully resolved then following error is displayed when importing QMobile library:

[ ERROR ] Expanding value of 'text' failed as '$str_key99 is a string and
          $int_key99 is a number' not fully resolved
The value substitution is processed when QMobile library is imported. The effective configuration contains only fully resolved values. I.e. one cannot do value substitutions at runtime (or when test are executing)

Examples

In this section, it is assumed that the reader is familiar with the Robot Framework and thus used options or parameters are not explained in detail here. Please see Robot Framework User Guide for reference.

In the execution part, the command output might be shortened to contain only the relevant parts. If you execute the test script on your computer the output might be longer and more detailed. Also commands are executed in Windows environment, so some changes might be required in other platforms (e.g. multi-line commands).

All the examples assumes that the test script is in the root of the project folder e.g. configuration. The configuration file (YAML) needs to be under resources folder of that project. The structure should look similar to this:

\---configuration
    |   example1.robot
    |   example2.robot
    \---resources
            example_config.yaml
            keywords.robot

When an example is executed the working directory has to be the project folder i.e. configuration in this case.

Example 1 — Value Substitution

Let’s have a configuration file example_config.yaml including examples shown in Attribute Value Substitution topic.

defaults:
  provider: local
  device: my_android
  session: example1
  value_overwritten: 'defaults section'
  EXECDIR: ~
providers:
  - provider: local
    value_overwritten: 'providers section'
    version: '1.7'
    capabilities:
      browserstack.appium_version: $version
devices:
  - device: my_android
    value_overwritten: 'devices section'
    capabilities:
      udid: ~
      platform: android
      automationName: UIAutomator2
sessions:
    - session: example1
      value_overwritten: 'sessions section'
      int_key: 1234
      float_key: 0.3456
      str_key1: This is a string
      str_key2: '1231234124124124'
      bool_key: true
      text: ~
      number: ~
      integer: $int_key
      float: $float_key
      text1: $str_key1
      text2: $text1 with substitution!
      text3_in_between: BEGIN${str_key2}END
      rf_variable: $EXECDIR
      capabilities: ~

In addition, let’s have a test script example1.robot to show the above configuration in action.

*** Settings ***
Library   QMobile       configs=${CURDIR}${/}resources${/}example_config.yaml
Library   Collections

*** Variables ***
${default_timeout}              666
${browserstack.appium_version}  1.1.1.1

*** Test Cases ***
Show Configuration
    [Documentation]     Logs effective configuration

    ${conf}     Get Variables           no_decoration=True
    ${conf}     Get Dictionary Items    ${conf}
    :FOR    ${config}       ${value}    IN      @{conf}
    \       Log To Console  ${\n}${config} = ${value}       no_newline=True

Show Capabilities
    [Documentation]     Logs effective capabilities

    ${caps}     Get Dictionary Items   ${capabilities}
    :FOR    ${capability}   ${value}    IN      @{caps}
    \       Log To Console  ${\n}${capability} = ${value}   no_newline=True

Execution: Show Configuration

> robot --quiet -t "Show Configuration" example1.robot

EXECDIR = C:/configuration
auto_scale = False
bool_key = True
browserstack.appium_version = 1.1.1.1
case_sensitive = False
configs = C:/configuration\resources\example1.yaml
default_timeout = 666
device = my_android
float = 0.3456
float_key = 0.3456
int_key = 1234
integer = 1234
legacy_config_mode = False
log_application = False
ocr_engine = RevisionEngine
ocr_scale = 0.3
provider = local
recognition_mode = appium
refresh_screenshot_every_nth_time = 1
rf_variable = C:/configuration
scroll = False
session = example1
str_key1 = This is a string
str_key2 = 1231234124124124
template_width = 0
text1 = This is a string
text2 = This is a string with substitution!
text3_in_between = BEGIN1231234124124124END
tolerance_text = 0.8
use_hti = False
value_overwritten = sessions section
version = 1.7
  • Observe attributes text2 and test3_in_between. Those show nicely how value substitution works (Layer II).

  • Observe the attribute value_overwritten. It shows how sections overlay works.

  • Observe the attribute browserstack.appium_version or default_timeout. It shows how configuration layers are working. In this case the final value comes from the Variables section (Layer III) of the test script instead of the configuration file.

  • Observe RF variable EXECDIR and attribute rf_variable. It shows how existing RF variables can be also used in configuration files.

Execution: Show Capabilities

> robot --quiet -t "Show Capabilities" example1.robot

automationName = UIAutomator2
browserstack.appium_version = 1.1.1.1
platform = android
  • Observe how capabilities from providers, devices and sessions sections are merged together

Execution: Command-line Substitution (Layer IV) with Show Configuration

## 1a. Simple substitution
> robot --quiet -t "Show Configuration" ^
  -v default_timeout:99 example1.robot

default_timeout = 99
value_overwritten = sessions section

## 1b. Simple substitution
> robot --quiet -t "Show Configuration" ^
  -v value_overwritten:"cmdline" example1.robot

default_timeout = 666
value_overwritten = cmdline

## 2. Substituting attribute which is used in other attributes
> robot --quiet -t "Show Configuration" ^
  -v str_key1:"This string is from command-line" example1.robot

default_timeout = 666
str_key1 = This string is from command-line
text1 = This string is from command-line
text2 = This string is from command-line with substitution!

## 3. Multiple substitutions to introduced attribute
> robot --quiet -t "Show Configuration" ^
  -v text:"None becomes: '$text2', float '$float' and boolean '$bool_key'" ^
  example1.robot

text = None becomes: 'This is a string with substitution!', float '0.3456' and boolean 'True'

## 4. Inner substitution from command-line
> robot --quiet -t "Show Configuration" ^
  -v text2:"Surprise!" -v bool_key:off ^
  -v text:"None becomes: '$text2', float '$float' and boolean '$bool_key'" ^
  example1.robot

text = None becomes: 'Surprise!', float '0.3456' and boolean 'False'

## 5. Error when substituting attribute which is not defined in configuration file
> robot --quiet -t "Show Configuration" ^
  -v new_attribute:"True" -v text:"This is $new_attribute" ^
  example1.robot

[ ERROR ] Expanding value of 'text' failed as 'This is $new_attribute' not
          fully resolved

new_attribute = True
text = This is $new_attribute

## 6. After definining new_attribute in configuration file (e.g. in defaults)
> robot --quiet -t "Show Configuration" ^
  -v new_attribute:"True" -v text:"This is $new_attribute"
  example1.robot

new_attribute = True
text = This is True

Example 2 — Run-time Configuration

It is important to acknowledge that building the effective configuration takes place when QMobile library is imported. The final values of that configuration become the default values of the both QMobile configuration as well as Robot Framework variables. However, from that point onwards those values are not connected (or synced) automatically. This might have some unexpected outcomes when actually running the test cases and when using Setup or Teardown and ResetConfig PaceWord.

This example shows some of the — maybe unexpected — outcomes and also recommendations.

Let’s still use the same configuration file example_config.yaml, but add new attributes to defaults section called use_suite_setup and use_test_setup:

defaults:
  provider: local
  device: my_android
  session: example1
  value_overwritten: 'defaults section'
  EXECDIR: ~
  use_suite_setup: false
  use_suite_setup: false

Next, let’s create a new file keywords.robot to resources folder. The custom keywords will be put to that file. Notice, how both QMobile configuration and RF variable are set to the same value — in other words, we try to manually keep them in sync.

*** Keywords ***
Conditional Suite Setup
    Run Keyword If  '${use_suite_setup}' == '${TRUE}'  Run Keywords  SetConfig   default_timeout   1000
    ...   AND   Set Suite Variable   ${default_timeout}   1000

Conditional Test Setup
    Run Keyword If  '${use_test_setup}' == '${TRUE}'  Run Keywords  SetConfig   default_timeout   9999
    ...   AND   Set Test Variable   ${default_timeout}   9999

Log Value
    [Arguments]     ${tc_name}

    ${config_timeout}=  GetConfig            default_timeout
    ${rf_timeout}=      Get Variable Value   ${default_timeout}
    Log To Console      ${tc_name} : default_timeout${\n}${SPACE*2}config: ${config_timeout}, rf_variable: ${rf_timeout}

Finally, let’s add a new test script example2.robot. The idea is just to show the effect of different approaches to the actual runtime value of a single attribute (or variable).

*** Settings ***
Library       QMobile       configs=${CURDIR}${/}resources${/}example_config.yaml
Library       Collections
Resource      ${CURDIR}${/}resources${/}keywords.robot
Suite Setup   Conditional Suite Setup

*** Variables ***
${default_timeout}              666

*** Test Cases ***
Check Suite Setup
    Log Value   ${TEST_NAME}

Check Test Setup
    [Setup]     Conditional Test Setup
    Log Value   ${TEST_NAME}

Check Test Setup With ResetConfig As Teardown
    [Setup]     Conditional Test Setup
    Log Value   ${TEST_NAME}
    [Teardown]  ResetConfig     default_timeout

Final Check
    Log Value   ${TEST_NAME}

Execution example2.robot

# 1. The plain test script
> robot --quiet example2.robot

Check Suite Setup : default_timeout
  config: 666, rf_variable: 666
Check Test Setup : default_timeout
  config: 666, rf_variable: 666
Check Test Setup With ResetConfig As Teardown : default_timeout
  config: 666, rf_variable: 666
Final Check : default_timeout
  config: 666, rf_variable: 666

# 2. Setting value from command-line
> robot --quiet -v default_timeout:2020 example2.robot

Check Suite Setup : default_timeout
  config: 2020, rf_variable: 2020
Check Test Setup : default_timeout
  config: 2020, rf_variable: 2020
Check Test Setup With ResetConfig As Teardown : default_timeout
  config: 2020, rf_variable: 2020
Final Check : default_timeout
  config: 2020, rf_variable: 2020
  • The test script is executed without any setup or teardowns.

  • Observe how the value of default_timeout comes from Variables section due to Layer III overwriting. When value is given from command-line then Layer IV is in effect and overwrites the value.

Execution with Suite Setup

# 1. Enable Suite Setup
> robot --quiet -v use_suite_setup:on example2.robot

Check Suite Setup : default_timeout
  config: 1000, rf_variable: 1000
Check Test Setup : default_timeout
  config: 1000, rf_variable: 1000
Check Test Setup With ResetConfig As Teardown : default_timeout
  config: 1000, rf_variable: 1000
Final Check : default_timeout
  config: 666, rf_variable: 1000

# 2. Enable Suite Setup and set value from command-line
> robot --quiet -v default_timeout:2020 -v use_suite_setup:on example2.robot

Check Suite Setup : default_timeout
  config: 1000, rf_variable: 1000
Check Test Setup : default_timeout
  config: 1000, rf_variable: 1000
Check Test Setup With ResetConfig As Teardown : default_timeout
  config: 1000, rf_variable: 1000
Final Check : default_timeout
  config: 2020, rf_variable: 1000
  • Observe how Suite Setup sets the effective value at run-time. As we saw in previous execution the default value after initialization is 666. Therefore after ResetConfig in Teardown the QMobile configuration value is set back to the default 666 which originates from Variables. Similarly, when the value is set from command line it becomes as the default value and is reset to it at teardown.

Execution with Test Setup

# 1. Enable Test Setup
> robot --quiet -v use_test_setup:on example2.robot

Check Suite Setup : default_timeout
  config: 666, rf_variable: 666
Check Test Setup : default_timeout
  config: 9999, rf_variable: 9999
Check Test Setup With ResetConfig As Teardown : default_timeout
  config: 9999, rf_variable: 9999
Final Check : default_timeout
  config: 666, rf_variable: 666

# 2. Enable Test Setup and set value from command-line
> robot --quiet -v default_timeout:2020 -v use_test_setup:on example2.robot

Check Suite Setup : default_timeout
  config: 2020, rf_variable: 2020
Check Test Setup : default_timeout
  config: 9999, rf_variable: 9999
Check Test Setup With ResetConfig As Teardown : default_timeout
  config: 9999, rf_variable: 9999
Final Check : default_timeout
  config: 2020, rf_variable: 2020
  • Observe that test case using Setup has specific value. The default value comes from either Variables or command line, if given.

Execution with Suite Setup and Test Setup

# 1. Enable both Suite Setup and Test Setup
> robot --quiet -v use_suite_setup:on ^
  -v use_test_setup:on example2.robot

Check Suite Setup : default_timeout
  config: 1000, rf_variable: 1000
Check Test Setup : default_timeout
  config: 9999, rf_variable: 9999
Check Test Setup With ResetConfig As Teardown : default_timeout
  config: 9999, rf_variable: 9999
Final Check : default_timeout
  config: 666, rf_variable: 1000

# 2. Enable both setups and set value from command-line
> robot --quiet -v default_timeout:2020 ^
  -v use_suite_setup:on ^
  -v use_test_setup:on example2.robot

Check Suite Setup : default_timeout
  config: 1000, rf_variable: 1000
Check Test Setup : default_timeout
  config: 9999, rf_variable: 9999
Check Test Setup With ResetConfig As Teardown : default_timeout
  config: 9999, rf_variable: 9999
Final Check : default_timeout
  config: 2020, rf_variable: 1000
  • Observe how the QMobile config value returns to either Variables or command line value depending on the scenario.

Recommendations

The following recommendations would minimize unexpected outcomes when the default configuration values have to be changed at run-time:

  • Prefer using Variables section or command-line to set the default values of QMobile configuration variables at run-time instead of Suite Setup or Test Setup

  • Use Suite Setup and Test Setup primarily for test related variables

  • Prefer using SetConfig and ResetConfig PaceWords directly in test steps

  • If the value needs to be changed only for couple of test steps then consider using it as PaceWord parameter e.g. TypeText username johndoe timeout=30

  • Use proper and minimum scope when setting RF variables

Example 3 — Real-life Configuration

This example shows how real test cases can be run locally on Android device and in BrowserStack. This is what is needed:

  • Download ApiDemos-debug.apk and place it to resources/app/ folder. It can be downloaded from Appium GitHub. Navigate to sample-code/apps

  • To run tests locally:

    • environment setup i.e. Appium, Android Studio and Pace Console

    • connect Android device using USB cable,

  • To run tests on BrowserStack:

    • an account is needed i.e. username and API Key

    • The app uploaded to BrowserStack and obtain the link bs://<hashed app-id> or custom_id

Let’s first create a new configuration file example3_config.yaml to resources folder. The defaults are adjusted for local test execution.

defaults:
  provider: local
  device: my_android
  session: test_apk
  no_reset: true
  full_reset: false
  EXECDIR: ~
providers:
  - provider: local
    host: localhost
    port: 4723
    appium_url: http://${host}:${port}/wd/hub
    capabilities:
      remote_url: $appium_url
  - provider: browserstack, bs
    username: ~
    apikey: ~
    appium_version: '1.17.0'
    bs_cloud_url: http://${username}:${apikey}@hub-cloud.browserstack.com/wd/hub
    capabilities:
      remote_url: $bs_cloud_url
      browserstack.appium_version: $appium_version
devices:
  - device: my_android
    capabilities:
      udid: ~
      platform: android
      automationName: UIAutomator2
  - device: cloud_device
    capabilities:
      deviceName: Samsung Galaxy S20
      os_version: ~
      platform: android
      automationName: UIAutomator2
sessions:
    - session: test_apk
      capabilities:
        noReset: $no_reset
        fullReset: $full_reset
        app: $EXECDIR/resources/app/ApiDemos-debug.apk
    - session: test_bs
      capabilities:
        adbexectimeout: 30000
        noReset: $no_reset
        fullReset: $full_reset
        project: Configurations
        build: Example 3
        name: ~
        app: 'bs://e73f7ca5467f4d3cd3af660be466c33e53c97a1b'

Secondly, let’s create a new custom keywords file example3_keywords.robot also under resources folder. This will facilitate the opening of the application and ensuring correct application state.

*** Variables ***
${First_Run}            ${True}

*** Keywords ***
Open Mobile App
    OpenApp
    Set Suite Variable  ${First_Run}   ${False}

AppState
    [Arguments]         ${state}
    Run Keyword IF      ${First_Run}==${True}         Open Mobile App
    ...    ELSE         Launch Application

    Run Keyword IF      '${state}'=='Home'    GoToAPIDemosHome
    ...    ELSE         Fail                  Unknown App State::${state}

GoToAPIDemosHome
    VerifyText      App

Finally, let’s create the actual test script example3.robot with 3 test cases to the project folder.

*** Settings ***
Resource          ${CURDIR}${/}resources${/}example3_keywords.robot
Library           QMobile   configs=${CURDIR}${/}resources${/}example3_config.yaml
Suite Setup       Open Mobile App
Suite Teardown    CloseApp

*** Test Cases ***
Verify Text Between Words
    [Tags]        VerifyText
    AppState      Home
    ClickText     Accessibility
    VerifyText    Node

Verify Text Inside Word
    [Tags]        VerifyText
    AppState      Home
    ClickText     Accessibility
    VerifyText    od

Click Item with Locator Text
    [Tags]        ClickItem
    AppState      Home
    ClickText     Views
    VerifyText    Focus
    SetConfig     Scroll          True
    VerifyText    Popup Menu
    ClickText     Popup Menu
    ClickItem     Make a Popup!
    VerifyText    Search
    ResetConfig   Scroll

Execution: example3.robot

# Local Android device when only 1 device connected
> robot example3.robot

# Specific local Android device when more than 1 device connected
> robot -v udid:<device unique ID> example3.robot

# Cloud Android device Samsung Galaxy S20 in BrowserStack
# NOTE: Change username and apikey
> robot -v provider:browserstack -v device:cloud_device -v session:test_bs ^
  -v username:<bs username> -v apikey:<bs apikey> ^
  example3.robot

# Specified cloud Android device with default OS version in BrowserStack
# NOTE: Change username and apikey
> robot -v provider:browserstack -v device:cloud_device -v session:test_bs ^
  -v username:<bs username> -v apikey:<bs apikey> ^
  -v deviceName:"Google Pixel 4" ^
  example3.robot

# Specified cloud Android device with defined OS version in BrowserStack
# NOTE: Change username and apikey
> robot -v provider:browserstack -v device:cloud_device -v session:test_bs ^
  -v username:<bs username> -v apikey:<bs apikey> ^
  -v deviceName:"Google Pixel 4" -v os_version:"11.0" ^
  example3.robot

Typically, it might be more feasible to create multiple configuration files like one for local testing and another for cloud testing. This would allow using the defaults sections for both separately. Also it is fine to define the username directly in the configuration file. In general, passwords or API keys should be given as command-line parameters.