Using Bitbar CI for Cross Platform Mobile Testing

Bitbar Continuous Integration is a SaaS product for mobile testing on iOS and Android platforms.

This post will guide you through a typical server-side iOS set up for a test run of two of the most popular frameworks for JavaScript (ECMAScript 6): Jasmine and Cucumber. Both will use Appium and Protractor as proxies between the mobile application API and test code.

These days we have a plenty of options to test web and mobile applications, but when the subject comes to JS-based frameworks, only a few have something impactful to offer. One of the leaders is Bitbar CI (https://bitbar.com) and their approach looks really modern. In the beginning, when you first set up a project, you will get a VMware (Docker) instance that comes with popular build and package management tools installed.

Bitbar supports the following languages (related mainstream frameworks are supported OOTB):

• C#
• Java
• JavaScript (NodeJS is also installed by default)
• Espresso
• Python
• Ruby
• XCode project support

Besides that, an instance of Appium can also be run on demand. The current version as of this publishing is 1.6.3.

Server-side project set up

On first login to the Bitbar CI management panel, navigate to the Project tab, fill in project name, and choose the Appium iOS server side option from implementation type of project.

That’s it! With set up complete, you are now ready to prepare test code and the application.

Application requirements

The application must meet the following requirements:

1. Application size should not exceed 500 MB.
2. Application must be properly signed with Ad Hoc provisioning profile. Applications with any other signature or missing signature will be rejected on runtime and tests will fail.

Now that the project is set up, you will need to prepare the test app:

1. Navigate to https://developer.apple.com/, login with your AppleID developer credentials, open the Certificates, Identifiers & Profiles page.
2. Open the App IDs tab in the Identities section. Add a new App ID for the test app. At this point, Apple might ask for a device to assign for this app; use any iOS device.*
3. Open the Distribution tab in Provisioning Profiles section, and add a new Ad Hoc distribution profile.
4. Open any iOS project in XCode > Preferences > Accounts > Add new account > input AppleID developer account.
5. In Project build properties, go to General > define developer account for signing, let XCode automatically manage application signatures. Then, change Display name to the one given during p.3. Change bundle identifier to match App ID following the pattern com.moduscreate.App ID where com.moduscreate is your personal domain.
6. Set generic iOS device destination, build .ipa file.
7. The application is now ready for testing.

*All the listed operations require administrator permissions. If any options are disabled and you are a member of a corporate Apple account, contact your group administrator.

Jasmine set up

You are free to use any custom framework, including Jasmine or Cucumber (not officially supported), though each requires proper configuring. By default, Bitbar CI cannot recognize test outcomes from any reporting tool. To make reports visible from the Bitbar control panel, provide test results in an XML file and post it into the specific directory. If screenshot on failure is also desired, you can use the Screenshot reporter module below.

A test run always starts from the run-tests.sh file, which must exist in the project root directory.
In the example of run-tests.sh below, I removed Gulp runner and simplified the initial run to the generic npm test command:

This is an example of setting things up with Appium and Protractor, using Jasmine test runner.

run-tests.sh:

#!/bin/bash
changeNpmCacheLocation(){
  mkdir .npm_cache
  npm config set cache ~/.npm_cache
}

startIOSWebkitProxy(){
  node /opt/appium/bin/ios-webkit-debug-proxy-launcher.js -c ${UDID}:27753 -d > ios-webkit-debug-proxy.log 2>&1 &
}

startAppium(){
  if [ "$(uname)" == "Darwin" ]; then
     startIOSWebkitProxy
     appium-1.6 -U ${UDID} --log-no-colors --log-timestamp --show-ios-log
  elif [ "$(expr substr $(uname -s) 1 5)" == "Linux" ]; then
     appium-1.6 --log-no-colors --log-timestamp
  else
     echo "Operating system not supported, exiting..."
     exit 1
  fi
  sleep 10
  ps -ef|grep appium
}

initializeTestRun(){
  unzip tests.zip
  npm install
  npm install -g npm
  rm -f tests/reports/*
}

executeTests(){
  if [ "$(uname)" == "Darwin" ]; then
     npm test
  elif [ "$(expr substr $(uname -s) 1 5)" == "Linux" ]; then
     node_modules/gulp/bin/gulp.js androidServerSide
  fi
  cp ./reports/*.xml TEST-all.xml

  if [ ! -f "TEST-all.xml" ]; then
     echo "Error occured during the test run";
     exit 1
  fi
}
changeNpmCacheLocation
startAppium
initializeTestRun
executeTests

package.json:

"scripts": {
 "test": "protractor configs/config.js"
}

Other system directories:

• <project root>/screenshots – the directory where Bitbar CI seeks for any screenshots created during test run. Cleaned on each run.
• <project root>/reports – the directory for XML report. If not provided, summary for test run will not be posted, analytics in dashboard cannot be used. Cleaned on each run.

The config below contains a typical installation for XML reports and screenshots for Protractor tests. Assuming that main config.js file is located at <project_root>/configs/, modify it this way:

const reporter = require('jasmine-reporters');
const ScreenshotReporter = require('./pathToScreenshotModule/screenshotReporter');

config = {
specs: ['../pathToTestsDirectory/*.js'],
seleniumAddress: 'http://localhost:4723/wd/hub',
framework: 'jasmine2',
allScriptsTimeout: 60000,
screenshots: {
   "enabled": true,
   "path": "screenshots"
},
capabilities: {
  	 browserName: "",
  	 bundleId:"com.moduscreate.<appName>",
  	 platformName: 'iOS',
   	deviceName: 'iPhone',
   	automationName: "XCUITest",
   	newCommandTimeout: "60",
},
onPrepare: function () {
   let junitReporter = new reporter.JUnitXmlReporter({
       consolidateAll: true,
       savePath: 'reports',
       filePrefix: 'xmloutput'
   });
   jasmine.getEnv().addReporter(junitReporter);
   jasmine.getEnv().addReporter(new ScreenshotReporter(__dirname.replace('configs', 'screenshots')));
},

Screenshot Reporter module:

const fs = require('fs');
const mkdirp = require('mkdirp');
const path = require('path');

const ScreenshotReporter = function (dir) {
   const screenshot = (testDescription) => {
       mkdirp(dir);
       browser.takeScreenshot().then((png) => {
           const stream = fs.createWriteStream(path.join(dir, testDescription + '.png'));
           stream.write(new Buffer(png, 'base64'));
           stream.end();
       });
   };
   this.specDone = (result) => {
       if (result.failedExpectations && result.failedExpectations.length > 0) {
           screenshot(result.fullName);
       }
   };
};
module.exports = ScreenshotReporter;

Please note:

1. You should never specify device name and OS version – Bitbar handles this itself.
2. Given capabilities are mandatory. Optional keys are supported the same as if it was a local Appium environment.

Test run

Return to the Bitbar management console and enter Appium Cloud test project.

1. Add new test run.
2. Upload test application on Application page.
3. Upload zip archive containing all the test files and run-tests.sh file.*

Choose devices to test on and click Start. Tests will run.
* Do not hesitate to add the entire node_modules directory into the archive. Regardless of the fact that Bitbar is going to install all dependencies, test preparation can be done 5-15 times faster if node_modules are already fetched and ready to use.

Upon completion, you will see the following report (test failed on purpose):

The screenshot is marked with the test name. The console log with Appium and Protractor test output is available, or download log files separately.

Cucumber set up

Initial settings for Cucumber almost completely match Jasmine set up above, except for reporting. Convert out-of-the-box CucumberJS report.json file to XML using any conversion tool, or use the snippet below:

configs.js

const cucumberJunit = require('cucumber-junit');
const fs = require('fs');
const path = require('path');

   onCleanUp: function () {
       let reportRaw = fs.readFileSync("./reports/report.json").toString().trim();
       let xmlReport = cucumberJunit(reportRaw);
       let junitOutputPath = path.resolve("./reports/", 'junit-report.xml');
       fs.writeFileSync(junitOutputPath, xmlReport);
   },
cucumberOpts: {
   require: '../pathToSteps/*.js',
   format:'json:./reports/report.json'
}

screenshot of test result

Summary

Now we have integrated Bitbar Cloud configuration into a Jasmine/CucumberJS based framework and are ready to write tests.

• Easy to use mobile CI with a wide range of devices for iOS (including iPhone 8/8 Plus, iPhone X) and Android.
• Ability to create any custom framework using supported programming languages.
• Multithreading comes OOTB, just pick devices from your test scope and run.
• Full JavaScript support (NodeJS, Jasmine, Cucumber, Protractor, NightWatch, WebDriverIO, Gulp).
• Appium server can be run on demand.
• Various subscription plans (starts from 99$ per 10 hours device time, monthly based).
• Only real devices.
• Fully customizable environment.
• Access to all devices in Manual Testing mode (remote screen interactions).
• Ability to run tests in Cloud or Local mode (in Local mode Bitbar provides Appium process and devices; framework, logs, etc. stay in local workstation).
• Built-in app crawler.
• Dashboard provides a lot of data from test runs for analysis.
• Notification integration with messengers and bug tracking tools (email, Slack, Atlassian Jira), API calls for custom messengers.
• API for remote management and debugging.

Ensure your software is secure, reliable, and ready to scale—explore our expert testing and QA services today!