Mobile App Testing: Real or Virtual Device?

Native Mobile vs Web Apps

The digital landscape has drastically changed in the last few years. More and more users access the web through their mobile devices and businesses must accommodate their needs to stay relevant. To meet users’ demands, businesses started to incorporate mobile application testing as a part of their testing strategy, along with web application testing. I have decided to look at Mobile testing and find out if there is a difference in testing on real versus virtual devices.

Native Mobile Applications are platform-specific apps that users can download from the Google Play Store or Apple App store. Native iOS apps are written in Swift or Objective-C and native Android apps are written in Java. This makes it hard and expensive to build and maintain these apps, as developers need to maintain two different codebases for Android and iOS.

Mobile Web Applications are standard web applications that can also be accessed through mobile browsers. Mobile web apps can be tested on any mobile device in different browsers like Chrome, Safari, Firefox, Edge or Opera.

Hybrid Applications are developed across all platforms, not only for a specific operating system. It is a combination of a native app and a web app. Hybrid Applications are built with HTML, CSS or JavaScript and run in a web view. These are easier for developers to maintain than native apps.

Native Mobile and Hybrid Applications are tested similarly. Functional testing includes not only verification that features work as expected, but we also test if the app can use built-in resources properly. Screen orientation testing checks Graphics and UI. We also need to test the ease of installing and uninstalling the app. Compatibility testing includes verification of the application behavior on different devices. Connectivity testing verifies that the app is working properly with the multiple types of network connections, that it can switch between different types of connections, and how it works without the network connection.

To test on my MacBook Pro, I have installed an Android Emulator, an iOS Simulator, and have two real devices to connect to my computer with cables: an iPhone 12 and a Samsung Galaxy Note20.

Virtual Devices

In absence of physical hardware, virtual devices can be used for testing. The Android Emulator simulates Android devices, so you can test applications on a variety of devices without the need to have each physical device present. I downloaded Android Studio to configure virtual Android devices needed for testing.

Android Virtual Device Manager

The iOS Simulator (which is a part of Xcode) can be used to test the basic functionality of iOS apps and UI layout. I used Xcode, which I had previously downloaded on my MacBook Pro, to configure the iPhone 11 simulator.

iOS Simulator

Real Devices

To connect an Android phone I used for testing, I downloaded an Android File Transfer so I can browse and transfer files between my Mac and my Android phone.

•  I connected the phone to my computer with the cable, changed its configuration to enable the developer mode, and then opened google.com in the Chrome browser.
• On a MacBook using Chrome browser, I entered chrome://inspect/ as the URL, which opened the DevTools for Devices. I clicked on ‘inspect’ and DevTools for google.com opened.

Android Chrome

While connecting my iPhone to Safari Web Inspector, I found a bug: Safari Web Inspector does not work well with the latest version of Big Sur. I found a temporary fix for this problem, for which I had to download Safari Technology Preview (a beta version of a new Safari).

• On the iPhone: Safari settings -> Advanced -> Enable Web Inspector
•  In Safari (Safari Technology Preview, in my case), change Advanced Preferences to “Show Develop menu in menu bar” and the Develop tab shows up in the top menu.

Safari Advanced Preferences

I connected my phone with the cable and opened google.com on Safari. The Develop tab showed my connected phone and let me pick one of the opened URLs to inspect, which then opened the Web Inspector.

Safari Web Inspector

Virtual vs Real Devices

I believe the advantage of using real devices for manual testing is the availability of the Web Inspector/DevTools for Devices. Virtual Devices lack these tools. Saving logs from web inspectors can also save developers time during debugging. To test the application across real-world scenarios, it is essential to use real mobile devices. Sanity testing and User Acceptance testing should be done with real devices, as it will give the most accurate result. It is not possible to test authentication (face ID, fingerprint, etc.) on a virtual device.

Virtual devices are great for unit testing and automating mobile UI tests. With frequent iterations that are necessary in the beginning stages of mobile web development, using virtual devices for testing can be more efficient. Virtual Devices work much faster than real devices when it comes to testing, which enables teams to practice shift-left testing, to test mobile applications earlier in the cycle. To avoid the risk of escaped defects, real devices should be used, as some defects can only be caught on real devices.

Nowadays, Mobile applications play a significant role in our daily life, so they require a lot of testing to make them work as they should. Companies need to find an optimum combination of simulators/emulators as well as the real devices necessary to maintain high standards and quality assurance.

About the Author:

Tanya Brant is a Consultant in our Milwaukee Development Center. She lives in Illinois with her husband and two children and does manual testing, UI test automation, mobile testing, API testing and API test automation.