A mobile application is a company's brand available on demand. It is a window into any service or product an organization offers. At Kobiton, they understand this—it is their mission to improve mobile applications through testing. 

Kobiton is a mobile testing platform that allows customers to perform manual and automated testing on real mobile devices from anywhere in the world. It started as a cloud service, but it recently utilized MinIO to provide an on-premises solution with AI features.

Real Device Testing In the Cloud

Testing on real mobile devices from within a CI/CD pipeline may seem impossible, but it is not. This is how it works using Kobiton. The pipeline starts when code is merged. The first step is to create a build for mobile testing. This is either an Android Package Kit (APK) file or an iOS App Store Package (IPA) file. These files contain all the elements needed for an app to be installed on a device, and they are the same files that need to be created for submission to application stores. The builds are uploaded to Kobiton’s application repository in the cloud via API calls. (Kobiton has integrations with Jenkins, CircleCI, and Bitrise, which makes this easy.) 

Once uploaded, the next step is to kick off an automated suite of tests previously set up within Kobiton. There are a variety of automation frameworks for setting up these tests. Appium is the most popular because it supports both Android and iOS. However, customers can also use XCUITest, which is the native iOS framework, and Espresso, which is the native Android framework. Once the tests are executed, customers can check the results by calling Kobiton’s APIs and then determine whether or not to move forward within their CI-CD pipeline.

Things get interesting when you consider that mobile applications have to run on several operating systems and a variety of devices. Take, for instance, an app that has to support the last 5 versions of the Android operating system and the last 5 versions of iOS. You would need one device for each version, which in this case is 10. But this number snowballs when you need to test different device versions, and the number increases even more when you want to test various operating systems and device combinations. The bottom line, this is a lot of devices to manage, and each test needs to run on each device - so there are also a lot of tests to manage. The good news is that tests within the Kobiton cloud have access to a rack of real mobile devices ready for testing, and Kobiton manages the tests.

A Market for an On-Premise Solution

When Kobiton started, they engineered their product in Amazon Web Services (AWS). However, they soon realized there was a market for on-premise deployments, so they set out to create an on-premise version of Kobiton.

The device exhaust mentioned in the previous section was stored in AWS S3. Consequently, their code took advantage of a lot of S3 interfaces. They also found it a lot harder to engineer for performance when you are also responsible for everything from the bare pedal up. In the cloud, the hardware is abstracted away from you, so getting the needed performance is easier. To solve their storage problems, they sought something that had an S3 interface and was highly performant, scalable, extensible, and fault-tolerant. That is what brought them to MinIO.

When they started looking deeper into MinIO, the main things that stood out were fault tolerance and performance. It was fault-tolerant beyond what they got from a RAID solution. They also found that they could extend or expand the storage capacity with no adverse impact on performance. 

“We needed an S3 compatible, performant, scalable, extensible, and fault-tolerant object store for Kobiton. When we found MinIO, it was one of those times in technology development when you found something that was a perfect fit for what you were looking for.”

-Stu Hume, Vice President of Engineering, Kobiton

Mobile Testing with AI

A lot of data is generated when Kobiton tests a mobile application since Kobiton does more than save the test results. Rather, they save metrics pertaining to everything going on within each device while tests are running. Device exhaust is what Kobiton refers to as all of the data emitted by a device during testing. This includes:

• Test steps with details in XML
• Screenshots
• Full video capture at 30 frames per second 
• Device logs 
• System metrics 
• Network payloads
• Device health statistics

One of Kobiton's differentiators is their ability to extract this exhaust efficiently while running testing. 

Wherever there is data, there is also an opportunity to provide greater insights using AI. Kobiton then used all this device exhaust (stored in MinIO for their on-premise offering) to create models capable of healing failed scripts. When a test script fails because the developer changed something in the app, instead of the script failing, a model trained on the device exhaust of previous tests can auto-correct and figure out the best matching element allowing the script to continue.

They also use deep learning models to help identify problems with the app’s user interface. For example, many UI-related tests use a baseline screenshot to represent a particular screen's correct look and feel. When a UI test is run, it will take a screenshot and compare it to the baseline. Often, the two screenshots will be subtly different when there is nothing wrong with the UI. A model that can help you be smart about the differences is important if you want to prevent annoying false negatives.

The Results

Today, Kobiton can offer customers who wish to test in their own data centers an on-premise solution. Using MinIO for S3-compliant object storage, Kobiton’s on-premise solution is fault-tolerant, scalable, and performant. Using all the device exhaust stored in MinIO, they can create machine-learning models that improve test results.

When I think back over the last couple of years at all the different issues I've had to resolve, not once have I had to resolve an issue related to MinIO. It just works. It has been the most fault-tolerant and reliable part of our platform.

-Stu Hume, Vice President of Engineering, Kobiton

To learn more about Kobiton, visit https://kobiton.com and to hear the podcast check out: MinIO and Kobiton Podcast - Revolutionizing Mobile Testing with Big Data and AI 

If you have any questions join our Slack Channel or drop us a note at hello@min.io.