IBM case study

“For us, the combination of “shake to report”, the integration with JIRA and the video recordings really was a WOW factor”– Marcelo Juliano Ramos, IBM Watson automation specialist.

OVERVIEW

MEET MARCELO JULIANO RAMOS

Marcelo is an a Watson automation specialist based in the IBM office in Brazil.

IBM Watson is an artificial intelligence technology, based on cognitive computing that understands natural language and is capable of answering questions posed in multiple languages.

As part of his role in IBM, Marcelo works with Volkswagen on an in-house innovative new project involving Watson technology for the new Volkswagen Virtus.
This new car model will offer a cognitive manual, which utilizes IBM Watson to respond to drivers’ questions about the vehicle, including information from the car manual. The app understands natural, informal speaking and “learns” more based on the interaction with the driver. The driver can either speak, write or take photos to interact with the app. Any “how to” questions about the car can be answered by this smart assistant, simply and quickly, in multiple languages. This solution allows a new way of interacting with the vehicle and offers a new technological experience to driving.

ABOUT TESTFAIRY

TestFairy is a mobile testing platform that enables teams to find, report and solve complex problems with their mobile apps, all from one dashboard. The platform includes an enterprise  app-distribution solution for iOS and Android apps, and provides an SDK that collects valuable information such as videos showing user behavior, app logs and crash reports. This information is collected and presented in an organized way, in a single, easy-to-use dashboard helping teams to quickly reproduce and understand problems, allowing for accurate and timely bug fixes.

THE CHALLENGE

“Users simply did not report on all problems because the manual process they had for reporting feedback was tedious and unfriendly”

As the Volkswagen Virtus project has many features and complex capabilities, and testing is done both in the lab and in the field for cars, the IBM and Volkswagen teams needed an organized way to manage testing and ensure the app is as quick and easy to use as the vehicle it accompanies. The app was to undergo testing by personnel from both companies, and so the teams needed to make sure they can both access and share the information collected throughout the process.

Reflecting on past experience working on an HR-focused app within IBM, Marcelo said their biggest problem was receiving enough useful feedback and managing this amount of data in a manner that allows the team to track problems, reproduce and fix them on time.
Users simply did not report on all problems because the manual process they had for reporting feedback was tedious and unfriendly, requiring a user testing the app to decide to take screenshots, attach them to an email and explain the situation.
As colleagues admitted to the team, it often put them off of reporting some issues, as the process required to much effort on their part. In addition, the received reports were not always useful, as sometimes the recordings may be partial or the information provided inaccurate or insufficient. This lead the team, on many occasions, to rework and waste time on recreating issues that didn’t have enough information provided on them.

Finally, a lot of data collected from the testing was eventually lost because there was no organized method of storing and using it in retrospect.As two teams are testing the app in its development process, they needed a way to efficiently manage testing throughout the development cycle and in transition from the IT development at IBM to a more user experience oriented approach at Volkswagen.

THE TESTFAIRY SOLUTION

“I no longer have to wait for users to tell me what they’ve experienced. Now I can see it for myself”

After the trying experience of unorganized testing, Marcelo’s team started using TestFairy, and that made all the difference. The fact that all sessions are automatically recorded, and contain all the relevant data the development team may need to solved their biggest problem.

What actually made the testers hop on board was how incredibly easy the reporting process has become. All they had to do is shake their phone, and a bug report with a full recording and attached screenshots, logs and metrics was automatically created and sent.
For Marcelo’s team, the fact that those reports automatically open up issues on their JIRA ensured that no more bugs are missed, and all the relevant data for fixing them is available right there on that ticket. For issues users do choose to report, having all the session recordings at hand enables the team to check if other users experienced them as well.

Now, with more reports flowing in with comprehensive and accurate information, the team is able to quickly reproduce and fix problems, as well as fully understand the actual user experience, all before releasing the app.

TO SUM IT UP:

Testing and getting feedback during the development process of an app is no easy task in any company or setting. The task is even harder when it involves a complex app and two developing companies. By using TestFairy, the IBM team is able to streamline this process and effectively run testing on their app, get more reports thanks to the super easy feedback it enables, and have urgent issues promptly sent to their JIRA. No more bugs missed, no more waiting for users – the teams can now watch the entire experience’s recordings from one dashboard.

Mobile Beta Testing – TestFairy is Integrating With Atlassian Products

TestFairy announces compatibility with JIRA Software, HipChat, Bitbucket, Bamboo and Atlassian Bitbucket Pipelines
It is not every day that you get invited by a company like Atlassian to spend a whole week in Amsterdam. And to make it a perfect geek party, you will be hanging out with developers, marketers and product managers from all product teams who will help you integrate with their platform. So are you coming or not? D’aah? And we landed in Amsterdam 🙂
To make a long story short, after less than 24 hours with the teams of JIRA Software, HipChat, Bitbucket and Bamboo, and one night without sleep, everything worked!
First came JIRA Connect.
One of TestFairy’s most useful features allows mobile testers to easily send bug reports to their developers. Testers can do that by logging in to their mobile dashboard, by using the TestFairy testers app, or by simply shaking their phone and filling in a feedback form that pops up.
Many of our customers asked us if we can post this feedback directly to JIRA Software, together with the video that shows what exactly happened on the app before the bug was reported. And so we did.
As a result, if someone says “It is not working”, and you’ll be surprised how many testers say that, you won’t have to ask them “What did you do?” “Did you click here or there?” “Did you have wifi or 3G?” “Can you send me some logs?”… Instead, just open the JIRA issue, rewind the video 10 seconds back and see exactly what happened before the user complained, and what exactly made them unhappy.
This is how a JIRA Software issue looks like with an embedded TestFairy video:

We did the same with crash reporting, where we pushed our video and logs together with every stacktrace, helping developers understand what exactly happened before their apps crashed.
It gets more interesting when the same crash occurs multiple times and the last thing you want is to have ten new issues on a crash that happened ten times (not mentioning thousands). We solved this problem by hashing every stack trace, later using it for grouping, so that one JIRA Software issue will be opened, regardless how many times your app crashed.

Issues created on JIRA Software create these magic properties. These properties include the name of the tester, the device he or she was using, operating system version, etc. You use these to later query JIRA by using JQL or simply by clicking on a property within an issue. JIRA Software will then automatically show you only the relevant issues, as in the screenshot above, only issues generated on “Apple iPhone 5”.
TestFairy for JIRA add-on is now available on Atlassian Marketplace.
Then came HipChat.

HipChat brings testers and developers together. Collaboration is instantaneous, and with TestFairy you can get testers’ feedback directly and immediately to your HipChat rooms. As a developer, you may also want to be notified when a new version has been uploaded, or a crash occurred.
Integration could not have been simpler — thanks to HipChat Connect.
All you need to do is click on
and choose the right HipChat room for your notifications. You may also configure which types of notifications you are interested in receiving.
We have found that it is a good practice to post tester’s feedbacks to one room, and crashes to another. Mainly for the reason that in most teams, different people handle different cases.
Now, when a user sends a feedback, we can easily see their notification in real time.
TestFairy for HipChat is available through the Atlassian Marketplace or directly from your account settings page on TestFairy.
Then came Bamboo.
Bamboo is where everything starts. Our plugin for Bamboo uploads your app to TestFairy whenever it is ready and automatically sends it a pre-defined group of testers. You can invite by email, or via a notification in a HipChat room.

This brings us to a platform that has TestFairy connected to JIRA, HipChat and Bamboo.
Then came Bitbucket.
Bitbucket Cloud holds the source code for your app. We closed the loop between source code and crash, by creating a link between the two. When showing a crash, TestFairy will automagically link the stack trace to your Bitbucket repository, at the right branch and revision.
And finally, Bitbucket Pipelines.
Bitbucket Pipelines is the continuous integration tool that glues the entire system together. Bitbucket Pipelines automatically compiles your mobile app with every commit. TestFairy integrates seamlessly to Bitbucket Pipelines to upload app artifacts, the .APK / .IPA files to TestFairy for distribution.
In the following example you can see an app that crashes.
The notification regarding this crash goes to HipChat in real time, together with a link to the actal TestFairy session.
Clicking on the crash will take you back to Bitbucket to the exact line that caused this crash.
We’ll be happy to get your feedback in the comments below.

App Responsiveness

First-of-its-kind Solution for Slow App Detection to be Launched by TestFairy at Google I/O
San Francisco, May 28th, 2015: TestFairy, the provider of the video-enabled beta testing platform for mobile apps, announced today a new service, a first of its kind, allowing mobile developers to understand when their apps are slow to respond.
TestFairy will demonstrate its new service at Google I/O 2015, an annual conference featuring inspirational talks, which gives developers a chance to hear more about Google’s latest developer products. Google selected TestFairy to present at Google I/O as one of the startups participating in its Launchpad program.
“App Responsiveness is a score given to the speed-ness of apps, indicating how fast they respond to user interface (UI) events,” Explained Gil Megidish, TestFairy Co-Founder & CTO. “Every time a user touches an on-screen button or scrolls down a list, the operating system pushes events into the main thread’s queue. Before the screen updates, this message queue must clear. Very often, app developers perform processing on the same main thread. This results in slugginess of the application and frustration of the user, as the app does not respond fast enough and seems slow and unreliable. TestFairy measures how slow your main thread is and, if a piece of code blocks it from processing user-interaction events, TestFairy will alert and provide the complete stack trace of the faulty code. These bugs are hard to trace as they may be caused by a bad network connection or long items in a list.”
TestFairy is the first company to provide a solution to this problem, as part of its mobile beta testing platform.
The TestFairy platform helps mobile developers by providing videos that show exactly what happened on their apps while being tested. In addition to videos, the service provides deep analysis data such as CPU usage, memory allocations, phone signal and wifi stats, along with logs and symbolicated crash reports. By providing developers with more information regarding bugs, TestFairy helps them fix bugs faster and cut the costs of testing and development.
The service is available for iOS and Android and will be available for Windows Phone later this year.
TestFairy, founded in 2013, was selected by many developers as the perfect TestFlight replacement, allowing mobile developers to efficiently test their apps. The company recently announced full compatibility with Xamarin Insights and Adobe Air. Developers on these platforms can now experience many improvements to their mobile app testing.
For more information please visit www.testfairy.com

TestFlight Replacement

TestFlight shutting down, TestFairy to the rescue!
TestFairy, the leader in deep analysis mobile beta testing, announces today its iOS platform following high developer demand after Apple closes testflightapp.com
Tel Aviv, February 16th, 2015 : TestFairy, a company that offers a deep analysis Android beta testing platform for companies such as as Orange, Telefonica, Flipboard, Citrix and over 10,000 other mobile developers is announcing today its launch on iOS, with a complete TestFlight alternative.
TestFairy is totally self-funded and has been working on its iOS solution for over a half a year. Due to high developer demand, the company decided to officially launch today.
The announcement comes at a critical time for developers, because just last week Apple announced that on February 26, The TestFlight website will be shut down and all iOS developers will be forced migrate their apps to the new TestFlight service available inside of iTunes Connect.
This migration is highly problematic for developers due to Apple’s lack of support of testing options for iOS7 that still runs on 27% of iOS devices, along with many other limitations such as limitation of two uploads per day or manual reviews. This developers to look for a TestFlight replacement.
Yair Bar-On, CEO and co-founder of TestFairy explains “We are excited to offer our industry leading technology to the hundreds of thousands of iOS developers looking for a robust app testing platform. TestFairy offers developers not only app distribution like other platforms do, but is focuses on telling the story behind the testing process and making it painless. TestFairy provides in-depth videos that show the developers what exactly happened during the test, as well as detailed graphs about CPU, memory, battery, logs, crash reports, heatmaps and a lot more. We are experiencing unprecedented growth and are looking to finalize our fundraising process shortly.
The app testing space has been heating up lately with some major acquisitions including Apple’s purchase of TestFlight, Microsoft’s acquisition of HokeyApp, Twitter and Crahlytics and Google’s announcement of their Appurify acquisition in the last Google I/O.
TestFairy has positioned itself as the leading independent beta testing platform and the only one that can provides videos, crashes and detailed insights about the testing process. All this saves time, and enables developers to deliver faster and dramatically improve their product quality. Finally, the company is in deep negotiations with major players in both the mobile and testing spaces regarding strategic partnerships.
About TestFairy
TestFairy, is the leader deep analysis mobile beta testing platform, providing app testing solutions for iOS and Android developers. The company was founded in 2013 by Gil Megidish, CTO and Yair Bar-On, CEO and has more than 10,000 developers on its network.
For more information please visit www.testfairy.com