Hi Guys and Gals!

I’ve been thinking about QA quite a bit today and I wanted to blog a little bit about it given some of the fails we have seen over the last 24 hours. Many people in the MDM space have had their share of negative feedback for their MDM vendor regardless of who they are (Microsoft, VMWare, Mobileiron, etc). On top of that, you have seen some major failures from individual app companies in recent memory.

I think as technologists we need to think critically and consider why over the last few years there have been some blatant failures on the front application life-cycle management. I will break this blog into a few key points: (1) some of the recent failures on the mobile app front that have created us pain, (2) a solid way for a company to perform their own QA/preparation for new OS versions, and (3) a brief discussion into some of the challenges and what we can do to bridge the gap.

Some Recent Missteps

So…yeah….vendors…you need to be better. I’m not that interested in airing anyone’s dirty laundry, but enterprise companies need people to be better. My opinion is that one of the major issues is that teams have become far too disparate. Sure we all love DevOps, but it’s not an excuse to completely silo two teams that clearly should be working closer together. I think that’s the danger of buzzwords in the IT industry today. There are so many words that C-level executives love like DevOps, collaboration, tactical, strategic, hybrid, and so many more that people like to use like a shield to deflect doing things the right way.

There is a right way and a wrong way to do everything. Obviously this is very subjective and everyone views it differently, but the goal is the same for everyone: be productive and do not hinder your workforce. It’s amazing to rollout new apps, technologies, advancements, and innovations, but if you create a bad first impression or negative impact it will disappear from a user’s device. People’s loyalty is fleeting. You can ask a product like Apple Notes, which before a year or two ago was mostly useless and could not even remotely compete with the OneNotesand Evernotes of the world.

Things get really interesting when something happens, such as a recent bug with OneNote where the app became corrupted after a password change. So, when you are a user with a broken application what do you do? You USE something else. That’s what happens. It’s the old adage that you are only as good as your last at bat. If you lose the trust of your users, people will use something else. This is very troubling to enterprise administrators. Once users start using an unmanaged application where we have no DLP or risk mitigation then you are opening yourself up to substantial troubles.

This is by no means a review of how this vendor sucks or this one sucks, but it is reality. People will always prefer to ask forgiveness than permission. One employee using an unapproved application in any regulated industry could cost you millions of dollars. That is burden put on vendors to not just innovate, but MAKE SURE that they release a better version of the application than the previous version. That ALSO means they must test these applications more effectively. As an AirWatch customer, we have seen our share of poop shows over the years with literally every application they release. Sometimes they “miss” something in testing.

I will never say definitively, but I wager that a lot of the challenges around the QA process could be that they are not investing heavily in QA. The two places that most companies cut/save money in is project management and QA. I’ve seen this myself from when I was at BlackBerry and its unfortunate. As a manager of mobile development, its hard to argue or justify the logic. Many companies believe in AGILE development practices usually iterating and often releasing monthly versions of their application. So in short, end users often end up being their crowd-sourced QA. This is a reality in the post BlackBerry world that the Android and iOS fanboys and girls will forgive hiccups in favor of their ability to play Angry Birds.

I won’t focus too much more on this, but I thought a recounting of a few of the bugs that I have seen in mobile apps over the last year can summarize what I’ve been talking about for the last few paragraphs. I think as consumers it is inevitably our fault. We forgive far too easily and don’t hold grudges. We just say “its okay.” The reality is that Steve Jobs is not walking back through that door (God rest his soul) and Android is a convoluted mess with far too many flavors. The Android pundits will say “but Android Oreo” and blah blah, but the reality is that many people cannot afford devices that can support Android Oreo. Despite the ever-growing pliability of mobility nothing is one size fits all. A few of the examples I mentioned earlier:

• App doesn’t work offline
• App crashes regularly
• Vendor fails to adopt new SDK required for a new OS
• UI inconsistencies
• App frequently doesn’t support new OS features for far too long
• Security risky behavior (corrupted key chain, not using HTTPs, etc)
• Airdrop breaking the ability to share attachments to iMessage

A blueprint for mobile QA

Quality Assurance in an enterprise is very interesting. First, we need to get into the pre-requisites. You need a few key things to get started. The initial ground work you must do are:

• Maintain a list of what will be in your MDM App Store
• You will need a developer account with Apple and/or be familiar with the developer preview process for Android
• Procure/set aside test devices for each form factor that your company will need to support

Your pre-reqs are really about covering your vectors. This tells you: (1) I know what device types, sizes, and models I need to validate, (2) I know what apps are the lifeblood of my company, and (3) I am able to get trusted versions of the OS to test what is coming out. Once you have done that, you need to work on covering the potential gaps. Most teams just go: “Here’s my list of apps and let’s see if I can open the app, login, and it looks okay?!” That is setting yourself up for failure.

Hopefully you are not responsible for testing/validating EVERY single Web App at your company. Testing apps really comes down to two different types: (1) Testing for a new OS version and (2) testing new features that the app team is deploying. In most circumstances, you will likely just need to make sure that your internal apps still work on the new version of the OS. Let’s start with the OS testing and then we’ll talk more about internal app testing, which many mobility engineers don’t understand how it works.

OS Testing

Operating System testing is mostly a manual process. The best way to approach OS testing is to write it out. I know that sounds simplistic, but you can break it down into a few simple areas. I usually recommend maintaining a spreadsheet/Wiki to track these areas, which will allow you to easily delegate to offshore/low-level resources. Those areas are:

• MDM Enrollment
• PIM Testing (Email, Contacts, Calendar, Tasks, etc)
• Restriction Validation (This entails testing if all of the MDM controls you are “controlling” actually work still) A few examples are:
  • Secure Open In
  • Restricting AirDrop
  • iMessage
  • iCloud Drive
• Popular App Validation (This is testing out the apps people use frequently to ensure they still work)
• Battery tests to ensure that a device’s battery usage is not deviating too much. I strongly suggest the native iOS battery app and theGSAM Battery Monitor for Android
• VPN App validation, specifically ensuring that your Split Tunnel/Per-App VPN/VPN On Demand works like it was previously
• Testing all of your MDM vendor apps to ensure they aren’t making your device throw up all over itself AND you.

App Testing

The hope is that your application teams are using something like Appiumwhich allows your app team to automate testing of their native or web apps in a structured matter. If you aren’t overly familiar, a team writes scripts and code with frameworks like Appium (hopefully using the Selenium drivers to really make the testing amazing) to automate their tests. Typically with a new OS, you should not need to do much more than open the application, make sure VPN/Proxy actually works and the application will let you go in and all of that jazz.

The way that I approach application testing is with a logical map of the expected user actions. I’m being purposefully vague as a company “recently” stole my idea and that app is coming out “very” soon. That’s okay though, karma is a funny little thing. What I do more or less is use screenshots and Visio to map out every potential action someone would take with my application, such as:

• Search for a person
• Change each of the settings
• Access each menu, etc

You get the hang of it! I document every potential scenario and then test each one of them when a new OS/Beta is released. Personally, I am a very visual person so that is a useful way for me to know each item has been accounted for. Testing/validation is usually about what works best for you. The key is to make sure you don’t miss something. The most important thing with internal apps is user adoption. You must earn people’s trust and build confidence. When you miss a vital bug, it could easily cause someone to stop using your application.

The Challenges in QA

So I have spent the last 1600 or so words talking about the missteps and how you can test your own applications. The question is “why does this happen?” We can all make excuses and say “this or that” but the reality is that it doesn’t matter. The question is “How can I prevent it?” That answer is not simple. It comes down to mitigation and not prevention. There are many struggles people are unaware of.

One struggle that I have seen with Microsoft for example is that the Intune SDK is completely separate from their mobile apps. A mobile app team could completely decide “we aren’t going to use the SDK” like the RDP app team decided. That is a major problem for companies heavily invested in DLP/risk mitigation aka EVERYONE. I’ve discussed on my other blogs about mitigating risk, but it doesn’t always come down to us. Sometimes you need to make a drastic change like dumping one MDM for another one if it can’t give you what you need. It’s NOT about doing what is cool and innovative! It’s about SOLVING the actual problem. I see people all the time wanting to use technologies that are new and cool without actually asking themselves “how does this actually help me?”

Application degradation and quality can often be summed up with “putting lipstick on a pig”. This is true for a few reasons. Sometimes companies try to keep a dying application alive to affect their bottom line and sometimes as customers we need to come up with innovative ways to make an app work for us. One way that I think can solve the gaps we often deal with on both internal and external apps is something likeAppDynamics that our friends at Cisco now own. AppDynamics can monitor your applications in real-time and notify you of application crashes, performance issues, and much more.

Overall, application issues are unavoidable, but it comes down to communication, mitigation, regularly assessing an application’s viability. As mobility engineers, we cannot be afraid to remove our “rubber stamp” on an app and remove it from the App Store entirely. That is a powerful statement to your end users. Firstly, removing it from the App Store shows you no longer trust the application and secondly you are no longer accountable for it. My personal feeling is that every application in my App Store must be re-evaluated like I would RFP competing MDM vendors. An App Store is a promise to your end users that says “I recommend these apps and you can count on them to make your life easier”

In Closing…

I hope that some of my words are useful for those of you that are constantly being challenged by the ineffective QA happening in our industry. I think it’s so important to take our ego’s out of it. That means every vendor, app team, and engineer must not take it personally. We are all responsible for QA! We can’t sit there and blame VMWare, Microsoft, Dell, the Easter Bunny, Santa Claus, Amazon, etc. for bugs. We must take it into our own hands and make sure we are NOT the last people to know. If we can mitigate these issues, then we MUST. People rely far too much on vendors today. We should trust but verify and work diligently to deliver a great experience to every user regardless if they are the CEO or an intern.