QA on Phones People Actually Own
A build that behaves perfectly on a simulator can still fail on a three-year-old handset with a warm battery and two bars of signal. Mobile QA is about the conditions a laptop never reproduces — and about the interruptions the web simply does not have.
A Simulator Is a Drawing of a Phone
It renders your screens faithfully and lies about everything else. Six things it cannot give you, all of which decide whether your app survives its first week.
Real heat
Phones throttle. Ten minutes of camera work, video playback or a busy map and the chip slows itself down to shed heat. Frame rates that looked fine in a thirty-second demo fall apart in the session your user actually has, and nothing on a desktop reproduces that curve.
Real networks
A simulator runs on your office fibre. Users are on a train, in a basement car park, or on a connection that is technically present and functionally useless. What matters is the slow request: a timeout you handle, a retry that is safe to repeat, and a screen that says something rather than spinning forever.
Real memory pressure
A budget handset with a dozen apps open has very little free RAM. Android will kill your process while it sits in the background, and iOS will do the same under pressure. If the app does not restore what the user was doing, they come back to an empty form and blame you rather than the operating system.
Real interruptions
A phone is first a phone. Calls, alarms, notification banners, the lock screen and another app stealing the foreground all arrive uninvited, often halfway through your checkout. This is the largest category of mobile bug and it barely exists on the web.
Real battery behaviour
There is no battery in a simulator, so there is no low-power mode, no background restriction, and no manufacturer battery manager quietly deciding your sync job is not important. Those settings change what your app is allowed to do, and they are switched on by default for a large share of real users.
Real storage speed
Cheap flash storage is slow, and a phone that is ninety percent full is slower again. Databases that open instantly on a developer machine take seconds on the handset a large part of your audience carries. Cold start is where this shows up first, and cold start is what people judge you on.
We still use simulators constantly — for layout iteration, for unit tests, for reproducing a specific screen size quickly. They are just never the thing that decides a build is ready.
Test the Handsets Your Users Have, Not the Ones on the Shelf
The most common mistake is testing on whatever the team happens to carry. Development teams carry recent phones with plenty of free storage and a fast connection, which is precisely the population that never sees your performance problems.
So we build the matrix from evidence. If you already have an app or a website, your analytics already know which devices and OS versions your audience is on, and that breakdown drives the list. If you are starting from nothing, we work from the market and the price bracket your customers buy in — a field service app for a logistics firm has a very different device profile from a premium fitness subscription.
The matrix then goes into the scope document in two tiers: must-pass devices that block a release, and best-effort devices that get checked before larger releases. That turns testing into a defined quantity of work rather than an argument at the end of the project.
| Tier | Device profile | What it proves | How often |
|---|---|---|---|
| Must pass | The oldest OS version you have agreed to support | That your stated minimum is real rather than aspirational, and that nothing in the build quietly requires a newer API | Every release candidate |
| Must pass | A budget Android handset with slow storage and little free RAM | Cold start, scroll smoothness, image memory, and whether the app survives being killed in the background | Every release candidate |
| Must pass | A current iPhone on the current iOS release | The platform your most vocal reviewers use, plus anything the newest OS has changed underneath you | Every release candidate |
| Must pass | The single handset most common in your own user base | The experience the largest share of your audience will get on launch day | Every release candidate |
| Best effort | The smallest screen still in meaningful circulation | Layout at the tightest width, with long translated strings and a large system text size | Before a feature release |
| Best effort | A tablet, large phone or foldable, if any is in scope | Layout stretched rather than scaled, and correct handling of a fold or a resize mid-task | Before a feature release |
| Best effort | One handset with an aggressive manufacturer battery manager | Whether background sync and notifications still arrive once the vendor layer starts making its own decisions | Before a feature release |
| Conditional | Hardware the app genuinely depends on: NFC, a specific camera class, biometrics, Bluetooth peripherals | The features that cannot be faked or stubbed, tested against the real thing | Whenever that feature changes |
An illustrative structure, not a fixed list. The real matrix is written for your audience and agreed before build starts — see how we build.
The Test Cases the Web Never Taught Anyone
A browser tab is rarely interrupted. A phone app is interrupted constantly, and almost always at the worst moment. Each of these is a scripted case run by hand on a real device.
A call arrives mid-task
The app is pushed aside, audio is taken away, and the user comes back three minutes later. Does the payment sheet still exist? Is the half-typed form still there? Did a video keep playing to nobody, and did the upload continue or silently stop?
The battery drops into low-power mode
Background refresh is curtailed, animations may be reduced, and network activity gets deferred. An app that assumed it could sync whenever it liked starts showing stale data with no explanation, which reads to the user as broken rather than as frugal.
Backgrounded halfway through
Someone switches to their messages to copy a verification code, then returns. The rule we test for is that nothing is lost and nothing is silently repeated — a half-submitted order must either complete once or not at all, never twice.
The OS kills the app outright
Process death is routine on Android and happens on iOS under pressure, and the user does not experience it as a crash. They experience it as the app forgetting them. We force it deliberately on a low-memory handset and check that state is restored, the session survives, and the app does not reopen on the first-run screen.
Signal disappears mid-request
The hardest of the set, because the request may have reached the server and only the reply was lost. That is why anything that changes data has to be safe to send twice, and why we test the lost-reply case specifically rather than only the clean offline case.
The screen locks and returns
Sensitive screens should not be legible in the app switcher, biometric re-authentication has to happen at the right moment, and timers, maps and live sessions have to pick up where they were instead of resetting to a loading spinner.
Every Permission Has a No Path, and It Is Rarely Tested
Most teams test the happy path where the user taps Allow. The store reviewer will test the other one, and so will a large share of your audience.
Plain refusal
The user declines the camera, location or contacts prompt. The screen that needed it must still be usable, or must explain calmly what is unavailable and why. Dead ends and blank screens here are a routine review rejection as well as a bad experience.
Allow once, then gone
iOS lets people grant access for a single session. Your app can hold a valid permission on Monday and none on Tuesday, so permission state has to be checked at the point of use, never cached at launch and trusted forever.
Revoked while you were away
Permissions can be switched off in system settings while the app sits in the background. On return the app has to notice, rather than carry on calling an API that now returns nothing at all.
Partial photo access
The user shares three photos rather than the whole library. An app written on the assumption of full access shows an empty grid and looks broken. Limited selection needs its own designed state and an obvious way to pick more.
Approximate location
Where the user grants a coarse position instead of a precise one, everything downstream has to degrade sensibly — a wider search radius, a map that does not zoom to a street corner it cannot see, and no nagging to change the setting before any value has been shown.
Notifications declined
Push permission declined at the wrong moment is usually declined forever. We test that the app still works without it, that anything critical has a second channel, and that the prompt appears only after the user has seen a reason to say yes.
Almost Nobody Tests the Update Path. It Is Where Real Users Live.
A clean install on a wiped device is the one installation experience almost none of your users will have. Everyone else is upgrading from the version they already had, with their data, their session and their settings still in place.
- Database migrations, run against a database that genuinely has old rows in it
- Stored credentials and tokens written by the previous version, in the old format
- Caches and downloaded files the new build may no longer understand
- Onboarding and permission prompts that must not reappear for an existing user
- Deep links and notification payloads still being sent in an older shape
- The version two releases back, because plenty of people update rarely
That last one matters more than it sounds. Not everyone has automatic updates switched on, and a migration chain only ever tested one step at a time will break for the person returning after six months away.
Run the App With the Screen Switched Off
Turning on VoiceOver or TalkBack and completing your main journey without looking at the screen finds more real problems in twenty minutes than any checklist.
Everything is announced
Icon-only buttons need labels, images that carry meaning need descriptions, and decorative ones should be skipped entirely rather than read out as a file name.
Focus order makes sense
Reading order should follow the visual order, modal sheets should trap focus, and closing one should return the cursor to where it came from.
Text scales
The largest system text size is a real setting that plenty of people use daily. Layouts have to reflow rather than clip, and buttons must not lose their labels.
Targets and contrast
Controls large enough to hit on a moving bus, and status never signalled by colour alone. Both are quick during build and awkward to retrofit.
Accessibility work tends to improve the app for everyone — labelled controls and reflowing layouts are also what make an app usable one-handed in bright sunlight.
Crash Reporting, and What a Crash-Free Rate Does Not Tell You
Crash reporting goes in with the first beta build, not after the first bad review. It is the only way to learn about failures on handsets you will never hold.
Readable stack traces
A crash report is useless unless the build symbols went up with the release, so a native crash resolves to a real file and line rather than hexadecimal. That upload belongs in the release process, not in somebody's memory.
Breadcrumbs before the crash
The stack tells you where it died. The trail of screens, network calls and user actions leading up to it tells you why, and that is usually what makes a one-in-a-thousand crash reproducible at all.
Freezes and not-responding
On Android, an app that stops responding is recorded separately from one that crashes, and both count towards your store standing. A frozen app feels worse to the user than a clean crash, yet it never shows up in a crash-free number.
Sessions or users
Crash-free sessions and crash-free users are different measurements and the second is always harsher. Agree which one you are reporting before anybody sets a threshold on it.
A gate on rollout
The number earns its keep during a staged release: watch it on the first slice of users, halt if it moves the wrong way, widen only when it holds. See launch and submission for how that sequence runs.
What it hides
Failed logins, sync that silently does nothing, a payment that never confirms — none of those are crashes. A high crash-free rate is a floor to clear, not evidence that the app works.
Real Testers, Real Phones, Before the Store Sees It
Internal QA finds the defects. A beta finds the misunderstandings — the screen nobody could interpret, the step everyone skipped, the wording that meant something different to your customers than it did to you.
TestFlight, on iOS
- Internal testers on your developer account receive builds almost immediately
- External groups are invited by email or a shareable link, after a lighter beta review
- Separate groups let you send a risky build to five people and the safe one to fifty
- Testers send feedback and a screenshot from inside TestFlight, tied to the build
- Builds expire after a set period, so stale versions stop circulating on their own
Closed tracks, on Google Play
- Internal, closed and open tracks, each with its own tester list and its own build
- The same store listing and install flow your public users will get, which is worth rehearsing
- Play runs newly uploaded builds on its own device farm and reports crashes and warnings back
- New developer accounts may have to complete a closed test before production access is granted, which is a schedule item rather than a formality
- Testers update through the Play Store, so the upgrade path gets exercised as well
Platform programmes and their rules change. We confirm the current requirements at kickoff rather than assuming last year still applies — particularly the Play testing requirement, which has moved more than once.
When Heavier QA Is Not Worth Your Money
We would rather say this now than bill for it later.
Where Testing Sits in the Rest of the Work
iOS App Development
Swift and SwiftUI, and the iOS-specific behaviour a QA pass has to account for.
Learn moreAndroid App Development
Where device diversity, background limits and vendor behaviour come from in the first place.
Learn moreSupport Plans
Regression passes against each new OS release, so the app keeps working after launch.
Learn moreTesting Questions We Get Asked
The simulator looks identical to the phone. Why test on hardware at all?
Because the simulator borrows your laptop resources and none of the phone constraints. It has desktop-class memory, a fast wired network, no battery, no thermal ceiling, nothing else competing for RAM, and no telephony stack to interrupt it. The bugs that matter most on mobile live in exactly those gaps: the animation that is smooth until the handset warms up, the upload that fails on a weak signal, the half-filled form that vanishes when Android reclaims memory. Simulators are genuinely useful for layout work and for running unit tests quickly, so we use them every day, but no build is called ready on the strength of one.
How do you decide which devices go in the matrix?
From who your users are, not from what is new. If you already have an app or a website, the device and OS breakdown in your analytics is the honest starting point. If you have neither, we work from the market you are selling into and the price bracket your customers buy in. The result is a written matrix with a must-pass tier that blocks a release and a best-effort tier that does not, so testing is a defined amount of work rather than an open promise. A flagship bought last month is the least informative device in the room, because it hides every performance problem you have.
What does a crash-free rate actually measure?
The share of sessions, or of users, that finished without the app crashing. Those are two different numbers and it is worth knowing which one a dashboard is showing you, because crash-free users is always the harsher figure. Teams generally treat a sharp fall in either as a reason to pause a rollout rather than to investigate calmly. It is a floor and not a goal: an app can be almost perfectly crash-free and still be unusable, because freezes, failed logins and silent sync failures never register as crashes at all. On Android we watch the not-responding rate alongside it.
Do you automate the tests?
Where automation pays for itself. Unit tests around business logic, snapshot tests on the components that change often, and a small suite of end-to-end journeys such as sign-up, the main task and purchase, all run on every build. We do not try to automate the whole app. Maintaining a large brittle interface suite for a six-screen product costs more than it saves, and the failures that hurt most on mobile, such as an interruption handled badly or a layout that collapses at the largest text size, are still found faster by a person holding the phone.
How do our own testers get the app before it is public?
Through TestFlight on iOS and a closed track on Google Play. Your team, your stakeholders and a handful of real customers install the build the way they would install any app, on their own phones and their own networks. Feedback comes back attached to a specific build number rather than as a screenshot in a group chat, and crash reports arrive automatically. We start sending builds out this way early in the project rather than once at the end, because that feedback is only worth having while there is still time to act on it.
Is accessibility testing part of the work or an extra?
Part of it. Every release candidate gets a screen reader run through the core journeys with VoiceOver and TalkBack, a pass at the largest system text size, a check that colour is never the only thing carrying meaning, and a check that tap targets are large enough to hit reliably while walking. Most of what this finds is cheap to fix while a screen is being built and expensive to retrofit afterwards: an unlabelled icon button, a focus order that jumps around the screen, a custom control that the screen reader announces as nothing at all.
Get Your App Tested on the Phones That Matter
A new build, or an app already live that keeps collecting one-star reviews about crashes. Either way, tell us what it does and who uses it, and you get a device matrix and a plan back.