8 December 2025
Sampling bias in in-app surveys
People who tap “rate this app” after a failed payment are not a window on the whole population. They are a queue at a particular door.
Survey tiles look democratic. A score between one and five, a sample size in the thousands, a sparkline. Cross-Platform App Reporting has to ask who was eligible to see the prompt. iOS may suppress it under a review guideline. Android may show it after a crash recovery flow. Web may bury it behind a cookie banner. Those are three sampling frames wearing one metric name.
Publish the frame
Every CSAT chart in a Thursday pack should name the trigger: random session, post-purchase, post-crash, or support follow-up. If triggers differ by client, do not average them. Sibling scores again. Privacy-Aware Funnel Notes spends a seminar on wording that does not claim representativeness you do not have.
Small n is not a moral failing
Web often has fewer in-product prompts. A score based on 40 responses is not “too small to show”; it is small, and the pack can say so. Hiding it and quoting only mobile produces a company mood that web users never voted on.
If you must show a blended score for a board pack, weight by eligible sessions, not by responses. Response-weighted blends reward the client that nags hardest.