Skip to content

34. Observability

The question. How does the team know what this app is doing on your device?

A backend runs on machines its team controls. When a service misbehaves, an engineer reads its logs, attaches a debugger or restarts it. A client runs on a device the team will never touch. The code executes on thousands of hardware models, under an operating system that suspends and ends it at will, on networks nobody can reproduce. Whatever the team learns about that execution, the client has to record and send back itself.

Four of the seven constraints (Chapter 2) shape how it does so.

Slow release is the reason the concern exists at all. A shipped version cannot be recalled, so a defect stays in the field until a new version passes review and users install it. The team needs to learn about the defect within hours, from the devices that hit it, and it needs enough detail to fix it without a second round of questions.

Process death and the unreliable network shape the mechanics. A record of what happened must survive the process that produced it, and it must wait on the device until a network is available. Observability is therefore a small offline system of its own, with a durable queue and an uploader, built on the ideas of Chapter 12.

Limited resources set the ceiling. Every record costs storage while it waits, and data and battery when it is sent. A client that reports everything spends the user’s resources on the team’s curiosity.

Platform rules add a gate. The operating system and the store require a maker to declare what an app collects, and the law in many regions requires the user’s agreement before some of it is collected. Chapter 22, Privacy, permissions and consent, covers that gate. This chapter covers what sits behind it.

This concern is the hardest in the book to see from outside. Observability is designed to be invisible to the user, and a well-built pipeline produces no screen, no delay and no error. Almost every claim you make from this chapter is Speculative. The chapter says so wherever it applies, and Section 34.10 lists what you can and cannot establish.

Observability is a set of independent channels. An app can have any combination of them. They are listed here from the least data to the most.

Channel 0. Nothing from the client. The team sees only what reaches the backend as ordinary requests. It knows which endpoints are called and how often they fail. It does not know about anything that happens without a request: a crash, a slow screen, a button nobody finds. Very small apps and apps with a strict privacy position work this way.

Channel 1. Reports the platform collects. The operating system records crashes and some performance figures for every app, and passes them to the maker when the user has agreed to share diagnostics at the device level. The app contains no code for this. The data is coarse and arrives late, and it costs the team nothing.

Channel 2. Crash reports. The client contains a component that records the state of the process when it fails and uploads that record. This is the first channel most teams add, because a crash is the worst outcome and the report is small.

Channel 3. Analytics events. The client records named events, such as a screen shown, a control tapped or a purchase completed, with a few properties each. Events answer product questions: which features are used, where users give up in a flow, whether an experiment changed behavior (Chapter 30, Remote config and experiments).

Channel 4. Performance measurements. The client times its own work on real devices: how long a cold start took, how long a screen took to show first content, how long each request took, how many frames were late during a scroll. The team reads these as distributions across all devices, not as single values.

Channel 5. Logs. The client keeps a running text record of what its components did. Logs are verbose and are rarely uploaded continuously. They are kept on the device and sent when something goes wrong or when the user sends feedback.

Channel 6. Session recordings. The client records what the user saw and did, in enough detail to replay it. This is the richest channel and the one with the most privacy risk. Teams use it for a small sample of sessions, or for specific flows where users fail and the events do not explain why.

ChannelWhat it recordsVolume per userUploadedPrivacy risk
0 NoneNothing beyond requestsNoneNeverLowest
1 Platform reportsCrashes and coarse performanceTinyBy the operating systemLow
2 Crash reportsProcess state at failureSmall, and only on failureOn the next launchLow to medium
3 Analytics eventsNamed actions with propertiesMediumIn batchesMedium
4 Performance measurementsDurations and countsMediumIn batches, often sampledLow
5 LogsText record of internal stepsLargeOn demandMedium to high
6 Session recordingsScreens and inputLargestSampledHighest

The channels share machinery. Each one records something, holds it on the device, and uploads it later. Section 34.3 describes that shared machinery, and Sections 34.4 to 34.7 describe what is particular to each channel.

Figure 34.1 shows the components. Few apps build all of them. Many use a third party for the whole pipeline, in which case the recorder and uploader arrive as a library and the receiving side belongs to someone else.

Figure 34.1. The components of a client observability pipeline.

Three properties of the figure matter for a teardown.

First, nothing in it touches the screen. The UI hands an event to the pipeline and continues. No screen waits on an upload, so no loading state reveals one.

Second, the pipeline is separate from the app’s own data path. Its queue is not the outbox of Chapter 12, and its uploads do not go through the sync engine. A failed upload never produces an error the user sees. The separation is deliberate: a fault in observability must not break the product.

Third, the pipeline has two destinations in the figure and can have more. Events may go to the team’s own backend, to a third party, or to both. From outside you cannot see which. You can only read what the maker declares, which Section 34.9 covers.

A process that has just failed cannot be trusted to open a connection, wait for a response and retry. It may have corrupted its own memory, and the operating system is about to remove it. A crash handler therefore does the least it can: it writes the state of the failing process to a file, using only operations that are safe in a broken process, and lets the process end.

The upload happens later, on the next launch. A startup step checks for a crash file, sends it, and deletes it once the backend confirms receipt.

Figure 34.2. A crash report is written at failure and uploaded on the next launch.

This timing has two consequences you can reason about. A user who crashes and never opens the app again never sends the report, so the worst crashes are under-counted. And a crash during startup is the hardest to report, because the next launch may crash before the upload runs. Careful designs upload the pending report before anything else at launch, or detect repeated startup failures and enter a reduced mode. Chapter 7 describes that startup sequence.

A report holds the call stack of each thread at the moment of failure, the app version, the operating system version, the device model, free memory and storage, and whether the app was in the foreground. The stack is a list of addresses. The backend holds a mapping file for each released build and translates addresses into function names. That translation is why the build number matters, and why a team keeps mapping files for every version still in use.

Most designs add context the stack lacks: a short list of recent events, called a trail in this chapter, such as the last screens visited and the last requests made. Some attach a user identifier so that support can connect a complaint to a report.

Three kinds of failure never run the crash handler.

  • A hang. The UI stops responding and the user force-quits, or the operating system ends the process for being unresponsive. A watchdog inside the client can notice that the main thread has been blocked for several seconds and record its stack.
  • Termination by the operating system. The process is ended for using too much memory or for overrunning a background time limit. No code runs at that moment. A client can detect this only by exclusion: at the next launch it notes that the previous run did not end cleanly, did not crash, and was not updated or restarted by the user.
  • A handled error. The code caught a failure and showed an error state. Nothing crashed, and the user was still let down. Teams record these as events, not as crash reports.

Most designs upload crash reports without asking, under whatever consent the user gave earlier. A few ask on the next launch whether to send the report. That prompt is one of the only direct views of this channel a user gets. If a crash happens to you naturally, the launch that follows is worth watching, and P34.8 records it. Do not try to cause a crash. A crash can lose your data, and a deliberate one tells you nothing a natural one would not.

An event is a name, a timestamp and a small set of properties. The recorder adds context that every event shares: an identifier for the installation or the user, an identifier for the current run of the app, the app version, the device class, and the experiment groups the user is in.

Two timestamps are needed. The device clock says when the event happened and can be wrong. The backend clock says when the batch arrived and says nothing about the delay before it. A sound design sends both the event time and the device’s time at upload, so the backend can compute how long ago each event happened by subtraction and place it on its own clock.

Sending each event as its own request would wake the radio hundreds of times in a session (Chapter 35, Performance and resource budgets). The recorder instead appends events to a durable queue, and an uploader sends them with batching.

function record(event):
if not consent.allows(event.category): return
if not sampler.keeps(event): return
queue.append(event) // on disk, survives process death
if queue.size >= batchSize: uploader.requestRun()
function upload():
batch = queue.peek(batchSize)
match send(batch, batchId): // batchId lets the backend drop repeats
Accepted:
queue.remove(batch)
Unreachable:
scheduleRetryWithBackoff()
Refused:
queue.remove(batch) // a bad batch must not block the rest

The uploader runs on a few triggers: the queue reaching a size, a timer, the app moving to the background, and the next launch. The move to the background matters most. It is the last moment the client is sure to run, so most designs try to send whatever is queued then, within the short time the operating system allows.

Events recorded offline wait in the queue. Unlike an outbox, this queue is allowed to lose data. It has a bound on size or age, and when the bound is reached the oldest or least important events are dropped. A team would sooner lose a day of events than fill the user’s storage.

When the network returns, the uploader drains the queue, often through a background task the operating system schedules (Chapter 23, Background work and notifications). This is the one place where the channel can leave a faint trace: a device that was used offline and then reconnected may show data sent by the app while the app was closed. P34.7 looks for that trace. An outbox draining looks the same, so the probe works only on a session in which you wrote nothing.

Analytics delivery is at-least-once with deduplication on the backend, or simply at-most-once. A batch whose response is lost is sent again, and the backend drops it by its batch identifier, which is the idempotency key of this pipeline. Order across batches is not guaranteed, and the backend sorts by event time. Nothing here is visible from outside.

Performance data from real devices answers a question the team’s own test devices cannot: how the app behaves on the slowest phones and the worst networks its users have. The client measures a small set of durations and counts.

MeasurementWhat it times or counts
StartupProcess start to first frame, and to first content
Screen loadNavigation to a screen until its content is drawn
RequestEach request’s duration, size and result, by endpoint
FramesHow many frames missed their deadline during scrolling or a transition
HangsPeriods in which the UI did not respond
ResourcesMemory in use, and terminations by the operating system

A single slow startup means nothing. The team reads percentiles across devices, split by app version, device class and region. The client therefore sends plain numbers with the same context as an event, usually through the same queue.

A finer tool is a trace: a record of the steps inside one operation with the time each took, such as the phases of a cold start. Traces are large and are collected for a small fraction of runs.

Measuring costs something. Timing every frame and every request adds work to the code paths that are already under pressure. That is one reason this channel is sampled, and Section 34.8 covers sampling.

A session recording lets a member of the team replay what a user saw. There are two designs.

  • Captured frames. The client takes images of the screen at intervals and uploads them as video or as a series of stills. This is simple and heavy. It costs processing time and a large upload, and it captures everything on screen unless areas are blanked out first.
  • Reconstruction. The client records the structure of each screen, as a tree of views with their positions and text, plus the input events. A viewer on the backend redraws the session from that record. This is far smaller and makes masking precise, because the recorder knows which element is a text field.

Both designs need masking: replacing sensitive content, such as typed text, payment details and messages, before anything leaves the device. Masking by default, with an explicit list of what may be shown, is the safe arrangement. Masking only what someone remembered to mark is the arrangement that leaks.

A recording is the channel a user is least likely to expect. You cannot observe it directly. The evidence available is the maker’s own statements in its privacy listing and policy, and occasionally a side effect: a device that grows warm or stutters in an app that draws nothing demanding, or unusually high data sent for the amount of use. Both have many other explanations. Any claim that an app records sessions is Speculative unless the maker states it.

Logs are kept in a ring: a fixed-size buffer in which new lines replace the oldest. The ring holds the last few minutes or megabytes of activity and costs a bounded amount of storage.

A feedback or problem-report form is where the ring surfaces. When you submit the form, the client can attach the ring, the device model and versions, the state of key settings, and a screenshot. Good forms say so and let you choose. This makes the feedback form the best window a user has onto this concern. The wording of the form tells you that logs exist, that they are held on the device, and whether the team treats their upload as something to ask about. P34.4 reads the form.

Some apps open the report form when you take a screenshot or shake the device. That trigger shows the form is wired into the app as a whole and not into one settings screen.

Collecting everything from every device is expensive at both ends, so most pipelines sample.

KindRuleUsed for
By userA fixed fraction of users report a channel in fullPerformance measurements, recordings
By sessionEach run of the app decides once whether it reportsTraces, recordings
By eventEach event is kept with some probabilityHigh-volume events such as scroll positions
By outcomeRecord everything locally, upload only if the run ended badlyLogs, trails before a crash

Sampling by user keeps each chosen user’s history complete, which matters when the question is about a sequence of actions. Sampling by event keeps totals accurate when scaled and breaks sequences. Sampling by outcome is the most economical: the ring and the trail cost nothing to upload unless a failure makes them worth reading.

Rates are set by remote config, not compiled into the app. A team raises the rate for a new version in its first days, lowers it once the version is stable, and can turn a channel off entirely for a version whose recorder is itself faulty. The same mechanism lets a team raise the detail collected from one account while it investigates a complaint, which is why a support agent sometimes asks you to reproduce a problem after they “enable diagnostics”.

Sampling is invisible. Two users of the same app may be reporting very different amounts, and neither can tell.

Observability collects data about a person’s behavior, so it sits behind the consent rules that Chapter 22 covers. Three design points belong to this chapter.

The gate comes first. In Figure 34.1 the consent gate sits before the recorder. An event the user has not agreed to is never recorded. The alternative is to record everything and filter at upload, which keeps data on the device that the user declined to give and risks sending it through a mistake.

Events before the choice. On first launch the user has not chosen yet. A design can drop everything until the choice is made, or hold early events in memory and either queue or discard them once the choice is known. The first is simpler and loses the data on first-launch behavior that product teams most want.

Categories. A consent screen usually separates what is needed for the app to work from what is optional. Crash reports are often placed in the first group and analytics in the second. The categories you are shown are a statement by the maker of which channels exist. So is the store’s privacy listing, which declares the kinds of data the app collects and whether they are linked to your identity.

These statements are the strongest evidence available for this chapter, and they need careful handling. They are published by the maker, and a statement is not an observation of behavior. Record them as what they are: Observed that the maker declares a category, and Speculative as to what the app sends. A listing that declares more than the app uses is as possible as one that declares less.

A setting that turns usage data off also tells you something about structure. If the change takes effect at once, the gate is consulted for each event. If the app says the change applies after a restart, the recorder reads the choice once at startup.

QuestionBest evidence availableConfidence you can reach
Does the maker declare crash or usage data collectionStore privacy listing, consent screenObserved, as a declaration
Is there an in-app controlSettings screenObserved
Do logs exist on the deviceFeedback form wordingInferred
Does the app upload while idleData figures over an idle dayObserved that data moved. Speculative as to what
Are events queued offlineData sent after reconnectingSpeculative
Are crash reports sentA prompt after a natural crashObserved if prompted. Otherwise Speculative
Are sessions recordedThe maker’s statement onlySpeculative without a statement
Which third parties receive dataThe maker’s policyObserved, as a declaration
What the sampling rates areNothingCannot be determined

Two designs are indistinguishable from outside throughout this chapter: a pipeline the team built and one supplied by a third party. Both queue, batch and upload in the same way.

Every probe in this chapter is passive. Run P34.1 to P34.5 in one sitting, since they only read screens. P34.6 and P34.7 need a day. P34.8 cannot be scheduled.

Open worksheet
Do
  1. Open the app's settings and look through every section, including privacy, data and about.
  2. Look for a control that mentions diagnostics, crash reports, usage data, analytics or improving the app.
  3. Read its description. Do not change it yet.
Watch

Whether a control exists, what it says is collected, and whether crash reports and usage data have separate controls.

Read

Select what you saw to record it in your worksheet. Select it again to clear it.

Do
  1. Open the app's page in the store.
  2. Find the privacy section the maker has filled in.
  3. Note whether it lists diagnostics, crash data, performance data or usage data, and whether each is linked to your identity.
Watch

Which kinds of data are declared and for which purposes, such as analytics or app functionality.

Read

Select what you saw to record it in your worksheet. Select it again to clear it.

Do
  1. Look in settings for a privacy choices or consent screen that can be reopened.
  2. If there is none, recall or note what the app asked on its first launch on your device.
  3. Read the categories offered and which are marked as required.
Watch

Whether analytics is a category you can refuse, and which category crash reports are placed in.

Read

Select what you saw to record it in your worksheet. Select it again to clear it.

Do
  1. Find the app's feedback, help or report-a-problem form.
  2. Read the whole form before typing anything, including small print near the send control.
  3. Look for a mention of logs, diagnostics, device details or a screenshot, and for a control to leave them out.
  4. Send a short, real piece of feedback only if you have one. You do not need to send anything to record the outcome.
Watch

What the form says it attaches, and whether you are given a choice.

Read

Select what you saw to record it in your worksheet. Select it again to clear it.

Do
  1. Open any ordinary screen of the app that shows nothing private.
  2. Take a screenshot with the device's own controls.
  3. Wait a few seconds and look for a prompt from the app.
Watch

Whether the app offers to report a problem or share the screenshot.

Read

Select what you saw to record it in your worksheet. Select it again to clear it.

Do
  1. Use the app normally, then close it.
  2. Open the device's data usage screen and note the figure for the app. Note the background figure separately if the device shows one.
  3. Leave the device connected and do not open the app for a full day.
  4. Read the figures again.
Watch

How much data the app used while you did not open it.

Read

Select what you saw to record it in your worksheet. Select it again to clear it.

Do
  1. Pick a feature that works offline for reading (Chapter 12). Skip this probe if the app shows nothing offline.
  2. Note the app's figure on the device's data usage screen.
  3. Turn on airplane mode and browse for several minutes. Read only. Do not create, edit, like or send anything.
  4. Close the app, then turn airplane mode off. Do not open the app.
  5. After an hour, read the data figure again.
Watch

Whether the app used data after reconnecting although you did not open it and had nothing of your own waiting to send.

Caution. This probe uses airplane mode, which interrupts calls and other apps on that device.

Read

Select what you saw to record it in your worksheet. Select it again to clear it.

Do
  1. When the app closes unexpectedly during normal use, note what you were doing.
  2. Open the app again.
  3. Watch the first few seconds and the first screen.
Watch

Whether the app asks to send a report, shows a message about the previous run, or returns you to where you were.

Caution. Never try to cause a crash. Run this probe only when the app has closed unexpectedly on its own.

Read

Select what you saw to record it in your worksheet. Select it again to clear it.

SignalDesign it points toConfidence
Settings offer a control for usage data or diagnosticsA consent gate in front of at least one client channelObserved
Separate controls for crash reports and for usage dataCrash reporting and analytics are separate channels with separate consentInferred
The store privacy listing declares diagnostics or usage dataThe maker states that the channel exists. A declaration, not an observation of trafficObserved
A usage-data setting says it applies after a restartThe recorder reads consent once at startupInferred
The feedback form says it attaches logsA bounded log kept on the device and uploaded on demandInferred
The feedback form lists device model and versionsContext is collected at the moment of the reportObserved
A report prompt appears after a screenshot or a shakeA report trigger installed at app levelObserved
A prompt to send a report on the launch after a crashCrash file written at failure and uploaded on the next launchObserved
An offer to restore your place after an unexpected closeThe client detects an unclean end to the previous runInferred
A small amount of data used on a day the app was not openedBackground upload, config check, sync or push handling. The channel cannot be identifiedSpeculative
Data sent after reconnecting from a read-only offline sessionA durable event queue drained in the backgroundSpeculative
Startup hangs on a poor network and is fast in airplane modeAn upload or config call on the launch path without a short timeoutSpeculative
Storage grows in an app that caches littleAn unbounded queue or logSpeculative
A support agent asks you to reproduce a problem after enabling diagnosticsPer-user detail raised by remote configInferred
Warmth and stutter on simple screensPossibly a recorder capturing frames. Many other causes fitSpeculative
The privacy policy names analytics or crash providersThird parties receive the data. Stated by the makerObserved

Search the signals from every chapter in the signal lookup.

Each channel needs a receiving side. Because the client evidence is weak, treat these as what must exist if the channel does.

  • Crash reports. A collection endpoint that accepts reports from every version ever shipped. A store of mapping files per build, and a step that turns addresses into names. Grouping, so that ten thousand reports of one defect appear as one issue with a count. Alerts when a new version’s crash rate rises, tied to the release process (Chapter 33, Releases and compatibility).
  • Analytics events. A collection endpoint built to accept and never to block: it acknowledges a batch quickly and processes it later. Deduplication by batch identifier. A definition of every event name and property, so that old versions sending old shapes can still be read. A large store that is queried in bulk.
  • Performance measurements. Aggregation into percentiles by version, device class and region. Raw values are rarely kept for long.
  • Logs and feedback. Storage for attachments, linked to the support case that carried them, with a short retention period.
  • Session recordings. Storage for the recordings, a viewer, and for the reconstruction design, a renderer that can redraw screens from every app version.
  • Control. Remote config entries for sampling rates and channel switches, per version and per user.
  • Deletion. A way to remove one person’s data from every one of these stores on request. This is easy to state and expensive to build, and it is the reason many teams avoid attaching a user identifier where an installation identifier will do.

When a third party supplies the pipeline, all of this exists on that party’s side, and the team’s backend may contain none of it.

Observability fails quietly, and a user sees its failures only as side effects.

  • A slow or hanging startup that clears when the network is turned off. A startup step is waiting on a collection endpoint. The upload was placed on the launch path without a short timeout.
  • Storage that grows in an app that caches little. A queue or a log with no bound, or an uploader that never succeeds and never gives up.
  • Data used every day by an app you never open. Background work that uploads, checks config or syncs. The channel cannot be identified from the figure alone.
  • The app is warm and stutters on simple screens. One possible cause among many is a recorder capturing frames. Speculative.
  • A feedback form that fails to send on a weak connection. The attachment is large and the transfer is not resumable (Chapter 18, File transfer: uploads and downloads).
  • The same crash on every launch, with no fix for days. Either reports are not arriving because the app fails before the upload runs, or the team has no alert on a rising crash rate.
  • A prompt to send a report appears long after the crash. The report waited on the device until the next launch, as Figure 34.2 shows.
  • A usage-data setting that says it takes effect after a restart. The recorder reads consent once at startup instead of for each event.

These are the fields this chapter adds to the teardown worksheet. Record the maker’s statements and your own observations in separate fields, and keep the confidence word on each.

Open worksheet
  • Observability is how a team learns what its code does on devices it cannot reach. Slow release makes it necessary, and limited resources and consent bound it.
  • The channels are platform reports, crash reports, analytics events, performance measurements, logs and session recordings. An app can have any combination.
  • All channels share one shape: record, hold on the device in a bounded queue, upload later in batches. The pipeline is separate from the app’s data path and never blocks a screen.
  • A crash report is written at failure and uploaded on the next launch. Hangs and terminations by the operating system need separate detection.
  • Sampling and remote config decide how much each device reports. Neither is visible.
  • The feedback form, the settings screen, the consent screen and the store’s privacy listing are the only direct evidence. The last two are statements by the maker, not observations of behavior.
  • Most claims in this concern are Speculative. A built pipeline and a third-party pipeline cannot be told apart from outside.
  • Never cause a crash to study it. Watch the launch after one that happens naturally.