REVIEW

A Review of Nonprobability Sampling Using Mobile Apps for Fishing

Effort and Catch Surveys

J. Michael Brick*

Westat, 1600 Research Boulevard, Rockville, Maryland 20850, USA

William R. Andrews and John Foster

National Oceanic and Atmospheric Administration, Fisheries, 1315 East-West Highway, Silver Spring, Maryland 20910,

USA

Abstract

ples have major advantages compared with nonprobability samples. We conclude that the application of nonprobabil-

ity sampling for fishing surveys faces serious challenges and should prove that it is fit for use before being adopted

more widely.

During th e last few decades, a number of developments

have greatly affected how we think about the ability of

data that are collected by different means to inform deci-

sion making and policy development across a wide range

of fields. Technological advancements have reshaped our

ideas of data and data sharing. “Big data” have become

commonplace, and methods for processing massive data

tive data that are collected by government agencies and

private-sector organizations as a source of data for ana-

lysts and researchers. The Foundations for Evidence-

Based Policymaking Act of 2018 requires the federal gov-

ernment to rethink its approach to data acquisition for

making policy decisions and to consider alternative data

sources. For example, in its review of the National Oce-

anic and Atmospheric Administration’s fishing effort and

catch surveys, the National Research Council (2006)

recommended the development of administrative records

data and statistics. For fisheries policy, this often involves

designing probability-based catch and effort surveys and

making inferences from the respondents. For example, the

National Oceanic and Atmospheric Administration

designed and implemented the Marine Recreational Infor-

mation Program (MRIP) using prob ability sampling so

The advancement of novel data sources has disrupted

this paradigm. A few examples illustrate the nature of the

novel data sources. One of the most discussed early appli-

cations for big data was the use of Google search data to

track flu prevalence in the United States (Pervaiz et al.

2012). The data were not “designed” specifically to study

flu prevalence, leading some to refer to this type of data

source as “organic.” Harvesting data from an existing

source to answer a policy question can result in substantial

savings in terms of cost and time compared with a more

(2015) give another example of using organic data. They

discuss the capture of mobile data from cell towers to

evaluate travel demand and infrastructure needs. Shi and

Abdel-Aty (2015) use a different novel data collection

technology that is not organic but is “passive” in the sense

that it does not require any efforts from respondents. They

discuss using data from microwave radar sensors to study

traffic safety and congestion. All of these examples collect

massive amounts of data by using modern data-processing

methods.

their web activity, are asked to complete a web survey.

In these nonprobability samples, the data items are

designed specifically to support a policy question and

the data are collected through an advanced technology

such as a mobile app or web survey. The main advan-

tage of these designs is that data collection can be done

quickly and more affordably than with probability sam-

ples. However, because the sampling is not controlled,

inferences from the data must rely completely on mathe-

matical models and assumptions that cannot be verified.

lection modes that are similar to traditional survey modes

such as face-to-face interviews, mail questionnaires, and

telephone interviews. While mobile apps have advantages

over hard copy surveys, they are not the primary distin-

guishing feature of the new era of data collection.

Our focus is on the method of deciding who should be

surveyed, not on the mode of data collection. The new

development we examine is the move away from probabil-

ity samples to samples of volunteers or persons who are

recruited without randomizing the selection process, a

collection and analysis methods for nonprobability sam-

ples such as quota sampling, matched sampling, and alter-

native statistical methods. The role of sampling in making

inferences has been debated in statistics (Stephan and

McCarthy 1958; Royall 1970), social science (Coleman

1958; Callegaro et al. 2014), and in official statistics (U.S.

Office of Management and Budget 2006). Our review con-

centrates on the relatively new and specific interest in

using nonprobability samples to estimate recreational, salt-

water fishing effort and catch. Specifically, we focus on

developed, and hybrid designs that combine probability

and nonprobability samples. The nature of the potential

biases arising from both probability and nonprobability

sampling methods are examined in the following section.

The fourth section examines the potential for using non-

probability samples by using mobile apps for addressing

data needs that supplement those from probability sam-

ples. The final section is a discussion and summary of our

view of the issues that face the application of nonprobabil-

ity sampling for fishing surveys.

desired estimate—total catch for each species. The Fishing

Effort Survey selects a random sample of households and

asks household residents to report the number of saltwater

recreational trips taken during the past 2 months. The

APAIS intercepts anglers at the conclusion of fishing trips

and collects data on the type and number of fish caught

on the trip. Both the effort and catch surveys use proba-

bility sampling and base inferences on those probabilities

of selection (Papacostas and Foster 2018). The MRIP sur-

veys have been examined by independent reviewers several

discuss the utility of angler apps and the challenges associ-

ated with using them. They define angler apps as mobile

apps “that allow anglers to record, share and network”

their activities.

From a fisheries management perspective, it would be

ideal if everyone who particip ates in recreational fishing

used an app to record their catch and shared their data,

resulting in a census of recreational fishing activity. How-

ever, Venturelli et al. (2017) identify recruitment and reten-

tion as major bar riers to useful app data. For example,

Similarly, Venturelli et al. (2017) note that only 5% of

those who begin to use a specific app still used it after 3

months, and Ahrens (2013) found angler retention for

iAngler, an app focused on saltwater fishing in south Flor-

ida, was very poor, with the majority of users only using

the app once. This rate of “stickiness” or persistence with

using an app is not unusual. For example, Yang et al.

(2020) found that 74% of health app users stopped using

an app by the tenth use and 26% used it only once. To

increase the percentage of anglers who volunteer to use

Although the premise is that the apps will result in a

census of fishing activity, this is extremely unlikely for rec-

reational fishing apps. Because apps are intended to be

used by ang lers who volunteer to do so, instead of a cen-

sus they constitute a nonprobability sample with no selec-

tion probabilities attached to those who do report. One

consequence of this method of collecting data is that no

techniques for controlling the respondents such as quotas

or matched samples (Baker et al. 2013) are possible.

Another consequence is that estimation methods that do

Even this simple adjustment is not feasible for fishing sur-

veys because the total number of anglers that could serve

as the population total is not known from the U.S. Census

Bureau or administrative records. For example, counts of

licensed anglers would exclude exempted categories, such

as kids and anglers who fish from licensed piers, and other

anglers who choose to fish without a license.

Another approach that has been discussed in the litera-

ture is using some combination of probability and non-

probability samples to gain the advantages of the lower

approach to use capture–recapture sampling methods for

fishing surveys. The capture

not have good solutions to the independence assumption

violation. As intercept survey coverage is generally limited

to publicly accessible fishing access sites, noncoverage

error is another potential source of bias in this hybrid

design. The extent to which trips returning to private

access sites differ from those returning to public sites in

terms of app reporting rates and catch characteristics, as

well as the proportion of total trips returning to private

sites, determine the potential for bias from th is noncover-

age error.

sites from intercept surveys is a source of noncoverage

error that could affect estimates from any program th at

incorporates that sampling design. Stokes et al. (2021)

speculated that nonresponse bias in the probability effort

survey was the greatest threat to the validity of its esti-

mates. In the next section, the potential biasing effects of

nonresponse are discussed for both probability and non-

probability samples.

suggesting that the potential nonresponse bias was not

large.

The mobile apps, on the other hand, have no capacity

to estimate the number of nonparticipants in the ratio in

equation (1), and consequently no capacity to estimate

fishing prevalence. The apps are designed for anglers to

record their activities while fishing; none of the suggestions

of Venturelli et al. (2017) for increasing the use of the apps

are intended for those who do not fish.

With this type of nonprobability sample, the estimator

The potential for bias becomes much greater with this

estimator compared with equation (1) for two reasons.

First, the estimator can no longer take advantage of the

auxiliary data, T, to reduce bias. Second, unless all peo-

ple who fishe d in the period use the app for every trip

(i.e., a census) the estimate will be biased downward.

Given the experiences of the use of apps that is described

in the previous section, th e bias will be large and consis-

makes it dif ficult to improve on the accuracy of equa-

tion (2). As mentioned earlier, administrative data on reg-

istered anglers are incomplete due to unlicensed fishing

activity, and even if it were complete the data would not

provide counts of the number of anglers who took trips

during the specified period.

offsetting. When this holds, the bias for the estimated

mean is reduced compared with separate estimates of the

numerator and denominator. Some researchers hav e con-

sidered using nonprobability samples for estimating

means, proportions, and other relationships because of

this observation. Notice that unlike effort estimates, the

catch rate estimates that are derived from mobile apps do

not require any data from those who do not fish during

the period.

While the bias for estimating means is generally lower

, (3)

where R is the percentage of the sample responding,

Y

is

the mean of the respondents, and

Y

is the mean of those

due to excluding private fishing access sites could be more

problematic.

A similar expression holds for nonprobability estimates

of a mean

We begin by examining the components of equation (3)

and equation (4). The quantity P refers to the entire popula-

tion, whereas R refers to the sample. In a probability sample

where equation (3) applies, data collection efforts are con-

centrated on getting the sampled units to respond by mak-

ing multiple conta ct attempts and using ot her tools like

monetary incentives. In a nonprobability sample where

equation (4) applies, any efforts to increase the first term,

making it closer to equation (1), requires doing the same

type of work for the whole population of anglers—encour-

and nonusers. The requirement is that catch rates are

independent of using the app.

The relative sizes of

Y

Y

depend on the specific application. For a catch survey, we

suspect that typically

Y

. This would hold if the likelihood of

responding and catch rate were not as highly correlated as

the likelihood of using the app and the catch rate. While

we suspect that this is th e case, the main contributor to

the bias mostly likely arises due to the differences between

R and P.

Throughout this discussion, we have described the bias

due to differences between the respondents and the full

population in terms of nonresp onse bias. It is more typical

to classify this bias in nonprobability samples as selection

probability samples. We used the nonresponse formulation

to be able to use the same structure for both types of sam-

ples. Meng (2018) and Kalton (2019) also discuss som e of

the issues with nonprobability sampling, but they primar-

ily show that the effective sample size in nonprobability

samples can be dramatically lower than the nominal sam-

ple size.

Even without resolving the magnitudes of the second

terms in the biases, the differences between R and P

would almost certainly result in considerably smaller

Papenfuss et al. (2015) proposed that apps could be

used to provide fine-scale movement data that are not

available from traditional surveys as well as a platform

for on-demand angler surveys and for quickly launching

projects to collect data passively. Venturelli et al. (2017)

discussed apps as a source of information on topics such

as bait and tackle, depth, fish kills, invasive species, and

water conditions. Jiorle et al. (2016) considered apps for

collecting data on discarded fish characteristics, greater

spatial resolution, and for low-effort or rarely encountered

the number, species, and length of the discarded fish. The

bias from the data that are collected in this manner could

be computed using equation (4) and speculating about dif-

ferent values for the quantities in that equation. Suppose

that 20% of anglers used the app (P = 0.2) and the mean

number of discards for those using the app is 1.5 and for

those not using the app is 1.2. The true mean number of

discards is 1.275, but the estimate based on just app users

is 1.5 (bias = 0.225, or 18% of the mean). The question is

whether an estimate with this magnitude of bias would be

in the spatial distributions (ranges) of species and the

potential link to climate change. Similarly, small-scale

localized studies might be mounted. These examples of

supplementary uses differ from the discard example given

above because population-level inferences are not the goal.

Rather, these types of supplementary uses might generate

hypotheses about processes that are worthy of further

investigation. Apps could also provide longitudinal data

for anglers over time. The main concern with this use is

the high attrition rate for those who do start reporting on

DISCUSSION

Fishing effort and catch surveys pose many challenges

irrespective of the sampling. However, the sampling

method has major implications. With probability samples,

the costs of data collection are generally considerably

higher than when nonprobability samples are used, and

the probability samples must use methods to ensure rela-

tively high-quality data if the advantages of probability

sampling are to be realized. By high quality, we mean

keeping both sampling errors and the nonsampling errors

rate so that the rate for those who do not fish is similar to

the rate for those who do fish. In catch surveys, techniques

can be instituted to sample and interview a sample that is

balanced with respect to catch and other important trip

characteristics to reduce bias.

Nonprobability fishing surveys face substan tial chal-

lenges if th ey are going to produce high-quality estimates

for statistical inferences. The challenges exist for both

effort and catch surveys, but finding ways to encourage

those who do not fish on a regular basis to use a fishing

If probability surveys for fishing effort are to be replaced,

methods other than using apps are needed.

effort, an app could be considered for estimating catch

rates because catch rate estimates only require input

ing errors and the conditional independence assumption

still need solutions. The role of noncoverage bias due to

excluding private fishing access sites in the probability

samples also requires some investigation for this

application.

Some data, especially for those who are without the

resources that are needed to collect high-quality data from

probability samples, seems better than no data at all. Per-

haps there are situations in which this holds, such as some

of the supple mentary uses where population inferences are

be open. The opposite result of allowing fishing when it

should be restricted could have even more grim conse-

quences. When the policy has such consequences, it is

important to have estimates and confidence intervals for

those estimates that can be trusted. Currently, the bes t

general approach to providing such estimates is by using

probability samples.

Our view is that nonprobability samples should first be

studied and evaluated in situations where the effects of

wrong decisions are not serious. Baker et al. (2013) refer

nonprobability samples and compare them with each other

or with benchmark estimates from some gold standard.

Callegaro et al. (2014) do a comprehensive review of such

studies. Yeager et al. (2011) experimented with using the

same instrument in multiple nonprobability samples and

in a probability sample. By highlighting issues that would

otherwise go undetec ted, such comparisons have value

even if they do not generalize directly to other applica-

tions. The existing studies have shown that the probability

samples typically produce estimates with smaller biases

were not the equivalent of those done in the social science

research noted above. They made only a few comparisons

and considered only a few species that have very limited

variation in catch rates and in very limited geographic

areas. Furthermore, the comparisons were to observations

from a probability sample not estimates from the sample.

Thus, this method of evaluation has not been explored

much for fishing surveys. Without robust evaluation of

nonprobability samples for fishing surveys, there is no evi-

dence that they are fit for use in providing population-

´

´