Welcome to my Thinking World…

Abstract:

Over the past few years, market researchers and clients working with data from non-probability online panels have voiced a number of data quality concerns over issues such as respondent identity, increased satisficing, and possible professionalization of survey taking. Recognizing the importance of having real, unique, and engaged panelists, panel companies are responding to these issues by introducing a variety of remedial measures such as name-address verification, email address verification, and validation of key demographic information against third-party databases. This study describes one such effort. In this study, for a cross-section of new panel members who were recruited online, we validated their identity (based on name and address) against the national databases of two data aggregators to identify match levels and patterns of matches and mismatches. We further explored the level of agreement between panel self-reported data and vendor files on selected demographic characteristics. We observed modest match and agreement rates that, importantly, differed between the two aggregators, with neither being superior to the other on all measures. This result suggests that such vendors do not provide a “gold standard” against which to verify panelist identities.  Multivariate analyses of the match and agreement rates suggested that older, more established panelists are more readily identified in the national databases. We conclude that, for the panel we studied, at least, negative match results can serve as a trigger to follow-ups with panelists, but not as a stand-alone reason to reject prospective panelists. This study is likely to have important implications for the data quality aspects of online panel recruitment, participation and retention.

Recommended Citation:

Rao, K., Kinnucan, M., & Dolson, T. (2011). Are you who you say you are? Using a Multisource Cross-validation Methodology for Panel Membership Information. Paper presented at the American Association for Public Opinion Research, Phoenix.

Attached Documents:

• AAPOR 2011 Program (see page #132 for the mention)
• For a copy of this paper, please send me a comment with your email address in the box below.