Tested levels of info on labels on IoT product boxes for data safety and privacy

Project Overview

Position: UX Researcher at Cylab, CMU
Timeline: Jan - Oct 2023
Advised by Professors: Lorrie Cranor and Yuvraj Agarwal
Status: Research paper submitted to CHI

Executive Summary

Our research project involved a comprehensive survey with 500+ participants, examining the impact of diverse privacy and security information on IoT product labels. We designed innovative labels, conducted multifaceted analyses, and our findings have influenced government understanding, sparking discussions and potential policy changes. Our work was referenced at the White House during the Cyber Trust Mark unveiling

Significance of Our Research

Background

Our research builds upon the existing body of knowledge on IoT labels in product packaging, which was initiated by Dr. Pardis Emami-Naeini. Her designs drew inspiration from nutrition labels. Our contribution extends this research by specifically investigating the influence of varying levels of information presented on such labels

Supplement Facts
Serving size 1 Tablet
Servings per container 60
Amount per serving
% daily value
Vitamin A
200 mg
**
Vitamin B
300 mg
**
Vitamin C
100 mg
**
Vitamin D
210 mg
50%
Vitamin E
110 mg
20%
Vitamin F
15 mcg
100%
Vitamin G
16 mg
107%
Vitamin H
15 mg
100%
Vitamin I
2,4 mcg
100%
Daily value not stablished

Methodology: Survey

We opted for a survey-based methodology as it allowed us to efficiently capture a wide spectrum of user perspectives. Surveys are a practical choice for gathering extensive insights from a diverse participant pool, aligning with our goal of comprehensively understanding user opinions regarding IoT product labels and their privacy and security information

Demographic

Segregating individuals into age groups of 18-35, 36-53, and 54+ who have made online or in-person purchases of IoT devices within the last three years

Decision-Making: Tools Selection and Design

Tools Selection Decision

The primary rationale for our utilization of Qualtrics was its capability to effortlessly segment respondents based on their answers to specific questions, as illustrated in the example below:

Scroll on this small prototype

Design Decisions

We conducted tests on four distinct label designs, each presenting varying levels and quantities of information. These label variations emerged from iterative testing during pilot phases to enhance their usability and user-friendliness

Research Goals: What We Investigated

Amount of information:

The study designed labels with different information levels, featuring a primary layer with a QR code and a secondary layer for more detailed IoT device info, all within packaging constraints for concise presentation.

Security and Privacy Attributes:

The labels had information about important things like how the device collects data, updates its security, and controls who can access it. People were asked to look at these labels on three different devices and see if they could tell which one was safer and more private.

Educational Intervention:

In the study, participants were given information about the U.S. Cyber Trust Mark and how to use QR codes on labels to access details, testing if this educational intervention improved their ability to understand and use privacy information.

Experiment

Prescreen Survey

We screened individuals who had purchased an IoT device, either online or in person, within the past three years

We review your designs in detail and provide you with a fixed-price-breakdown for each page and a timeline for the project.

Main Survey

We conducted a between-subjects study, categorizing the main survey participants into age groups, depending on whether they received an intervention or not. We then randomly assigned them to respond to questions using one of three labels (label 1, label 2, or label 3). The survey was distributed via Prolific, and at the conclusion, we presented all three labels to gather their feedback

If you’re happy with the price, we’ll start the work right away, by building each page across mobile, tablet and desktop.

Qualitative and Quantitaive Analysis

Qualitative:
For open-ended responses, we performed single coding using a jointly developed codebook by three authors. In earlier formative and pilot studies, we used double and triple coding with a high agreement rate of 85% to 95%. In the main study, one author independently conducted the coding process

Quantitative:
We used statistical tests like Kruskal-Wallis and Pearson's Chi-square to analyze how treatment conditions (label complexity and educational intervention) and demographics (age and technical background) influenced participant responses in our mostly quantitative survey. For multiple-choice questions, we used Chi-squared with a 0.05 significance threshold. In Likert-scale questions, we used Kruskal-Wallis at 𝛼 = 0.05 significance. For multi-select questions, we treated options as binary choices (True/False) and used Pearson's Chi-squared for significance testing

Once we’re 80% of the way through to completion, we will share a link for you to review the website on a staging site and provide feedback.

Web log Analysis

We verified people's claims of QR code scanning behavior by cross-referencing their statements with the log files to confirm if they had indeed scanned the QR code within the survey

We provide  you with post-30 day launch support and a library of personalised tutorials on how to use your new website so that you are well equiped to take control of the website.

User Quotes

"it's a lot faster for me to read the label that is already there,
as opposed to scanning a QR code. Also I am a little wary of scanning random
QR codes unless I already know that I can trust the source, as I have
heard about malicious QR codes." (referencing high complexity label)

"It had the right amount of information.
Others not enough or to much.
Very neverland or goldilocks happenings." (referencing high complexity label)

"I think it has just the right amount of information.
Label 4 is definitely too much. Label 1 and 2 is not
enough for people like my aunt who has a cell phone,
but is still not super tech savvy."

"It's the best of the bunch. The first label may as well not be there since its just a qr code.
The second label is somewhat useful but useful info is still hidden behind the qr code.
While the 3rd label provides data that I don't necessarily need while at the store,
it also gives information I'm interested in having.
The 4th label bombards you with far too much information -
only some of which would have any bearing on your general consumer. "

The details should be directly on the label. No business
should expect a customer to scan some random QR code
(with reference to label 1 having no upfront information)

"What data elements are collected &
with whom they are shared; any options
for me as a user to customize this" (on being asked what information is meaningful)

"Because I want the information on the box and to not
have to scan with a QR code and parse through information"

"Include more information without having the user scan the QR code.
As in, show who the data is shared with, and what the Trust Mark actually represents."

"Who is the data being shared with - and how is it secured? How can
I trust it not to spy on me, does it have a hardware switch for the various
sensors in case I needed to be discrete? What are the standards for the
cyber trust mark? A url alternative in-case the QR isn't working or doesn't seem trustworthy."

The right side should include more detail regarding the "collected"
and "shared" section. It should at the bare minimum provide the frequency
in which the data is collected, who is the data being shared to, and the general
purpose of collecting and sharing. The above image is just too simple to make use of it.

Pause

High Level Findings and Recommendations

Consumers prefer having detailed label directly on product packaging, with a strong preference for high-complexity labels, according to the study, indicating a universal demand for more information regardless of age or technical knowledge.

While some participants in the low-complexity condition scanned QR codes when no other information was available, most were unwilling to do so due to inconvenience and security concerns, highlighting the difficulties of comparing labels on small screens.

Participants in the study stressed the importance of including privacy-related information about data collection, usage, sharing, and how data will be used directly on product labels, which is currently lacking in the security-focused criteria for the Cyber Trust Mark.

Project 1
Project 1
Project 3
Project 3