This repository contains supplemental material to enable replication of our study as submitted to the artifact evaluation for ACSAC 2022.
- Both versions of the survey as QSF-files
- Analysis script(s) used for quantitative analysis
- Codebooks used for qualitative analysis
- scripts for computing IRR and a breakdown of calculated IRR by round and IRR results
- html: (html, js) source files and scripts used to run the study recruitment page.
- scripts: (python) scripts used for data preprocessing and analysis
- analysis: analysis artifact files, mainly from our qualitative analysis. These include a copy of our qualitative codebooks and records of our inter-rater reliability while coding.
- survey_instruments: The surveys as given to Reddit and Prolific participants as Qualtrics export (QSF) files (see below for details).
Each QSF-file contains survey questions exported from Qualtrics and can easily be re-imported into there. There are two versions of the survey:
- The Survey as presented to participants from Reddit (survey_instruments/reddit/Smart_DNS.qsf)
- The prescreen (survey_instruments/prolific/SmartDNS-Prescreen.qsf) and main survey (survey_instruments/prolific/SDNS_-Main_Survey.qsf) as presented to Prolific participants.
Note that all of the surveys require Javascript and therefore will not work with free Qualtrics accounts. The surveys have multiple sections as outlined in the paper. These sections are modeled in the Qualtrics survey. Refer to the Qualtrics documentation in order to learn how to import the survey back into Qualtrics.
Though we did not include it in the QSF files, the original surveys contained a hidden question in which participants computers accessed (and thus requested the DNS resolution for) a domain under our control of the form <unique_id>.smartdnsstudy.com, where <unique_id> was a nonce assigned to the each unique participant session and *.smartdnsstudy.com was a wildcard domain for which we owned the authoritative name server.
(Per the conditions of our IRB approval and to protect participant privacy, we only stored records of the DNS resolution requests in our server logs.)
We then compared the DNS resolver IP addresses to a set of known SDNS resolvers to try and determine whether participants had configured SDNS to run on their computers and forgotten to disable it.
However, since our results were inconclusive, we ultimately decided not to report them in our paper and have omitted them from our artifacts.
Rahel A. Fainchtein, Adam J. Aviv, and Micah Sherr. User Perceptions of the Privacy and Usability of Smart DNS. The 2022 Annual Computer Security Applications Conference (ACSAC 2022).