Workshop on the evaluation of vaccine impact

Workshop learning objectives

• To understand the strengths and weaknesses of routinely collected administrative data

• To distinguish between vaccine impact and vaccine effectiveness studies

• To critically evaluate different methods for evaluating vaccine impact

• To format and analyze real world time series data

Prerequisites

• Knowledge of basic statistics (e.g., regression)

• Familiarity with R is recommended but not required

Structure of the course

• Recorded didactic presentations

• “Live” interactive data analysis sessions via RStudio Cloud and ZOOM, with recordings archived on Microsoft Stream

  • Accessibility: automatic English subtitles will be used on recorded videos

Communication

• Please join the Slack workspace for this course to participate in discussions about course material

How to navigate the course

1. For R novices only: complete the recorded introduction to R session video and the accompanying tutorial

2. Watch the recorded video on Microsoft Stream prior to the lives sessions to obtain background on the topic

3a) If you were selected to attend the live sessions, log onto ZOOm at the designated time and follow along with rstudio.cloud

3b) If you were not selected to attend the live session, view the recordings at a later time on Microsoft stream

Schedule for live sessions (max 1.5-2 hours each)

• US/Africa/Europe: January 4, 6, 11, 13

  • 9:30am US ET (New York); 2:30pm GMT (london); 4:30pm Johannesburg

• Asia/Pacific: January 5, 7, 12, 14

  • 12pm Melbourne Australia; 9am Manila and Beijing; 8am Hanoi (5pm US Pacific time/8pm New York on the previous day)

Course textbook

• This workshop will closely follow the material developed for a draft WHO guidance document on the use of administrative data for evaluating the impact of pneumococcal conjugate vaccines. That text can be accessed here.

For novices:

• Create a free account on https://rstudio.cloud This is a web-based version of RStudio, with all packages, data, and code pre-loaded

• Step 2: Join the Vaccine Evaluation Course workspace by clicking this link

• Step 3: Access the Vaccine Evaluation Course Workspace

  

• 4. Open a project by clicking on its name. For example: ‘Session-0-Intro-to-R’
  

For advanced users:

• Run the workshop materials on your local machine on RStudio. All workshop materials are provided as a set of RProjects, which can be access from https://github.com/orgs/VaccineEvaluationWorkshop

• You can either directly download and then open the projects on your computer by clicking the .rproj file in the corresponding directory, or you can download via Git, if installed on your computer

0. Introduction and Welcome to the course

Watch this brief video (10 minutes) for an introduction to the workshop (for subtitles, click the CC button at the bottom of the video; for translated subtitles, Click Settings/Subtitles/Select your Language)

1. Hands on: Introduction to R

This video and tutorial (42 minutes) is designed for R novices to introduce you to the R statistical programming language and the RStudio interface

• Watch this Video to orient you to RStudio and the R programming language. For subtitles, click the CC button at the bottom of the video; for translated subtitles, Click Settings/Subtitles/Select your Language)

• For novices, navigate to Session-0-Intro-to-R on the RStudio Cloud work space (see instructions above about how to set up an account on RStudio Cloud). You can follow along with the video to see how to get started, and then either follow the video or complete the exercises on your own.

• If you want to run the materials on your own computer, you can directly download the project here, unzip it, and click the .Rproj file to launch the project. Then open the .Rmd file and click ‘Run document’ at the top’

2. Recorded Presentation: What is ‘vaccine impact’?; and Administrative data: challenges and opportunities for evaluation studies

In this video (24 minutes), we will:

• Discuss the distinction between vaccine effectiveness and vaccine impact;

• Discuss types of administrative data and sources for these types of data

• Discuss limitations of administrative data

• (for subtitles, click the CC button at the bottom of the video; for translated subtitles, Click Settings/Subtitles/Select your Language)

3. Live session 1: Data Wrangling

-In this hands on session, we will start with a database of individual-level administrative data, create time series, and do some basic visualization

Join the ZOOM link provided and follow along on RStudio.cloud or on your own computer

• Recommended: navigate to Session1-Data-Wrangling on the RStudio Cloud work space (see instructions above about how to set up an account on RStudio Cloud

• If you want to run the materials on your own computer, you can directly download the project here, unzip it, and click the .Rproj file to launch the project

-For those who were unable to join for this session, a recording can be found here.

4. Recorded presentation: Interrupted time series analysis

In this video (28 minutes), we will discuss:

• The goals for trend analysis and interrupted time series analysis

• Different approaches for using ITS

• Critical evaluation of ITS methods

• How much data do you need?

• (for subtitles, click the CC button at the bottom of the video; for translated subtitles, Click Settings/Subtitles/Select your Language)

5. Live session 2: Trend analysis and interrupted time series analysis

Join the ZOOM link provided and follow along on RStudio.cloud or on your own computer

• Recommended: navigate to Session2-Trend-Analysis on the RStudio Cloud work space (see instructions above about how to set up an account on RStudio Cloud

• If you want to run the materials on your own computer, you can directly download the project here, unzip it, and click the .Rproj file to launch the project

6. Live session 3: Case study: PCV impact in The Gambia

In this live session, we will go through an example of applying interrupted time series analysis to data from The Gambia for the purspose of evaluating the impact of pneumococcal conjugate vaccines.

Join the ZOOM link provided and follow along on RStudio.cloud or on your own computer

• Recommended: navigate to Session3-Hands-on-Example-Gambia on the RStudio Cloud work space (see instructions above about how to set up an account on RStudio Cloud

• If you want to run the materials on your own computer, you can directly download the project here, unzip it, and click the .Rproj file to launch the project

7. Recorded presentation: Using controls and synthetic controls

-In this video (19 minutes), we will discuss:

• The goals for using control outcomes or control covariates

• Different approaches that are used for incorporating controls into your analyses

• Criteria for selecting candidate controls

• Synthetic controls methodology

• (for subtitles, click the CC button at the bottom of the video; for translated subtitles, Click Settings/Subtitles/Select your Language)

8. Live Session 4: Synthetic controls analysis (Case study: South Africa); presenting results and dealing with uncertainty

Join the ZOOM link provided and follow along on RStudio.cloud or on your own computer

• Recommended: navigate to Session4-Synthetic-Controls on the RStudio Cloud work space (see instructions above about how to set up an account on RStudio Cloud

• If you want to run the materials on your own computer, you can directly download the project here, unzip it, and click the .Rproj file to launch the project

9. Presenting results and dealing with uncertainty

-In this video (TODO), we will discuss:

• Principles for making nice plots

• Presenting uncertainty

• Which pieces of data should you focus on, and for which audiences

• (for subtitles, click the CC button at the bottom of the video; for translated subtitles, Click Settings/Subtitles/Select your Language)