There has been a lot of talk about the IHME Covid-19 projection model. @EpiEllie & I have a chat about it in tomorrow& #39;s @casualinfer episode; here is a quick description of what is going on here with a focus on the *uncertainty*

https://abs.twimg.com/emoji/v2/... draggable="false" alt="🔗" title="Link symbol" aria-label="Emoji: Link symbol"> https://covid19.healthdata.org/united-states-of-america

1/14">https://covid19.healthdata.org/united-st...

When I look at models, I usually start with two things:

https://abs.twimg.com/emoji/v2/... draggable="false" alt="📈" title="Chart with upwards trend" aria-label="Emoji: Chart with upwards trend"> What method is being used?
https://abs.twimg.com/emoji/v2/... draggable="false" alt="📥" title="Inbox tray" aria-label="Emoji: Inbox tray"> What data is it based on?

Let& #39;s start with the methods!

2/14

https://abs.twimg.com/emoji/v2/... draggable="false" alt="📈" title="Chart with upwards trend" aria-label="Emoji: Chart with upwards trend"> The IHME model is estimating the log of the cumulative death rate for a given state at a given time
https://abs.twimg.com/emoji/v2/... draggable="false" alt="🌊" title="Water wave" aria-label="Emoji: Water wave"> Using curve fitting¹
https://abs.twimg.com/emoji/v2/... draggable="false" alt="📏" title="Straight ruler" aria-label="Emoji: Straight ruler"> parametrized with info about the state& #39;s social distancing

--
¹ in particular it is a non-linear mixed effects model https://abs.twimg.com/emoji/v2/... draggable="false" alt="ℹ️" title="Information source" aria-label="Emoji: Information source"> https://ihmeuw-msca.github.io/CurveFit/methods/
3/14">https://ihmeuw-msca.github.io/CurveFit/...

https://abs.twimg.com/emoji/v2/... draggable="false" alt="📈" title="Chart with upwards trend" aria-label="Emoji: Chart with upwards trend"> Since the IHME model is trying to estimate a *curve* there are https://abs.twimg.com/emoji/v2/... draggable="false" alt="✌️" title="Victory hand" aria-label="Emoji: Victory hand"> two important pieces:

https://abs.twimg.com/emoji/v2/... draggable="false" alt="1️⃣" title="Keycap digit one" aria-label="Emoji: Keycap digit one"> When will deaths "peak"
https://abs.twimg.com/emoji/v2/... draggable="false" alt="2️⃣" title="Keycap digit two" aria-label="Emoji: Keycap digit two"> How many deaths will there be at the "peak"

4/14

https://abs.twimg.com/emoji/v2/... draggable="false" alt="📥" title="Inbox tray" aria-label="Emoji: Inbox tray"> To estimate when these, the IHME model has two sources of info:

https://abs.twimg.com/emoji/v2/... draggable="false" alt="⏱" title="Stopwatch" aria-label="Emoji: Stopwatch"> the current death rate over time for the state
https://abs.twimg.com/emoji/v2/... draggable="false" alt="📏" title="Straight ruler" aria-label="Emoji: Straight ruler"> the social distancing measures being implemented

5/14

This information is combined with some https://abs.twimg.com/emoji/v2/... draggable="false" alt="🌍" title="Earth globe europe-africa" aria-label="Emoji: Earth globe europe-africa"> global info as well

https://abs.twimg.com/emoji/v2/... draggable="false" alt="👶" title="Baby" aria-label="Emoji: Baby"> In the short run, the model is impacted more by the state& #39;s data
https://abs.twimg.com/emoji/v2/... draggable="false" alt="👴" title="Older man" aria-label="Emoji: Older man">in the long run, they use info from locations that have seemingly already reached a peak: Wuhan, 5 in Italy, 2 in Spain

6/14

OKAY now that we know what the IHME model is doing, let& #39;s get to the good stuff - where is the uncertainty?

https://abs.twimg.com/emoji/v2/... draggable="false" alt="1️⃣" title="Keycap digit one" aria-label="Emoji: Keycap digit one"> There is uncertainty that the model itself will accurately predict what will happen (it& #39;s based on a Gaussian error function - is that right?)

7/14

https://abs.twimg.com/emoji/v2/... draggable="false" alt="2️⃣" title="Keycap digit two" aria-label="Emoji: Keycap digit two"> There is uncertainty in the distributional assumptions of the model
https://abs.twimg.com/emoji/v2/... draggable="false" alt="3️⃣" title="Keycap digit three" aria-label="Emoji: Keycap digit three"> Even if the model is correctly specified, there is uncertainty in the parameter estimation (this is a mixed effects model, so there is uncertainty associated with fixed and the random effects)

8/14

https://abs.twimg.com/emoji/v2/... draggable="false" alt="4️⃣" title="Keycap digit four" aria-label="Emoji: Keycap digit four"> There may be systematic uncertainty in the reported state-by-state death data. Why? Fewer deaths may be reported on weekends, if systems are overrun, COVID-19 related deaths may go unreported¹, etc

--
¹ https://www.npr.org/sections/coronavirus-live-updates/2020/04/08/829506542/after-deaths-at-home-in-nyc-officials-plan-to-count-many-as-covid-19

9/14">https://www.npr.org/sections/...

https://abs.twimg.com/emoji/v2/... draggable="false" alt="5️⃣" title="Keycap digit five" aria-label="Emoji: Keycap digit five"> There may be random uncertainty in the reported state-by-state death data
https://abs.twimg.com/emoji/v2/... draggable="false" alt="6️⃣" title="Keycap digit six" aria-label="Emoji: Keycap digit six"> There is uncertainty in the reported information coming from cities that seem to have already peaked

10/14

So let& #39;s recap on the uncertainty in the IHME model:

https://abs.twimg.com/emoji/v2/... draggable="false" alt="1️⃣" title="Keycap digit one" aria-label="Emoji: Keycap digit one"> model choice
https://abs.twimg.com/emoji/v2/... draggable="false" alt="2️⃣" title="Keycap digit two" aria-label="Emoji: Keycap digit two"> model parameters
https://abs.twimg.com/emoji/v2/... draggable="false" alt="3️⃣" title="Keycap digit three" aria-label="Emoji: Keycap digit three"> model estimation
https://abs.twimg.com/emoji/v2/... draggable="false" alt="4️⃣" title="Keycap digit four" aria-label="Emoji: Keycap digit four"> data from the states (systematic)
https://abs.twimg.com/emoji/v2/... draggable="false" alt="5️⃣" title="Keycap digit five" aria-label="Emoji: Keycap digit five"> data from the states (random)
https://abs.twimg.com/emoji/v2/... draggable="false" alt="6️⃣" title="Keycap digit six" aria-label="Emoji: Keycap digit six"> data from the "peaked" locations

11/x

In the original model (pre-last week) the error bands you saw only accounted for https://abs.twimg.com/emoji/v2/... draggable="false" alt="3️⃣" title="Keycap digit three" aria-label="Emoji: Keycap digit three">, since then the model was updated so that the uncertainty also accounts for out-of-sample uncertainty, which I believe covers https://abs.twimg.com/emoji/v2/... draggable="false" alt="5️⃣" title="Keycap digit five" aria-label="Emoji: Keycap digit five">

12/14

The shaded red region in the model is the *uncertainty* the model accounts for, just two of the 6:

https://abs.twimg.com/emoji/v2/... draggable="false" alt="❌" title="Cross mark" aria-label="Emoji: Cross mark">https://abs.twimg.com/emoji/v2/... draggable="false" alt="1️⃣" title="Keycap digit one" aria-label="Emoji: Keycap digit one"> model choice
https://abs.twimg.com/emoji/v2/... draggable="false" alt="❌" title="Cross mark" aria-label="Emoji: Cross mark">https://abs.twimg.com/emoji/v2/... draggable="false" alt="2️⃣" title="Keycap digit two" aria-label="Emoji: Keycap digit two"> model parameters
https://abs.twimg.com/emoji/v2/... draggable="false" alt="✅" title="White heavy check mark" aria-label="Emoji: White heavy check mark">https://abs.twimg.com/emoji/v2/... draggable="false" alt="3️⃣" title="Keycap digit three" aria-label="Emoji: Keycap digit three"> model estimation
https://abs.twimg.com/emoji/v2/... draggable="false" alt="❌" title="Cross mark" aria-label="Emoji: Cross mark">https://abs.twimg.com/emoji/v2/... draggable="false" alt="4️⃣" title="Keycap digit four" aria-label="Emoji: Keycap digit four"> data from the states (systematic)
https://abs.twimg.com/emoji/v2/... draggable="false" alt="✅" title="White heavy check mark" aria-label="Emoji: White heavy check mark">https://abs.twimg.com/emoji/v2/... draggable="false" alt="5️⃣" title="Keycap digit five" aria-label="Emoji: Keycap digit five"> data from the states (random)
https://abs.twimg.com/emoji/v2/... draggable="false" alt="❌" title="Cross mark" aria-label="Emoji: Cross mark">https://abs.twimg.com/emoji/v2/... draggable="false" alt="6️⃣" title="Keycap digit six" aria-label="Emoji: Keycap digit six"> data from the "peaked" locations

13/14

This is not unusual or bad! It is just good to keep in mind the uncertainty that these projections carry with them. If all of the uncertainty we& #39;ve talked about today was quantified, it& #39;s possible we& #39;d basically have no answers to go off of https://abs.twimg.com/emoji/v2/... draggable="false" alt="🤷‍♀️" title="Woman shrugging" aria-label="Emoji: Woman shrugging">

https://twitter.com/hspter/status/1246955939946803202?s=20

14/14">https://twitter.com/hspter/st...

Think I missed something important? Please let me know! https://abs.twimg.com/emoji/v2/... draggable="false" alt="🙏" title="Folded hands" aria-label="Emoji: Folded hands">