Instructor: Amanda Hickman
Presenting: Sarah El Safty and Josh Slowiczek
You’re often going to find yourself working with numbers that were given to you by a source who has a vested interest in how your story turns out. Ask lots of questions. Be skeptical.
What do you think is the fastest way to reduce the number of unsolved rape cases in your precinct?
If there's a meaningful reward for moving the numbers, there's a real incentive to move the numbers without changing the underlying issue at all.
And if you see a hospital that advertises high surgical success rates, does that mean they have the best surgeons? Or that they only take easy cases?
VA Hospitals are addicted to metrics and they almost always turn out to be gameable, often in ways that make problems worse.
90% of fetuses diagnosed with Down Syndrome are aborted.
It is made by people. People take shortcuts. They interpret things and make calls.
The NYT -- often accidental gun deaths are classified as homicides and suicides -- one core takeaway is that it is up to the coroner to make a call, and they don't always make the call you'd expect. One result is that accidental gun deaths are wildly under counted.
Kansas: https://splinternews.com/how-an-internet-mapping-glitch-turned-a-random-kansas-f-1793856052 / https://splinternews.com/this-is-the-new-digital-center-of-the-united-states-1793856143 / https://source.opennews.org/articles/distrust-your-data/ (esp https://www.buzzfeed.com/ryanhatesthis/who-watches-more-porn-republicans-or-democrats and https://www.vox.com/2014/4/21/5636040/whats-the-matter-with-kansas-and-porn )
The Fires (Joe Flood) -- The Fires is one of a few excellent stories about the famous burning Bronx of the 1970s. One thing he talks about, which others have also talked about, is the direct relationship between radical cutbacks in FDNY station funding in the Bronx and the rise in fires -- maybe The Bronx was burning because the infrastructure to put fires out had been decimated. RAND's statistical team analyzed precicted citywide response times and determined that the city could safely close a lot of Bronx firehouses -- without acknowledging that those firehouses were some of the busiest in the city and often weren't responding to fires in their immediate vicinity because they were already out fighting fires. So a very "pure" data-driven approach conveniently rationalized shuttering a lot of stations in poor areas.
538 had to retract a story on broadband reach because they didn't understand the data they were working from.
IP addresses as a proxy for location will give you a ton of hits in Kansas.
I've talked about this before, so I won't dwell on it, but 311 calls are not a random sample of lived experiences.
This is a classic in statistics texts: in the 70s, researchers looked at admissions rates at UC Berkeley and found that women were far, far more likely to be rejected. A closer examination revealed that women were more likely to apply to more competitive programs, so department by department, there wasn’t evidence of discrimination. http://vudlab.com/simpsons/ --
“The only real way to maybe avoid this and similar potential land mines in a statistical model is to thoroughly understand your subject area and to be skeptical of any result." John Perry @ AJC
This is closely related to the ecological fallacy: if I tell you that states with more foreign born residents have more wealthy households, what’s your next question? (Are foreign born people more likely to be wealthy? No.) An older study showed that there was a positive state-by-state correlation between literacy and foreign born populations: areas with high immigrant populations were likely to be more literate. What you don’t know is whether immigrants are likely to be more literate.
Question order changes how people respond to questions. This is over a decade old now, but in '03, Americans were more likely to say they support civil unions if you already asked them if they support gay marriage.
Pew has a lot of great research about survey design.
Who is stuck? Let's brainstorm getting unstuck.