Find me
The Developmental Aspects of Sexual Health Laboratory
  • Home
  • People
  • Research Projects
  • Blog
  • Publications
  • Presentations

Responsible reporting

2/12/2015

0 Comments

 
Last week in Professional Development we talked about ethical data management. This week we moved to issues with reporting, including data analysis and interpretation. We discussed an interesting paper by John, Loewenstein, and Prelec (2012) in which they surveyed academic psychologists about their engagement in 10 different “questionable research practices” (some of which overlapped with our discussion last week). They told some of the participants that they would be using Bayesian Truth Serum (BTS) – that is, they would use a scoring algorithm to assess the pattern of their scores and determine the honesty of their responses, and then would make a donation to the charity of the participant’s choice based on how truthful their responses were. They did find some differences (based on the algorithm) between the two groups, with the BTS group reporting higher levels of questionable practices than the control group. Our discussion, however, focused more on the percentages of participants who admitted to specific behaviors. It was interesting to see which behaviors students thought were surprisingly high, or were surprised to see on a list of questionable behaviors. It led to a good discussion of whether certain practices, such as not reporting all dependent measures, were questionable/unethical. One student was also very surprised at the rate (43% in BTS group; 38% in control group) of admitting to rounding p-values, and also, in the ratings of how defensible it was to round p-values.

We talked about defamation, and the extent to which it is okay for journalists or researchers to attack researchers in the public domain. And whether there’s a point where it’s okay for the researcher to sue for defamation. We used the Michael Mann case as an example.

We talked about reporting of fMRI data. I confess it is out of my comfort zone to talk about methods for researching fMRI. But we read a paper by Vul, Harris, Winkielman, and Pashler (2009) to spur the discussion. Fortunately, one of my students works with fMRI data, and it was great to have her perspective.

And we discussed some issues to consider in responsible reporting. Such as:

Correcting for number of tests/Type I error.

Statistical significance vs. practical importance/meaningfulness. Studies with enormous sample sizes can demonstrate significant results that, when thought of in effect sizes, are essentially meaningless. On the flip-side, there are sometimes meaningful differences that may not reach the magical .05.

p-hacking/fishing. Gelman has discussed this issue repeatedly on his blog, as have many others. Gelman has also talked about what he refers to as researcher degrees of freedom (decisions made that don’t involve statistical fishing but may still be questionable). I think that fishing is a very common issue in research, particularly with secondary data analysis. It is really useful for students to think about it early on, and to learn how to formulate hypotheses and research questions before running analyses.

Comparing two analyses without statistical comparisons. I have railed about this issue for many years. Gelman and Stern (2006) wrote a great paper called “The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant” (Yes, I always tell my students not to use article titles in their writing; I make an exception here). The issue is that researchers often say that two things were correlated, and two other things were not correlated, and therefore they are meaningfully different. Or, two things were correlated for one group, but not the other. For instance, maybe parent-child conflict correlated with substance use at .25, p < .05, whereas parent-child closeness correlated with substance use at .22, p > .05. And the researcher/author might then conclude that conflict matters for substance use, but closeness does not. Not okay!  In a paper of mine, I once ran regressions predicting sexual behavior from a set of gendered attitudes, and was interested to see if the gendered attitudes mattered more for men’s or women’s sexual behavior. So I included interactions between each gendered attitude and biological sex. A reviewer then said that I needed to instead run the regressions separately for men and women to see what was significant. That was at least 5 years ago, and I clearly still haven’t gotten over it.

Causal conclusions when not warranted.

I learned the term HARKING.

Preregistration: We discussed arguments for and against pre-registration of hypotheses. And, taking pre-registration a step further, the idea of pre-registering hypotheses AND using simulated data to test hypotheses before actually running analyses in the actual data.   

Students also generated the additional issues to consider:

Treating p < .05 as a magical/meaningful cutoff

How/when to report marginal significance/trend level findings.

“The post Responsible reporting first appeared on Eva Lefkowitz’s blog on February 12, 2015.”

0 Comments

Your comment will be posted after it is approved.


Leave a Reply.

    Eva S. Lefkowitz

    I write about professional development issues (in HDFS and other areas), and occasionally sexuality research or other work-related topics. 

    Looking for a post doc? 
    List of HDFS-relevant post docs
    Looking for a fellowship? 
    List of HDFS relevant fellowships, scholarships, and grants
    Looking for an internship?
    List of HDFS-relevant internships
    Looking for a job?
    List of places to search for HDFS-relevant jobs

    Categories

    All
    Adolescent Development
    Being A Grad Student
    Conferences
    Excel
    Gmail
    Grant Proposals
    Job Market
    Mentoring
    Midcareer
    Networking
    PowerPoint
    Publishing
    Research
    Reviewing
    Sexual Health
    Social Media
    SPSS
    Teaching
    Theses & Dissertations
    Transitions
    Undergraduate Advice
    Word
    Work/life Balance
    Writing

    Archives

    February 2022
    May 2021
    January 2021
    July 2019
    June 2019
    May 2019
    January 2019
    November 2018
    October 2018
    September 2018
    August 2018
    July 2018
    June 2018
    May 2018
    March 2018
    October 2017
    November 2016
    June 2016
    May 2016
    April 2016
    February 2016
    January 2016
    December 2015
    November 2015
    May 2015
    March 2015
    February 2015
    January 2015
    December 2014
    November 2014
    October 2014
    September 2014
    August 2014
    July 2014
    June 2014
    May 2014
    April 2014
    March 2014
    February 2014
    January 2014
    December 2013
    November 2013
    October 2013

    Tweets by @EvaLefkowitz

    RSS Feed

    View my profile on LinkedIn

    Enter your email address:

    Delivered by FeedBurner

    Blogs I Read

    Female Science Professor

    The Professor is in

    APA Style Blog

    Thinking About Kids

    Tenure She Wrote

    Prof Hacker

    Andrew Gelman

    Claire Kamp Dush
Proudly powered by Weebly