Everything Hertz

Dan and James chat with Jon Brock (Cognitive scientist at Frankl) about the use of blockchain technology for open science. Here's what they cover: What is the blockchain? Why Jon made the jump from academia to Frankl A cryptocurrency for open science What do institutional review boards think about using blockchain for data collection and storage? Autism heterogeneity How will this approach scale to biological signals and genetics data? What’s something that Jon’s changed him mind about in regards to academia? Links Frankl https://frankl.io Five reasons Frankl has a token https://medium.com/franklopenscience/why-does-frankl-need-a-frankl-token-4129d718ab74 Bjoern Brembs blog post http://bjoern.brembs.net/2018/05/after-24-years-when-will-academic-culture-finally-shift/ An explainer on cryptographic hashes https://en.wikipedia.org/wiki/Cryptographichashfunction#Illustration Frankl in a nutshell https://medium.com/franklopenscience/frankl-in-a-nutshell-9b488c554dea Frankl for autism https://medium.com/franklope

Dan and James talk about the recent SIPS conference answer a listener question on "salami slicing" the outcomes from one study into multiple papers. Here's what they cover: What is the SIPS conference? [0:24] A SIPS proposal for Google scholar to highlight commentaries and replication attempts on specific articles [15:42] James and Dan’s favourite Hertz episodes [20:43] We answer a listener question on Salami slicing [28:45] Can you publish too much? [48:10] Links - SIPS conference: https://www.improvingpsych.org/SIPS2018/ - Reproducibilitea podcast: https://soundcloud.com/reproducibilitea - Salami slicing tweet: https://twitter.com/academicswrite/status/1008719899940786176 - Cumulative impact factors: http://khakhalin.blogspot.com/2012/11/cumulative-impact-factor-benchmarking.html - A working document from SIPS on making replications discoverable (including Google scholar) https://osf.io/57zxa/ Find us on Twitter: www.twitter.com/hertzpodcast www.twitter.com/dsquintana www.twitter.com/jamesheathers Music cr

Dan and James chat about science journalism with Brian Resnick (@b_resnick), who is a science reporter at Vox.com. Here’s what they cover: Should scientists be worried that their work will be misrepresented when talking to the media? [0:58] How Brian approaches science journalism [8:53] It’s ok to challenge the assumptions of science journalists [16:57] How do you write a great headline? [19:13] How does Brian appraise the quality of research? [29:50] Should psychiatrists (or journalists) diagnose the US President? [32:50] Stories in science that no one knows the answer to [36:58] How to promote your research without going via your institution’s media department [40:24] The best way to pitch your research to a science journalist [44:25] How pre-preprints are great for research addressing current events [48:45] How scientists can improve their science communication writing [53:15] Dick jokes in science writing — yes or no? [54:30] What has Brian changed his mind about? [56:37] Brian’s book recommendation [58

Dan and James chat about the adoption of open science practices with Dorothy Bishop, Professor of Developmental Neuropsychology at the University of Oxford. Here are some highlights from the show: Why Dorothy starting adopting open science practices Forking paths of analysis Dorothy’s experience with her first registered report Issues around data deposition When someone finds an error in your data What happens when a senior researcher is using questionable research practices? What to do when you’re caught doing something wrong Detecting errors in other papers The potential for open data to be weaponised How error detection is interpersonally difficult Carving out time for non-work pursuits The benefits of writing fiction when you're a scientist Using video for science communication James Heathers: Cat dealer Promoting your research vs. promoting yourself Dorothy’s book recommendation Links Dan Gilbert’s paper: http://science.sciencemag.org/content/351/6277/1037.2 Merchants of doubt [book]: https://www.amazo

Dan and James chat with Greg Nuckols, who is grad student in exercise physiology, strength coach, and writer at strongerbyscience.com What they cover in this episode: Why Greg blogs his papers before preprints How Greg combines his business with his grad study Getting your research to your audience without publishing in scientific journals The limitations of traditional publishing Addressing popular misconceptions in research Are questionable research practices as bad in sports science as they are in psychology? Being an “academic outsider” can be tough, but it has some advantages The work that goes into exercise physiology studies How practical are multilab research projects in sports science? Exercise “experts” on Instagram Using Instagram to disseminate research Greg’s go-to resources for learning about open science What Greg’s changed his mind about How Greg’s planning on funding his future research without grants Links Scihub - whereisscihub.now.sh Greg on Twitter - twitter.com/GregNuckols Greg's we

Dan and James answer listener questions on academic conferences, getting abreast of the literature, and conflicts of interest. Here are more details of what's on this episode: How question times during conference seminars are useless Choosing which conferences to attend as a PhD student Feedback from our Registered Reports episode with Chris Chambers (Episode 56) People that have binged our entire back catalogue The amount of reading do you need to do to keep track of the field you work in PhD students need time to make time to read the literature People sending out half-arsed work hoping that peer-review will “fix it” Guest authorship When you’re a native English speaker and get asked to have your manuscript proofed by a native English speaker Is it a conflict of interest to a review a paper with that includes someone you’ve co-authored with in the past on a different topic? The Frontiers journal model Reviewing papers so that authors are actually grateful for your criticism Links Nuzzle: http://nuzzel.

Despite cosmetic changes, scientific journals haven't changed that much over the past few decades. So what if we were to completely rethink how a scientific journal should operate in today's environment? Dan and James are joined by Rickard Carlsson (Linnaeus University, Sweden), who is the Co-Editor of the new "Meta-Psychology" journal. Here's what they cover: Why start a new psychology journal? What’s new about this journal? How does the journal have no subscription fees and no article processing fees? How does a new journal increase its profile? The difficulties in publishing negative results The limits of study pre-registration Are data archiving requirements unrealistic? Open polices and the Swedish constitution How can we make data anonymous? What’s the hardest thing about starting a journal? What would success look like for this new journal? What has Rickard changed his mind about recently? What’s one book or paper that Rickard thinks everyone should read Links The Daniel Lakens blog post on JPSP (T

Dan and James are joined by Simine Vazire (University of California, Davis and co-host of the Black Goat podcast) to chat about the role of podcasting in scientific communication. Dan's wife also starts going into labor during the episode, so this is an extra special one - make sure you listen through the ENTIRE episode. Here's what the cover: Why Simine started podcasting The perils of being a "methodologist terrorist" researcher Why podcast when you could blog or tweet? Dan and James’ favourite things about podcasting The current role of blogs Navigating the public/private crossover of science communication How much do we censor our podcasts? Should Journal editors tweet and podcast in a personal capacity? Should early career researchers podcast? The costs of not speaking above your station What equipment does we use to record podcasts? Two vs. three podcast hosts? How do you know when you have a good podcast? What type of person is suited to podcasting? What book does Simine think everyone should read? Wh

Dan and James are joined by Rebecca Willén (Institute for Globally Distributed Open Research and Education) to discuss transparency in scientific research and how she started her own independent research institute in Bali. Here's what they cover: Rebecca explains the story behind her practice of sharing disclosure statements for her published work Many people are changing their research practices for the better for current research - but what about their past research? The 21 word solution Using disclosure statements in your pHD The state of research openness in forensic psychology The flexibility in determining a primary outcome How and why Rebecca founded the IGDORE research institute The drawbacks to starting your own research institute Rebecca’s recommendation for getting started with open science The story behind the RONIN institute Links - IGDORE https://igdore.org Rebecca’s website https://rmwillen.info 21 word solution https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2160588 PNAS article questi

Dan and James are joined by Chris Chambers (Cardiff University) to discuss the Registered Reports format. Here’s an overview of what they covered: What is a registered report and why should we implement them? [1:47] The impact of conscious and unconscious bias on scientific publication [6:17] Common objections to registered reports [8:21] The slippery slope fallacy [14:33] The advantages of registered reports for early career researchers [15:47] The generational divide for embracing methodological reforms [19:13] The launch of registered reports in 2013 [23:30] The “tone debate” in psychology [24:50] Dealing with publishing decisions as an early career researcher [27:30] Using registered reports to disarm your research rivals [30:52] A peek behind the curtain of peer-review [34:40] How do we convince journals to take up the registered report format? [36:28] Using registered reports for meta-analysis [38:40] What’s something that Chris has changed his mind about recently? [43:14] What’s Chris’ favourite f

In this episode, Dan and James discuss the US National Institutes of Health's new definition of a “clinical trial”, which comes into effect on the 25th of January. Here’s the new definition: “A research study in which one or more human subjects are prospectively assigned to one or more interventions (which may include placebo or other control) to evaluate the effects of those interventions on health-related biomedical or behavioural outcomes”. Over the course of this episode, they cover the pros and cons of this decision along with the implications for researchers and science in general. Here are a few things they cover: The traditional definition of a clinical trial We go through James’ old work to determine if he’s been a clinical trialist all along The lack of clarity surrounding the new definition Why are adopting a clinical trial approach when this approach has obvious weaknesses? What do you actually have to do when running a clinical trial? Will institutions also adopt this new definition, thus putt

In this episode, James sits in the guest chair as Dan interviews him on his recent work find and exposing inconsistent results in the scientific literature. Stuff they cover: How James got into finding and exposing inconsistent results The critiques of James’ critiques How James would do things differently, if he were start over again? Separating nefarious motives from sloppiness The indirect victims of sloppy science Grants that fund sloppy science take resources from responsible science projects If people actually posted their data and methods, James’ job would be much easier Registered reports improve the quality of science If James could show one slide to every introductory psychology lecture what would it say? The one thing James believes that others think is crazy What James has changed his mind about in the last year Links The Sokal hoax: https://en.wikipedia.org/wiki/Sokal_affair James’ Psychological Science paper: http://journals.sagepub.com/doi/full/10.1177/0956797615572908 The @IamSciComm Tweet

Dan and James discuss whether you need to have “skin in the game” to critique research. Here's what else they cover in the episode: Should scientists be required to communicate their science? If your research is likely to be misinterpreted try and get out of in front of what's going to be said Will science communication just become another metric? The distinction between “science communication” and “science media” Who’s going to pay for all science communicators that we’ll need to communicate everyone’s science? Dan and James mispronounce Dutch and German names and give a formal apology to the nation of The Netherlands Outcome switching in clinical trials Does having skin in the game guarantee expertise, or just wild biases? James’ recent desk rejection from a Journal Editor Dan’s method to invite manuscript reviewers as an Associate Editor Links: The science communication Twitter thread https://twitter.com/ocaptmycapt/status/927193779693645825 ERC comics https://www.erccomics.com The “skin in the game”

In this episode, Dan and James welcome back Daniel Lakens (Eindhoven University of Technology) to discuss his new paper on justifying your alpha level. Highlights: Why did Daniel write this paper? Turning away from mindless statistics Incremental vs. seismic change in statistical practice The limitations to justifying your alpha The benefits of registered reports Daniel’s coursera course What’s better? Two pre-registered studies at .05 or one unregistered study at .005? Testing at the start of semester vs. the end of semester Thinking of controlling for Type 1 errors as driving speed limits Error rates mean different things between fields What if we applied the “5 Sigma” threshold used in physics to the biobehavioral sciences? What about abandoning statistical significance How did Daniel co-ordinate a paper with 88 co-authors? Using time zones to your benefit when collaborating How can junior researchers contribute to these types of discussions? Science by discussion, not manifesto The dangers of blanket

In this episode, Dan and James are joined by Jessica Polka, Director of ASAPbio, to chat about preprints. Highlights: What is ASAPbio? Differences between the publication processes in the biological sciences vs. the biomedical sciences Common concerns with preprints Media embargoes How peer review isn’t necessarily a mark of quality Do preprints make it harder to curate information? Specialty preprint servers vs. broad servers? How well do you need to format your preprint? How do you bring up preprints to lab heads and PIs? An example of a good preprint experience from Dan Using preprints for your grant applications What Jessica has changed her mind about The one article that Jessica thinks everyone should read Links Jessica's Twitter account - @jessicapolka ASAPbio - http://asapbio.org & @asapbio_ Rescuing Biomedical science conference 2014 resources - http://rescuingbiomedicalresearch.org/events/ Sherpa/Romeo - http://www.sherpa.ac.uk/romeo/index.php PaleoArxiv - https://osf.io/preprints/paleorxiv Principl

Dan and James celebrate their 50th episode with a live recording! They cover a blog post that argues grad students shouldn’t be publishing, what’s expected of today’s postdocs, and the ‘tone’ debate in psychology. BONUS: You can also watch the video of this episode on the Everything Hertz podcast channel (link below) Other stuff they cover: James offends a sociologist, as is his wont The argument for why grad students shouldn’t publish Gatekeepers controlling what’s being published Editors that Google authors before sending papers out for review Judging researchers on their institution’s location James on networking How do you challenge reviewers when they say you are "too junior" The standards of Frontiers papers Writing review papers for the wrong reasons Why are there so many meta-analyses? Pre-registering your meta-analysis Registered reports vs. pre-registration What’s expected of today’s postdocs How many papers should you peer review? How James tried to ward off review requests Things that millennia

In this episode Dan and James discuss a forthcoming paper that's causing a bit of a stir by proposing that biobehavioral scientists should use a 0.005 p-value statistical significance threshold instead of 0.05. Stuff they cover: A summary of the paper and how they decided on 0.005. Whether raising the threshold the best way to improve reproducibility? Is 0.005 too stringent? Would this new threshold unfairly favour “super” labs? If we keep shifting the number does any threshold really matter? Dan and James’ first impressions of the paper A crash course on Mediterranean taxation systems What would a 0.005 threshold practically mean for researchers? Links The paper https://osf.io/mky9j/ ENIGMA consortium http://enigma.ini.usc.edu Music credits: Lee Rosevere freemusicarchive.org/music/Lee_Rosevere/

Dan and James are joined by Jason Hoyt, who is the CEO and co-founder of PeerJ, an open access journal for the biological and medical sciences. Here's some of what they cover: PeerJ’s model and how it got started What goes into running a journal Impact factors vs. low-cost publishing When the journal user experience is too good Getting a quick reviewer turnaround The need scientists to change their practices (not publishers) PeerJ’s membership model Glamour journals Future plans for PeerJ Predatory journals Researchers don’t want cheap journals, only impact factors Links - PeerJ: https://peerj.com - The Phoenix project: https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business-ebook/dp/B00AZRBLHO - The Goal: https://www.amazon.com/Goal-Process-Ongoing-Improvement-ebook/dp/B002LHRM2O/ref=pdsim3512?encoding=UTF8&psc=1&refRID=EMTE1M9W2XW5Q24X4GE8 Music credits: Lee Rosevere freemusicarchive.org/music/Lee_Rosevere/ Special Guest: Jason Hoyt.

In this episode, Dan and James are joined by Anne Scheel (LMU Munich) to discuss open science advocacy. Highlights: - How Anne became an open science advocate - Open science is better science - Methodological terrorists/freedom fighters - The time Anne stood up after a conference keynote and asked a question - Asking poor PhD students to pay for conference costs upfront and then reimbursing them 6 months later - Is it worth if for early career researchers to push open science practices? - How to begin with implementing open science practices - Power analysis should be normal practice, it shouldn’t be controversial - Anne’s going to start a podcast - The 100%CI: A long copy blog with 4 writers - The benefits of preprints and blogging - Science communication in English for non-native English speakers - Doing stuff that interests you vs. stuff that’s meant to advance your career Twitter accounts of people/things we mentioned: @dalejbarr - 2:10 @siminevazire - 2:45 @lakens - 2:45 @nicebread303 (Felix Schön

In this episode, Dan and James are joined by Andy Field (University of Sussex), author of the “Discovering Statistics” textbook series, to chat about statistical literacy. Highlights: The story behind Andy’s new book SPSS and Bayesian statistics Andy explains why he thinks the biggest problem in science is statistical illiteracy Researcher degrees of freedom and p-hacking The story behind the the first version of ‘Discovering statistics’ How to improve your statistical literacy Does peer review improve the statistics of papers Researchers will draw different conclusions on the same dataset The American Statistical Association’s statement on p-values How has the teaching of statistics for psychology degrees changed over the years Andy fact checks his own Wikipedia page Andy’s thoughts on Bayesian statistics and how he applied it in a recent paper The peer review of new statistical methods Andy’s future textbook plans The rudeness of mailing lists/discussion forums What is something academia or stats-related