Normally I\’m a big Ben Goldacre fan

But I think he got something a little wrong here.

On top of that, we also know that researchers can change their stated goal, or \”primary outcome\”, after their trial has finished. You might do a trial on a blood pressure pill, for example, stating that you will look to see if it can reduce heart attacks, but find that it doesn\’t. Then you might retrospectively change the purpose of your study, ignore the heart attacks, pretend it was only ever about blood pressure, and glowingly report a reduction in blood pressure as if this was what you were always interested in. Or you might measure so many different things that some of them will show up as positive simply by chance.

The reason is this:

Researchers have found that the use of drug immediately after a stroke can reduce damage to brain cells by as much as two thirds.

Strokes are blood clots in the brain that starve the surrounding cells of oxygen eventually leading to their death and then to irrevocable brain damage.

The immediate injection of antibiotic minocycline protects the cells and reduces the swelling, saving many from losing brain cells.

It is particularly effective on nerve cells that process and transmit information in the brain.

Dr Cesar Borlongan from the University of South Florida, who led the team, said the drug was already available as it was used for acne and arthritis.

He said the results showed the technique could be much “more effective” than current treatments.

Now I agree that this was the result of a trial specifically to test this. But that\’s not quite my point.

If, in the course of spending that $800 million to design and test a drug the researchers find that it doesn\’t do what they thought it would, but it does do something else (like Viagra, not all that great at reducing angina, great at giving people stiffies) what are they supposed to do?

Throw away the $800 million and the evidence they\’ve collected?

Or say, hey, well, guess what we found out? Our acne treatment stops people dribbling and falling over after a stroke. Ain\’t that interesting?

Information is information, knowledge is knowledge: I\’m not sure it matters all that much what you were looking for. What you\’ve found is surely the most important part?

9 thoughts on “Normally I\’m a big Ben Goldacre fan”

  1. I don’t think that is what Ben is saying. many drugs have followed a course similar to your two examples. What Ben is saying is that you shouldn’t use a study designed to investigate one endpoint as a definitive study for a different endpoint thatappeared during the trial. You should of course use it as indicative of an unsuspected effect but then that effect needs to be tested in a study designed to look at that endpoint.

  2. “Wasn’t a certain impotence drug that I can’t mention”

    Certainly was. In fact, that’s how the patents were undone: the effect was noted as a curio in the scientific literature before Pzifer thought “hang on a minute..”

  3. That’s right. In fact a male patient once put a nitrate patch (for angina) on his penis, I can’t recall why, and then required his wife to deal with the subsequent tumescence via sexual intercourse. Following which, having absorbed a massive amount of nitrates, she was incapacitated by headache.

    But Ben’s point is straightforward. You should look at the tables of secondary outcomes and other parameters in my anaesthetic journals. There are potentially dozens of these in a large trial. Keeping in mind that the standard measure of statistical significance means only that the result is only 5% likely to be due to chance, there is a one in 20 risk that any particular ‘outcome’ is just dumb luck. So if you have 20 outcomes to be measured such as lower pain scores, blood pressure, happiness and so on, and one of them is unexpectedly fruitful, you haven’t proven anything. That doesn’t mean it can’t be investigated further, but you wouldn’t bet your house on it.

  4. If your column is disappearing into the ether perhaps you may want to research certain drugs.

    Obvious and trivial – excellent!

  5. I don’t think anyone’s quite got Ben’s point, which is…

    Say you run 20 trials on a drug you think will will reduce heart attacks from which you get 2 positives and 18 negatives, but solid secondary evidence that it reduces blood pressure, and report all trials in full, including the negatives.

    What you get is a new drug you can bring to market for reducing blood pressure but not one you can market as reducing heart attacks because the trial data does not prove its efficacy.

    If, on the other hand, you run the same trials on the same drug, but instead of reporting that 18 proved negative for reducing heart attacks you report those trials as if they were only ever about blood pressure, then the drug is likely to go to market on the basis that it can reduce heart attacks and reduce blood pressure, even though the data from all trials fails to support the former conclusion.

    Reporting secondary outcomes is not the problem here.

    Failing to negative results relating to primary outcomes is.

  6. hi tim,

    like various others i think you’ve got the wrong end of the stick here.

    nobody’s saying that you shouldn’t be allowed to report incidental and secondary findings, that they should be silenced, just that you shouldnt pretend that those findings were always supposed to be your primary outcome.

    there are lots of reasons for this, one of the most important is the issue of correction for multiple comparisons. the other is the issue of people quietly sweeping negative findings for the primary outcome under the carpet by pretending that they were never supposed to be the primary outcome, a good explanation of this from the comments elsewhere:

    “An analogy: A given road is particulaly bad for fatal car crashes. The council paints confusing markings on the road, on the theory that it will cause people to slow down, and so be less likely to die from a crash. They monitor the traffic, and find that indeed the cars are now slower, and pat themselves on the back and conclude the plan was a success. However, there are actually more crashes now, and even with a higher survival rate, more deaths. If the council claimed they would measure deaths, but found the speed was good but the deaths were bad, and so published only the speed data and pretended they never collected the deaths data, that an example of changing the stated outcome.”

    again: nobody is saying censor or don’t report secondary and incidental findings. just don’t pretend they were always your primary outcome. pretty simple.

  7. “just don’t pretend they were always your primary outcome. ”

    I take it you play the version of pool where you nominate the pocket beforehand rather than saying “aimed for!” after a fluke. Rather like Dave Lister in Red Dwarf episode “White Hole” (series 4).

    I’m worried. I think Red Dwarf has become my I Ching.

Leave a Reply

Your email address will not be published. Required fields are marked *