Italian e-cig study does not support the conclusion that e-cigarettes stimulate smoking cessation

A paper entitled "EffiCiencyand Safety of an eLectronic cigAreTte (ECLAT) as Tobacco Cigarettes Substitute: A Prospective 12-Month Randomized Control Design Study" was just published in PLoS One that is being interpreted as indicating that e-cigarettes lead to cessation of conventional cigarettes among people who are not planning to quit smoking regular cigarettes.
 
For the reasons discussed below, this is not an appropriate interpretation of the results in this study.
 
This study involved randomizing people in Italy who said they were using conventional cigarettes and not interested in quitting to three groups: one receiving e-cigarettes with higher nicotine (Group A), one receiving e-cigarettes with a lower level of nicotine (Group B), and a third group receiving zero nicotine e-cigarettes (Group C, which the paper considered the “control” group). Based on comparing quit rates of conventional cigarettes one year later, the authors concluded that “in smokers not intending to quit, the use of e-cigarettes, with or without nicotine, decreased consumption and elicited enduring tobacco abstinence.”  This conclusion is not supported by the data in the paper for two reasons.
 
First, and most important, despite the fact that the title describes the paper as a “randomized control design,” there is not a control group of people who were not using e-cigarettes that would allow assessment of spontaneous quit rates.  By not having a true control group that would account for spontaneous quitting without using e-cigarettes one cannot say anything about whether e-cigarettes affected quitting.
 
This is a very important point because, as noted in my textbook Primer of Biostatistics (7ed, McGraw-Hill, 2012, p. 250), “To reach meaningful conclusions about the efficacy of some treatment, one must compare the results obtained in the individuals who receive the treatment with an appropriate control group that is identical to the treatment group in all respects except the treatment. Clinical studies often fail to include adequate controls. This omission generally biases the study in favor of the treatment.”
 
Second, there are issues with the statistical analysis which, when corrected, eliminate the reported statistically significant results. 
 
The authors state that “At week 52 quitters were 22/200 (11.0%) in Groups A-B [the two groups of nicotine e-cigarette users combined]  and 4/100 (4.0%) in Group C [the zero nicotine e-cigarette users] (p = 0.04), which is the basis for the “enduring abstinence” conclusion.  The authors based this conclusion on the fact that a chi-square test of a 2 x 2 contingency table (smoking or not smoking vs nicotine or non-nicotine e-cigarettes) reached statistical significance (p = 0.04, which is less than 0.05, the cutoff for conventional statistical significance).  The problem is that the authors failed to include the required Yates correction* in their calculation of the chi-square test statistic and associated p value.  Recalculating the test properly yields p = 0.07, which is no longer statistically significant.  Thus, the correct conclusion is that there is no statistically significant difference between the nicotine and non-nicotine e-cigarettes.
 
Probably a more appropriate comparison – which follows the experimental design – would be to treat the data as a 2 x 3 contingency table (smoking or not smoking vs the three different kinds of e-cigarettes).  The chi-square analysis of the 3 x 2 contingency table yields p = 0.08, which is even further from statistical significance.
 
Thus, combining the fact that there is not a non-e-cigarette control group and correcting the statistics means that the appropriate conclusion to draw about quitting smoking based on these data is that the level of nicotine in the e-cigarette (including zero nicotine) has no detectable effect on quitting smoking conventional cigarettes.
 
These data cannot be used to support any statement about the efficacy of e-cigarettes for stimulating smoking cessation one way or the other. 
 
*NOTE ON THE YATES CORRECTION (from Primer of Biostatistics, page 84): “… when analyzing 2 × 2 contingency tables, the value of chi-square computed [from the data] and the theoretical chi-square distribution leads to P values that are smaller than they ought to be. Thus, the results are biased toward concluding that the treatment had an effect when the evidence does not support such a conclusion. The mathematical reason for this problem has to do with the fact that the theoretical chi-square distribution is continuous whereas the set of all possible values that the chi-square test statistic can take on is not. To obtain values of the test statistic that are more compatible with the critical values computed from the theoretical chi-square distribution [for a 2 x 2 contingency table], apply the Yates correction (or continuity correction) to compute a corrected chi-square test statistic …. This correction slightly reduces the value of chi-square associated with the contingency table and compensates for the mathematical problem just described.”
I have posted essentially the same comment (without some of the introductory explanation) as a comment on the paper on the PLoS One website.

Comments

Polosa too $316,050 from Philip Morris

Riccardo Polosa, one author of the ECLAT e-cigarette study, took $316,060 from Philip Morris for a two year research project between 2003 and 2005.  (Related news story.)

What the two good population-based studies found on quitting

In response to my posting of this comment on the PLoS One website, the president of the "Consumer Advocates for Smokefree Alternatives Association" selectively quoted the results of the two population-based studies of the effects of using e-cigarettes on quitting conventional cigarettes and represented these papers as not saying what I said they said.  Here is what these two papers actually say.

On page 213 Adkison et al state: "ENDS users stated that they used ENDS as a tool to help them quit smoking, although only 11% of current ENDS users report having quit since Wave 7. Quitting did not differ between users and non-users, chi-square (2, n=4136) = 0.442 (P=0.516).

The Results section of Vickerman et al states, "Both e-cigarette user groups were significantly less likely to be tobacco abstinent at the 7-month survey compared with participants who had never tried e-cigarettes (30-day point prevalence quit rates: 21.7% and 16.6% vs. 31.3%, p < .001)."

These statements seem pretty clear to me.

Ordered treatments and e-cigarettes

The test for effect of ordered treatments (which they were - by nicotine content) gives a p-value of 0.027.

A no e-cig control group could not be blind. a zero nicotine e-cig control group is perfectly sensible.

The fundamental problem is the wrong control group

The authors did treat the zero nicotine e-cigarette group as the control group; that is my point. What the paper compares is people using different levels of nicotine e-cigs on quitting smoking among people using e-cigs.  Regardless of whether the differences are statistically significant or not, the data in this paper can only be used to draw conclusions among the three groups that were tested. Since there was not a control group of people not using e-cigarettes, there is no information on the background level of spontaneous cessation among cigarette smokers (not using e-cigarettes) to make any statements about whether e-cigarettes affect conventional cigarette cessation. There are two population-based studies of the relationship between e-cigarette use and cessation: as one found no effect and the other showed e-cigarette smokers were less likely to quit. 

Wrong control group

I take your point that a control group of subjects with no e-cigarette use would be relevant to the question  - but the study shows that increased levels of nicotine delivery in e-cigarettes leads to higher levels of quitting.

I'm sure you're aware that many studies of human behaviour go to great lengths to create dummy interventions that mimic the active treatment but do not include the key element of the intervention - e.g. trials of behaviour therapy. Seems to me that this is similar, with the active intervention being the delivery of nicotine via the e-cigarette.

It is of course always possible that if Ss were recruited to a group given no e-cigarettes that their smoking cessation rate would be high because they knew they were in a trial, and I assume this is your point. Seems to me we could get a handle on this by looking at rates of cessation in control groups in trials of pharmaceutical interventions.

Doesn't solve the design problem with the Italian study

One could have designed such a study, but the Italian group didn't do it.

Chapman estimates that the 1 year unassisted quit rate for conventional cigarettes is 7%, which is within the 95% confidence interval for the quit rate for the high nicotine e-cig users -- 6.3% to 19.7% -- so taking the approach you suggest would still fail to support the conclusion that the e-cigarettes were significantly increasing quit rates.

The important point remains that all the authors can talk about is how different levels of nicotine in the e-cigarettes affected quitting among people using e-cigarettes.  Their design does not permit saying anything about how e-cigarettes affect quitting in an absolute sense, i.e., compared to smokers who do not use them.

Chapman estimates are for smokers trying to quit

The ECLAT study recruited smokers not intending to quit. It can be argued that some or all recruits may have been participating with a subconscious desire to quit, but we just don't know. Still seems as if Chapman is an orange to the ECLAT apple.   -akh

Yes

That is correct and why I did not cite the Chapman number in my original post.

I was citing it because the comment I was responding to suggested that such a comparison might be appropriate.

I agree that the appropriate control group for the ECLAT study to draw conclusions about whether e-cigarettes promoted quitting would have been smokers not intending to quit.  They didn't do that, which is why their study cannot be used to support a conclusion about the effects of e-cig use on quitting.

Nicotine levels

There are lots of things wrong with this study - that's not surprising - but *if* they had only been interested in the 52 wk outcome of quitting and hadn't screwed things up with reduction outcomes and multiple testing they could have concluded that increasing nicotine level in e-cigarettes was significantly  associated with increasing smoking cessation.

I would also point out that dummy (no nicotine) patches and dummy (no nicotine) gum are widely used in clinical trials...

Not really

I don't agree, since you don't know what the background level of cessation is.  There is a possibility that the e-cigs could have reduced smoking cessation among the no nicotine e-cig users and had no effect on the high dose e-cig users.