Part one – published June 29, 2018
As a journalist and as the son of a mathematician, I view studies, especially those trumped up by interest groups and governments, with an eye of suspicion.
This is especially true when a government says a study has come out that validates everything it has said.
I’m generally suspicious, and generally look to verify, or at least get response to, anything a ruling or opposition politician says. After all, a lot of politics is about spin.
But when you add to that a study saying one thing, I get even more suspicious.
In part, because a good study is rarely so declarative. To prove something, you have to control for other variables. And in studies of things like health and proposed policies, many of those variables are impossible to control.
My father is a university math professor. He has taught calculus, linear algebra and probability. He taught me about statistics. My mother is a nurse and avid reader of nursing journals. Between the both of them, I have developed a healthy scepticism of headlines contained in newspapers and non-trade magazines about the responses of studies, especially ones that are ground-breaking or include conclusions no one else has reached before.
In journalism school, we’re taught to assess studies before race off to report on them.
The things you always have to report on any study, or any poll, is both the methodology, the author, the funder and, in the case of polls, the margin of error.
Often, the way a poll question or study topic is framed can have a massive impact on what it determines.
It is with all of this background that I clicked on a headline from a provincial government news release that insisted the provincial government’s carbon tax arguments are backed up by the University of Regina.
I opened the press release and looked through it, like I have with so many other studies, and a few things stuck out to me.
Firstly, it was peculiar that there were no quotes from the author of the study. In fact, the document’s author was neither in the press release nor the actual analysis itself, just the institute where he worked.
Second, it seemed strange that nowhere did it indicate who had funded this report, important when considering an issue with as many potential legal, political, environmental and economic consequences as the carbon tax debate.
Third, I found it unusual that I couldn’t find the analysis independently when I went to the University of Regina’s website, or to the website of the Institute for Energy, Environment and Sustainable Communities. If the findings were as big as the province said they were, wouldn’t the University want to show them off to the world?
As it turns out, this was not an independent study of the University of Regina. Provincial Environment Minister Dustin Duncan, when questioned by reporters, admitted as much.
The piece of analysis was funded by the province, which had commissioned the team through a request for proposals.
“The Ministry of Environment did put out a call, an RFP, to researchers to put together a model that would help to demonstrate the argument that we believed we could make,” he told Arthur White-Crummey of the Regina Leader-Post.
The study, then, despite is length, isn’t peer-reviewed, and in reading it, the document contradicts itself, saying both that a carbon tax is ineffective, and stating, at one point, that “there is no doubt that carbon tax is an effective way to reduce the GHG emissions.”
Further, it’s riddled with the kind of typos you wouldn’t expect in a serious economic study, such as referring to the macroeconomic impacts as “marco” factors, reverting back to macro on the next page.
Yeah yeah, I know. Newspapers have typos too. But that kind of mistake wouldn’t be one you would see in a serious, independent economic analysis.
It’s not the only inconsistency either.
Parts of the analysis are written like an academic document, whereas others read as if they were ripped straight from a government press release touting the provincial climate change plan. That plan still has no defined targets, and without targets, how can you possibly assert, or even assess, how effective it will be?
The simple answer — you can’t.
The press release isn’t much better. It also quotes a second study out of the University of Calgary, one where the author said her numbers and comments have been taken out of context. In fact, she wrote a second blog post clarifying her numbers as upper-limit estimates, and not an accurate representation of what the Trudeau carbon tax would actually cost. That’s partly because the calculations weren’t based on the Trudeau plan. They were made prior to the federal government announcing what the plan would be.
I, for one, would actually be interested in a peer-reviewed, well-researched analysis of what the Saskatchewan plan will actually do and what it will actually cost. But we don’t have that. We don’t even know what the targets will look like. (The Premier told reporters that those targets are “coming.”)
What I’m not interested in is a shoddy study that even non-economists can detect as flawed, especially one paid for out of the government’s own budget.
I approached the provincial government with my concerns. It’s a complicated issue, and a response wasn’t ready by the time I had to send today’s paper to press.
But that’s ok, because there’s an element of this I still want to explore myself.
Part two comes tomorrow, with more from me and a response from the folks at the Ministry of Environment.
For now, if you see a study that asserts something to be true that seems ground-breaking or out of place, ask yourself this:
- Who wrote the study?
- Who funded it?
- What questions did they ask and what were they out to prove?
- How did they do it? And
- Does this seem fair?
If not, it’s likely dubious. To see the provincial government pushing something like this is disappointing, but not overly surprising. Of course, the internal documents and analyses always exist, and need to exist, but pushing it as a ‘university analysis’ that proves the government’s point, instead of an internal document designed to inform its own argument, is misleading.
At best, it’s an overzealous attempt to prove one’s own point. At worst, it’s a sad example of political gamesmanship.
Part 2 – published June 30, 2018
Yesterday I took a look at what makes for a good study, and how I approach reporting on such studies.
I indicated that a part two was coming today with responses from the government.
I got my responses.
The Ministry of Environment grouped my questions into five sections, and provided five answers. For some, they addressed my concerns with an explanation.
For others, not so much.
If you missed yesterday, let me recap a bit.
Earlier this week the Ministry of Environment touted a study that seemed to, on the surface, indicate the province’s argument that a carbon tax is ineffective and would hurt Saskatchewan was backed up with scientific evidence.
Intrigued, I took a look, examining in the study what I always look at, namely, who wrote it, who funded it, how it went about finding conclusions and what those conclusions actually were.
There’s a lot of good information in that study, but I disagreed with the claim that it backs up the province’s argument.
That’s for a few reasons, including the fact that the province in fact funded the study. It hasn’t been peer-reviewed, and other experts interviewed by the Leader-Post found it to be lacking.
The study also contradicts itself, stating both that a carbon tax is and isn’t effective.
I’m curious to see what targets the province comes up with, and whether its plan will result in reduced emissions without any economic damage. The plan, as it stands, lacks targets or accountabilities, making it difficult to assess.
Essentially, I’m trying to take a stance on the study without taking a stance on the carbon tax. If the province wants to argue a carbon tax isn’t effective, it needs something to back up its argument.
Unfortunately, while the study does compare a carbon tax to the status quo, in terms of promoting the province’s plan, its deeply flawed and weakens the province’s argument. The study, I should note, was publicly-funded. So this is our tax dollars at work producing a sub-par analysis riddled with inconsistencies, spelling errors and conclusions that don’t have any argumentative or factual basis.
Here’s what I mean by that.
The study asserts the Saskatchewan plan is better without assessing the Saskatchewan plan, or putting it through the same modelling as the federal government’s plan.
The NDP brought this point up too – the study compares the status quo to a carbon tax, and then the government compares its plan to the findings that a carbon tax would be more expensive than the status quo.
This would be like me saying “this study shows that current apples are cheaper than taxed apples. Therefore, my oranges are the best possible solution.”
The province responded to this when I asked:
“This is not a study to determine the “best” approach to addressing climate change; it is a study of the effect of a carbon tax, at the price now enacted by the federal government, on Saskatchewan’s economy.”
Fair. Then don’t use it to argue your plan is better.
But let me quote from the study:
“In 2017, the Government of Saskatchewan proposed the A Made in Saskatchewan Climate Change Strategy. In this strategy, the Government of Saskatchewan builds upon actions we have already taken and introduces measures to strengthen our province and build resilience to climate change. Our plan is bold, broad and made for Saskatchewan. It is the best way for our province to harness our valuable resources while contributing solutions to an issue that affects the entire world”
According to whom?
This is an assertion without a factual or argumentative basis anywhere else in the document.
It’s hard to take something billed as a serious study seriously when it includes assertions without the arguments or stats to back it up.
That’s especially true when the study also says this:
“There is no doubt that carbon tax is an effective way to reduce GHG emissions.”
Yet, later, it says this:
“A carbon tax would make it more difficult for our province to respond effectively to climate change because a simple tax will not result in the innovations required to actually reduce emissions.”
Do you see my confusion?
I brought this up in my questions to the ministry. I asked about the inconsistency in the report. This is the answer I received.
“Some jurisdictions may select a carbon tax as one way to reduce emissions. This research supports that a carbon tax is not the most efficient or cost-effective way to reduce emissions in Saskatchewan.
In government’s June 27 news release, Minister Duncan stated that the federal government has not accounted for energy-intensive, trade-exposed industries that drive Saskatchewan’s economy.
Saskatchewan’s Prairie Resilience strategy includes developing and implementing sector-specific output-based performance standards on large emitting facilities, such as those in oil and gas, and mining. These standards are being developed in consultation with industry throughout 2018 and will recognize actions already taken by industry to reduce emissions. “
I should also note this – the research did not compare a carbon tax to other ways to reduce emissions. If this is the research they’re citing, it certainly doesn’t show what they say it shows.
The study was to take a look at potential carbon tax impacts. Nothing more. So quit it with the false equivalencies.
One of the great ironies of this is the province, in its release, quoted Dr. Jennifer Winter. Winter, from the University of Calgary, provided senate testimony later republished in a blog prior to Trudeau’s carbon tax backstop about potential costs at potential price points.
The province said her numbers prove what they have been saying.
But Winter says that’s not true. Winter has gone on the record in response to others using her numbers that her findings represent an upper bound on potential costs to households, isn’t an exact figure, is based on many assumptions and is not representative of Trudeau’s plan.
The province said it provided a link to her work so people could get more information.
While they cited her figures (without the context), they missed this from her testimony:
“The policy options we have available are pricing, regulations, or a combination of both. Any policy action has costs and benefits, winners and losers, and political consequences. From an economic perspective, we should, where possible, strive for policies that achieve the maximum benefit at the minimum cost. That is why economists are almost universally in favour of emissions pricing via taxes or a cap and trade system. By contrast, political feasibility often depends on the visibility of costs and benefits, which is why regulatory solutions are often preferred due to their less explicit costs.
“A key principle of economic efficiency is treating all individuals and firms the same, so they face the same incentives. Policies such as phasing out coal or the cap on oil sands emissions effectively create a two-tier system, where specific types of economic activity are favoured over others.”
The provincial government is moving forward with a regulatory plan with caps, as opposed to a taxation regime. While Winter does call for flexibility, she also warns that regulatory regimes can be even more expensive.
You might wonder why I care so much. The answer is this – I’m not yet sold on carbon tax. Yet, most studies show it should be effective. I say should, because we’re just getting going on putting these policies in place on a wide scale, and haven’t really tested alternatives.
This study was provincially funded, from our money, and yet it doesn’t prove what the government says it does. If we want to argue our plan is better, we need REAL data that backs that up, not a comparison to nothing. If this study compared one plan to another it would have had way more value.
In terms of advocating for the made-in-Saskatchewan plan, this misses the mark. If the government uses this flawed study to argue its position, no one will take it seriously.
Also, I care because I care about what the government has to say and how it says it.
I’m used to political games and cherry-picked statements.
But when a government can commission a study then insist it says what it doesn’t, and say their argument is backed by science, when they play fast and loose with the facts, I get upset.
As I wrote on my personal Facebook page before delving into this:
“If there’s one thing I hate more than government obfuscation or cherrypicking of data, it’s obfuscation and cherrypicking of data masked as academic research.”
