Do Millennials have no friends?

I recently read an article claiming that 22% of Millennials say they have no friends. And then many other articles with the same figure. This made me feel sad. Some of the articles further distinguished between “close” and “best” friends, so here we’re presumably talking about just any friend of any level at all.

Sure, being a human I have felt peaks and troughs of loneliness over my life-history to date, but I’m not sure I’d cope well if I felt like I had no-one in the ‘friend’ category at all. The thought that nearly a quarter of the young-but-definitely-adult generation feel that way today was quite shocking and depressing to me.

Outside of my personal feelings, it is an increasingly well established fact that loneliness – which I have to imagine strongly associates with having no friends – is not only extremely unpleasant for most folk, but actually harmful to ones physical and mental health. People with stronger social connections may literally live longer. One can go too far with the ‘you may as well take up smoking’ type headlines, but there does seem to be something potentially life-and-death within this subject.

But before getting too upset for the local young adults, I did want to check in on the data itself. Millennials do get a bad rap. Most famously perhaps, we’re all supposed to believe that the reason young people don’t own houses is nothing to do with the fact houses cost an insane amount of money, and everything to do with their high expenditure on avocado toast. Somehow a stereotype seems to have developed in some quarters that the reason not every millennial has a job is because they’re lazy (nothing to do with the supply side of the job market, naturally), they’re selfish, narcissistic and constantly going around maliciously killing various industries and destroying other venerable and much-loved institutions, including DUIs, divorce and porn. After all of that, headlines that involve the word “Millennials” do tend to induce a slight level of skepticism in me.

Reassuringly, it turns out friendship data was from a survey conducted by a reputable enough survey company, YouGov. And they were good enough to publish the full, albeit heavily aggregated, results of the survey itself. OK, in a horrible PDF format, but it didn’t take too long to extract the details of the folk who responded ‘zero’ to the “How many friends do you have?” question in a way conducive to constructing a few breakdowns of these folks below, and satisfying a bit of personal curiosity.

TLDR: Yes, 22% of Millennials did say they had no friends, the highest of all surveyed generations. But it’s not clear to me at all that it’s because they’re Millennials. For example, 27% of black people said the same. And what does ‘friend’ even mean in this survey?

Have some reading time on your hands? Well, in accordance to the guidance given in the original data file, any groups where the number of participants surveyed was less than 50 will not be shown, because these very small samples are considered by YouGov to be statistically unreliable. I will however note what the ‘missing’ categories are, in case it helps clarify who is or isn’t in each category.

Unfortunately there didn’t seem to be a whole lot of other statistical significance info in the data file; no confidence intervals or the like. So it’s not clear to me to what extent small percentage differences should be considered “real”. But they are a reputable enough company who have at least taken the time to re-weight the respondents to represent a base of all US adults and talk about the limitations of too-small samples, so I’m going to go wild and assume that we might care about at least the larger differences.

By generation

Missing categories:
– Gen Z (people born in the year 2000 and later)
– Pre-Silent generation (1927 and earlier)

So this is the data the articles focussed on. Sure enough, Millennials were more likely to report having no friends than the other groups. Are we seeing a uniquely lonely generation? Well, it’s possible. However, to be honest, it’s not possible to tell if Millennials are “special” here from this data.

There are other potential explanations, including that – by definition – each generation here must have been a different age when they were surveyed.

The survey was carried out in 2019, and YouGov here defines a Millennial as being someone born between 1982 and 1999 (the exact definition varies depending on who you ask – so always best to check the data source!). So these folk were between 20 and 37 years old. Compare that to the seemingly more friend-enabled ‘Silent Generation’, who in this analysis would have been between 74 and 91 years old.

Perhaps – and I’m not presenting any evidence here to suggest you should believe this over any other hypothesis – it’s just normal that older people are less likely to report having no friends than younger people.

Are changes in the number of people reporting having no friends really a ‘cohort effect’, which is what a lot of the headlines about this survey imply? More data-digging would be needed to determine that, as opposed to whether this is, for instance, an aging effect.

An aging effect is a change in variable values which occurs among all cohorts independently of time period, as each cohort grows older.

A cohort effect is a change which characterizes populations born at a particular point in time, but which is independent of the process of aging.

A period effect is a change which occurs at a particular time, affecting all age groups and cohorts uniformly.

Source: Distinguishing aging, period and cohort effects in longitudinal studies of elderly populations

By gender

Not a whole lot to say here. Males seem slightly more likely to report having no friends than females, but the gender differences are much less than between generations. Without knowing the confidence intervals of the responses it’s also hard to know how significant these differences are.

By region

Again, only relatively small differences are seen when the respondents are split up into what region of the US they live in.

By race

OK, here are some large differences again!

The difference between Black and White respondents – 16 percentage points – is actually the same level of difference as between Millennials and the generation with the very lowest % of people reporting having no friends.

It’s interesting that many of the articles reporting on this survey focused on the generation as opposed to the race. There may be a legitimate reason why, but it’s not self-evident to me. It seems if we’re worried that Millennials may be lonely, the same concern might be needed for non-white folk too.

By education level

The big differences keep on coming! At a glance, this looks like a strong positive link between having higher levels of education and having a friend.

By income

Can money buy you friends? Traditionally we tend to say no. But having a higher income sure does seem to reduce the likelihood of you feeling like you have no friends at all.

(I am not sure exactly what income was asked for – from the values, I’d assume this is something like annual household income, but should verify before stating that to be the case!)

By urbanity of area lived in

The traditionalist’s view that city living includes being surrounded by hordes of other people, but feeling personally lonely, seems directionally borne out in these results. Univariately at least, urban dwellers are more likely to report having no friends.

By marital status

Missing categories
– Civil partnership
– In a relationship, not living together
– Separated
– Other
– Prefer not to say

So is getting married the end of all friendships, with the happy couple dumping their pals so as to get on with journeying their way through mortgages, careers and other misc adulting? Seemingly not. People who are, or were, once married were a lot less likely to report having no friends than those who were not.

By whether being a parent or guardian of any children

Missing categories:
– Don’t know / Prefer not to say

Likewise, parenting doesn’t appear to remove your entire friendship circle (or at least if it does, maybe you end up replacing them all with new parenty-friends over the years). Having kids, especially ones who are now adults, seems to make you less likely to report having no friends.

So, to summarise:

Are millenials more likely than other well-represented generations to report having no friends in this survey? Yes, they are. But we need more data to understand if this is a “Millennial generation” phenomenon vs a “being in your 20-30s” phenomenon. After all, a 40-year-old has had longer to find a friend out there in the wilderness!

Millennials aren’t the only group to report having no friends

Applying the same analysis to the rest of the survey results shows the existence of several other ‘risk factors’ for reporting no friends.

Excluding the variables that show only a couple of percentage point differences between categories, these added-risk groups include:

  1. not being white
  2. having a low level of education
  3. having a low income
  4. living in an urban area
  5. not being or having been married
  6. not having children, especially of adult age

So there’s potential for confounding here. Let’s imagine a world where being born in the 1980-90s did not actually affect your friend count. If any of the above factors are over-represented in Millennials in comparison to other groups, we could still see the same overall effect.

I don’t intend to dig up all the stats correlating age with the 6 bullet points above for this post, but even a modicum of websearching reveals sensible-sounding sources with claims like:

Relative to members of earlier generations, millennials are more racially diverse, more educated, and more likely to have deferred marriage; these comparisons are continuations of longer-run trends in the population. Millennials are less well off than members of earlier generations when they were young, with lower earnings, fewer assets, and less wealth.

Source: Are Millennials Different?

So that’s risk factors 1, 3 and 5 confounding away, mitigated perhaps by reverse-risk factor 2.

For reasons of age alone, it’s unlikely many millennials have adult children yet, and they don’t seem to be in a particular hurry to have any children at all. All this, whilst enjoying urban life, if they’re able to.

So, how to differentiate the root cause? Well, with the level of data published – and don’t get me wrong YouGov, I’m grateful any was! – it’s not really possible to. A more complex analysis using data at the individual person level, allowing us to look at the effect of generation controlling for other variables, and ideally comparing also with previous time series, would be the obvious start. Whilst that type of observational study is usually not able to prove causation beyond doubt, we might get closer towards understanding the likely fundamentals.

It didn’t escape my notice – although of course I am not going to prove this to be true here – that many of the higher risk groups are those that society has often appeared to value less highly – poor people, non-white people, less-educated people, the unmarried; think of the sections of society sometimes defined by or over-represented in low “socioeconomic status” groups. Perhaps policies designed to assist those our current social structure apparently does not help so much may have a bonus side effect in the realm of strengthening social connections – and all the health and life benefits that go alongside that.

What is a friend anyway?

After discussing this statistic with a friend (see what I did there? True story though, and thank you, correspondent) I was reminded that the definition of a friend is itself rather woolly. One person’s friend is another person’s acquaintance, colleague or window-cleaner.

“Friendship is difficult to describe,” said Alexander Nehamas, a professor of philosophy at Princeton, who in his latest book, “On Friendship,” spends almost 300 pages trying to do just that. 

Source: Do Your Friends Actually Like You?

So another scenario in which millennials could be more likely than others to report having no friends – even if in reality they had the same level of social connections – would be if they define ‘friend’ differently, especially more stringently, than others. Some more qual-side digging into whether disparate generations define friends differently to each other would be useful to look into that hypothesis.

Related to definitions, there could also be something in how the question was asked. I did not see the original survey, but the Yougov write-up implies the question asked was

“Excluding your partner and any family members, how many of each of the following do you have?”

followed by a list comprising of “acquaintances”, “friends”, “close friends” and “best friends”.

Most of the articles focus only on the “friends” result, where the 22% zero figure is seen. OK, fine – taken in isolation, “close friends” and “best friends” sound like subsets of friends, right?

But if presented with each of those options on the same screen, perhaps respondents might categorise each person they know into an exclusive category. So if you pop your pal Jimmy into the “best friends” box, perhaps you don’t also add him into the basic “friends” box.

Perhaps you feel close to all your friends, so have 10 close friends but no “non-close” friends, except those you categorise as acquaintances. In this way, a very close-friend-fulfilled person might be included in the no friends bucket when analysed one question at a time.

If this was the case, whilst the articles aren’t reporting anything untrue; it may be misleading when taken in isolation. All this is only a theory of course, as I haven’t seen the precise flow of the original survey. Perhaps the questions were asked in a a way less likely to cause this issue. But we can tell that the 4 categories aren’t being used entirely as subsets of each other, as 25% of Millennials report having no acquaintances, vs only 22% having no friends; i.e. more Millennials have friends than acquaintances.

(Out of interest, 27% of Millennials reported no close friends, and 30% having no best friends – and yes, pedants, some people did report having more than one ‘best’ friend).

Anyhow, wild hypothesising aside: More knowledge does give us a higher chance of developing effective remedies, if remedies are indeed needed. Which they likely are, in my opinion, no matter what the precise count of Millennials involved is or why, given the dramatic impact of loneliness on people’s lives. This is an important line of research that should likely be pursued with the full resources and rigour that serious issues around health and well-being deserve.

But in the mean time, associations between loneliness and all sorts of negative health and well-being effects have been repeatedly demonstrated. So if we’ve societal levers to pull, or personal practices to enact that have the potential to reduce any level of friendlessness, let’s get on and do it.

What people claim to believe: Hillary Clinton edition

Back to political opinion polls today I’m afraid. Yep, the UK’s Brexit is all done and dusted (haha) but now our overseas friends seem to be facing what might be an even more unlikely choice in the grand US presidential election 2016.

Luckily, the pollsters are on hand to guide us through the inner minds and intentions of the voters-to-be. At last glance, it was looking pretty good for a Clinton victory -although, be not complacent ye Democrats, given the lack of success in the field of polling with regards to the afore-mentioned Brexit or perhaps the 2015 General Election here in the UK.

Below is perhaps my favourite most terrifying poll of recent times. It’s a recent poll carried out by the organisation “Public Policy Polling” concerning residents of the state of Florida. As usual, they asked several questions about the respondents’ characteristics and viewpoints, which lets us divide up the responses into those coming from Clinton supporters vs those coming from Trump supporters.

There are many insidious facts one could elucidate here on both sides, but given that at the moment the main polls are very in favour of a Clinton win (but see previous comment re complacency…), let’s pick out some that might hold relevance in a world where Clinton semi-landslides to victory.

Firstly, it shouldn’t particularly matter, but one can’t help but notice that Clinton is of the female persuasion. But, hey, rational voters look at policies, competence, experience or similar attributes, so a basic demographic fact alone doesn’t matter, right?

Wrong: the survey shows that just 69% of all respondents thought that gender didn’t make a difference. And, predictably, twice as many thought that the US would be better off with a male president than those who thought it would be better of with a female president. The effect is notably strongest within Trump supporters, where nearly 20x the proportion of people think the US would be better with a male president than with a female one.


Now, I can imagine some kind of halo effect where it’s hard for people to totally differentiate “my favourite candidate is a man and I can’t imagine having a favourite candidate that is not like him” from “my favourite candidate is a man but the fact he happens to be a man is incidental”.

But that nearly 40% of Trump supporters here claim that generically the president should be a man (implying that if it was Ms Trump vs Mr Clinton, they might vote differently), it seems potentially a stronger signal of inequality than that, especially when compared to the lower bias between Clinton supporters and preferring a woman – which is equally as illogical, but at least has a lower incidence. We can note also a pro-male bias in the “not sure” population too.

Of course we don’t actually have an example of what the US is like when it has a female president, because none of the 43 serving presidents to date have been women.

But we do know part of what Hilary Clinton is already presidentially responsible for apparently. “Coincidentally” (hmm…) her husband was one of the previous 43 male presidents, and apparently the majority of Trump supporters think it’s perfectly right to hold her responsible for his “behaviour”.

Yep, anything he did, for good or bad (which, let’s face it, is probably biased towards the bad for those people who support the opposing party and/or don’t appreciate cheating spouses) is in some sense his wife’s fault, for the Trumpians.


But if she’s so obviously bad, then why does she actually poll quite well, at the time of writing? Well, of course there can be only one reason. The whole election is a fraud. And given we haven’t actually had the election yet, I guess the allegation must also entail that poll respondents are also lying about their intentions, and/or that all the publishers of polls are equally as corrupt as the electoral system of the US.


Yes, THREE-QUARTERS of Trump supporters polled here apparently believe that if, as seems quite likely, Clinton wins then it can only be because the election was rigged. The whole democratic process is a sham. The US has fallen prey to semi-visible forces of uber-powerful corruption. We should presumably therefore ignore the result and give Trump the golden throne (to fit inside his golden house). Choice of winner aside, this is a pretty scary indictment on the respect that citizens feel for their own democratic system. This is not to say whether they are right or wrong to feel this way; to us Brits, I think it sometimes seems that in the US money has even greater hold over some theoretically democratic outcomes in the US than it does over here – but that so many have so little regard for the system is surely…a concern.

But wait, it’s not just that she may hypothetically commit electoral fraud in the near future. She has apparently already committed crimes serious enough that she should already be locked up in prison.


Over EIGHTY PERCENT of Trump supporters polled here think she should literally go to prison; and this isn’t predicated on her winning. Well, there’s no shortage of bad things that can be laid at her door I’m sure, she has after all been serving at a high level of politics for a while already and, without being an expert, it seems like there are many serious allegations that people lay at the Clintons’ feet. But it’s perhaps quite surprising that the large majority of her opponent’s supporters want to throw someone who is likely to be their next president in jail. I don’t think even the Blair war-crimes movement ever got quite that far!

Unless…well. I’m only sad they didn’t ask the same question about Trump. Perhaps we could be more at ease if at least the same proportion of people thought he should be locked up. An oft overlooked fact is that analysis is often meaningless without some sort of carefully-chosen comparison. Perhaps there’s a baseline figure of people that think any given prominent politician should be jailed (but I’ve not seen research on that).

It’s hard to imagine though that the fact Trump has himself actually appeared to threaten her with jail doesn’t play some role here with his supporters though. It is apparently unprecedented for a major party nominee to have said publicly that his opponent should be jailed – but say it he did, most famously during their second presidential debate. As the Guardian reports:

Trump, embracing the spirit of the “lock her up” mob chants at his rallies, threatened: “If I win I am going to instruct my attorney general to get a special prosecutor to look into your situation – there has never been so many lies and so much deception,” he threatened.

Clinton said it was “awfully good” that someone with the temperament of Trump was not in charge of the law in the country, provoking another Trump jab: “Because you’d be in jail.”

Eric Holder, who once was the US attorney general, didn’t really seem to like that plan.

So we’ve established that in the eyes of the average Florida Trump supporter polled here that if Clinton wins then the whole shebang was fraudulent, she already should have been locked up in prison, and, besides, the fact that she’s a women should probably ban her from applying to the office of the president in the first place. That’s a strong indictment. But, of course, there’s another level to explore.

Is Hillary Clinton a malevolent paranormal entity, intent on destroying humankind?


Erm…2 out of every 5 Trump supporters here think yes, she definitely is an actual demon. And the majority aren’t sure that she is not an actual demon.

Even only just over 50% of the “not sure” supporters are also sure she’s not an actual demon. It’s also entertaining to contemplate the c. 10% of her supports that think she might be demonic yet still fancy her as president.

The lower figures might be down to some variant of the excellent StarSlateCodex’s concept of the “Lizardman’s Constant” which can perhaps be summed up as there’s a lower bound % of people who will believe, or claim to believe, any polled sentiment.

But there they benchmark that at around 4%, and ten times that proportion of Trump supporters here respond that they are certain that Clinton is a literal demon. There are many ways to introduce biases that lead to this sort of result, which StarSlateCodex does go over. But 40% is…big…if this poll is even remotely respectable.

So, where has this idea that she’s a demon come from? Have Trump supporters as a collective seen some special evidence that proves this must be true, that somehow the rest of us have overlooked? Surely each individual doesn’t randomly become subject to these thoughts which even believers would probably term an unusual state of affairs -is there no smoke without fire? (pun intended)

Well, perhaps it has something to do with a subset of famous-enough people have stated that she is.

Trump himself did refer to her as a devil, although in fairness that just maybe possibly might be an unfortunate turn of phrase, if we want to be charitable. After all, to his credit, evidence suggests he’s not great at following a script (or at least not one you’d imagine a typical political spinner would write).

Perhaps more pertinently, for certain a certain subsection of viewers anyway, is presenter Alex Jones of “Infowars” fame (a website that apparently gets more monthly visitors than e.g. the Economist or Newsweek), he who Trump says of “your reputation is amazing…I will not let you down”, who did go on a bit of a rant on this subject.

MediaMatters have kindly transcribed:

She is an abject, psychopathic, demon from Hell that as soon as she gets into power is going to try to destroy the planet. I’m sure of that, and people around her say she’s so dark now, and so evil, and so possessed that they are having nightmares, they’re freaking out… I mean this woman is dangerous, ladies and gentleman. I’m telling you, she is a demon. This is Biblical.

There’s so much more if you’re into that sort of stuff; see it all on this video, including the physical evidence he presents of Clinton’s demonness (spoiler alert: she smells bad, and Obama is obviously one too because sometimes flies land on him).

Unfortunately I’m not aware of time series data on perception of Clinton’s level of demonicness – so I’m afraid there’s no temporal analysis to present on causal factors here.

At first glance some of this might seem kind of amusing in a macabre way – especially to us foreigners for whom the local political process is hugely less pleasant or equitable than it should be, but it doesn’t usually come with claims of supernatural possession. But the outcome may not be so funny. In the likely (but not certain) event that Clinton wins, Florida at least seems to have a significant bunch of people who think the whole debacle was rigged, and Clinton should have a gender change, an exorcism and a long spell in jail before even being considered for for the presidency.

Update 1: this sort of stuff probably doesn’t help matters – from former Congressmen / Radio host Joe Walsh:

Update 2: the polls are a lot closer now then they were when I started writing.

Is the EU referendum actually a great conspiracy?

(Sorry to anyone bored by the great/hideous Brexit referendum – this is the last post on the topic, well, at least until the event actually happens 🙂 )

Today is the day!  All us UK citizens can cast our direct-democracy vote as to whether the UK should remain in the EU, or say goodbye. It’s been a long, torrid, at times revolting, journey in terms of output from the campaigners, politicians and media. “It is as though the sewers have burst”, said Nick Cohen in the Observer, somewhat accurately. But the vote is today and it’ll therefore all be over soon.

Or will it? Yougov have surveyed on many, many EU referendumy topics. One of the latest included questioning respondents on various conspiracy-esque statements about the result of the referendum. I don’t use the word “conspiracy” in a necessarily derogatory tone – some perceived “conspiracies” turn out to be true, although many do not.

Anyway, here were the statements offered up to the public to pronounce on whether they thought they were probably true, probably false or don’t know.

  • There are plans for further EU integration and enlargement that the EU are deliberately not announcing till after the referendum
  • The BBC & ITN are not commissioning an exit poll in order to allow the vote to be fixed without anyone telling
  • MI5 is working with the UK government to try and stop Britain leaving the EU
  • It is likely that the EU referendum will be rigged

I have listed them in my perception of order of seriousness, although several are open to interpretation regarding the scope and intentionality they imply. The first just relates to the timing of announcing EU events, the last implies the literal undermining of the entire democratic process, implying a pointless referendum beholden to corrupt, criminal actors.

But what did the respondents think of these? Did anyone seriously think that MI5 spies are secretly influencing the result? (*) That the whole referendum is a fraudulent scam?

(*) Well, it’s not quite MI5, but when the Conservative peer Baroness Warsi recently changed her view from Leave to Remain, there were people suggesting she was a  secret Remain campaign plant all this time. Amongst other far more horrific diatribes that I am reluctant to reproduce on this site.

Well, it turns out the answer is yes, a fair amount of people do agree with these statements. Please click through and interact with the visualisation below in order to see the proportion of people agreeing with each statement, with the ability to break it down by age, gender, social grade, region, which political party they voted for in the 2015 general election, and – perhaps most interestingly -how they reported that they intend to vote for in the EU referendum itself: leave vs remain.

EU referendum conspiracy theory poll2


A few things I noticed:

There’s a sizeable amount of people that agree with every one of those statements. That’s not to say that they are the same single cohort of people in each case, as the data is too high level to determine that, but every statement has at least 15% of people in favour. There’s not one statement that over half the surveyed people thought was probably false. Not one.

To take perhaps the most dramatic one – nearly a third of the surveyed population think that it’s likely that the EU referendum will be rigged. If this implies “direct” rigging i.e. fiddling with the results, then this is quite a terrifying indictment on our view of the legitimacy of our democratic process.

Sidenote: There does seem to be a movement to “bring your own pen” to the voting stations today, under the premise than the pencils that poll booths traditionally offer leave marks on the ballot papers that could be easily erased and replaced. Although this seems like one of the most annoying and time consuming ways I could imagine of fixing an election result! If you’re going to believe in an over-arching conspiracy here, then I suspect MI5 could have far more efficient methods…

When splitting by demographics and behaviour, clear differences emerge. Flicking through the interactive version will show you the full details not represented in the below text – but in summary, for most statements:

  • A fairly similar proportion of females and males believe they are true. But for those that don’t, females are more likely to say they don’t know whereas males are more likely to go for probably false.
  • Those of social grades ABC1 are generally less likely to think any of the statements are probably true than C2DE, and more likely to think they’re probably false.
  • There is a strong difference in the beliefs of the voters based on whether they’re likely to vote for Leave or Remain. Without exception, the Leavers are more likely to think the statements are probably true than the Remainers.  The proportion of Leavers who think the referendum is likely rigged is over four times the proportion of Remainers.

    EU referendum conspiracy theory poll

  • Digging down deeper into (the somewhat correlated, but not fully so) variable of which political party they supported in the 2015 election, there is one hugely obvious outlier. Those who voted UKIP are way more likely to agree with the statements than others, particularly regarding whether the EU referendum will be rigged. A majority, nearly two thirds, of UKIP voters believe this to be true, in comparison to between 14 and 23% of voters for other parties.

So, what does this mean?

Well, it shows a distinct lack of faith in the system set up for this referendum and trust in the “powers that be” – which is perhaps somewhat understandable, considering the ways the various campaigns have been run.

At first glance, the sheer level of disbelief in the overall integrity of the system seems a notable unhealthy sign of the times though – although I would like to see similar stats taken over previous years in order to determine whether the figure of 28% believing the referendum will be rigged is “normal” for every year. If so, it could certainly explain the non-amazing turnout the UK generally sees in elections.

…except that there’s a curious interaction regarding voting intention, political party and turnout. In a previous post here, we saw that UKIP supporters are one of the subsections of society that appear to be most likely to say that they will turn up and participate in the referendum. However, this is also the segment that is by far the most sceptical of the result being legitimate. UKIP was likely also one of the driving forces that led towards the referendum being called in the first place: if there was no visible block of desire to leave the EU, an issue that UKIP was originally set up to dedicate itself to, then there would have been no reason for a referendum.

That’s not to say other political parties don’t have members with anti-EU views in them, who are individually in places where they might be expected to have a higher influence in political shenanigans than the average UKIP candidate. Two of the highest profile Leave campaigners, Boris Johnson and Michael Gove, are both high-up members of the Conservative party.

But, simplistically, the people who most demanded the referendum and are most likely to go and vote in it also seem to be the people who are least likely to believe its results. Are we seeing a political form of Pascal’s wager?!

It would also suggest that no matter what the result is, the debate will be far from over. Particularly if the result goes to remain, it seems like nearly half of those voting to leave may feel that it has been rigged (of course people are likely forgive rigging more if it produces the answer they want). And even if it goes to Leave, one in ten Remainers are seemingly sceptical of its legitimacy already, which is a sizeable number of people who, even without the psychology surrounding losing a vote to those with different beliefs, believe that the entire system is invalid.

So recently we have learned:

Hey, it’s almost as if it isn’t really the time or place for such a consequential question about the future of the UK to be determined in this manner. Is it too late to call the whole thing off? (answer: yes, I guess it is).

The EU referendum: do voters understand what they’re voting on?

The UK’s EU referendum is now less than a week away. We’re each going to individually vote on something that could dramatically affect the future of our lives and even the structure of society the UK, so it’s a potentially important one. Recent sick tragedies have added to the mess that is the provably wrong claims from the Leave camp, and the responses that seem largely ineffective, and possibly not much less biased, from the Remain camp.

Clear-cut facts seem in short supply within the public consciousness; and yet surely one of the assumptions behind the validity of a direct-democracy referendum is that those who are enfranchised to participate in the decision have something akin to “perfect knowledge” about what they are to vote on. Or at least pretty-good knowledge, if we want to grant some leniency.

If one knows almost nothing, or, worse yet, holds false beliefs around the issues to be balloted on, then to choose the option most in-line to the priorities of the voter themself, irrespective of what they are, becomes a matter of chance, a dangerous reliance on instinct, or a matter of fallible heuristics. Logically, one might assume then that the voters with the most true knowledge about the relevant issues would be in a position to make “better” decisions.

So, do we, the British population have a decent knowledge of the key issues that apparently govern the EU battleground? Battles are being fought between camps on  economics, immigration, legislative power and democratic credentials.There have been various polls on this issue, trying to establish the knowledge of the electorate. Below I have chosen one from Ipsos Mori, who asked various subsections of the eligible voting population to give their views on several “EU facts”.

One of the subsections they divided on was the self-reported response as to whether the respondent was thinking to vote for Leave, Remain or was currently undecided. This opens up the possibility I wanted to investigate: is one side more well informed about the relevant facts than the other? One might – arguably – then risk a claim that this is the side that may be executing more effectively with regard to “data driven decision making”, being technically more “qualified” to participate in decisions relating to matters of this domain.

This is admittedly arguable for several reasons. Firstly, the precise choice of the questions may not be an accurate reflection of the points of highest relevance to this decision. Ipsos Mori could not ask about every possible EU fact, so there is a possible selection bias here. However, they did ask questions on most of the topics that each side specifically campaigns on, so it seems in line with what the campaigners think are the priorities driving people’s decisions.

Another question is, with the confusing contradictory mess of the claims being put out there, is it really safe to say there is a “correct” answer? For some potential questions, my view is no: establishing a true net economic value of the EU seems beyond us at the moment for instance. However, Ipsos Mori did at least work with an external “fact checking charity”, Full Fact, to try and establish a set of questions and respective answers that could be held as independently true.

Unfortunately, Ipsos seem to have decided to release the detailed results of the survey (which is nice) in a 500+ page PDF (which is not). So to get to the bottom of my question, it seemed appropriate to extract and visualise some of this data.

Let’s get to it!


Please tell me whether you think the following statement is true or false: The UK annually pays more into the EU’s budget than it gets back

Most of the respondents were correct to imagine that the UK pays more in to the EU budget than it receives back (directly back is what is implied, I believe, more on this later). 90% of Leave fans believed this, although only just over half of Remain and Undecideds chose the correct option. Both of those were more uncertain, although a quarter of Remain campaigners incorrectly though we received back more than we put in.

Correct answer according to Ipsos Mori: TRUE

Sheet 1

There is a widely-known argument that the financial benefits of being in the EU are nonetheless  net-positive due to things like the increase ease of business, investment and so on. The StrongerIn campaign writes:

And we get out more than we put in. Our annual contribution is equivalent to £340 for each household and yet the CBI says that all the trade, investment, jobs and lower prices that come from our economic partnership with Europe is worth £3000 per year to every household.

The whole financial aspect of the decision is one heavily campaigned on, very selectively, by the different sides to the point where they seem to contradict each other directly (not rare). It’s possible in the resulting confusion that some respondents may have included those non-direct factors, which could make the statement true.  It might have been helpful if the question had made it very clear that it was about direct transfers of money with no external factors.

Winner: Leave

What proportion of Child Benefit claims awarded in the UK do you think children living outside the UK in other countries in the European Economic Area (EEA)?

Organisations such as MigrationWatch, and the obvious media outlets that like to cause drama with such figures, have stated that the UK is paying a pile of expensive-sounding child benefits to children that live outside the UK, in the EU. It’s true that this, in accordance with the current law, is happening. But what proportion of child benefit is actually going abroad like that? Is it a worrying amount? (if one could set a mark as to when it would be worrying…).

Correct answer according to Ipsos Mori: 0.3%

Sheet 1

OK, we’re all way out here! Only 11% of both the Leave and Remain camps got this right. Leave were more likely to estimate stupendously high amounts. Almost half of Leave though it was at least 13%, which would overstate reality by 43x  (and 20% though it was nearly a third, a whole two orders of magnitude higher than real life.).

That’s not to say the Remainers were correct,  over half still over-estimated it to various degrees, and c. 10% understated it.

Winner: Remain.

In 2014, international investment into the UK was £1,034bn. To the best of your knowledge, what share of this total amount do you think comes from businesses based in the following countries or regions?

Part of the Remain case for remaining in the EU is the supposed positive effects it has on investment in the UK. A letter from a bunch of business folk publicised by the SrongerIn campaign says:

…almost three-quarters of foreign investors cite access to the EU’s single market as a key reason for their investment in Britain.

The Vote Leave campaign disagrees on the significance:

Trade, investment and jobs will benefit if we Vote Leave… Today the USA is a more important source of investment in the UK than the EU is.

This “share of investment” is not the only metric of significance to this discussion, but it is relevant. Does the UK get a lot of investment from EU, or is it mere pennies? How well do we know where investment comes from today?

Correct answer according to Ipsos Mori: The EU provided 48% of international investment into the UK in 2014.

Sheet 2

All groups very much underestimate the percentage of international investment that comes from the EU. The Leavers at the most extreme, with a median response of 28% vs the Remainers 35%. In both cases this seems largely down to wildly overestimating the amount of investment that comes from China.

Winner: Remain


To the best of your knowledge, what share of this budget do you think was spent on staff, administration and maintenance of buildings?

We accept that for the EU to exist, it has to have a budget, that has to be paid for by those within it (and arguably a few of those outside of it, but that’s a different story). But is it spent in way likely to make effective impact, or does the majority of it go on bureaucratic administration tasks and staff costs?

Correct answer according to Ipsos Mori: 6%

Sheet 2

Ha, way out, on both sides. Leave people are most inaccurate, thinking that the proportion of EU budget going on admin, staff and buildings as actually 5x larger than it is. Remain aren’t that much better though, estimating it to be over 3x reality.

Winner: Remain

Please identify the top 3 contributors to the EU budget in 2014

Respondents were then given a list of 10 countries and asked to identify which were the top 3, in descending order, in terms of contribution to the EU budget – i.e. what was the direct financial cost of the EU to them.

Correct answer according to Ipsos Mori: 

  1. Germany
  2. France
  3. Italy


Sheet 4

Second most:

Sheet 4

Third most:

Sheet 4

Well, both Leave and Remain did better than half marks when stating which country made the highest contribution to the EU budget – Germany. But identifying #2 and #3 were trickier; no subpopulation got even half marks on identifying the correct answer.

Of course, probably the most relevant datapoint driving people’s voting decisions is where voters think the UK sits in the ranking of budget pay-ins. It isn’t actually in the top 3 (it is in fourth place, after Italy). However most respondents clearly thought it did feature in the top 3 contributors. Leave were particularly bad for this, with nearly a third thinking it was the single top contributor, and around 90% convinced it was in the top 3. The figures for Remain don’t show a huge pile of knowledge though – 17% and 80% respectively.

Winner: Remain

Please identity the three which received the most from the EU in 2014.

The above covers putting money into the budget, but part of what the EU does is give money back to countries directly, for example to support farming or development of the more deprived areas of a country. So, same 10 countries, can we identify those the top three in terms of receiving money directly from the EU budget?

Correct answer according to Ipsos Mori: 

  1. Poland
  2. France
  3. Spain

Top:Sheet 4

Second most:Sheet 4

Third most:Sheet 4

Hmm, we’re even worse at understanding the money flowing back into EU countries! Around half of all populations got that Poland receives the most, OK. But after that the uncertainty was huge, with no more than 1 in 5 people of any sub-population coalescing around any answer, right or not.

Focusing on the UK, about 9% of Leavers thought it was somewhere in the top 3 recipients, whereas the Remainers were much more wrong about this, with 22% claiming the UK was in that list.

Winner: Leave


Please tell me whether you think the following statement is true or false: The members of the European Parliament (MEPs) are directly elected by the citizens of each member state they represent

MEPs are our representatives in Europe, and yes, they are elected by us. The last election was in 2014, although with a pathetic turnout of 34% it does sound like the majority of Britain didn’t notice. But do we at least know these people for whom we should have voted for 2 years ago exist?

Correct answer according to Ipsos Mori: Yes

Sheet 1

Umm…not really. At least a little over half of Leave and nearly two thirds of Remain knew that they elect MEPs, but that still leaves a highly significant number of people who either are convinced that MEPs are unelected, or don’t know. The next chance to elect UK MEPs is likely to be in 2019, so let’s hope we can spread the word before then.

Winner: Remain (but not by a lot)

Laws and regulations

Which of the following, if any, are laws or restrictions that are in place, due to be put in place, or are suggested by the EU for implementation in the UK?

Ah, the EU laws craziness! Did you know, Europe bans us from <<insert anything fun>> and makes us do <<insert anything miserable>>? Well, in honesty, it does have some influence on what will later be entered into British law.

Below are a list of a few fun potential legislative bits and pieces. Which ones are actually true and somehow related to the EU? As there are quite a few of them, the answers according to Ipsos Mori are inline.

Sheet 3

Actually, we all did better than I expected. Only 8% of Leavers thought we’d have to rename our sausages as “emulsified high fat offal tubes”, which funnily enough the EU hasn’t made us do. Maybe we should anyway. Sausages aren’t that great for you.

Perhaps more interestingly, most of us don’t realise the restrictions that the EU has influenced us towards – although the list is perhaps “summarising” it a bit. The classic “Bendy Bananas Ban” has been categorised here as true (which only 35% of Leavers thought was the case, vs an even worse 15% of Remainers).

You’ll no doubt will be amazed to hear that the law doesn’t actually read “you can’t have bananas that are too bendy”. But it does actually come from somewhere in terms of real legislation. To be exact (brace yourself for excitement): the COMMISSION IMPLEMENTING REGULATION (EU) No 1333/2011.

It states that:

…subject to the special provisions for each class and the tolerances allowed, the bananas must be…free from malformation or abnormal curvature of the fingers…

But that’s only really for “top class” bananas. Go for a class 2 and you can expect that:

The following defects of the fingers are allowed, provided the bananas retain their essential characteristics as regards quality, keeping quality and presentation:
— defects of shape,
— skin defects due to scraping, rubbing or other causes, provided that the total area affected does not cover more than 4 cm-sq of the surface of the finger.

So if you get an abnormally curved top class banana, the EU has let you down. However one measures that.

Winner: 5:3 to Remain (although there’s an obvious pattern that drives this results: Leave are always more likely to think any law is made by EU, whereas Remain don’t think any law is – so Remain are lucky there are more false statements than true statements really).

To the best of your knowledge, which of these laws or taxes in force in the UK are as a result of EU regulations?

And now for current laws. Again,  the answers according to Ipsos Mori are inline.

Sheet 3

Hmm…we’re less good at knowing the truth of this one, whether Leave or Remain. In fact in some instances the results are strikingly similar between groups. Around 60% of both Leave and Remain know that EU regulations surround the cap in working hours (although actually there are exemptions for certain types of jobs). But only 23% of each side understand that 2 year guarantees are a result of such regulations.

Believing that the national living wage is a result of EU regulations is similarly thought true by 19% of both populations, even though it’s false. All in all, the differences between Leave and Remain are probably lower than the general level of ignorance on this topic.


Winner: 5:3 to Leave, by my count, although some questions are super-close.

Which of the following, if any, do you think are areas where only the EU has power to pass rules, and not individual EU countries?

We’ve covered which existing or proposed regulations and laws are influenced by the EU above – but what about topics where on the whole only the EU has the power to legislate?

Answers according to Ipsos Mori are inline.

Sheet 3

Hey, we’re not too shoddy on this one compared to some of the other questions. Both Leave and Remain beat 50% on knowing which domains were EU-regulated, and likewise both sides did even better at knowing which ones were done so domestically.

Again, some of the differences in responses between group were pretty small. The most notable ones were perhaps that Leave were 8 percentage points better at knowing that the EU has the power to rule on fishing industry controls, whereas the Remainers were better at knowing that the EU does not control laws around sentences for crimes committed by non-British nationals by 7 pp.

Winner: 4:2 to Remain (again, differences between groups are very small in some cases).


Another hot topic in the debate; an argument that, at the most despicable end of the Leave campaigners boils down to “if we remain in the EU then the UK will be overrun with nasty foreigners who just don’t deserve all the good things we have”. Of course many Leavers are far less obnoxious in their views, and may have more benign concerns around resourcing and space. The Remainers, depending on their views, might pursue the argument that immigration is net benefit to the UK (or at least not net detriment), the more ethical option, or that leaving the EU is not likely to make so much difference anyway.

But are we deriving  our viewpoints with accurate knowledge as to the incidence of migration into the UK?

Out of every 100 residents in the UK, about how many do you think were born in an EU member state other than the UK?

Correct answer according to Ipsos Mori: 5%

Sheet 2

Looks like the median respondent is way out again: all subpopulations over-estimate the percentage dramatically. Leave produces the most out-there answer, thinking one in five UK residents where born in an EU country other than the UK; 400% of the real value.

Remain do better, but still come up with a median answer that is double that of reality.

Winner: Remain


Horray, we’re done. Did we learn anything? Well, when totting up the scores the final Leave vs Remain results by my slightly rough scoring method above are:

  • Leave: 3
  • Remain: 8

Overall winner: Remain

So, can we go so far as to say that well-informed voters are more likely to make the choice to Remain?  And hence, if we assume a perfect electorate should have perfect knowledge, then is Remain the correct way to swing?

Well, it is surely the case that the Remain voters were more likely to be a bit more accurate as a population in most of the questions above by my measure, but that conclusion is still rather a strong one to draw.

What really shows through here is the general level of ignorance in all populations; whilst it would be nice to say that 100% of Remainers got things right and 100% of Leavers got things wrong and hence Remain is the only decision we could say was based on evidence, the reality was far more mixed. There were plenty of questions where the majority of both groups got it wrong.

This is quite concerning if one has an ambition that the results of a referendum are predicated on voters basing their choice on some semblance of the reality of the present or potential future. In fact, I’ve had Brennan’s book on ‘The Ethics of Voting‘ on my to-read list for a while, and I’m now a little scared to read it in case it makes me decide that Churchill was actually wrong to imagine that democracy was even the least worst form of Government! Perhaps we are simply not yet in a place where it makes sense to hold a referendum on this topic, although there is certainly no stopping it now.

It’s also apparent that there are voters on each side that hold their opinions “despite” what they think they know about certain domains. That is to say: we can infer that nearly 1 in 5 of the Remain voters are committed to remain, despite the fact that they (incorrectly) think the UK pays the highest amount of the EU budget, and/or (incorrectly) think that the proportion of EU born people living in the UK is actually twice as high as it really is. Although the data is not available in a granular enough fashion to perform a per-respondent analysis on it to see if these two subsections of people consist of the same individuals, this does suggest that there are reasons not elucidated in any one of these questions regarding why one might choose to vote stay or go, and hence the conclusion is incomplete.

That said, for those of us currently desiring a Remain verdict, it seems that it would do no harm to try and spread some of the more validated “truths” to the nay-sayers. Given the mess that both sides have created whilst campaigning, it may be debatable how effective that can be amongst the noise; but, if we want to believe in the validity of referendum politics, then we must try to believe that true knowledge has some impact on one’s voting choices.

However, there are yet further psychological forces to counteract even the most ardent advocate of facts driving decisions: given research suggests that we tend to disregard anyone whose opinion disagrees with us, and that we  often make up reasons to explain our behaviour after we’ve executed it (Kahneman writes excellently on this), the war for votes requires something more than simply winning the battle to expound the truth.



The Sun and its dangerous misuse of statistics

Here’s the (pretty abhorrent) front cover of yesterday’s Sun newspaper.


Bearing in mind that several recent terrorist atrocities are top of everyone’s mind at the moment, it’s clear what the Sun is implying here.

The text on the front page is even more overt:

Nearly one in five British Muslims have some sympathy with those who have fled the UK to fight for IS in Syria.

The implication is obviously a starkly ominous claim that 20% of Britain’s Muslims support the sort of sick action that IS have claimed responsibility for in Paris and other places that have experienced horrific, crazed, assaults in recent times.

This is pushed even more fervently by the choice to illustrate the article solely with a photo of Jihadi John, an inexcusably evil IS follower “famous” for carrying out sick, cruel, awful beheadings on various videos.

Given the fact that – most thankfully – there are far, far fewer people featuring on videos of beheadings than there are people travelling to Syria to join fighters, this is clearly not a representative or randomly chosen photo.

Sun writer Anila Baig elaborates in a later column:

It beggars belief that there are such high numbers saying that they agree with what these scumbags are doing in Syria. We’ve all seen the pictures of how innocent Yazidi girls are treated, how homosexuals are thrown off tall buildings. It’s utterly abhorrent.

The behaviours she describes are undoubtedly beyond abhorrent.

But of course, this 1 in 5 statistic isn’t true – even in the context of the small amount of data that they have used to support this story.

It is however a very dangerous claim to make in the current climate, where “Islamophobia” and other out-group prejudices make the lives of some people that follow Islam – or even look like a stereotype of someone that might – criminally unpleasant. Britain does have statutes that cover the topic of hate speech after all, and hate speech can come from many quarters.

Anyway, enough of the rants (kind of) and onto the statistics.

There are three main points I describe below, which can be summarised as:

  • The survey this headline is based on did not ask the question that the paper implies.
  • The paper deliberately misleads its readers by failing to give easily-available statistical context to the figures.
  • The methodology used to select respondents for the survey was anyway so inadequate that it’s not possible to tell how representative the results are.



The Sun’s claim that 1 in 5 British Muslims have sympathy for Jihadis (the headline) and those who fight for IS (in main text) comes from a survey they commissioned, which was executed by professional polling company Survation. You can read a summarised version of the quantitative results here.

The “20% sympathise with Isis” and those implications are based on responses to question 6 (page 9 of the PDF results above), which asked people to say which of a set of sentences they agreed with the most. The sentences were of the form “I have a [lot/some/no] of sympathy with young Muslims who leave the UK to join fighters in Syria”.

Results were as follows, and the Sun presumably added up the first 2 boxes to get to their 20%, which isn’t a bad approximation. Note though that an equally legitimate  headline would have been “95% of Muslim respondents do not have a lot of sympathy for jihadis”.

Response % selected
I have a lot of sympathy with young Muslims who leave the UK to join fighters in Syria 5.3%
I have some sympathy with young Muslims who leave the UK to join fighters in Syria 14.5%
I have no sympathy with young Muslims who leave the UK to join fighters in Syria 71.4%
Don’t know 8.8%

However, compare the claim that they have sympathy with those ‘who have fled the UK to fight for IS’ and ‘they agree with what these scumbags are doing…homosexuals thrown off tall buildings’ (even ignoring the implications of supporting the sort of criminal mass murder seen in Paris) with the question actually asked.

There was no mention of IS or any particular act of terrorism or crime against human rights in the question whatsoever.

The question asks about joining fighters in Syria. Wikipedia has a list of the armed groups involved in the Syrian war. At the time of writing, they have been segmented into 4 groups: the Syrian Arab Republic and allies; the Syrian Opposition + al-Qaeda network and allies; the Kurnodish self-administration and allies; and ISIL (aka IS) and allies.

There are perhaps in the order of 200 sub-groups (including supporters, divisions, allies etc.) within those divisions, of which the huge majority are not affiliated with IS. Even the UK is in the “non-lethal” list, having donated a few million to the Syrian opposition groups.

To be fair, the question did ask about joining fighters rather than the c. 11 non-lethal groups. But we should note that – as well as the highly examined stream of people apparently mesmerised by evildoers to the extent of travelling to fight with them – there was also  a Channel 4 documentary a few months ago showing a different example of this. In it, we saw 3 former British soldiers who had decided to travel to Syria and join fighters – the ones who fight against IS. I do not know what religion, if any, those 3 soldiers followed – but is it possible someone might feel a little sympathy towards the likes of them?

It is not necessarily a good thing for a someone to be travelling abroad to join any of these groups with a view to violent conflict, and I am convinced that some people do travel to join the most abhorrent of groups.

But, the point is that, despite what the Sun wrote, the question did not mention IS or any of their evil tactics, and could have in theory suggested some sort of allegiance to very many other military-esque groups.

The question only asks whether the respondent has sympathy for these young Muslims who travel abroad.

To have sympathy for someone does not mean that you agree with the aims or tactics of the people they are persuaded to go and meet.

Google tells us that the definition of sympathy is:

feelings of pity and sorrow for someone else’s misfortune.

One can quite easily imagine a situation where, even if you believe these people are travelling to Syria specifically to become human weapons trying to mass target innocent victims you can have some sympathy for the young person involved.

It seems plausible to have some sympathy for a person that has been brainwashed, misguided, preyed on by evildoers and feels that they have such quality of life that the best option for their future is to go and join a group of fighters in a very troubled country. Their decisions may be absurd, perhaps they may even end up involved in some sick, awful, criminal act for which no excuses could possibly apply – but you could have some sympathy for a person being recruited to a twisted and deadly cause, whilst virulently disagreeing with the ideology and actions of a criminal group that tries to attract them.

And, guess what, the Sun handily left out some statistics that might suggest that is some of what is happening.

For every such survey that concentrates on the responses of a particular population, it’s always important to measure the base rate, or a control group rate. Otherwise, how do you know whether the population you are concentrating on is different from any other population? It’s very rare that any number is meaningful without some comparative context.

As it happens, a few months ago, the same survey group did a survey on behalf  of Sky News that asked the same question to non-Muslims. The results can be seen here, on page 8, question 5, reproduced below.

As the Sun didn’t bother providing these “control group” figures, we’re left to assume that no non-Muslim could ever “show sympathy”to the young Muslims leaving the UK to join fighters. But…

Response % selected
I have a lot of sympathy with young Muslims who leave the UK to join fighters in Syria 4.3%
I have some sympathy with young Muslims who leave the UK to join fighters in Syria 9.4%
I have no sympathy with young Muslims who leave the UK to join fighters in Syria 76.8%
Don’t know 9.6%

So, 14% of non-Muslims respond that they have a lot or some sympathy with this group of travellers. Or as the Sun might headline it: “1 in 7 Brit non-Muslims sympathy for jihadis” (just below the same picture of a lady in a bikini, obviously).

14% is less than 20% of course – but without showing the base rate the reader is left to assume that 20% is “20% compared to zero” which is not the case.

Furthermore in some segments of the surveyed population, the sympathy rates in non-Muslims are higher than in Muslims.

The Sun notes:

The number among young Muslims aged 18-34 is even higher at one in four.

Here’s the relevant figures for that age segment from a) the poll taken to support the Sun’s story, and b) the one that asked the same question to non-Muslims.

Response Muslims aged 18-34 Non-Muslims aged 18-34
I have a lot of sympathy with young Muslims who leave the UK to join fighters in Syria 6.9% 10.9%
I have some sympathy with young Muslims who leave the UK to join fighters in Syria 17.6% 19.2%
I have no sympathy with young Muslims who leave the UK to join fighters in Syria 66.2% 52.2%
Don’t know 9.3% 17.7%

So-called “Jihadi sympathisers” aged 18-34 make up a total of 24.5% of Muslims, and 30.1% of non-Muslims.

Another headline, dear Sun writers: “1 in 3 Brit non-Muslims youth sympathy for jihadis thankfully moderated by less sympathetic Muslim youth?”

A similar phenomenen can be seen when broken down by non-Islamic religions. Although some of these figures are masked due to small samples, one can infer from the non-Muslim survey that a greater than 20% proportion of the sample that classified themselves as some religion outside of Christian, Muslim and “not religious” were at least somewhat sympathetic to these young Muslims who go to Syria.

As a final cross-survey note, it so happens that the Muslim-focussed survey was also carried out targetting a Muslim population earlier in the year, in March, too, again with Survation and Sky News. Here’s the results of that one:

Response % selected
I have a lot of sympathy with young Muslims who leave the UK to join fighters in Syria 7.8%
I have some sympathy with young Muslims who leave the UK to join fighters in Syria 20.1%
I have no sympathy with young Muslims who leave the UK to join fighters in Syria 61.0%
Don’t know 11.1

Totting up the support for having some/a lot of sympathy for the young Muslims leaving the UK for Syria, we see that the proportion showing any form of sympathy fell from 27.9% in March to 19.8% now in November.

That’s a relatively sizeable fall, again not mentioned in the Sun’s article (because that would spoil the point they’re trying to make the reader conclude). Here’s another headline I’ll donate to the writers: ‘Dramatic 30% fall in Brit Muslims sympathy for jihadis‘.

Next up, time to talk about the method behind the latest survey.

Regular readers of the Sun might have noticed that normally the formal surveys they do are carried out by Yougov. Why was it Survation this time? The Guardian reports that Yougov declined the work because

it could not be confident that it could accurately represent the British Muslim population within the timeframe and budget set by the paper.

So rather than up the timeframe or the budget, the Sun went elsewhere to someone that would do it cheaper and quicker. Survation apparently complied.

Given most surveys cannot ask questions to every single person alive, there’s a complex science behind selecting who should be asked and how to process the results to make it representative of what is claimed.

Here’s the World Bank on how to pick respondents for a household survey. Here’s an article from the Research Methods Knowledge Base on selecting a survey method. There is far more research on best practice on polling, and this is one reason why resourcing using professional pollsters is often a necessity if you want vaguely accurate results, especially on topics that are so controversial.

However, none of the research I’ve seen has ever suggested that one should pick out a list of people whose surname “sounds Muslim” and ask them the questions, which is, incredibly, apparently the shortcut Survation used given they didn’t have the time or money to do the detailed and intricate work that would be necessary to generate a statistically representative sample of all British Muslims.

It might be that Survation did happen to choose a perfectly representative sample, but the problem is we just do not know. After talking to other pro-pollsters, the Guardian noted:

It cannot be determined how representative the Survation sample is because of a lack of various socioeconomic and demographic details.

Even if the rest of the story had been legit, then – being generous with the free headlines – the Sun would have been more accurate to write “1 in 5 out of the 1500 people we had the time to ring that had “Muslim sounding surnames” and did in fact agree that they were Muslim’s sympathy for jihadis“. But it’s a bit less catchy and even the non-pros might notice something methodologically curious there.

So, to summarise – the Sun article, which seems to me to tread dangerously near breaking various legislation on hate speech in principle if not practice, is misleadingly reporting on the results of a questionnaire:

  • that did not even ask the question that would be appropriate to make the claims it is headlining.
  • without providing the necessary comparative or historical perspective to make the results in any way meaningful.
  • that was carried out in inadequate, uncontrolled fashion with no view to producing reliable, generalisable results.

We have to acknowledge that, on the whole, people going away to join fighter groups is a bad, sad event and one that the world needs to pay attention to. Infinitely moreso, obviously, if they are travelling to commit atrocities with groups that can be fairly described as evil.

But for the Sun imply such dramatic and harmful allegations about a section of the British population to whom prejudice against is already very apparent (note the  300% increase in  UK anti-Muslim hate crime last week)  to its huge readership – who will now struggle to entirely forget the implication during their dinner-table conversation even if they wanted to –  is not only extremely poor quality data analysis, but also downright irresponsible and even dangerous.