A really great post has been brought to my attention on a blog I wasn’t previously following, and I’d like to share it with readers here. There are two current bugbears at the moment causing me irritation that anyone who follows me on Twatter will be aware of. The first is the woeful, shambolic nature of the local railway, and it’s use of performance figures. The second is the use of statistics by Walsall Police Superintendent Keith Fraser.
A brilliant, sharp and lucid blog. Hat tip to Inspector Guilfoyle. Click on the image to visit the blog.
On the surface, neither of these are interesting or worthy of note, but as a self-confessed statistics monkey, I beg to differ. Most folk don’t understand the weight and bias that can be subtly and effectively applied to numbers in discussion, and many who quote numbers that look appealing don’t understand how easily they can misrepresent things.
London Midland Trains, currently experiencing large numbers of cancellations and service alterations due to staffing issues, tweet performance data daily. They do this for the whole network as one figure. By amortising every single train from every line, this neatly disguises poor performance hotspots like the woeful Cross City Line, by tying them onto solid runners like Stourbridge Town – Junction. Further, it was recently pointed out by one clued up Network Rail employee that on one Thursday, London Midland avoided 44 cancelled services being counted in their performance stats by filing their removal as short-notice timetable alterations, which can be registered up to 10:00pm the night before they’re due to run. They can legally omit these from performance stats.
When called on the matter, London Midland moved to include those cancelled services in future figures, but still won’t give line by line performance, or supply data measured at peak times in isolation.
It might be an unavoidable staff shortage to them, but it’s a royal pain in the arse when you’re stuck at some freezing station waiting for a train home.
Since challenging Superintendent Fraser weeks ago, the situation I previously highlighted with police statistics he quotes has moved no further forward; I was asked to mail his press office with my concerns, which I did. I’ve never been dignified with a reply, and now it is being insisted that I telephone him in person. There seems to be no will to understand or comprehend why certain figures cannot be quoted as absolutes. It doesn’t take a detective to work out that I’m being fobbed off.
To Superintendent Fraser, I will point out that I don’t do deals in private, and anything said on this matter can be discussed openly, and in public. This is, after all, about transparency.
@BrownhillsBob great, crime is important to people so I wont stop regarding that. Re public statisfaction, Crime 80% and ASB cira 79%.
— Keith Fraser (@keithfraser2017) October 12, 2012
On the whole, I support Walsall and, by extension, West midlands Police. But statistics like the above one are a good illustration of why data awareness is so desperately needed. The above figure is calculated by calling back complainants and surveying their opinion. This represents a specific group, and leaves out those who fail to get service from the police in the first place. The data quoted is therefore misleading.
For an organisation that claims to be reforming away from a reliance on arbitrary statistics, this is quite depressing.
In their ‘About Us’ piece, West Midlands Police assert:
Traditionally, police forces have measured how they are performing by purely focusing on crime data – how many offences have taken place and if crime has gone up or down.
From April 2011, West Midlands Police took the decision to move away from this sort of counting and introduced a new way of judging its performance against a framework based on what the public tell us they want from their police service. This new way of measuring our performance is based on our vision, our values and the behaviours we expect of staff.
It is hugely important that numbers quoted in public discourse are easy to understand, traceable and carry veracity. They should also be, as far as is possible, presented in an impartial manner. All I’m asking of both Walsall Police and London Midland is to issue full datasets of such numbers they quote, preferably made available online. These datasets need to be accompanied by explanations of how they’re gathered. Many public bodies do this. That way, we can fact check the numbers we’re being told, and decide if the use to which they are employed is fair and accurate.
It is therefore with a degree of trepidation I commenced reading Inspector Guilfoyle’s blog post about misleading and pointless data and it’s visualisation. However, what I found was a clever, solid and intelligent demolition of nonsense numbers. In short, it’s a brilliant read and I commend it to anyone, not just numbers geeks like me. I tip my hat to the Inspector, and hope and pray he might have a word with his colleagues and try and improve statistics literacy in Walsall. Lord knows, someone needs to…
Statistics don’t lie. People who quote them, however…
RSS - Posts
80% of statistics are misleading….
Well in my head I’m sure that’s right…
Loving your conversation with LM, I am quite surprised they are keeping it going. Sometimes it just proves what a bunch of incompetent people run our most important services… Which is very similar to the Council still. Those high up with the expensive salaries have no clue about what it is really like at the sharp end for us plebs…
Statistics are like bikinis. What they reveal is suggestive, but what they conceal is vital. ~Aaron Levenstein
So important, this. Meaningless statistics are treated as some arcane truth by people not confident enough in their numeracy to challenge them, and as long as that is the case, time and money will be wasted by organisations on propaganda, lies, and, well, illogical rubbish. Presumably numeracy skills are being inadequately taught in school….but confounding this is the willingness of people (particularly women) who would be appalled if you pulled them up on their grammar or cultural awareness, to titter and profess their hopelessness at maths and science.
The victim satisfaction surveys are a legacy of the performance management obsession of the last government, and sat alongside a whole raft of measures that aimed to be a sort of ‘balanced scorecard’. These surveys still have to be done, though, and it’s unclear whether the current government is serious about removing all these performance measures (which they promised), as organisations such as the HMIC continue to base their inspections on such data.
Each force must sample a set % of victims of certain offences only. There are fairly strict rules about how the sample is selected. The questions are also prescribed, and forces are not allowed to vary from the script. ‘Taking all things into account, are you very satisfied/satisfied/neither satisfied nor dissatisfied/dissatisfied/very dissatisfied with the service you received?’
The de facto purpose is in order to compare the performance of one force against another. Generally speaking, forces with better satisfaction scores do seem to provide better levels of service, but even if a force was scoring 99%, would you be reassured if you were one of the 1%?
The 80% figure quoted by the superintendent relates to those who say they are very satisfied or just satisfied. The ‘neither-nors’ do not count as ‘satisfied’. So it’s likely that only around 5-10% of the sample say they are either dissatisfied or very dissatisfied.
The surveys have helped raise the importance of victim service. They are done with genuine independence and the sample is randomised. Almost all forces use private market research companies to run them, so they are fairly reliable. Forces also get the (anonymised) feedback comments, so they can act on what has been said. But they do not represent all victims, and don’t cover other service users (ie, those who are not victims but nevertheless have views on the policing they receive). They are usually representative at a force level, but may not be at a local borough level. As an average % figure, there may well be sections of the community that are consistently less satisfied than others.
Forces have genuinely tried to improve service during recent years. But while 80% is good compared to many other public services, that still leaves a fair number who are unhappy.
Pingback: Get sociable with Walsall Police! | BrownhillsBob's Brownhills Blog