Library Stats that are wrong

I get really upset when I see bad data – statistics that are poorly defined, improperly analysed, badly presented or just plain wrong. Most of the time they come from poor data literacy and ignorance, not deliberate misrepresentation or political spin (although this is becoming more common).* I’ll admit to having made mistakes along the way, but as a statistician I feel a professional obligation to use my superpower for good not evil, and to not go down the “I can prove anything with statistics” path – even though I could.

Why does it matter I hear you ask? It matters because our world is increasingly been driven by data. Evidence-based decision-making is a good thing and data driving behaviour is a good thing, but only if the data is good in the first place. When it’s not we get misunderstanding, distortions in behaviour and wasted resources and energy (and as it is we need to make the most with what resources we have). I know I can’t change the world by shouting about the improper use of statistics. But I can try to raise awareness.

This blog records three library statistics and pieces of analysis that annoy me (two from Victoria, one from NSW). And if I get in trouble for raising them – so be it. [By the way, this might get a bit heavy, so if you need some help in understanding it please get in touch.]

1. Library efficiency, VAGO Report into Council Libraries, November 2019

Do you hate it when someone starts a sentence with “I don’t mean to be rude, but …”. Well, in 2019 the Victorian Auditor-General’s Office conducted an audit of Council library services to “examine whether councils and Regional Library Corporations deliver services efficiently and effectively”. In grappling with what efficiency and effectiveness might look like in a public library VAGO undertook some Data Envelop Analysis, a technique that “uses multiple inputs and outputs to benchmark efficiency across a group of organisations that deliver similar services.”

And so, to benchmark and rank the ‘efficiency’ of every Victorian public library in delivering ‘core library services’ the VAGO analysis used two measures of input (the number of staff FTE and the size of the collection) and two measures of output (the number of loans and the number of opening hours). Any issues?

• Loans and collection size are already combined to produce a widely used efficiency indicator called ‘turnover’ (i.e. average loans per item). But smaller (often regional and rural) libraries typically have a larger per capita collection to ensure customers have a variety of reading options, and consequently tend to have lower turnover rates. Therefore, this analysis penalises smaller libraries who try to offer their customers a quality service.

• The report noted that “Longer opening hours mean that members have more time to visit the library and access its services.” But the number of opening hours is only useful when you also take into account the number of branches, the spread of opening hours and the geographic spread of a library service. For example, in 2018-19 Greater Dandenong had two large branches open for a total of 144 hours per week and Kingston had 9 smaller branches open for a total of 336 hours per week. Kingston wins on physical access – there’s a library on every corner open for 37 hours a week. Greater Dandenong wins on spread of access – each library is open for 72 hours a week. But VAGO says Kingston is nearly more than 2½ times more efficient.

The kicker came in the notes on ‘Limitations’. “Our analysis measures efficiency. It does not account for the quality or range of services that council libraries deliver. … We limited our analysis to the common outputs of libraries.” It then listed some of the presumably ‘uncommon’ or ‘non-core’ services offered by public libraries, which included computer and internet access, customer service and enquiries, and home library services. The resultant ranking table was pretty meaningless. Which is no criticism of the libraries that appeared at the top of the list – just an observation that this type of analysis leads to a conclusion that the most efficient library is the one with no staff. A soulless building open 24/7 with a few bookshelves that stock popular reads. No life, no learning, no connections, no community. No thanks.

2. Cost of Library Service per Population, Local Government Performance Reporting Framework, Victoria 2020

I mentioned this one in an earlier blog but it really irks me so it gets another go. In 2014 Local Government Victoria established a set of indicators and now publish them annually on the Know Your Council website so residents can compare the performance of their Council against all the others. It’s the league tables for rubbish collection, food handling, dog catchers and libraries.

In 2019-20 the LGPRF replaced one poorly defined indicator (LB3 Cost per visit) with another poorly defined ‘efficiency’ indicator.

LB5 – Cost of library service per population

Definition: The direct cost of the library service per population.

Numerator: Direct cost of the library service … Denominator: Population

It is in principle a very good indicator (the cost of running a library for every person who lives in the LGA) but it all goes pear-shaped when it comes to defining the cost of the library service. Because LB5 deliberately excludes “capital purchases such as library collection items” and public library funding contributions from the Victorian State Government’s annual recurrent grant. On average, these two items represent 15% and 16% respectively of the total cost per capita of providing library services in Victoria (PLV Annual Survey 2018-19). However, libraries that do not capitalise their collection and purchase collection items from recurrent funds will have to include the cost of purchasing physical collection items in the LB5 cost base.

Therefore, this indicator: a) does not represent the actual cost of delivering library services to the community as it excludes on average 31% of library costs (and up to 40% for some library services); and b) could vary by 10-20% between library services depending on the accounting treatment of physical collection purchases. LB5 is both INCONSISTENT (between library services) and INCOMPLETE (as it does not reflect the actual cost of providing a library service to the population).

But most importantly, what’s the point of a library efficiency measure that deliberately excludes the cost of providing the books. Yes Minister … St Edmund’s Hospital … anyone?

3. Number of Program Participants, NSW Public Libraries Statistical Return, 2019-20

This one is less about the quality of the statistic than the message it sends. It comes from NSW Public Libraries Statistical Return for 2019-20, which is an annual data collection that contributes to publication of service and benchmarking data (thanks SLNSW – I love it) and the compilation of national library statistics.

In the instructions to libraries for reporting the number of participants in library programs is the following note.

“Only count number of people who participate in the program, e.g. for pre-school story time - only count the number of children, carers should not be included; for a lapsit or baby story time program, both carer and child should be counted as both participate.”

My concern is that pre-school Story Times have two very important sets of participants. One is the young children who are brought along by their carers and hopefully leave the session with a positive experience of the library and having learned a story, a rhyme and a crafty skill. The second (and in my view more important) set of participants is the carers. I believe the most important outcomes from a Story Time session are that parents and carers develop skills and confidence to read to their child at home, and that they leave the library with a pram or bag full of books.

The research is clear. Children who are read to regularly have a literacy level almost a year ahead of children who are not read to at home. The child may only attend one Story Time session a week, but a public library can turn that into habitual reading at home in a literacy-rich environment if it explicitly engages carers as active participants and shared learners. The return on investment is huge.

Statistically, I could take the NSW number and multiply it by about 1.8 (assuming some carers have more than one child with them) to get a more relevant estimate of ‘participation’. But by excluding carers from the counting process this sends a message that pre-school story time is only about the kids – and it’s NOT! (Well, it is ultimately, but only through the agency of the parents and carers and the quality of the wonderful presenters).

* One of my all-time favourites in the ‘ignorant’ classification is an Opposition Spokesman for Education who was outraged that nearly 50% of schools in the state were performing below the average (median). Think about it – keeping in mind that the ‘median’ is by definition ‘the middle value’.