The imperfect art of library benchmarking
Data has become a critical management tool in the last 30-40 years. It always was integral to thorough analysis of business performance, but its importance has been heightened by decision-makers, funders and stakeholders who want to know objectively why they should pursue a chosen path or continue to invest in a particular program or product. Hence the rise of benchmarking.
Being a statistician I think this is a fabulous thing. But … and there is always a but … what we statisticians also know is that data is not always pure, not always objective and not the only input to good decision-making. [Cue discussion on Australia’s just released first ever ‘wellbeing’ report – Measuring What Matters, https://www.abc.net.au/news/2023-07-21/jim-chalmers-releases-national-wellbeing-framework/102629490 ]
We also know that too much analysis can cloud otherwise obvious conclusions. The winner of this year’s AFL Grand Final may NOT be the team with the best D50 to F50 conversion rate, optimum kick to handball ratio, or the best mix of inside and outside mids. It WILL be the team that kicks the highest score! And Stuart Broad, games of cricket are not awarded to the team with the most ‘impetus’. It’s the team that scores the most runs. Sometimes it’s that simple. [Yeah - I know, data geek and sports geek.]
So … to library benchmarking.
I am wrapping up a benchmarking exercise for a client that wanted to know how their library service was performing. And to be honest it’s been a mess. In theory I just take the data on a few key library indicators (say visits, loans and funding per capita) and compare the numbers. But consider this.
Should I compare this library against the national APLA/ALIA standards, the sector-wide averages in their state/territory or libraries of similar size and population demographics? All 3 are relevant, but they might (and did) give me slightly different answers.
Curiously, there are some significant differences in the profile of library use between Australian states and territories.
There were inconsistencies and gaps in some of the data, so it wasn’t really an apples vs apples comparison.
There are slight differences in how different jurisdictions and library services and systems count things, so again Granny Smiths vs Red Delicious.
Some library indicators can be interpreted differently. Is it better to have high or low funding per capita (quality vs efficiency)?
And some indicators carry greater weight for some stakeholders than others (operating efficiency vs scope of service).
Should I compare this library’s performance over time? Is their performance getting better or worse?
Well there was COVID – remember that! So library use in every financial year from 2018-19 has been compromised to varying degrees. What is normal?
Two of this library’s largest branches has been closed for renovations for 12 and 18 months respectively and been operating out of temporary facilities. Apples and bananas.
Their population is growing at 3-5% per year so the mix of library users and services demanded/needed is changing over time.
And finally, the world itself is changing and this impacts what library services are used. More downloads of eResources than ever before, fewer fixed PCs and more wifi hours, social inclusion is as important as literacy and lifelong learning.
You can see how this plays out. In a desire to objectively understand how this library performs against standards, benchmarks and peers I reached multiple, divergent and somewhat unreliable conclusions. You wanted a straight answer … well, it depends on what you think straight looks like (or what ingredients you like in your fruit salad).
It is in these circumstances that I am reminded that number-crunching is as much an art as it is a science. And that if you want to make sense of an imperfect world you first have to acknowledge and accept the imperfection. Only then can things become clear.
So how do we find meaning in this mess and convince others that our conclusions are valid? The answer of course is in the statistics, because statistics are all about probability, and the more data you have the more likely it is that it starts to tell a consistent story. Not through any single indicator but through the combination of them all. It’s a bit like circumstantial evidence in a criminal trial (“indirect evidence that does not, on its face, prove a fact in issue but gives rise to a logical inference that the fact exists”). If enough bits of evidence suggest ‘the butler did it’, then he/she probably did. And if enough indicators suggest your library is performing a little above or below par, then it probably is.
In this case I presented the different and competing data, but found a narrative that seemed to hold true whichever way I looked at things. It was that the library provided a good service to the community, yet usage was below expectations. The indicators on funding and staffing and collections and facilities were all about where they should be (+/- a margin of error), but the per capita visits and loans and program participation, etc. tended to be on the down side of the benchmarks.
When we talked it through there came a realisation that with all of the upheavals of the last few years the library had lost a little of its deep connection with the community. Community engagement, marketing and promotion and service partnerships needed a bit more focus, especially in a fast-growing community. And as they re-open their newly renovated library there is an opportunity to make a noise and bring people back in even greater numbers.
So what’s my point?
Firstly, as a numbers man I always go to the data, but I understand and accept that it might not always jump out at me with a clear story. However, I know that 95% of the time the story is in there if I make the effort to look for it. I look for trends, I look for corroboration, and I trust my gut, because library benchmarking is a combination of science and art. And I think 95% is pretty good odds.
Secondly, the story is what the story is. Basic statistical theory has an extraordinary and proven capacity to reflect what is actually happening in the real world. So if the numbers are down because of COVID don’t shy away from that. If you did something great or dropped the ball it will probably show up in the numbers. Keep an open mind, be honest, and the data will ultimately be your friend.