Approaching the reality of the silicon state

Has Margaret Thatcher's favourite philosopher finally met his match? Friedrich Hayek famously argued that centralised planning was doomed to fail because a bureaucracy could never gather and interpret sufficient information to make sensible decisions. If attempted, the outcome would always be waste and resource misallocation on a grand scale. Instead, wherever possible the invisible hand of the market should be preferred to the dead hand of the state.

Hayek won a Nobel Prize for his efforts, but more importantly his thinking went on to influence generations of economists, politicians and policymakers right up to the present day. In the 50 years since he first developed his ideas, the policies of devolution, marketisation and privatisation have been embraced by governments around the world, as well as promoted by international bodies such as the International Monetary Fund and the World Bank.

But Hayek is wrong.

Interestingly, he wasn't always wrong but something dramatic has happened in the last few years that should give his proponents pause for thought. The rise of big data and advanced analytics now means that Hayek's fundamental contention - that information is overwhelming - is increasingly challengeable, and the implications for policymakers will be profound.

How centralisation can be a winning strategy

A revolution powered by big data and cognitive computing has been heralded in the private sector for several years. Companies like Google and Facebook are famously founded on business models that require the massive collection and interpretation of data on a scale simply unimaginable to Hayek, but more traditional industries are being disrupted too.

For example, real estate companies that oversee large national portfolios of properties in the US have moved away from a decentralised approach, in which local managers set rents, to centralised datasets and price-setting software. The centralised approach consistently outperforms the devolved solution. Similar examples can be found in industries ranging from logistics (where the routes of UPS vehicles are determined by software rather than drivers) to retail (where the staffing rotas and stocking schedules in supermarkets are determined by algorithms that predict customer demand).

Situations that lend themselves particularly well to such an approach are likely to involve repeated decisions, based on diverse information, to achieve measurable outcomes. When this process is handled by a so-called learning algorithm, with the capacity to draw inferences between action and outcome, the result is turbo-charged learning at a level that rapidly outstrips human capabilities. See for example DeepMind's mastery of a classic Atari video game - from absolute beginner to world champion in a matter of hours.

The public sector is awash with problems that are particularly amenable to cognitive computing, given the complexity of public services and the social problems they attempt to address. For example, advanced traffic management systems are already used in countries like Japan to help reduce congestion by automatically controlling traffic lights and speed restrictions. Such applications are relatively uncontroversial but, as technology advances, it will be increasingly possible to apply similar approaches to interventions in health, education, welfare and justice. This isn't a question of ‘if' but rather ‘when'.

The case for the silicon state

The implications of all this for policymakers are many and varied. I'll focus on just three here.

Firstly, the assumption that markets and devolution are preferable to greater central control can be challenged afresh on practical and empirical grounds rather than relying on philosophical or political arguments. For example, it is no longer clear that multiple local commissioners are better placed to make decisions about public services than a centralised agency. Similarly, how certain are we that multiple outsourced providers are better placed to aggregate the knowledge required to improve public services than a single in-house provider?

Secondly, the emphasis on data and outcomes in public services gains a new resonance. Data is the fuel upon which advanced analytics depends, and higher quality, richer datasets lead to deeper insights and better decisions. We are no longer limited by the data-absorption capacity of humans or, indeed, the need for data to be presented in a structured manner. Policymakers should therefore redouble their efforts to ensure that as much information as possible about the inputs, activities and outcomes of public services is collected and shared in a way that unleashes innovation and service improvement.

Thirdly, the moral and privacy issues raised are profound but unavoidable, given the march of technology, so we must start grappling with them now. Would it be acceptable for my unemployment benefits and sanctions to be determined by an algorithm predicting my propensity to work? Will I always be able to tick the box to say I can opt out and retain complete control of my personal data, even if it means less effective, more costly public services?

In his Nobel acceptance speech Hayek observed that if man is to avoid doing more harm than good he must learn that “he cannot acquire the full knowledge which would make mastery of events possible”. The contrary is now increasingly true, and we must quickly learn how to harness the power that such knowledge can offer us all.

FURTHER READING

• Open data: unlocking development potential in Africa and Asia. Dr Savita Bailur explains how open data has the potential to empower ordinary people to participate in development, not just as its beneficiaries
• Power to the people. Few countries have embraced the digital era as successfully as New Zealand. We talk to one of its government's key digital transformation leaders, Richard Foy,about how they've done it
• Computer says yes. Governments are increasingly reliant on digital technology to deliver public services - and Australia's myGov service is a potential game-changer, says Gary Sterrenberg
• Digital dawn. It may not be obvious, but US policymakers have had an important role to play in the creation of today's digital era, says David Dean
• Core workouts. With governments increasingly seeking to transform their core systems,Andrew Arcuri explains how they can move from achieving good results, to truly great
• Online, on track. BCG's Miguel Carrasco looks at how policymakers can improve the delivery of digital services
• Open all hours. Liz Carolan explains how open data can help accelerate development progress around the world
• Moving beyond open data. Adrian Brown says governments should take a strategic approach to open data
• Data deliverers. International policymakers set out their priorities as they promote open data in their governments
• Extended opening hours. Open data is increasingly being used to revolutionise public services. Here, we take a look at examples from Nigeria, Ecuador and New Zealand