Data Development in India — Scope and Challenges

If information is the oil of the 21st Century then analytics is its combustion engine. What makes analytics quintessential is its ability to turn data into information, and further, turn information into insight.

India is at a critical juncture in the data analytics industry. Since we are in the formative years of nation-building, one of the inevitable tools to accomplish the same is through the relay of vital information and dissemination, which in turn could instrumentalise the chains of growth.

However, our tryst with data science has been relatively torpid. Although we have successfully established important institutions to garner and publish data, we are far from cultivating dedicated sources and channels that assemble and contribute to the process of building specialised agencies of information.

In this article, we look into the annals of data generation in India, systemic and institutional barriers faced in the process of doing so and try to enumerate ways in which the emerging data needs of a burgeoning economy like ours can be addressed.

Role of Data Science in Economic Growth

The world seems to be witnessing what they call the “Fourth Industrial Revolution” (the first three being heralded by the advent of steam engines, fossil fuels and computers respectively). Data science and artificial intelligence are the precursors of this age wherein the ability to extract, refine and deploy new sources of value to the global economy through digitisation holds the key to progress.

More and more sectors are adopting data as an asset to enable innovation across a host of services like processing, efficiency-building, automation, modernisation, statistics, management etc.

And that is just the private sector. The Government’s reliance on data has become increasingly fundamental as well. Management of both natural and human resources through computational data derived from Census or other formats is on the rise. Think about it. The state depends on multi-source data collection and collaboration processes. Building them into coherent frames of references and common taxonomies (e.g. welfare, banking, privacy, security, education, etc.) is crucial for policy formulation.

Rising Demand for Data Analytics

With rise in Big Data and new-age industries, the growth of technological services and solutions is commensurately rising. Companies are relying more and more on the employability of data science to drive efficiencies and per-capita productivity.

This has in turn led to a steady increase in the job market. According to an IBM report, in 2020, the job demand for data science and analytics professionals was projected to have increased by 364,000 openings to over 2.7 million.

But, what is important to keep in mind is the absence of skilled professionals to fill this void. Lack of upskilling capabilities is a major impediment to recruitment, which is why, even though one can find no substantial dearth of individuals pursuing data science as a career, positions like those of specialists, project coordinators, visualisation experts etc. in large corporations remain seemingly vacant as the junior crop of programmers, developers and designers are unsuited to fill these proficient positions.

India’s Glaring Data-Gap

As a country, we may boast of advanced data-collection and reporting methods that have been witnessed as a prelude to the early onset of the statistical age since ancient times (Arthashastra, Ain-i-Akbari etc. have dedicated chapters on legalised measurements, crop-yield patterns and land classification). However, the growth of inductive reasoning, which is an element of modern statistics, remains limited.

For instance, the methodology and base year used to estimate GDP growth was called into question recently. The method doesn’t take into account important indices like price deflators which are important to adjust GDP for inflation. There are also issues pertaining to transparency and incoherence of Government data which led to the incongruence of official and leaked values of unemployment rate in 2017–18. Not to mention, there are rising apprehensions regarding the proposed merger of the Central Statistics Office (CSO) and the National Sample Survey Office (NSSO), which indicates deficiencies at the structural level.

This also has a direct correlation with more-than-ideal number of vacancies in the data science industry. At the end of August 2020, close to 93,000 jobs lay vacant. The growth of data analytics and management is the precursor to growth and reform in the Government data sector. Inclusivity, upskilling, data architecture and adaptability of new-age techniques in data growth in the private corporations which consequently propel the ship of standardisation to sail successfully towards the public sector.

State of Statistics in Other Countries

There’s hardly any doubt that data processing and management is now a developmental issue. Big data, in particular, has been cited as immensely impactful in driving up the economies of developing nations. The utopian vision of a farmer in the Hindukush mountains keeping track of his cattle herd through installation of RFID in his smartphone is one that makes for a brilliant pursuit of growth in data and analytics.

Developed nations certainly have an edge when it comes to building data infrastructures that have political, social as well as economic dimensions. In a 2019 study conducted by Harvard Business Review, India ranked 24th on the list of “new” data-driven world order.

The study takes into account four standards: data availability, volume, quality and usability. Agreed, the absence of other important variables related to data privacy, localisation, transparency etc. were excluded from this study, it still reveals a stark digital divide between the developing and the developed world.

How To Unleash the Data Evolution?

The startup ecosystem in India is the potential mecca for bringing about a resolute reform in the development of data analytics and Big Data in India. Entrepreneurship in Big Data by local firms who produce services, software and solutions to Western clients is a promising window which can be tapped into by incentivising output and investment channels into them.

Need for upskilling is the next big step. Building dedicated campuses to enable courses with data visualisation, machine learning and artificial intelligence should be incorporated as a target objective of higher education policy making and funding processes. Innovation is the key to reform in the data sector and thus personnel-training is the most important condition precedent to achieving the same.

Let us also not forget that enlisting Big Data’s help shouldn’t blind us to much-needed reform in “small” data. Decentralisation of data development institutions which focus on collecting raw and primary data at the rural and grassroots level shouldn’t go remiss.

And please, let us endeavour to modernise our methodologies that go into computing and processing the data. To that end, the Ministry of Statistics and Programme Implementation’s proposal to create a one-stop real-time official data platform for all things concerning the Indian economy is a significant step forward.

The Pandemic Angle

The COVID-19 pandemic has ushered changes in many facets of our lives and governance. Even though it posed challenges to data collection during the two-month lockdown period, it has illustrated the need to develop more evolved data collection, analysis and accounting strategies in the future. Growth of dedicated data institutions is crucial to building a vibrant economy that allows us to build on the deficiencies pin-pointed to us in the form of credible and impactful numbers.

If it wasn’t for the contribution of UK Biobank, a prominent institution storing biological data samples, and its role in ensuring rapid linkage of the country’s surveillance system, expedited response to the pandemic would have been far from reality. Let us not stand at the similar edge of vulnerability in the future and start working towards building vibrant and data-driven democracies.

(Originally published January 28th 2021 in transfin.in)