How open data is changing the world

Open data is more than a fad, becoming a potentially very practical tool for everyone. In what aspects is the world changing by the open data and how is it possible to measure its scope and influence? We analyze it below.

The transformation capacity of society that has open data is reflected in all its facets. Public entities and companies, private companies and citizens can take advantage of the value offered by the opening of data for a permanent social evolution in various ways.

In order to measure in some way the level of impact that open data initiatives have, experts have developed analyzes that set out the essential factors for this purpose. In the following lines we will study the degree of influence of open data in our society, but not before stopping to study their current situation.

Status of open data

Before delving into the full potential of open data to establish a new panorama in society, let’s stop to visualize the current situation of data openness.

First, we can go to the Report on the Maturity of Open Data in Europe, prepared by Capgemini Invent for the European Open Data Portal. In this, 4 dimensions are measured: policies, portals, impact and quality.

The total maturity quantified by this study is 65%. If we disaggregate these results according to the aforementioned dimensions, we have policies with 82% maturity, portals with 63%, quality 62% and 50% impact.

As improvement measures, the report echoes the need to focus efforts on raising awareness of the relevance of open data and its reuse.

As a second source of information on the situation of open data, we have the Open Data Barometer, published by the World Wide Web Foundation, which focuses on the analysis of Open Data measures taken by the 30 governments that are fully committed in this aspect.

Among the positive aspects embodied in this study, it should be noted that two thirds of governments have made two-digit progress in more than five years and that more than one third has increased their rating by more than 50%.

Regarding the facets in which an improvement is needed, mention that less than 20% of the existing data sets are open. In addition, some countries, such as the United Kingdom and the United States have taken steps backwards in the area of ​​open data.

How the impact of open data is measured

How the impact of open data is measured

To really know how open data is transforming our reality, we must establish certain parameters that serve to measure this impact. For this, it will be pertinent to resort to the GovLab analysis of the University of New York.

This analysis sets 5 useful factors to measure the level of impact of open data initiatives. These are: definition of the problem and demand, capacity and culture, governance, collaborations and risks.

Problem and demand definition

This requires an in-depth knowledge of the situation that gives rise to the need to create an open data initiative to solve it.

Here a detailed study of the users who are going to benefit from this initiative is required, as well as the causes that originate the need, the definition of the objectives, the way to address the problem and how the data will be audited.

Capacity and culture

Here the cultural, civil and institutional aspects that can be an impediment to the benefit of open data are considered.

At this level, aspects such as those related to software and hardware necessary for its development, human, economic and all kinds of resources, as well as variables related to the level of digital culture of the agents involved are considered.

Governance

This aspect includes actions such as performance evaluations and the risks associated with data privacy, default opening policies and free access to information, minimum quality requirements and responsiveness to changes that may occur between Users and their needs.

Collaborations

When we talk about collaborations we are referring to joint actions with institutions and individuals within the world of open data.

More specifically, for these collaborations we must take into account data managers, intermediaries, experts in the field of dealing with open data in particular, as well as many other agents in the open data field.

Risks

Finally, it is not negligible to contemplate everything related to the security of open data, to be more exact we can talk about what implies privacy, guaranteeing the anonymity and security of sensitive information.

In this section we must also mention as a risk those initiatives that open data that only seek to give a good image, without trying to offer greater benefits to society.

How are open data changing the world?

Continuing with the study of the GovLab of the University of New York, it reflects 4 essential aspects in which open data is changing the world: improving governments, empowering citizens, creating opportunities and solving public problems.

The improvement of governments

Here the greatest transparency, the reduction of corruption, the improvement of public services and the distribution of economic, human and material resources are considered.

With the opening of data, governments clearly face a public audit by citizens, which will require full transparency and legality in their actions.

On the other hand, the results presented thanks to open data initiatives make the needs of the population better known, so that the provision of services for their satisfaction will be much more accurate and proportionate.

The impact of open data on society

Citizen empowerment

Thanks to the free availability of the information that brings open data, to which we must add new forms of communication, we have a much better informed society.

This population can make decisions strongly based on information, as well as decide new ways of mobilizing socially.

Creation of opportunities

Proper management and understanding of open data opens the door to greater innovation and growth of business initiatives, as companies can dispose and reuse this data to develop and market new products and services.

All this leads to economic growth of the productive apparatus of the region, as well as higher employment rates.

Resolution of public problems

This greater resolution capacity by citizens and public management organizations is the result of the development of policies to evaluate social problems thanks to the study of data.

For example, open data is useful in the management of health crises or those produced by natural or environmental disasters.

As we have been able to verify, the opening of data for consultation and analysis opens up endless opportunities for all types of institutions and people. From Ogoov, with our open data module we want to provide a simple, attractive, scalable and affordable solution to facilitate the deployment of open data initiatives to entities.

The 8 Open Government Data Principles

It would be strange to find someone who has not heard about open data at this point, as well as related concepts, such as Linked Open Data. Its influence reaches a broad spectrum of fields, from administrative and government to many areas of the private sector. 

This expansion of open data, as with many other aspects, causes its definition to be distorted, publishing with the open data label that does not meet the requirements established for it. What are these requirements? We describe them below.

How did it all begin?

To know the origin of the requirements to open data we have to go back to December 2007 and move to Sevastopol, in California. That was when during a meeting of the Open Government Working Group the 8 fundamental pillars that support the concept of open data was born.

Why was it so important to define what open data is? In addition to the conceptual confusion we have explained above, we must emphasize the importance of these data for the evolution of democracy and society in general.

This information made available to the public through the rise of the Internet allows everyone to participate actively in governance and, in addition, allows third parties to reuse this data to develop all kinds of tools with which to obtain an economic benefit and / or providing an integration of them into non-existent services and, with a global vision, contributes to a more advanced society.

But that’s not all, guided by these 8 principles for open data, administrations around the world and at all levels (municipal, provincial, regional or state) are more effective and more transparent.

What are the 8 basic principles of open data?

Once we have placed ourselves regarding your need and your applications, let’s analyse in depth the premises that, if fulfilled, allow you to name some data with the open category.

1.- Complete

All public data is made available. Public data is data that is not subject to valid privacy, security or privilege limitations.

With this condition that urges that all information produced or collected by governments be available to the entire population, it is intended that it may make the best decisions for the general good and / or reuse them in the interest of themselves or third parties.

This availability will lead to an evolution in multiple social facets, such as the Smart Cities.

In addition, we can say that each of us has full right to know and reuse the content of the data that governments contain, since we have paid for them through our taxes.

One of the main impediments we can face when applying this first principle is the fact that much of this government information has not yet been digitized. Therefore, this conversion to reusable electronic format is pressing for administrations that are lagging behind in this regard.

2.- Primary

The data is collected at the source, with the highest level of granularity, there are no aggregations or modifications.

This principle requires that for open data to be such, they must adopt three inescapable qualities:

  • High level of detail
  • Virgins, that is, they must not have suffered any screening or previous treatment. They must be presented exactly as they emerged from the primitive source.
  • Their origin and the references they contain can be checked, so that anyone can verify their validity.

 We understand that this principle bases its reason on objectivity and transparency, and that thanks to this second condition it is facilitated that the reuser of this information can process the non-cooked data as it suits him best.

3.- Opportunities

The data will be made available as soon as necessary to preserve the value of the data.

Some information only has value for a certain time, for this reason it must be published as soon as possible and kept updated periodically. Of course, if these data are provided in real time, we will face an optimal situation.
 
For this reason, in OGoov the Real Time Open Data (RTOD) module has been developed, a middleware that allows real-time access to content provided by different sources in an automated way, incorporating them in various formats to the organization’s open data catalog .
 
The amount of open data in real time is not as abundant as it should be, this being an improvement to take into account, as indicated in the Report on Maturity of Open Data in the European Union.
Principios de los datos abiertos

4.- Accessible

Data is available for the widest range of users with the widest range of purposes.
 
It is essential that open data can be consulted by all citizens, this implies people with some type of disability.
 
But that is not all, the information regarding Open Data must be interoperable, or what is the same, must be accessible independently of the software and hardware with which it is accessed. To achieve this goal, it must be published with the current standards and protocols set by the industry, as well as in alternative formats when required for reuse.
 
The data is not accessible if it can only be obtained through a web form or if the automated tools cannot access it due to a robots.txt file, or any other policy or technological restriction.

5.- Processable by machines

The data is reasonably structured to allow automated processing.

When publishing data we must always include formats whose coding allows automatic processing. These formats must be properly documented and clarified.

This means that the publication of data in unstructured formats, such as free texts, PDF files or those of a graphic nature, such as JPG or PNG, should be avoided, and if they are exposed, they should.

6.- Non discriminatory

The data is available to anyone, without registration.
 
For a data set to be open, it must be accessible without the need to provide any identifying information, such as a username and password, among other authentication methods. This is because, through these practices, the distribution of public content is being hindered.
 
All access facilities must be offered without registration or authentication, both through web interfaces and public APIs for this purpose.

7.- Non owners

The data is available in a format over which no entity has exclusive control.

It is necessary that the published data are in an open format, in order to facilitate its free access to all, as well as its reuse. If there is a situation that proprietary data is cataloged, they must have their counterpart in an open format.

Therefore, files with extensions such as CSV, XML, SVG, etc. will have priority.

 8.- Free of license

 The data is not subject to any copyright, patents, trademarks or regulation. Reasonable privacy and security restrictions are allowed.
 
To allow the redistribution, reuse and commercialization of this data, they must have an open license such as CC0CC-BYPDDLODC-BY, etc.
 
This condition excludes impediments arising from copyright or patent rights. On the other hand, this does not imply that the policies related to the privacy of personal information should not be taken into account.
 
Through open data portals, such as those deployed using the OGoov platform, Public Administrations can make available to the public data that have an open license that facilitates their redistribution, reuse and commercialization.

Are these principles fulfilled?

Once we have made it clear what conditions open data must meet to be considered as such, it is appropriate to analyze the extent to which countries comply with them. For this purpose, the report that analyzes the level of maturity of the open data initiatives of the European Union, which we mentioned above, will be very useful.
 
In this report 4 aspects are evaluated within the open data, policies, impact, portals and quality. On average, the results obtained were:
  • Polítics: 82%.
  • Impact: 50%.
  • Gateways: 63%.
  • Quality: 62%.
In what concerns Spain, this is at the forefront in terms of maturity of its open data, its results being: policies (93%), impact (97%), portals (78%) and quality (78%).
 
From this report we can extract that, although we are immersed in a positive trend, the margin for improvement is wide. For example, there are data sets that contain errors in their distributions, that are incomplete or poorly detailed. This makes it impossible for citizens and reusers to make practical use of them.
 
From OGoov we offer solutions to facilitate that the different data sets available to public bodies and that should be available to everyone in portals and catalogs such as those deployed by our platform, meet these requirements (and Spanish and European legislation ) and that all this can be implemented with a high level of automation and minimal impact on the organization that publishes them.
 
In this way we will take a very important step in a better use of information of incalculable value for the progress of society.

Is data journalism possible in Spain?

Data journalism has taken a giant step in recent years, benefiting from the legal and technological measures of data openness and transparency. What is the current panorama of this journalistic branch and what do the different entities do to facilitate it?

The journalistic specialty based on the collection and analysis of large amounts of data has its germ in the 60s, then using the limited technological resources they had.

It was at that time when communication professionals began to be aware of everything that could offer them to truthfully inform society and perform the work of social counter-power and auditors of the political powers that characterize the journalistic profession.

As it is easily deductible, data journalism has been exponentially enhanced with the popularization of the internet, the emergence of open data and new data processing technologies, such as Big Data.

At this point, we should ask ourselves in what situation is data journalism in Spain and if this current situation allows its development with absolute normality. We look for the answer to this question below.

Data journalism and transparency

One of the main issues that we must resolve is whether there are sufficient levels of transparency by Spanish public entities to be able to develop data journalism normally.

To draw valuable conclusions, the work done by Leonardo La-Rosa and Teresa Sandoval-Martín, from the Carlos III University, entitled The insufficiency of the Transparency Law for the exercise of data journalism in Spain, which tries to analyze the consequences of Law 19/2013 and the Spanish government’s transparency portal to data journalism, as well as the reaction of journalists to these initiatives. To this end, interviews were conducted with journalists from different media and a thorough investigation of the newspaper archive.

As a summary of the interviews, we can conclude that, as the title of the research indicates, the Transparency Law and the government’s transparency portal should help much more than it currently does in the exercise of data journalism and instead, in some cases, it even makes it difficult.

The interviewees point out that the deadlines defined by the law are not met, the requested information arriving too late, if it is not denied with relative ease. Their responses also reflect the fact that the data offered in the respective transparency portals lack practical utility for the development of their work.

Regarding the study of journalistic publications, 93 articles were analyzed, what most attracts attention is that only 9 news did not have any interested opinion as a source, such as political parties and institutions directly involved.

Data journalism

How Spanish journalists access public data?

One of the factors that are worth studying further is the modus operandi of journalists when accessing these public management data. For this, we will use the comparative study on access to public data by Spanish and Swedish journalists, entitled The Promise of the Transparency Culture.

To carry out this study, semi-structured interviews were used with ten journalists working in five Spanish and five Swedish news organizations. In addition, the results were supported by a survey of 96 Spanish and 84 Swedish data journalists that contained 17 questions on data and public data journalism.

The results reveal that 50% of Spanish journalists have requested information from public authorities several times, and almost a fifth of Spanish data journalists have never requested public information.

In reference to the main challenges of Spanish data journalists, the study highlights access to the data itself, statistical skills and public transparency. In addition, it highlights that the simplification of the process of access to public data does not meet expectations.

As in the study mentioned in the previous section, these interviewees also show their complaint about the slow delivery and the denials of certain information. They also put on the table a conflict between the Transparency Law and other laws that may have priority over the former.

Initiatives

Just take a look at the mainstream media to be aware that data journalism is reaching certain levels of prominence. Articles of various kinds, often accompanied by a significant amount of graphics and infographics attest to this trend.

That is why it is intended to make visible and encourage this discipline, as well as the professionals who exercise it. To this end, the professors of the Master in Innovation in Journalism of the Miguel Hernández University (UMH for its acronym in Spanish) of Elche have shaped the first list of professionals related to data journalism in Spain and Latin America, which has more than 300 members. This tool also aims to increase the interaction between these types of information professionals, so that synergies of high value for the sector originate.

As an example of the application of data journalism, we can highlight the Empty Spain Workshop, in which the Medialab-Prado data journalism group compiled the results of the work carried out on Empty Spain, a project that demonstrates the demographic evolution of the country from open data. From this data, various multidisciplinary groups developed information, applications and projects.

Some open data portals even reflect in some of its sections how the information they offer in the journalistic field affects, such as that of the city of Madrid, which shows a compilation of the published news that has been based on the different datasets offered by said portal.

In short, in response to the question that gives title to this article, we can say that data journalism of course is possible in Spain, as evidenced by the greater presence of works related to the study of data and the various related initiatives.

The blame for this growth in data journalism is open data portals. Providing added value to those who facilitate the analysis of published data presented clearly through tables, graphs and/or maps, which facilitates and streamlines the work of the communication professional and subsequent understanding by the citizens. An example of this can be found in OGoov, the Platform that allows the display of Open Data Portals developed by Viafirma.

However, there is ample room for improvement, especially by public institutions and certain reluctance to provide the requested information, as well as the slow bureaucratic procedures to obtain such information and the lack of truly interesting content in Open data catalogs. Defects must begin to settle as soon as possible.

Last edition of the Open Data Barometer

Many countries have committed to open data. The latest edition of the Open Data Barometer focuses on these governments to analyze if they are really taking the necessary measures and if real progress is being made in this area. Next, we analyze this latest report published by the World Wide Web Foundation.

Year after year, the World Wide Web Foundation, a foundation founded by Tim Berners-Lee, creator of the World Wide Web, devotes a report to analyzing the current state of open data around the world. 2018 was an important year for this organization as it marked the tenth anniversary since they met to establish the eight essential principles for the opening of government data.

Since then, the foundation has been working closely with governments to open their information to the general public and offer citizens new ways to participate in their community.

The Edition of the Leaders

The last edition of the Open Data Barometer, published in September 2018, is entitled “Barometer of Open Data – Edition of Leaders”, as it focuses on the performance of leading governments (thirty, in total) throughout of this decade and establishes basic guidelines to continue on the right path.

It has been named leaders in this report to these thirty governments because they assumed some commitment to open data, either by signing the Open Data Charter or by adhering to the Open Data Principles for the fight against Corruption of the G20.

This does not imply that these governments have made great progress in this matter, since none of them has yet carried out the organizational changes necessary for open data to be the norm.

Even so, these commitments have been shown to be important and make a difference. The Leader Edition scores are two to three times higher than in the Fourth Edition (we analyzed it in depth at the time), which had 115 countries in total.

The conclusions of this edition are quite encouraging for the leading countries, but they are not exempt from worrisome trends that the report points out. To begin with, two-thirds of governments have made two-figure progress in more than five years and more than a third have increased their score by more than 50%.

However, as we said, not all is good news and the report points out several issues that need major improvements. For example, less than 1 in 5 datasets are open, which is a disservice to the opening of government data.

In addition, governments that were once leaders are no longer leaders, such as the United Kingdom. This country, formerly number one in this aspect, has been reducing its score over the last five years. The United States is another of the countries highlighted as being unable to maintain its position as leader. Its score has fallen eleven points.

Classification of leaders

In this latest edition of the Open Data Barometer, Canada and the United Kingdom are in the first position with a total score of 76, followed by Australia (75), France (72) and South Korea (72). Although the United Kingdom is in a privileged position, its score since the first edition has decreased by four points, while Canada has won eighteen in these years.

Ranking Open Data Barometer

In general, this edition is characterized by progress in the opening of data and greater approval and improvement of open data policies and practices, although there are still differences in rhythms.

Mexico is the fastest growing country, increasing by 33 points in just five years. There are also other countries with an enviable rate of progress such as South Korea, Colombia, Ukraine, Japan and Uruguay, with more than 20 points since the first edition.

Historic score improvements

However, there are also other countries that are not in such a good position and there has not been remarkable progress in recent years. The United Kingdom, Germany, the United States, Chile and Costa Rica are some of the governments that have retreated or have improved very slightly. Keep in mind that all these executives have committed to opening the data.

The most negative conclusion that the report leaves us is that governments still do not consider open data as a priority, but as a secondary project. That is why more investment is needed in the governance of open data and not simply promises that end in broken bag. The subtitle of this year’s report refers to this very thing: “From the promise to progress”.

In view of this current situation on open data, the report concludes that the most important action for governments to make the necessary progress is to invest the necessary resources to create the policies, practices and infrastructure that will serve as the basis for the promotion of the opening of data.

The foundation has also issued three specific recommendations for improvement. To begin, carry out a plan that has the default opening, which must be supported by a series of plans, guidelines and procedures that make it possible to share this data proactively.

The construction and consolidation of infrastructures is also necessary, thus improving the quality and interoperability of open data. Finally, the publication of the data must have a purpose, so it is necessary to collaborate with groups of citizens to identify the problems that open data are capable of solving. Once published the relevant sets, you must analyze what has been its impact and whether it has met the desired objective.

Lessons learned from past editions

The five years in which the Open Data Barometer has been published have offered a lot of interesting information that helps us to have a general idea of ​​the situation, both because of its temporal and geographic scope.

Although open data has become “relatively popular”, the actual opening has not occurred, as shown by the fact that less than 10% of the analyzed data sets are open. In addition, even in cases where data is available, they are often incomplete or not of sufficient quality.

Political will can be decisive for the opening of data is a success or a failure. The most complicated moment is in the allocation and investment phase. In some cases, it is committed to a false opening, through which the information is published selectively without taking into account the principles of open data and with the sole purpose of offering a better image.

Despite the fact that leading governments have committed themselves to creating a solid infrastructure and a community around them, these promises do not become real and do not materialize into a tangible investment.

As for the legislative part, the open data is not receiving much support in recent years. In the absence of strong legislation on the right of access to information, citizens cannot demand results from governments and they do not need to be held accountable.

Finally, it is not acting correctly in the analysis phase, so we do not have enough information to know the real impact and benefit of open data. Most programs do not have a post evaluation and the few studies that exists focuses on anecdotal parts.

One more year, the Open Data Barometer helps us to know the real situation of open data around the world. In this Edition of Leaders, we are left with the remarkable progress that some countries have made in this matter, although a real commitment on the part of the governments to create a solid infrastructure, with sufficiently large investments, is not yet perceived.