CCSI Data Observatory

ESRS in the Film, Television & Music Industry

Wed, 28 Feb 2024 15:00:00 +0100

Save the date.

IDCC24 Lightning Talk Session

Tue, 20 Feb 2024 16:15:00 +0000

Open Music Europe is a Horizon Europe project that aims to build a working prototype of the planned European Music Observatory.

Interested in our automated data observatories? Let us meet in Edinburgh on the 18th International Digital Curation Conference and discuss who you could use our open-source, collaborative data infrastructure and know-how.

The EU, UN, or other international bodies have recognised or initiated at least 60 data observatories that carry out long-term data collection on various domains; we have not found any good policies or practices on how to place these observatories on data infrastructures that are interoperable towards open science and open government. We are creating a data management and governance model and a working MVP that coordinates data collection and statistical data production among scientific, private and official statistical actors.

Our most crucial pilot project wants to showcase a best practice for using privately-held data, i.e., data of music organisations and surveys carried out by scientific and business actors, to improve the quality of government statistics. We show how the guidelines on using private data as an ‘administrative data source’ and an ex-ante harmonisation of governmental surveys with open scientific surveys can result in high-quality datasets that fully complement the pre-existing official statistical products and commercial products.

As a coordination tool, we started developing a Data Management Plan to increase transparency from the outset. Apart from applying Horizon Europe’s OpenAIRE recommendations and FAIR requirements, we use the Open Policy Analysis Guidelines to bring open science transparency into the less standardised policy analysis area. We implement this following various UN/EU Guidelines on statistical production, creating a three-way reconciliation and interoperability, i.e., scientific research, public policy design and official statistics.

Click through to our working paper (available in PDF, epub, and html).

Our work contributes to sharing outputs earlier using Open Research platforms because we are building a framework supported by research automation that integrates open science, business, and official governmental data. We develop a software ecosystem complementing the R statistical environment and language, the lingua franca of official and scientific statistics, to make the data curation, pre-processing, processing, and eventual quality-controlled statistical data release open, transparent, and much timelier.

Our project follows an open collaboration framework that we design so that private music NGOs and enterprises, statistical offices and open science research groups can work together on the curation and design, production and release and use of data assets in the cultural domain. By opening the statistical infrastructure with our open-source production code and implementing the statistical data and metadata exchange standards simultaneously with other metadata standards and standardisation techniques like ex-ante and retrospective survey harmonisation, we hope to combine them in novel ways like never before while making them available sooner.

https://music.dataobservatory.eu/documents/open_music_europe/slovakia/slovak-cult-stat-pilot.html

Our showcase product will be a twin, linked open data resource: the Slovak Comprehensive Music Database. It will connect in unprecedented detail information about musical works and their sound recordings and notations in music libraries, heritage organisations and individual and collective rights management organisations. We will derive the Slovak Music Industry Registry from this linked open resource that we will convert into a structural business register satellite as an interface between the privately-held data of music management and music heritage institutions and the national/satellite account system of the Slovak Republic, particularly the Slovak Cultural and Creative Satellite Accounts.

Let’s get in touch if you are interested.

Futureproof Film Production Data for Integrated ESG Reporting

Fri, 24 Nov 2023 09:00:00 +0000

Click through to the presentation slides (available in PDF, epub, and html).

Daniel Antal presented the concept of Open Productions for producers and film professionals at the National Film Institute of Budapest, Europe’s largest film production cluster. Our planned project, which received one of the highest scores in the EIT CCI KIC’s Breakthrough Lab and received a conditional funding promise, aims to create an open, computer-actionable knowledge base to support the integration of film production budgeting, rights management, financial bookkeeping and accounting systems with environmental (CO2, H2O, waste, pollution, biodiversity risk map) and social sustainability data (workforce work-life balance, gender pay gap, etc.)

Let’s get in touch!

We are looking for film productions to test our system and report easier, cheaper and better the CO2, H2O, CH4 footprint, tackle their health and safety, gender paygap and royalty management problems.

Open Productins: Integrating Financial and ESG Reporting for Film

Fri, 24 Nov 2023 09:00:00 +0100

Next: ️> or Space | Previous :️<
Start: Home | Finish: End
Overview: Esc| Speaker notes: S
Zoom: Alt + Click️| Fullscreen: F

🖱 Highlighted text: clickable link (to our project page.) Best view with Full screen.

Questions?

Ask: Email | Keybase

LinkedIn: Daniel Antal, CFA | Reprex

Semantic Enriching of The Historical Photo Collection

Mon, 20 Nov 2023 15:30:00 +0100

See the use case: Open Foto Ateliers

The slides are only available in the Hungarian language.

Categorizing Cultural & Creative Sectors: An Impossible Task?

Wed, 15 Nov 2023 09:00:00 +0100

Daniel Antal, as the co-founder of Reprex is honoured to be invited to the final event of the wonderful Creative FLIP program. Our Digital Music Observatory in many ways gained momentum from the CCS Ecosystems: Evidence-Driven CCI Policy & The Central & Eastern European Music Industry Report discussion in 2020, just before the terrible pandemic hit us all.

Blogpost for discussion: Categorizing Cultural & Creative Sectors: An Impossible Task? Creating a more flexible and useful business and occupational categorisations for the music and film industries.

More than two years have passed since Creative FLIP kicked-off a new project cycle, continuing to pursue its overall goal, to support a healthy ecosystem for the cultural and creative sectors and industries (CCSI). This is how we work – the launch of the online platform on status and working conditions of artists and cultural professionals is only one of many highlights to expect! Cruise along our flagship initiative, Creatives Unite, and discover its new look and tools, engage in discussions about the power of transformation policies, cross-sectoral cooperation, and platforms for peer exchanges. We will inquire if we should categorize CCSI occupations and why we need to talk about social inclusion, sustainability, greening and more transversal topics when discussing the future of the sector.

In this panel on the FLIP FORWARD Final Conference organised together with the CHARTER project (See: Factsheets: Families of competences pdf) we want to discuss the benefits but also the dangers of categorisation related to the self-image, the measurement, but also the visibility of the CCSI: categorisation enables a highlighting of similarities, but it also has the potential to exclude the unforeseen, the evolving, the fringes.

Our discussion will focus on cultural-creative occupations and be organised around the following questions:

To what extent do the categoriser’s viewpoint and goals shape the categories?
What are the limitations and dangers of categorising the occupational scope of the CCS?
Is it possible to develop an occupational categorisation of the CCS that serves all purposes? If not, how do you deal with this dilemma?

Are you in Brussels on 15-16 November? Interested? Get in touch.

Categorizing Cultural & Creative Sectors: An Impossible Task?

Sun, 12 Nov 2023 17:48:00 +0100

I am honoured to be invited to the final event of the wonderful Creative FLIP program. Our Digital Music Observatory in many ways gained momentum from the CCS Ecosystems: Evidence-Driven CCI Policy & The Central & Eastern European Music Industry Report discussion in 2020, just before the terrible pandemic hit us all. Are you in Brussels on 15-16 November? Interested? Get in touch and let us meet at the event.

It is one of our most fundamental cognitive abilities to identify shared features or similarities of objects, events, or ideas and then to group (“categorise”) these to make sense of the world. Without categorisation, learning, language, memory, and decision making (to name a few only) would be impossible. The production of cultural statistics also depends on categorisation, distinguishing what is to be measured and what is not.

Session: Categorizing CCS - an impossible task?

In this panel on the FLIP FORWARD Final Conference organised together with the CHARTER project (See: Factsheets: Families of competences pdf), we want to discuss the benefits but also the dangers of categorisation related to the self-image, the measurement, but also the visibility of the CCSI: categorisation enables a highlighting of similarities, but it also has the potential to exclude the unforeseen, the evolving, the fringes.

Our discussion will focus on cultural-creative occupations and be organised around the following questions:

To what extent do the categoriser’s viewpoint and goals shape the categories?
What are the limitations and dangers of categorising the occupational scope of the CCS?
Is it possible to develop an occupational categorisation of the CCS that serves all purposes? If not, how do you deal with this dilemma?

Are you in Brussels on 15-16 November? Interested? Get in touch.

One of the most fundamental problems of developing creative businesses in Europe and developing public policies to support them is that we have far less business and policy planning information available then any business in fruit farming, manufacturing of cars or providing banking services. Cultural statistics have a very low coverage in music or film. Indicators that an apple producer, a car component manufacturer or a bank supplier can take for granted, such as total or average gross value added in a country, employment, or wage statistics, do not exist.

The lack of statistical coverage makes it particularly difficult for such creative businesses and their industry organisations to comply with the expectations of the Corporate Social Responsibility Directive or participate in the advantages of green financing and the European Green Deal.

The production of statistics for the music or the film industry, like any other industry, relies on three fundamental categorisations:

the categorisations of economic activities that creative businesses and self-employed people do;
the categorisations of their products and services (such as films or sound recordings);
the categorisation of occupations and jobs available or filled with such occupations.

These categorisation do not work well in the cultural and creative sectors, and perhaps in some other services sector either.

To help small enterprises in music or film, we need different data collection and statistical or indicator production procedures and more flexible categorisation processes to support their business development on enterprise or a regional/national level. Photo: Jan Demiralp on Unsplash.

Why normal business and occupation categories fail to work in the creative sectors?

Unfortunately, these categorisations cannot be successfully applied in the music or the film industry for many reasons:

The entities are too small.
They have mixed activities.
They mainly employ people atypically without even checking their formal qualifications.

A small recording studio that works for films or the music industry, or a small company that offers stage or in-studio or on-set lightning typically employs people who learned their skills on the job; they often do various related activities, so their NACE code is not very characteristic. Also, because of their size, they hardly ever participate in governmental statistical surveys.

On a more fundamental level, in the creative industries, the dominant firm size is the microenterprise, where the management functions are usually not specialised and often not filled by people with business administration degrees: they do not have an HR department or even an HR function that would categorise their jobs; they do not have a management controlling system that would rely on a sophisticated bookkeeping or a well-categorised inventory of sales. This is what the International Labour Organization, or employment policymakers, describe as “informal enterprises”: they do not have formal internal structures, processes, and often even contracts that in larger enterprises enable a professional management of product, career, finances, or sustainability.

In Open Music Europe, building on a decade of experience at the Digital Music Observatory, we are using statistical processes and data innovation to serve the needs of the music industry by providing data on royalty collection, marketing, average artist remuneration, or gross value added. Our approach is based on innovative statistical procedures and recent regulatory changes that allow national statistical authorities to embrace them.

We do not have average musician remuneration statistics or gross value-added figures for music labels because even though musicians and record labels participate in the national statistical system, they are invisible as they do not have a clear category. The NACE categorisation system does not contain the activities of the “music industry”, only categories like “audiovisual” or “art performances” categories that mix up the numbers of the music industry with television, video production, or theatres. The same can be said of most occupations where music businesses employ workers or self-employed professionals. Even though they have a chance to participate in statistical surveys, which call upon a randomly selected list of small enterprises (in the case of business surveys) or people (in the case of the labour force surveys), they do not get an appropriate “musician” or “music business” label.

The categorisation or labelling of “performing arts” or “audiovisual” is unsuitable for a musician or filmmaker. If “audiovisual” average wages rise in a region, who are they supposed to know if this is due to outsourcing some film post/production to the region or an uptake in music recording activities? After all, film sound studios and music recording studios work for different markets and employ different people.

On the other hand, many creative industries know who belongs to their industry. Architects usually must fulfil severe qualification requirements and register into a local chamber of architects. Musicians must register with author’s and performer’s collective management societies if they want to receive copyright or neighbouring right royalties. If we can align such representative creative organisations with the statistical processes, we will get indicators that follow relevant categorisations for musicians, record labels or architects.

Several statistical and data innovations can make such an alignment or data coordination possible.

See: United Nations Guidelines on Statistical Business Registers pdf. European Business Statistics Methodological Manual for Statistical Business Registers. 2021 Edition.

Sample surveying, when musicians (as members of the working force) or their enterprises (as small businesses) are selected by a lottery to fill out a questionnaire, had been the most important source of official statistics between 1970 and 2010, but they are less and less used. Before the 1970s, we only had fewer statistics on full census-like questionnaires with no random sampling. In the last decade, we have seen more and more often that statistical offices directly connect to various “registers” or databases initially created for business or tax administration purposes to retrieve the data. Why ask a musician about her royalties if we can access her royalty account directly?

Why ask a musician about her royalties if we can access her royalty account directly? Accessing data that is available in a well-managed administrative source, such as a national royalty accounting system is cheaper, faster and more accurate.

From a royalty account, we can access more precise and timely information than from the memory of an artist.

In the last decade, UNECE and Eurostat have been the forerunners with several European national statistical offices in finding new ways to tap into so-called “privately held” registers, for example, the royalty accounts of copyright management societies, who have a full view of musicians in the country and precise and relevant information about their earnings, and indirectly about their employment or gross value added.

Economy of music in Europe: Novel data collection methods and indicators

In Slovakia, we are working out a process where the register and framework of the official data collection could be aligned with a private register set up by the music industry’s nationally representative stakeholders. Such a scheme would have many advantages both for the private and the public parties:

The collection of the statistical data would be faster, cheaper and more precise.
The data could be securely mapped to and from the NACE/ISCO categorisation used by the government and the relevant categorisations used by the music industry.
Whenever the music sector – a relatively minor part of the services business sector – would be under-sampled in a statistical survey methodology to produce reliable, music sector-specific data, additional, harmonised surveys or administrative data could ensure that minimum amount of music enterprise and musician data is present for quality-controlled statistical indicators.
The music businesses could professionalise their company planning HR processes, and, critically, have relevant benchmarks for environmental and social sustainability management; they could participate in the European Green Deal.

Are you in Brussels on 15-16 November? Interested? Get in touch!

Cover photo by Clem Onojeghuo on Unsplash.

Roundtable Discussions on Greening European Film Policy

Tue, 06 Jun 2023 10:00:00 +0200

Greening European Film Policy (GEFP) explores the burgeoning field of environmentally sustainable - ‘green’ - production in the European film industries. The project views green production as both an ethical proposition and a means to ensure the competitiveness of the European film industry - under which we include film, television, and streaming media. We position green production as a unique and as-yet unexplored competitive advantage for the film sector, where advanced development, embedding, and dissemination of green production practices is a crucial factor in maintaining the competitiveness, if not the very viability, of the European film industries.

GEFP focuses on mapping ongoing policy initiatives for green production and evaluate how the dynamics of top-down governance and horizontal policy and communication incentivise, deploy, and assess on-the-ground practice. We collaborate with key stakeholders across the UK and European green film and television sector, including policymakers, environmental consultants, production companies, and creatives to provide policy recommendations to incentivize a more sustainable sector.

Daniel Antal, CFA, was participating in the panel presentations and discussions on behalf of Reprex and presented the ideas behind the Eviota project.

The Budapest event was orgnanised by Dr Pietari Kääpä, University of Warwick, Policy Support Fund, Laurent, and GreenEyes Productions.

Questions for Film Funds and Policies

What are the current norms regarding funding and policy requirements around sustainability?
What challenges do you face in implementing or imposing sustainability policies?
Are there ways in which you might benefit from more communication or understanding of on-set experience from a sustainability POV?
Are there ways in which you might benefit from more regional or international collaboration, or is funding an inherently localized issue?
We mainly talked about sustainability from a production POV so far, where do you think featuring it as part of the storyline has relevance / potential? Is the film fund considering similar changes to what happened internationally (BFI, Eurimage etc.) where the funding applications require to elaborate on any planned sustainable practices?
What models do you think work well in other countries, that you think would be realistic to implement in the region and what is needed in order to implement it?

Open Data Day in The Hague 2023

Sun, 05 Mar 2023 13:00:00 +0100

Reprex is a finalist for The Hague Innovators Award 2022, and the prize of the audience, in the startup category with our respectable competitors, Sibö, WECO, STHRIVE and ECOBLOQ.

Reprex will organize events on the International Open Data Day. Save the date in your calendar.

About Data Observatories

Sat, 04 Mar 2023 00:00:00 +0000

More than 60 EU, UN, or OECD-recognized data/social science observatories exist worldwide.

Reprex’s data observatories are modern, 21st-century re-imagination of the observatory concept. In the 18-19th century, observatories were the centers of structured observation and data collection about the natural world. Since the 20th century, the EU and UN organizations created or acknowledged about 80 observatories ranging from natural sciences to social sciences and data observatories.

Table of Contents

New and maturing open data observatories

Digital Music Observatory
Cultural Creative Sectors Industries
Green Deal Data Observatory
Open Collections Net Digitized information and knowledge that is not shaped as datasets: books, texts, still images, sound recordings, films.

Watch Our 2-min Introduction with the example of music. (We do not only work with music data!)

⚙️/ Subtitles/ 🇳🇱 🇬🇧 🇧🇦 🇨🇿 🇭🇺 🇩🇪 🇱🇹 🇫🇷 🇸🇰 🇪🇸 🇹🇷 + Catalan.

Mission statement

Currently, almost unregulated machine learning is taking over newer and newer roles in human creation, jobs, tasks, and groups with the historically low quantity or quality of data representation are facing a new digital divide. (➡️ project page, slides.)

Our observatories aim to help small businesses, NGOs, foundations, research groups, and universities that do not have a data engineering team and cannot finance large, proprietary data collection and processing programs in social sciences and digital humanities. We help them to access big data, poolin their research needs, and provide open-source access to research automation.

Data feminism: fighting the gender bias of AI is a blueprint to correct other forms of injustice

Human- and machine-readable information about women is less quantity than men: the algorithm can find less suitable audiences for female writers and composers or suggests services or content to female readers.

Cultural and linguistic diversity: AI should not only benefit English-speaking countries and individuals

As books, films, and music is increasingly sold by algorithms on global platforms, inventories and catalogs that are not perfectly machine-readable or not available in English are sidelined by the autonomous recommender or sales systems.

Competitiveness of SMEs, NGOs, small countries, small research groups

Market, policy or scientific researchers who have no reliable access to big data assets and research automation become cannot compete with organizations that employ an army of data engineers, data scientists in research, innovation, or business development.

Services

Our Observatory Data Labs project tries to find new innovation and business models to fight this imbalance (➡️ project page, slides.)

An open-source, linked, open data-driven platform for open science, open government, and the commercial use of open data-driven innovation.
A governance model that complies with the European Data Governance strategy and regulations and provides a good practice of jointly exploiting data and knowledge among public and private, small and large organizations.
An API that fuels a growing range of open-source scientific and statistical software.
A data integration model that allows the joint exploitation of surveys, datasets, statistical data, and business indicators created by a citizen scientist, individual artists, small companies, research groups, and statistical authorities.

Our observatories collect, link, process, validate open data. They are very suitable for market/policy/scientific research organizations (⬇️ join the observatories). We offer open-source scientific and statistical software to exploit our open data and knowledge (➡ Reprex software and rOpenGov.). We are working with an increasing number of innovators who place our open data and open-source software into more friendly, commercial apps which remain affordable for non-profit users (➡ join our Data Observatory Labs.)

The observatories are open collaboration projects, following the principles of open-source software development, open science, linking open data, and the Open Policy Analysis Guidelines. The observatory partners collaborate on various projects that are either self-funded or granted. They jointly create datasets and databases, online collections of digital objects and knowledge, and open-source software.

History & Timeline

2014 CEEMID (2014-2020) is now the Digital Music Observatory(➡ project overview, multi-country research, survey harmonization, European best practice).
2018 First release of iotables, an reproducible research tool.
- why
- rOpenGov
2020 Automated data observatory product/market fit validation
- Early concept of an automated data observatory—blogpost with video.
- Product/Market fit development in the world’s second ranked university-backed incubator, the Yes!Delft AI+Blockchain Lab (➡ see announcement, Demo Music Observatory, Dutch AI Coalition.)
- First release of retroharmonize, a survey recycling and harmonization tool for better, cheaper, collaborative surveying and data recycling (➡ announcement, benefits.)
2021
- Further concept development in the JUMP Music Market Accelerator program for the music observatory: (➡ announcement, closing) .
- Green Deal Data Observatory concept
- Establishment of the Cultural and Creative Sectors Industries Data Observatory as a joint project of IViR (UvA) and Reprex ( website, announcement, collection().)
2022
- Several small projects in the
- Open Music Europe:
- First release of statcodelists and early release of dataset, two R libraries to
- Audience Prize in The Hague Innovators Challenge (➡ blogpost.)
2023
- Kick-off for Open Music Europe
- Data Observatories Lab

Governance & Budget

The observatories do have not a permanent budget, they are currently financed by Reprex and various user projects. Reprex will release a first business plan in June 2023 that contains a long-term public-private partnership plan based on financial, environmental, and social sustainability, and an ethical, inclusive, and open governance plan.

Read more on budgetary and goverance principles

The Principles of Open Scholarly Infrastructure is an important source of inspiration for our plans. We will evaluate the suitability of this model, bearing in mind that our observatories will not only serve scholarly (educational and research) but commercial purposes, too.
Open Data Directive is an important piece of EU legislation. Our observatories were designed to optimalize this legal instrument for in public benefit and for-profit use cases, too.
Data Governance Act is a new EU law that aims to make more data available by regulating the re-use of publicly/held, protected data, by boosting data sharing through the regulation of novel data intermediaries and by encouraging the sharing of data for altruistic purposes. It enters into force in September 2023.
Open Music Europe is an EU-funded project that aims to bring the Digital Music Observatory from a prototype to an EU-recognized, official European Music Observatory. We will experiment with a governance model that follows the EU’s helix-model to serve as a basis of further observatories.

Observatories

Please join one of our observatories:

Digital Music Observatory: This is a maturing project with an increasing number of open and private data assets, users, stakeholders, aiming for an official EU recognition by 2025.
Cultural Creative Sectors Industries Observatory: Our third observatory is currently seeking early collaborators and funded projects to further develop the idea and release a large enough number of data assets with use cases and publications. We want to create a momentum in 2023.
Green Deal Data Observatory: Our second observatory is in an early stage, but the recent adoption of the CSRD Directive and the need for reliable environmental sustainability data will provide us with excellent growth opportunities.
Social Sustainability Data Observatory: this project is in early concept phase.
Competition Data Observatory: this project is in early concept phase.
Open Collections Net: This project is in it inception phase, but it builds on many years of work with music-related collections.

More detailed overview here.

Join

Institutional

Private

Data curator: Help us curate data – tell us what sort of information is missing from your research agenda. Challenge us and collaborate with us in the crafting of valuable datasets that combine domain knowledge with reproducible, open data research practices (how?—contact.)
Developer: the development works are currently carried out by Reprex and the Turku Data Science Group with open-source collaborators (more—contact.)

Data Observatory Labs

Thu, 02 Mar 2023 18:12:00 +0100

Next: ️> or Space | Previous :️<
Start: Home | Finish: End
Overview: Esc| Speaker notes: S
Zoom: Alt + Click️| Fullscreen: F

🖱 Highlighted text: clickable link (to our project page.)

Questions?

Ask: Email | Keybase

LinkedIn: Daniel Antal | Reprex

Data Observatory Labs

Thu, 02 Mar 2023 13:40:00 +0100

Check out our slides. We are still looking for certain partners.

DataInnoLab is a cooperation of 6 innovative SMEs and startups, a non-profit cultural venue and non-profit cultural heritage organization and ESCP, a leading European Business school which aims to support two important Creative Europe strategic objectives.

DataInnoLab wants to increase the competitiveness and economic potential of the cultural and creative sectors, in particular the audio-visual sector, in a way that mainstreams environmental and social sustainability management.
We also want to help the development and promotion of European cooperation on cultural diversity and heritage in a way that removes institutional, technical, and semantic barriers between the cooperation of the commercial music and audiovisual industry and the music and audiovisual heritage organizations and their supporting institutions.

The DataInnoLab Consortium aims to provide business development tools for the music and audiovisual industry and its supporting heritage and other organizations to leverage the existing instruments of the Culture and Media Strand of the Creative Europe program and the broader New European Bauhaus instruments with higher market revenues and better access to green financing, green investments and insurance.

Our innovators bring into this project early stage products, prototypes, and open source software that can be further developed, packaged into services which are tested in multiple European markets, and eventually scaled for the entire EU internal market and beyond.

In this project, European innovative startups and SMEs will work with the ESCP Business School, a transnational business school for higher education with urban campuses in Germany, France, Great Britain, Italy, Spain and Poland, and its prestigious Blue Factory, its transnational start-up incubator. Established in 1819, ESCP is the world’s oldest business school and celebrated its 200th anniversary in 2019. ESCP Business School has particular expertise in three key academic areas, namely sustainability, entrepreneurship and digitalisation with a foundation of international management.

Our Proposal Data: Open Innovation Labs

Project Acronym: DataInnoLab
Call: CREA-CROSS-2023-INNOVLAB
Funding scheme: CREA-PJG
Proposal ID: SEP-210944562

Table of Contents

WP 1 Coordination

WP 2 Sustainability

WP 3 Competitiveness

WP 4 Open Data

WP 5 Scaling Up

Potential partners

Creative enterprises with good YouTube (+Vimeo and other) distribution and rights management track record.
Market research, particularly survey based in film and television.
Services for private collections/collectors.

Who are we?

In 90 secs: conceptual introduction

⚙️/ subtitles/ 🇳🇱 🇬🇧 🇧🇦 🇨🇿 🇭🇺 🇩🇪 🇱🇹 🇫🇷 🇸🇰 🇪🇸 🇹🇷 + Catalan.

Music Eviota

Tue, 28 Feb 2023 11:13:00 +0200

Next: ️> or Space | Previous :️<
Start: Home | Finish: End
Overview: Esc| Speaker notes: S
Zoom: Alt + Click️| Fullscreen: F

🖱 Highlighted text: clickable link (to our project page.)

Introduction 2. Our Approach 3. Main Features 4. Connected Financial and Sustainability Reporting 5. Why are we developing it? 6. Is There a Film/TV Industry Version? 7. Questions

Introduction

Our approach

Main Features

Connected Financial and Sustainability Reporting

Why Are We Developing Eviota for Music?

Is There A Film Industry Version?

Yes, it is coming! Ask for a demo on Email | Keybase | LinkedIn.

Questions?

Ask: Email | Keybase

LinkedIn: Daniel Antal | Reprex

LineCheck, Milano, IT

Thu, 24 Nov 2022 15:15:00 +0000

Table of Contents

Green Deal, this time for real

Daniel Antal, co-founder of Reprex will participate in the panel about on Thursday, November the 24th at 3.15pm about environment and sustainability in the music industry and introduce Eviota, our simplified, connected financial and sustainability reporting tool.

At LineCheck X, you can find out more about our projects

You can get a free, total value chain based sustainability report. (Get in touch with us before 18 November)
We will demonstrate how you can eradicate packaging waste from your music events by making the packaging edible.
And you can find out how we are trying to find where energy is going down the sewage, literally, with Thermowatt and the Green Deal Data Observatory.

Beta test our sustainabilitiy reporting tool

We are looking for beta testers for our simplified sustainability report that is made from your trial balance. We sign and NDA about your accounting data if you test with us, and you can test with older data, too. Beta testing is free.

Our solution benefits the music MSMEs and CSOs in several ways:

The European Commission estimates that the cost of connecting sustainability and finanical reporting will cost on average €10,000 for corporations. We want to bring down the voluntary reporting costs for MSMEs below €500 euro to benefit from green loans, green insurance and other green financing.
It provides them with a size adequate sustainability management and reporting tool that helps first the management of greenhouse gas emissions, and later sustainable water use, pollutions, biodiversity, and recycling in their entire value chain (for example, it flags environmental risks in the supply base of a festival including equipment rentals, transport, security firms, catering facilities, etc.) by connecting standard accounting documents of the MSME with SNA and EEA science based benchmarks.
Our system will be extendible to management of social sustainability. We are also showcasing harmonized gender inequality data collection with our other project (see below.)

Digital Music Observatory: survive in the music business without a data engineer

⚙️/ subtitles/ 🇳🇱 🇬🇧 🇧🇦 🇨🇿 🇭🇺 🇩🇪 🇱🇹 🇫🇷 🇸🇰 🇪🇸 🇹🇷 + Catalan. If you are there, please leave a 👍, too :)

We will introduce Surveyharmonies, a survey recycling and survey harmonization tool. You can participate in our international gender bias survey in January 2023. Our team will also participate in the KeyChange Creative Lab on Friday, 25 November at 11.00.
You can learn about our Listen Local project, which helps local music ecosystems remain visible on global platforms.
We will offer you a cup of tea or coffee in an edible cup.

Harmonizing surveys on gender (in)equaliaty

You can create better surveys with less cost: you only need to ask the information, or change of information, that is not included in our harmonized datasets. Shorter, better questionnaires, smaller samples sizes, huge cost savings.
When you make questionnaire-based research, you immediately get a history (the same question asked years ago) and an international comparison (the same question asked in other countries.)
Often you do not even have to pay for the survey, because somebody else has already made a similar taxpayer funded research and we can just get the data for you.

Survey harmonization is a powerful research tool to increase the usability of questionnaire-based empirical research. Read more on our blog.

Create Datasets that are Easy to Combine and Reuse

Tue, 22 Nov 2022 09:09:00 +0100

The latest Reprex R package, dataset was released today on the Comprehensive R Archive Network. It is a very early, conceptual package that will help make scientific achievements more open, governmental data easier to find, and store information that can be better combined.

Data interoperability is almost a buzzword, yet we see very few comprehensive, good solutions to apply it. Try to find information on open government portals or on big open science repositories—apart from a few good examples, most datasets are as disorganized as any PC’s hard disk that is collecting dust in a shed.

The dataset package aims to bring together the best practices of data semantics, data organization, and the use of standard metadata to make sure that whatever you store in a data table, it will be immediately available for data analysis, activation, or combination in any new database.

Ambitious? It is, and dataset 0.1.9 is a very experimental product. While our other packages are aimed at intermediate users with a clear use case in mind, dataset at this point is aimed at package developers. Casual or even heavy R users are unlikely to download it as a standalone product. Instead, dataset aims to be a stable developer basis for our existing products, rOpenGov packages, and many new uses.

Download dataset

The metadata aim of dataset it to add standardized metadata to r data.frames, tibbles, data.tables and other similar structured, tabular objects. The organization and semantic objectives are to bring the tidy data concept closer to the datacube model, which is the basis of all statistical data exchanges, and W3C standards, which foster machine-to-machine data communications on the traditional web APIs and the semantic web.

Makes data importing easier and less error-prone;
Leaves plenty of room for documentation automation, resulting in far better reusability and reproducibility;
The publication of results from R following the FAIR principles is far easier, making the work of the R user more findable, more accessible, more interoperable and more reusable by other users;
Makes the placement into relational databases, semantic web applications, archives, repositories possible without time-consuming and costly data wrangling (See From dataset To RDF).

The first official release offers little immediate benefits. However, if you are an R package developer, we can bring you a few steps nearer to releasing your data products in a way that conforms the FAIR metadata principles. We can make a few steps to streamline your data wrangling. Make integration with relational databases easier. To make a step towards the semantic web.

Reprex Wins Audience Prize in The Hague Innovators Challenge

Tue, 15 Nov 2022 18:00:00 +0100

Reprex won the Audience Prize in The Hague Innovators Challenge - with about as many votes as the 7 other contenters combined. We really wanted to prove that big data can work for small organizations, too.

Congratulations for WECO for the startup prize, and our partners, BeeSage, for the pre-startup price. You will hear from BeeSage and Reprex more in the near future.

The winners of our yearly The Hague Innovators challenge are announced! These three startups are shaking up their industry:
1. #WECO
2. #Beesage
3. #Reprex
Congratulations to all of them! 🎉#doinggoodanddoingbusiness #TheHagueInnovators pic.twitter.com/GnRdjRrL5P
— ImpactCity (@impactcity) November 15, 2022

Reprex Nominated for The Hague Innovators Award

Tue, 15 Nov 2022 09:00:00 +0000

Reprex is a finalist for The Hague Innovators Award 2022, and the prize of the audience, in the startup category with our respectable competitors, Sibö, WECO, STHRIVE and ECOBLOQ.

The transition towards a sustainable and inclusive economy depends on collaboration. That is why we are bringing together startups, scale-ups, investors, policymakers, and other impact makers from around the world in The Hague for the 7th edition of ImpactFest.

With the The Hague Innovators Challenge, the municipality of The Hague challenges startups, scale-ups, and students to present their innovative ideas for global issues, as described in the UN Sustainable Development Goals (SDGs).

Vote Reprex

New Data Curators Wanted

Wed, 09 Nov 2022 11:46:00 +0100

A data curator is a contributor in our open collaboration who will be named as a co-creator of tidy, standardized, reusable, FAIR, datasets in his/her field of expertise. Our curators help us vocalize the needs of their domain, be it data-driven beekeeping, or detecting algorithmic biases of recommender systems, and evaluates if the data that we come up with is directly usable and actionable. A data curator is a similar co-author as a “contributor” to open source software or a co-author of a journal article.

Engage with us on LinkedIn @DigitalMusicObs or check out our open data and open repositories, code, tutorials!

Table of Contents

Boost your career without a conflict of interest

Being a data curator does not mean a commercial affiliation with any observatory partners, it is an affiliation to jointly create intellectual property. All our data curators are identified by their ORCiD ideas and named as co-creators in the open science repositories where we make our data available.

We create CC0 data that can be used for commercial, academic, and policy purposes. However, we want to honor the intellectual investment into a shared intellectual property by

delaying the release (for remaining competitive in academic publishing, if our curator is using the data in new articles; or NGOs for their campaign)
creating hybrid assets for commercial users where some elements, particularly the ones that use their proprietary data, may not become open data.

FAIR: Findable, Accessible, Interoperable, and Reusable Digital Assets

Our observatories do not only work with open data.

We want to make data findable, interoperable (to be used in many applications), accessible, and eventually reusable (FAIR), but that does not mean that all data used must be free. Creating and especially regularly updating high-quality data assets requires plenty of intellectual and monetary investment.

We gladly add commercially available data to our observatory if we can share a large enough subset that our peer-reviewers can attest to the data’s high quality, usability, and actionability.

How to become a data curator?

Our handbook for curators a bit of a work in progress, but the onboarding process is clear. Do not worry if you do not use GitHub, it is not necessary, but we story and co-create our assets, including the curator’s handbook on this digital co-working place.

This is an open book that we co-create on GitHub, and if you find any roadblocks, you do not understand something, or have a better idea on how to illustrate or explain things, just make a for to this repo, improve it, add new photos, and send us a pull request. (You need an invite first for editing!)
Here is a starter repository on GitHub. Not mandatory, but if you use GitHub, start here.

In a nutshell:

Please read the entire covenant here.
We need a very brief biography. Name, affiliation, education details, one-line and short biography. Please, send back this bio_template.txt text file. If you know markdown, use this version. The files are identical, but your word processor may not know how to open an .md file.
Your ORCiD to resolve ambiguity with similarly named people. You may use different library or publication service IDs, such as Google Scholar, Publeon, etc, you may provide them, too, but we do need an ORCiD ID, because most of the EU open science infrastructure and the R ecosystem uses this one. If you do not have it, please create one—it only takes a few minutes. Please add it to the bio_template.txt.
Your LinkedIn ID, add it to the bio_template.txt.
You should follow our file naming conventions, and avoid the use of special characters in any file names at all times: , $, :,;,,,., ", ' tick or backtick.
You must send a ile picture that is at least 500px wide (jpg or png format.) It can be bigger, and preferably not a very “narrow” cut, as all avatars will be behind a circular mask (see other curators.)

Find inspiration from other contributors

Why data observatories?

Our data observatories (platform products) cover our R&D and platform costs while giving us access to an expanding range of prime clients. We use 21-st century open-source data engineering solutions, a decentralized data governance method, and web 3.0 technologies to avoid conflicts of interest and prevent the data Sisyphus of error-prone human data wrangling. There is little competition on this service level (there are about 60 UN/EU/OECD recognized data observatories, and almost all of them are managed by a different operator.) This layer is already monetized, and we have proven success. Our unique advantage is a combination of legal and technological skills: understanding legally open data, web 3.0, and data modeling, and the ability to participate in the open-source statistical /scientific software creator community.
We create open-source software applications that fuel our data observatories with unprocessed, open, linked data. We create software for the R statistical environment, which is used in both official statistics and in many business and academic organizations. The production of R software components is a competitive field, but we believe that our position is strong: the vast majority of R packages are lightly or not at all serviced because of the lack of financing.

Reprex produces open-source scientific software, and various collaborative data engineering infrastructures to get legally open governmental data and open science data in a timely, usable format to ecological researchers, and ecotech innovators.

We provide bespoke analytics solutions to our institutional partners in our data observatories. Such bespoke solutions iterate over our existing software components, helping us design better applications within an ever-expanding ecosystem. Providing tailored data-science services would require a large organization without a clear focus. We provide these services on an ad-hoc basis only among institutional partners and users of our data observatories. In these circles, which are often prime clients, we face little or no competition because we are trusted partners and data and solution providers. This is a key to our revenue and market growth.
We develop high-value software-as-service applications that leverage our data observatory assets and our software solution into a novel, commercially valuable uses. Our applications are built around our family of open-source software and generalize our bespoke analytics solutions. We are in a late prototype phase where we already have some revenue and are trying to prepare for scaling up at the correct price with three of our applications. All of our applications are entering into highly competitive market segments. We are building on our ‘unfair’ advantage that we are bundling our solutions with data that is not accessible to competitors, and we can test them in the protected ecosystems of our observatories.

Good to know

FAIR Principles: improve the Findability, Accessibility, Interoperability, and Reuse of digital assets.
DataCite: A persistent, standardized approach to access, identification, sharing, and re-use of datasets—this is our favored way of describing data for future use according to the FAIR principles. Many EU open science repositories will ask your publications with this documentation.
Biblatex is a standard text file used by citation engines, bibliography management tool, and in scientific publication templates. (See for example the Overleaf Biblatex tutorial.
Dublin Core is an older international standard than DataCite, but the two standards greatly overlap. Dublin Core was originally developed by libraries. You often may need to fill out Dublin Core properties for publication.

Watch Our 2-min Introduction

⚙️/ Subtitles/ 🇳🇱 🇬🇧 🇧🇦 🇨🇿 🇭🇺 🇩🇪 🇱🇹 🇫🇷 🇸🇰 🇪🇸 🇹🇷 + Catalan.

Vote Reprex :)

Go to Cast your vote for The Hague Innovators challenge 2022! and choose Reprex :)

Engage with us on LinkedIn @DigitalMusicObs or check out our open data and open repositories, code, tutorials!

OpenMuse Consortium

Sun, 06 Nov 2022 17:52:00 +0100

OpenMuse brings together music industry stakeholders and researchers from 12 European countries. Our partners represent the diversity of the industry, as well as the shared need to find financially, socially, and environmentally sustainable policy and business models in multiple, sometimes-fragmented streams (e.g., live music, composers/publishers, and recordings with producers and performers).

OpenMuse is centerred around the Digital Music Observatory as an open, scholarly data infrastructure that can be used both commercially and non-commercially. We want to replicate our success story in music in the broader creative and cultural industries.

Big Data for All - Building Collaborative Data Observatories

Thu, 03 Nov 2022 17:30:00 +0000

Reprex’s co-founder, Daniel Antal talked in the Eindhoven Innovation Café about these issues. You can watch the recorded version of the the livestream that starts at 5 minutes and 22 seconds:

This is a past event. Check out our forthcoming events or write to Daniel Antal or to antaldaniel. Or send an email.

The event invitation text and links

Big data and AI creates inequalities. It puts historically marginalized people, like ethnic minorities, and womxn, at a disadvantage. Because AI and checking on AI require plenty of data, usually only giant corporations, the wealthiest governments, and university entities can make it work for them. Reprex is a Hague-based, international startup that wants to impact various sustainable development goals by enabling smaller organizations to join their smaller datasets, use open data, create linked available data, and collaboratively make a change.

Reprex is a finalist for the Hague Innovation Award for impact startup (please 🙏, vote for us!). Daniel Antal, one of the co-founders, will talk about their approach to building an international coalition of music organizations to pool data and challenge data monopolies using organizational techniques, a collaboration ethos, and data from the open-source developer world.

Using the example of independent music creators, who often find themselves in a position where it is more expensive to claim their money from global platforms, he will talk about how to reduce inequalities in the world of big data and AI with collaboration on web 3.0. In the Q&A he will take questions on how to apply their know-how, and generally linked open data to other art+tech or creative segments or problems for which everybody is too small, like meeting the Paris Accord greenhouse gas targets bit by bit, small company by small company.

In the Q&A, we can discuss many things

How can Reprex help an individual creator in music, or in fashion and design, or any other area?
What sort of help it can give to researchers, research institutes, specialist consultancies, law firms, and other knowledge-based actors?

What sort of partners is Reprex looking for in Eindhoven?

Check out our projects

Digital Music Observatory and Listen Local
Cultural & Creative Sectors and Industries Observatory and short call for potential partners.
Green Deal Data Observatory and simple, connected, financial and sustainability reporting for creative enterprises and others

Reprex: the impact startup

Check out our accomplishments since the foundation in 2020

Big Data for All: Building Collaborative Data Observatories

Thu, 03 Nov 2022 17:30:00 +0000

Engage with us on LinkedIn @CultDataObs or check out our open data and open repositories, code, tutorials

Table of Contents

Reprex’s co-founder, Daniel Antal talked in the Eindhoven Innovation Café about these issues. You can watch the recorded version of the the livestream that starts at 5 minutes and 22 seconds:

This is a past event. Check out our forthcoming events or write to Daniel Antal or to antaldaniel. Or send an email.

The event invitation text and links

In the Q&A, we can discuss many things

How can Reprex help an individual creator in music, or in fashion and design, or any other area?
What sort of help it can give to researchers, research institutes, specialist consultancies, law firms, and other knowledge-based actors?

What sort of partners is Reprex looking for in Eindhoven?

Check out our projects

Digital Music Observatory and Listen Local
Cultural & Creative Sectors and Industries Observatory and short call for potential partners.
Green Deal Data Observatory and simple, connected, financial and sustainability reporting for creative enterprises and others

Reprex: the impact startup

Check out our accomplishments since the foundation in 2020

Listen Local Nederland

Thu, 03 Nov 2022 17:00:00 +0000

We will carry out the research outlined in our feasibility study within the [OpenMuse] project from 1 January 2023 in an open collaboration with cultural policymakers, the local music ecosystem, and open source developers. Our partner, MXF tries to put the findings into actionable data services in Lithuania and Ukraine with the help of MusicAire. We develop open tools that can be applied in Utrecht, the Hague, Budapest, Tallinn, Vilnius, Bratislava, or anywhere.

This is a past event. Check out our forthcoming events or write to Daniel Antal or to antaldaniel. Or send an email.

Our ongoing project since 2014 is Listen Local. We would like to find out how can a local music ecosystem, or a small scene avoid being colonized by global players on streaming platforms, radio, or in-store music. How can we make sure that music recommendations connect bands in Utrecht with fans in Utrecht? If a Polish band is visiting Utrecht, music lovers will find their show?

We are arriving at opening to Klein Berlijn at 17.00 where you can sit down with a coffee or a beer to chat. No reservation needed. At 19.30 we are getting on our OV fiets and cycle over to the Vechtclub where we will meet people of the local indie scene, Tiny Rooms, and see two independent bands on stage.

As the sales and promotion of recorded music get fully automated, and even the curation of music for live events gets more and more influenced by machine learning outcomes or social media metrics, how can a DIY label remain relevant? How you can run a small club or a label without having to invest in a multi-million euro data engineering team? Make sure that the algorithm will learn successfully your music offering.

‘I feel at home where people care’ Read Gabija’s interview and check out the band with us.

Reprex’s motto is big data for all. We want to fight data inequalities, data monopolies, and make big data and AI work for self-released artists or small labels, too.

Gabija and Daniel are visiting Utrecht to meet DIY musicians, labels, researchers, fans and friends to find new partners for our Listen Local projects. On an intellectual level, we are interested in trustworthy AI, data feminism, and providing a proper digital representation to music and live performances for all. On a more emotional level, we want to meet musicians and music lovers from the indie scene.

We want to help independent artists to find their next audience, and fans to find their next favorite record or a truly fulfilling live music experience. Gabija had a conversation with The Kurws in our Listen Local Interviews series. She asked the band about where they are coming from, where they want to go? Where they are local? And what they have to offer to the people who will join us on 31 October 2022 in Utrecht?

Check out our projects

Digital Music Observatory and Listen Local
Cultural & Creative Sectors and Industries Observatory and short call for potential partners.
Make interviews, get interviewed, write blog posts, or syndicate our content to and from Data & Lyrics.
Don’t forget to vote for Reprex in the Hague Innovators Award competition 2022. The audience voting starts on 1 November 2022.
Download our submission for the competition.
Get in touch!

Reprex: Big Data For All

Mon, 31 Oct 2022 12:00:00 +0000

Reprex is the Hague-based impact startup developing decentralized, modern, web 3.0-compatible data observatories. Our mission is to fulfill parts of the SDG 16 and 17 goals: based on the open collaboration method of open-source software development and open knowledge management, we would like to enable impact makers to contribute to other SDG goals by making AI and big data work for them.

Engage with us on LinkedIn @CultDataObs or check out our open data and open repositories, code, tutorials. 🇭🇺 magyarul

Table of Contents

Watch Our 2-min Introduction

⚙️/ Subtitles/ 🇳🇱 🇬🇧 🇧🇦 🇨🇿 🇭🇺 🇩🇪 🇱🇹 🇫🇷 🇸🇰 🇪🇸 🇹🇷 + Catalan.

How To Vote?

Go to Cast your vote for The Hague Innovators challenge 2022! and choose Reprex :)

Make sure you select Reprex and write in your email (it is safe here.) You need to tick `✅ I’m not a robot’ to be able to select companies. Further instructions 🇬🇧 here 🇭🇺 magyarul itt.

What do we do in the cultural and creative sectors and industries?

We realized in 2021 that most of the hard-to-get and difficult-to-process information sources of music are identical or very similar to those in film, gaming, books, and even fashion. We created a consortium with some of our partners in the music observatory (notably, the Institute for Information Law Research or the University of Amsterdam, and Europeana, the EU’s cultural heritage harmonization body) to launch a broader data observatory for the cultural and creative sector.
We want to use our PPP funding consortium experience, data governance experience, and excellent EU tendering track record of building a broader sister observatory for our music observatory.
Like every one of our observatories, contributes to the SDG 17 partnership goal as we use the open collaboration method developed in open source software and open knowledge development. We contribute to policy coherence and significantly increase the availability of high-quality, timely, and reliable data disaggregated by income, gender, age, race, ethnicity, migratory status, disability, geographic location, and other characteristics relevant to national contexts.
Our data coverage already includes some of the least developed countries.

Product

Our data observatories (platform products) cover our R&D and platform costs while giving us access to an expanding range of prime clients. We use 21-st century open-source data engineering solutions, a decentralized data governance method, and web 3.0 technologies to avoid conflicts of interest and prevent the data Sisyphus of error-prone human data wrangling. There is little competition on this service level (there are about 60 UN/EU/OECD recognized data observatories, and almost all of them are managed by a different operator.) This layer is already monetized, and we have proven success. Our unique advantage is a combination of legal and technological skills: understanding legally open data, web 3.0, and data modeling, and the ability to participate in the open-source statistical /scientific software creator community.
We create open-source software applications that fuel our data observatories with unprocessed, open, linked data. We create software for the R statistical environment, which is used in both official statistics and in many business and academic organizations. The production of R software components is a competitive field, but we believe that our position is strong: the vast majority of R packages are lightly or not at all serviced because of the lack of financing.

We provide bespoke analytics solutions to our institutional partners in our data observatories. Such bespoke solutions iterate over our existing software components, helping us design better applications within an ever-expanding ecosystem. Providing tailored data-science services would require a large organization without a clear focus. We provide these services on an ad-hoc basis only among institutional partners and users of our data observatories. In these circles, which are often prime clients, we face little or no competition because we are trusted partners and data and solution providers. This is a key to our revenue and market growth.
We develop high-value software-as-service applications that leverage our data observatory assets and our software solution into a novel, commercially valuable uses. Our applications are built around our family of open-source software and generalize our bespoke analytics solutions. We are in a late prototype phase where we already have some revenue and are trying to prepare for scaling up at the correct price with three of our applications. All of our applications are entering into highly competitive market segments. We are building on our ‘unfair’ advantage that we are bundling our solutions with data that is not accessible to competitors, and we can test them in the protected ecosystems of our observatories.

Plans in The Hague?

Our message is simple: doing business and doing good for the city of the Hague means a vote for Reprex. We would like to win the Hague Innovators Challenge in 2022 because we believe we could multiply our growth in partnership with the Hague. We have a significant budget to develop our observatories, and our company is already located in the Hague, in Apollo 14, – but most of our team members, not to mention the observatory’s non-data personnel are not based in our beautiful and smart city. The observatories are important platforms for our growth, and they could create a lot more jobs and impact in the city than in our startup company. Should we win the prize, we would spend the 25,000 euros on one thing: to develop our observatories into a real public-private partnership in the Hague, with a permanent office in Apollo 14 or the Hague Humanity Hub.

Reprex’s data observatories, particularly the Green Deal Data Observatory are public-private partnerships that foster the collective collection, processing, peer-review, and reuse of novel big data, like BeeSage’s beehive data, and reusable statistical and environmental data. We hope to place the permanent institution of this PPP in the Hague, which is already the World’s Smartest City, and which wants to remain a global centre of excellence of peace, justice, and sustainability in the era of big data and AI.

Please Choose Reprex in The Hague Innovators Award Online Vote

Sat, 29 Oct 2022 16:17:00 +0200

Engage with us on LinkedIn @CultDataObs or check out our open data and open repositories, code, tutorials. 🇭🇺 magyarul

Table of Contents

How to vote?

Go to Cast your vote for The Hague Innovators challenge 2022!

Make sure you select Reprex and write in your email (it is safe here.) You need to tick `✅ I’m not a robot’ to be able to select companies.

Your vote is not final yet, you must click on a confirmation link to prove that it was you who voted. Go to your email. (Your email is only recorded to avoid double voting, they will not add your address to any marketing databases.)

Please make sure you chose Reprex and click on link to the confirmation your vote.

You receive a text that your code is recorded and you have nothing else to do. (Your address is safe with the municipality of The Hague.)

Thank you very much!

Please share our video message on YouTube among your colleagues and friends.

By pressing the ⚙️ and choosing subtitles you can choose your language. If you are there, please leave a 👍, too :)

The message is the message! We are an ethical data and AI company, and one of our topics is detecting if algorithms are biased towards the English language speakers. We want to teach the computer to understand small languages, and of course, everyone, who is under-represented in data: womxn, former colonial nations.

🇳🇱 🇬🇧 🇧🇦 🇨🇿 🇭🇺 🇩🇪 🇱🇹 🇫🇷 🇸🇰 🇪🇸 🇹🇷 + Catalan.

2 Retweet our appeal from one of our observatory Twitter accounts. For cultural audiences:

🤎 We ask you, humbly, to support us with a vote or by sharing our appeal. We're part of the #opensource, #opendata, and #openscience movement that depends on your support to stay online and thriving, but many of our users simply look the other way🤦🏻‍♀️https://t.co/zdU2i0Jvvn
— Creative Cultural Data Observatory (@CultDataObs) November 1, 2022

The same message for music audiences: @DigitalMusicObs; for green audiences: @GreeDealObs.

Like our LinkedIn page and share our appeal.
Or just send the link to this post from the browser your colleagues and friends.

Why vote for us?

We are finalists in The Hague Innovation Awards with a product offering and a message that big data and AI should work for all: ethnic minorities, small nations, small languages, womxn. We are measuring why certain artists are not getting recommended and paid on global streaming platforms, or why NGOs do not find the right data about fighting greenwashing. We want to help small businesses, civil society organizations, and NGOs who cannot hire a data engineer and a data scientist to fight data monopolies. Who cannot defend themselves from the dark patterns of greedy algorithms?

Get in touch

Check out our events or write to Daniel Antal or to antaldaniel. Or send an email. Thank you!

Listen Local Nederland Meetup in Utrecht

Wed, 26 Oct 2022 16:17:00 +0200

‘I feel at home where people care’ Read Gabija’s interview and check out the band with us.

Reprex’s motto is big data for all. We want to fight data inequalities, data monopolies, and make big data and AI work for self-released artists or small labels, too.

Engage with us on LinkedIn @CultDataObs or check out our open data and open repositories, code, tutorials

Check out our projects

Digital Music Observatory and Listen Local
Cultural & Creative Sectors and Industries Observatory and short call for potential partners.
Make interviews, get interviewed, write blog posts, or syndicate our content to and from Data & Lyrics.
Don’t forget to vote for Reprex in the Hague Innovators Award competition 2022. The audience voting starts on 1 November 2022.
Download our submission for the competition.
Get in touch!

Creative Economy

Thu, 06 Oct 2022 14:40:00 +0200

In the case of Pillar 1 – Music & Economy, we are usually able to locate the source of the data, and we have some experience in processing the data and bringing it to light. We have mapped many perceived data gaps to the various data harmonization projects of GESAC and CISAC, and we will seek cooperation with these organizations and their members to find a secure and voluntary way to retrieve the data.

Within the OpenMuse project, we will demonstrate the capacity of improved data to increase artist revenues within a short- to medium-term timeframe. This pilot project will directly build upon results of CEEMID and other prior projects undertaken by Artisjus, SOZA, and Reprex. The partners will replicate a full market model used to measure the use of recorded music in Hungary and Slovakia. Artisjus and SOZA will revise and validate this full market model using fully standardised data, collected and harmonised via the OpenMusE music data software ecosystem.

By bringing the model into line with the standards of Open Policy Analysis, open science, and open-source software development, the consortium will enable its transfer to the Bulgarian market with partner Musicautor. In all three countries, the Consortium will generate replicable, detailed “live policy documents” including gross value added, employment, and other important economic variables which will be integrated into a precise valuation of all uses of recorded music within the national context. These valuations will depend on standardised data from GESAC, CISAC, and potentially from IFPI, Aepo-Artis, and SCAPR, and provided via partners Artisjus, SOZA, and MUSICAUTOR. Because the “live policy documents” will be generated semi-automatically using the software ecosystem introduced above, similar documents could be created nearly instantly for any GESAC/CISAC/IFPI national member organisation that provided comparable data on recorded music within their national context. The OpenMusE Report on the European Music Economy will be published as a “live policy document” including the Bulgarian, Hungarian, and Slovakian valuations, as well as valuations for any national market in which a representative organisation consents to provide data.

In other cases, we will rely on on the 2019/1024/EU Open Data Directive to access, process, document, and disseminate the data.

The music.dataobservatory.eu monitors the music markets with an economic methodology: we not only measure market volumes and prices, but we also measure both demand- and supply side indicators so that we can forecast future market volumes or prices. (See Innovation pillar — 4.2 Forecasting)

Music is not a purely market activity. Music and music services have consumers who pay for it one way or another, but there are non-market forms of music (for example, in liturgy) and music markets are overshadowed by a very large illegal market. Therefore, instead of “music consumption” we use the more appropriate statistical terms of access and participation. we do measure non-market form,s but currently we do not address teh data gap the “impact of the non-profit sector”.

Pillar 1 - Music Economy
Topic	Description
pillar	problem	availability	feasibility
Pillar 1	Value of music sector	One-off	EY study on the cultural and creative industries (2015)
Pillar 1	Value of music sector	One-off or one recast	CEEMID studies on national music economies
Pillar 1	Employment	One-off	EY study on the cultural and creative industries (2015)
Pillar 1	Employment	One-off or one recast	CEEMID national music industry reports
Pillar 1	Employment	Annual	Eurostat, lacking granularity
Pillar 1	Employment	Data gap	Absence of granularity on the employment of the various sub-sectors, in particular in defining the roles of the various sub-sectors and the importance of the not-for-profit sector in terms of employment.
Pillar 1	Employment	Data gap	HU, SK pilot successful to add granularity.
Pillar 1	Value of music sector	Data gap	No EU-level assessment since 2015
Pillar 1	Strucutre of the market	Data gap	Absence of pan-European data detailing the number of companies, employees, revenues for the sector and the subsectors.
Pillar 1	The impact of the not-forprofit sector on the overall economy of the music sector	Data gap	No data available on the specific impact of the not-for-profit sector, especially in the live music sub-sector
Pillar 1	Recorded music	Subject to partnership with IFPI	IFPI
Pillar 1	Authors and publisher’s stream	CISAC partnership	CISAC
Pillar 1	Authors and publisher’s stream	GESAC	GESAC
Pillar 1	Recorded music stream - performer rights	Data gap	No aggregated data on neighbouring rights collections Partner with AEPO-ARTIS and SCAP.
Pillar 1	Music publishing	Data gap	No aggregated data on the music European music publishing business
Pillar 1	Synchronisation rights	Data gap	IFPI data available on the recorded music side but not on the publishing side.
Pillar 1	Independent music companies	Data gap	No aggregated data on the independent music sector (value, number of companies, employees, etc.)
Pillar 1	Live music	Data gap	Some data is compiled by Live DMA, ETEP or Yourope, but there is no aggregated data on the pan-European live music sector listing the value of the market, the number and size of venues and shows, number of festivals, share of European artists, among other data points.
Pillar 1	Exports	Data gap	No pan-European data on the export flows between EU countries and outside the EU.
Pillar 1	Exports	Data gap	Embedded cultural tourism export.
Pillar 1	Music retail	Data gap	Granular data on some countries via retail associations (UK, France, Germany) but no pan-European aggregated data.
Pillar 1	Financing of the music sector	Data gap	No aggregated data on how the sector is financed (from investment fund to bank loans and subsidies).
Pillar 1	Live music regulation	Data gap	No aggregated information available on the various legal and tax systems within the EU applied to the live music sector.
Pillar 1	Copyright regulations and evolution of copyright regimes	Data gap	Although many copyright laws applicable in Europe originate from the Commission, there are few instruments available to monitor the state of copyright regulation across the EU

Art+Tech

Thu, 06 Oct 2022 08:12:00 +0200

We will connect to several innovation topics in music by providing research, data, and evidence. Building on the work of the Blockchain & Society research project of our Consortium member, the Institute for Information Law IViR at the University of Amsterdam, we will address issues related to NFT and blockchain technologies in music.
Connecting to the work of various music stakeholders in Finland, we will experiment with the safe data sharing of confidential data, avoiding the extreme pitfalls of ‘radical openness’ and ‘closed data’. With the University of Bologna, and eight years of experience in sensitive data sharing with the Digital Music Observatory (formerly: CEEMID), we will create various statistical methods to create anonymized and synthetic datasets that do not endanger business or private data.
We are going to design indicators and carry out a landscape mapping with Music Tech Europe, a new organization that aims to become the representative of music startups and music ecosystems in the European Union.


Topic	Description
pillar	problem	availability	feasibility
Pillar 4	Several reports cover the evolution of music and tec	Available	Music Ally, Midia Research…
Pillar 4	Blockchain and music	Data gap	No authoritative assessment of the impact of Blockchain on the music sector and of the EU-powered initiatives linking Blockchain and music.
Pillar 4	Artificial intelligence, machine learning and music	Data gap	No authoritative assessment of the impact of AI and machine learning on the music sector and of the EUpowered initiatives linking AI and music.
Pillar 4	Future of streaming	Data gap	No authoritative assessment of the future development of streaming and its impact on the EU music sector
Pillar 4	Digital revenues in the music sector and the relevant business models	Data gap	In the UK the report ‘Dissecting the digital dollar’ commission by the MMF to CMU, but no similar European study.
Pillar 4	Mapping the flow of digital revenues in the music sector and the relevant business models in Europe	Data gap	Digital distribution of music has introduced new complex business models that are not always transparent.
Pillar 4	Music start-ups in the EU	Data gap	No authoritative mapping of start-ups involved in music at EU level.
Pillar 4	The impact of artists’ ‘do it yourself’ culture on the economy of the sector	Data gap	Although more artists are going to market with their music without the traditional support of labels, there is no overview of how deep the trend is, how it affects the music ecosystem or what policy frameworks can or should support such activity.
Pillar 4	Funding mechanisms for music	Data gap	No regular overview of the funding schemes in Europe relevant to the music business.

Cultural Diversity & Circulation

Thu, 06 Oct 2022 08:12:00 +0200

WP2 Music diversity and circulation will culminate in a pilot project conducted in Slovakia, Lithuania, and Ukraine by SOZA, MXF, and MEU.

The WP2 pilot project will demonstrate the feasibility of increasing local artist representation in radio and streaming, via trustworthy AI and evidence-based regulatory policy. In the “Listen Local” project, partners SOZA and Reprex created the Slovak Demo Music Database (SDMDb) as a reference database to measure the market share of Slovak music in radio and streaming, and created a prototype of a recommendation system that meets trustworthy AI criteria and is able to achieve similar cultural diversity goals as laid out by Slovak media legislation in 2015: i.e., to achieve 15% “Slovak” market share in radio playlists.

Culture And Society

Thu, 06 Oct 2022 08:12:00 +0200

In the OpenMuse WP3 Music, society, and citizenship will culminate in a pilot project conducted in Italy by MIH. The WP3 pilot project will demonstrate the ability of reproducible research and innovation tools to help MSMEs comply with emerging social and environmental sustainability regulations, at a viable cost. Within the pilot project, MIH will utilise the OpenMuse open music data software ecosystem to create an environmental impact report – as will soon be required of many European MSMEs. For this, we will add new functionality to software that was originally developed for economic and environmental impact assessments for the Hungarian and Slovak Music Industry Reports. In addition, more detailed social and governance reporting will be carried out with MIH. As founders of the Keychange program for gender equality in music, MIH is dedicated to transferable piloting on environmental sustainability (covering greenhouse gas and water use at a minimum), social sustainability (focusing on the gender pay gap), and governance sustainability (focusing on live performance kickbacks and recorded performance payola schemes as corruption risks in the music industry).

The environmental, social, and governance sustainability data collection will be piloted in Italy and transferred within the project lifespan to Bulgaria. As in the WP1 and WP2 pilot projects, the methods and software utilised will be fully open source, and thus open to, and replicable by, any national stakeholders who can collect the inputs required.

Human agency and oversight

AI systems should empower human beings, allowing them to make informed decisions and fostering their fundamental rights. At the same time, proper oversight mechanisms need to be ensured, which can be achieved through human-in-the-loop, human-on-the-loop, and human-in-command approaches

Technical Robustness and Safety

AI systems need to be resilient and secure. They need to be safe, ensuring a fall back plan in case something goes wrong, as well as being accurate, reliable and reproducible. That is the only way to ensure that also unintentional harm can be minimized and prevented.

Privacy and Data Governance

Besides ensuring full respect for privacy and data protection, adequate data governance mechanisms must also be ensured, taking into account the quality and integrity of the data, and ensuring legitimised access to data.

Transparency

The data, system and AI business models should be transparent. Traceability mechanisms can help achieving this. Moreover, AI systems and their decisions should be explained in a manner adapted to the stakeholder concerned. Humans need to be aware that they are interacting with an AI system, and must be informed of the system’s capabilities and limitations.

Diversity, Non-Discrimination and Fairness

Unfair bias must be avoided, as it could could have multiple negative implications, from the marginalization of vulnerable groups, to the exacerbation of prejudice and discrimination. Fostering diversity, AI systems should be accessible to all, regardless of any disability, and involve relevant stakeholders throughout their entire life circle.

Societal and Environmental Well-Being

AI systems should benefit all human beings, including future generations. It must hence be ensured that they are sustainable and environmentally friendly. Moreover, they should take into account the environment, including other living beings, and their social and societal impact should be carefully considered.

Accountability

Mechanisms should be put in place to ensure responsibility and accountability for AI systems and their outcomes. Auditability, which enables the assessment of algorithms, data and design processes plays a key role therein, especially in critical applications. Moreover, adequate an accessible redress should be ensured.

Application Development

Tue, 30 Aug 2022 13:52:00 +0200

We are building and ecosystem of open data, open software and trustworthy algorithms around our data observatories. We collaborate with scientific software developers, and their communities, such as rOpenGov. Our software tools have many thousand users, but they require coding skills. We will soon start to deploy more user-friendly applications that can be used by sustainability reporting professionals, or lawyers preparing factual green disclosures.

We are currently developing two applications.

Our Eviota project, which builds a multi-language reporting interface for preventing greenwashing. It will have a user-frontend for our environmental (and economic) impact assessment software tool, iotables. The iotables R package implements most of the functionality laid out in the economic and environmental input-output analysis features in the Eurostat and the respective UN statistical technical guidance on the topic.
Our smart document are always alive: they contain code that searches for data updates or changes in the law. It is a document that includes reproducible research code and makes sure that your document contains the latest information. It is a resource that, when uploaded to the web 3.0, cannot only be downloaded, but finds itself its audience, places itself into global libraries, data exchanges, automated websites.

Application Development

Tue, 30 Aug 2022 13:52:00 +0200

The first application is developed in our Eviota project, which builds a multi-language reporting interface for preventing greenwahsing. It will have a user-frontend for our environmental (and economic) impact assessment software tool, iotables. The iotables R package implements most of the functionality laid out in the economic and environmental input-output analysis features in the Eurostat and the respective UN statistical technical guidance on the topic.

Please stay tuned – we will showcase our first app soon.

Parntership Proposal for Europeana & Network

Wed, 17 Aug 2022 12:00:00 +0000

We are competing for The Hague Innovators’ Award as an impact startup with the support letter of Europeana, trying to find alternative funding for our shared project (next slide.)
2-minute, popular 🖱️ explainer video with ⚙️/subtitles/ 🇳🇱 🇬🇧 🇧🇦 🇨🇿 🇭🇺 🇩🇪 🇱🇹 🇫🇷 🇸🇰 🇪🇸 🇹🇷 🇵🇱 🇸🇪 + Catalan. (🙏🏻 share it).
Please 🙏🏿 vote for Reprex by clicking here, and if you like us, share the appeal from 🖱️ here on Twitter or LinkedIn, too.

Open Call for Collaboration on Cultural Heritage Data

Next >️ or Space| Previous <
Start Home | Finish End
Overview Esc | Fullscreen F
Zoom Alt + Click

The 🖱️ CCSI Data Observatory already has some data assets on Zenodo, and we can upgrade its API (both as Rest API with datacube and with a simple RDF serialization)
We are creating high quality, SDMX compatible, linkable, open science data about cultural heritage objects and other cultural heritage policy data. We would like to connect on knowledge graphs to actual content.
Revisit some modest deliverables of 🖱️ RECREO and seek new funding.

The 🖱️ CfR HORIZON-CL2-2023-HERITAGE-01-03: Re-visiting the digitisation of cultural heritage: What, how and why?. Our appeal is to avoid duplication of cultural heritage and cultural industries digital infrastructure, and join forces with rights management, and commercial knowledge graphs to make heritage more reusable and valued.
🖱️ CfR HORIZON-CL2-2023-HERITAGE-01-02: Cultural and creative industries for a sustainable climate transition. We have our open source software and app for environmental impact analysis and value chain analysis of the music industry that can be applied to the entire cultural sector.

Listen Local Feasibility Sutdy | Listen Local Judaica Demo

What are data observatories?

There are more than 60 functional, and about 20 already discontinued data observatories, i.e. long-term, usually triangular (business, academic, policy) data collection institutions recognized by the EU, OECD or UNESCO, including the European Observatory on Infringements of Intellectual Property Rights of the EU or the European Audiovisual Observatory of the Council of Europe.

Do it Smarter

They usually do not exchange standard data with statistical agencies, they are not synchronized on knowledge graphs of the Europeana or national libraries, and their research output is usually not to be found on open science repositories.
The Hague is the winner of the World Smart City Award 2021, and we would like to attract the planned European Music Observatory and other, EU/UNESCO recognized institutions into the town building on the innovations of Reprex and the ecosystem of the Hague.

Strategic objectives

Develop our data observatories as Open Scholarly Infrastructure.
Place our Digital Music Observatory, Cultural and Creative Data Observatory, and Green Deal Data Observaotry on knowledge graphs of Europeana, Wikidata, and other open knowledge systems.
Harmonize research artefacts with open repositories such as Zenodo and Figshare.
Achieve EU/UNESCO/OECD recognition for our self-governing, triangular, science-policy-bushiness triangular data ecosystems as data observatories.

Possible Collaboration

Connect national collective management organization, national library, and various services (Spotify, YouTube) to make the national repertoire more visible
Create use statistics for cultural diversity policies and monitoring local content regulations
Provide best practice example and open source tools for replication
Support research automation

Web 3.0 & FAIR

FAIR metadata: Dublin Core & DataCite referential metadata
Integration to FigShare and Zenodo for automated releases and publications
Supported with optional, open source APIs to retrieve the data
Supported with RDF serialization

Dissemination Support

Support automated publishing and releasing of data, visualizations, newsletters, and long-form documentation in auto-refreshing websites, blogposts, or articles, or even books.
Develop an ecosystem of open source software that helps the professional collection, processing, documentation of data conforming the Data Governance Act, and supporting data sharing and data altruism.

Questions?

LinkedIn: Daniel Antal - Digital Music Observatory - Cultural Creative Sectors Industries

Slides

Wed, 17 Aug 2022 12:00:00 +0000

Data Observatory 3.0

Controls

Next: Right Arrow or Space
Previous: Left Arrow
Start: Home
Finish: End
Overview: Esc
Speaker notes: S
Fullscreen: F
Zoom: Alt + Click
PDF Export: E

What are data observatories?

There are more than 60 functional, and about 20 already discontinued data observatories, i.e. long-term, usually triangular (business, academic, policy) data collection institutions recognized by the EU, OECD or UNESCO, including the European Observatory on Infringements of Intellectual Property Rights of the EU or the European Audiovisual Observatory of the Council of Europe.

Do it Smarter

They usually do not exchange standard data with statistical agencies, they are not synchronized on knowledge graphs of the Europeana or national libraries, and their research output is usually not to be found on open science repositories.
The Hague is the winner of the World Smart City Award 2021, and we would like to attract the planned European Music Observatory and other, EU/UNESCO recognized institutions into the town building on the innovations of Reprex and the ecosystem of the Hague.

Strategic objectives

Develop our data observatories as Open Scholarly Infrastructure
Place our Digital Music Observtory, Cultural and Creative Data Observatory, and Green Deal Data Observaotry on knowledge graphs of Europeana, Wikidata, and other open knowledge sytems
Harmonize research artefacts with open repositories such as Zenodo and Figshare.
Achieve EU/UNESCO/OECD recognition for our self-governing, triangular, science-policy-busines triangular data ecosystems as data observatories

Digital Music Observatory

Listen Local in Horizon Europe OpenMuse WP Diversity, Creative Europe MusicAIRE: connected and curated data on 10,000s of music works
Our aim is to describe the entire, currently legally available music repertoire of Slovakia and Lithuania at first, and a large part of Ukraine.
Connected with name authorities, web services.

Possible Use Case

Connect national collective management organization, national library, and various services (Spotify, YouTube) to make the national repertoire more visible
Create use statistics for cultural diversity policies and monitoring local content regulations
Provide best practice example and open source tools for replication

Creative and Cultural Sectors Industries Data Observatory

Technical Features

Reprex | Documentation

FAIR

FAIR metadata: Dublin Core & DataCite referential metadata. Our dataset R software package is under review in rOpenSci and will be added to the rOpenGov ecosystem. It will greatly facilitate the automatic release of any research output from the R statistical system.
Integration to FigShare and Zenodo for automated releases and publications.

Web 3.0

supported with optional, open source APIs to retrieve the data
supported with RDF serialization

Dissemination Support

support automated publishing and releasing of data, visualizations, newsletters, and long-form documentation in auto-refreshing websites, blogposts, or articles, or even books.
develop an ecosystem of open source software that helps the professional collection, processing, documentation of data conforming the Data Governance Act, and supporting data sharing and data altruism.

Research Automation

Research automation

support research automation

Questions?

LinkedIn: Daniel Antal - Digital Music Observatory

Slides

Wed, 17 Aug 2022 12:00:00 +0000

Data Observatory 3.0

Controls

Next: Right Arrow or Space
Previous: Left Arrow
Start: Home
Finish: End
Overview: Esc
Speaker notes: S
Fullscreen: F
Zoom: Alt + Click
PDF Export: E

What are data observatories?

There are more than 60 functional, and about 20 already discontinued data observatories, i.e. long-term, usually triangular (business, academic, policy) data collection institutions recognized by the EU, OECD or UNESCO, including the European Observatory on Infringements of Intellectual Property Rights of the EU or the European Audiovisual Observatory of the Council of Europe.

Do it Smarter

They usually do not exchange standard data with statistical agencies, they are not synchronized on knowledge graphs of the Europeana or national libraries, and their research output is usually not to be found on open science repositories.
The Hague is the winner of the World Smart City Award 2021, and we would like to attract the planned European Music Observatory and other, EU/UNESCO recognized institutions into the town building on the innovations of Reprex and the ecosystem of the Hague.

Strategic objectives

Develop our data observatories as Open Scholarly Infrastructure
Place our Digital Music Observtory, Cultural and Creative Data Observatory, and Green Deal Data Observaotry on knowledge graphs of Europeana, Wikidata, and other open knowledge sytems
Harmonize research artefacts with open repositories such as Zenodo and Figshare.
Achieve EU/UNESCO/OECD recognition for our self-governing, triangular, science-policy-busines triangular data ecosystems as data observatories

Digital Music Observatory

Listen Local in Horizon Europe OpenMuse WP Diversity, Creative Europe MusicAIRE: connected and curated data on 10,000s of music works
Our aim is to describe the entire, currently legally available music repertoire of Slovakia and Lithuania at first, and a large part of Ukraine.
Connected with name authorities, web services.

Possible Collaboration

Connect national collective management organization, national library, and various services (Spotify, YouTube) to make the national repertoire more visible
Create use statistics for cultural diversity policies and monitoring local content regulations
Provide best practice example and open source tools for replication

Creative and Cultural Sectors Industries Data Observatory

Possible Collaboration

The CCSI Data Observatory already has some data assets on Zenodo, and we can upgrade its API (both as Rest API with datacube and with a simple RDF serialization)
Create use statistics for cultural heritage objects and other cultural heritage policy data
Revisit some modest deliverables of RECREO and seek new funding.

Green Deal Data Observatory

Possible Collaboration

The Green Deal Data Observatory is currently developed to provide free or very accessible environmental, social and governance reporting tools to the cultural sector.
It could also be used to provide ecological context to cultural heritage objects (CHO) for greater awareness.

Technical Features

Reprex | Documentation

FAIR

FAIR metadata: Dublin Core & DataCite referential metadata
Integration to FigShare and Zenodo for automated releases and publications

Web 3.0

supported with optional, open source APIs to retrieve the data
supported with RDF serialization

Dissemination Support

support automated publishing and releasing of data, visualizations, newsletters, and long-form documentation in auto-refreshing websites, blogposts, or articles, or even books.
develop an ecosystem of open source software that helps the professional collection, processing, documentation of data conforming the Data Governance Act, and supporting data sharing and data altruism.

Research Automation

Research automation

support research automation

Questions?

LinkedIn: Daniel Antal - Digital Music Observatory

dataset: Create Interoperable FAIR Datasets

Thu, 11 Aug 2022 00:00:00 +0000

Interoperable, FAIR datasets

The primary aim of dataset is create well-referenced, well-described, interoperable datasets from data.frames, tibbles or data.tables that translate well into the W3C DataSet definition within the Data Cube Vocabulary in a reproducible manner. The data cube model in itself is is originated in the Statistical Data and Metadata eXchange, and it is almost fully harmonzied with the Resource Description Framework (RDF), the standard model for data interchange on the web[^1].

A mapping of R objects into these models has numerous advantages:

Makes data importing easier and less error-prone;
Leaves plenty of room for documentation automation, resulting in far better reusability and reproducability;
The publication of results from R following the FAIR principles is far easier, making the work of the R user more findable, more accessible, more interoperable and more reusable by other users;
Makes the placement into relational databases, semantic web applications, archives, repositories possible without time-consuming and costly data wrangling (See From dataset To RDF).

Our package functions work with any structured R objects (data.fame, data.table, tibble, or well-structured lists like json), however, the best functionality is achieved by the (See The dataset S3 Class), which is inherited from data.frame().

Contact

For contact information, contributors, see the package homepage.

Code of Conduct

Please note that the dataset project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.

stacodelists: use standard, language-independent variable codes to help international data interoperability and machine reuse in R

Wed, 29 Jun 2022 08:12:00 +0100

The goal of statcodelists is to promote the reuse and exchange of statistical information and related metadata with making the internationally standardized SDMX code lists available for the R user. SDMX – the Statistical Data and Metadata eXchange has been published as an ISO International Standard (ISO 17369). The metadata definitions, including the codelists are updated regularly according to the standard. The authoritative version of the code lists made available in this package is https://sdmx.org/?page_id=3215/.

Purpose

Cross-domain concepts in the SDMX framework describe concepts relevant to many, if not all, statistical domains. SDMX recommends using these concepts whenever feasible in SDMX structures and messages to promote the reuse and exchange of statistical information and related metadata between organisations.

Code lists are predefined sets of terms from which some statistical coded concepts take their values. SDMX cross-domain code lists are used to support cross-domain concepts. What are these cross-domain coded concepts?

Geographical codes, like NL: the Netherlands in the CL_AREA code list.
Standard industry codes J631 for Data processing, hosting and related activities in Europe. (NACE Rev 2 in Europe, beware, it is J592in Australia and New Zealand, see CL_ACTIVITY_ANZSIC06.)
Occupations, like OC2521 for Database designers and administrators in CL_OCCUPATIONS
Time fomatting standards, like CCYY for annual data series in CL_TIME_FORMAT.

Check out the available codlists on the package homepage.

The use of common code lists will help users to work even more efficiently, easing the maintenance of and reducing the need for mapping systems and interfaces delivering data and metadata to them. A very obvious advantage of using the code systems is that you can retrieve data from national sources indifferent of the natural language used in North Macedonia, Japan, the U.S. or the Netherlands. While the data labels may change to be locally human-readable, computers and geeks can read the codes and understand them immediately. Provided that they use the standard codes.

Our data observatories are rolling out SDMX coding across all datasets to help data ingestion and interoperability, data findability and data reuse. statcodelists can help the use of standard SDMX codes in your R workflow–both for downloading data from statistical agencies and to produce publication-ready datasets that the rest of the world (and even APIs) will understand.

Installation

You can install statcodelists from CRAN:

install.packages("statcodelists")

Further recommended code values for expressing general statistical concepts like not applicable, etc., can be found in section Generic codes of the Guidelines for the creation and management of SDMX Cross-Domain Code Lists.

For further codelists used by reliable statistical agency but not harmonized on SDMX level please consult the SDMX Global Registry Codelists page.

The creator of this package is not affiliated with SDMX, and this package was has not been endorsed by SDMX.

Code of Conduct

Please note that the statcodelists project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Engage with us on LinkedIn @CultDataObs or check out our open data and open repositories, code, tutorials!

links:

icon: twitter icon_pack: fab name: Follow url: https://twitter.com/CultDataObs/
icon: linkedin icon_pack: fab link: https://www.linkedin.com/company/80644612/
icon: fa-solid fa-code icon_pack: fas name: Code & Tutorials link: https://statcodelists.dataobservatory.eu/
icon: github icon_pack: fab name: Contributions, Feedback & Bug Reports link: https://github.com/antaldaniel/statcodelists/

statcodelists: Use Standardized Statistical Codelists

Tue, 28 Jun 2022 00:00:00 +0000

Retrospective data harmonization

The aim of retroharmonize is to provide tools for reproducible retrospective (ex-post) harmonization of datasets that contain variables measuring the same concepts but coded in different ways. Ex-post data harmonization enables better use of existing data and creates new research opportunities. For example, harmonizing data from different countries enables cross-national comparisons, while merging data from different time points makes it possible to track changes over time.

Retrospective data harmonization is associated with challenges including conceptual issues with establishing equivalence and comparability, practical complications of having to standardize the naming and coding of variables, technical difficulties with merging data stored in different formats, and the need to document a large number of data transformations. The retroharmonize package assists with the latter three components, freeing up the capacity of researchers to focus on the first.

Specifically, the retroharmonize package proposes a reproducible workflow, including a new class for storing data together with the harmonized and original metadata, as well as functions for importing data from different formats, harmonizing data and metadata, documenting the harmonization process, and converting between data types. See here for an overview of the functionalities.

The new labelled_spss_survey() class is an extension of haven’s labelled_spss class. It not only preserves variable and value labels and the user-defined missing range, but also gives an identifier, for example, the filename or the wave number, to the vector. Additionally, it enables the preservation – as metadata attributes – of the original variable names, labels, and value codes and labels, from the source data, in addition to the harmonized variable names, labels, and value codes and labels. This way, the harmonized data also contain the pre-harmonization record. The stored original metadata can be used for validation and documentation purposes.

The vignette Working With The labelled_spss_survey Class provides more information about the labelled_spss_survey() class.

In Harmonize Value Labels we discuss the characteristics of the labelled_spss_survey() class and demonstrates the problems that using this class solves.

We also provide three extensive case studies illustrating how the retroharmonize package can be used for ex-post harmonization of data from cross-national surveys:

The creators of retroharmonize are not affiliated with either Afrobarometer, Arab Barometer, Eurobarometer, or the organizations that designs, produces or archives their surveys.

We started building an experimental APIs data is running retroharmonize regularly and improving known statistical data sources. See: Digital Music Observatory, Green Deal Data Observatory, Economy Data Observatory.

Citing the data sources

Our package has been tested on three harmonized survey’s microdata. Because retroharmonize is not affiliated with any of these data sources, to replicate our tutorials or work with the data, you have download the data files from these sources, and you have to cite those sources in your work.

Afrobarometer data: Cite Afrobarometer Arab Barometer data: cite Arab Barometer. Eurobarometer data: The Eurobarometer data Eurobarometer raw data and related documentation (questionnaires, codebooks, etc.) are made available by GESIS, ICPSR and through the Social Science Data Archive networks. You should cite your source, in our examples, we rely on the GESIS data files.

Citing the retroharmonize R package

For main developer and contributors, see the package homepage.

This work can be freely used, modified and distributed under the GPL-3 license:

citation("retroharmonize")
#> 
#> To cite package 'retroharmonize' in publications use:
#> 
#>   Daniel Antal (2021). retroharmonize: Ex Post Survey Data
#>   Harmonization. R package version 0.1.17.
#>   https://retroharmonize.dataobservatory.eu/
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Manual{,
#>     title: {retroharmonize: Ex Post Survey Data Harmonization},
#>     author: {Daniel Antal},
#>     year: {2021},
#>     doi: {10.5281/zenodo.5006056},
#>     note: {R package version 0.1.17},
#>     url: {https://retroharmonize.dataobservatory.eu/},
#>   }

Contact

For contact information, contributors, see the package homepage.

Code of Conduct

Please note that the retroharmonize project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.

Developing a software-as-service solution for micro-, and small enterprises

Thu, 09 Jun 2022 09:40:00 +0100

Engage with us on LinkedIn @DigitalMusicObs or check out our open data and open repositories, code, tutorials

The music sector must increase its environmental and social (ESG) sustainability management to meet the challenges of the climate emergency and to make the music sector a fairer, more just workplace for womxn and artists coming from minorities, small countries. The European Union will make target setting and audited reporting mandatory in environmental and social sustainability for large companies. The application of these new accounting, reporting and disclosure rules are optional for the music sector where almost all entities are micro-, or small enterprises and civil society organizations.

Even if music organizations are not pushed by regulators to adopt these new standards, it is in their best interest to take the initiative on the principle of subsidiarty, and develop tools that can be applied as an extension to their simplified financial and tax reporting. Music organizations and businesses that can prove that they are making progress in reducing their carbon footprint, making their water use more sustainable, and they provide equal opportunities for womxn, they will be eligible for new, green bank and insurance products (which are particularly important in live music) and can attract new sponsors and donors.

Compliance with these new rules is very costly, because tools are being developed for stock-exchange listed big companies and financial institutions. The Commission’s impact assessment (SWD/2021/150 final) estimates the cost of compliance with the Corporate Social Responsibility Directive exceeding 4 bn euros for the European companies or around 10,000 euros per company. Reprex, working together with large accounting, audit and value-based banking partners, scientific, research and industry partners in the Digital Music Observatory open knowledge collaboration, hopes to bring down this cost below 500 euros, which will immediately pay off when a music organization receives green money.

We are working on a simple interface that can connect the accounting system of micro and small enterprises with new methodologies, starting with greenhouse gas reporting with Reprex’s open source EEIO application iotables. We will keep many aspects of our software and data solution open, so that later methodological innovations and scientific achievements can be easily incorporated into the system. Reprex’s minimum viable product will be created in four iteration rounds in Malta, Czechia, Bulgaria and Belgium. However, our testing is open for any amount of donations to any music entities in the European Union who can provide input data in English or Dutch, or be able to pay for their translation and localization costs.

Link: Final List of Awareded Projects by MusicAIRE

Music Eviota

Thu, 09 Jun 2022 09:40:00 +0100

Our minimum viable product will create sustainability reports (or report components) for greenhouse gases and sustainable water use with applying the Global GHG Accounting & Reporting Standard for the Financial Industry and EFRAG’s proposed concept on connecting European accounting standards and information with sustainability. We will help small music organizations in their sustainability reporting, where detail data and reporting standards are only available for greenhouse gas emissions. The Music Eviota project is supported by the MusicAIRE.

Open collaboration

Our project is based on open collaboration. Our proposal, if funded, will provide us with resources to supply further music businesses, music civil society organizations and researchers with high-quality data (during the duration of the project for free.) We are already looking for interested parties to put our data and research projects into use and validate their usability and quality in real-life policy or business development scenarios.

Why are we developing this service?

The European Green Deal, which includes the proposed Corporate Sustainability Reporting Directive, and the sustainable finance package, aims to set the European economy on a permanent decarbonization and sustainability increasing path with adjusting the rules how economic activities are financed by bank loans, insurance, investments, and direct subsidies. From 2023, it will be cheaper to get loans, insurance, and other types of funding for organizations that can prove that they follow the environmental, social and governance path set out in the Paris Agreement and other UN, OECD, and EU agreements.

Correct and reliable sustainability management will come with many financial advantages and increased responsibility. The European Financial Reporting Advisory Board is currently preparing the new combined financial and sustainability reporting standard that will be used in banks, insurance, investment, granting, and the large companies of Europe in their entire supply and purchaser chain. The European Commission estimates that compliance costs until the end of 2023 will amount to 4 billion euros, with reporting and auditing costs mounting 10,000 euros per organization. While music small and medium sized organizations (MSMEs) and limited liability civil society organizations (CSOs) will be exempted from mandatory sustainability management and audited reporting, they can still comply in a non-audited and voluntary way.

Our solution benefits the music MSMEs and CSOs in several ways:

It provides them with a size adequate sustainability management and reporting tool that helps first the management of greenhouse gas emissions, and later sustainable water use, pollutions, biodiversity, and recycling in their entire value chain (for example, it flags environmental risks in the supply base of a festival including equipment rentals, transport, security firms, catering facilities, etc.) by connecting standard accounting documents of the MSME with SNA and EEA science based benchmarks.
Our system will be extendible to management of social sustainability. Our previous research shows that particularly the live music industry that needs a large workforce, suffers from underuse of, and discrimination of female workers in various technical and even managerial roles. Our system will be able to flag risks of gender paygap and related issues in the entire value chain and of course, provide good benchmarks for internal activities.

Why Eviota?

We named Eviota after a small fish that lives symbiotically among the tentacles of the mushroom coral. It not only rhymes with our software, iotables, but it refers to the fragile ecosystem of coral reefs: miraculous and beautiful forms of life under threat from global warming. The first step of saving our planet is to objectively detect where your organization’s stakeholders – suppliers, buyers, workers, technology – leave an impact on the environment.

Our review of the environmental, social and governance risk management (ESG sustainability management) suggests that complying with ESG standards is not only a pre-requisite to get cheaper loans (less important) and cheaper insurance (very important in live music), but also a requirement by corporate sponsors of events, and even a large part of the audience. While some music organizations already provide sustainability reporting, they are not standardized and are less factful as they are not connected to accounting information at every point. Our solution aims to give much credibility to both the sustainability reports and non-financial disclosures of the financial reports (which are not mandatory for MSMEs but increase their trustworthiness on an elective basis if they are included.)

Green Recovery in the Music Sector

The objectives of the MusicAIRE GREEN recovery program is increasing the music sector’s environmental sustainability and ecological awareness with a view to greening the music industry, in particular live acts, festivals and touring, as well as supporting innovative start-ups aiming at decreasing the environmental footprint of online data storing and music distribution.

Surveyharmonies

Thu, 09 Jun 2022 09:40:00 +0100

Surveyharmonies is a joint project of Reprex and Sinus. Reprex integrates open-source surveying tools, open-source software as a service, and open data as a service. SINUS carries out survey design, fieldwork, and analysis on demand to maximize the value of survey recycling and the reuse of open data. Surveyharmonies unleashes the benefits of research automation, open science, and open data: you get more responses, higher quality datasets, and better documentation for less money. Surveyharmonies is an offering of Reprex with Sinus.

Value proposition

Surveyharmonies increase the value of your surveying work, whether self-managed or outsourced.

Unleash the power of open and linked data: more data is collected by adding comparative answers from open datasets. You pay for one fieldwork and receive the output of many fieldworks.
More information, more robust and valuable inference: compare your respondent’s answers with responses collected in other regions or earlier years. You gather data in one timeframe and region but receive data from more timeframes and many regions.
Better connection, more users: We make your work more usable and more citable with standardized metadata. Your data products are easy to find, import, reuse, and cite.
Higher data quality: our data products are easier to review. We offer more tailored and cheaper quality control than proprietary survey providers. We use transparent, open, peer-reviewed algorithms.
Apply the best practices of open science or open government recommendations.

Our services

Outsource services where you have no competitive in-house capacity. We aim to use only open-source components, so you can insource as much of the workflow as you can carry out in high-quality in-house. We integrate high-value survey design, fieldwork, data processing, analysis, and documentation services.

While we thrive on being fully open-source, at this stage, we are still looking for good open-source solutions for carrying out fieldwork tasks. We support some widely used and inexpensive fieldwork applications like LimeSurvey or SurveyMonkey, or the fieldwork software of your choice. We support using Google Forms for simple activities because it is easy to use and free for end-users.

Offering

For commercial partners and NGOs: we offer the best value for money with survey recycling. We unleash the power of open data and add already collected responses from taxpayer-funded research in the European Research Area or the United States. With a more limited offering, we work in other regions.
For research consortia and projects: we help you carry out your social sciences or digital humanities surveying tasks with the best value for money. For larger tasks, as consortium partners, and for smaller tasks (and retrospective data processing from pre-existing sources) as subcontractors.
Data altruism: we offer limited free services when our partners are willing to make the results open.

Pan-European Creator Survey

Sun, 20 Mar 2022 14:00:00 +0100

Engage with us on LinkedIn @CultDataObs or check out our open data and open repositories, code, tutorials

The Cultural Creative Sectors Industries CCSI Data Observatory has grew out of the Digital Music Observatory, which had harmonized and collected surveys in all European countries about how musicians work and earn their living, and how audiences differ in various countries, metropolitan regions in terms of typical age, visiting probability, spending capacity, and other important factors. We helped music organizations to significantly increase the royalty pay-outs of artists in two countries, and we contributed for the advocacy of fairer compensation and fairer taxation in others.

This year we teamed up with the reCreating Europe research consortium of renowned experts in the field of copyright, geography and economics of creativity, sociology of innovation, communication and media studies, cultural policies, Open Knowledge and access to culture, cultural policies, minority rights and disability rights. We are carrying out a truly pan-European survey in all EU languages about important topics for creators and performers, including but not limited to musicians, songwriters, composers, photographers, video artists, designers, actors, illustrators, authors, directors, journalists and screenwriters:

Understanding how much creators are aware of the legal and financial terms that online platforms use that distribute their work
Understanding and interest in how AI algorithms contribute to the successful or not successful dissemination of their work, or can support their creative processes
Understanding and evaluation of fairness in the way internet platforms distribute earnings to creators
Attitudes to plagiarism, piracy, and copyright protection
Access to Covid-19 relief funds
No personal data is involved.

What do we ask from individual creators, bands, collectives?

The researchers of the reCreating Europe Consortium would like to carry out an interview with you in English.
If you cannot or do not want to be interviewed in English, we would be grateful if you would still fill out the survey form (about 10 minutes) in any of the EU official languages,
And invite a colleague/friend confident in English to do the same and volunteer for the interview at the end of the survey.

Creator organizations

We would like to ask you to ask your members to do the same above, in your newsletter, Facebook page, or other means of communication. The research with free datasets, visualizations, infographics will be placed for free in the Digital Music Observatory and the Cultural and Creative Sectors Industries Data Observatory and can help your own HR, advocacy, education work. (We have many other similar data assets that are already there, or we can give to you.) Therefore, it is fully compliant with GDPR—we do not want to know who fills out our survey, the interviews are voluntary, and sharing the survey can contribute to your core business.

Bloggers

We would be very happy if you would write about this survey or invite creators to participate in our research. We can give you infographics, data visualizations prior to academic publication to write about the results.

Our research principles

We believe in transparency, openness, and high-quality work. We carry out an open collaboration with representatives of music professionals, NGOs, and universities. Because in the European music ecosystem, most professional and artists are freelancers or micro-entrepreneurs, we also try to form collaborations with individuals. All our data is open, interoperable, reusable data that comes with the highest quality of documentation and help for reuse.

We would appreciate it immensely if you would support this important research by disseminating the call to participate in this study.

History

Before you start a new questionnaire-based research, get in touch with us! Maybe we have history for your questionnaires. We can make your survey cheaper, better, and more informative.

The Digital Music Observatory and its predecessor, CEEMID, has been working with harmonized surveys for 8 years. We have compiled the biggest database of interview transcripts with concert audiences (more than 70,000 interviews in all European countries, soon to be extended to more than 100k) and the world’s biggest harmonized survey dataset about music creators (4000 responses from 12 European countries.) We use the Open Data Directive, originally for government-funded research data, recently extended to taxpayer funded scientific research, to access datasets that are invisible for the music industry.

What is Survey Harmonization?

Mon, 14 Mar 2022 13:00:00 +0100

Engage with us on LinkedIn @CultDataObs or check out our open data and open repositories, code, tutorials

Survey harmonization is a powerful research tool to increase the usability of questionnaire-based empirical research. When the same questions are asked from similarly selected German and French people, music audiences and musicians, then we can make meaningful comparisons between the public opinions of the two countries, or the different perceptions of fans and makers of music. And if we repeat this procedure time after time, we can see if opinion is shifting more in Germany or France, or the living conditions of musicians are catching up with the rest of the country population.

You can create better surveys with less cost: you only need to ask the information, or change of information, that is not included in our harmonized datasets. Shorter, better questionnaires, smaller samples sizes, huge cost savings.
When you make questionnaire-based research, you immediately get a history (the same question asked years ago) and an international comparison (the same question asked in other countries.)
Often you do not even have to pay for the survey, because somebody else has already made a similar taxpayer funded research and we can just get the data for you. Harmonizing surveys requires advanced data science and statistics knowledge, which is what we provide with the scientific partners of the Cultural Creative Sectors Industries Data Observatory. We have developed an open-source software, free to use, for this purpose. It is not for the faint heart – but users of our observatory can just leave their data for us and let us run the code.

How can you join this year’s survey?

Since 2020, a consortium of nine European universities led by Sant’Anna Pisa is working on the research project reCreating Europe funded by the European Union’s Horizon 2020 research and innovation programme. reCreating Europe aims at bringing a ground-breaking contribution to the understanding and management of copyright in the DSM, and at advancing the discussion on how IPRs can be best regulated to facilitate access to, consumption of and generation of cultural and creative products.

As a part of this project, the University of Amsterdam is conducting a survey which targets a wide range of creators and performers, including but not limited to musicians, songwriters, composers, photographers, video artists, designers, actors, illustrators, authors, directors, journalists and screenwriters.

Tpoics covered in the survey: (1) overall experiences with digitization of cultural content; (2) experiences with publishers, platforms and aggregators: contract and market power issues, accessibility, role of algorithms; (3) experiences with reversion clauses in copyright contracts (4) content removal and website blocking, role of algorithms in this; (5) income development over past years: total income, income from creative profession and distribution of sources within profession; impact of Covid and Covid-related support schemes (6) experiences with piracy and plagiarism; (7) competition from AI producing content.

The survey is available in all official EU languages.

We would appreciate it immensely if you would support this important research by disseminating the call to participate in this study.

History

Before you start a new questionnaire-based research, get in touch with us! Maybe we have history for your questionnaires. We can make your survey cheaper, better, and more informative.

Ensuring the Visibility and Accessibility of European Creative Content on the World Market: The Need for Copyright Data Improvement in the Light of New Technologies

Tue, 01 Mar 2022 00:00:00 +0000

In the European Strategy for Data, the European Commission highlighted the EU’s ambition to acquire a leading role in the data economy. At the same time, the Commission conceded that the EU would have to increase its pools of quality data available for use and re-use. In the creative industries, this need for enhanced data quality and interoperability is particularly strong. Without data improvement, unprecedented opportunities for monetising the wide variety of EU creative and making this content available for new technologies, such as artificial intelligence training systems, will most probably be lost. The problem has a worldwide dimension. While the US have already taken steps to provide an integrated data space for music as of 1 January 2021, the EU is facing major obstacles not only in the field of music but also in other creative industry sectors. Weighing costs and benefits, there can be little doubt that new data improvement initiatives and sufficient investment in a better copyright data infrastructure should play a central role in EU copyright policy. A trade-off between data harmonisation and interoperability on the one hand, and transparency and accountability of content recommender systems on the other, could pave the way for successful new initiatives.

The published article: https://www.jipitec.eu/issues/jipitec-13-1-2022/5515

Preprint version

The earlier preprint version on SSRN our for direct download here on Data & Lyrics. Senftleben, Martin and Margoni, Thomas and Antal, Daniel and Bodó, Balázs and Gompel, Stef van and Handke, Christian and Kretschmer, Martin and Poort, Joost and Quintais, João and Schwemer, Sebastian Felix, Ensuring the Visibility and Accessibility of European Creative Content on the World Market - The Need for Copyright Data Improvement in the Light of New Technologies and the Opportunity Arising from Article 17 of the CDSM Directive (February 12, 2021). Available at SSRN: https://ssrn.com/abstract=3785272 or http://dx.doi.org/10.2139/ssrn.3785272

Open Policy Analysis

Sun, 06 Feb 2022 14:00:00 +0000

Our ambition is to truly maximize transparency, (re)usability, scientific, policy, and business impact while embracing the best practices laid out in the the recommendations of the Reproducibility of scientific results scoping report, and the Progress on Open Science: Towards a Shared Research Knowledge System policy documents of the European Commission’s DG Research & Innovation, as well as the best practices outlined in the evidence-based Knowledge4Policy K4P platform of the European Commission. For the first time in Europe, we will apply and contextualize the Open Policy Analysis Guidelines, which grew out of several initiatives in research transparency with the aim of maximizing benefits in the context of the Foundations for Evidence-based Policy Making Act of 2018 initiative in the United States. We want to ensure that by relying not only on the best European practices, but considering trans-Atlantic experiences, we will make the most out of the opportunities offered by the European Open Data Directive of 2019. This will not only mean rendering a dramatically increased data availability for our partners, as well as increased quality assurance and transparency in our work, but also immediate data access. Following the EU & US best practices we have already placed before sumbitting our proposal our important software tools, methodologies, and sample data in the Zenodo repository.

Our new software will continue to run in the cloud, depositing all of our findings—Findable, Accessible, Interoperable and Reuseable digital assets, including our well-designed and user-tested indicators in 41 data gap fields—into our Digital Music Observatory, which already hosts a modern REST API similar to the Eurostat Rest API. We are still adjusting this service in order to find a way to best implement SDMx and other data standards while maintaining ease of use. We anticipate enhanced usability by April 2022.

Music and Creative Industry Indicators from Needs Assessment to Automated Production

Mon, 03 Oct 2022 12:42:00 +0200

Invitation for an open collaboration

Digital Music Observatory | Listen Local Slovakia

Controls

Next: Right Arrow | Space | Previous: Left Arrow
Start: Home| Finish: End | Overview: Esc
Speaker notes: S | Fullscreen: F | Zoom: Alt + Click

OpenMuse

Presentation outline

Part I: Daniel Antal introduces the OpenMuse project [general aims, methodology, work packages], and our offering to Slovak CCSI stakeholders.
Part II: Mária Kmety Barteková our offering for indicator design and creation, methodology, timeline, quality control.
Part III: Daniel Antal our offering for cultural diversity policies, radio quotas and beyond
Part IV: Quesitons and identification of partners and data sources.

Part I: OpenMuse offering

Offering Tools

Smart policy documents
Dashboards
Open source software

Offering Methodology

Open Policy Analysis, “Live Policy Documents”
Indicator design
Slovak music market share
Harmonized data collection, when needed

Open Polcy Analysis

Live Policy Documents

Trustworthy Cultural AI: make big data work for Lithuania artists and small labels, publishers and CCIs
Synchronize the (meta)data of Lithuania heritage and present cultural production as linked open data, so that autonomous systems of Spotify, YouTube, Wikipedia, Library of Congress, etc, can read it.
Practical, open source solutions that can immediately applied
Access to the OpenMuse know-how

Past excellence

CEEMID
Slovak Music Industry Report

Work Packages

WP1 Economy - CCSI indicators, valuations (IViR, EUBA)
WP2 Diversity - market share of Slovak music in radios, streaming, Listen Local Slovakia (SOZA, Reprex.)
WP3 Music & Society - harmonized data collection where needed
WP4 Innovation - providing policymakers with tools

WP1 Slovak creators, including music creators, should earn as much as the European average. Their income should level up to German and Finnish levels.

WP2 Lithuanian people should find good Lithuanian music in Lithuania on Spotify, YouTube and the radio. Foreign labels must not increase their market share and colonize the local creative ecosystem.

Joining the European Union

Why should creators earn less than other skilled occupations and knowledge workers?
Why should artists get less paid than the average in Slovakia?
Why should Slovaks get paid less than the average European?

Harmonized data collecton

Valuations

Income & empirical comparisons | Setting the Value of Slovak Music

Part II: Indicators

Maria on Eurostat indicator cycle

Maria on Slovak indicator cycle

explain the cylce in Slovakia, with a timline
what will they get by October 2022
waht will they get by October 2023

Maria on Open Policy Analysis

explain what the application of the OPA guidelines will mean to your work
when will you release the information, where, what are the quality/transparency guarrantees

Maria on indicator topics

What kind of indicators are we going to produce
Just brief, like employment, etc.

Maria on stakeholders

To whom we are designing the indicators
When will we ask them for initial needs assessment
For data
For feedback

Maria on example data needs

What sort of raw data we will ask for
Give a few examples

Part III: WP2

Download in English | | Download in Slovak

Is the algorithm cheating? ➡️ trustworthy AI policy, enabling courts, competition and consuerm protection agencies to protect the Lithuanian CCIs
Is the algorithm learning from bad data? ➡️ national heritage policy, consolidate Lithuanian CCI data as linked, open data, and synch it daily with the world.
Can we ask for 20% Lithuanian music in radio, television, YouTube or Spotify?

LaLa Database

Opt-in, opt-out database to make sure that you are in control
Health check of artist data from Wikipedia to YouTube
Make changes that avoid missing payments, recommendations errors.

Listen Local Database

Write-in database that we do not forget about any new or legacy artists, crowdsourcing for fans, journalists, musicians, educators to add artists, bands, ensembles
Minimum requirement one commercially released recording, one published music composition, a Wikipedia page. Other commercially not released artists, or their heirs must explicitly register to LaLa (if they are not public, GDPR applies)

Topics: 
 - European AI regulation and ethics, music AI, 
 - non-discriminiation and fair remuneration for 
 Lithuania's creators.
 - Smart and future proof local content and media 
 regulation, music export, cultural heritage. 

Features: 
 - The policy brief itself expressed as a resource, 
   not a static file that is once finished and downloaded.
 - It is an educational document in itself, it is 
 self-refreshing with data, legal and policy texts
 - We educate policy-makers and music professionals
 to talk, write and think about AI in a smart way.

Invitation for an open collaboration

Click for Contacts: Digital Music Observatory | Listen Local Lithuania | LaLa for Artists | Lala for Fans | Open Policy Analysis

Part IV Questions?

Get in Touch: Daniel - Valuations, Metadata, AI Policy | Ioanna - Fashion & Blockchain | Borbala - Fair competition and fighting dark algorithms | Mark - MXF

Market analysis

Local content regulation & goals

Unpaid revenue rate
Local content regulations for radios
Goal setting on streaming

Avoiding negative outcomes

Trustworthy AI

Data biases

The data about the Lithuanian national cultural output and heritage shoulld be availabe in Lithuania
Should be daily synchronized with the world via the Europena, Dbpedia, MusicBrainz and other open linked data solutions that connect YouTube, Spotify, and other platforms to daily, up-to-date knowledge to train their algorithms.
AI is machine learning

Listen Local Lithuania

open linked data: We aim to sync with AGATA, LATGA, the Martynas Mažvydas National Library of Lithuania, MIC and internationally via the Europeana, Music Brainz and Zenodo with global libraries, music services, and scientic repositories.
open and smart policies: In sync with the the Cultural Ministry of Lithuania, providing a blueprint for film, television, books, newspapers, fashion, architecture, design and other cultural and creative industries.

Surveyharmonization

Sun, 25 Sep 2022 12:00:00 +0000

Survey Harmonization Workflow

Controls

Next: Right Arrow or Space
Previous: Left Arrow
Start: Home | Finish: End
Overview: Esc | Speaker notes: S | Fullscreen: F

Principles

Generic concept of surveying, i.e. examining and record the area and features of (an area of land) to construct a map, plan, or description.
Structured data collection of the missing information, harmonization of knowledge.
Reproducibility and not automation. On a small scale, anything can be done with Ctrl C + Ctrl V. But it should be recorded, documented for future Ctrl C + Ctrl V.

Timeline

Concepts

We harmonize knowledge concepts. Because knowledge concepts are very abstract, the harmonization of concepts requires an iteration of desired output and questionnaire or form items, and it will be carried on throughout the project. The harmonization of concepts will allow us to link our survey data to pre-existing survey data, financial information, or any other source of information.

Data Model

Data modeling enables us to place the information we gain from existing sources, for example, by recycling pre-existing questionnaire items and answers to a knowledge graph together with our data. A knowledge graph is a more flexible, future-proof, generalized database that connects pre-existing information with new information.

Question Bank

The questionnaire harmonization includes the harmonization of the question or entry form label (In the past 12 months, how many times have you been to a concert) and the response scale (1, 2, Do not remember, Decline to say). The harmonization must be made with other knowledge concepts (i.e. the concept of the concert) and survey questionnaires or annual report information fields.

Translations

We must be able to work with translators and standardized translated labels. We must have the question bank ready by the end of October. Ensure that we do not use URIs but IRIs for identifying questionnaire items. Labesl are translated or localized. A generic ‘French’ label is often unsuitable for French speakers in Belgium.

Fieldwork

We must carry out fieldwork, i.e. surveying music-related problems. We will conduct the fieldwork with a cheap online tool (LimeSurvey or SurveyMonkey). The fieldwork will likely remain fully online or may contain a small, hybrid online interview element. The integration of fieldwork implementation is the least important task for us. Use whatever is convenient.

Code and Save

We must record the information into coded datasets that are saved into files. The success of the output harmonization will depend on the use of harmonized coding (we will use, whenever possible, SDMX code definitions, such as ‘F" = ‘female’) and the use of machine-readable, open, portable file formats. Potential users are small entities, and we will avoid the use of databases and favor the use of knowledge graphs instead.

Output harmonization

We will harmonize the data, which means that we will join the coded answers considering the question labels, the value labels, and various forms of missing information across all languages (i.e., English or German versions of a question and answer options.)

Presentation

We will report the harmonized information using graphic visualizations, tables placed into presentation slides, books or web pages. We should have the templates based on test data ready in January.

Questions?

LinkedIn: Daniel Antal - Digital Music Observatory

Dutch AI Coalition Working Group Culture and Media

Thu, 22 Sep 2022 09:30:00 +0000

Reprex is presenting its Digital Music Observatory and Cultural Creative Sectors Industries Data Observatory as platforms for developing and evaluating trustworthy AI in the cultural domains. We hope to find new partners within the NLAIC community to join our open, collaborative projects.

Learn more about the Dutch AI Coalition’s Cultur and Media Working Group (in Dutch:)

Listen Local

Thu, 08 Sep 2022 14:30:00 +0000

Daniel Antal will introduce Listen Local Lithuania, a new offspring of the Digital Music Observatory’s trustworthy AI program, Listen Local, and show what artists, managers, labels, and cultural policymakers can do to ideas about balancing the post-Covid revenues with increased sales on global platforms.

Based on our experience in Slovakia (see: Feasibility Study On Promoting Slovak Music In Slovakia & Abroad), and new data from Lithuania and Ukraine we show what data management problems make the Lithuanian music invisible for the AI algorithms of YouTube, Spotify and other platforms; how artists from Vilnius or Kaunas can find their new release on Forgetify, and app that plays songs that were never played. We show why music revenues decreased in the past years for independent and small country catalogues, and we give practical advice on increasing the value and visibility of the Lithuanian music repertoire. We show an open, data federation model to connect the Lithuanian national library the system of collective management, distributors, and small data of artists and labels.

Prepare your questions

Do you want to know why your music is not recommended by streaming platforms? Are you interested in finding new Lithuanian music that fits into your radio or festival programming but your are stuck? You can send questions prior to the conference to the speaker, Daniel Antal, and we’ll address your problem with examples in the talk.

Opt-in

You can opt-in into our database before or during the conference, and we will give you tailored analysis about your data quality, and your opportunities to place your music to new audiences in Lithuania and abroad.

Venue

The talk will be in the what’s next in music conference or follow them on Facebook. Participation is free but requires registration. (We will post shortly more details on this.)

The time set for the talk is EEST Vilnius time (GMT+02.00).

Not available?

You can meet the project’s manager and the speaker on 10 September in Kaunas, at the closing event of what’s next in music.

Background

Listen Local Lithuania is supported by the European Union’s MusicAIRE, an EU-funded project that aims to provide tailored-made instruments for the sector and contribute to a green, digital, just, and resilient recovery of the music ecosystem. It is managed by MXF in collaboration with Music Export Ukraine and Reprex.

MXF and the outcomes of the Listen Local Lithuania will participate in the OpenMusE project, an international research and innovation project with prestigious universities, music industry stakeholders, collective management organizations, and innovative SMEs to design tools that promote the diversity of music on global platforms, increase the revenues and decrease the data costs of music organizations, provide cultural policymakers with the Open Policy Analysis framework the first time in Europe. This open collaboration project funded for 2023-2025 is open for small and large music organizations, collective management organizations, libraries, and cultural policymakers.

Smart Policy Documents

Sun, 04 Sep 2022 00:00:00 +0000

Stop uploading your work onto the web 2.0 in a pdf, epub, or word file. Build a self-refreshing resource that re-refreshes the statistics, legal texts, tables, visualizations, footnotes and bibliography instead. This way you’ll have a greater impact: you’ll connect to global knowledge graphs, and your work can be reused much better.

Our “smart document” is always live: it contains code that searches for data updates or changes in the law. It is a document that includes reproducible research code and makes sure that your document contains the latest information. It is a resource that, when uploaded to the web 3.0, cannot only be downloaded, but finds itself its audience, places itself into global libraries, data exchanges, automated websites.

📊 Downloads or updates reliable statistical data, utilizing the SDMX standard, for example, Eurostat datasets, which is placed into your text as a table, a data visualization, and its source as a footnote and bibliographic citation.

⚖️ Updates legal citations from Eur-LEX, i.e. contains the latest form of policy or legal documents, and flags for the researchers any important changes in the lifecycle of the citation (legal text goes out of force, amended, important court decisions get connected).

🎨 Exchange cultural Digital Objects with Europeana or other cultural heritage or knowledge organizations, such as out-of-print books, music works, or 3D design objects.

We admit that at this point we mainly serve policy wonks or scientists who are very tech savvy. But we are working hard to package it into a highly usable end-user product.

What is reproducible research?

Reproducible research is what it is: it can be reproduced. Easier said than done. We create open source software that accompanies the entire research workflow, from the fieldwork or big data collector, downloading documents via the analysis, visualizations, citations, web dissemination and publication. We do all those little steps that computers do better than humans: logging, documenting, testing, validating, archiving. This allows our users to do what humans do best: think.

Our data observatories are open scholarly platforms that support reproducible research. Our policy documents bring this functionality to your personal computer, and make it available for an NGO, a lawyer, a consultant, or an individual researcher.

Feature list

Each external resource, i.e. a policy document, a legal text, a cultural heritage object, a catalog entry, a dataset is clearly identified and downloaded, processed for the document into a footnote, citation, table or standard visualization.
New artifacts, such as visualizations, tables, receive a unique document object identifier (DOI) that clearly states their source, the person who oversaw the creation, the date, and the version.
These artifacts are added into the text in pre-defined places, such as the “Chart on GDP growth” placeholder containing the latest chart on GDP growth, while the citation in the bibliography contains the new version of the artifact (i.e. the chart with a DOI.)
The artifacts, such as datasets, tables, visualizations, codebooks, reference lists, are uploaded with a new version to an open science repository such as FigShare or Zenodo. This ensures that the creator’s intellectual rights are respected, and different, unauthorized versions of the table, chart, or other artifact in unknown news outlets, social media, are not connected to the creator or publisher.
Zenodo, via OpenAIRE, connects your work with global libraries, and if they meet the quality criteria, they are often immediately placed into the catalogues of global libraries. Our smart policy documents are not only uploaded onto the web, but connect to global knowledge graphs, or the web 3.0.
The author and publisher of the ‘smart policy document’ receives notifications of significant changes, i.e. non-trivial new data at Eurostat, or non-trivial connecting policy documents, court decisions, which should trigger a revision of the smart policy document’s textual contents. Needless to say, behind the scenes, we handle those trivial changes, too.
After human review of the new version, it is created as Word docx, EPUB, and PDF file, and with a new version and DOI, it is uploaded to open science repositories that synchronize this publication with global library systems.

Trustworthy AI

Sat, 03 Sep 2022 13:37:00 +0200

Visit the documentation website of statcodelists on statcodelists.dataobservatory.eu/.

Reporting the impacts of the entire value chain.

Requirements for connecting financial and sustainability reporting.

Growing interest for ESG in select countries.

Co-funded by the European Union

More data is collected by adding comparative answers from open datasets. You pay for one fieldwork and receive the output of many fieldworks.

In our Central & Eastern European Music Industry Report 2020 as a case-study on evidence-based policymaking we compared how musicians and their audiences live in several countries.

We using survey harmonization and data integration techniques to join hundreds of questionnaire-based research in Europe on music audiences. We are now improving our capacities to bring analysis to sub-national level, like in the example of Wales.

In our Central & Eastern European Music Industry Report 2020 as a case-study on evidence-based policymaking we compared how musicians and their audiences live in several countries.

Layer	Goal	Target	Example
Open Output	Ensure unified output	We comply with the level 3 requirements and we will create a showcase how to do this best following EU open science recommendations.	See our example.
Open Output	Establish a clear link between input and output	We will produce more than 100 outputs, some only as indicators, and others in form of policy analysis, we will comply with level 1,2,3 as necessary.	Our affiliated music industry partners will create cases studies with interactive tools (level 3). See our Slovak case study which came with a Shiny App that analyzed music recommendations.
Open Analysis	Provide clear accounts of all methodological procedures in a way that is easily interpreted by an informed reader.	We accomplish level 3 with placing the code in clearly documented. into a dynamic document, or open notebook	See for example our blogpost on automatic forecasting for the music industry.
Open Analysis	Share raw (or analytic) data and materials in a way that the analysis is reproducible with minimal effort.	We will accomplish level 3 through trusted repositories following EU recommendations. We will use the Zenodo repository developed by CERN and the EU’s OpenAIRE project.	See our solution on Zenodo.
Open Analysis	Share an open report that includes clear accounts of all methodological procedures, data, and assumptions.	We would like to go beyond the level 3 requirements of the OPA with using standardized documentation languages, such as SDMX statistical metadata and its standardized codebooks, and comply with both Dublin Core and DataCite extended, recommended standarized reporing.	See our example An Empirical Analysis of Music Streaming Revenues and Their Distribution created for the UK Intellectual Property Office’s evidence-based policy effort in music streaming.
Open Materials	Standardize the file structure so that materials are organized in a way that is accessible to an informed reader.	We comply with the level 3 requirements. Our versioned controled output is on Github.	See an example on Github.
Open Materials	Label and document each input, including data, research, and guesswork.	We will go beyond level 3 requirements, because we want to make sure that our labelling and documentation is interopreable, and we apply various metadata standards for this purpose.	See our example explaining how we document our datasets in our API.
Open Materials	Ensure that code/spreadsheets are reproducible.	All our spreadsheets are machine generated for the convenience of the user who uses spreadsheet applications, but everything can be run with a click, which accomplishes level 3, and maintains the convenience of level 1-2 for the user. We go further with creating authoritative copies of each dataset and visualization with DOIs. We also produce an API which gives programatic or single table access to both the data and standardized codebooks.	See our API. All our datasets are described in detail on Zenodo and Figshare, too.
Open Materials	Use a version control strategy.	We use Git version control, and we employ various repositories and project documentation tools on Github. These are linked with the Zenodo EU open repository and our data API.	See our example intergration.

Percentage of Regional Population Who Reads Books

Mon, 15 Nov 2021 19:00:00 +0100

The indicator is created from the Eurobarometer 79.2 survey’s GESIS datafile using regional subsamples. The regional subsamples were recoded to the NUTS 2016 regional boundary definitions with the regions R package. In the larger countries, where only NUTS1 level information was present (for example, in Germany and the United Kingdom), we imputed the NUTS1 territorial average values to the constituent NUTS2 regions.

We placed the authoritative copy with metadata on the Zenodo open repository.

A ‘dirty averaging’ was used to create regional averages, with scale national post-stratification weights to an expected value of 1. Al respondents who read at least one book in the previous 12 months were coded to have read a book.

This indicator was used in the Balázs Bodó, Dániel Antal, Zoltán Puha: Can scholarly pirate libraries bridge the knowledge access gap? An empirical study on the structural conditions of book piracy in global and European academia, in Plos ONE (Published: December 3, 2020.)

Engage with us on LinkedIn @CultDataObs or check out our open data and open repositories, code, tutorials

How We Add Value to Public Data With Better Curation And Documentation?

Mon, 08 Nov 2021 09:00:00 +0000

In this example, we show a simple indicator: the Turnover in Radio Broadcasting Enterprises in many European countries. This is an important demand driver in the Music economy pillar of our Digital Music Observatory, and important indicator in our more general Cultural & Creative Sectors and Industries Observatory. We show a very similar example in our Green Deal Data Observatory with environmental R&D public spending in Europe.

Engage with us on LinkedIn @CultDataObs or check out our open data and open repositories, code, tutorials

This dataset comes from a public datasource, the data warehouse of the European statistical agency, Eurostat. Yet it is not trivial to use: unless you are familiar with national accounts, you will not find this dataset on the Eurostat website.

The data can be retrieved from the Annual detailed enterprise statistics for services NACE Rev.2 H-N and S95 Eurostat folder.

Our version of this statistical indicator is documented following the FAIR principles: our data assets are findable, accessible, interoperable, and reusable. While the Eurostat data warehouse partly fulfills these important data quality expectations, we can improve them significantly. And we can also improve the dataset, too, as we will show in the next blogpost.

Findable Data

Our data observatories add value by curating the data–we bring this indicator to light with a more descriptive name, and we place it in context with our Digital Music Observatory and Cultural & Creative Sectors and Industries Observatory. While many people may need this dataset in the creative sectors, or among cultural policy designers, most of them have no training in working with national accounts, which imply decyphering national account data codes in records that measure economic activity at a national level. Our curated data observatories bring together many available data around important domains. Our Digital Music Observatory, for example, aims to form an ecosystem of music data users and producers.

We added descriptive metadata that help you find our data and match it with other relevant data sources.

We added descriptive metadata that help you find our data and match it with other relevant data sources. For example, we add keywords and standardized metadata identifiers from the Library of Congress Linked Data Services, probably the world’s largest standardized knowledge library description. This ensures that you can find relevant data around the same key term (radio broadcasting) in addition to our turnover data. This allows connecting our dataset unambiguosly with other information sources that use the same concept, but may be listed under different keywords, such as Radio–Broadcasting, or Radio industry and trade, or maybe Hörfunkveranstalter in German, or Emitiranje radijskog programa in Croatian or Actividades de radiodifusão in Portugese.

Accessible Data

Our data is accessible in two forms: in csv tabular format (which can be read with Excel, OpenOffice, Numbers, SPSS and many similar spreadsheet or statistical applications) and in JSON for automated importing into your databases. We can also provide our users with SQLite databases, which are fully functional, single user relational databases.

Tidy datasets are easy to manipulate, model and visualize, and have a specific structure: each variable is a column, each observation is a row, and each type of observational unit is a table. This makes the data easier to clean, and far more easier to use in a much wider range of applications than the original data we used. In theory, this is a simple objective, yet we find that even governmental statistical agencies–and even scientific publications–often publish untidy data. This poses a significant problem that implies productivity loses: tidying data will require long hours of investment, and if a reproducible workflow is not used, data integrity can also be compromised: chances are that the process of tidying will overwrite, delete, or omit a data or a label.

Tidy datasets are easy to manipulate, model and visualize, and have a specific structure: each variable is a column, each observation is a row, and each type of observational unit is a table.

While the original data source, the Eurostat data warehouse is accessible, too, we added value with bringing the data into a tidy format. Tidy data can immediately be imported into a statistical application like SPSS or STATA, or into your own database. It is immediately available for plotting in Excel, OpenOffice or Numbers.

Interoperability

Our data can be easily imported with, or joined with data from other internal or external sources.

All our indicators come with standardized descriptive metadata, and statistical (processing) metadata. See our API

All our indicators come with standardized descriptive metadata, following two important standards, the Dublin Core and DataCite–implementing not only the mandatory, but the recommended descriptions, too. This will make it far easier to connect the data with other data sources, e.g. turnover with the number of radio broadcasting enterprises or radio stations within specific territories.

Our passion for documentation standards and best practices goes much further: our data uses Statistical Data and Metadata eXchange standardized codebooks, unit descriptions and other statistical and administrative metadata.

Reuse

All our datasets come with standardized information about reusabililty. We add citation, attribution data, and licensing terms. Most of our datasets can be used without commercial restriction after acknowledging the source, but we sometimes work with less permissible data licenses.

In the case presented here, we added further value to encourage re-use. In addition to tidying, we significantly increased the usability of public data by handling missing cases. This is the subject of our next blogpost.

Are you a data user? Give us some feedback! Shall we do some further automatic data enhancements with our datasets? Document with different metadata? Link more information for business, policy, or academic use? Please give us any feedback!

How We Add Value to Public Data With Imputation and Forecasting

Mon, 08 Nov 2021 10:00:00 +0100

Engage with us on LinkedIn @CultDataObs or check out our open data and open repositories, code, tutorials

Public data sources are often plagued by missng values. Naively you may think that you can ignore them, but think twice: in most cases, missing data in a table is not missing information, but rather malformatted information. This approach of ignoring or dropping missing values will not be feasible or robust when you want to make a beautiful visualization, or use data in a business forecasting model, a machine learning (AI) applicaton, or a more complex scientific model. All of the above require complete datasets, and naively discarding missing data points amounts to an excessive waste of information. In this example we are continuing the example a not-so-easy to find public dataset.

In the previous blogpost we explained how we added value by documenting data following the FAIR principle and with the professional curatorial work of placing the data in context, and linking it to other information sources, such as other datasets, books, and publications, regardless of their natural language (i.e., whether these sources are described in English, German, Portugese or Croatian). Photo: Jack Sloop.

Completing missing datapoints requires statistical production information (why might the data be missing?) and data science knowhow (how to impute the missing value.) If you do not have a good statistician or data scientist in your team, you will need high-quality, complete datasets. This is what our automated data observatories provide.

Why is data missing?

International organizations offer many statistical products, but usually they are on an ‘as-is’ basis. For example, Eurostat is the world’s premiere statistical agency, but it has no right to overrule whatever data the member states of the European Union, and some other cooperating European countries give to them. And they cannot force these countries to hand over data if they fail to do so. As a result, there will be many data points that are missing, and often data points that have wrong (obsolete) descriptions or geographical dimensions. We will show the geographical aspect of the problem in a separate blogpost; for now, we only focus on missing data.

Some countries have only recently started providing data to the Eurostat umbrella organization, and it is likely that you will find few datapoints for North Macedonia or Bosnia-Herzegovina. Other countries provide data with some delay, and the last one or two years are missing. And there are gaps in some countries’ data, too.

See the authoritative copy of the dataset.

This is a headache if you want to use the data in some machine learning application or in a multiple or panel regression model. You can, of course, discard countries or years where you do not have full data coverage, but this approach usually wastes too much information–if you work with 12 years, and only one data point is available, you would be discarding an entire country’s 11-years’ worth of data. Another option is to estimate the values, or otherwise impute the missing data, when this is possible with reasonable precision. This is where things get tricky, and you will likely need a statistician or a data scientist onboard.

What can we improve?

Consider that the data is only missing from one year for a particular country, 2015. The naive solution would be to omit 2015 or the country at hand from the dataset. This is pretty destructive, because we know a lot about the radio market turnover in this country and in this year! But leaving 2015 blank will not look good on a chart, and will make your machine learning application or your regression model stop.

A statistician or a radio market expert will tell you that you know more-or-less the missing information: the total turnover was certainly not zero in that year. With some statistical or radio domain-specific knowledge you will use the 2014, or 2016 value, or a combination of the two and keep the country and year in the dataset.

Our improved dataset added backcasted (using the best time series model fitting the country’s actually present data), forecasted (again, using the best time series model), and approximated data (using linear approximation.) In a few cases, we add the last or next known value. To give a few quantiative indicators about our work:

Increased number of observations: 65%
Reduced missing values: -48.1%
Increased non-missing subset for regression or AI: +66.67%

If your organization is working with panel (longitudional multiple) regressions or various machine learning applications, then your team knows that not havint the +66.67% gain would be a deal-breaker in the choice of models and punctuality of estimates or KPIs or other quantiative products. And that they would spent about 90% of their data resources on achieving this +66.67% gain in usability.

If you happen to work in an NGO, a business unit or a research institute that does not employ data scientists, then it is likely that you can never achieve this improvement, and you have to give up on a number of quantitative tools or visualizations. If you have a data scientist onboard, that professional can use our work as a starting point.

Can you trust our data?

We believe that you can trust our data better than the original public source. We use statistical expertise to find out why data may be missing. Often, it is present in a wrong location (for example, the name of a region changed.)

If you are reluctant to use estimates, think about discarding known actual data from your forecast or visualization, because one data point is missing. How do you provide more accurate information? By hiding known actual data, because one point is missing, or by using all known data and an estimate?

Our codebooks and our API uses the Statistical Data and Metadata eXchange documentation standards to clearly indicate which data is observed, which is missing, which is estimated, and of course, also how it is estimated. This example highlights another important aspect of data trustworthiness. If you have a better idea, you can replace them with a better estimate.

Our indicators come with standardized codebooks that do not only contain the descriptive metadata, but administrative metadata about the history of the indicator values. You will find very important information about the statistical method we used the fill in the data gaps, and even link the reliable, the peer-reviewed scientific, statistical software that made the calculations. For data scientists, we record the plenty of information about the computing environment, too-–this can come handy if your estimates need external authentication, or you suspect a bug.

Avoid the data Sisyphus

If you work in an academic institution, in an NGO or a consultancy, you can never be sure who downloaded the Annual detailed enterprise statistics for services (NACE Rev. 2 H-N and S95) Eurostat folder from Eurostat. Did they modify the dataset? Did they already make corrections with the missing data? What method did they use? To prevent many potential problems, you will likely download it again, and again, and again…

See our The Data Sisyphus blogpost.

We have a better solution. You can always rely on our API to import directly the latest, best data, but if you want to be sure, you can use our regular backups on Zenodo. Zenodo is an open science repository managed by CERN and supported by the European Union. On Zenodo, you can find an authoritative copy of our indicator (and its previous versions) with a digital object identifier, in this case, 10.5281/zenodo.5652118. These datasets will be preserved for decades, and nobody can manipulate them. You cannot accidentally overwrite them, and we have no backdoor access to modify them.

Recreo Consortium

Sat, 06 Nov 2021 16:00:00 +0000

The Scuola Superiore di Studi Universitari e di Perfezionamento Sant’Anna and Università degli Studi di Trento (Italy); University of Glasgow (United Kingdom); Universiteit van Amsterdam and Stichting Europeana from the Netherlands; the National University of Ireland Maynooth (Ireland); Tartu Ulikool (Estonia); Szegedi Tudományegyetem (Hungary); Fundacion Santa Maria La Real del Patrimonio Historico from Spain; the Katholieke Universiteit Leuven, (Belgium); Culture Action Europe AISBL and IDEA Strategische Economische Consulting (Belgium) and Reprex created the REshaping CCSI REsearch: Open data, policy analysis and methods for evidence-based decision-making consortium consortium, which will mainly develop new policy evidence in the field of innovation and inclusiveness for the creative and cultural sectors, industries. The Consortium applies for a Horizon Europe grant with the HORIZON-CL2-2021-HERITAGE-01-03 Cultural and creative industries as a driver of innovation and competitiveness call of the European Commission.

Policymakers face challenges when trying to implement a strict evidence-based approach to decision-making in the field of cultural and creative sectors and industries (CCSI). This is mostly due to four phenomena:

Evidence dissonances in mapping, measuring and analysis of key indicators, which lead to improper generalizations and gaps in decisionmakers’ knowledge and stakeholders’ awareness
Fragmentation of hubs of production and concentration of platforms, which create statistical biases and have features that hardly fit with traditional impact assessment methods;
Datafication, which is revolutionizing CCSI but remains difficult to investigate, thus broadening knowledge gaps; and
Stakeholders’ fragmentation and conflicting interests, which hinders their engagement, awareness-raising and uptake of policy inputs.With its cross-disciplinary consortium of academics, practitioners and a strong network of stakeholders, engaged via participatory research strategies, RECREO will help policymakers and stakeholders tackling such challenges, by generating new knowledge and methods to fill in knowledge and awareness gaps. RECREO will achieve this goal through four actions.

First, it will generate a wide array of horizontal and sector-specific datasets, made openly accessible via the CCSI Data Observatory and the Evidence Synthesis Platform. Second, it will offer an unprecedented EU and comparative mapping and impact assessment of key regulatory and policy measures relevant for CCSI, made available on the Law and Policy Observatory. Third, it will develop innovative methods to measure and assess CCSI innovation, competitiveness and spill-over effects, emphasizing inclusiveness, diversity and sustainability. Last, it will offer policy recommendations and best practices aimed at supporting the sustainable growth and competitiveness of culturally diverse CCSI, and their cross-fertilization with cultural heritage promotion and preservation.

Reprex Joins RECREO Research Consortium To Develop Innovation Indicators

Sat, 06 Nov 2021 16:00:00 +0000

Engage with us on LinkedIn @CultDataObs or check out our open data and open repositories, code, tutorials

Evidence dissonances in mapping, measuring and analysis of key indicators, which lead to improper generalizations and gaps in decisionmakers’ knowledge and stakeholders’ awareness
Fragmentation of hubs of production and concentration of platforms, which create statistical biases and have features that hardly fit with traditional impact assessment methods;
Datafication, which is revolutionizing CCSI but remains difficult to investigate, thus broadening knowledge gaps; and
Stakeholders’ fragmentation and conflicting interests, which hinders their engagement, awareness-raising and uptake of policy inputs.With its cross-disciplinary consortium of academics, practitioners and a strong network of stakeholders, engaged via participatory research strategies, RECREO will help policymakers and stakeholders tackling such challenges, by generating new knowledge and methods to fill in knowledge and awareness gaps. RECREO will achieve this goal through four actions.

Streaming Economics: Where Are We Really Going?

Wed, 03 Nov 2021 15:00:00 +0200

Artists in the UK had a difficulty explaining in Westminster how they are losing out in streaming, so we have created a streaming price index, like the Dow Jones, if you like, that explains the economic factors of the devaluation of music in the last 5 years in 20 countries. (See our report.)
Music organizations in Slovakia and Hungary were frustrated that their politicians and journalists believed music to be taxpayer funded, so we showed with data that they contribute more proportionally to the national budget than car manufacturers, the darling of local politicians (See our reports in Hungary (recast several times) and in Slovakia.)
We successfully challenged with data restaurant associations, hotel chains, telecom corporations and broadcasters who wanted to bring music prices down in court and via lobbying.

Presentation Slides

You can see some of the topics we will bring into the conversation here.

Digital Music Observatory on MaMA 2021

Thu, 14 Oct 2021 12:15:00 +0000

Use Cases

Public advocacy reports, scientific uses

Use Cases 2

Business Confidential Reports with Digital Music Observatory

Damage claims in private copying
Royalty setting for restaurants, hotels, broadcasting
Music streaming market indicators
Evidence for competition law / regulatory affairs

Questions?

LinkedIn: Daniel Antal - Digital Music Observatory

Open Data - The New Gold Without the Rush

Thu, 14 Oct 2021 12:15:00 +0000

Use Cases

Public advocacy reports, scientific uses

Use Cases 2

Business Confidential Reports with Digital Music Observatory

Damage claims in private copying
Royalty setting for restaurants, hotels, broadcasting
Music streaming market indicators
Evidence for competition law / regulatory affairs

Questions?

LinkedIn: Daniel Antal - Digital Music Observatory

Horizon Europe Consortium

Thu, 14 Oct 2021 10:00:00 +0000

The OpenMusE Consortium brings together music industry stakeholders and researchers from 12 European countries. Our partners represent the diversity of the industry, as well as the shared need to find financially, socially, and environmentally sustainable policy and business models in multiple, sometimes-fragmented streams (e.g., live music, composers/publishers, and recordings with producers and performers).

Good policies require supporting evidence, and good business planning and copyright exploitation require high-quality data. OpenMusE will provide such data through novel, open-source tools that leverage automation to enable transfer to nearly any European market. We will disseminate our data via “live policy documents” and visualisations hosted on our Open Music Observatory, which automatically update when data sources are refreshed. This will set a concrete precedent and a clear path towards a more competitive, fair, and sustainable European music ecosystem.

We hope to get funding from the Horizon Europe HORIZON-RIA HORIZON Research and Innovation Action Research and innovation on cultural heritage and CCIs - 2022 (HORIZON-CL2-2022-HERITAGE-01) Towards a competitive, fair and sustainable European music ecosystem with our competitive tender bid.

Our aim is to partially fill 41/45 identified European data gaps within the Music Moves Europe framework. (See our data coverage below.)

We are looking for end-users and dissemination partners to help us define the European music industry’s data needs, help us collect the data, and review our work in the context of the Horizon Europe call Towards a competitive, fair and sustainable European music ecosystem.

Data that we can offer

By providing continually updated data on five industry pillars – innovation, music and society, music economy, music diversity and circulation, and music policy – we will accurately capture the economic and social value created by the sector, as well as providing stakeholders with data-driven means of maximizing this value.

For copyright and neighbouring rights-holders and their organizations: Value of music, euro value of the value gap, private copying, privacy—we have data and make estimates with all known music value estimation techniques.
For live music organizations: Pan-European audience surveys; demographic comparisons of all European audiences; attendance predictions, attendance profiling.
For granting authorities: we have the world’s largest survey-based dataset on how musicians work and live; we create ex ante needs assessment and grants evaluation indicators and ex-post grant evaluation indicators.
For labels and music publishers: we make big data understandable; we create price and volume indexes of streaming markets; we tell you about price developments and selection probabilities in different cities and countries; we understand the biases of the Spotify and YouTube algorithms; and we measure (meta)data problems and offer scalable solutions.
For music export offices: we measure domestic and foreign market shares in streaming; we create actionable key performance indicators; and we do geographical targeting.

Open collaboration

Presenting our Central & Eastern European Music Industry Report 2020 as a case-study on evidence-based policymaking in the two-day high-level stakeholder event jointly organized by Geothe-Institute and the DG Education and Culture.

The Data Sisyphus

Thu, 08 Jul 2021 09:00:00 +0000

Sisyphus was punished by being forced to roll an immense boulder up a hill only for it to roll down every time it neared the top, repeating this action for eternity. This is the price that project managers and analysts pay for the inadequate documentation of their data assets.

When was a file downloaded from the internet? What happened with it sense? Are their updates? Did the bibliographical reference was made for quotations? Missing values imputed? Currency translated? Who knows about it – who created a dataset, who contributed to it? Which is an intermediate format of a spreadsheet file, and which is the final, checked, approved by a senior manager?

Big data creates inequality and injustice. On aspect of this inequality is the cost of data processing and documentation – a greatly underestimated, and usually not reported cost item. In small organizations, where there are no separate data science and data engineering roles, data is usually supposed to be processed and documented by (junior) analysts or researchers. This a very important source of the gap between Big Tech and them: the data usually ends up very expensive, ill-formatted, not readable by computers that use machine learning and AI. Usually the documentation steps are completely omitted.

“Data is potential information, analogous to potential energy: work is required to release it.” – Jeffrey Pomerantz

Metadata, which is information about the history of the data, and information how it can be technically and legally reused, has a hidden cost. Cheap or low-quality external data comes with poor or no metadata, and small organizations lack the resources to add high-quality metadata to their datasets. However, this only perpetuates the problem.

The hidden cost item behind the unbillable hours

As we have shown with our research partners, such metadata problems are not unique to data analysis. Independent artists and small labels are suffering on music or book sales platforms, because their copyrighted content is not well documented. If you automatically document tens of thousands of songs or datasets, the documentation cost is very small per item. If you, do it manually, the cost may be higher than the expected revenue from the song, or the total cost of the dataset itself. (See our research consortiums’ preprint paper: Ensuring the Visibility and Accessibility of European Creative Content on the World Market: The Need for Copyright Data Improvement in the Light of New Technologies)

In the short run, small consultancies, NGOs, or as a matter of fact, musicians, seem to logically give up on high-quality documentation and logging. In the long run, this has two devastating consequences: computers, such as machine learning algorithms cannot read their documents, data, songs. And as memory fades, the ill-documented resources need to be re-created, re-checked, reformatted. Often, they are even hard to find on your internal server or laptop archive.

Engage with us on LinkedIn @CultDataObs or check out our open data and open repositories, code, tutorials

Metadata is a hidden destroyer of the competitiveness of corporate or academic research, or independent content management. It never quoted on external data vendor invoices, it is not planned as a cost item, because metadata, the description of a dataset, a document, a presentation, or song, is meaningless without the resource that it describes. You never buy metadata. But if your dataset comes without proper metadata documentation, you are bound, like Sisyphus, to search for it, to re-arrange it, to check its currency units, its digits, its formatting. Data analysts are reported to spend about 80% of their working hours on data processing and not data analysis – partly, because data processing is a very laborious task that can be done by computers at a scale far cheaper, and partly because they do not know if the person who sat before them at the same desk has already performed these tasks, or if the person responsible for quality control checked for errors.

Uncut diamonds need to be cut, polished, and you have to make sure that they come from a legal source. Data is similar: it needs to be tidied up, checked and documented before use. Photo: Dave Fischer.

Undocumented data is hardly informative – it may be a page in a book, a file in an obsolete file format on a governmental server, an Excel sheet that you do not remember to have checked for updates. Most data are useless, because we do not know how it can inform us, or we do not know if we can trust it. The processing can be a daunting task, not to mention the most boring and often neglected documentation duties after the dataset is final and pronounced error-free by the person in charge of quality control.

Our observatory automatically processes and documents the data

The good news about documentation and data validation costs is that they can be shared. If many users need GDP/capita data from all over the world in euros, then it is enough if only one entity, a data observatory, collects all GDP and population data expresed in dollars, korunas, and euros, and makes sure that the latest data is correctly translated to euros, and then correctly divided by the latest population figures. These task are error-prone,and should not be repeaeted by every data journalist, NGO employee, PhD student or junior analyst. This is one of the services of our data observatory.

The tidy data format means that the data has a uniform and clear data structure and semantics, therefore it can be automatically validated for many common errors and can be automatically documented by either our software or any other professional data science application. It is not as strict as the schema for a relational database, but it is strict enough to make, among other things, importing into a database easy.
The descriptive metadata contains information on how to find the data, access the data, join it with other data (interoperability) and use it, and reuse it, even years from now. Among others, it contains file format information and intellectual property rights information.
The processing metadata makes the data usable in strictly regulated professional environments, such as in public administration, law firms, investment consultancies, or in scientific research. We give you the entire processing history of the data, which makes peer-review or external audit much easier and cheaper.
The authoritative copy is held at an independent repository, it has a globally unique identifier that protects you from accidental data loss, mixing up with unfinished an untested version.

Cutting the dataset to a format with clear semantics and documenting it with the FAIR metadata concep exponentially increases the value of data. It can be publisehd or sold at a premium. Photo: Andere Andre.

While humans are much better at analysing the information and human agency is required for trustworthy AI, computers are much better at processing and documenting data. We apply to important concepts to our data service: we always process the data to the tidy format, we create an authoritative copy, and we always automatically add descriptive and processing metadata.

The value of metadata

Metadata is often more valuable and more costly to make than the data itself, yet it remains an elusive concept for senior or financial management. Metadata is information about how to correctly use the data and has no value without the data itself. Data acquisition, such as buying from a data vendor, or paying an opinion polling company, or external data consultants appears among the material costs, but metadata is never sold alone, and you do not see its cost.

In most cases, the reason why there is no gold rush for open data is that fact that while the EU member states release billions of euros’ worth data for free, or at very low cost, annually, it comes without proper metadata.

Data-as-Service

Reusable, legal, easy-to-import, interoperable, always fresh data in tidy formats with a modern API. Photo: Edgar Soto.

If the data source is cheap or has a low quality, you do not even get it. If you do not have it, it will show up as a human resource cost in research (when your analysist or junior researcher are spending countless hours to find out the missing metadata information on the correct use of the data) or in sales costs (when you try to reuse a research, consulting or legal product and you have comb through your archive and retest elements again and again.)

The data, together with the descriptive and administrative metadata, and links to the use license and the authoritative copy can be found in our API. Try it out!

Metadata

Wed, 07 Jul 2021 00:00:00 +0000

Adding metadata exponentially increases the value of data. Did your region add a new town to its boundaries? How do you adjust old data to conform to constantly changing geographic boundaries? What are some practical ways of combining satellite sensory data with my organization’s records? And do I have the right to do so? Metadata logs the history of data, providing instructions on how to reuse it, also setting the terms of use. We automate this labor-intensive process applying the FAIR data concept.

In our observatory we apply the concept of FAIR (findable, accessibe, interoperable, and reusable digital assets) in our APIs and in our open-source statistical software packages.

The hidden cost item

Metadata gets less attention than data, because it is never acquired separately, it is not on the invoice, and therefore it remains an a hidden cost, and it is more important from a budgeting and a usability point of view than the data itself. Metadata is responsible for industry non-billable hours or uncredited working hours in academia. Poor data documentation, lack of reproducible processing and testing logs, inconsistent use of currencies, keywords, and storing messy data make reusability and interoperability, integration with other information impossible.

FAIR Data and the Added Value of Rich Metadata we introduce how we apply the concept of FAIR (findable, accessibe, interoperable, and reusable digital assets) in our APIs.

Organizations pay many times for the same, repeated work, because these boring tasks, which often comprise of tens of thousands of microtasks, are neglected. Our solution creates automatic documentation and metadata for your own historical internal data or for acquisitions from data vendors. We apply the more general Dublin Core and the more specific, mandatory and recommended values of DataCite for datasets – these are new requirements in EU-funded research from 2021. But they are just the minimal steps, and there is a lot more to do to create a diamond ring from an uncut gem.

Map your data: bibliographis, catalogues, codebooks, versioning

Updating descriptive metadata, such as bibliographic citation files, descriptions and sources to data files downloaded from the internet, versioning spreadsheet documents and presentations is usually a hated and often neglected task withing organization, and rightly so: these boring and error-prone tasks are best left to computers.

Already adjusted spreadsheets are re-adjusted and re-checked. Hours are spent on looking for the right document with the rigth version. Duplicates multiply. Already downloaded data is downloaded again, and miscategorized, again. Finding the data without map is a treasure hunt. Photo: © N.

The lack of time and resources spend on documentation over time reduces reusability and significantly increases data processing and supervision or auditing costs.

Our observatory metadata is compliant with the Dublin Core Cross-Domain Attribute Set metadata standard, but we use different formatting. We offer simple re-formatting from the richer DataCite to Dublin Core for interoperability with a wider set of data sources.
We use all mandatory DataCite metadata fields, all the the recommended and optional ones.
It complies with the tidy data principles.

In other words: very easy to import into your databases, or join with other databases, and the information is easy to find. Corrections, updates can automatically managed.

What happened with the data before?

We are creating Codebooks that are following the SDMX statistical metadata codelists, and resemble the SMDX concepts used by international statistical agencies. (See more technical information here.)

Small organizations often cannot afford to have data engineers and data scientists on staff, and they employ analysts who work with Excel, OpenOffice, PowerBI, SPSS or Stata. The problem with these applications is that they often require the user to manually adjust the data, with keyboard entries or mouse clicks. Furthermore, they do not provide a precise logging of the data processing, manipulation history. The manual data processing and manipulation is very error prone and makes the use of complex and high value resources, such as harmonized surveys or symmetric input-output tables, to name two important source we deal with, impossible to use. The use of these high-value data sources often requires tens of thousands of data processing steps: no human can do it faultlessly.

What is even more problematic that simple applications for analysis do not provide a log of these manipulations’ steps: pulling over a column with the mouse, renaming a row, adding a zero to an empty cell. This makes senior supervisory oversight and external audit very costly.

Our data comes with full history: all changes are visible, and we even open the code or algorithm that processed the raw data. Your analysts can still use their favourite spreadsheet or statistical software application, but they can start from a clean, tidy dataset, with all data wrangling, currency and unit conversion, imputation and other low-priority but important tasks done and logged.

Survey Harmonization

Mon, 05 Jul 2021 08:00:00 +0000

We provide retrospecitve, ex post, and ex ante survey harmonization to our partners.

The aim of retrospective survey harmonization is to pool data from pre-existing surveys made with a similar methodology in different points in time and different countries or territories. Ex post survey harmonization is in a way a passive form of pooling research funding because you can utilize information from surveying that were made on somebody else’s expense.

The Arab Barometer surveys do not have a consolidated codebook, but our retroharmonize software created one, and put together data from three years and collected in many countries about various public policy issues.

The aim of ex ante survey harmonization is to maximize the value from future retrospective harmonization; in a way, it is an active form of pooling research funding, because you benefit from money spent on related open governmental and open science survey programs.

In this example we designed a survey representative among music professionals that it can be compared with large-sample, national surveys on living conditions and attitudes, and with occupational groups. Nationally representative surveys do not question enough musicians to allow such specific use; musician only surveys do not allow comparison.

retorhamonize is a peer-reviewed, scientfic statistcal software that allows the programmatic retrospective harmonization of surveys, such as the last 35 years of all Eurobarometer microdata, or all Afrobarometer microdata. Eurobarometer grew out of certain CEE member states’ need for comparable data about their music and audiovisual sectors. We commissioned surveys following ESSNet-Culture guidelines and combined our survey data with open access European microdata-level surveys.

regions solves the problems caused by Europe’s shifting regional boundaries, which have undergone changes in several thousand places over the last twenty years, meaning member states’ and Eurostat’s regional statistics are not comparable over more than two to three years. This software validates and, where possible, changes the regional coding from NUTS1999 until the not yet used NUTS2021, opening up vast, valuable, untapped data sources that can be used for longitudinal analysis or for panel analysis far more precise than what national data alone would allow. It was originally designed in a research project at IVIR in the University of Amsterdam to understand the geographical dynamics of book piracy. Because of the needs this software fills, it had 700 users in the first month after publication. It is particularly useful to re-code old surveys, as regional boundaries are changing in each decade several hundred times in Europe.

Open Data - The New Gold Without the Rush

Fri, 18 Jun 2021 17:00:00 +0000

Engage with us on LinkedIn @CultDataObs or check out our open data and open repositories, code, tutorials

If open data is the new gold, why even those who release fail to reuse it? We created an open collaboration of data curators and open-source developers to dig into novel open data sources and/or increase the usability of existing ones. We transform reproducible research software into research- as-service.

Every year, the EU announces that billions and billions of data are now “open” again, but this is not gold. At least not in the form of nicely minted gold coins, but in gold dust and nuggets found in the muddy banks of chilly rivers. There is no rush for it, because panning out its value requires a lot of hours of hard work. Our goal is to automate this work to make open data usable at scale, even in trustworthy AI solutions.

There is no rush for it, because panning out its value requires a lot of hours of hard work. Our goal is to automate this work to make open data usable at scale, even in trustworthy AI solutions.

Most open data is not public, it is not downloadable from the Internet – in the EU parlance, “open” only means a legal entitlement to get access to it. And even in the rare cases when data is open and public, often it is mired by data quality issues. We are working on the prototypes of a data-as-service and research-as-service built with open-source statistical software that taps into various and often neglected open data sources.

We are in the prototype phase in June and our intentions are to have a well-functioning service by the time of the conference, because we are working only with open-source software elements; our technological readiness level is already very high. The novelty of our process is that we are trying to further develop and integrate a few open-source technology items into technologically and financially sustainable data-as-service and even research-as-service solutions.

Our review of about 80 EU, UN and OECD data observatories reveals that most of them do not use these organizations’s open data - instead they use various, and often not well processed proprietary sources.

We are taking a new and modern approach to the data observatory concept, and modernizing it with the application of 21st century data and metadata standards, the new results of reproducible research and data science. Various UN and OECD bodies, and particularly the European Union support or maintain more than 60 data observatories, or permanent data collection and dissemination points, but even these do not use these organizations and their members open data. We are building open-source data observatories, which run open-source statistical software that automatically processes and documents reusable public sector data (from public transport, meteorology, tax offices, taxpayer funded satellite systems, etc.) and reusable scientific data (from EU taxpayer funded research) into new, high quality statistical indicators.

We are taking a new and modern approach to the ‘data observatory’ concept, and modernizing it with the application of 21st century data and metadata standards, the new results of reproducible research and data science

We are building various open-source data collection tools in R and Python to bring up data from big data APIs and legally open, but not public, and not well served data sources. For example, we are working on capturing representative data from the Spotify API or creating harmonized datasets from the Eurobarometer and Afrobarometer survey programs.
Open data is usually not public; whatever is legally accessible is usually not ready to use for commercial or scientific purposes. In Europe, almost all taxpayer funded data is legally open for reuse, but it is usually stored in heterogeneous formats, processed into an original government or scientific need, and with various and low documentation standards. Our expert data curators are looking for new data sources that should be (re-) processed and re-documented to be usable for a wider community. We would like to introduce our service flow, which touches upon many important aspects of data scientist, data engineer and data curatorial work.
We believe that even such generally trusted data sources as Eurostat often need to be reprocessed, because various legal and political constraints do not allow the common European statistical services to provide optimal quality data – for example, on the regional and city levels.
With rOpenGov and other partners, we are creating open-source statistical software in R to re-process these heterogenous and low-quality data into tidy statistical indicators to automatically validate and document it.
We are carefully documenting and releasing administrative, processing, and descriptive metadata, following international metadata standards, to make our data easy to find and easy to use for data analysts.
We are automatically creating depositions and authoritative copies marked with an individual digital object identifier (DOI) to maintain data integrity.
We are building simple databases and supporting APIs that release the data without restrictions, in a tidy format that is easy to join with other data, or easy to join into databases, together with standardized metadata.
We maintain observatory websites (see: Digital Music Observatory, Green Deal Data Observatory, Economy Data Observatory) where not only the data is available, but we provide tutorials and use cases to make it easier to use them. Our mission is to show a modern, 21st century reimagination of the data observatory concept developed and supported by the UN, EU and OECD, and we want to show that modern reproducible research and open data could make the existing 60 data observatories and the planned new ones grow faster into data ecosystems.

We are working around the open collaboration concept, which is well-known in open source software development and reproducible science, but we try to make this agile project management methodology more inclusive, and include data curators, and various institutional partners into this approach. Based around our early-stage startup, Reprex, and the open-source developer community rOpenGov, we are working together with other developers, data scientists, and domain specific data experts in climate change and mitigation, antitrust and innovation policies, and various aspects of the music and film industry.

Our open collaboration is truly open: new data curators,developers and service designers, even volunteers and citizen scientists are welcome to join.

Our open collaboration is truly open: new data curators, data scientists and data engineers are welcome to join. We develop open-source software in an agile way, so you can join in with an intermediate programming skill to build unit tests or add new functionality, and if you are a beginner, you can start with documentation and testing our tutorials. For business, policy, and scientific data analysts, we provide unexploited, exciting new datasets. Advanced developers can join our development team: the statistical data creation is mainly made in the R language, and the service infrastructure in Python and Go components.

Analyze Locally, Act Globally: New regions R Package Release

Wed, 16 Jun 2021 12:00:00 +0000

The new version of our rOpenGov R package regions was released today on CRAN. This package is one of the engines of our experimental open data-as-service Green Deal Data Observatory, Economy Data Observatory, Digital Music Observatory prototypes, which aim to place open data packages into open-source applications.

In international comparison the use of nationally aggregated indicators often have many disadvantages: they inhibit very different levels of homogeneity, and data is often very limited in number of observations for a cross-sectional analysis. When comparing European countries, a few missing cases can limit the cross-section of countries to around 20 cases which disallows the use of many analytical methods. Working with sub-national statistics has many advantages: the similarity of the aggregation level and high number of observations can allow more precise control of model parameters and errors, and the number of observations grows from 20 to 200-300.

The change from national to sub-national level comes with a huge data processing price: internal administrative boundaries, their names, codes codes change very frequently.

Yet the change from national to sub-national level comes with a huge data processing price. While national boundaries are relatively stable, with only a handful of changes in each recent decade. The change of national boundaries requires a more-or-less global consensus. But states are free to change their internal administrative boundaries, and they do it with large frequency. This means that the names, identification codes and boundary definitions of sub-national regions change very frequently. Joining data from different sources and different years can be very difficult.

Our regions R package helps the data processing, validation and imputation of sub-national, regional datasets and their coding.

There are numerous advantages of switching from a national level of the analysis to a sub-national level comes with a huge price in data processing, validation and imputation, and the regions package aims to help this process.

You can review the problem, and the code that created the two map comparisons, in the Maping Regional Data, Maping Metadata Problems vignette article of the package. A more detailed problem description can be found in Working With Regional, Sub-National Statistical Products.

This package is an offspring of the eurostat package on rOpenGov. It started as a tool to validate and re-code regional Eurostat statistics, but it aims to be a general solution for all sub-national statistics. It will be developed parallel with other rOpenGov packages.

Get the Package

You can install the development version from GitHub with:

devtools::install_github("rOpenGov/regions")

or the released version from CRAN:

install.packages("regions")

You can review the complete package documentation on regions.dataobservaotry.eu. If you find any problems with the code, please raise an issue on Github. Pull requests are welcome if you agree with the Contributor Code of Conduct

If you use regions in your work, please cite the package as: Daniel Antal. (2021, June 16). regions (Version 0.1.7). CRAN. http://doi.org/10.5281/zenodo.4965909

Download the BibLaTeX entry.

Join us

Join our open collaboration Green Deal Data Observatory team as a data curator, developer or business developer. More interested in antitrust, innovation policy or economic impact analysis? Try our Economy Data Observatory team! Or your interest lies more in data governance, trustworthy AI and other digital market problems? Check out our Digital Music Observatory team!

Join Copernicus Climate Data Store Data with Socio-Economic and Opinion Poll Data

Sun, 06 Jun 2021 10:00:00 +0000

In this series of blogposts we will show how to collect environmental data from the EU’s Copernicus Climate Data Store, and bring it to a data format that you can join with Eurostat’s socio-economic and environmental data. We have shown in a previous blogpost how to connect this to survey (opinion poll) and tax data, and a real policy problem in Belgium. We will create now subsequent tutorials to do more!

Engage with us on LinkedIn @CultDataObs or check out our open data and open repositories, code, tutorials

But first, why are we doing this? The European Union and its members states are releasing every year more and more data for open re-use since 2003, yet these are often not used in the EU’s data dissemination projects (the observatories) or in EU-funded research. We believe that there are many reasons behind this. Whilst more and more people can conduct business, scientific or policy analysis programmatically or with statistical software, knowledge how to systematically collect the data from the exponentially growing availability is not everybody’s specialty. And the lack of documentation, and high re-processing and validation need for open data is another drawback.

rOpenGov has long been producing high-quality, peer-reviewed R packages to work with open data, but their use is not for all. In an open collaboration, where you can join, too, rOpenGov teamed up with open source developers, knowledgeable data curators, and a service developer team lead by the Dutch reproducible research start-up Reprex to create a sustainable infrastructure that is permanently collecting, processing, documenting and visualizing open data. What we do is that we access open data (that is not always available for direct download) and re-process it to usable data that is tidy to be integrated with your existing data or databases. We are competing for the EU Datathon Challenge 1: supporting a European Green Deal agenda with open data as a service, and research as a servcie, and you are more than welcome to join our effort as a developer, a data curator, or as an occasional contributor to open government packages.

Register to the Copernicus Climate Data Store

Koen Hufkens, Reto Stauffer and Elio Campitelli created the ecmwfr R package for programmatically accessing the Copernicus Data Store service. Follow the CDS Functionality vignette to get started.

You will need to create a Register yourself for CDS services after accepting the Terms and conditions.

wf_set_key(user: "12345", 
           key: "00000000-aaaa-b1b1-0000-a1a1a1a1a1a1", 
           service: "cds")

You can check if you were successful with:

ecmwfr::wf_get_key(user: "12345", service: "cds")

Get the Data

Let us formulate our first request:

request_lai_hv_2019_06 <- list(
  "dataset_short_name": "reanalysis-era5-land-monthly-means",
  "product_type"  : "monthly_averaged_reanalysis",
  "variable"      : "leaf_area_index_high_vegetation",
  "year"          : "2019",
  "month"         :  "06",
  "time"          : "00:00",
  "area"          : "70/-20/30/60",
  "format"        : "netcdf",
  "target"        : "demo_file.nc")

lai_hv_2019_06.nc  <- wf_request(user: "<your_ID>",
                     request: request_lai_hv_2019_06 ,
                     transfer: TRUE,
                     path: "data-raw",
                     verbose: FALSE)

Effective Leaf Area Index

You can find this data either in global computer raster images, or in re-processed monthly averages. Working with the raw data is not very practical – in case of cloudy weather you have missing data, and the files are extremely huge for a personal computer. For the purposes of our Green Deal Data Observatory the monthly average values are far more practical, which are called monthly_averaged_reanalysis product types.

For compatibility with other R packages, convert the data with the from raster package from rSpatial.org.

lai_file <- here::here( "data-raw", "demo_file.nc")
lai_raster <- raster::raster(lai_file)

## Loading required namespace: ncdf4

Let us convert this to a SpatialDataPointsDataFrame class, which is an augmented data frame class with coordinates.

LAI_df <- raster::rasterToPoints(lai_raster, fun=NULL, spatial=TRUE)

Get The Map

With the help fo rOpenGov, we are creating various R packages to programmatically access open data and put them into the right format. The popular eurostat package is not only useful to download data from Eurostat, but also to map it.

In this case, we want to create regional maps. Europe has five levels of geographical regions: NUTS0 for countries, NUTS1 for larger areas like states, provinces; NUTS2 for smaller areas like countries, NUTS3 for even smaller areas. The LAU level contains settlemens and their surrounding areas.

Country borders change sometimes (think about the unification of Germany, or the breakup of Czechoslovakia and Yugoslavia), but they are relatively stable entities. Sub-national regional border change very-very frequently – since 2000 there were many thousand changes in Europe. This means that you must choose one regional boundary definition. The latest edition is NUTS2021 but most of the data available is still in the NUTS2016 format, and often you will find NUTS2013 or even NUTS2010 data around. Our Green Deal Data Observatory uses the NUTS2016 definition, because it is far the most used in 2021. An offspring of the eurostat package, regions helps you take care of NUTS changes when you work, and can convert your data to NUTS2021 if you later need it.

## sf at resolution 1:60 read from local file

## Warning in eurostat::get_eurostat_geospatial(resolution: "60", nuts_level =
## "2", : Default of 'make_valid' for 'output_class="sf"' will be changed in the
## future (see function details).

plot(map_nuts_2)

Our measurement of the average Effective Leaf Area Index is a raster data, it is given for many points of Europe’s map. What we need to do is to overlay this raster information of the statistical map of Europe. We use the excellent sp: R Classes and Methods for Spatial Data package for this purpose. The sp::over() function decides if a point of Leaf Area Index measurement falls into the polygon (shape) of a particular NUTS2 regions, for example, Zuid-Holland or South Holland in the Netherlands, or Saarland in Germany, or not. Then it averages with the mean() function those measurements falling in the area.

LAI_nuts_2: sp::over(sp::geometry(
  as(map_nuts_2, 'Spatial')), 
  LAI_df,
  fn=mean)

Let’s call the average LAI index lai, and bind it to the Eurostat map:

names(LAI_nuts_2)[1] <- "lai"
LAI_sfdf <- bind_cols ( map_nuts_2, LAI_nuts_2 )

If you want to work with the data in a numeric context, you do not need the geographical information, and you can “downgrade” the SpatialDataPointsDataFrame to a simple data frame.

set.seed(2019) #to always see the same sample
LAI_sfdf %>%
  as.data.frame() %>%
  select ( all_of(c("NUTS_NAME", "NUTS_ID", "lai")) ) %>%
  sample_n(10)

##                      NUTS_NAME NUTS_ID lai
## 281                       Vest    RO42  NA
## 125                     Kassel    DE73  NA
## 69              Friesland (NL)    NL12  NA
## 237 Agri, Kars, Igdir, Ardahan    TRA2  NA
## 273                East Anglia    UKH1  NA
## 119                Prov. Liège    BE33  NA
## 61                   Bourgogne    FRC1  NA
## 275                      Essex    UKH3  NA
## 282                   Istanbul    TR10  NA
## 174                    Leipzig    DED5  NA

We’ll plot the map with ggplot2.

library(ggplot2)
library(sf)
ggplot(data=LAI_sfdf) + 
  geom_sf(aes(fill=lai),
          color="dim grey", size=.1) + 
  scale_fill_gradient( low ="#FAE000", high: "#00843A") +
  guides(fill: guide_legend(reverse=T, title: "LAI")) +
  labs(title="Leaf Area Index",
       subtitle: "High vegetation half, NUTS2 regional avareage values",
       caption="\ua9 EuroGeographics for the administrative boundaries 
                \ua9 Copernicus Data Service, June 2019 average values
                Tutorial and ready-to-use data on greendeal.dataobservatory.eu") +
  theme_light() + theme(legend.position=c(.88,.78)) +
  coord_sf(xlim=c(-22,48), ylim=c(34,70))

Data Integrity

Our Green Deal Data Observatory has a data API where we place the new data with metadata for programmatic download in CSV, JSON or even with SQL queries. For data integrity purposes, we are placing an authoritative copy on Zenodo (Green Deal Data Observatory Community). You can use this for scientific citations. We are also happy if you place your own climate policy related research data here, so that we can include it in our observatory. In our subsequent tutorials, we will show how to do this programmatically in R. This particular dataset (not only with the month June, which we selected to streamline the tutorial) is available here with the digital object identifier doi.org/10.5281/zenodo.4903940.

Join us

Economic and Environment Impact Analysis, Automated for Data-as-Service

Thu, 03 Jun 2021 16:00:00 +0000

We have released a new version of iotables as part of the rOpenGov project. The package, as the name suggests, works with European symmetric input-output tables (SIOTs). SIOTs are among the most complex governmental statistical products. They show how each country’s 64 agricultural, industrial, service, and sometimes household sectors relate to each other. They are estimated from various components of the GDP, tax collection, at least every five years.

SIOTs offer great value to policy-makers and analysts to make more than educated guesses on how a million euros, pounds or Czech korunas spent on a certain sector will impact other sectors of the economy, employment or GDP. What happens when a bank starts to give new loans and advertise them? How is an increase in economic activity going to affect the amount of wages paid and and where will consumers most likely spend their wages? As the national economies begin to reopen after COVID-19 pandemic lockdowns, is to utilize SIOTs to calculate direct and indirect employment effects or value added effects of government grant programs to sectors such as cultural and creative industries or actors such as venues for performing arts, movie theaters, bars and restaurants.

Making such calculations requires a bit of matrix algebra, and understanding of input-output economics, direct, indirect effects, and multipliers. Economists, grant designers, policy makers have those skills, but until now, such calculations were either made in cumbersome Excel sheets, or proprietary software, as the key to these calculations is to keep vectors and matrices, which have at least one dimension of 64, perfectly aligned. We made this process reproducible with iotables and eurostat on rOpenGov

Our iotables package creates direct, indirect effects and multipliers programatically. Our observatory will make those indicators available for all European countries.

Accessing and tidying the data programmatically

The iotables package is in a way an extension to the eurostat R package, which provides a programmatic access to the Eurostat data warehouse. The reason for releasing a new package is that working with SIOTs requires plenty of meticulous data wrangling based on various metadata sources, apart from actually accessing the data itself. When working with matrix equations, the bar is higher than with tidy data. Not only your rows and columns must match, but their ordering must strictly conform the quadrants of the a matrix system, including the connecting trade or tax matrices.

When you download a country’s SIOT table, you receive a long form data frame, a very-very long one, which contains the matrix values and their labels like this:

## Table naio_10_cp1700 cached at C:\Users\...\Temp\RtmpGQF4gr/eurostat/naio_10_cp1700_date_code_FF.rds

# we save it for further reference here 
saveRDS(naio_10_cp1700, "not_included/naio_10_cp1700_date_code_FF.rds")

# should you need to retrieve the large tempfiles, they are in 
dir (file.path(tempdir(), "eurostat"))

dplyr::slice_head(naio_10_cp1700, n: 5)

## # A tibble: 5 x 7
##   unit    stk_flow induse  prod_na geo       time        values
##   <chr>   <chr>    <chr>   <chr>   <chr>     <date>       <dbl>
## 1 MIO_EUR DOM      CPA_A01 B1G     EA19      2019-01-01 141873.
## 2 MIO_EUR DOM      CPA_A01 B1G     EU27_2020 2019-01-01 174976.
## 3 MIO_EUR DOM      CPA_A01 B1G     EU28      2019-01-01 187814.
## 4 MIO_EUR DOM      CPA_A01 B2A3G   EA19      2019-01-01      0 
## 5 MIO_EUR DOM      CPA_A01 B2A3G   EU27_2020 2019-01-01      0

The metadata reads like this: the units are in millions of euros, we are analyzing domestic flows, and the national account items B1-B2 for the industry A01. The information of a 64x64 matrix (the SIOT) and its connecting matrices, such as taxes, or employment, or C**O₂ emissions, must be placed exactly in one correct ordering of columns and rows. Every single data wrangling error will usually lead in an error (the matrix equation has no solution), or, what is worse, in a very difficult to trace algebraic error. Our package not only labels this data meaningfully, but creates very tidy data frames that contain each necessary matrix of vector with a key column.

iotables package contains the vocabularies (abbreviations and human readable labels) of three statistical vocabularies: the so called COICOP product codes, the NACE industry codes, and the vocabulary of the ESA2010 definition of national accounts (which is the government equivalent of corporate accounting).

Our package currently solves all equations for direct, indirect effects, multipliers and inter-industry linkages. Backward linkages show what happens with the suppliers of an industry, such as catering or advertising in the case of music festivals, if the festivals reopen. The forward linkages show how much extra demand this creates for connecting services that treat festivals as a ‘supplier’, such as cultural tourism.

Let’s seen an example

## Downloading employment data from the Eurostat database.

## Table lfsq_egan22d cached at C:\Users\...\Temp\RtmpGQF4gr/eurostat/lfsq_egan22d_date_code_FF.rds

and match it with the latest structural information on from the Symmetric input-output table at basic prices (product by product) Eurostat product. A quick look at the Eurostat website already shows that there is a lot of work ahead to make the data look like an actual Symmetric input-output table. Download it with iotable_get() which does basic labelling and preprocessing on the raw Eurostat files. Because of the size of the unfiltered dataset on Eurostat, the following code may take several minutes to run.

sk_io <-  iotable_get ( labelled_io_data: NULL, 
                        source: "naio_10_cp1700", geo: "SK", 
                        year: 2015, unit: "MIO_EUR", 
                        stk_flow: "TOTAL",
                        labelling: "iotables" )

## Reading cache file C:\Users\..\Temp\RtmpGQF4gr/eurostat/naio_10_cp1700_date_code_FF.rds

## Table  naio_10_cp1700  read from cache file:  C:\Users\..\Temp\RtmpGQF4gr/eurostat/naio_10_cp1700_date_code_FF.rds

## Saving 808 input-output tables into the temporary directory
## C:\Users\...\Temp\RtmpGQF4gr

## Saved the raw data of this table type in temporary directory C:\Users\...\Temp\RtmpGQF4gr/naio_10_cp1700.rds.

The input_coefficient_matrix_create() creates the input coefficient matrix, which is used for most of the analytical functions.

a_i**j: X_i**j / x_j

It checks the correct ordering of columns, and furthermore it fills up 0 values with 0.000001 to avoid division with zero.

input_coeff_matrix_sk <- input_coefficient_matrix_create(
  data_table: sk_io
)

## Columns and rows of real_estate_imputed_a, extraterriorial_organizations are all zeros and will be removed.

Then you can create the Leontieff-inverse, which contains all the structural information about the relationships of 64x64 sectors of the chosen country, in this case, Slovakia, ready for the main equations of input-output economics.

I_sk <- leontieff_inverse_create(input_coeff_matrix_sk)

And take out the primary inputs:

primary_inputs_sk <- coefficient_matrix_create(
  data_table: sk_io, 
  total: 'output', 
  return: 'primary_inputs')

## Columns and rows of real_estate_imputed_a, extraterriorial_organizations are all zeros and will be removed.

Now let’s see if there the government tries to stimulate the economy in three sectors, agricultulre, car manufacturing, and R&D with a billion euros. Direct effects measure the initial, direct impact of the change in demand and supply for a product. When production goes up, it will create demand in all supply industries (backward linkages) and create opportunities in the industries that use the product themselves (forward linkages.)

direct_effects_create( primary_inputs_sk, I_sk ) %>%
  select ( all_of(c("iotables_row", "agriculture",
                    "motor_vechicles", "research_development"))) %>%
  filter (.data$iotables_row %in% c("gva_effect", "wages_salaries_effect", 
                                    "imports_effect", "output_effect"))

##            iotables_row agriculture motor_vechicles research_development
## 1        imports_effect   1.3684350       2.3028203            0.9764921
## 2 wages_salaries_effect   0.2713804       0.3183523            0.3828014
## 3            gva_effect   0.9669621       0.9790771            0.9669467
## 4         output_effect   2.2876287       3.9840251            2.2579634

Car manufacturing requires much imported components, so each extra demand will create a large importing activity. The R&D will create a the most local wages (and supports most jobs) because research is job-intensive. As we can see, the effect on imports, wages, gross value added (which will end up in the GDP) and output changes are very different in these three sectors.

This is not the total effect, because some of the increased production will translate into income, which in turn will be used to create further demand in all parts of the domestic economy. The total effect is characterized by multipliers.

Then solve for the multipliers:

multipliers_sk <- input_multipliers_create( 
  primary_inputs_sk %>%
    filter (.data$iotables_row == "gva"), I_sk )

And select a few industries:

set.seed(12)
multipliers_sk %>% 
  tidyr::pivot_longer ( -all_of("iotables_row"), 
                        names_to: "industry", 
                        values_to: "GVA_multiplier") %>%
  select (-all_of("iotables_row")) %>%
  arrange( -.data$GVA_multiplier) %>%
  dplyr::sample_n(8)

## # A tibble: 8 x 2
##   industry               GVA_multiplier
##   <chr>                           <dbl>
## 1 motor_vechicles                  7.81
## 2 wood_products                    2.27
## 3 mineral_products                 2.83
## 4 human_health                     1.53
## 5 post_courier                     2.23
## 6 sewage                           1.82
## 7 basic_metals                     4.16
## 8 real_estate_services_b           1.48

Vignettes

The Germany 1990 provides an introduction of input-output economics and re-creates the examples of the Eurostat Manual of Supply, Use and Input-Output Tables, by Jörg Beutel (Eurostat Manual).

The United Kingdom Input-Output Analytical Tables Daniel Antal, based on the work edited by Richard Wild is a use case on how to correctly import data from outside Eurostat (i.e., not with eurostat::get_eurostat()) and join it properly to a SIOT. We also used this example to create unit tests of our functions from a published, official government statistical release.

Finally, Working With Eurostat Data is a detailed use case of working with all the current functionalities of the package by comparing two economies, Czechia and Slovakia and guides you through a lot more examples than this short blogpost.

Our package was originally developed to calculate GVA and employment effects for the Slovak music industry, and similar calculations for the Hungarian film tax shelter. We can now programatically create reproducible multipliers for all European economies in the Digital Music Observatory, and create further indicators for economic policy making in the Economy Data Observatory.

Environmental Impact Analysis

Our package allows the calculation of various economic policy scenarios, such as changing the VAT on meat or effects of re-opening music festivals on aggregate demand, GDP, tax revenues, or employment. But what about the C**O₂, methane and other greenhouse gas effects of the reopening festivals, or the increasing meat prices?

Technically our package can already calculate such effects, but to do so, you have to carefully match further statistical vocabulary items used by the European Environmental Agency about air pollutants and greenhouse gases.

The last released version of iotables is Importing and Manipulating Symmetric Input-Output Tables (Version 0.4.4). Zenodo. https://doi.org/10.5281/zenodo.4897472, but we are already working on a new major release. (Download the BibLaTeX entry.) In that release, we are planning to build in the necessary vocabulary into the metadata functions to increase the functionality of the package, and create new indicators for our Green Deal Data Observatory. This experimental data observatory is creating new, high quality statistical indicators from open governmental and open science data sources that has not seen the daylight yet.

rOpenGov and the EU Datathon Challenges

rOpenGov, Reprex, and other open collaboration partners teamed up to build on our expertise of open source statistical software development further: we want to create a technologically and financially feasible data-as-service to put our reproducible research products into wider user for the business analyst, scientific researcher and evidence-based policy design communities.

rOpenGov is a community of open governmental data and statistics developers with many packages that make programmatic access and work with open data possible in the R language. Reprex is a Dutch-startup that teamed up with rOpenGov and other open collaboration partners to create a technologically and financially feasible service to exploit reproducible research products for the wider business, scientific and evidence-based policy design community. Open data is a legal concept - it means that you have the rigth to reuse the data, but often the reuse requires significant programming and statistical know-how. We entered into the annual EU Datathon competition in all three challenges with our applications to not only provide open-source software, but daily updated, validated, documented, high-quality statistical indicators as open data in an open database. Our iotables package is one of our many open-source building blocks to make open data more accessible to all.

Join our open collaboration Economy Data Observatory team as a data curator, developer or business developer. More interested in economic policies, particularly computation antitrust, innovation and small enterprises? Check out our Economy Music Observatory team! Or your interest lies more in data governance, trustworthy AI and other digital market problems? Check out our Digital Music Observatory team!

Data API

Tue, 01 Jun 2021 11:00:00 +0000

Our observatory has a new data API which allows access to our daily refreshing open data. You can access the API via api.economy.dataobservatory.eu (apologies for the ugly, temporary subdomain masking!).

All the data and the metadata are available as open data, without database use restrictions, under the ODbL license. However, the metadata contents are not finalized yet. We are currently working on a solution that applies the FAIR Guiding Principles for scientific data management and stewardship, and fulfills the mandatory requirements of the Dublic Core metadata standards and at the same time the mandatory requirements, and most of the recommended requirements of DataCite. These changes will be effective before 1 July 2021.

The Competition Data Observatory temporarily shares an API with the Economy Data Observatory, which serves as an incubator for similar economy-oriented reproducible research resources.

Indicator table

The indicator table contains the actual values, and the various estimated/imputed values of the indicator, clearly marking missing values, too.

api.economy.dataobservatory.eu: indicator retrieval

You can get the data in CSV or json format, or write SQL querries. (Tutorials in SQL, R, Python will be posted shortly.)

Description metadata table

Processing Metadata table

The metadata table contains various data processing information, such as the first and last actual observation of the indicator, the number of approximated, forecasted, backcasted values, last update at source and in our system, and so on.

api.economy.dataobservatory.eu: processing metadata

Authoritative Copies

Greendeal Data Observatory on Zenodo

Metadata

Tue, 01 Jun 2021 11:00:00 +0000

The Competition Data Observatory temporarily shares an API with the Economy Data Observatory, which serves as an incubator for similar economy-oriented reproducible research resources.

api.economy.dataobservatory.eu: processing metadata

Descriptive Metadata


Identifier	An unambiguous reference to the resource within a given context. (Dublin Core item), but several identifiders allowed, and we will use several of them.
Creator	The main researchers involved in producing the data, or the authors of the publication, in priority order. To supply multiple creators, repeat this property. (Extends the Dublin Core with multiple authors, and legal persons, and adds affiliation data.)
Title	A name given to the resource. Extends Dublin Core with alternative title, subtitle, translated Title, and other title(s).
Publisher	The name of the entity that holds, archives, publishes prints, distributes, releases, issues, or produces the resource. This property will be used to formulate the citation, so consider the prominence of the role. For software, use Publisher for the code repository. (Dublin Core item.)
Publication Year	The year when the data was or will be made publicly available.
Resource Type	We publish Datasets, Images, Report, and Data Papers. (Dublin Core item with controlled vocabulary.)

Recommended for discovery

The Recommended (R) properties are optional, but strongly recommended for interoperability.


Subject	The topic of the resource. (Dublin Core item.)
Contributor	The institution or person responsible for collecting, managing, distributing, or otherwise contributing to the development of the resource. (Extends the Dublin Core with multiple authors, and legal persons, and adds affiliation data.) When applicable, we add Distributor (of the datasets and images), Contact Person, Data Collector, Data Curator, Data Manager, Hosting Institution, Producer (for images), Project Manager, Researcher, Research Group, Rightsholder, Sponsor, Supervisor
Date	A point or period of time associated with an event in the lifecycle of the resource, besides the Dublin Core minimum we add Collected, Created, Issued, Updated, and if necessary, Withdrawn dates to our datasets.
Related Identifier	An identifier or identifiers other than the primary Identifier applied to the resource being registered.
Rights	We give SPDX License List standards rights description with URLs to the actual license. (Dublin Core item: Rights Management)
Description	Recommended for discovery.(Dublin Core item.)
GeoLocation	Similar to Dublin Core item Coverage

The Subject property: we need to set standard coding schemas for each observatory.
Contributor property:
- DataCurator the curator of the dataset, who sets the mandatory properties.
- DataManager the person who keeps the dataset up-to-date.
- ContactPerson the person who can be contacted for reuse requests or bug reports.
The Date property contains the following dates, which are set automatically by the dataobservatory R package:
- Updated when the dataset was updated;
- EarliestObservation, which the earliest, not backcasted, estimated or imputed observation.
- LatestObservation, which the earliest, not backcasted, estimated or imputed observation.
- UpdatedatSource, when the raw data source was last updated.
The GeoLocation is automatically created by the dataobservatory R package.
The Description property optional elements, and we adopted them as follows for the observatories:
- The Abstract is a short, textual description; we try to automate its creation as much as a possible, but some curatorial input is necessary.
- In the TechnicalInfo sub-field, we record automatically the utils::sessionInfo() for computational reproducability. This is automatically created by the dataobservatory R package.
- In the Other sub-field, we record the keywords for structuring the observatory.

Optional

The Optional (O) properties are optional and provide richer description. For findability they are not so important, but to create a web service, they are essential. In the mandatory and recommended fields, we are following other metadata standards and codelists, but in the optional fields we have to build up our own system for the observatories.


Language	A language of the resource. (Dublin Core item.)
Alternative Identifier	An identifier or identifiers other than the primary Identifier applied to the resource being registered.
Size	We give the CSV, downloadable dataset size in bytes.
Format	We give file format information. We mainly use CSV and JSON, and occasionally rds and SPSS types. (Dublin Core item.)
Version	The version number of the resource.
Rights	We give SPDX License List standards rights description with URLs to the actual license. (Dublin Core item: Rights Management)
Funding Reference	We provide the funding reference information when applicable. This is usually mandatory with public funds.
Related Item	We give information about our observatory partners’ related research products, awards, grants (also Dublin Core item as Relation.) We particularly include source information when the dataset is derived from another resource (which is a Dublin Core item.)

In the Language we only use English (eng) at the moment.
By default We do not use the Alternative Identifier property. We will do this when the same dataset will be used in several observatories.
The Size property is measured in bytes for the CSV representation of the dataset. During creations, the software creates a temporary CSV file to check if the dataset has no writing problems, and measures the dataset size.
The Version property needs further work. For a daily re-freshing API we need to find an applicable versioning system.
The Funding reference will contain information for donors, sponsors, and co-financing partners.
Our default setting for Rights is the CC-BY-NC-SA-4.0 license and we provide an URI for the license document.
In the RelatedItem we give information about:
- The original (raw) data source.
- Methodological bibilography reference, when needed.
- The open-source statistical software code that processed the data.

Administrative (Processing) Metadata

Administrative Metadata

Like with diamonds, it is better to know the history of a dataset, too. Our administrative metadata contains codelists that follow the SXDX statistical metadata standards, and similarly strucutred information about the processing history of the dataset.

See for further reference The codebook Class.


Observation Status	SDMX Code list for Observation Status 2.2 (CL_OBS_STATUS), such as actual, missing, imputed, etc. values.
Method	If the value is estimated, we provide modelling information.
Unit	We provide the measurement unit of the data (when applicable.)
Frequency	SDMX Code list for Frequency 2.1 (CL_FREQ) frequency values
Codelist	Euros-SDMX Codelist entries for the observational units, such as sex, etc.
Imputation	SDMX Code list for Frequency 2.1 (CL_IMPUT_METH) imputation values
Estimation	The estimation methodology of data that we calculated, together with citation information and URI to the actual processing code
Related Item	We give information about the software code that processed the data (both Dublin Core and DataCite compliant.)

See an example in the The codebook Class article of the dataobservatory R package.

Data Sharing

Sun, 16 May 2021 00:00:00 +0000

we would like to actively encourage the sharing of data assets.

Open Data

Sun, 16 May 2021 00:00:00 +0000

Many countries in the world allow access to a vast array of information, such as documents under freedom of information requests, statistics, datasets. In the European Union, most taxpayer financed data in government administration, transport, or meteorology, for example, can be usually re-used. More and more scientific output is expected to be reviewable and reproducible, which implies open access.

What’s the Problem with Open Data?

How We Add Value?

Is There Value in It?
If it’s money on the street, why nobody’s picking it up?

Datasets Should Work Together to Give Information
Data is only potential information, raw and unprocessed.

What’s the Problem with Open Data?

“Data is stuff. It is raw, unprocessed, possibly even untouched by human hands, unviewed by human eyes, un-thought-about by human minds.” [1]

Most open data cannot be just “downloaded.”
Often, you need to put more than $100 value of work into processing, validating, documenting a dataset that is worth $100. But you can share this investment with our data observatories.
Open data is almost always lacking of documentation, and no clear references to validate if the data is reliable or not corrupted. This is why we always start with reprocessing and redocumenting.

How We Add Value?

We believe that even such generally trusted data sources as Eurostat often need to be reprocessed, because various legal and political constraints do not allow the common European statistical services to provide optimal quality data – for example, on the regional and city levels.
With rOpenGov and other partners, we are creating open-source statistical software in R to re-process these heterogenous and low-quality data into tidy statistical indicators to automatically validate and document it.
Metadata is a potentially informative data record about a potentially informative dataset. We are carefully documenting and releasing administrative, processing, and descriptive metadata, following international metadata standards, to make our data easy to find and easy to use for data analysts.
We are automatically creating depositions and authoritative copies marked with an individual digital object identifier (DOI) to maintain data integrity.

Is There Value in Open Data?

A well-known story tells of a finance professor and a student who come across a $100 bill lying on the ground. As the student stops to pick it up, the professor says, “Don’t bother—if it were really a $100 bill, it wouldn’t be there.”

But this is not the case with open data. Often, you need to put more than $100 into processing, validating, documenting a dataset that is worth $100.

In the EU, open data is governed by the Directive on open data and the re-use of public sector information - in short: Open Data Directive (EU) 2019 / 1024. It entered into force on 16 July 2019. It replaces the Public Sector Information Directive, also known as the PSI Directive which dated from 2003 and was subsequently amended in 2013.

Open Data is potentially useful data that can potentially replace costlier or hard to get data sources to build information. It is analogous to potential energy: work is required to release it. We build automated systems that reduce this work and increase the likelihood that open data will offer the best value for money.

Most open data is not publicy accessible, and available upon request. Our real curatorial advantage is that we know where it is and how to get this request processed.
Most European open data comes from tax authorities, meteorological offices, managers of transport infrastructure, and other governmental bodies whose data needs are very different from yours. Their data must be carefully evaluated, re-processed, and if necessary, imputed to be usable for your scientific, business or policy goals.
The use of open science data is problematic in different ways: usually understanding the data documentation requires domain-specific specialist knowledge. Open science data is even more scattered and difficult to access than technically open, but not public governmental data.

From Datasets to Data Integration, Data to Information

“Data is only potential information, raw and unprocessed, prior to anyone actually being informed by it.” ^[2]

We are building simple databases and supporting APIs that release the data without restrictions, in a tidy format that is easy to join with other data, or easy to join into databases, together with standardized metadata.

Our service flow and value chain

FAQ

Why Downloading Does Not Work?

Most open data is not available on the internet.
If it is available, it is not in a form that you can easily import into a spreadsheet application like Excel or OpenOffice, or into a statistical application like SPSS or STATA.
Even the data quality of trusted web sources, like the Eurostat website, can be very low. Eurostat just publishes what it gets from governments, and often has no mandate to fix errors. The data is full with missing information, and in the case of regional statistics, faulty region codes and region names that make matching your data or placing them on a map impossible.
Adjusting euros with millions of euros, correctly translating dollars to euros, pounds to kilograms requires plenty of work. This is a very error-prone process when done by humans.

Can Open Data be Used in Machine Learning and AI?

Most public and open data sources have many missing observations; machine learning models usually cannot hanlde missingness. These points must be carefully imputed with approximations, which can be very challenging when the data has geographical dimension.
Removing missing values makes samples extremely biased and your model will learn from omissions, not information.

Photo Credits

What’s the Problem with Open Data? illustration is a photo by Cristina Gottardi How We Add Value? illustration is a photo by Nana Smirnova. Is There Value Left in It? is a photo by Imelda Datasets Should Work Together to Give Information is a photo by Lucas Santos

Footnote References

[1] Pomerantz, Jeffrey. 2021. “Metadata.” MIT Press essential knowledge series. MIT Press. Cambridge, Massachusetts ; London, England : The MIT Press, [2015]

[2] Pomerantz, Jeffrey. 2021. “Metadata.” MIT Press essential knowledge series. MIT Press. Cambridge, Massachusetts ; London, England : The MIT Press, [2015]

Feasibility Study On Promoting Slovak Music In Slovakia & Abroad

Thu, 25 Mar 2021 11:00:00 +0000

Download the study in Slovak or in English.

In 2015, realizing the low visibility and income-generating potential of Slovak music, the legislation introduced an amendment to the broadcasting act to regulate local content in radiostreams. The Slovak content promoting policy was well-intended but not based on any impact assessment, and it reached its goal only partially.

The Slovak broadcasting quotas in comparison with other national quotas a very simple, and they are impossible to measure, which makes both compliance and enforcement very difficult. Radio editors do not get any help to find music that fits into the playlists and fulfill the quota obligations – in many cases, it is impossible for them to find out if a song actually meets the quota requirements. For the same reason, neither is enforcement possible.

Another deficiency of the broadcasting quotas is that because of its fuzzy target, it is not clear whom it tries to help, and it has few friends. It is unclear how performers, composers or Slovak music producers can benefit from the system. Furthermore, it only helps a few genres, and it decreases the chances of other Slovak music in instrumental and non-Slovak language genres (for example, classical, jazz, rock) to be heard.

What should a good local content policy (radio quota, recommendation system, streaming quota) achieve?

And at last, radio is losing its importance in music discovery. New generation find the music during their music discovery age on YouTube and digital streaming platforms. A Slovak content promoting policy that does not work on digital streaming platforms will be obsolete when radio content providers will switch to digital streaming in the foreseeable future.

Our Feasibility Study follows the following logic: In the first chapter we introduce various music recommendation systems in the context of local content promotion polices, like local mandatory content quota regulations.

We aimed to create a demo version of a utility-based, transparent, accountable recommendation system.

In the second chapter, we consider the market-based or creative industry economy supporting policy goals, measurements, and potential support given to artists and producers.

We then turn in the third chapter to content-based local regulations promoting the use of the Slovak language or Slovak music content, irrespective of the performers and producers nationality, residence or ethnicity.

We introduce the idea of the Slovak Music Database, a comprehensive, mainly opt-in, opt-out database that of Slovak artists and Slovak music that should be supported by the local content regulation and other policies. We also create a Demo Slovak Music Database to understand the problem and scope of the creation of the comprehensive version.

The project website contains the Demo Slovak Music Database.

We also created a Demo Recommendation System which is currently not online. We explain here why.

Screenshot of the first verison of the demo app.

Research questions

Why are the total market shares of Slovak music relatively low both on the domestic and the foreign markets?
How can we measure the market share of the Slovak music in the domestic and foreign markets?
How can we measure the value gap between what some media platforms, most particularly the biggest YouTube, does not pay out to the Slovak stakeholders within Slovakia?
What is the interplay of the various definitions on market share and national quota targets?
How ‘shadow-markets’ of home copying and unlicensed media platforms, such as YouTube impact market shares directly and national quotas indirectly?
How can modern data science, predictive microeconomics and statistics help increase the market share of Slovak music in Slovakia and abroad?

We would like to continue this work in more depth in Slovakia, and include new countries and regions, for example, Estonia, Hungary, the Netherlands, Flanders and Wales in the next versions. We are also planning city-version.

Thanks for the entire Reprex team who contributed to the English version:

Budai, Sándor, programming and deployment
Dr. Emily H. Clarke, musicologist
Stef Koenis, musicologist, musician
Dr. Andrés Garcia Molina, data scientist, musicologist, editor
Kátya Nagy, music journalist, research assistant;

and the Slovak version:

Dáša Bulíková, musician, translator
Dominika Semaňáková, musicologist, editor, layout.

Special thanks to Tammy Nižňanska.

Project Website, Demo App

Listen Local is an open initiative for musicians and music organizations to create new, open AI applications that make locally relevant music visible online and offline. Listen Slovak is our first demonstration project, accompanied with a Feasibility Study. It was created jointly with the Slovak Performing Rights Society, and with the support of the Slovak Arts Council and Consolidated Independent, a music distributor company for independent labels and artist. Our aim was to understand why some Slovak music cannot be heard on radio and on streaming platforms, and to create a database and a demo application that makes that music discoverable.

Regional Climate Change Awareness Datasets: A Regional Geocoding Harmonization Case Study

Sat, 06 Mar 2021 00:00:00 +0000

library(regions)
library(lubridate)
library(dplyr)

if ( dir.exists('data-raw') ) {
  data_raw_dir <- "data-raw"
} else {
  data_raw_dir <- file.path("..", "..", "data-raw")
  }

Engage with us on LinkedIn @DigitalMusicObs or check out our open data and open repositories, code, tutorials

Going beyond the national level

Let’s start with a dirty averaging by sub-national unit. The w1 weighting variable contains the post-stratification weight for the national samples. The Eurobarometer samples represent nations (with the exception of East and West Germany, Northern Ireland and Great Britain.) The average of the w1 variable is 1.00 for each sample, but it is not necessarily 1 for smaller territorial units. If sum(w)>1 for say, AT23 it only means that the AT23 region was undersampled relatively to the rest of Austria, and responses must be over-weighted in post-stratification.

There is no way to make the samples become regionally representative, and a correct post-stratification would require further data about the sampel design. But we can simply adjust to over/undersampling by making sure that oversampled territorial averages are proportionally increased and undersampled ones are decreased. [Another ‘dirty’ averaging would be the use of an unweighted average, but our method is better, because it more-or-less adjusts gender and education level biases, but leaves intra-country regional biases in the sample.]

panel <- readRDS((file.path(data_raw_dir, "climate-panel.rds")))

climate_data <-  panel %>%
  mutate ( year: lubridate::year(date_of_interview)) %>%
  select ( all_of(c("isocntry", "geo", "w1")), 
           contains("problem")
  )  %>%
  mutate ( 
    # use the post-stratification weights for national samples
    serious_world_problems_first: w1*serious_world_problems_first , 
    serious_world_problems_climate_change: w1*serious_world_problems_climate_change) %>%
  group_by (  .data$geo ) %>%
  summarise( serious_world_problems_first: mean(serious_world_problems_first, na.rm=TRUE),
             serious_world_problems_climate_change: mean (serious_world_problems_climate_change, na.rm=TRUE),
             mean_w1: mean(w1)
             ) %>%
  mutate ( 
    # adjust for post-stratification weight bias due to regional over/undersampling
    climate_first: serious_world_problems_first / mean_w1, 
    climate_mentioned: serious_world_problems_climate_change / mean_w1
    )

So, we averaged, weighted and adjusted the mentioning of climate change as the world’s most serious, or one of the most serious problems by NUTS regions.

Aggregation level

The problem is that most statistical data is available in for the NUTS regional boundaries according to the NUTS2016 definition. However, GESIS uses NUTS2013 regions, so 252 regional codes in the four survey waves are invalid. Some data is available only on national level, but it can be projected to regional level, because small countries like Luxembourg have no regional divisions. Larger countries like Germany are divided only on state level (NUTS1), while small countries are divided on NUTS3 level.

This leads to various problems. Many data is available only on NUTS2 level, in which case NUTS1 data should be projected to its constituent smaller NUTS2 regions, and NUTS3 level data must be aggregated up to larger, containing NUTS2 levels.

Of course, we also must choose if we use `NUTS2013 or NUTS2016 boundaries. Sub-national boundaries have changed many thousand times in the EU27 countries alone since 1999.

## # A tibble: 5 x 2
##   validate         n
##   <chr>        <int>
## 1 country         15
## 2 invalid        252
## 3 nuts_level_1   132
## 4 nuts_level_2   452
## 5 nuts_level_3   141

Recoding the Regions

Our regions package was designed to keep track of sub-national regional boundary changes. It can validate regional data codes, and to some extent carry out recoding, imputation or simple aggregation.

Recoding means that the boundaries are unchanged, but the country changed the names/codes of regions, because there were other boundary changes which did not affect our observation unit.
Imputation must not be done with usual, general imputation tools, because our data is regionally structured. However, some imputations are very simple, because we can use equality equasions like MT = MT0, MT00.
Often the boundary change is additive, and merged territorial units can simple aggregated for comparison in earlier data.

regional_coding_2016 <- panel %>%
  mutate ( year: lubridate::year(date_of_interview)) %>%
  select (  all_of(c("isocntry", "geo", "region", "year") ) ) %>%
  distinct_all() %>%
  recode_nuts()

regional_coding_2013 <- panel %>%
  mutate ( year: lubridate::year(date_of_interview)) %>%
  select (  all_of(c("isocntry", "geo", "region", "year") ) ) %>%
  distinct_all() %>%
  recode_nuts( nuts_year: 2013)

climate_data_recoded <- climate_data %>% 
  left_join ( regional_coding_2016, by: 'geo' ) %>%
  left_join ( regional_coding_2013 %>% 
                select ( all_of(c("geo", "code_2013"))), 
              by: "geo") %>%
  distinct_all()

saveRDS ( climate_data_recoded , file.path(tempdir(), "climate_panel_recoded_agr.rds"), version: 2)

# not evaluated
saveRDS( climate_data_recoded , file: file.path("data-raw", "climate_panel_recoded_agr.rds"))

Where Are People More Likely To Treat Climate Change as the Most Serious Global Problem?

Sat, 06 Mar 2021 00:00:00 +0000

library(regions)
library(lubridate)
library(dplyr)

if ( dir.exists('data-raw') ) {
  data_raw_dir <- "data-raw"
} else {
  data_raw_dir <- file.path("..", "..", "data-raw")
  }

The first results of our longitudinal table were difficult to map, because the surveys used an obsolete regional coding. We will adjust the wrong coding, when possible, and join the data with the European Environment Agency’s (EEA) Air Quality e-Reporting (AQ e-Reporting) data on environmental pollution. We recoded the annual level for every available reporting stations [not shown here] and all values are in μg/m3. The period under observation is 2014-2016. Data file: https://www.eea.europa.eu/data-and-maps/data/aqereporting-8 (European Environment Agency 2021).

Recoding the Regions

Recoding means that the boundaries are unchanged, but the country changed the names and codes of regions because there were other boundary changes which did not affect our observation unit. We explain the problem and the solution in greater detail in our tutorial that aggregates the data on regional levels.

panel <- readRDS((file.path(data_raw_dir, "climate-panel.rds")))

climate_data_geocode <-  panel %>%
  mutate ( year: lubridate::year(date_of_interview)) %>%
  recode_nuts()

Let’s join the air pollution data and join it by corrected geocodes:

load(file.path("data", "air_pollutants.rda")) ## good practice to use system-independent file.path

climate_awareness_air <- climate_data_geocode %>%
  rename ( region_nuts_codes : .data$code_2016) %>%
  left_join ( air_pollutants, by: "region_nuts_codes" ) %>%
  select ( -all_of(c("w1", "wex", "date_of_interview", 
                     "typology", "typology_change", "geo", "region"))) %>%
  mutate (
    # remove special labels and create NA_numeric_ 
    age_education: retroharmonize::as_numeric(age_education)) %>%
  mutate_if ( is.character, as.factor) %>%
  mutate ( 
    # we only have responses from 4 years, and this should be treated as a categorical variable
    year: as.factor(year) 
    ) %>%
  filter ( complete.cases(.) )

The climate_awareness_air data frame contains the answers of 75086 individual respondents. 17.07% thought that climate change was the most serious world problem and 33.6% mentioned climate change as one of the three most important global problems.

summary ( climate_awareness_air  )

##                  rowid       serious_world_problems_first
##  ZA5877_v2-0-0_1    :    1   Min.   :0.0000              
##  ZA5877_v2-0-0_10   :    1   1st Qu.:0.0000              
##  ZA5877_v2-0-0_100  :    1   Median :0.0000              
##  ZA5877_v2-0-0_1000 :    1   Mean   :0.1707              
##  ZA5877_v2-0-0_10000:    1   3rd Qu.:0.0000              
##  ZA5877_v2-0-0_10001:    1   Max.   :1.0000              
##  (Other)            :75080                               
##  serious_world_problems_climate_change    isocntry    
##  Min.   :0.000                         BE     : 3028  
##  1st Qu.:0.000                         CZ     : 3023  
##  Median :0.000                         NL     : 3019  
##  Mean   :0.336                         SK     : 3000  
##  3rd Qu.:1.000                         SE     : 2980  
##  Max.   :1.000                         DE-W   : 2978  
##                                        (Other):57058  
##                                    marital_status         age_education  
##  (Re-)Married: without children           :13242   18            :15485  
##  (Re-)Married: children this marriage     :12696   19            : 7728  
##  Single: without children                 : 7650   16            : 5840  
##  (Re-)Married: w children of this marriage: 6520   still studying: 5098  
##  (Re-)Married: living without children    : 6225   17            : 5092  
##  Single: living without children          : 4102   15            : 4528  
##  (Other)                                  :24651   (Other)       :31315  
##    age_exact                      occupation_of_respondent
##  Min.   :15.0   Retired, unable to work       :22911      
##  1st Qu.:36.0   Skilled manual worker         : 6774      
##  Median :51.0   Employed position, at desk    : 6716      
##  Mean   :50.1   Employed position, service job: 5624      
##  3rd Qu.:65.0   Middle management, etc.       : 5252      
##  Max.   :99.0   Student                       : 5098      
##                 (Other)                       :22711      
##             occupation_of_respondent_recoded
##  Employed (10-18 in d15a)   :32763          
##  Not working (1-4 in d15a)  :37125          
##  Self-employed (5-9 in d15a): 5198          
##                                             
##                                             
##                                             
##                                             
##                        respondent_occupation_scale_c_14
##  Retired (4 in d15a)                   :22911          
##  Manual workers (15 to 18 in d15a)     :15269          
##  Other white collars (13 or 14 in d15a): 9203          
##  Managers (10 to 12 in d15a)           : 8291          
##  Self-employed (5 to 9 in d15a)        : 5198          
##  Students (2 in d15a)                  : 5098          
##  (Other)                               : 9116          
##                   type_of_community   is_student      no_education     
##  DK                        :   34   Min.   :0.0000   Min.   :0.000000  
##  Large town                :20939   1st Qu.:0.0000   1st Qu.:0.000000  
##  Rural area or village     :24686   Median :0.0000   Median :0.000000  
##  Small or middle sized town: 9850   Mean   :0.0679   Mean   :0.008151  
##  Small/middle town         :19577   3rd Qu.:0.0000   3rd Qu.:0.000000  
##                                     Max.   :1.0000   Max.   :1.000000  
##                                                                        
##    education       year       region_nuts_codes  country_code  
##  Min.   :14.00   2013:25103   LU     : 1432     DE     : 4531  
##  1st Qu.:17.00   2015:    0   MT     : 1398     GB     : 3538  
##  Median :18.00   2017:25053   CY     : 1192     BE     : 3028  
##  Mean   :19.61   2019:24930   SK02   : 1053     CZ     : 3023  
##  3rd Qu.:22.00                EL30   :  974     NL     : 3019  
##  Max.   :30.00                EE     :  973     SK     : 3000  
##                               (Other):68064     (Other):54947  
##      pm2_5             pm10               o3              BaP        
##  Min.   : 2.109   Min.   :  5.883   Min.   : 66.37   Min.   :0.0102  
##  1st Qu.: 9.374   1st Qu.: 28.326   1st Qu.: 90.89   1st Qu.:0.1779  
##  Median :11.866   Median : 33.673   Median :102.81   Median :0.4105  
##  Mean   :12.954   Mean   : 38.637   Mean   :101.49   Mean   :0.8759  
##  3rd Qu.:15.890   3rd Qu.: 49.488   3rd Qu.:110.73   3rd Qu.:1.0692  
##  Max.   :41.293   Max.   :123.239   Max.   :141.04   Max.   :7.8050  
##                                                                      
##       so2              ap_pc1            ap_pc2             ap_pc3       
##  Min.   : 0.0000   Min.   :-4.6669   Min.   :-2.21851   Min.   :-2.1007  
##  1st Qu.: 0.0000   1st Qu.:-0.4624   1st Qu.:-0.49130   1st Qu.:-0.5695  
##  Median : 0.0000   Median : 0.4263   Median : 0.02902   Median :-0.1113  
##  Mean   : 0.1032   Mean   : 0.1031   Mean   : 0.04166   Mean   :-0.1746  
##  3rd Qu.: 0.0000   3rd Qu.: 0.9748   3rd Qu.: 0.57416   3rd Qu.: 0.3309  
##  Max.   :42.5325   Max.   : 2.0344   Max.   : 3.25841   Max.   : 4.1615  
##                                                                          
##      ap_pc4            ap_pc5        
##  Min.   :-1.7387   Min.   :-2.75079  
##  1st Qu.:-0.1669   1st Qu.:-0.18748  
##  Median : 0.0371   Median : 0.01811  
##  Mean   : 0.1154   Mean   : 0.06797  
##  3rd Qu.: 0.3050   3rd Qu.: 0.34937  
##  Max.   : 3.2476   Max.   : 1.42816  
##

Let’s see a simple CART tree! We remove the regional codes, because there are very serious differences among regional climate awareness. These differences, together with education level, and the year we are talking about, are the most important predictors of thinking about climate change as the most important global problem in Europe.

# Classification Tree with rpart
library(rpart)

# grow tree
fit <- rpart(as.factor(serious_world_problems_first) ~ .,
   method="class", data=climate_awareness_air %>%
     select ( - all_of(c("rowid", "region_nuts_codes"))), 
   control: rpart.control(cp: 0.005))

printcp(fit) # display the results

## 
## Classification tree:
## rpart(formula: as.factor(serious_world_problems_first) ~ ., 
##     data: climate_awareness_air %>% select(-all_of(c("rowid", 
##         "region_nuts_codes"))), method: "class", control: rpart.control(cp: 0.005))
## 
## Variables actually used in tree construction:
## [1] age_education                         isocntry                             
## [3] serious_world_problems_climate_change year                                 
## 
## Root node error: 12817/75086: 0.1707
## 
## n= 75086 
## 
##          CP nsplit rel error  xerror      xstd
## 1 0.0240566      0   1.00000 1.00000 0.0080438
## 2 0.0082703      3   0.92783 0.92783 0.0078055
## 3 0.0050000      5   0.91129 0.91425 0.0077588

plotcp(fit) # visualize cross-validation results

summary(fit) # detailed summary of splits

## Call:
## rpart(formula: as.factor(serious_world_problems_first) ~ ., 
##     data: climate_awareness_air %>% select(-all_of(c("rowid", 
##         "region_nuts_codes"))), method: "class", control: rpart.control(cp: 0.005))
##   n= 75086 
## 
##            CP nsplit rel error    xerror        xstd
## 1 0.024056592      0 1.0000000 1.0000000 0.008043837
## 2 0.008270266      3 0.9278302 0.9278302 0.007805478
## 3 0.005000000      5 0.9112897 0.9142545 0.007758824
## 
## Variable importance
## serious_world_problems_climate_change                              isocntry 
##                                    31                                    26 
##                          country_code                                   BaP 
##                                    20                                     8 
##                                 pm2_5                                ap_pc1 
##                                     4                                     3 
##                         age_education                                  pm10 
##                                     2                                     2 
##                             education                                ap_pc2 
##                                     2                                     1 
##                                  year 
##                                     1 
## 
## Node number 1: 75086 observations,    complexity param=0.02405659
##   predicted class=0  expected loss=0.1706976  P(node) =1
##     class counts: 62269 12817
##    probabilities: 0.829 0.171 
##   left son=2 (25229 obs) right son=3 (49857 obs)
##   Primary splits:
##       serious_world_problems_climate_change < 0.5          to the right, improve=2214.2040, (0 missing)
##       isocntry                              splits as  RRLLLRRRLLRLRLLLLLLLLLLRRLLLRLL, improve= 728.0160, (0 missing)
##       country_code                          splits as  RRLLLRRLLRLLLLLLLLLLRRLLLRLL, improve= 673.3656, (0 missing)
##       BaP                                   < 0.4300347    to the right, improve= 310.6229, (0 missing)
##       pm2_5                                 < 13.38264     to the right, improve= 296.4013, (0 missing)
##   Surrogate splits:
##       age_education splits as  ----RRRRRR-RRRRRRRRRR-RRRRRRRRRR-RRRRRRRRRR-RRRRRRRRRR-RRRRRL-RRR-RRRRRRRRR--RRRLLR--R-R, agree=0.664, adj=0, (0 split)
##       pm10          < 7.491315     to the left,  agree=0.664, adj=0, (0 split)
## 
## Node number 2: 25229 observations
##   predicted class=0  expected loss=0  P(node) =0.3360014
##     class counts: 25229     0
##    probabilities: 1.000 0.000 
## 
## Node number 3: 49857 observations,    complexity param=0.02405659
##   predicted class=0  expected loss=0.2570752  P(node) =0.6639986
##     class counts: 37040 12817
##    probabilities: 0.743 0.257 
##   left son=6 (34631 obs) right son=7 (15226 obs)
##   Primary splits:
##       isocntry     splits as  RRLLLRRRLLRLRLLLLLLLLLLRRLLLRLL, improve=1454.9460, (0 missing)
##       country_code splits as  RRLLLRRLLRLLLLLLLLLLRRLLLRLL, improve=1359.7210, (0 missing)
##       BaP          < 0.4300347    to the right, improve= 629.8844, (0 missing)
##       pm2_5        < 13.38264     to the right, improve= 555.7484, (0 missing)
##       ap_pc1       < -0.005459537 to the left,  improve= 533.3579, (0 missing)
##   Surrogate splits:
##       country_code splits as  RRLLLRRLLRLLLLLLLLLLRRLLLRLL, agree=0.987, adj=0.957, (0 split)
##       BaP          < 0.1749425    to the right, agree=0.775, adj=0.264, (0 split)
##       pm2_5        < 5.206993     to the right, agree=0.737, adj=0.140, (0 split)
##       ap_pc1       < 1.405527     to the left,  agree=0.733, adj=0.126, (0 split)
##       pm10         < 25.31211     to the right, agree=0.718, adj=0.076, (0 split)
## 
## Node number 6: 34631 observations
##   predicted class=0  expected loss=0.1769802  P(node) =0.4612178
##     class counts: 28502  6129
##    probabilities: 0.823 0.177 
## 
## Node number 7: 15226 observations,    complexity param=0.02405659
##   predicted class=0  expected loss=0.4392487  P(node) =0.2027808
##     class counts:  8538  6688
##    probabilities: 0.561 0.439 
##   left son=14 (11607 obs) right son=15 (3619 obs)
##   Primary splits:
##       isocntry      splits as  LL---LLR--L-L----------LL---R--, improve=337.5462, (0 missing)
##       country_code  splits as  LL---LR--L-L--------LL---R--, improve=337.5462, (0 missing)
##       age_education splits as  ----LLLLLL-LLLRRRRRRR-RRRRRRRRRL-RRRRRRLLRR-RRRRLLRLRL-RRLRRR-RRR-LLLLRRR-----LR-----L-R, improve=294.0807, (0 missing)
##       education     < 22.5         to the left,  improve=262.3747, (0 missing)
##       BaP           < 0.053328     to the right, improve=232.7043, (0 missing)
##   Surrogate splits:
##       BaP           < 0.053328     to the right, agree=0.878, adj=0.485, (0 split)
##       pm2_5         < 4.810361     to the right, agree=0.827, adj=0.271, (0 split)
##       ap_pc2        < 0.8746175    to the left,  agree=0.792, adj=0.124, (0 split)
##       so2           < 0.3302972    to the left,  agree=0.781, adj=0.078, (0 split)
##       age_education splits as  ----LLLLLL-LLLLLLLRLR-LRRLRRRRRR-RRRRLLLLLR-LRLRLLRRLL-LLRLLR-LLR-RRLLLLL-----RR-----R-L, agree=0.779, adj=0.071, (0 split)
## 
## Node number 14: 11607 observations,    complexity param=0.008270266
##   predicted class=0  expected loss=0.3804601  P(node) =0.1545827
##     class counts:  7191  4416
##    probabilities: 0.620 0.380 
##   left son=28 (7462 obs) right son=29 (4145 obs)
##   Primary splits:
##       age_education                    splits as  ----LLLLLL-LRRRRRRRRR-RRLRRLRRLL-RRRRLRLLRR-RLRLLLRLRL-RR-RR--RRL-L-LLRRR------------L-R, improve=123.71070, (0 missing)
##       year                             splits as  R-LR, improve=107.79460, (0 missing)
##       education                        < 20.5         to the left,  improve= 90.28724, (0 missing)
##       occupation_of_respondent         splits as  LRRLRRRRRLRLLLRLLL, improve= 84.62865, (0 missing)
##       respondent_occupation_scale_c_14 splits as  LRLLLRRL, improve= 68.88653, (0 missing)
##   Surrogate splits:
##       education                        < 20.5         to the left,  agree=0.950, adj=0.861, (0 split)
##       occupation_of_respondent         splits as  LLLLRLLRRLRLLLRLLL, agree=0.738, adj=0.267, (0 split)
##       respondent_occupation_scale_c_14 splits as  LRLLLLRL, agree=0.733, adj=0.251, (0 split)
##       is_student                       < 0.5          to the left,  agree=0.709, adj=0.186, (0 split)
##       age_exact                        < 23.5         to the right, agree=0.676, adj=0.094, (0 split)
## 
## Node number 15: 3619 observations
##   predicted class=1  expected loss=0.3722023  P(node) =0.04819807
##     class counts:  1347  2272
##    probabilities: 0.372 0.628 
## 
## Node number 28: 7462 observations
##   predicted class=0  expected loss=0.326052  P(node) =0.09937938
##     class counts:  5029  2433
##    probabilities: 0.674 0.326 
## 
## Node number 29: 4145 observations,    complexity param=0.008270266
##   predicted class=0  expected loss=0.4784077  P(node) =0.05520337
##     class counts:  2162  1983
##    probabilities: 0.522 0.478 
##   left son=58 (2573 obs) right son=59 (1572 obs)
##   Primary splits:
##       year                     splits as  L-LR, improve=40.13885, (0 missing)
##       occupation_of_respondent splits as  LRLLRRRRRLRLLLRLLL, improve=18.33254, (0 missing)
##       marital_status           splits as  LRRRLRRRLRRLRLLRRRRRRLRLRLLRR, improve=17.86888, (0 missing)
##       type_of_community        splits as  LRLRL, improve=17.55254, (0 missing)
##       age_education            splits as  ------------LLRRRRRRR-RR-RL-RR---LRRR-R--LR-R-R---R-R--RR-RR--RR------RRR--------------R, improve=14.66121, (0 missing)
##   Surrogate splits:
##       type_of_community splits as  LLLRL, agree=0.777, adj=0.412, (0 split)
##       marital_status    splits as  RRLLLLLRLLLLLLLRRRLLLLLLRLRLL, agree=0.680, adj=0.155, (0 split)
##       isocntry          splits as  LL---LL---L-R----------LL------, agree=0.669, adj=0.127, (0 split)
##       country_code      splits as  LL---L---L-R--------LL------, agree=0.669, adj=0.127, (0 split)
##       o3                < 83.06345     to the right, agree=0.650, adj=0.076, (0 split)
## 
## Node number 58: 2573 observations
##   predicted class=0  expected loss=0.4240187  P(node) =0.03426737
##     class counts:  1482  1091
##    probabilities: 0.576 0.424 
## 
## Node number 59: 1572 observations
##   predicted class=1  expected loss=0.43257  P(node) =0.02093599
##     class counts:   680   892
##    probabilities: 0.433 0.567

# plot tree
plot(fit, uniform=TRUE,
   main="Classification Tree: Climate Change Is The Most Serious Threat")
text(fit, use.n=TRUE, all=TRUE, cex=.8)

## Warning in labels.rpart(x, minlength: minlength): more than 52 levels in a
## predicting factor, truncated for printout

saveRDS ( climate_awareness_air , file.path(tempdir(), "climate_panel_recoded.rds"), version: 2)

# not evaluated
saveRDS( climate_awareness_air, file: file.path("data-raw", "climate-panel_recoded.rds"))

Climate Awareness Change in Europe 2013-2019

Fri, 05 Mar 2021 00:00:00 +0000

Retrospective survey harmonization comes with many challenges, as we have shown in the introduction to this tutorial case study. In this example, we will work with Eurobarometer’s data.

Engage with us on LinkedIn @CultDataObs or check out our open data and open repositories, code, tutorials

Please use the development version of retroharmonize:

devtools::install_github("antaldaniel/retroharmonize")

library(retroharmonize)
library(dplyr)       # this is necessary for the example 
library(lubridate)   # easier date conversion

## Warning: package 'lubridate' was built under R version 4.0.4

library(stringr)     # You can also use base R string processing functions

Get the Data

retroharmonize is not associated with Eurobarometer, or its creators, Kantar, or its archivists, GESIS. We assume that you have acquired the necessary files from GESIS after carefully reading their terms and you placed it on a path that you call gesis_dir. The precise documentation of the data we use can be found in this supporting blogpost. To reproduce this blogpost, you will need ZA5877_v2-0-0.sav, ZA6595_v3-0-0.sav, ZA6861_v1-2-0.sav, ZA7488_v1-0-0.sav, ZA7572_v1-0-0.sav in a directory that you will name gesis_dir.

#Not run in the blogpost. In the repo we have a saved version.
climate_change_files <- c("ZA5877_v2-0-0.sav", "ZA6595_v3-0-0.sav",  "ZA6861_v1-2-0.sav", 
                          "ZA7488_v1-0-0.sav", "ZA7572_v1-0-0.sav")

eb_waves <- read_surveys(file.path(gesis_dir, climate_change_files), .f='read_spss')

if (dir.exists("data-raw")) {
  save ( eb_waves,  file: file.path("data-raw", "eb_climate_change_waves.rda") )
}

if ( file.exists( file.path("data-raw", "eb_climate_change_waves.rda") )) {
  load (file.path( "data-raw", "eb_climate_change_waves.rda" ) )
} else {
  load (file.path("..", "..",  "data-raw", "eb_climate_change_waves.rda") )
}

The eb_waves nested list contains five surveys imported from SPSS to the survey class of retroharmonize. The survey class is a data.frame that retains important metadata for further harmonization.

document_waves (eb_waves)

## # A tibble: 5 x 5
##   id            filename           ncol  nrow object_size
##   <chr>         <chr>             <int> <int>       <dbl>
## 1 ZA5877_v2-0-0 ZA5877_v2-0-0.sav   604 27919   139352456
## 2 ZA6595_v3-0-0 ZA6595_v3-0-0.sav   519 27718   119370440
## 3 ZA6861_v1-2-0 ZA6861_v1-2-0.sav   657 27901   151397528
## 4 ZA7488_v1-0-0 ZA7488_v1-0-0.sav   752 27339   169465928
## 5 ZA7572_v1-0-0 ZA7572_v1-0-0.sav   348 27655    80562432

Beware the object sizes. If you work with many surveys, memory-efficient programming becomes imperative. We will be subsetting whenever possible.

Metadata analysis

As noted before, prepare to work with nested lists. Each imported survey is nested as a data frame in the eb_waves list.

Metadata: Protocol Variables

Eurobarometer calls certain metadata elements, like interviewee cooperation level or the date of a survey interview as protocol variable. Let’s start here. This will be our template to harmonize more and more aspects of the five surveys (which are, in fact, already harmonization of about 30 surveys conducted in a single ‘wave’ in multiple countries.)

# select variables of interest from the metadata
eb_protocol_metadata <- eb_climate_metadata %>%
  filter ( .data$label_orig %in% c("date of interview") |
             .data$var_name_orig == "rowid")  %>%
  suggest_var_names( survey_program: "eurobarometer" )

# subset and harmonize these variables in all nested list items of 'waves' of surveys
interview_dates <- harmonize_var_names(eb_waves, 
                                       eb_protocol_metadata )

# apply similar data processing rules to same variables
interview_dates <- lapply (interview_dates, 
                      function (x) x %>% mutate ( date_of_interview: as_character(.data$date_of_interview) )
                      )

# join the individual survey tables into a single table 
interview_dates <- as_tibble ( Reduce (rbind, interview_dates) )

# Check the variable classes.

vapply(interview_dates, function(x) class(x)[1], character(1))

##             rowid date_of_interview 
##       "character"       "character"

This is our sample workflow for each block of variables.

Get a unique identifier.
Add other variables
Harmonize the variable names
Subset the data leaving out anything that you do not harmonize in this block.
Apply some normalization in a nested list.
When the variables are harmonized to same name, class, merge them into a data.frame-like tibble object.

Now finish the harmonization. Wednesday, 31st October 2018 should become a Date type 2018-10-31.

require(lubridate)
harmonize_date <- function(x) {
  x <- tolower(as.character(x))
  x <- gsub("monday|tuesday|wednesday|thursday|friday|saturday|sunday|\\,|th|nd|rd|st", "", x)
  x <- gsub("decemberber", "december", x) # all those annoying real-life data problems!
  x <- stringr::str_trim (x, "both")
  x <- gsub("^0", "", x )
  x <- gsub("\\s\\s", "\\s", x)
  lubridate::dmy(x) 
}

interview_dates <- interview_dates %>%
  mutate ( date_of_interview: harmonize_date(.data$date_of_interview) )

vapply(interview_dates, function(x) class(x)[1], character(1))

##             rowid date_of_interview 
##       "character"            "Date"

To avoid duplication of row IDs in surveys that may not be unique in different surveys, we created a simple, sequential ID for each survey, including the ID of the original file.

set.seed(2021)
sample_n(interview_dates, 6)

## # A tibble: 6 x 2
##   rowid               date_of_interview
##   <chr>               <date>           
## 1 ZA7488_v1-0-0_7016  2018-10-28       
## 2 ZA7488_v1-0-0_19187 2018-11-02       
## 3 ZA6861_v1-2-0_1218  2017-03-18       
## 4 ZA6861_v1-2-0_4142  2017-03-21       
## 5 ZA7572_v1-0-0_12363 2019-04-17       
## 6 ZA7572_v1-0-0_8071  2019-04-18

After this type-conversion problem let’s see an issue when an original SPSS variable can have two meaningful R representations.

Metadata: Geographical information

Let’s continue with harmonizing geographical information in the files. In this example, var_name_suggested will contain the harmonized variable name. It is likely that you have to make this call, after carefully reading the original questionnaires and codebooks.

eb_regional_metadata <- eb_climate_metadata %>%
  filter ( grepl( "rowid|isocntry|^nuts$", .data$var_name_orig)) %>%
  suggest_var_names( survey_program: "eurobarometer" ) %>%
  mutate ( var_name_suggested: case_when ( 
    var_name_suggested == "region_nuts_codes"     ~ "geo",
    TRUE ~ var_name_suggested ))

The harmonize_var_names() takes all variables in the subsetted, geographical metadata table, and brings them to the harmonized var_name_suggested name. The function subsets the surveys to avoid the presence of non-harmonized variables. All regional NUTS codes become geo in our case:

geography <- harmonize_var_names(eb_waves, 
                                 eb_regional_metadata)

If you are used to work with single survey files, you are likely to work in a tabular format, which easily converts into a data.frame like object, in our example, to tidyverse’s tibble. However, when working with longitudinal data, it is far simpler to work with nested lists, because the tables usually have different dimensions (neither the rows corresponding to observations or the columns are the same across all survey files.)

In the nested list, each list element is a single, tabular-format survey. (In fact, the survey are in retroharmonize’s survey class, which is a rich tibble that contains the metadata and the processing history of the survey.)

The regional information in the Eurobarometer files is contained in the nuts variable. We want to keep both the original labels and values. The original values are the region’s codes, and the labels are the names. The easiest and fastest solution is the base R lapply loop.

geography <- lapply ( geography, 
                      function (x) x %>% mutate ( region: as_character(geo), 
                                                  geo   : as.character(geo) )  
)

Because each table has exactly the same columns, we can simply use rbind() and reduce the list to a modern data.frame, i.e. a tibble.

geography <- as_tibble ( Reduce (rbind, geography) )

Let’s see a dozen cases:

set.seed(2021)
sample_n(geography, 12)

## # A tibble: 12 x 4
##    rowid               isocntry geo   region              
##    <chr>               <chr>    <chr> <chr>               
##  1 ZA7488_v1-0-0_7016  SI       SI012 Podravska           
##  2 ZA7488_v1-0-0_19187 PL       PL63  Pomorskie           
##  3 ZA6861_v1-2-0_1218  DK       DK02  Sjaelland           
##  4 ZA6861_v1-2-0_4142  FI       FI1B  Helsinki-Uusimaa    
##  5 ZA7572_v1-0-0_12363 SE       SE12  Oestra Mellansverige
##  6 ZA7572_v1-0-0_8071  IT       ITH   Nord-Est [IT]       
##  7 ZA6861_v1-2-0_6145  IE       IE021 Dublin              
##  8 ZA6861_v1-2-0_24638 RO       RO31  South [RO]          
##  9 ZA7488_v1-0-0_11315 CY       CY    REPUBLIC OF CYPRUS  
## 10 ZA6595_v3-0-0_27568 HR       HR041 Grad Zagreb         
## 11 ZA7572_v1-0-0_17397 CZ       CZ06  Jihovychod          
## 12 ZA6861_v1-2-0_10993 PT       PT17  Lisboa

The idea is that we do similar variable harmonization block by block, and eventually we will join them together. Next step: socio-demography and weights.

Socio-demography and Weights

There are a few peculiar issues to look out for. This example shows that survey harmonization requires plenty of expert judgment, and you cannot fully automate the process.

The Eurobarometer archives do not use all weight and demographic variable names consistently. For example, the wex variable, which is a projected weight for the country’s 15 years old or older population is sometimes called wex, sometimes wextra. The individual survey’s post-stratification weight is the w1 variable, but this is not necessarily what you need to use.

The suggest_var_names() function has a parameter for survey_program: "eurobaromater" which normalizes a bit the most used variables. For example, all variations of wex, wextra wil be noramlized to wex. You can ignore this parameter and use your own names, too.

eb_demography_metadata  <- eb_climate_metadata %>%
  filter ( grepl( "rowid|isocntry|^d8$|^d7$|^wex|^w1$|d25|^d15a|^d11$", .data$var_name_orig) ) %>%
  suggest_var_names( survey_program: "eurobarometer")

As you can see, using the original labels would not help, because they also contain various alterations.

eb_demography_metadata %>%
  select ( filename, var_name_orig, label_orig, var_name_suggested ) %>%
  filter (var_name_orig %in% c("wex", "wextra") )

##            filename var_name_orig                                  label_orig
## 1 ZA5877_v2-0-0.sav        wextra      weight extrapolated population 15 plus
## 2 ZA6595_v3-0-0.sav        wextra      weight extrapolated population 15 plus
## 3 ZA6861_v1-2-0.sav           wex weight extrapolated population aged 15 plus
## 4 ZA7488_v1-0-0.sav           wex weight extrapolated population aged 15 plus
## 5 ZA7572_v1-0-0.sav           wex weight extrapolated population aged 15 plus
##   var_name_suggested
## 1                wex
## 2                wex
## 3                wex
## 4                wex
## 5                wex

demography <- harmonize_var_names ( waves: eb_waves, 
                                    metadata: eb_demography_metadata )

Socio-demographic variables like level of highest education or occupation are rather country-specific. Eurobarometer uses standardized occupation and marital status scales, and a proxy for education levels, age of leaving full-time education.

This is a particularly tricky variable, because it’s coding in fact contains three different variables - school leaving age, except for students, and except for people who did not finish their compulsory primary school. And while school leaving age was a good proxy since the 1970s, in the age when the EU is promoting life-long-learning becomes less and less useful, as people stop and re-start their education throughout their lives.

example <- demography[[1]] %>%
  mutate ( across ( -any_of(c("rowid", "w1", "wex")), as_character) ) %>%
  mutate ( across (any_of(c("w1", "wex")), as_numeric) )
unique ( example$age_education )

##  [1] "22"                     "25"                     "17"                    
##  [4] "19"                     "12"                     "23"                    
##  [7] "18"                     "20"                     "21"                    
## [10] "14"                     "24"                     "16"                    
## [13] "26"                     "15"                     "Still studying"        
## [16] "DK"                     "31"                     "29"                    
## [19] "27"                     "13"                     "32"                    
## [22] "28"                     "30"                     "53"                    
## [25] "42"                     "62"                     "40"                    
## [28] "No full-time education" "Refusal"                "37"                    
## [31] "39"                     "34"                     "35"                    
## [34] "47"                     "36"                     "45"                    
## [37] "51"                     "33"                     "43"                    
## [40] "38"                     "49"                     "46"                    
## [43] "41"                     "57"                     "7"                     
## [46] "48"                     "44"                     "50"                    
## [49] "56"                     "8"                      "11"                    
## [52] "10"                     "9"                      "75 years"              
## [55] "6"                      "3"                      "54"                    
## [58] "55"                     "60"                     "64"                    
## [61] "2 years"                "58"                     "52"                    
## [64] "72"                     "61"                     "4"                     
## [67] "63"

The seamingly trival age_exact variable has its own issues, too:

unique ( example$age_exact)

##  [1] "54"       "66"       "56"       "53"       "33"       "72"      
##  [7] "83"       "62"       "86"       "77"       "64"       "46"      
## [13] "44"       "59"       "60"       "67"       "63"       "20"      
## [19] "43"       "37"       "78"       "49"       "90"       "45"      
## [25] "28"       "29"       "30"       "39"       "51"       "38"      
## [31] "41"       "71"       "25"       "48"       "79"       "88"      
## [37] "61"       "85"       "70"       "35"       "81"       "52"      
## [43] "57"       "27"       "47"       "15 years" "21"       "42"      
## [49] "32"       "68"       "36"       "34"       "19"       "31"      
## [55] "26"       "23"       "24"       "22"       "16"       "84"      
## [61] "65"       "18"       "55"       "40"       "50"       "73"      
## [67] "69"       "87"       "89"       "74"       "75"       "98 years"
## [73] "76"       "80"       "58"       "82"       "17"       "93"      
## [79] "91"       "92"       "95"       "94"       "97"

Let’s see all the strange labels attached to age-type variables:

collect_val_labels(metadata: eb_demography_metadata %>%
                     filter ( var_name_suggested %in% c("age_exact", "age_education")) )

##  [1] "2 years"                  "75 years"                
##  [3] "No full-time education"   "Still studying"          
##  [5] "15 years"                 "98 years"                
##  [7] "96 years"                 "[NOT CLEARLY DOCUMENTED]"
##  [9] "74 years"                 "99 and older"            
## [11] "Refusal"                  "87 years"                
## [13] "DK"                       "88 years"

We must handle many exception, so we created a function for this purpose:

remove_years  <- function(x) { 
  x <- gsub("years|and\\solder", "", tolower(x))
  stringr::str_trim (x, "both")}

process_demography <- function (x) { 
  
  x %>% mutate ( across ( -any_of(c("rowid", "w1", "wex")), as_character) ) %>%
    mutate ( across (any_of(c("w1", "wex")), as_numeric) ) %>%
    mutate ( across (contains("age"), remove_years)) %>%
    mutate ( age_exact: as.numeric (age_exact)) %>%
    mutate ( is_student: ifelse ( tolower(age_education) == "still studying", 
                                   1, 0), 
             no_education: ifelse ( tolower(age_education) == "no full-time education", 1, 0)) %>%
    mutate ( education: case_when (
      grepl("studying", age_education) ~ age_exact, 
      grepl ("education", age_education)  ~ 14, 
      grepl ("refus|document|dk", tolower(age_education)) ~ NA_real_,
      TRUE ~ as.numeric(age_education)
    ))  %>%
    mutate ( education: case_when ( 
      education < 14 ~ NA_real_, 
      education > 30 ~ 30, 
      TRUE ~ education )) 
}

demography <- lapply ( demography, process_demography )

## Warning in eval_tidy(pair$rhs, env: default_env): NAs introduced by coercion

## Warning in mask$eval_all_mutate(quo): NAs introduced by coercion

## Warning in eval_tidy(pair$rhs, env: default_env): NAs introduced by coercion

## Warning in eval_tidy(pair$rhs, env: default_env): NAs introduced by coercion

## Warning in eval_tidy(pair$rhs, env: default_env): NAs introduced by coercion

## Warning in eval_tidy(pair$rhs, env: default_env): NAs introduced by coercion

## WE'll full join and not use rbind, because we have different variables in different waves.
demography <- Reduce ( full_join, demography )

## Joining, by: c("rowid", "isocntry", "w1", "wex", "marital_status", "age_education", "age_exact", "occupation_of_respondent", "occupation_of_respondent_recoded", "respondent_occupation_scale_c_14", "type_of_community", "is_student", "no_education", "education")
## Joining, by: c("rowid", "isocntry", "w1", "wex", "marital_status", "age_education", "age_exact", "occupation_of_respondent", "occupation_of_respondent_recoded", "respondent_occupation_scale_c_14", "type_of_community", "is_student", "no_education", "education")
## Joining, by: c("rowid", "isocntry", "w1", "wex", "marital_status", "age_education", "age_exact", "occupation_of_respondent", "occupation_of_respondent_recoded", "respondent_occupation_scale_c_14", "type_of_community", "is_student", "no_education", "education")
## Joining, by: c("rowid", "isocntry", "w1", "wex", "marital_status", "age_education", "age_exact", "occupation_of_respondent", "occupation_of_respondent_recoded", "respondent_occupation_scale_c_14", "type_of_community", "is_student", "no_education", "education")

Now let’s see what we have here:

set.seed(2021)
sample_n(demography, 12)

## # A tibble: 12 x 14
##    rowid    isocntry    w1    wex marital_status        age_education  age_exact
##    <chr>    <chr>    <dbl>  <dbl> <chr>                 <chr>              <dbl>
##  1 ZA7488_~ SI       0.828  1428. (Re-)Married: withou~ 19                    43
##  2 ZA7488_~ PL       1.01  32830. (Re-)Married: withou~ 19                    64
##  3 ZA6861_~ DK       0.641  3100. (Re-)Married: withou~ 22                    78
##  4 ZA6861_~ FI       1.83   8601. (Re-)Married: childr~ 30                    38
##  5 ZA7572_~ SE       0.342  2645. (Re-)Married: withou~ 17                    68
##  6 ZA7572_~ IT       0.630 32287. (Re-)Married: childr~ 20                    40
##  7 ZA6861_~ IE       0.868  3054. (Re-)Married: childr~ 32                    42
##  8 ZA6861_~ RO       0.724 11805. (Re-)Married: withou~ 14                    59
##  9 ZA7488_~ CY       0.691  1013. (Re-)Married: childr~ 18                    67
## 10 ZA6595_~ HR       0.580  2098. Single living w part~ 27                    30
## 11 ZA7572_~ CZ       1.86  16908. Single: without chil~ still studying        20
## 12 ZA6861_~ PT       0.932  7448. Widow: with children  no full-time ~        84
## # ... with 7 more variables: occupation_of_respondent <chr>,
## #   occupation_of_respondent_recoded <chr>,
## #   respondent_occupation_scale_c_14 <chr>, type_of_community <chr>,
## #   is_student <dbl>, no_education <dbl>, education <dbl>

Harmonizing Variable Labels

So far we have been working with metadata, weights and socio-demography. In other words, we have not even started the desired harmonization of climate change awareness. The methodology is the same, but here we really must look out for the answer options in the questionnaire. (Refer to our data summary again here.)

climate_awareness_metadata <- eb_climate_metadata %>%
  suggest_var_names( survey_program: "eurobarometer" ) %>%
  filter ( .data$var_name_suggested  %in% c("rowid",
                                            "serious_world_problems_first", 
                                             "serious_world_problems_climate_change")
  ) 

hw <- harmonize_var_names ( waves: eb_waves, 
                            metadata: climate_awareness_metadata )

The retroharmoinze package comes with a generic harmonize_values() function that will change the value labels of categorical variables (including binary ones) to a unitary format. It will also take care of various types of missing values.

First, let’s go back to our metadata and collect all value labels that will show up with collect_val_labels():

collect_val_labels(climate_awareness_metadata)

##  [1] "Climate change"                            
##  [2] "International terrorism"                   
##  [3] "Poverty, hunger and lack of drinking water"
##  [4] "Spread of infectious diseases"             
##  [5] "The economic situation"                    
##  [6] "Proliferation of nuclear weapons"          
##  [7] "Armed conflicts"                           
##  [8] "The increasing global population"          
##  [9] "Other (SPONTANEOUS)"                       
## [10] "None (SPONTANEOUS)"                        
## [11] "Not mentioned"                             
## [12] "Mentioned"                                 
## [13] "DK"

In this case, we want to select Climate change as the mentioned most serious problem, and Climate change taken from a list of three serious problems. The first question type is a single-choice one, where Climate change is either mentioned, or the alternative answer is labeled as Not mentioned. In the multiple choice case, the alternative may be something else, for example, Spread of infectious diseases, as we all well know by 2021.

We want to see who thought Climate change was the most serious problem, or one of the most serious problems, so we label each mentions of Climate change as mentioned and we pair it with a numeric value of 1. All other cases are labeled as not_mentioned, with the exceptions of various missing observations, which in these cases are Do not know answers, Declined to answer cases, and Inappropriate cases [The latter one is Eurobarometer’s label for questions that were for one reason or other not asked from a particular interviewee – for example, because the Turkish Cypriot community received a different questionnaire.]

# positive cases
label_1: c("^Climate\\schange", "^Mentioned")
# missing cases 
na_labels <- collect_na_labels( climate_awareness_metadata)
na_labels

## [1] "DK"                             "Inap. (10 or 11 in qa1a)"      
## [3] "Inap. (coded 10 or 11 in qc1a)" "Inap. (coded 10 or 11 in qb1a)"

# negative cases
label_0 <- collect_val_labels( climate_awareness_metadata)
label_0 <- label_0[! label_0 %in% label_1 ]

The harmonize_serious_problems() function harmonizes the labels within the special labeled class of retroharmonize. This class retains all information to give categorical variables a character or numeric representation, and various processing metadata for documentation purposes. While this class is very reach (it contains whatever was imported from SPSS’s proprietary data format and the history), it is not suitable for statistical analysis. We could, of course, directly call the harmonize_values() from the retroharmonize package, but the parameterization would be very complicated even in a simple function call, not to mention a looped call. Because this function is the heart of the retroharmonize package, it has a tutorial article on its own.

harmonize_serious_problems <- function(x) {
  label_list <- list(
    from: c(label_0, label_1, na_labels), 
    to: c( rep ( "not_mentioned", length(label_0) ),   # use the same order as in from!
            rep ( "mentioned", length(label_1) ),
            "do_not_know", "inap", "inap", "inap"), 
    numeric_values: c(rep ( 0, length(label_0) ), # use the same order as in from!
                       rep ( 1, length(label_1) ),
                       99997,99999,99999,99999)
  )
  
  harmonize_values(x, 
                   harmonize_labels: label_list, 
                   na_values: c("do_not_know"=99997,
                                 "declined"=99998,
                                 "inap"=99999), 
                   remove: "\\(|\\)|\\[|\\]|\\%"
  )
}

Our objects are rather big in memory, so first, let’s remove the surveys that do not contain these world problem variables. In this cases, the subsetted and harmonized surveys in the nested list have only one columns, i.e. the rowid.

hw <- hw[unlist ( lapply ( hw, ncol)) > 1 ]

Now we have a smaller problem to deal with. With many surveys, it is easy to fill up your computer’s memory, so let’s start building up our joined panel data from a smaller set of nested, subsetted surveys.

hw <- lapply ( hw, function (x) x %>% mutate ( across ( contains("problem"), harmonize_serious_problems) ) )

Our lapply loop calls an anonymous function which in turn calls the harmonize_serious_problems parameterized version of the harmonize_values() on all variables that have problem in their names.

once we are done, our variables have harmonized names, and harmonized values, and harmonized label, but they are stored in the complex retroharmonize_labelled_spss_survey class, inherited from the haven_labelled_spss in haven.

We reduced our single and multiple choice questions to binary choice variables. We can now give them a numeric representation. Be mindful that retroharmonize has special methods for its special labeled class that retains metadata from SPSS. This means that as_character and as_numeric knows how to handle various types of missing values, whereas the base R as.character and as.numeric may coerce special values to unwanted results. This is particularly dangerous with numeric variables – and this is the reason why we introduced a new set of S3 objects and methods in the package.

We will ignore the differences between various forms of missingness, i.e. the person said that she did not know, or did not want to answer, or for some reason was not asked in the survey. In a more descriptive, non-harmonized analysis you would probably want to explore them as various ‘categories’ and use a character representation.

hw <- lapply ( hw, function(x) x %>% mutate ( across ( contains("problem"), as_numeric) ))

hw <- Reduce ( full_join, hw) # we must use joins instead of binds because the number of columns vary.

Let’s see what we have:

set.seed(2021)
sample_n (hw, 12)

## # A tibble: 12 x 3
##    rowid             serious_world_problems_fi~ serious_world_problems_climate_~
##    <chr>                                  <dbl>                            <dbl>
##  1 ZA6595_v3-0-0_23~                          0                               NA
##  2 ZA7572_v1-0-0_70~                          0                                0
##  3 ZA6595_v3-0-0_18~                          0                               NA
##  4 ZA6861_v1-2-0_27~                          0                                0
##  5 ZA6595_v3-0-0_26~                          0                               NA
##  6 ZA7572_v1-0-0_19~                          0                                1
##  7 ZA5877_v2-0-0_16~                          0                                0
##  8 ZA6861_v1-2-0_12~                          0                                0
##  9 ZA7572_v1-0-0_17~                          0                                0
## 10 ZA5877_v2-0-0_17~                          0                                1
## 11 ZA6861_v1-2-0_41~                          0                                0
## 12 ZA6861_v1-2-0_61~                          0                                1

Creating the Longitudional Table

Now we just need to join the partial table by the rowid together:

#start from the smallest (we removed the survey that had no relevant questionnaire item)
panel <- hw %>%
  left_join ( geography, by: 'rowid' ) 

panel <- panel %>%
  left_join ( demography, by: c("rowid", "isocntry") ) 

panel <- panel %>%
  left_join ( interview_dates, by: 'rowid' )

And let’s see a small sample:

sample_n(panel, 12)

## # A tibble: 12 x 19
##    rowid  serious_world_pr~ serious_world_pr~ isocntry geo   region    w1    wex
##    <chr>              <dbl>             <dbl> <chr>    <chr> <chr>  <dbl>  <dbl>
##  1 ZA686~                 0                 0 ES       ES41  Casti~ 1.21  46787.
##  2 ZA686~                 0                 0 RO       RO31  South~ 0.724 11805.
##  3 ZA686~                 0                 0 SK       SK02  Zapad~ 0.774  3499.
##  4 ZA757~                 0                 1 PT       PT16  Centr~ 1.11   9336.
##  5 ZA659~                 1                NA HR       HR041 Grad ~ 0.580  2098.
##  6 ZA659~                 1                NA RO       RO21  North~ 1.21  20160.
##  7 ZA686~                 0                 0 PT       PT17  Lisboa 0.932  7448.
##  8 ZA659~                 0                NA GB-GBN   UKI   London 0.994 50133.
##  9 ZA757~                 0                 0 CY       CY    REPUB~ 0.594   874.
## 10 ZA686~                 0                 0 LT       LT003 Klaip~ 0.623  1564.
## 11 ZA757~                 0                 0 IE       IE013 West ~ 0.490  1651.
## 12 ZA659~                 0                NA LT       LT003 Klaip~ 1.16   2917.
## # ... with 11 more variables: marital_status <chr>, age_education <chr>,
## #   age_exact <dbl>, occupation_of_respondent <chr>,
## #   occupation_of_respondent_recoded <chr>,
## #   respondent_occupation_scale_c_14 <chr>, type_of_community <chr>,
## #   is_student <dbl>, no_education <dbl>, education <dbl>,
## #   date_of_interview <date>

saveRDS ( panel, file.path(tempdir(), "climate_panel.rds"), version: 2)

# not evaluated
saveRDS( panel, file: file.path("data-raw", "climate-panel.rds"), version=2)

Putting It on a Map

This is not the end of the story. If you put all this on a map, the results are a bit disappointing.

Why? Because sub-national (provincial, state, county, district, parish) borders are changing all the time - within the EU and everywhere. The next step is to harmonize the geographical information. We have another CRAN released package to help you with. See the next post: Regional Climate Change Awareness Dataset.

What is Retrospective Survey Harmonization?

Thu, 04 Mar 2021 00:00:00 +0000

Reproducible ex post harmonization of survey microdata

Retrospective survey harmonization allows the comparison of opinion poll data conducted in different countries or time. In this example we are working with data from surveys that were ex ante harmonized to a certain degree—in our tutorials we are choosing questions that were asked in the same way in many natural languages. For example, you can compare what percentage of the European people in various countries, provinces and regions thought climate change was a serious world problem back in 2013, 2015, 2017 and 2019.

Engage with us on LinkedIn @CultDataObs or check out our open data and open repositories, code, tutorials

We developed the retroharmonize R package to help this process. We have tested the package with about 80 Eurobarometer, 5 Afrobarometer survey files extensively, and a bit with Arabbarometer files. This allows the comparison of various survey answers in about 70 countries. This policy-oriented survey programs were designed to be harmonized to a certain degree, but their ex post harmonization is still necessary, challenging and errorprone. Retrospective harmonization includes harmonization of the different coding used for questions and answer options, post-stratification weights, and using different file formats.

Eurobarometer, Afrobaromer, Arab Barometer and Latinobarómetro make survey files that are harmonized across countries available for research with various terms. Our retroharmonize is not affiliated with them, and to run our examples, you must visit their websites, carefully read their terms, agree to them, and download their data yourself. What we add as a value is that we help to connect their files across time (from different years) or across these programs.

The survey programs mentioned above publish their data in the proprietary SPSS format. This file format can be imported and translated to R objects with the haven package; however, we needed to re-design haven’s labelled_spss class to maintain far more metadata, which, in turn, a modification of the labelled class. The haven package was designed and tested with data stored in individual SPSS files.

The author of labelled, Joseph Larmarange describes two main approaches to work with labelled data, such as SPSS’s method to store categorical data in the Introduction to labelled.

Two main approaches of labelled data conversion.

Our approach is a further extension of Approach B. Survey harmonization in our case always means the joining data from several SPSS files, which requires a consistent coding among several data sources. This means that data cleaning and recoding must take place before conversion to factors, character or numeric vectors. This is particularly important with factor data (and their simple character conversions) and numeric data that occasionally contains labels, for example, to describe the reason why certain data is missing. Our tutorial vignette labelled_spss_survey gives you more information about this.

In the next series of tutorials, we will deal with an array of problems. These are not for the faint heart – you need to have a solid intermediate level of R to follow.

Tidy, joined survey data

The original files identifiers may not be unique, we have to create new, truly unique identifiers. Weighting may not be straightforward.
Neither the number of observations or the number of variables (which represents the survey questions and their translation to coded data) is the same. Certain data may be only present in one survey and not the other. This means that you will likely to run loops on lists and not data.frames, but eventually you must carefully join them.

Class conversion

Similar questions may be imported from a non-native R format, in our case, from an SPSS files, in an inconsistent manner. SPSS’s variable formats cannot be translated unambiguously to R classes. retroharmonize introduced a new S3 class system that handles this problem, but eventually you will have to choose if you want to see a numeric or character coding of each categorical variable.
The harmonized surveys, with harmonized variable names and harmonized value labels, must be brought to consistent R representations (most statistical functions will only work on numeric, factor or character data) and carefully joined into a single data table for analysis.

Harmonization of variables and variable labels

Same variables may come with dissimilar variable names and variable labels. It may be a challenge to match age with age. We need to harmonize the names of variables.
The harmonized variables may have different labeling. One may call refused answers as declined and the other refusal. On a simple choice, climate change may be ‘Climate change’ or Problem: Climate change. Binary choices may have survey-specific coding conventions. Value labels must be harmonized. There are good tools to do this in a single file - but we have to work with several of them.

Missing value harmonization

There are likely to be various types of missing values. Working with missing values is probably where most human judgment is needed. Why are some answers missing: was the question not asked in some questionnaires? Is there a coding error? Did the respondent refuse the question, or sad that she did not have an answer? retroharmonize has a special labeled vector type that retains this information from the raw data, if it is present, but you must make the judgment yourself – in R, eventually you will either create a missing category, or use NA_character_ or NA_real_.

That’s a lot to put on your plate.

It is unlikely that you will be able to work with completely unfamiliar survey programs if you do not have a strong intermediate level of R. Our package comes with tutorials for Eurobarometer, Afrobarometer and our development version already covers Arab Barometer, highlighting some peculiar issues with these survey programs, that we hope to give a head start for less experienced R users.

Eurobarometer Surveys Used In Our Project

Wed, 03 Mar 2021 00:00:00 +0000

In our tutorial series, we are going to harmonize the following questionnaire items from five Eurobarometer harmonized survey files. The Eurobarometer survey files are harmonized across countries, but they are only partially harmonized in time.

Engage with us on LinkedIn @CultDataObs or check out our open data and open repositories, code, tutorials

All data must be downloaded from the GESIS Data Archive in Cologne. We are not affiliated with GESIS and you must read and accept their terms to use the data.

Eurobarometer 80.2 (2013)

GESIS Data Archive, Cologne. ZA5877 Data file Version 2.0.0, https://doi.org/10.4232/1.12792

Data file: ZA6595 data file (European Commission 2017).
Questionnaire: Eurobarometer 83.4 Basic Bilingual Questionnaire
Citation: ZA6595 Bibtex

QA1a Which of the following do you consider to be the single most serious problem facing the world as a whole? (single choice)

QA1b Which others do you consider to be serious problems? (multiple choice)

QA2 And how serious a problem do you think climate change is at this moment? Please use a scale from 1 to 10, with '1' meaning it is "not at all a serious problem (scale 1-10)

QA4 To what extent do you agree or disagree with each of the following statements? - Fighting climate change and using energy more efficiently can boost the economy and jobs in the EU (agreement-disagreement 4-scale)

QA4 To what extent do you agree or disagree with each of the following statements? - Reducing fossil fuel imports from outside the EU could benefit the EU economically (agreement-disagreement 4-scale)

QA5 Have you personally taken any action to fight climate change over the past six months? (binary)

Eurobarometer 83.4 (2015)

European Commission, Brussels; Directorate General Communication COMM.A.1 ´Strategy, Corporate Communication Actions and Eurobarometer´GESIS Data Archive, Cologne. ZA6595 Data file Version 3.0.0, https://doi.org/10.4232/1.13146

Data file: ZA6595 data file (European Commission 2018).
Questionnaire: Eurobarometer 83.4 Basic Bilingual Questionnaire
Citation: ZA6595 Bibtex

Eurobarometer 87.1 (2017)

European Commission, Brussels; Directorate General Communication, COMM.A.1 ‘Strategic Communication’; European Parliament, Directorate-General for Communication, Public Opinion Monitoring Unit GESIS Data Archive, Cologne. ZA6861 Data file Version 1.2.0, https://doi.org/10.4232/1.12922

Data file: ZA6861 data file.
Questionnaire: Eurobarometer 90.2 Basic Bilingual Questionnaire
Citation: ZA6861 Bibtex

QC1a Which of the following do you consider to be the single most serious problem facing the world as a whole? (single choice)

QC1b Which others do you consider to be serious problems? (multiple choice)

QC2 And how serious a problem do you think climate change is at this moment? Please use a scale from 1 to 10, with '1' meaning it is "not at all a serious problem (scale 1-10)

Qc4 To what extent do you agree or disagree with each of the following statements? - Fighting climate change and using energy more efficiently can boost the economy and jobs in the EU (agreement-disagreement 4-scale)

Qc4 To what extent do you agree or disagree with each of the following statements? - Promoting EU expertise in new clean technologies to countries outside the EU can benefit the EU economically (agreement-disagreement 4-scale)

Qc4 To what extent do you agree or disagree with each of the following statements? - Reducing fossil fuel imports from outside the EU can benefit the EU economically (agreement-disagreement 4-scale)

Qc4 To what extent do you agree or disagree with each of the following statements? - Reducing fossil fuel imports from outside the EU can increase the security of EU energy supplies (agreement-disagreement 4-scale)

Qc4 To what extent do you agree or disagree with each of the following statements? - More public financial support should be given to the transition to clean energies even if it means subsidies to fossil fuels should be reduced. (agreement-disagreement 4-scale)

Qc5 Have you personally taken any action to fight climate change over the past six months? (binary)

Eurobarometer 90.2 (2018)

European Commission, Brussels; Directorate General Communication, COMM.A.3 ‘Media Monitoring and Eurobarometer’ GESIS Data Archive, Cologne. ZA7488 Data file Version 1.0.0, https://doi.org/10.4232/1.13289

Data file: ZA7488 data file (European Commission 2019a)
Questionnaire: Eurobarometer 90.2 Basic Bilingual Questionnaire
Citation: ZA7488 Bibtex

QB5 To what extent do you agree or disagree with each of the following statements? - Fighting climate change and using energy more efficiently can boost the economy and jobs in the EU (agreement-disagreement 4-scale)

QB5 To what extent do you agree or disagree with each of the following statements? - Promoting EU expertise in new clean technologies to countries outside the EU can benefit the EU economically (agreement-disagreement 4-scale)

QB5 To what extent do you agree or disagree with each of the following statements? - Reducing fossil fuel imports from outside the EU can benefit the EU economically (agreement-disagreement 4-scale)

QB5 To what extent do you agree or disagree with each of the following statements? - Reducing fossil fuel imports from outside the EU can increase the security of EU energy supplies (agreement-disagreement 4-scale)

QB5 To what extent do you agree or disagree with each of the following statements? - More public financial support should be given to the transition to clean energies even if it means subsidies to fossil fuels should be reduced. (agreement-disagreement 4-scale)

Eurobarometer 91.3 (2019)

European Commission, Brussels; Directorate General Communication, COMM.A.3 ‘Media Monitoring and Eurobarometer’ GESIS Data Archive, Cologne. ZA7572 Data file Version 1.0.0, https://doi.org/10.4232/1.13372

Data file: ZA7572 data file (European Commission 2019b).
Questionnaire: Eurobarometer 91.3 Basic Bilingual Questionnaire
Citation: ZA7572 Bibtex

QB4 To what extent do you agree or disagree with each of the following statements? - Taking action on climate change will lead to innovation that will make EU companies more competitive (N) (agreement-disagreement 4-scale)

QB4 To what extent do you agree or disagree with each of the following statements? - Promoting EU expertise in new clean technologies to countries outside the EU can benefit the EU economically (agreement-disagreement 4-scale)

QB4 To what extent do you agree or disagree with each of the following statements? - Reducing fossil fuel imports from outside the EU can benefit the EU economically (agreement-disagreement 4-scale)

QB4 To what extent do you agree or disagree with each of the following statements? - Adapting to the adverse impacts of climate change can have positive outcomes for citizens in the EU (agreement-disagreement 4-scale)

QB5 Have you personally taken any action to fight climate change over the past six months? (binary)

References

European Commission, Brussels. 2017. “Eurobarometer 80.2 (2013).” GESIS Data Archive, Cologne. ZA5877 Data file Version 2.0.0, https://doi.org/10.4232/1.12792. https://doi.org/10.4232/1.12792.

———. 2018. “Eurobarometer 83.4 (2015).” GESIS Data Archive, Cologne. ZA6595 Data file Version 3.0.0, https://doi.org/10.4232/1.13146. https://doi.org/10.4232/1.13146.

———. 2019a. “Eurobarometer 90.2 (2018).” GESIS Data Archive, Cologne. ZA7488 Data file Version 1.0.0, https://doi.org/10.4232/1.13289. https://doi.org/10.4232/1.13289.

———. 2019b. “Eurobarometer 91.3 (2019).” GESIS Data Archive, Cologne. ZA7572 Data file Version 1.0.0, https://doi.org/10.4232/1.13372. https://doi.org/10.4232/1.13372.

Music Streaming: Is It a Level Playing Field?

Tue, 23 Feb 2021 11:00:00 +0000

Our article, Music Streaming: Is It a Level Playing Field? is published in the February 2021 issue of CPI Antitrust Chronicle, which is fully devoted to competition policy issues in the music industry.

The dramatic growth of music streaming over recent years is potentially very positive. Streaming provides consumers with low cost, easy access to a wide range of music, while it provides music creators with low cost, easy access to a potentially wide audience. But many creators are unhappy about the major streaming platforms. They consider that they act in an unfair way, create an unlevel playing field and threaten long-term creativity in the music industry.

Our paper describes and assesses the basis for one element of these concerns, competition between recordings on streaming platforms. We argue that fair competition is restricted by the nature of the remuneration arrangements between creators and the streaming platforms, the role of playlists, and the strong negotiating power of the major labels. It concludes that urgent consideration should be given to a user-centric payment system, as well as greater transparency of the factors underpinning playlist creation and of negotiated agreements.

You can read the entire issue and the full text of our article on Competition Policy International in pdf.

Reprex introduction in IVIR

Tue, 02 Feb 2021 10:10:00 +0000

IViRtual 9 April 2021

Data Curation

Thu, 21 Jan 2021 00:00:00 +0000

If you cannot find the right data for your policy evaluation, your consulting project, your PhD thesis, your market research, or your scientific research project, it does not mean that the data does not exist, or that it is not available for free. In our experience, up to 95% of available open data is never used, because potential users do not realize it exists or do not know how to access it.

Every day, thousands of new datasets become available via the EU open data regime, freedom of information legislation in the United States and other jurisdictions, or open science and scientific reproducibility requirements — but as these datasets have been packaged or processed for different primary, original uses, they often require open data experts to locate them and adapt them to a usable form for reuse in business, scientific, or policy research.

The creative and cultural industries often do not participate in government statistics programs because these industries are typically comprised of microenterprises that are exempted from statistical reporting and that file only simplified financial statements and tax returns. This means that finding the appropriate private or public data sources for creative and cultural industry uses requires particularly good data maps.

Data curation means that we are continuously mapping potential data sources and sending requests to download and quality test the most current data sources. Our CEEMID project has produced several thousand indicators, of which a few dozen are available in our Demo Music Observatory.If you have specific data needs for a scientific research, policy evaluation, or business project, we can find and provide the most suitable, most current, and best value data for analysis or for [ethical AI applications](/service/Trustworthy AI/).

Data Processing

Thu, 21 Jan 2021 00:00:00 +0000

Data analysts spend 80% of their time on data processing, even though computers can perform these task much faster, with far less errors, and they can document the process automatically. Data processing can be shared: an analyst in a company and an analyst in an NGO does not have to reprocess the very same data twice*

See our blogpost How We Add Value to Public Data With Imputation and Forecasting?.

Data-as-Service

Thu, 21 Jan 2021 00:00:00 +0000

We want to ensure that individual researchers, artists, and professionals, as well as NGOs and small and large organizations can benefit equally from big data in the age of artificial intelligence.

Big data creates inequality and injustice because it is only the big corporations, big government agencies, and the biggest, best endowed universities that can finance long-lasting, comprehensive data collection programs. Big data, and large, well-processed, tidy, and accurately imputed datasets allow them to unleash the power of machine learning and AI. These large entities are able to create Algorithms that decide the commercial success of your product and your artwork, giving them a competitive edge against smaller competitors while helping them evade regulations.

Check out our iotables software that helps the use of national accounts data from all EU members states to create economic direct, indirect and induced economic impact calculation, such as employment multipliers or GVA affects of various cultural and creative economy policies.

Check out our regions software that helps the harmonization of various European and African standardized surveys.

Check out our retroharmonize software that helps the harmonization of various European and African standardized surveys.

Can scholarly pirate libraries bridge the knowledge access gap? An empirical study on the structural conditions of book piracy in global and European academia

Thu, 03 Dec 2020 11:00:00 +0000

PLOS One is the fourth most influential multidisciplinary journal after Nature, and Science, and Proceedings of the National Academy of Sciences of the United States of America (based on H index.) On December 3, 2020 it published a paper co-authored by Dr. Balazs Bodo, associate professor at the Institute for Information Law (IViR), Daniel Antal (Reprex, Demo Music Observatory), a data scientist interested in reproducible research, as an independent researcher, and Zoltan Puha, a Data Science PhD at Tilburg University, JADS. PLOS (Public Library of Science) is a nonprofit Open Access publisher, empowering researchers to accelerate progress in science and medicine by leading a transformation in research communication.

The article utilizes the our reproducible datasets created with our regions package, and builds on many years of expertise in empirical research on the field of music and audiovisual piracy, home copying and private copying compensation (see for example Private Copying in Croatia.) Our aim is to provide reliable, high quality indicators for the creative industries not only on national, but provincial, state, regional and metropolitan area level, too, because these levels are often more relevant for creators, performers and policy-makers.

The topic of the paper is Library Genesis (LG), the biggest piratical scholarly library on the internet, which provides copyright infringing access to more than 2.5 million scientific monographs, edited volumes, and textbooks. The paper uses advanced statistical methods to explain why researchers around the globe use copyright infringing knowledge resources. The analysis is based on a huge usage dataset from LG, as well as data from the World Bank, Eurostat, and Eurobarometer, to identify the role of macroeconomic factors, such as R&D and higher education spending, GDP, researcher density in scholarly copyright infringing activities.

We created a global and a far more detailed European model for pirate book downloads.

The main finding of the paper is that open access, even if it is radical, is not a panacea. The hypothesis of the research was that researchers in low-income regions use piratical open knowledge resources relatively more to compensate for the limitations of their legal access infrastructures. The authors found evidence to the contrary. Researchers in high income countries and European regions with access to high quality knowledge infrastructures, and high levels of funding use radical open access resources more intensively than researchers in lower income countries and regions, with less resourceful libraries. This means that while open knowledge is an important resource to close the knowledge gap between centrum and periphery, equality in access does not translate into equality in use. Structural knowledge inequalities are both present and are being reproduced in the context of open access resources.

The paper is unique not just because of the data it is based on. It also sets new standards in interdisciplinary legal research by publishing the paper, the data and the software code in the same time in open access repositories, following reproducible research best practices — the practices that we want to promote in our Digital Music Observatory and further data observatories to serve business, evidence-based policy and scientific research.

Our research was funded from the Horizon Europe 2020 Research grant #710722 “OPENing UP new methods, indicators and tools for peer review, dissemination of research results, and impact measurement”.

Valuing Music

Thu, 12 Nov 2020 11:00:00 +0000

The copyright and neighboring right royalty is the wage-like earning of the music performer, the composer and producer. In popular music performers usually play their own composition and sing their own lyrics. If the musician is self-published, all the three royalties directly pay the artist’s work. If a professional publisher is helping the artist as a composer, or a record producer as a recorded performer, they usually share the respective royalty revenues. In emerging markets fewer, in mature markets more composers are represented by publishers and performers by record labels.

International Comparisons

The CEEMID international comparison models are based on the implicit econometric model set out in the AKKA/LAA vs Konkurences padome case which goes far beyond the regional comparison suggested by the earlier Czech Léčebné lázně Mariánské Lázně v OSA case for comparing royalty tariffs in the European Union.

The EUCJ declared that “it is appropriate to compare its rates with those applicable in neighbouring Member States as well as with those applicable in other Member States adjusted in accordance with the PPP index, provided that the reference Member States have been selected in accordance with objective, appropriate and verifiable criteria and that the comparisons are made on a consistent basis.”

In our view, setting a royalty tariff, especially when comparison of other market prices is taken into consideration, should follow the principles of the IFRS 13 Fair Value Measurement international accounting standard. This accounting standard was endorsed as an official interpretation of accounting principles by the EU Commission Regulation 1255/2012, which is in turn a binding interpretation in the light of the directly applicable 1126/2008 EC Regulation. This document gives a practical guidance on how to use objective data in valuations. Although the language of the standard is mainly aimed at large corporation with heterogeneous balance sheets, as a canonical, cross-country reading of sound accounting principles it is applicable for establishing the fair value of an assets or liabilities, including intangible assets such as royalty-earning copyright and neighboring rights.

On the top level of the hierarchy we find publicly observable prices in frequently traded assets, which is not applicable in our case. The second level of the hierarchy contains more reliable, publicly verifiable data sources and the third level of the hierarchy non-public information that is objective. Whenever possible, publicly available and verified information should be used.

The structure of the music and audiovisual industries makes the use of Level 1 input data almost impossible in a European context, because the vast majority of the industry players are microenterprises and sole solicitors, and even larger entities such as collective management societies and TV stations usually fall in the scope of the EU small- and medium size enterprise category. With few exceptions, this is the case with HORECA tariff payers, i.e. restaurants, pensions, hotels and other similar establishments. Micro-, small and medium sized enterprises are usually subject to simplified and limited financial reporting, tax reporting rules and mandatory official statistical reporting rules. Most of the EU economic statistics are created from tax and financial reports, and in other cases from mandatory statistical reports. Because creative and cultural enterprises usually do not fall into the threshold set for most enterprise statistics data collection, and because of the simplified tax and accounting reports, less official statistics can be created.

In the absence of high-level valuation input data, we must use level 2 (public) and level 3 (private) data sources that are by nature subject to interpretation. Such interpretation is subjective in the language of the standard because it is personally made by an objective analyst. The reliability of the valuation depends not only on the quality of the data, which is the case with level 1 data, but on the sound analysis of a competent analyst.

CEEMID made a large effort in the last 5 years to use high quality, publicly observable data and private data to create meaningful and objective social, economic and cultural indicators as set out in the EUCJ preliminary decision.

Market Comparators

The best models are based on market data and use information on prices and quantities agreed by independent, arms-length contractual partners. Our market comparator models process all collectively managed and estimated individually licensed uses of music and audiovisual content in a country. It is than possible to compare private copying values, value transfer to media platforms, or individual uses such as radio or television to the total market value of music listening per hour or film watching per hour. These models require the most data input.

We used such estimates in the Slovak Music Industry Report, in the calculation of private copying benefits to consumers and value transfer to YouTube in Hungary and Croatia.

Direct Estimates & Hypothetical Evaluations

Direct estimates are measuring quantities of unknown uses and they relate them to known market prices. Hypothetical evaluations ask the willingness of users to pay for uses. These models are easier to create, require less information, but they can be more easily challenged by regulators, licensees or auditors, because they do not rely only on independent and arms-length transaction data.

Hedonic Pricing

Hedonic price models are econometric price models that establish the residua value of music in complex services like hotel or restaurant services. We have used hedonic price modelling to establish the value of music use in the HORECA sector. Technically these models could be used to evaluate the use in television or radio but the data requirements are extensive.

Music Reports

Thu, 12 Nov 2020 08:00:00 +0000

Our Digital Music Observatory is powering policy, business reports and scientific publication with high quality, internationally comparable data on

The observatory grew out of the Central and Eastern European Music Industry Databases out of necessity following a CISAC Good Governance Seminar for European Societies in 2013. The adoption of European single market and copyright rules, and the increased activity of competition authority and regulators required a more structured approach to set collective royalty and compensations tariffs in a region that was regarded traditionally as data-poor. We needed to collect and integrate data that is often collected in more advanced markets by government entities, and we needed to present them as evidence in the policy making and in courts.

Public Reports

The first Hungarian Music Industry Report, a 144-pages business strategy and policy advocacy report, which became the basis of annual reports in the Hungarian music industry.
The first Slovak Music Industy Report, a 227-pages advocacy report with business strategy and evidence-based policy recommendations. Several royalty pricing and other fact-based industry work was commissioned by Slovak stakeholders which are not publicly available. This report is based on reproducible research standards - i.e. most of the report can be automatically updated as new data comes in.
Private Copying in Croatia is an advocacy report for re-setting the remuneration of private copying, and measuring the value transfer to media platforms such as YouTube. In Hungary, more technical and detailed reports were made for Artisjus, Mahasz, EJI, Hungart and Filmjus, which are not available to the public. This is also a reproducible report.
Central & Eastern European Music Industry Report 2020 is the first Hungarian, Slovak, Croatian and Czech music industry reports are compared with Armenian, Austrian, Bulgarian, Lithuanian, Serbian and Slovenian data and findings. It put on stage on as a good example of evidence-based policy making on a two-day high-level stakeholder event jointly organized by Geothe-Institute and the DG Education, Youth, Sport and Culture of the European Commission with the Creative FLIP project. (See a brief summary of the presentation and our use case.) This is our first reproducible report that covert primary information flowing in in 12 languages.
The Welsh Music Industries In a Post-Covid World is first of a planned series of post-pandemic recovery reports created by Dr Paul Carr. While the UK Music Industry is generally seen among the world leader, it’s power is mainly concentrated in London and England. Wales appears to share more problems with emerging national music scenes like the Slovakian or Croatian — small, recently independent nations that are seeking their best way to place their national repertoire in an increasingly competitive regional and global music market.
Metropolitan Area, Regional, or Sub-National Reports: Because music and culture are often managed at the level of cities, regions and communities, we want to give you all the data on sub-national levels, whether for regions, metropolitan areas or smaller divisions.

Non-Public Reports

Our observatories do not only collect, but also integrate data. We own many data assets, we make many open data assets available, but we also can integrate highly proprietary data with our assets. Our novel approach compared to other observatories is that we can connect members of the observatory, join their data, create a new insight, and then close the data connection. This way data synergies can be exploited without the conflict of interest troubles related by change of data ownership.

Valuation Reports: for price reviews, copyright tribunals, competition cases, regulatory oversight. We want your evidence to stand a fighting chance against large teams of professional legal and economics teams on the other side with proper valuations and damage claims.
Advocacy Reports: We want you to be able to prove to your fans, the press, your economy minister that music in many countries has not been at the mercy of the taxpayer, but has been carrying far heavier tax burdens than manufacturers. We want to make your case that the music industry plays a vital role in the European economic recovery and job creation, because we can create economic impact assessments on GDP, employment, tax, import and export effects automatically.
Grant Reports: Ex ante grant design in a comprehensive, logical framework, supported by empirical evidence, artist and ecosystem data and needs, direct and indirect (effect) indicators for program controls. We would like to persuade grant managers that ex ante evaluation must be followed by ex post valuation. Original expectations must be contrasted with the effects of the grant and fed back to design better grants in the following year. (See more in Smart Grants)

Szavazz a Reprexre!

Tue, 29 Sep 2020 10:00:00 +0000

Table of Contents

Hogyan tudsz szavazni?

Klikkelj a Cast your vote for The Hague Innovators challenge 2022 oldalra!

A jobb oldali görgetővel scrollozz le addig, amíg a szavazólapot nem látod. Válaszd ki a Reprex címet és írd be az email címedet. Ahhoz, hogy engedjen a gép választani a cégek közül, ki kell pipálnod a `✅ I’m not a robot’ cimkét, ahogyan a képen mutatjuk.

A szavazatod még nem érvényes, azt meg kell erősítened. Lépj be abba az email fiókba, amit az előbb megadtál. (Az email címedet nem használják marketing célokra, nem adják át másnak, csak azért rögzítik, hogy kizárják a többszöri szavazást.)

Ellenőrizd, hogy valóban a Reprex vállatnév van az angol nyelvű szövegben és klikkelj az angol szöveg utáni linkre. Ezzel megerősíted, hogy valóban ránk akarsz szavazni (Köszi!)

Egy megerősítő szövegre leszel átirányítva, ami hollandul (vagy angolul) annyit mond, hogy a szavazatodat megerősítették és nincsen több tennivalód.

Ha ezt látod, akkor nagyon szépen köszönjük a fáradozásodat. De azért nekünk még segíthetsz ezzel-azzal…

Hogyan terjesztheted ezt a felhívást?

Oszd meg kérlek a videó bemutatkozónkat a YouTube oldalról a barátaiddal, kollégáiddal, alkotótársaiddal.

A ⚙️ jel megnyomásával több opció jelenik meg. A subtitles segít kiválasztani a feliratozás nyelvét, ebből a Hungarian lesz a magyar:) Ha már arra jársz, kérlek, dobj ránk egy 👍 lájkot is, jól esik :)

Az üzenetünk maga az üzenet. Egy olyan etikus startup vállalkozás vagyunk, aminek az egyik célja az, hogy csökkentse a nem angol nyelvű alkotók, a nők, a kisebbségek hátrányát a nagy, big data és AI vezérelte globális internet platformokon, például a YouTube vagy a Spotify oldalán. Ezek az oldalak egyre inkább gépi tanulással működnek, és ha nincsen elég információjuk a magyarokról, nőkről, székelyekről, akkor senkinek nem fogják a mi tartalmainkat ajánlani.

Ezeken a nyelveken is elérhető az üzenetünk:

🇳🇱 🇬🇧 🇧🇦 🇨🇿 🇭🇺 🇩🇪 🇱🇹 🇫🇷 🇸🇰 🇪🇸 🇹🇷 + Catalan.

2 Oszd meg a Twitteren (ha használod ezt a közösségi médiát) a felhívásunknak. Zenei körökben lehetőleg ezt:

🤎 We ask you, humbly, to support us with a vote or by sharing our appeal. We're part of the #opensource, #opendata, and #openscience movement that depends on your support to stay online and thriving, but many of our users simply look the other way🤦🏻‍♀️https://t.co/zdU2i0Jvvn
— Creative Cultural Data Observatory (@CultDataObs) November 1, 2022

Zenei körökben inkább ezt: @DigitalMusicObs; zöldeknek meg ezt: @GreeDealObs.

Lájkold a LinkedIn oldalunkat és ha teheted, ott is oszd meg a felhívásunkat néhány kedves magyar szó kíséretében.
Vagy csak egyszerűen küld el ennek az oldalnak az URL linkjét a böngésződből barátaidnak, kollégáidnak, alkotótársaidnak.

Miért szavazz ránk?

Döntősök vagyunk egy nagyon fontos hollandiai versenyben: a The Hague Innovation Awardért vagyunk versenyben 7 másik olyan vállalkozással, amelyik fenntarthatóbbá, jobbá akarja tenni a világot. Mi azért harcolunk, hogy a big data és az AI mindenkinek működjön: az etnikai kisebbségek, kis országok, nők, ne újabb versenyhátrányba kerüljenek, hanem egyenelő esélyeket kapjanak az algoritmusoktól. Azzal foglalkozunk, hogy az olyan kis országok zenészeit, mint Magyarország, miért nem ajánlják a globális streaming szolgáltatók még magyaroknak sem, vagy a civilek miért nem találnak bizonyítékokat a hamis öko-reklámok ellen. Minden olyan kisvállalkozásnak, egyéni alkotónak, civil szervezetnek szeretnénk segíteni, akiknek nincsen pénzük arra, hogy adattudósokat, adat mérnököket, statisztikusokat alkalmazzanak, hogy megvédjék magukat a sötét és kapzsi algoritmusokkal szemben. Ehhez olyan kollektív megoldásokat kínálunk, amiket a zenészek, a kulturális szféra szereplői, vagy a zöld szervezetek közösen tudnak használni.

Lépjünk kapcsolatba!

Írj a Daniel Antal vagy a antaldaniel címre, vagy egyszerűen küldj egy emailt. Köszönjük szépen a segítségedet!

retroharmonize R package for survey harmonization

Tue, 25 Aug 2020 00:00:00 +0000

Retrospective data harmonization

The vignette Working With The labelled_spss_survey Class provides more information about the labelled_spss_survey() class.

In Harmonize Value Labels we discuss the characteristics of the labelled_spss_survey() class and demonstrates the problems that using this class solves.

We also provide three extensive case studies illustrating how the retroharmonize package can be used for ex-post harmonization of data from cross-national surveys:

The creators of retroharmonize are not affiliated with either Afrobarometer, Arab Barometer, Eurobarometer, or the organizations that designs, produces or archives their surveys.

Citing the data sources

Citing the retroharmonize R package

For main developer and contributors, see the package homepage.

This work can be freely used, modified and distributed under the GPL-3 license:

citation("retroharmonize")
#> 
#> To cite package 'retroharmonize' in publications use:
#> 
#>   Daniel Antal (2021). retroharmonize: Ex Post Survey Data
#>   Harmonization. R package version 0.1.17.
#>   https://retroharmonize.dataobservatory.eu/
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Manual{,
#>     title: {retroharmonize: Ex Post Survey Data Harmonization},
#>     author: {Daniel Antal},
#>     year: {2021},
#>     doi: {10.5281/zenodo.5006056},
#>     note: {R package version 0.1.17},
#>     url: {https://retroharmonize.dataobservatory.eu/},
#>   }

Contact

For contact information, contributors, see the package homepage.

Code of Conduct

Please note that the retroharmonize project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.

iotables R package for working with symmetric input-output tables

Wed, 03 Jun 2020 00:00:00 +0000

iotables processes all the symmetric input-output tables of the EU member states, and calculates direct, indirect and induced effects, multipliers for GVA, employment, taxation. These are important inputs into policy evaluation, business forecasting, or granting/development indicator design. iotables is used by about 800 experts around the world.

Code of Conduct

Please note that the iotables project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.

ESCO Focus Group on CCI Occupations and Skills

Wed, 11 Mar 2020 14:16:00 +0000

Daniel Antal, the founder of the Digital Music Observatory was be participating in the ESCO Focus group on Cultural and Creative Occupations and Skills of the Creative FLIP project.

CEEMID has a large enough answer pool for an empirical clustering of occupations. We are able to perform clustering of job descriptions, earnings, educational levels, and work output (such as films, documentaries, TV programmes, live music performances, music compositions, etc) to empirically create groups with a “set of jobs whose main tasks and duties are characterised by a high degree of similarity” in terms of economic activity and work output. Further details in the blog post here.

Central & Eastern European Music Industry Report 2020

Wed, 29 Jan 2020 16:00:00 +0000

The predecessor of Reprex’s Demo Music Observatory, CEEMID together with the independent music distributor Consolidated Independent presented and discussed with stakeholders the Central & Eastern European Music Industry Report 2020 as a case-study on national and comparative evidence-based policymaking in the cultural and creative sector on the CCS Ecosystems: FLIPPING THE ODDS Conference – a two-day high-level stakeholder event jointly organized by Geothe-Institute and the DG Education and Culture of the European Commission with the Creative FLIP project.

The CEE Report builds on the results of the first Hungarian, Slovak, Croatian and Czech music industry reports are compared with Armenian, Austrian, Bulgarian, Lithuanian, Serbian and Slovenian data and findings.

Our research findings were earlier presented and discussed in Vienna, Prague, Budapest and Bratislava with stakeholders.

You can find the earlier presentations in the blog section of the website.

Executive Summary

The first Central European Music Industry Report is the result of a co-operation that started among stakeholders in three EU countries five years ago to measure the economic value added of music – the basis of a modern royalty pricing system. This gave birth to CEEMID, originally the Central & Eastern European Music Industry Databases, a data integration programme that now in 2020, covers all of Europe. CEEMID fulfils similar roles to the planned European Music Observatory and supports all pillars of the future pan-European system.

The comparison of Western and Eastern music audiences reveals key demographic differences that make the unchanged adoption of business practices from mature markets in the region questionable. Chapter 2 of this report will show these differences and their consequences on music markets, in terms of visiting and acquisition likelihood, frequency, seasonality and purchasing capacity. This is an example of how CEEMID fulfils the role of Pillar 3 (music, society and citizenship) in the planned European Music Observatory.

Chapter 3 contrasts market demand with the supply strategies of musicians. CEEMID has been surveying music professionals, including artists, technicians and managers about their working conditions, market conditions and plans for five years across a growing number of countries. In 2019 we invited 100 national and regional stakeholders to distribute our surveys. In some countries, our surveys already have several years of historic data, making the resulting musician database probably the largest ever source of data about how music is produced and how musicians live. We are constantly looking for partners to roll out this survey to new countries in new languages.

The CEE region has comparative advantages in big music events like festivals, and it has become one of the most important hubs for cultural tourism in the world. We explain this phenomenon in Chapter 4 by showing the differences in demand composition, demography and supply of venues in the second chapter. The lack of a modern and dense network of permanent music venues gave rise to magnificent music festivals in the CEE. Open’er, Sziget and Exit are among the biggest and best festivals in the world, closely followed by several smaller festivals in all countries. The share of festivals in the live music market is many times higher than in Western Europe and they provide vital export revenues to the local music economies. However, they play a limited role in finding new audiences for local artists, as they are increasingly programming for Western audiences by providing shows of international hits. They can only very partially fill in the gaps left by the small venue problem that hit the emerging markets harder than the UK or Australia, where policy action had been already taken to reverse the decline of the availability of smaller live music venues.

On the recording side, our analysis shows that modern digital services are growing at a faster rate than in mature markets. Because of lower repertoire competition, streaming quantities are similar for a typical Austrian, Czech, Hungarian, Polish or Slovak track than in the mature markets. However, revenue growth is limited because of the interplay of several analysed factors. Our analysis of the live and recorded music markets shows that CEEMID fulfils the roles of the Pillar 1 (music economy) of the planned European Music Observatory.

Most recorded music sales revenue in the region comes from streaming platforms, just like in the mature markets. Successful sales strategies require a solid knowledge of the global marketplace and the ability to understand and train sales Algorithms. Micro-enterprises, such as independent labels, have very limited ability to cope with these functions, given that they do not have market research or R&D functions. CEEMID and Consolidated Independent have started initiating open, national R&D consortia to create the necessary concentration in data assets, analytical capacity and budgets to close this gap. As a first step, CEEMID and Consolidated Independent have created a large, independent music dataset based on hundreds of millions of royalty statement entries to create our market indexes, styled after stock market and bond market indexes. Streaming opportunities are fast changing as roll-out of streaming services is happening at a different rate in various territories; subscription charges and the exchange rate to the producer’s currency vary and repertoire competition emerges in the market. Our volume and revenue indexes in Chapter 5.3 are aimed at creating sales Algorithms that optimize sales volumes and expected revenues. We believe that this analysis also reveals that CEEMID partially fulfils the roles of Pillar 2 (music diversity and circulation) and feeds important data into Pillar 4 (innovation).

The region has far bigger untapped potential than most music business executives believe. Households in the region spend a significantly lower share of their recreational budget on music than their Western, Southern or Nordic peers. The region has a lot of untapped cultural purchasing power because servicing is particularly challenging in both the live and recorded sides of the business.

This upside potential cannot be tapped without better pricing. Royalty levels are often very low in the region. Due to many combined effects analysed in this short report, the gap between royalties earned in the CEE and Western Europe is several times bigger than the difference in GDP or national average wage. These gaps are partly caused by special interests preventing collective management from charging appropriate tariffs for restaurants, media companies or electronic appliance importers and manufacturers, and partly by unfavourable taxation of cultural products and services.

CEEMID was designed to create economic evidence on royalty pricing, private copying compensation and the creation of economic value added in the industry. In the first Hungarian Music Industry Report of ProArt and in the first Slovak Music Industry Report we have shown that economic and taxation policies of the CEE countries aimed to support car and electronics manufacturing create a distorted, unfavourable economic regime for creative industries. We want to help local stakeholders with economic evidence to correct these discriminatory policies during the overhaul of the EU VAT system. We have been helping various national organizations with economic evidence, presented in the light of latest EU jurisprudence, to improve their pricing activities. Our thousands of indicators were also used in ex ante evaluations of granting schemes.

In 2020, all EU member states will change their copyright administration legislation because of the national implementations of the 2019/790 Digital Single Market directive. CEEMID provides evidence in several countries about the size and impact mechanism of the value transfer, and generally the widespread use of the copyright exemption for private copying. We believe that the thousands of pan-European music industry indicators that we have aggregated over the five years will play a vital role in these regulatory processes.

CEEMID fulfils its roles with a very thorough exploitation of the EU’s 17-years-old Open Data regime with the re-use of public sector information, and a very careful mapping of the music industry. These maps help us conduct annual surveys among musicians and the audience, and they help us connect (always with pre-approval and with a user mandate) to industry databases. We do not only cover the EU countries, but increasingly (potential) candidate countries and neighbourhood countries.

In our vision, this data collection and integration, i.e. Pillars 1-3 should be available for all music stakeholders, should remain public and publicly funded. The last Pillar of the observatory, innovation, is where private entities should compete. The founders of CEEMID and Consolidated Independent believe that this report demonstrates the business and policy benefits of such a system with the analysis of the Central & Eastern European music markets. We believe that this way CEEMID is in a position to serve most of the planned functions of the envisioned European Music Observatory, and we are looking for ways to make either our thousands of indicators, or our data collection and integration software open source and available for all stakeholders in the EU and its neighbours. CEEMID was born out of necessity to level out the different levels of public research and statistical coverage of the EU member states. In our view, private entities in the future should focus their investments in Pillar 4 of the planned observatory, i.e. competing in innovation with creating new models, Algorithms and services based on data that is available throughout the European Union without giving further advantage to the already mature markets.

Central European Music Industry Report 2020

Wed, 22 Jan 2020 14:16:00 +0000

CEEMID & Consolidated Independent will present to the audience the Central European Music Industry Report 2020 (will be availale in all formats on the presentation day) on the CCS Ecosystems: FLIPPING THE ODDS Conference – a two-day high-level stakeholder event jointly organized by Geothe-Institute and the DG Education and Culture of the European Commission with the Creative FLIP project.

Creative FLIP stands for Finance, Learning, Innovation and Patenting is a Pilot project co-funded by the EU whose main objective is to support healthy and sustainable ecosystems for Cultural and Creative Industries (CCIs) with respect to these four key policy areas.

Executive Summary

Most recorded music sales revenue in the region comes from streaming platforms, just like in the mature markets. Successful sales strategies require a solid knowledge of the global marketplace and the ability to understand and train sales algorithms. Micro-enterprises, such as independent labels, have very limited ability to cope with these functions, given that they do not have market research or R&D functions. CEEMID and Consolidated Independent have started initiating open, national R&D consortia to create the necessary concentration in data assets, analytical capacity and budgets to close this gap. As a first step, CEEMID and Consolidated Independent have created a large, independent music dataset based on hundreds of millions of royalty statement entries to create our market indexes, styled after stock market and bond market indexes. Streaming opportunities are fast changing as roll-out of streaming services is happening at a different rate in various territories; subscription charges and the exchange rate to the producer’s currency vary and repertoire competition emerges in the market. Our volume and revenue indexes in Chapter 5.3 are aimed at creating sales algorithms that optimize sales volumes and expected revenues. We believe that this analysis also reveals that CEEMID partially fulfils the roles of Pillar 2 (music diversity and circulation) and feeds important data into Pillar 4 (innovation).

In our vision, this data collection and integration, i.e. Pillars 1-3 should be available for all music stakeholders, should remain public and publicly funded. The last Pillar of the observatory, innovation, is where private entities should compete. The founders of CEEMID and Consolidated Independent believe that this report demonstrates the business and policy benefits of such a system with the analysis of the Central & Eastern European music markets. We believe that this way CEEMID is in a position to serve most of the planned functions of the envisioned European Music Observatory, and we are looking for ways to make either our thousands of indicators, or our data collection and integration software open source and available for all stakeholders in the EU and its neighbours. CEEMID was born out of necessity to level out the different levels of public research and statistical coverage of the EU member states. In our view, private entities in the future should focus their investments in Pillar 4 of the planned observatory, i.e. competing in innovation with creating new models, algorithms and services based on data that is available throughout the European Union without giving further advantage to the already mature markets.

Slides

Tue, 21 Jan 2020 00:00:00 +0000

Data Curation With Reprex

Reprex | Data Curation

Presentation Controls

Next: Right Arrow or Space
Previous: Left Arrow
Start: Home
Finish: End
Overview: Esc
Speaker notes: S
Fullscreen: F
Zoom: Alt + Click
PDF Export: E

Data Curation

If you do not find the data for your research, reporting or evaluation project, it does not mean that the data does not exist. It does not mean that the data is not available for free.

Fragments

Make content appear incrementally

One Two Three

A fragment can accept two optional parameters:

class: use a custom style (requires definition in custom CSS)
weight: sets the order in which a fragment appears

Speaker Notes

Add speaker notes to your presentation

{{% speaker_note %}}
- Only the speaker can read these notes
- Press `S` key to view
{{% /speaker_note %}}

Press the S key to view the speaker notes!

Themes

black: Black background, white text, blue links (default)
white: White background, black text, blue links
league: Gray background, white text, blue links
beige: Beige background, dark text, brown links
sky: Blue background, thin dark text, blue links

night: Black background, thick white text, orange links
serif: Cappuccino background, gray text, brown links
simple: White background, black text, blue links
solarized: Cream-colored background, dark green text, blue links

Custom Slide

Customize the slide style and background

{{< slide background-image="/media/boards.jpg" >}}
{{< slide background-color="#0000FF" >}}
{{< slide class="my-style" >}}

Custom CSS Example

Let’s make headers navy colored.

Create assets/css/reveal_custom.css with:

.reveal section h1,
.reveal section h2,
.reveal section h3 {
  color: navy;
}

Questions?

Ask

Documentation

Slides

Tue, 21 Jan 2020 00:00:00 +0000

Data Curation With Reprex

Reprex | Data Curation

Presentation Controls

Next: Right Arrow or Space
Previous: Left Arrow
Start: Home
Finish: End
Overview: Esc
Speaker notes: S
Fullscreen: F
Zoom: Alt + Click
PDF Export: E

Data Curation

If you do not find the data for your research, reporting or evaluation project, it does not mean that the data does not exist. It does not mean that the data is not available for free.

Fragments

Make content appear incrementally

One Two Three

A fragment can accept two optional parameters:

class: use a custom style (requires definition in custom CSS)
weight: sets the order in which a fragment appears

Speaker Notes

Add speaker notes to your presentation

{{% speaker_note %}}
- Only the speaker can read these notes
- Press `S` key to view
{{% /speaker_note %}}

Press the S key to view the speaker notes!

Themes

black: Black background, white text, blue links (default)
white: White background, black text, blue links
league: Gray background, white text, blue links
beige: Beige background, dark text, brown links
sky: Blue background, thin dark text, blue links

night: Black background, thick white text, orange links
serif: Cappuccino background, gray text, brown links
simple: White background, black text, blue links
solarized: Cream-colored background, dark green text, blue links

Custom Slide

Customize the slide style and background

{{< slide background-image="/media/boards.jpg" >}}
{{< slide background-color="#0000FF" >}}
{{< slide class="my-style" >}}

Custom CSS Example

Let’s make headers navy colored.

Create assets/css/reveal_custom.css with:

.reveal section h1,
.reveal section h2,
.reveal section h3 {
  color: navy;
}

Questions?

Ask

Documentation

Slides

Tue, 21 Jan 2020 00:00:00 +0000

Research Automation With Reprex

Reprex | Research Automation

Presentation Controls

Next: Right Arrow or Space
Previous: Left Arrow
Start: Home
Finish: End
Overview: Esc
Speaker notes: S
Fullscreen: F
Zoom: Alt + Click
PDF Export: E

Research Automation

We believe that if you have already done the same data processing or controlling step twice in your team, the problem will show up the third and the fourth time, too.

If your junior researchers have downloaded the same dataset at least twice, or made the same processing steps in Excel or Power BI already twice, then it is high-time to automate the process, because each manual process is time-consuming and error-prone.
For your senior researchers, automation of data validation and unit-testing can save equally many hours.

Fragments

Make content appear incrementally

One Two Three

A fragment can accept two optional parameters:

class: use a custom style (requires definition in custom CSS)
weight: sets the order in which a fragment appears

Speaker Notes

Add speaker notes to your presentation

{{% speaker_note %}}
- Only the speaker can read these notes
- Press `S` key to view
{{% /speaker_note %}}

Press the S key to view the speaker notes!

Themes

black: Black background, white text, blue links (default)
white: White background, black text, blue links
league: Gray background, white text, blue links
beige: Beige background, dark text, brown links
sky: Blue background, thin dark text, blue links

night: Black background, thick white text, orange links
serif: Cappuccino background, gray text, brown links
simple: White background, black text, blue links
solarized: Cream-colored background, dark green text, blue links

Custom Slide

Customize the slide style and background

{{< slide background-image="/media/boards.jpg" >}}
{{< slide background-color="#0000FF" >}}
{{< slide class="my-style" >}}

Custom CSS Example

Let’s make headers navy colored.

Create assets/css/reveal_custom.css with:

.reveal section h1,
.reveal section h2,
.reveal section h3 {
  color: navy;
}

Questions?

Ask

Documentation

Slides

Tue, 21 Jan 2020 00:00:00 +0000

Trustworthy AI With Reprex

Reprex | Data Curation

Presentation Controls

Next: Right Arrow or Space
Previous: Left Arrow
Start: Home
Finish: End
Overview: Esc
Speaker notes: S
Fullscreen: F
Zoom: Alt + Click
PDF Export: E

Trustworthy AI

The European Union has adopted the Ethics Guidelines for Trustworthy AI with seven important principles:

Human agency and oversight
Technical robustness and safety
Privacy and Data governance
Transparency
Diversity, non-discrimination and fairness
Societal and environmental well-being
Accountability

In the EU, in the US or in the UK creative industries usually do not have access to trustworthy AI in 2021.

Our Focus

Make content appear incrementally

Transparency: we use open data, open source software, and we make sure that our clients understand what is happening with their products

Diversity: we know that a Slovak, Hungarian or Estonian song has less chances than an American one. We see that many corporate algorithms reinforce racism and patriarchy.

Societal well-being: we understand that the algorithm is maximizing the well-being and profit of the entity that trains it.

Accountability: algorithm-driven robotic sales must be held accountable, and must comply with consumer-protection, competition-protecion and non-discrimination rules.

A fragment can accept two optional parameters:

class: use a custom style (requires definition in custom CSS)
weight: sets the order in which a fragment appears

Speaker Notes

Add speaker notes to your presentation

{{% speaker_note %}}
- Only the speaker can read these notes
- Press `S` key to view
{{% /speaker_note %}}

Press the S key to view the speaker notes!

Themes

black: Black background, white text, blue links (default)
white: White background, black text, blue links
league: Gray background, white text, blue links
beige: Beige background, dark text, brown links
sky: Blue background, thin dark text, blue links

night: Black background, thick white text, orange links
serif: Cappuccino background, gray text, brown links
simple: White background, black text, blue links
solarized: Cream-colored background, dark green text, blue links

Custom Slide

Customize the slide style and background

{{< slide background-image="/media/boards.jpg" >}}
{{< slide background-color="#0000FF" >}}
{{< slide class="my-style" >}}

Custom CSS Example

Let’s make headers navy colored.

Create assets/css/reveal_custom.css with:

.reveal section h1,
.reveal section h2,
.reveal section h3 {
  color: navy;
}

Questions?

Ask

Documentation

Music Exports & Imports

Fri, 27 Sep 2019 00:00:00 +0000

Europe, like most regions of the world, is a net importer of music, but has very signficant export potential. Intra-European export is also very significant, especially from Germany, France, Sweden and the Netherlands.

The Central and Eastern European region is relatively weak in exporting recorded and live performances, but it is among global leaders in creating export revenues from festivals.

Currently I am involved in creating predicitve models that help finding the best potential foreign market for music, and I make economic impact assessments of cultural tourism.

Panel Discussion of the Central Euroepan Music Industry Report 2019

Fri, 20 Sep 2019 10:16:00 +0000

The panel discussion of the first comprehensive Central and Eastern European report (Albania, Armenia, Austria, Bulgaria, Croatia, Czechia, Estonia, Lithuania, Poland, Romania, Slovakia, Slovenia) will be in held on the Waves Music Showcase Festival and Conference.

The presentation 10Mb self-contained HTML file, takes some time to load

You can read a related blogpost here

Quoted from the conference agenda:

CEEMID – Central & Eastern European Music Databases was created out of necessity: many Central and Eastern European countries are lacking effective cross-industry or governmental statistics on music or on cultural industries in general. The lack of data makes the work of tour planning, record promotion, grant evaluation or royalty tariff setting very difficult in those countries where the need would be even greater for such activities than in the most advanced markets.

Musicians are the most known source of information

Daniel Antal developed a survey distributed among music professionals asking them about their experience. These surveys were designed to fill the gaps in the data collection of official statistics and other music and film industry sources. The first survey was made in 2014 in Hungary, and in 2019 it already included 11 countries. They had been filled out in the past years by more than 5000 creative industry professional, in 2019 in Austria, too.

Daniel Antal believes that the music business in most of Europe, including Austria, has many common, similar challenges and would need a lot more regional cooperation to solve them. With the CEEMID-report, presented at the Waves Vienna Conference. Daniel Antal gives a fact-based comparison and will suggest some possible policy and strategic solutions to be debated after.

Slovak Music Industry Report

Tue, 26 Mar 2019 00:00:00 +0000

The whole report can be downloaded for free from the homepage of SOZA.

Press reports in Slovak:

SME: [SOZA predstavila prvú Správu o slovenskom hudobnom priemysle](Čítajte viac: https://domov.sme.sk/c/22084685/soza-predstavila-historicky-prvu-spravu-o-slovenskom-hudobnom-priemysle.html)

Kreatívna Europa: Historicky prvá Správa o slovenskom hudobnom priemysle prináša okrem analýz aj odporúčania pre budúcnosť

In Hungarian: Új Szó: Tudományosan a hazai zeneiparról

Slides

Spáva o hudobnom priemysle na Slovensku from Dániel Antal

CEEMID

Wed, 27 Feb 2019 00:00:00 +0000

The Feasibility study for the establishment of a European Music Observatory (in short: EMO Feasibility Study)[1] has identified four critical data gaps related to the diversity and circulation of European music. This is a key business planning and policy problem, and of course, a serious shortcoming for better music research in Europe.

According to the EMO Feasibility Study, “there is already an existing pool of data that would allow the European Music Observatory to start compiling information about the European music sector. Various potential sources and providers of data are … [available, such as our Digital Music Observatory.]. Many of these providers have already indicated support for a future European Music Observatory, and are willing to collaborate and provide data. However, some data is not collected or is not aggregated in a way that it can be compared across Europe[2].

From CEEMID to the Digital Music Observatory

The Digital Music Observatory (formerly CEEMID) has been identified among these data sources[3]. CEEMID was originally an acronym for a data integration project that aimed to Create a Regional Music Database to Support Professional National Reporting, Economic Valuation and a Regional Music Study[4]. According to the European Commission’s feasibility study, “CEEMID can transfer thousands of indicators that are reproducible and verifiable, open-source software that creates them to a European Music Observatory. In particular, CEEMID provides a useful and interesting approach to harnessing the possibilities of open data in Europe in relation to the music sector, which should be further explored by the European Music Observatory in its start-up phase.”[5]

The Digital Music Observatory is built on the gradual, and open extension of the original CEEMID consortium, and builds on practical research and innovation that contributed to many music policy goals in an increasing number of countries since 2014. The original CEEMID consortium delivered the first Hungarian music industry studies (followed by several subsequent ones), the Slovak national music industry study, Private Copying in Croatia, and eventually the regional Central European Music Industry Report.[6] All of these industry studies addressed the problem of shrinking competitiveness in of the small language repertoire in the streaming era.

Historically CEEMID started out as the Central and Eastern European Music Industry Databases out of necessity following a CISAC Good Governance Seminar for European Societies in 2013, and eventually grew out of an abandoned GESAC project. The adoption of European single market and copyright rules, and the increased activity of competition authority and regulators required a more structured approach to set collective royalty and compensations tariffs in a region that was regarded traditionally as data-poor with lower quantity of industry and government data sources available.

The EMO Feasibility Study is built on the consensus of many European music organizations around a shared understanding “that a European Music Observatory should act as a centralised music data and an intelligence hub at European level.” However, as a pilot project and policy brief carried out in Finland[7], or the CEEMID project in Central and Eastern Europe has shown, has its limits. We believe that the existing data availability is better than that described in the Feasibility study, and since the creation of this report, it has only got better.

Since 2014, the Digital Music Observatory (former CEEMID) successfully piloted data collection, standardization and harmonization procedures that can fill all the data gaps identified in the music diversity and circulation pillar of the future European Music Observatory. CEEMID was transformed into a “Demo Music Observatory” on 15 September 2020[8] and eventually named Digital Music Observatory in 2021. Its policy relevance was also recognized in the United Kingdom after Brexit[9].

The Digital Music Observatory follows the Open Policy Analysis Guidelines[10] The OPA Guidelines give practical guidance on how to improve the transparency, replicability, and as a result, the reusability of the policy-making work with a scientific underpinning.

The EMO Feasibility Study acknowledges the importance of open data. The “EMO should also take advantage of open data where possible. Open data is data that can be freely used, re-used and redistributed by anyone. […] Using some form of open source software where relevant would allow an EMO to access a continuous peer-review of data ingestion, processing, corrections and indicator creation by statisticians, data scientists and academics.”[11]

As stated in this final report, the 2019 Open Data Directive further extended the availability of re-usable public sector information (PSI) with open science data. . In both PSI, as open government data, and in open science data, there is a huge potential to fill in the data gaps without new data collection—the fact that data can be reused instead of being recollected is the main aim of the directive. However, open data does not mean public data. Open data means that taxpayer-funded research data can be repurposed, reprocessed, and reused—the Digital Music Observatory is an expert on such data science techniques.

The Digital Music Observatory is not an alternative to the future European Music Observatory, just its open source, open data driven voluntary, based on open collaboration with music industry stakeholders, music policy makers and users, academic researchers, individual and citizen scientists, and individual artists. Because we create fully automated research tools that are creating open datasets, and open source software, even our research infrastructure can be fully transferred to a future European Music Observatory[12].

The Central European Music Industry Report 2020 concluded the CEEMID project showing that building meaningful statistical indicators for the live performance, recording and publishing sides of the music industry is possible even in seemingly data poor emerging and future markets. Our Report was presented as a best practice by the European Commission and the Geothe Institute in 2020.

We are not planning to make a competing data observatory. We want to provide a minimum viable model of creating at least a hundred useful indicators—selected from hundreds of indicator-candidates with user feedback— that goes through the unit-testing of data science and computer science, the peer-review of open-source scientific algorithm/software development, the methodological peer-review of science, and eventually user verification from music industry users. This will make sure that the indicators can be reproduced, refreshed, and placed in a future European Music Observatory with at least as good data quality as one would expect from a governmental statistical source or Eurostat.

The aim of the Digital Music Observatory is to maximize transparency by introducing to the policy context of Music Moves Europe, and the wider European music ecosystem the Open Policy Analysis framework. (See Open Collaboration, Open Policy Analysis, and Open Data.)

Open Collaboration, Open Policy Analysis, and Open Data

We aim to maximize transparency by introducing to the policy context of Music Moves Europe, and the wider European music ecosystem the Open Policy Analysis framework. (BITSS 2019; Hoces de la Guardia, Grant, and Miguel 2020b).

Our policy tools will make possible the first, large scale European application of the Open Policy Analysis, which grew out from several initiatives in research transparency, such as the Berkeley Initiative for Transparency in the Social Sciences, the Data Access and Research Transparency (DA-RT) group, the Center for Open Science and their TOP Guideline, the Meta-Research Innovation Center at Stanford University. Globally, the World Bank promotes this framework (Hoces de la Guardia, Grant, and Miguel 2020b; Berkeley Initiative for Transparency in the Social Sciences et al. 2020) and they are fully in line with the Open Science objectives of the European Union (Commission et al. 2020).

Our ambition to increase transparency with introducing the Open Policy Analysis into the European music policies and collaborative data use in the European industry.

We believe that existing data availability is better than that described in the Feasibility study. As stated in this final report, the 2019 Open Data Directive further extended the availability of re-usable public sector information (PSI) with open science data. In both PSI, as open government data, and in open science data, there is a huge potential to fill in the data gaps without new data collection—the fact that data can be reused instead of being recollected is the main aim of the directive. These open data sources are legally open but are not accessible without further investment, and our Consortium wants to make this investment, and produce about 50% of all the data needs of the future European Music Observatory.

It also mentioned the work CEEMID, a bottom-up initiative originally started by three collective management societies, and eventually joined by more than 60 stakeholders in 12 European countries to fill in some of these gaps with a) voluntary data integration among partners b) open data re-processing c) co-financed data collection. Our proposal wants to put the former CEEMID, currently named Digital Music Observatory, a more solid scientific and methodological foundation, to make it more user-friendly, and to exploit state-of-art statistical, data science and computer science methods to provide more comprehensive, timely, and accurate services for the European music sector.

References

Antal, Daniel. 2015. “A Proart zeneipari jelentése. [The Music Industry Report of Proart].” ProArt Szövetség a Szerzői Jogokért Egyesület. http://zeneipar.info/letoltes/proart-zeneipari-jelentes-2015.pdf.

———. 2017. “The Growth of the Hungarian Popular Music Repertoire: Who Creates It And How Does It Find An Audience.” In Made in Hungary, 1st ed. Studies in Popular Music. New York, NY: USA: Routledge.

———. 2019a. “Private Copying in Croatia.” https://www.zamp.hr/uploads/documents/Studija_privatno_kopiranje_u_Hrvatskoj_DA_CEEMID.pdf.

———. 2019b. “Slovak Music Industry Report [Správa o slovenskom hudobnom priemysle].” https://doi.org/10.17605/OSF.IO/V3BE9.

———. 2020. “Central And Eastern European Music Industry Report 2020.” CEEMID, Consolidated Independent. https://doi.org/10.13140/RG.2.2.21450.31686.

Artisjus, HDS, SOZA, and Candole Partners. 2014. “Measuring and Reporting Regional Economic Value Added, National Income and Employment by the Music Industry in a Creative Industries Perspective. Memorandum of Understanding to Create a Regional Music Database to Support Professional National Reporting, Economic Valuation and a Regional Music Study.”

Berkeley Initiative for Transparency in the Social Sciences, Aleksandar Bogdanoski, Carson Christiano, Joel Ferguson, Fernando Hoces de la Guardia, Katherine Hoeberling, Edward Miguel, Emma Ng, and Lars Vilhuber. 2020. “Guide for Accelerating Computational Reproducibility in the Social Sciences.” Berkeley Initiative for Transparency in the Social Sciences. https://bitss.github.io/ACRE/intro.html.

BITSS. 2019. “Guidelines for Open Policy Analysis.” Berkeley Initiative for Transparency in the Social Sciences. http://www.bitss.org/wp-content/uploads/2019/03/OPA-Guidelines.pdf.

Commission, European, Directorate-General for Research, Innovation, L Baker, I Cristea, T Errington, K Jaśko, et al. 2020. Reproducibility of Scientific Results in the EU : Scoping Report. Edited by W Lusoli. Luxembourg: Publications Office of the European Union. https://doi.org/doi/10.2777/341654.

EUR-Lex. 2019. “Directive (EU) 2019/790 of the European Parliament and of the Council of 17 April 2019 on Copyright and Related Rights in the Digital Single Market and Amending Directives 96/9/EC and 2001/29/EC (Text with EEA Relevance.).” Official Journal of the European Union OJ L (130): 92–125. http://data.europa.eu/eli/dir/2019/790/oj.

European Commission, Directorate-General for Education, Youth, Sport and Culture, M Clarke, P Vroonhof, J Snijders, A Le Gall, B Jacquemet, et al. 2020. Feasibility Study for the Establishment of a European Music Observatory : Final Report. Publications Office of the European Union. https://doi.org/doi/10.2766/9691.

Hoces de la Guardia, Fernando, Sean Grant, and Edward Miguel. 2020b. “A framework for open policy analysis.” Science and Public Policy 48 (2): 154–63. https://doi.org/10.1093/scipol/scaa067.

———. 2020a. “A framework for open policy analysis.” Science and Public Policy 48 (2): 154–63. https://doi.org/10.1093/scipol/scaa067.

Osimo, David, Pujol Priego Laya, Turo Pekari, and Ano Sirppiniemi. 2019. “A Symphony, Not a Solo. How Collective Management Organisations Can Embrace Innovation and Drive Data Sharing in the Music Industry.” Teosto. https://www.teosto.fi/app/uploads/2020/10/27134714/a-symphony-not-a-solo-policy-brief-final-09012019.pdf.

state51 Music Group. 2020. “Written Evidence Submitted by The state51 Music Group. Economics of Music Streaming Review. Response to Call for Evidence.” UK Parliament website. https://committees.parliament.uk/writtenevidence/15422/html/.

[1] Feasibility study for the establishment of a European Music Observatory (European Commission et al. 2020).

[2] Feasibility study for the establishment of a European Music Observatory (European Commission et al. 2020, pp31–34).

[3] Our observatory is mentioned to be ablet to participate with “data collection and integration system based on open data, opensources and online surveys” in all pillars of the future European Music Observatory, including Diversity & Circulation (European Commission et al. 2020, p40.)

[4] The original aim of CEEMID was Measuring and Reporting Regional Economic Value Added, National Income and Employment by the Music Industry in a Creative Industries Perspective Three national collective management societies and their consultant signed Memorandum of Understanding about this in 2014 (Artisjus et al. 2014).

[5] Feasibility study for the establishment of a European Music Observatory (European Commission et al. 2020, pp147–148).

[6] A ProArt zeneipari jelentése (Antal 2015) and The Growth of the Hungarian Popular Music Repertoire: Who Creates It And How Does It Find An Audience (Antal 2017); Správa o slovenskom hudobnom priemysle (Antal 2019b), Private copying in Croatia (Antal 2019a), Central Euroepan Music Industry Report (Antal 2020).

[7] See the Policy Brief A Symphony, not a Solo (Osimo et al. 2019).

[8] Antal-Szentirmay: Launching Our Demo Music Obervatory (blogpost)

[9] Written evidence submitted by The state51 Music Group. Economics of music streaming review. Response to call for evidence (state51 Music Group 2020).

[10] The Open Policy Analysis Guidelines which grew out from several initiatives in research transparency, such as the Berkeley Initiative for Transparency in the Social Sciences, the Data Access and Research Transparency (DA-RT) group, the Center for Open Science and their TOP Guideline, the Meta-Research Innovation Center at Stanford University (BITSS 2019; Hoces de la Guardia, Grant, and Miguel 2020a).

[11] In the EU, open data is governed by Directive (EU) 2019/1024 on open data and the re-use of public sector information (EUR-Lex 2019), which replaced the Public Sector Information Directive, also known as the ‘PSI Directive’ which dated from 2003 and was subsequently amended in 2013. Feasibility study for the establishment of a European Music Observatory (European Commission et al. 2020, p38).

[12] In other words, our aim, fully in line with the intentions of the Commission expert who were encouraging us to make this proposal, and with the Feasibility Study, is to create open source scientific/statistical software and pilot open data that can be utilized in 4/4 identified data gaps related to music diverstiy and circulation in the European Music Observatory.

CEEMPC session: Licensing, How To Tackle The Value Gap And A Discussion On Licensing Initiatives

Fri, 29 Sep 2017 22:00:00 +0000

More information in the blogpost.

The Growth of the Hungarian Popular Music Repertoire: Who Creates It And How Does It Find An Audience

Sun, 22 Jan 2017 00:00:00 +0000

Being visible in the world is always difficult in the Central and Eastern European region. Made in Hungary is the first book in the Popular Music Studies series of Routledge from the region. A description of our first datasets, the motivation of the research and the CEEMID concept is laid out as a closing, quantitative chapter in the book.

regions R package to create sub-national statistical indicators

Sun, 01 Jan 2017 00:00:00 +0000

Installation

You can install the development version from GitHub with:

devtools::install_github("rOpenGov/regions")

or the released version from CRAN:

install.packages("devtools")

regions currently takes care of 20,000 sub-divisional boundary changes in Europe since 1999. Comparing departments of France in 2013, with 2007 vojvodinas of Poland and 2018 megyék in Hungary? This extremely errorprone work is automated, as a result, you can compare 110-260 regions for far better analysis. regions was downloaded about 600 researchers in the first month after release.

You can review the complete package documentation on regions.dataobservatory.eu. If you find any problems with the code, please raise an issue on Github. Pull requests are welcome if you agree with the Contributor Code of Conduct

If you use regions in your work, please cite the package.

Motivation

Working with sub-national statistics has many benefits. In policymaking or in social sciences, it is a common practice to compare national statistics, which can be hugely misleading. The United States of America, the Federal Republic of Germany, Slovakia and Luxembourg are all countries, but they differ vastly in size and social homogeneity. Comparing Slovakia and Luxembourg to the federal states or even regions within Germany, or the states of Germany and the United States can provide more adequate insights. Statistically, the similarity of the aggregation level and high number of observations can allow more precise control of model parameters and errors.

The advantages of switching from a national level of the analysis to a sub-national level comes with a huge price in data processing, validation and imputation. The package Regions aims to help this process.

Sub-national Statistics Have Many Challenges

Frequent boundary changes: as opposed to national boundaries, the territorial units, typologies are often change, and this makes the validation and recoding of observation necessary across time. For example, in the European Union, sub-national typologies change about every three years and you have to make sure that you compare the right French region in time, or, if you can make the time-wise comparison at all.
Hierarchical aggregation and special imputation: missingness is very frequent in sub-national statistics, because they are created with a serious time-lag compared to national ones, and because they are often not back-casted after boundary changes. You cannot use standard imputation algorithms because the observations are not similarly aggregated or averaged. Often, the information is seemingly missing, and it is present with an obsolete typology code.

Package functionality

Generic vocabulary translation and joining functions for geographically coded data
Keeping track of the boundary changes within the European Union between 1999-2021
Vocabulary translation and joining functions for standardized European Union statistics
Vocabulary translation for the ISO-3166-2 based Google data and the European Union
Imputation functions from higher aggregation hierarchy levels to lower ones, for example from NUTS1 to NUTS2 or from ISO-3166-1 to ISO-3166-2 (impute down)
Imputation functions from lower hierarchy levels to higher ones (impute up)
Aggregation function from lower hierarchy levels to higher ones, for example from NUTS3 to NUTS1 or from ISO-3166-2 to ISO-3166-1 (aggregate; under development)
Disaggregation functions from higher hierarchy levels to lower ones, again, for example from NUTS1 to NUTS2 or from ISO-3166-1 to ISO-3166-2 (disaggregate; under development)

Vignettes / Articles

Feedback?

Raise and issue on Github or get in touch. Downloaders from CRAN:

Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.

Creating better national cultural statistics with Eurobarometer datasets and ESSNet-Culture technical recommendations

Mon, 09 Nov 2015 00:00:00 +0000

You can download the poster presentation here

or you can zoom into it with prezi.

Hungarian Music Industry Report

Fri, 01 May 2015 00:00:00 +0000

A ProArt zeneipari jelentése from Dániel Antal

Market size of the re-usable public sector information in Hungary

Tue, 15 Dec 2009 00:00:00 +0000

Original title in Hungarian: A közintézmények újrahasznosítható információinak piaca Magyarországon.

Mon, 01 Jan 0001 00:00:00 +0000

Robin Nagy

Mon, 01 Jan 0001 00:00:00 +0000

Robin is a freelancer working on projects using emerging technologies. Main areas of experties are Business Development, Project Management, Innovation, Governance and Corporate culture. Recent interest: Exponential technologies, Exponential business.

CCSI Data Observatory

ESRS in the Film, Television & Music Industry

IDCC24 Lightning Talk Session

Futureproof Film Production Data for Integrated ESG Reporting

Open Productins: Integrating Financial and ESG Reporting for Film

Slide navigation

Questions?

Semantic Enriching of The Historical Photo Collection

Categorizing Cultural & Creative Sectors: An Impossible Task?

Categorizing Cultural & Creative Sectors: An Impossible Task?

Why normal business and occupation categories fail to work in the creative sectors?

Roundtable Discussions on Greening European Film Policy

Open Data Day in The Hague 2023

About Data Observatories

Mission statement

Services

History & Timeline

Governance & Budget

Observatories

Join

Institutional

Private

Data Observatory Labs

Slide navigation

Questions?

Data Observatory Labs

WP 1 Coordination

WP 2 Sustainability

WP 3 Competitiveness

WP 4 Open Data

WP 5 Scaling Up

Potential partners

In 90 secs: conceptual introduction

Music Eviota

Slide navigation

Contents

Introduction

Our approach

Main Features

Connected Financial and Sustainability Reporting

Why Are We Developing Eviota for Music?

Is There A Film Industry Version?

Questions?

LineCheck, Milano, IT

Green Deal, this time for real

Beta test our sustainabilitiy reporting tool

Digital Music Observatory: survive in the music business without a data engineer

Harmonizing surveys on gender (in)equaliaty

Create Datasets that are Easy to Combine and Reuse

Reprex Wins Audience Prize in The Hague Innovators Challenge

Reprex Nominated for The Hague Innovators Award

New Data Curators Wanted

Boost your career without a conflict of interest

FAIR: Findable, Accessible, Interoperable, and Reusable Digital Assets

How to become a data curator?

Find inspiration from other contributors

Why data observatories?

Good to know

Watch Our 2-min Introduction

Vote Reprex :)

OpenMuse Consortium

Big Data for All - Building Collaborative Data Observatories

The event invitation text and links

In the Q&A, we can discuss many things

Check out our projects

Reprex: the impact startup

Big Data for All: Building Collaborative Data Observatories

The event invitation text and links

In the Q&A, we can discuss many things

Check out our projects

Reprex: the impact startup

Listen Local Nederland

Check out our projects

Reprex: Big Data For All

Watch Our 2-min Introduction

How To Vote?

What do we do in the cultural and creative sectors and industries?

Product

Plans in The Hague?

Please Choose Reprex in The Hague Innovators Award Online Vote