A KPI framework to standardize the measurement of a country’s progress in bringing quantum computing into application

Abstract

Quantum computing (QC) is a new and disruptive technology with large economic potential especially in application and downstream value creation stages. Hence, it is important for an economy to understand the current implementation state and to know the ecosystem to support the successful industrial application of this technology. Regularly identifying potential areas of improvement and then defining appropriate actions is necessary to ensure a leading position. Therefore, the Quantum Technology and Application Consortium (QUTAC) has developed a Key Performance Indicator (KPI) framework consisting of 24 KPIs that represent a country’s performance in applying QC. Detailed measurement guidelines and clear data sources ensure transparency of measurement, reproducibility of KPI values and comparability over time. An aggregation method allows summarizing the results of all KPIs. Thus, it is possible to assess the performance of each stakeholder involved and to calculate a single composite indicator that represents the country’s performance. The KPI framework can be adapted to any country and enables the comparison of the performance of different countries. It is a proposal for standardizing the evaluation of QC and its ecosystem on a national level. Thus, strengths and weaknesses can be identified and measurements for improvement derived. The paper highlights the development of the framework, its main features and the application of the framework to Germany. Based on the results, we will discuss the current state of QC application in Germany and make possible suggestions for improvement.

1 Introduction

Quantum computing (QC) is a key technology that has the potential to drive a range of disruptive innovations [1]. Potential is seen in multiple use cases across different industries: From automotive and finance to pharma [2], solving challenges in the fields of engineering and design, material science and production and logistics [3]. In the future, QC can help solve currently seemingly insolvable problems in a wide range of industries [4]. Among other things, QC can contribute significantly to decarbonization and therefore help to reach the goal of a net-zero economy [4]. Broadness of applications results in estimated cost savings and revenue opportunities of more than 450 billion dollars annually [5]. Software and services are expected to account for the majority of value added [6]. By 2030, global value added in the field of QC software will be twice as high as in QC hardware [6], emphasizing the need for an application focus.

Global QC activities are increasing, especially since quantum supremacy has been first postulated by Google in 2019 [7]. Although the problem solved in that case was purely academic and still without practical benefits, and the question of whether quantum supremacy actually exists is still a scientific dispute, ongoing scientific progress is increasing the QC engagement in many countries. In 2021, the volume of global private investments in quantum technology (QT) surpassed 1.4 billion dollars. In the second half of 2021 1.9 billion dollars of public funding was announced [8]. The United States of America (USA), Canada (CA), and the United Kingdom (UK) lead in private investments. In contrast, China has reported the highest government investment for QC, amounting to 10 billion dollars since 2013 [9]. Germany has increased its activities with the goal to become a world leader in quantum systems and to ensure technological sovereignty [10]. In 2021, the federal ministry of education and research published a strategic roadmap [11] and leading German companies founded the Quantum Technology and Application Consortium (QUTAC), a national industry consortium dedicated to the application of QC [3]. In the same year, the German government invested in QC with 2 billion euros, resulting in multiple funded projects [12, 13]. All these initiatives show that many countries are investing in building a QC ecosystem by publishing strategies, increasing funding and starting collaborations. To benefit from the great economic potential of QC and to secure the future of Germany and Europe as a business location, these activities must be pursued consistently. Only then it will be possible to further strengthen Europe’s position in this important technology of the future and catch up to the USA and China [9].

QUTAC has set itself the goal of playing a decisive role in shaping the development of a sovereign QC ecosystem in Germany in order to remain competitive in the future and to strengthen the business landscape in both Germany and Europe. In order to achieve this, it is crucial to accelerate the advancements in the practical implementation of QC. It is therefore recommended to regularly monitor the QC ecosystem and to identify potential areas of improvement [14, 15]. Despite multiple ongoing activities in the field of QC, there is not yet a standardized approach to capture the impact of these activities. Already published key performance indicators (KPIs) [10, 16] focus mainly on QC hardware and related long-term targets. However, as QC is still a new technology when it comes to industrialization, it is crucial to monitor the entire ecosystem’s evolution, rather than solely focusing on hardware advancements and application outcomes.

QUTAC developed a KPI framework including all relevant stakeholders of the QC ecosystem. These stakeholders are government, industry, academia, investors and society. To ensure transparency and comparability with other countries, QUTAC proposes the developed KPI framework as standard to measure a country’s performance in applying QC. Through an aggregation methodology, we can summarize a country’s performance in one score. This enables an easy and transparent comparison of the progress of different countries in applying QC. By comparing the current performance with a target or with other countries, it is possible to identify strength and weaknesses and derive appropriate need for action. The developed KPIs can be used for any country, however detailed measurement guidelines, data sources, target values and thus resulting values are currently only defined for Germany.

We have structured the paper as follows: In Sect. 2, we present the methodology on which the development of the KPI framework is based. Then, in Sect. 3, we explain the resulting KPI framework. In Sect. 4, we apply the KPI framework to Germany (Sect. 4.1), derive potential areas for improvement (Sect. 4.2), indicate the limitations of the KPI framework (Sect. 4.3) and discuss the potential application to other countries (Sect. 4.3). Section 5 concludes the paper.

2 Methodology

The goal of our work is to develop a set of meaningful KPIs that quantify a country’s performance in applying QC. The KPIs need to support all relevant stakeholders in assessing their contribution to building a QC ecosystem and deriving the need for action.

We followed a three-step process to develop the KPI framework (see Fig. 1). At first, we created multiple ideas for potential KPIs through brainstorming. The leading questions were: ‘What are the key characteristics of an ecosystem accelerating the application of QC? What is needed to get there? What challenges will be faced?’. Brainstorming was supported by expert interviews and literature research, including QC reports and articles on existing composite indicators in related areas. The result of this first phase of the three-step process was a list of all relevant stakeholders and drivers of a successful QC ecosystem (see Sect. 3.1) and more than 60 KPI ideas. Secondly, we selected the final KPIs. The selection of the final KPIs was based on the criteria simplicity, measurability, relevance, relatability, actionability and uniqueness (see Sect. 3.2). The selected 24 KPIs are included in the final framework. In the third and final phase, we examined all remaining 24 KPIs in detail (see Sect. 3.3). We identified corresponding stakeholders and drivers for each KPI. In addition, we have set a target value for each KPI and defined a clear data source and measurement guideline. For the final aggregation of all 24 KPIs into one score, we evaluated the impact and data quality of each KPI. Experts accompanied all steps of this phase.

Figure 1

Development process of the KPI framework

All experts involved in phase 1 and 3 have knowledge in QC, similar high-tech innovations or performance measurement. They come from either the public sector, the private sector, academia or industry (see Acknowledgements).

3 Resulting KPI framework

In this section we describe the KPI framework, developed according to the three phases explained in Sect. 2. Section 3.1 outlines the generation of KPI ideas and the general structure of the framework, which is based on the two dimensions stakeholders and drivers. We then defined and applied criteria to the KPI ideas to determine the final KPIs of the framework (Sect. 3.2). To evaluate and act upon these KPIs, we defined data sources, target values, weights, and an aggregation methodology (Sect. 3.3).

3.1 General structure of the framework: stakeholder and driver

Based on the interviews conducted, the following aspects are necessary to build a holistic QC ecosystem:

• Startups to accelerate innovation

• Financial support by investors and government

• Collaboration within academia, within industry and between industry and academia

• Development of QC workforce and prevention of brain-drain

• Exploration of use cases across industries

• Clear communication and expectation management

Several QC reports similarly emphasize the development of startups, public and private investments and application potentials across different industries [2, 15, 17]. Furthermore, the national strategy papers of the USA [18] and Canada [19] highlight the importance of (a) education and training to increase the potential future workforce, (b) large investments to build a strong ecosystem, (c) domestic and international cooperation and collaboration, (d) focusing on application and (e) ensuring security. Other indices assessing the performance of new technologies at a national level, such as the Global AI Index [20], consist of KPIs measuring similar aspects: Talent, infrastructure, startup activity, investments, governmental commitment, and research and development.

To structure these insights, we defined two dimensions to build the framework: Stakeholders and drivers. Stakeholders are those who primarily influence progress in applying QC. Drivers explain how the respective stakeholders can contribute. Based on our expert interviews and literature research we identified five stakeholders and eight drivers of importance for the application of QC (see Table 1). We allocated all 60 KPI ideas that resulted from the interviews and further research to these two categories, whereby one KPI can belong to multiple stakeholders and multiple drivers.

Table 1 Stakeholder and driver to structure the KPI framework

3.2 The 24 KPIs of the framework

We defined several criteria to select the right KPIs for the framework out of the long list of generated KPI ideas. A KPI of high quality has to fulfill several criteria [20–24], which are summarized in Fig. 2 and explained in the following:

• Simple: All KPIs are easy understandable and applicable through their naming, description, and measurement instructions.

• Measurable: All KPIs have a precisely defined measurement guideline. The precise definition supports the interpretability of the KPI and keeps the manipulability of the KPI low. Accessibility to the required data is a prerequisite and is guaranteed to an audience as broad as possible in order to achieve high transparency.

• Relevant: All KPIs serve the objective of the QC KPI framework, to contribute to representing a country’s performance in applying QC. We conducted expert interviews to assess the relevance of the respective KPIs.

• Relatable: All KPIs are relatable through their allocation to stakeholders (see Table 1 and Table 2). If targets are not met, it is clear who is the responsible party to approach for improvement.

  Table 2 KPIs of the QC KPI framework

• Actionable: All KPIs are actionable through their relatability to stakeholders.

• Unique: All KPIs are unique, as they capture different aspects of the QC ecosystem. We discussed correlation and similarity between KPIs with experts when choosing the final KPIs.

Figure 2

Criteria for high quality KPIs

We identified 24 KPIs that meet all of the above criteria. The final KPIs are summarized in Table 2, including a short description, the corresponding stakeholder and driver and the intention of the KPI, explaining why it is relevant.

We decided to normalize the KPIs by gross domestic product (GDP) or working age population (WAP), in case the KPI depends on the country’s economic strength or population. We thus focus our assessment on what a country should be able to achieve. For example, if a country is assessing its availability of people with QC skills, the values normalized by WAP provide a better measure for comparison with other countries than the non-normalized values. However, if the KPI-framework is used to identify the absolute best performing country or to rank countries by their absolute performance, KPI values should not be normalized by GDP or WAP. For some KPIs we assume no correlation to GDP or WAP. Those are not normalized and include ratios (KPIs: Industry representation; National participation; Workforce attrition; QC excellence clusters; Collaboration funding; High-school curriculum), durations (KPIs: Hiring duration; Political support horizon) qualitative KPIs (KPI: Society’s sentiment) and the KPIs ‘Reach of associations’ and ‘National strategy’.

In Fig. 3, we illustrated the allocation of all KPIs to their stakeholders and drivers. Some KPIs have been assigned to more than one stakeholder and driver. It can be seen that the KPIs cover all identified stakeholders and drivers (see Sect. 3.1) that are important in the QC ecosystem. Furthermore, Fig. 3 highlights the main drivers for each stakeholder. For example, government contributes to the progress of applying QC primarily by providing strategy, funding, education and enabling collaboration. The application focus of the framework becomes clear when looking at the driver “hardware” that is only covered by industry. This indicates that we did not develop specific KPIs related to hardware development, as otherwise industry, but especially academia, would be increasingly involved in the KPI framework. We did not leave out the driver hardware completely, as some KPIs (QC companies; QC startups) cover hardware and software equally. Furthermore, this leaves open the possibility of adding additional hardware focused KPIs in future.

Figure 3

Coverage of stakeholders and drivers by the 24 KPIs of the QC KPI framework. Please note that some KPIs have been assigned to more than one stakeholder and driver

3.3 Definition of data sources

To meet the criterion measurability (see Sect. 3.2), it is necessary to define explicit data sources for each KPI. Explicit data sources ensure transparency and reproducibility of the KPIs. For the determination of the data sources we have defined the following criteria:

• Relevancy: The data source needs to fit the intention of the KPI.

• High accessibility: Publicly accessible data sources are preferred. If there is no publicly accessible data source, software commonly used by companies in Germany is selected as the data source.

• Durability: Only those data sources are considered that experts estimate will be available in at least the next three years.

Data sources should not be changed over time to ensure comparability of KPI values, which were collected at different points in time, and to enable measurement of progress.

3.4 Aggregation methodology

In order to condense the information, we have used an aggregation method to combine all KPIs into a single composite indicator. This simplifies the interpretation of the KPIs and helps to assess a country’s progress over time [26]. This is the key for further discussions and the derivation of needs for action. In the following, we introduce the aggregation method used, which consists of three steps: Normalization with defined targets values, weighting according to impact and data quality and calculation of the composite indicators (see Fig. 4).

Figure 4

Aggregation method

Step 1: Normalization with defined target values

Normalization is required because the KPIs have different measurement units [26]. As we normalize the KPIs by using target values as reference, we explain how to set targets for each KPI before introducing the normalization formulas. Targets ensure that KPIs can be acted upon: Deviations from the target value can be analyzed in order to derive the need for action. If the framework is used to compare the performance of different countries, all target values should be set at the value of the best performing country. In this way, it is possible to determine which stakeholders or drivers have potential for improvement compared to other countries. If data for other countries is either not available, resource-intensive to gather or a country wants to set itself individual targets, target values can be set independently from the performance of other countries.

We identified the target values for Germany in three different ways, based on:

• Recommendations from experts (REX)

• Comparison with achievements in the field of artificial intelligence (CAI)

• Comparison with achievements in the field of QC with other countries (CC)

Countries that are considered as benchmark are USA, CA, Japan (JP) and UK, as they are currently leading in the field of QC [17]. Therefore, their performance is taken into consideration when determining target values based on CC. China, as one of the leading countries in QC, is not included as comparison country because China’s activities and progress in the field of QC is not disclosed transparently.

To normalize the KPIs we have chosen the ‘distance from the group leader’ [21] normalization method, since the KPI framework should clearly describe the deviation of the actual state from the defined target state. For the QC KPI framework, the ‘group leader’ is the defined target value. If the defined target is achieved, the normalized KPI has a value of 1. Depending on the target type, we applied different formulas, as shown in Table 3, to calculate the normalized KPI value.

Table 3 Normalization formulas (CV = Current KPI value; TV = Target KPI value)

Step 2: Weighting

We have weighted all KPIs according to two aspects: Their ‘impact’ $w_{i}$ on the progress of applying QC and the ‘data quality’ $w_{d}$ of their underlying data source. We have chosen those two weighting factors to reflect the contribution of each KPI to an overall composite indicator and the quality of the underlying data source. KPI weights are also used for aggregation to the Global AI Index [20] and proposed by M. Freudenberg [21]. Both weighting values, ‘impact’ and ‘data quality’, are set to values between 1 and 3, whereby 1 indicates low and 3 high impact or data quality. We have set clear criteria for the definition of the weighting factors, to ensure transparency, comparability and to reduce bias (see Table 4). The criteria for ‘impact’ resulted from discussions with experts. To assess ‘data quality’, we used the 16 dimensions of data quality by L. L. Pipino et al. [27] as a basis. The dimensions relevancy, accessibility, understandability and interpretability are already a prerequisite for the choice of the data source (see Sect. 3.3: Definition of data sources). We condensed the remaining 12 dimensions to four: Believability (including the aspects of ease of manipulation, free-of-error, objectivity, reputation and security), completeness, timeliness, and value-added (including the aspects of concise and consistent representation because a lack in conciseness and consistency reduces the added value of the KPI). Amount of data was not considered, as for our use it is relevant only that the data source is complete.

Table 4 Criteria to define the values of the weighting factors

After normalizing and defining weighting factors, the KPIs are multiplied by the sum of their weights, as shown in formula (1). This is how we take into account the respective ‘impact’ and ‘data quality’ values of each KPI.

$$ \mathrm{KPI}_{\mathrm{norm}, \mathrm{weighted}} = \mathrm{KPI}_{\mathrm{norm}} * ( w_{i} +w_{d} ) $$

(1)

Step 3: Calculation of composite indicators

We determine three different types of composite indicators: Stakeholder scores (SH_score), driver scores (D_score) and the national score. The stakeholder score represents a stakeholder’s performance in applying QC. The driver score shows the strength of a country in certain activities contributing to the application of QC. The national score is the overall composite indicator that reflects the country’s performance in applying QC. A stakeholder score is calculated by adding all weighted KPIs assigned to one stakeholder, divided by the sum of the weights of these KPIs (see formula (2)).

$$ \mathrm{SH}_{\mathrm{score}} = \frac{\sum_{n=1}^{\text{Number of KPIs per SH}} \mathrm{KPI}_{\mathrm{norm}, \mathrm{weighted}, n}^{\mathrm{SH}}}{\sum_{n=1}^{\text{Number of KPIs per SH}} ( w_{i_{n}} +w_{d_{n}} )} $$

(2)

In the same way, the driver score is calculated for each driver:

$$ D_{\mathrm{score}} = \frac{\sum_{n=1}^{\text{Number of KPIs per D}} \mathrm{KPI}_{\mathrm{norm}, \mathrm{weighted}, n}^{D}}{\sum_{n=1}^{\text{Number of KPIs per D}} ( w_{i_{n}} +w_{d_{n}} )} $$

(3)

For the calculation of the national score, we consider all 24 KPIs of the framework (see formula (4)). If KPIs are allocated to multiple stakeholder or driver, they are only considered once in the calculation of the national score to avoid a higher weighting of individual KPIs.

$$ \text{National score}= \frac{\sum_{n=1}^{\text{Number of KPIs}=24} \mathrm{KPI}_{\mathrm{norm}, \mathrm{weighted}, n}}{\sum_{n=1}^{\text{Number of KPIs}=24} ( w_{i_{n}} +w_{d_{n}} )} $$

(4)

4 Application to Germany: results and discussion

In Sect. 4, we apply the developed KPI framework to Germany. Based on the defined data sources, target values and weighting factors (see Table 5), we determined the KPIs for Germany in October 2022 and calculated the composite indicators (see Sect. 4.1). In Sect. 4.2, we discuss the resulting values and give recommendations for potential improvements. In addition, we indicate the limitations of the framework and provide an outlook on a possible improvement of the framework by extending it with additional KPIs (Sect. 4.3). We conclude Sect. 4 by pointing out what to consider when applying the framework to other countries (Sect. 4.4).

Table 5 Definitions for Germany and evaluation of KPIs for Germany

4.1 Resulting KPI values and composite indicators for Germany

For the KPI evaluation, data sources, target values, and weighting factors need to be defined. For Germany, we provide these values for each KPI in Table 5. Since the weights have to be set qualitatively, we discussed their definitions and set their final values in collaboration with experts. For other countries, ‘data quality’ weighting factors would need to be reassessed if other data sources were used. For Germany, we measured all 24 KPIs in October 2022 and summarized the values in Table 5. Figure 5 gives an overview about the normalized KPIs.

Figure 5

Normalized KPI values for Germany based on a KPI measurement in October 2022

The aggregation of all KPI values of Germany, based on a measurement performed in October 2022 (see Table 5), results in a national score of 0.54 (range from 0 (worst) to 1 (best)). This value represents the performance of Germany in bringing QC into application. The aggregation of the KPIs to stakeholder scores (see Fig. 6) provides additional information about the performance of the individual stakeholders. The leading stakeholder is government with a score of 0.60. Investors have greatest potential for improvement, with the lowest score of 0.28. In between there are academia with a score of 0.54, and society and industry, both with a score of 0.49. We discuss the values in the Sect. 4.2.

Figure 6

Stakeholder scores for Germany based on a KPI measurement in October 2022

The driver scores shown in Fig. 7 indicate which activities need more attention in Germany to build a balanced QC ecosystem and accelerate bringing QC into application. Innovation with the lowest score of 0.15 needs special attention. Hardware yield in the second lowest score at 0.22, but interpretation must take into account the limitations of the framework presented (see Sect. 4.3). Further improvement potential is identified for software with a score of 0.27 and education with a score of 0.35. For funding the driver score results in 0.51. The best performance in Germany is seen in the field of collaboration, workforce and strategy. For collaboration and workforce the driver score results in 0.67. With 0.75 the highest driver score in Germany is achieved for strategic activities. Driver scores will also be discussed in Sect. 4.2.

Figure 7

Driver scores for Germany based on a KPI measurement in October 2022

4.2 Discussion of improvement potentials for Germany

By analyzing the results of the KPI framework applied to Germany in October 2022 (see individual KPI values in Table 5; normalized KPI values in Fig. 5; aggregated results of stakeholders in Fig. 6 and drivers in Fig. 7), we are able to identify the strengths of the German QC ecosystem and potential for improvement. The results are summarized in Table 6 and explained in the following.

Table 6 Positive contributions to the German QC ecosystem and potential for improvement by stakeholder based on calculated KPI values

Government is making a positive contribution to the German QC ecosystem. Publications about the strategic direction [10], as well as the financial support for QC with 2 billion euros [12] are important measures to bring QC into application. We see potential for improvement in the strategic alignment of the different ministries, as so far, no strategic document on QT topics was published by the national government or in collaboration between different ministries. Moreover, it could be beneficial if the share of funding for collaborative projects involving industry were increased to ensure that the funded projects are application-oriented. Based on information provided by the German federal ministry for education and research (BMBF) on ongoing projects [41], 52% of BMBF funding is currently awarded to QT projects involving at least one partner from industry.

For academia and government an aspect to focus on is the education of future employees. Both, universities and universities of applied sciences should establish QC degree programs to focus on the fundamentals and the application of this new technology. We assume that this measure would increase the number of people with QC skills available for the job market and thus lead to shorter times to fill vacancies. Additionally, fundamentals of quantum physics and quantum mechanics should be integrated into the high school curriculum, to provide a basic understanding of statistics, probabilities and the physics behind QC. We see the responsibility for adapting the education system with academia and government.

In all areas of the QC ecosystem where academia is the only stakeholder, academia is performing well: There are already more than 1800 scientists with an H-index above 40 in Mathematics, Computer Science, Chemistry, Electronics and Electrical Engineering, Engineering and Technology and Material Science in Germany available [37]. This number of scientists in not dedicated to quantum related research as the used data source does not provide the necessary details for a quantum specific evaluation. However, high-quality and successful research in STEM field potentially attracts quantum researchers, which is why we consider this KPI as relevant for the framework. Still, normalized by WAP, there are less scientists with an H-index above 40 in Germany than in CA and UK. Academia plays a crucial role in the development of hardware. When evaluating the stakeholder academia, it should be noted that the progress of hardware development is not reflected in the presented KPI framework (see Sect. 4.3).

High quality education in QC will lead to a larger QC community. This combined with more attractive job opportunities will prevent people with QC experience to move abroad. To offer attractive job opportunities for QC talents, industry needs to be involved. A screening of the QC market shows that in Germany there are less QC companies than in USA, CA and UK. This applies both to the absolute number of companies and to the number of companies per GDP according to Capital IQ data. In terms of the number of startups per GDP as reported by Crunchbase, CA and UK perform better than Germany.

One influencing factor is that, according to Crunchbase, fewer business angels, venture capital funds and private equity firms invest in German companies than in American or Canadian companies. Germany should focus on supporting companies in developing QC products and services as well as using QC to improve products and services. As QC is a new and disruptive technology, the commitment of larger companies is just as important as the support of startups. We see the responsibility for the commitment and the support with industry, investors and government.

Society already shows interest and engagement in QC topics, as, for example, activity in Meetup groups and contributions via GitHub are increasing. However, compared to CA, not even a third of the activity is seen in Germany.

The collaboration of German companies in QC topics is expandable, as only 31% of the companies in the German stock market index DAX are represented in QUTAC or QuIC, the two most important industry consortia for QT topics in Germany and Europe. The influence of existing QC associations is clearly visible, as the number of LinkedIn followers of associations such as Munich Quantum Valley, Munich Center for Quantum Science and Technology and Quantum Business Network has increased by more than 15% in the last year.

The lack of innovation activity that can be derived from the analysis of the driver value for innovation, reflect the fact that Germany has only 15 % of the number of patents per WAP of CA. To achieve a leading position in QC, more attention should be paid to securing innovations.

4.3 Limitations of the framework

Various limitations of the QC KPI framework were identified both in discussions with experts and in the application of the framework to Germany. These include limitations in the field of hardware and missing KPIs due to lack of accessible and reliable data sources. In addition, we discuss limitations in the interpretation of the KPIs.

No focus on hardware

We developed the QC KPI framework with a focus on bringing QC into application. Therefore, we did not focus on quantifying the progress and performance of the QC hardware. The driver hardware is included in the framework, because screening of QC companies is part of the framework and some of the companies are active in QC software and hardware. When interpreting the results of the driver hardware, it has to be taken into account that important aspects such as the performance and accessibility of QC hardware are not represented in the KPIs. It is possible to extend the KPI framework in the future with additional KPIs such that the progress in QC hardware development is also covered.

Missing KPIs due to lack of accessible and reliable data sources

One of the greatest challenges in developing the KPIs was defining accessible and reliable data sources to ensure transparency and comparability of the measurements. For some relevant topics, it was not possible to define accessible and reliable data sources and thus measurable KPIs. We have summarized those KPIs in Table 7. As soon as it is possible to access reliable data source for the listed KPIs, they will be included in the KPI framework.

Table 7 Missing KPIs in the QC KPI framework due to lack of accessible and reliable data sources

Interpretation of KPIs

We developed the QC KPI framework with the intention of showing potential improvements for the QC ecosystem of a specific country. The structure of the framework by stakeholders and drivers clearly shows who can make a positive contribution to the QC ecosystem through which activities. Since many stakeholders and activities influence each other, the results achieved must always be interpreted in the overall context of the specific country. Additionally, it is crucial to consider the limitations of the framework as well as the ‘data quality’ and ‘impact’ of the individual KPIs when deriving need for action.

The defined targets of the individual KPIs should not be seen as absolute. They set the direction and relate the current to an optimal state. However, optimizing individual KPIs has only limited success – the entire ecosystem has to make progress. This is stated by Goodhart’s Law: ‘When a measure becomes a target, it ceases to be a good measure.’ [51]

4.4 Application to other countries

It is possible to apply the QC KPI framework to countries other than Germany. So far, we have only evaluated 11 of the 24 KPIs (see Table 5). For other countries as for those, the data sources used provided the necessary data not only for Germany but also for the analysis of the comparison countries. For the evaluation of the remaining KPIs and the subsequent calculation of the composite indicators, for countries other than Germany, explicit data sources would still need to be defined, as was done for Germany. The data sources should meet the criteria mentioned in Sect. 3.3: Relevancy, high accessibility, and durability. Furthermore, it is necessary to provide a detailed measurement guide to ensure transparency and comparability over time. If the data sources chosen differ from those chosen in Germany, the weighting factor for ‘data quality’ has to be reassessed.

The target values of the KPIs set for Germany can also be used for other countries as they are normalized by GDP and WAP. By setting all target values to the KPI value of the best performing country, it would be possible to analyze the differences between the various countries. The strengths of each country could be identified and give insights into which country’s activities could be applied to improve the overall QC ecosystem. The QC KPI framework presented in this paper would add additional value if shared among several countries.

5 Conclusion

KPIs are an important tool for quantifying progress and deriving the need for action. Several individual KPIs have been published for QC and its ecosystem by different organizations in recent years. Many of them contain only very limited information on how to measure the KPIs and cover only parts of the whole QC ecosystem. This leads to limitations in the comparability of the measurements and actionability of the KPIs.

Therefore, we developed the presented QC KPI framework. It provides 24 clearly defined KPIs to quantify QC and its ecosystem on a national level. It allows to capture the current state of QC and its ecosystem and helps to derive need for action to bring QC into application. Since data sources are defined for each KPI, it ensures comparable measurement over time and high transparency. If the KPIs are used as a standard for evaluating country performance, there is also comparability between different countries. This would allow direct comparison between multiple countries to identify strengths and weaknesses. Thus, countries could use the QC KPI framework to analyze how they can improve their QC ecosystem and accelerate bringing QC into application.

Through the application of the framework for Germany, we identified improvement potential for all stakeholders. Special attention should be paid on the encouragement of investors to fund QC startups in Germany, the building of quantum workforce through education, the provision of a coordinated strategy by all ministries involved in QT topics and the participation in QC collaborations to strengthen knowledge exchange. For Germany, we will apply the framework on a regular basis, to track progress, reflect on ongoing activities, react to changes and build on identified improvement potential.