Background
Taking the experience and lessons learned from more than two decades of ocean energy technology and market development into account, a detailed monitoring of progress and success should have the following characteristics:
- It is necessary to differentiate between the various needs of the development stages from R&D, Prototype, Demonstration, to Pre-Commercial and Industrial Roll-out;
- Criteria need to be defined which are specific to each development stage;
- A connection must be made between the performance criteria and the availability of certain types of support in the form of public and private funding;
- The process should use continued feasibility-checks on the OE technology potential with an increasing focus on LCOE as the technology matures.
After this initial period of focussing on the technological feasibility (Can it be done? Does it work?), where the only metric used was the successful technology evolution to higher TRL levels, economics and other social acceptance criteria have been identified that need to be considered at an early development stage for ocean energy technology to be commercially successful.
The current situation is characterised by an ongoing international collaboration on the development of ocean energy technology and the management of the various stage-gated technology development programmes run by funders around the world. This has led to an ongoing need to develop a process for defining appropriate and rigorous metrics for measuring success in a number of critical target areas of ocean energy technology development.
Furthermore, associated numerical success thresholds must be identified for each metric which can be applied at each stage of development as stage gate success thresholds. The objective of these activities is to build clarity, information and understanding to support the definition of a fully defined set of metrics and success thresholds. These “stage gate measurements of success” in interrelated topic areas are linked to a top-level optimisation of Levelised Cost of Energy (LCOE) – the key metric into which all others inevitably feed. This provides a significant international challenge since a final definition of metrics and success thresholds will, to some extent, be specific to a technology area and perhaps the type of market targeted.
OBJECTIVES
Performance Metrics (also known as Task 12) addresses the ongoing need to define appropriate and rigorous metrics for measuring success in a number of critical target areas of ocean energy technology development. The objective is to establish a common international stage gate metrics framework to be used by technology developers, investors and funders.
An internationally accepted approach provides device developers, national and international funding organisations, and the development community the following benefits:
- The ability to measure technology development progress and success;
- A methodology to assist in the management of competitive innovation calls that can compare the viability of competing technologies;
- An approach for ensuring appropriate allocation of funding to the most promising technologies;
- A set of metrics to measure technology progress to illustrate the impact of funding;
- An internationally accepted and credible marker of success to aid in building technology confidence in investors and other stakeholders;
- The ability to make cross technology funding comparisons to help avoid replication or repetition of funding of technologies by numerous funders with similar objectives;
- Decision making assistance for private and public funders.
International collaboration in the development of tools such as metrics creates value through the facilitation of cross-funder comparisons and benchmarking. Common agreement of the method for assessing technologies and the associated success thresholds allows various funding organisations to learn from others, compare programmes, and avoid repetition and replication. This way the international collaboration encourages collaboration among developers and funders, and fosters standardisation of processes and designs.
The result over time would be to publish an international framework which can be applied by all relevant stakeholders.
When completed, the framework should cover the following topics and achieve the results described in item 5 below:
- Briefly describe the existing metrics and their application, benefits, and shortfalls;
- Explain the process of defining a target development outcome, using stages, metrics and applying thresholds to identify stage gate completion and success;
- Identify the different levels of detail needed to successfully use the stage gate metrics;
- Identify the stage gate metrics needed to align a device developer’s road map and business plan with the strategy of an investor and that of the technology, market and energy policy targets of public funding programmes;
- Identify and make available a methodology for evaluating and benchmarking OE technology concepts using a series of development stage gates and related evaluation metrics with the relevant thresholds needed to verify stage gate completion for use by the international community in developing commercially viable ocean energy technologies. This methodology, stage gates and metrics may already exist for some approaches such as “the structured innovation method” being developed by NREL and Sandia Labs for the U.S. DoE which is an advanced tool for guiding innovative development of new concepts.
A report was published discussing the benefits of common evaluation approaches in the ocean energy sector and the use of common language to help build consensus. It considers how the Evaluation Criteria and, more commonly, the Evaluation Process changes throughout the technology development process. The technology development process has been split into three parts, which are closely associated with the Technology Readiness Level scale:
- Early-stage - Low level of detail – represents technology before any detailed sub-system, device, or array numerical modelling;
- Mid-stage – Medium level of detail – represents technology measured by numerical modelling or tank testing;
- Late stage – High level of detail – represents technology measured by open water testing at specified sites.
A technology and its application can be assessed from different perspectives: system effectiveness, process efficiency, technology performance, cost effectiveness and commercial attractiveness. These perspectives can be broken down into various Evaluation Areas. These Evaluation Areas are categorised as measures of “Technology Performance” or “Process Efficiency”, which can be combined into an evaluation of “System Effectiveness”.
For each Evaluation Area, a set of Evaluation Criteria for Early, Mid and Late stage in the technology development process has been developed.
The Framework document represents a certain amount of international consensus. It has been the subject of significant stakeholder engagement with the IEA-OES contracting parties and all key user groups. However, in order to expand consensus and deliver its full value, the framework recommendations need wider uptake by the target user groups as well as other key stakeholders such as test centres and consenting bodies. The group expected to be instrumental in driving wider uptake of the Framework is the public funders, whose application of the recommendations in public funding schemes would automatically drive uptake by applicant technology developers. However, to ensure this alignment between funders and developers, and to achieve a seamless transfer of technology developers from public funding schemes to compliance with standards, certification and the expectations of private investors, engagement with all users will be required.
The next steps are expected to deliver the following results:
- Immediate Framework improvement requirements identified and delivered, as required to allow wider uptake by public funders;
- Increased adoption by public funders achieved and foundations built for wider uptake by other target users;
- Further Framework improvement needs clarified, prioritised and delivered, including inclusion of sustainability considerations;
- Framework integrated with the work of standards and certification institutions and the concept of a ‘Technology Passport’ developed;
- Future Framework improvement and hosting arrangements defined.
