Energy Efficiency Program Evaluation Guides by National Action Plan for Energy Efficiency and DOE/EERE

Other Evaluation Guides

Other Evaluation Resources
   Background Materials
   Papers and Reports
   Databases

Acknowledgements

There are several documents available that explain energy efficiency program evaluation and provide guidance on how to conduct impact evaluations of efficiency programs in the United States. The goal of this "Guide to the Guides" is to clarify the relationships among the documents. It focuses on documents with a national scope, particularly the DOE/EERE's Impact Evaluation Framework for Technology Deployment Programs and the National Action Plan for Energy Efficiency's Model Energy Efficiency Program Impact Evaluation Guide, two important national-level documents about energy efficiency program impact evaluation that were released in 2007. This "Guide to the Guides" also identifies other evaluation guides written for particular regions or states that may have broader implications, and other useful documents and resources related to evaluation.

The Impact Evaluation Framework for Technology Deployment Programs and the Model Energy Efficiency Program Impact Evaluation Guideare complementary in their approaches and scope, and together provide a fairly comprehensive overview of impact evaluation for all types of program approaches.

The Impact Evaluation Framework for Technology Deployment Programs uses the term "technology deployment programs" instead of "energy efficiency programs." This can be confusing to anyone focusing on the administration or evaluation of ratepayer-funded efficiency programs. This term is used because the document is meant to provide guidance on evaluating clean energy programs and renewables programs as well as efficiency programs. For efficiency program evaluation purposes, however, this language can simply be interpreted as "energy efficiency programs."

The Impact Evaluation Framework for Technology Deployment Programs is based on the premise that "identifying the linkages between outputs and outcomes"—that is, how what the program does (its activities or "outputs") are translated by partners and target audiences into actions that produce a variety of impacts or "outcomes," including but not limited to energy and demand impacts—"is one of the most critical and most difficult problems in program evaluation." ^[1] To help managers and evaluators address this problem, it provides specific tools to use in identifying the linkages between program activities or outputs and the resulting impacts or outcomes. Identifying these linkages helps to clarify and prioritize what should be measured in the evaluation, thus enabling evaluators to apply with greater effectiveness the more technically oriented measurement and analysis tools presented in guides such as the Model Energy Efficiency Program Impact Evaluation Guide. The identification of outputs, outcomes, and the linkages among them also helps to separate program-induced impacts from the same effects that may be generated by other factors. The ability to separate program-induced impacts from other factors will become increasingly important as more players enter the field in which ratepayer funded energy efficiency programs used to play alone, offering messages, programs, or services designed to reduce energy use for a variety of different reasons.

The Model Energy Efficiency Program Impact Evaluation Guide provides technical guidance for calculating energy and demand savings and avoided emissions from energy efficiency programs via a set of practical processes and methodologies. It focuses on evaluation specifically for program approaches relying primarily on direct energy savings. It lays out clearly the steps involved in selecting the appropriate measurement and analysis approach for the program and evaluation goals. This includes, but may not be limited to, the use of billing analysis, deemed savings, and project- or facility-level data collection, monitoring and analysis (M&V). It also provides important context and background for implementing the International Performance Measurement and Verification Protocol (IPMVP) as part of evaluation. It provides some basic approaches to including limited market effects measurement in impact evaluation for the calculation of net savings. It refers users to the Impact Evaluation Framework for Technology Deployment Programs for more extensive treatment of market effects evaluation, and for the evaluation of programs relying mostly or exclusively on indirect effects, such as market transformation programs and education or training programs.

All programs are carried out in a larger structural context—that is, in the real world of society, the economy, and markets. Off the drawing board and in the real world, many factors unrelated to the program and beyond the control of program administrators can affect program outcomes. The technical guidance offered in Model Energy Efficiency Program Impact Evaluation Guide is likely to result in a more robust evaluation when implemented in the context of a clear understanding of the linkages between program activities and outcomes as well as the other factors that could affect energy savings, demand savings, and other potential program outcomes. Whenever possible, program managers and evaluators should strive to develop and communicate a clear understanding of these factors and linkages as part of program and evaluation planning.

Other Evaluation Guides

International Performance Measurement and Verification Protocol (IPMVP)

The IPMVP, a product of the Efficiency Valuation Organization, is a set of framework documents used to develop strategies and plans for quantifying energy and water savings at the project level—that is, in individual facilities or groups of facilities—for retrofits and new construction. This document is referred to Model Energy-Efficiency Program Impact Evaluation Guide described above, which explains the relevance of the IPMVP to efficiency programs and when to use each of its parts. The IPMVP describes appropriate approaches to selecting facilities for measurement and verification (M&V); measuring and verifying equipment installation and usage; monitoring indoor environmental quality under different circumstances; and addressing how to quantify avoided emissions from facilities and projects.

National Action Plan for Energy Efficiency

The National Action Plan for Energy Efficiency is a joint effort of the U.S. Environmental Protection Agency, U.S. Department of Energy, and more than 80 energy, environmental, and other organizations. Its goal to create a sustainable, aggressive national commitment to energy efficiency through gas and electric utilities, utility regulators, and partner organizations. Action Plan participants are identifying key barriers limiting greater U.S. investment in energy efficiency, and developing and documenting sound business practices for removing these barriers. The Action Plan's initial report identifies key barriers to limiting greater investment in energy efficiency and reviews sound business practices for removing these barriers, detailing findings and outlining recommendations and options for overcoming them. In addition to the Model Energy Efficiency Program Impact Evaluation Guide discussed above, the Action Plan is producing a series of documents to help guide regulators and program administrators in planning for and evaluating energy efficiency as a resource. These include two documents useful for planning energy-efficiency portfolios and programs, the Guide for Conducting Energy-Efficiency Potential Studies and Guide to Resource Planning with Energy Efficiency. For a comprehensive list of Action Plan guides and papers, go to http://www.epa.gov/cleanenergy/energy-programs/napee/resources/guides.html.

U.S. DOE Evaluation Publications

In addition to the Impact Evaluation Framework for Technology Deployment Programs described above, there is a twelve page overview and an application example designed to help in learning how to implement the approach described in this document. Also on the DOE website are:

• Overview of Evaluation Methods for R&D Programs. This booklet introduces managers to a variety of methods for evaluating R&D programs. Related to this are A Toolkit for Evaluating Public R&D Investment Models, Methods, and Findings from ATP's First Decade and a patent-searching database.
  
• EERE Guide for Managing General Program Evaluation Studies. This Guide focuses on the management and use of general program evaluation studies performed by outside experts and contractors. In addition to outcome and impact evaluation, the guide addresses process, cost-benefit evaluation, and market assessment evaluation.
  
• EERE Peer Review Guide. The peer review guide describes steps to plan, design, and implement external peer reviews.

The Evaluation Protocols are the official document that evaluation professionals are to rely upon when planning and conducting their evaluation efforts for California programs.

Ontario Power Authority Evaluation Framework & Protocols

The Ontario Power Authority (OPA) is in the process of finalizing an evaluation framework, protocols, and other related guidance documents for conducting energy efficiency program evaluation of OPA programs. The evaluation framework is to provide the background for a consistent and systematic methodology for tracking, reporting and evaluating the impacts and effectiveness of Conservation programs funded by the OPA.

New England Forward Capacity Market M&V Manual

As part of the newly formed wholesale capacity market in New England, or Forward Capacity Market (FCM), demand resources, including energy efficiency, load management, DG, real-time demand response, and real-time emergency generation, can compete alongside supply resources in the annual auctions. The New England Independent System Operator (ISO-NE) has developed an M&V Manual for demand resources which sets forth the standards that demand resource providers must meet in order to qualify their resources and receive capacity payments during the commitment period.

California Demand Response Evaluation

The California Public Utilities Commission has posted a document that is expected to form the basis of the state's protocols on load impact estimation for demand response. This document is helpful in outlining issues regarding the evaluation of demand response programs, and also discusses forecasting the effects of demand response for program planning. It should be available for downloading in September. Check the proceedings section of the California Public Utilities Commission's website for this document. (Refer to Rulemaking 07-01-041.)

Other Evaluation Resources

Background Materials

CEE's Evaluation Web Pages

CEE's evaluation web pages include a wealth of materials to support the evaluation of publicly funded energy efficiency programs and help new program administrators and evaluators to understand and keep up with the evolving field of energy efficiency program evaluation. After February 27, 2008, these will include materials supporting CEE's evaluation webinar, such as descriptions of categories and types of evaluation, examples of program portfolios and program logic models, and a downloadable recording of CEE's evaluation webinar. Other materials currently available include links to a calendar of evaluation training opportunities, links to organizations that offer evaluation training on-line or on a regular basis, results of seven years of CEE's annual survey of household awareness of the ENERGY STAR label, and an annual state-by-state summary of energy-efficiency budgets and a regional summary of savings impacts from CEE members. This "Guide to the Guides" is also available on CEE's evaluation web pages.

California Evaluation Framework (2001)

The predecessor to the current (2004) California Evaluation Framework, California's 2001 Framework for Planning and Assessing Publicly-Funded Energy Efficiency , provides a thorough overview of energy-efficiency program evaluation theory and approaches. It includes a glossary of energy-efficiency evaluation terms and useful explanations of concepts and measurement challenges that are rarely treated elsewhere. This and the 2004 Framework are excellent background and reference documents for both novice and experienced evaluators.

EPA's Clean Energy-Environment Guide to Action

The Clean Energy-Environment Guide to Action identifies and describes 16 clean energy policies and strategies that states have used to meet their clean energy objectives. It describes how states are successfully expanding the role of clean energy in the U.S. energy system and shares the experience and lessons learned from successful state clean energy policies.

Best Practices Benchmarking for Energy Efficiency Programs

The Energy Efficiency Best Practices Project seeks to identify and communicate effective energy-efficiency practices nationwide to enhance the design, implementation, and evaluation of energy-efficiency programs. The Web site contains a database of best practices reports and individual program summary reports.

Emerging Technologies Coordinating Council

The Emerging Technologies Coordinating Council is an organization run by California utilities to assess promising new technologies for California's energy customers. This website contains a database of current and recent new technologies projects undertaken by the council.

Energy Information Administration (EIA)

The Energy Information Administration (EIA) generates an annual survey of electric utilities. The EIA Web site contains the collected data from form EIA-861, which includes the following information: peak load, generation, electric purchases, sales, revenues, customer counts and demand-side management programs. Annual results are available for download.

Papers & Reports

CEE Evaluation Clearinghouse

CEE's Market Assessment & Program Evaluation (MAPE) Clearinghouse is a database of energy-efficiency program evaluation reports, potential studies, and related documents. The documents are all publicly available and most can be downloaded directly from the CEE Web site as PDF files. Most are supplied by CEE members. It also offers full-text searching of IEPEC conference proceedings. While the Clearinghouse was designed for the use of CEE's membership, in the interest of improving program evaluation across the entire energy-efficiency industry, CEE and its members make the Clearinghouse available to the public.

California Measurement Advisory Council (CALMAC)

CALMAC provides a forum for the development, implementation, presentation, discussion, and review of regional and statewide market assessment and evaluation (MA&E) studies for California energy efficiency programs conducted using Public Goods Charge funds. CALMAC hosts a large searchable database of downloadable evaluation reports from the state's three IOUs and other energy efficiency program administrators.

International Energy Program Evaluation Conference (IEPEC)

IEPEC is a biennial professional conference for energy program implementers, evaluators of those programs, federal and state agency representatives, and academic researchers. The purpose of the conference is to provide a forum for the presentation, critique and discussion of objective evaluations of energy programs. Abstracts of IEPEC documents can be searched on-line at IEPEC's website or via CEE's Clearinghouse. Proceedings are available for purchase on the IEPEC website.

American Council for an Energy-Efficient Economy (ACEEE)

ACEEE is a nonprofit organization dedicated to advancing energy efficiency as a means of promoting both economic prosperity and environmental protection. They host two bi-annual conferences that include evaluation-related papers, and conduct and publish their own research. A list of publications—many available for free downloading—and conference proceedings, and information about their conferences, are available on-line.

American Journal of Evaluation

The American Journal of Evaluation is an interdisciplinary social science magazine which covers diverse topics within the field of evaluation. This website offers both current and archived articles for subscribers.

The New York State Energy Research and Development Authority (NYSERDA)

The NYSERDA Web site contains information about the organization's evaluation approach and reporting. Users can access the full text of annual and quarterly evaluation reports filed with the New York State Public Service Commission over the past five years, as well as other special reports such as Cost-Effectiveness and Low-Income Program evaluation results. Also available is a listing of 100-plus detailed reports completed by NYSERDA's evaluation contractors in recent years that can be obtained by request.

Northeast Energy Efficiency Partnerships

NEEP is a nonprofit organization that conducts policy and program research as well as evaluation related to energy-efficiency activities from Maine to New Jersey. The Web site includes links to a clearinghouse of energy-efficiency data for the Northeast, updates on developments in energy-efficiency policy, results of regional protocols research, and information on regional initiatives.

The Northwest Energy Efficiency Alliance (NEEA)

NEEA is a nonprofit corporation supported by Bonneville Power Administration, electric utilities, public benefits administrators, state governments, public interest groups and energy-efficiency industry representatives. Its mission is to catalyze the Northwest marketplace to embrace energy-efficient products and services. Publications, including evaluation reports and market research reports pertaining to the Pacific Northwest, are available on the Research and Reports page of the NEEA Web site.

Energy Trust of Oregon

The Energy Trust of Oregon is a public-purpose organization dedicated to energy efficiency and renewable energy generation. They host an on-line library as well as a list of evaluation reports.

Wisconsin Department of Administration

The following items are available on the Wisconsin Department of Administration Web site: FAQs, facts about Wisconsin, fact sheets, document library (forms, publications and other working documents) and listing of other on-line resources related to Wisconsin program evaluations. Also available are evaluation reports of Focus on Energy, the statewide residential efficiency program.

Energy Center of Wisconsin

The Energy Center of Wisconsin, a private, non-profit organization dedicated to improving energy sustainability including support of energy efficiency, renewable energy, and environmental protection, also hosts an extensive assortment of evaluation reports in their library.

Databases

Database for Energy Efficient Resources (DEER)

The Database for Energy Efficient Resources (DEER) is sponsored by the California Energy Commission and California Public Utilities Commission (CPUC). It is designed to provide well-documented estimates of energy and peak demand savings values, measure costs, and effective useful life (EUL) all with one data source. DEER has been has been designated by the CPUC as its source for deemed and impact costs for program planning.

California Residential Appliance Saturation Survey (RASS)

The California Residential Appliance Saturation Survey (RASS) is a research product containing statewide and utility specific results, including data on all appliances, equipment and general usage habits. All of the survey data is available on the Web site.

U.S. DOE's Online Search Tool for Patents

For use with R&D evaluation. Search patents from 1940-present.

Acknowledgements

This document was developed with input from the Steve Schiller of Schiller Consulting, author of the Model Energy Efficiency Program Impact Evaluation Guide, and John Reed of Innovologie and Ed Vine of Lawrence Berkeley National Laboratory, two co-authors of the Impact Evaluation Framework for Technology Deployment Programs

[1] Reed, J., Jordan, G. and E. Vine. 2007. Impact Eval Framework for Technology Deployment Programs. U.S. Department of Energy, Office of Energy Efficiency and Renewable Energy. Page iv.

Back to top