Evaluating Impact

Measuring Success

A program of any scale will have a significant amount of data to track—customer information and contact dates, audit results, measures implemented, energy savings, and costs. A well-designed data tracking system can organize this data and provide reports that will streamline program management and evaluation, measurement, and verification (EM&V) efforts. The specific data collected will vary by program, but should at least include: participating customer information, specific measure information, and program costs.

There are a number of data tracking tools and programs that utilities and co-ops can use:

• A customer relationship management system. Using off-the-shelf systems that are not specifically designed for energy efficiency will require customization to make it work for the program.

• A specific data platform developed for energy efficiency programs, like energyOrbit, Energy Savings Platform, Direct Technology’s Energy Efficiency Collaborative Platform (EECP), or EnergySavvy’s Optix.

• Some third-party program management contractors have customized software that they bring to their utility projects.

  Your team, utility leadership, program funders, customers, and other program stakeholders will all expect updates about the impact of your program. Measuring the impact of your program will allow program administrators to determine if key program goals are being met and—if not—what aspects of the program may need to be modified. Measuring impact ranges from tracking total energy savings resulting from the program to customer satisfaction with the program.
  

Evaluation, Measurement, and Verification (EM&V)

Evaluating, measuring, and verifying the impact of your program will allow you to understand how program design assumptions compare with real-world performance. Some degree of evaluation, measurement and verification (EM&V) is also often required to meet state mandates or to continue to secure funding for your program. Depending on the funding source of your program or possible regulatory reporting requirements, EM&V procedures can be very specific.

Regardless of the reporting requirements of your program, all programs would benefit from tracking these EM&V metrics:

• Participation rates

• Conversion rates from audit to upgrade

• Demographic information about participants

• Building stock characteristics of participating homes/buildings

• Total and per home/building energy savings resulting from upgrades

• Total and per home/building carbon reductions resulting from upgrades

• Total project cost per upgraded home

• Cost per kWh saved or other cost-effectiveness tests of your program

• Participant billing impacts

Setting up reliable data tracking systems prior to program launch is essential for conducting accurate EM&V. In many cases, utilities will need to work with a third-party firm or organization that can conduct an independent EM&V. Third-party EM&V is often required by state regulations or grant funders to show objectivity in evaluations. Although EM&V is conducted after the program is underway, be sure to arrange your EM&V team and process well in advance of program launch. Retroactively tracking program information is much more challenging and costly than setting up data tracking and measurement systems at the design phase — before the outset of the program.

Quality Assurance, Quality Control, and Process Improvement

A quality assurance (QA) and quality control (QC) protocol is key, especially for any building energy improvement program. Improper installations will severely undermine any program, so it is best to catch these problems early and then correct any systemic problems that may lead to additional installation issues. Poor quality work may also damage the building, and even endanger the occupants. Energy retrofits must be completed in a manner that not only improves the health and comfort of the resident but also meets local, state, and national standards.

Establishing a strong QA/QC process early in a program will help program administrators identify areas where more contractor training is needed and avoid headaches down the road. For example, a program could opt to perform a detailed QA check on each contractor’s first five projects. While initially time-consuming, this helps the program administrators and contractors to get on the same page (and identify contractors who are not qualified or otherwise not a fit for the program). The QA process can then be reduced to occasional spot checks.

When weighing the cost-benefits of QA/QC processes, don’t overlook the benefits that strong QA/QC offers participants. Program participants can feel overwhelmed or underinformed when dealing with contractors. Participants can be made to feel much more comfortable and satisfied when program representatives serve as third-party reviewers of contractors’ proposals and work. This will help spread word-of-mouth about the program and reduce issues with unsatisfied participants.

Surveys

Many utilities offer beneficial electrification programs to provide a new and valuable service to their customers. There is often an assumption that offering a beneficial electrification program will increase customer or member satisfaction—but it is important to verify that assumption. With a new program, it is helpful to follow up with participants directly after the retrofit is complete to capture immediate reactions and feedback, as well as a year later after the customer has experienced all four seasons in the newly retrofitted home.