Audit of Information to Support Performance Reporting

Vetted Report

February 2015

Access to Information Assessment

An assessment of this report with respect to provisions in the Access to Information Act produced no exemptions; therefore this report is presented in its entirety.

Table of Contents

Acronyms and Abbreviations

APPS
Annual Performance Planning System
BI
Business Intelligence
BSC
Balance Scorecard
DPP
Detachment Performance Plan
DPR
Departmental Performance Report
EPP
Executive Performance Plan
MAF
Management Accountability Framework
MRRS
Management, Resources and Results Structures
OCG
Office of the Comptroller General
OPSM
Organizational Priorities and Strategic Management
PAA
Program Alignment Architecture
PMF
Performance Measurement Framework
RBAP
Risk-Based Audit Plan
RCMP
Royal Canadian Mounted Police
RPP
Report on Plans and Priorities
SEC
Senior Executive Committee
SPPD
Strategic Policy and Planning Directorate
TB
Treasury Board
TBS
Treasury Board of Canada Secretariat
UPP
Unit Performance Plan

Executive Summary

A well designed performance management model is a key tool to aid an organization in managing its priorities and plans and to report on its progress in achieving established performance objectives. Key to the design of such a system is determining what information should be collected, and how it will add value to the various stakeholders. Within the RCMP, development of such a system is complicated in that ideally, it should meet local, divisional, provincial, headquarters, and federal requirements. A well-designed system would allow information to be entered once and subsequently amalgamated and viewed from different perspectives to meet a host of management and reporting requirements.

The current approach to performance management and performance reporting within the RCMP would benefit from additional strategic planning and direction to ensure it provides consistent, accurate, value-added information to stakeholders. A more streamlined and consistent approach to gathering and collating the required information is warranted, to ensure maximum benefits are derived from the resources expended. While efforts are currently underway to modify the existing Annual Performance Planning System, careful planning and consideration of the required information, its format, and desired reporting capabilities is needed prior to investing further in any technology changes. Only through clearly articulating the objectives, and ensuring all users have a consistent understanding of its purpose, can an appropriate system be developed to meet the need.

This audit report includes recommendations that seek to improve existing performance reporting processes and which could result in improved information for decision-making as well as on-going efficiency gains. The management response demonstrates the commitment from senior management to address the findings and recommendations. A detailed management action plan is currently being developed. Once approved, RCMP Internal Audit will monitor its implementation and undertake a follow-up audit if warranted.

Management's Response to the Review

Strategic Policy and Planning Directorate (SPPD) agrees with the findings of the Audit of Information to Support Performance Reporting. SPPD recognizes that improvements are required to the current approach and tools used for performance management in the RCMP in order to meet the requirements of the RCMP's programs and activities, at the federal and provincial/territorial levels. A comprehensive plan is required to ensure that performance management and reporting is consistent, efficient and value-added across the RCMP.

Communication is a key component of successful implementation, as is providing effective policies, procedures and tools. A discussion on the findings of this audit and the challenges associated with the APPS will be held with Commanding Officers and other senior managers at the February 2015 meeting of the Commissioner's Senior Management Team. This discussion will assist SPPD in its examination of the current approach and the development of an appropriate strategy which clearly articulates performance reporting objectives. A focus will be placed on ensuring the resulting approach meets the needs of stakeholders and creates efficiencies in reporting requirements.

SPPD commits to taking action on the recommendations during the next fiscal year. A detailed management action plan which addresses the report recommendations will be developed for review by the Departmental Audit Committee prior to their June 2015 meeting.

Rennie Marcoux
Chief Strategic Policy and Planning Officer

1 Background

Performance management is a key element in the delivery of effective policing and is crucial to the RCMP's ability to report on its commitment to safe homes and communities for all Canadians. Central to performance management is the development of performance measures, the collection of performance data, and the compilation, analysis and reporting of performance information.

Performance measures are collected for four principal purposes. Two of these are federal government requirements related to performance management reporting: The Treasury Board (TB) Policy on Management, Resources and Results Structures (MRRS) and the 2009 Policy on Evaluation. The policy on MRRS calls for each department to create a Performance Measurement Framework (PMF) that outlines the expected results, including performance measures, for reporting on programs identified in the Program Alignment Architecture (PAA). The Policy on Evaluation requires the development of Program Performance Measurement Strategies, which include performance measures to assess the success of a program and ensure data availability for evaluations. As well, RCMP contract policing divisions collect extensive performance information to satisfy municipal, provincial and territorial requirements. Finally, each RCMP unit and detachment is expected to be engaged in business planning and to track performance metrics in order to improve processes and results.

Performance enhancement and continuous improvement is supported by the Strategic Policy and Planning Directorate (SPPD). Their role includes the coordination of RCMP performance reporting to central agencies and Parliament and ensuring RCMP compliance with the Policy on MRRS. It also includes identifying and assessing corporate risks that may affect the delivery of RCMP policies, programs and services.

Until 2014, the Balance Scorecard (BSC) was the RCMP's main performance management tool to compile strategic performance data. As per the SEC Record of Decisions of December 2013, the BSC was phased out. To replace it, SPPD is developing an Executive Performance Plan (EPP) within the Annual Performance Plan System (APPS) to record the performance management information necessary to report on the PMF. The APPS has been in place since 2007. It was designed for detachments in contract provinces to create Detachment Performance Plans (DPP) that establish priorities, initiatives and targets to report to municipalities, provinces and territories. Some federal and administrative units also use the APPS to create Unit Performance Plans (UPP) but this has not been mandatory.

Restructuring within the RCMP required both the PAA and PMF to be recently revised. This revision included some changes to performance indicators and information sources. The transition from the BSC to the APPS, and the revised PAA and PMF, are all part of the new performance management approach currently being implemented in the RCMP.

In April 2014, the Commissioner approved an audit of information to support performance reporting as part of the 2014-17 Risk-Based Audit Plan (RBAP). The scope of the audit included an assessment of the progress on the management response to the Office of the Comptroller General (OCG) Horizontal Internal Audit of Compliance with the Policy on MRRS. In response to the audit, SPPD committed to a new performance management approach for the RCMP. The management action plan stipulated that SPPD would develop a business plan to incorporate the MRRS within all aspects of planning and resource allocation by April 1, 2014.

2 Objective, Scope, Methodology and Statement of Conformance

2.1 Objective

To assess the effectiveness and efficiency of the processes and procedures being put in place for strategic performance management within the RCMP.

2.2 Scope

The audit scope included an assessment of performance management in the RCMP with a focus on the types of performance information collected, how the information is reported and how the information is used at all levels of the organization. The audit also included a follow-up to the Management Action Plans resulting from the OCG Horizontal Internal Audit of Compliance with the Policy on MRRS.

2.3 Methodology

Planning for the audit was completed in September 2014. Interviews were conducted with SPPD staff and senior management and relevant policies and guidance documents regarding performance management were examined. Audit criteria used to develop audit tests were based on RCMP policies and guidelines, and the Audit Criteria related to the Management Accountability Framework: A Tool for Internal Auditors developed by the Office of the Comptroller General. Audit objectives and criteria are available in Appendix A.

The examination phase, which concluded in November 2014, employed various auditing techniques including: the review of performance management related documents; interviews with senior management and SPPD staff; and the analysis of Detachment Performance Plans (DPP) and Unit Performance Plans (UPP); as well as information obtained through questionnaires sent to 48 detachment and unit commanders, planners and line officers across four Divisions (E, H, O, V) and one business line (SPS). These sites were judgmentally selected from those that had completed a DPP or UPP over the past two years.

2.4 Statement of Conformance

The audit engagement conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program.

3 Audit Findings

3.1 Performance Management Strategy

An overarching performance management strategy needs to be further developed and communicated to ensure that maximum benefits are realized.

The Policy on MRRS requires that all federal departments and agencies collect performance information in order to measure, monitor and assess their successes and realize opportunities for improvements. This information is used to set new priorities, objectives and targets for the following year. The auditors expected that a strategy for performance management would be in place that would define the requirements to collect performance information needed for reporting and program enhancement. Performance information collected at the detachment and unit levels was expected to align with the RCMP PMF, divisional priorities and program evaluation requirements as well as support local business planning. Ideally the plans would allow for consolidation to fulfill reporting requirements and support decision-making at all levels.

The audit first examined whether an approach and a strategy had been established and then whether a plan had been developed to ensure that the right performance data is captured at all levels of the organization. Federal government reporting requirements and the contract policing requirements to report to the provinces were then reviewed. Lastly, the team assessed how the information is collected and consolidated to complete the required reporting.

SPPD has a mandate to support enhanced program performance and continuous improvement as well as to report performance management information at the department level. In response to the 2013 OCG Horizontal Internal Audit of Compliance with the Policy on MRRS, SPPD established a management action plan that included a commitment to implement a new performance management approach for the RCMP by April 2014. The approach was to incorporate the MRRS within all aspects of planning and resource allocation. While a detailed implementation plan with associated milestones has not been developed, a new performance management approach that includes a revised PAA, a revised PMF and new tools to facilitate reporting is under development.

Performance Management Requirements

Following the revision of the PAA, a new PMF was submitted to TBS for approval in November 2014. The PMF has been substantially revised to include improved performance indicators that measure outcomes and efficiency using statistically based data and client-based surveys. The PAA and the PMF are not static documents and will be reviewed again in the future, giving the opportunity to further refine some of the PMF indicators to include additional measures of operational activities.

Federal units constitute the majority of PMF reporting requirements. These units continue to face challenges in developing metrics that measure the impact of their work. This is understandable given the diversity and complexity of the work they undertake. Some units are tracking established targets and indicators that measure outputs as opposed to outcomes. Efforts are required to ensure that all units are collecting and recording performance information in their UPP in support of PMF reporting and to facilitate performance improvement. Currently this is not the case as UPPs are not being used to provide the information required for PMF reporting. Organizational Priorities and Strategic Management (OPSM) has provided assistance to some administrative and Federal units to establish strategic plans, however, their focus to date has been on reviewing detachment level plans. To establish a foundation for future evaluations, the RCMP Evaluation Group, in conjunction with OPSM has provided some assistance at the unit level to develop and improve Federal unit performance indicators. Establishing relevant, measurable metrics is a key step prior to considering the mechanism to be used to capture and analyze the resulting data.

Under the terms of the RCMP contracts with provinces to provide policing services, divisions are required to report on the implementation status of their objectives, priorities and goals to their respective provinces. In addition to reporting on performance, this information is used to set future priorities. The challenge at this level is to meet the requirements of the individual provinces. Most Divisional Commanding Officers stated they are satisfied with the performance management information they receive from their divisional planners or strategic staff however they had limited awareness of the processes in place to gather this information, or of the level of effort involved.

Although units, detachments and divisions are all making progress in improving performance management, the efforts remain largely siloed and manual. This process is very labour intensive and increases the risk that performance information required at all levels is not readily available. There has been limited effort to determine if the information could be obtained in a more replicable, efficient manner.

Communication and Guidance

Recognizing the importance of communication, in particular during a period of change, SPPD chairs a working level advisory committee that engages planners in discussions on national issues related to strategic performance and communicates proposed improvements. The terms of reference for this committee indicate that proposed improvements and changes are to be communicated by the attendees within their respective Division, Business Line or Corporate Service Sector. While this is a positive step, efforts are needed to ensure that a consistent message with clear endorsement from the policy centre is disseminated to all units and divisions.

A successful performance management strategy requires clearly communicated expectations, clearly defined roles and responsibilities and a monitoring and oversight mechanism to ensure the implementation meets the requirements. SPPD is developing a Strategic Performance Management Guide to meet this requirement. The current draft guide would benefit by clearly detailing roles and responsibilities, including the requirement for monitoring and oversight. The guide should indicate how each element in the process is integrated into reporting at the various levels. In addition, it should include information on how the EPP will be used to capture program level performance information to support the new PMF. CO's indicated that they do not receive feedback from SPPD on the quality of their plans within APPS and expectations regarding the completion of plans are not well communicated, especially in terms of meeting federal performance management requirements. Actively reaching out to plan owners, detachments and units to ensure that they all understand how the transition will impact their work, would contribute to a smooth implementation. Increased support to planners and plan owners would strengthen the performance management process throughout the organization by ensuring that all have the guidance, tools and feedback necessary to implement the new approach.

3.2 Performance Management Tools

A common, streamlined tool for performance management is needed.

In December 2013, the discontinuation of the BSC and the development of a BI tool was discussed and approved by Senior Executive Committee (SEC). Subsequent to this decision, SPPD announced in February 2014 that the APPS would become the primary strategic performance management tool used by the RCMP. The APPS in conjunction with Business Intelligence (BI) software, was intended to deliver accurate and more timely performance information. Whether the APPS can fulfill this role in the future depends on the capabilities of the system and the quality of the data stored in it. Of concern, the APPS does not currently have the capacity to search for or consolidate information. To be efficient and effective, the APPS must allow information to be consolidated from all levels, and facilitate the compilation and extraction of data for use in reporting and decision-making. Only recently, a plan has been developed outlining necessary changes to the APPS to support the inclusion of an EPP. However, the plan does not outline the changes required to the APPS to ensure that it can be used effectively with a BI tool.

Currently, Divisions use several methods and systems to collect, track and assess performance information such as spreadsheets and local databases because of the APPS limitations. As a result most reports are generated manually, limiting the planners' capability to respond quickly to requests or monitor progress on an ongoing basis. Standardizing the tools used by detachments and units to record, tabulate and report performance information and increasing access to these tools would improve efficiency and allow for consolidation of the information across the Division. This would facilitate performance monitoring and the fulfillment of reporting requirements.

Analysis of APPS data highlighted areas for improvement with respect to completeness and comparability. A quantitative analysis of the 2013-2014 plans indicated that essential performance information was not always being recorded. For example, 16% did not include issues, objectives and measures; 18% of plans did not identify strategic priorities and initiatives; and 39% did not include a risk assessment. In addition, the data was not recorded in a standardized format limiting analysis and comparability. Some plan owner's recorded progress as a percentage, while others measured progress in raw numbers. Moreover, a large portion of the fields use text rather than numeric formats. In many cases a diary or log book approach has been used and lengthy dialogues are provided to outline progress in implementing initiatives. This use of text complicates analysis and validation of information by planners, reduces efficiency and reliability, making it difficult to determine if progress occurred. Performance indicators used across detachments and units varied greatly, further inhibiting the consolidation of data across the divisions.

There is currently no mechanism in place for SPPD to be able to confirm the number of plans that should be completed each year. Quantitative analysis indicated that the recorded number of plans (DPPs and UPPs) in the system decreased over the past 5 years. Between 2012-2013 and 2013-2014 the number of plans decreased by 17%, from 1,134 to 939 plans. Of the 939 plans, 376 were federal UPPs. By comparison, 570 federal UPPs were completed in 2009-2010.

Detachments are currently the main users of the APPS and although they see the value in completing performance plans, users indicated that the APPS is cumbersome and not user-friendly. Plan owners experienced IT-related challenges when developing their plans and compiling data. APPS reporting limitations, combined with inconsistency in the information recorded, make it difficult for Division Commanding Officers to monitor progress on a regular basis and take any necessary corrective action in a timely manner. Rather, they must rely on manually aggregated data from planners to ensure detachment targets are met. Similarly, manual reports must be produced by detachments and units in response to the needs of divisional planners. This is very labour intensive, results in duplication of effort and increases the risk of data inaccuracy.

A plan to modify the APPS to include EPPs and improve the ease of use and reporting function has recently been developed although it is not yet approved. This plan also includes the modification of the APPS to align with PMF requirements. The plan documents the transition of the application over a three-year period however it does not address the requirement to review and standardize the format of information within the system nor to change it from a data repository to a dynamic searchable tool. This is a necessary step before a BI tool can be developed to access the data. The use of a BI tool will improve the ability to extract, compile and report data related to performance management. However, careful planning is required to ensure that the right data is captured at the source and can be compiled to support different reporting requirements.

4 Conclusion

Performance management is a key part of public governance and the RCMP must be in a position to comply with MRRS and Evaluation requirements by collecting and analyzing data related to its performance. To be successful, the performance management plan should include strong performance indicators; be supported by a tool that allows consolidation and reporting to both Parliament and the provinces; and define the expectations and accountabilities at the unit and detachment level.

Several changes have been made over the time period of this audit and performance management practices continue to be revised. The BSC was phased out before having a new performance management system in place, making it difficult for the organization to report on its accomplishments during this period without significant manual intervention.

The transition to a new performance management approach in the RCMP would benefit from improved planning and communication across the organization. Policies and guidance documents should also be amended to reflect the changes. The involvement and approval by senior management in this transition is essential to ensure acceptance of the changes.

The implementation of the new performance management approach, currently in progress, is an opportunity to develop strong performance indicators and streamline the way we collect, record and report performance information. With a comprehensive plan, an approach that meets all departmental requirements should be achievable. Such a plan would include a common framework that would allow the same data in a dynamic searchable tool to be used for multiple purposes. Adopting a standardized performance management approach would result in efficiencies by reducing duplication of effort and would provide a sound basis to report performance at all levels.

5 Recommendations

1) The Chief Strategic Policy and Planning Officer should:
Develop an overarching plan for collecting and reporting performance information across the RCMP that: efficiently meets reporting requirements at all levels; allows for consolidation of data; and facilitates performance monitoring and management.
2) The Chief Strategic Policy and Planning Officer should:
Continue to develop and implement tools, including policies, user guides and systems, appropriate to achieve the plan.
3) The Chief Strategic Policy and Planning Officer should:
Develop additional mechanisms for communicating performance management objectives and requirements to Senior Management.

Appendix A - Audit Objectives and Criteria

Objective: The objective of this audit engagement is to assess the effectiveness of the processes and procedures being put in place for strategic performance management within the RCMP. Criterion 1: A plan was established to implement the new strategic performance management approach.
Criterion 2: Policies and procedures are in place for strategic performance management.
Criterion 3: Monitoring and oversight mechanisms for strategic performance management are in place.
Criterion 4: Performance information meets the needs of Senior Management and the Divisions/Detachments/Units for strategic decision-making

Appendix B - Policies, Guidelines and Report References

Treasury Board Policies, Directives, and Guide
Policy on Management, Resources and Results Structures (MRRS)
Policy on Evaluation
RCMP Annual Reports
Department Performance Report (DPR)
Report on Plans and Priorities (RPP)
External Reports
Office of the Comptroller General (OCG) Horizontal Internal Audit of Compliance with the Policy on MRRS, March 2013
Top of page
Date modified: