The final piece of the “line-by-line” audit is the assessment of program necessity and effectiveness. Resource constraints demand that each agency must manage its resources as effectively and efficiently as possible. As such, we undertook a review of the State of California’s strategic planning, performance measurement and performance based budgeting. This review provides an understanding of how the state plans its programs, measures its results and makes budget adjustments accordingly. Further, it shows the direction the state is moving in budgeting, fiscal systems, and program management; we must recognize the state’s efforts to date in measuring and monitoring performance.

In 1994, the State Government Strategic Planning and Performance Review Act (Chapter 779, Statutes of 1994) was passed, adding Government Code (GC) Sections 11810–11817. This act required that all state agencies consider strategic planning, and also required DOF to survey agencies concerning their strategic planning efforts. This charge commenced in 1995 and was to be updated annually thereafter.

DOF’s annual survey and report to the Joint Legislative Budget Committee showed that a means for implementation and enforcement was necessary. In 1996, DOF, through Management Memo 96-23, required all agencies to develop strategic plans and have them completed and approved by July 1, 1997. The memo further identified the minimum components of strategic plans and stated that the plans would be the basis of each state agency’s budget. In addition, all agencies’ strategic plans were to be approved by and filed with the Governor’s Office. Beginning with the FY 1998–1999 budget, strategic plans were to be linked to the budget process. Furthermore, BCPs would be considered for approval only if consistent with an approved strategic plan.

GC Section 11813 (part of the act mentioned above), discusses the Legislature’s intent that strategic planning should include performance measurement, performance budget contracts, and performance reviews. DOF implemented the code via Budget Letter 98-07, which required that budget adjustments requested in the preparation of the FY 1999–2000 budget be linked to the entity’s strategic plan.

We met with DOF, the Bureau of State Audits, the State Controller’s Office, the Department of Motor Vehicles, a former Deputy Director of Parks and Recreation (a strategic planning expert) and other California Performance Review team members to discuss the current status of strategic planning. After reviewing GC Sections, Budget Letter 98-07 and conducting our interviews, we determined that the legislation and direction is still valid; departments are required to implement strategic planning efforts. However, the Davis Administration did not support strategic planning and consequently neither DOF nor the Governor’s Office has actively monitored/tracked compliance beyond the first year of implementation. Currently, the state has no centralized oversight of state strategic planning efforts.

In January 1993, the Governor’s Budget introduced a Performance Budgeting Pilot Project. Four agencies participated in the pilot beginning in FY 1995–1996, at which time DOF was asked to review and evaluate the pilot and report to JLBC. We address DOF’s review separately, in a later section of this report. The Davis Administration did not extend the pilot beyond FY 1998–1999. However, the agencies that participated in the pilot continue to use performance measurement in their budgeting processes.

Budget Letter 98-07 requires an updated strategic plan (plan) whenever a change to a goal in a prior plan, a change to an objective in a prior plan, or a proposed budget request that does not tie to an existing plan occurs. Otherwise, a plan need not be revised if the department director certifies that there have been no changes.

We first contacted the State Library to obtain a list of agencies that submitted strategic plans or performance reports in the past five-year period and to obtain the current strategic plans for each agency. Since 1998, only 11 plans have been cataloged at the State Library. However, when the strategic planning act was first introduced in 1994, the State Library received 48 plans. The number of updates and renewals has steadily declined as Davis Administration interest and DOF oversight waned from FY 1999–2000 through 2002–2003. To augment our identification of agencies’ strategic planning activity, we surveyed state agencies and inquired as to their use of strategic planning, performance measures, and performance based budgeting. We sent out 106 survey forms to various state agencies (with staff size of 20 or more Personnel Years (PY)) including boards, commissions, departments and offices. Eighty-two (77 percent) of the 106 state agencies responded to our survey. Note that the results we report below are anecdotal in nature; that is, this is what the respondents told us. We did not audit the strategic planning or performance measurement of these agencies. Any conclusions we draw and recommendations we make must be taken with this in mind.

Our survey asked whether agencies had strategic plans and the latest date of their plan update. Seventy-four of the 82 agencies (90 percent) reported that they had developed strategic plans. Moreover, most have updated their plans regularly; 64 (86 percent) of the 74 agencies with plans have updated their plans since 1998. See Exhibit A below.

We asked how agencies developed their strategic plans. Forty-four of the 74 agencies (60 percent) used only internal staff to develop their plans, 24 agencies (32 percent) used a combination of consultants and internal staff and five agencies (6 percent) employed consultants alone to prepare their plans. The enabling legislation allocated no additional funding; the agencies continuing to develop and update their plans have done so with existing resources. See Exhibit B below.

Budget Letter 98-07 requires that strategic plans be submitted to DOF for approval as well as to the Governor’s Office for signature. Of the 74 agencies that reported they had developed plans, only 30 (40 percent) indicated that they sent their plans to DOF while 17 (23 percent) were sent to the Governor’s Office. This correlates to the small number of plans cataloged at the State Library over the last five years.

We sought to understand whether agency goals and objectives were tied to any higher level strategic plans. Thirty-two agencies (43 percent) stated that their plans were tied to a higher level plan. In addition, we asked whether an agency’s strategic plan was supported by other plans from within the organization. Fifty-three agencies (71 percent) indicated that their plans were supported by sub-plans under their responsibility.

We asked what, if any, obstacles agencies may have experienced in developing their strategic plans. Sixty of the 74 agencies (81 percent) provided us with at least one obstacle. The single largest obstacle, identified by 24 (40 percent) respondents, was that a lack of resources limited their strategic planning efforts. Twelve (20 percent) responded that a lack of “group acceptance” to strategic planning led to difficulties in preparing plans, while 11 (18 percent) responded that “management acceptance” was an obstacle. See Exhibit C below.

To understand what went well, we asked what successes could be identified as a result of an agency’s development of a strategic plan. Seventy respondents indicated that they had achieved successes; the largest number, 23 (33 percent), indicating an increased focus on mission, vision and goals in their organization. Another 18 (26 percent) responded that their business operations became more effective as a result of planning, while 9 (13 percent) indicated that they increased their focus on meeting objectives and efficiently using resources. See Exhibit D below.

If an agency did not prepare a strategic plan or has not updated their plan, we asked them to explain why they did not. Twenty-six agencies gave various reasons for not updating or developing strategic plans with “Lack of resources” being the most common response given by seven agencies. The remaining reasons varied from office turnover to some offices using other methods of planning.

DOF’s 1998 Budget Letter on Strategic Planning Requirements provided agencies with instructions regarding the format and components of strategic plans. Specifically, DOF indicated that departments should, at a minimum, develop at least one key performance measure for each strategic plan objective. DOF also suggested that agencies develop plans to monitor and track performance. The second part of our survey focused on state agencies’ activities related to performance measures. We summarized the survey results below.

Responding agencies indicated that they use benchmarks and performance measures, to varying extents, in managing their programs. Specifically, 69 agencies or 84 percent have developed benchmarks and performance measures that are linked to their strategic goals and objectives for all or some of their key program areas. Respondents’ comments revealed other interesting points, as follows:

• Several agencies’ performance measures were either mandated, required by contract provisions, or piloted by the federal government.
• Some agencies developed their performance measures independent of their strategic plan.
• A few agencies that developed performance measures do not use benchmarks.
• Performance measures are sometimes limited to measuring timeframes.

Approximately 90 percent of the agencies have automated information systems or manual processes in place to gather and measure performance data. Three of those agencies currently use web-based systems. The chart at Exhibit E below identifies agencies’ use of manual processes versus automated systems.

Fifty-four or 66 percent of the agencies monitor their program performance on at least a quarterly basis. Most agencies regularly generate reports detailing actual performance against targets. Almost half responded that the reports are being analyzed by management. Also, several agencies responded that their reports are being analyzed by internal staff such as strategic planning, statistical, budget or audit staff. Alternatively, many agencies’ reports are being analyzed by external consultants, stakeholders, and/or state or federal agencies.

We asked the agencies if they regularly analyzed the viability or propriety of their measures. Although 57 percent indicated that they do monitor and adjust measures, their timeframes for regular review and adjustment of measures ranged from an “as-needed” to “biennial” basis. When asked what action management takes if measurable operating objectives are not met, 44 percent of the agencies indicated that they determined the cause for not achieving the objective and adjusted the existing action plan or implemented a new one accordingly. While specific actions varied, the following were recurring themes: reallocating staff or resources, implementing training and documenting staff performance evaluations. Furthermore, 59 percent of the agencies indicated that the information was integrated into the following year’s strategic plan.

We asked agencies what obstacles they overcame or are trying to overcome to make their performance measuring efforts a success. Sixty-five agencies responded to this question; most identified multiple obstacles. As indicated within the chart at Exhibit F below, insufficient resources and the inability to develop adequate performance measures are key obstacles.

We asked what general successes agencies experienced from their performance measuring efforts. Most of the 63 agencies that responded to this question cited success in improving their focus on strategic goals and objectives. The agencies also identified various other successes, as indicated in the table at Exhibit G below.

Exhibit G
Successes Resulting From Performance Measuring Efforts

In addition, some agencies attributed more specific, statewide successes to their performance measuring efforts. These included the following:

• Contributing to affordable housing
• Decreasing the mileage death rate
• Better control of fires in urban areas
• Overall improvement in the health of children

The third part of our survey inquired about state agencies’ use of performance based budgeting processes.

Ten agencies responded that they had implemented a performance based budgeting (PBB) process for one or more of their programs. Four of these agencies had participated in the State’s Performance Budgeting Pilot Project mentioned earlier in this section of the report. As indicated within the table at Exhibit H below, other agencies currently use PBB to some extent or have used it in the past.

Exhibit H
Number of Agencies That Use Performance Budgeting

Those who use outcomes indicated that their operational budgets link to their strategic plans, at least partially. In most cases, they reported success in using performance measures to make budget decisions. Several agencies commented that their performance measurements helped them to direct resources to those areas most in need or to identify program activities to maintain, reduce, or eliminate.

Typically, these agencies included requirements for meeting financial targets within their strategic plans. When asked to identify the consequences for not meeting those requirements, responses included the following: loss of federal funds, increased oversight or discretion over budget, staff evaluations or bonuses affected and action plans required.

Eighty-four percent of the agencies indicated that their budgetary responsibilities are established and reviewed regularly. However, when asked if they had a formal contingency plan in place to deal with budget deficits, less than half responded “yes.”

Lack of resources is the main obstacle to implementing performance budgeting When asked what obstacles prevented agencies from considering a PBB process, agencies most often responded that insufficient resources prohibited them from implementing a PBB process. The chart at Exhibit I below identifies agencies’ responses.

Other individual concerns must be addressed if performance budgeting is implemented statewide. For example, agency program budgeting may not correspond to functions that cross program lines; the state’s employee performance appraisal process may not be in accord with agency performance measures; and agency performance may be contingent on obtaining accurate data from local entities.

Finally, we asked agencies how well prepared they were to implement a PBB process in the future. Not surprising, only 20 percent indicated that they were well prepared or prepared and ready to implement. Others did not know, did not respond, or identified concerns that mirrored the PBB obstacles identified above.

Forty-nine agencies (60 percent) indicated that at least some of their program performance or performance measures had been reviewed or audited. These audits have been conducted by both internal and/or external entities. Specifically, BSA and DOF conducted reviews/audits in 15 of those agencies (31 percent) and federal audit groups conducted reviews/audits in 12 (24 percent). Additionally, internal staff or internal auditors conducted reviews/audits in 17 of the agencies (35 percent). Furthermore, some reviews/audits had been conducted by other various external state agencies, contractors, or other entities.

BSA spends approximately 76 percent of its effort conducting performance audits. This type of audit is an objective and systematic examination of evidence to provide an independent assessment of a government organization, program, activity, or function in order to provide information to improve public accountability and facilitate decision-making by responsible parties. We identified seven BSA performance audits conducted in the past three fiscal years that reviewed state agency or county departmental strategic planning efforts or specific performance measures. BSA identified key elements missing from strategic plans and identified one agency that failed to use adequate strategic planning in managing a major project. Some recommendations for improving performance measures addressed the need for more complete performance measurements that include all cost components, and better tracking of data.

According to OSAE’s Directory of State Internal Audit Organizations for 2004, 32 internal audit units operate within state agencies. Seventeen of those audit units indicated that they performed program reviews and/or performance audits during FY 2002–2003. On average, the units spend 20 percent of their time on such audits. We did not examine the reports these auditors produced, but recognize the potential for this work to positively influence the development and review/revision of effective performance monitoring.

In January 1996, DOF evaluated and reported on the Performance Budgeting Pilot (pilot), which included the California Conservation Corps, and the Departments of General Services, Consumer Affairs, and Parks and Recreation. DOF viewed the pilot as a short-term success, but deemed it too early to issue a final evaluation of its results.

DOF identified two key implications for the state’s budget process, as follows:

• The usefulness and application of performance budgeting will depend on whether the allocation of budget dollars to specific outcomes or performance targets is for information purposes, or for appropriation control.
• While performance budgeting will facilitate the setting of budget priorities across programs and departments, it should not displace consideration of other factors, i.e., basic public needs, workload of case-driven programs and changing social and program conditions.

DOF also identified several objectives that should be satisfied before the pilot could be fully evaluated and rolled out to other non-pilot agencies. They included the following:

• Replacing all activity targets with performance measures (to the extent possible).
• Reviewing the linkage between all outcome oriented performance measures and departmental responsibilities.
• Establishing cost accounting processes, which link expenditures to dollars budgeted for outcomes or other performance targets and are capable of predicting alternative service levels associated with alternative budget allocations.
• Identifying, verifying and tracking cost savings associated with the program innovations that arise from the performance budgeting decision-making process.
• Implementing internal and external evaluation systems to assess the integrity and reliability of the performance and cost data.
• Using external evaluations to determine if any statutory program requirements have been de-emphasized simply because their performance levels are not easily measurable, or, conversely, whether some activities that are represented in performance indicators are unduly emphasized.
• Requiring external and internal audits to ensure that appropriate internal controls are in place to prevent any abuses associated with the administrative flexibilities granted to agencies.
• Collecting information on the cost-savings and improved performance, if any, resulting from each item of administrative relief and share this information with the appropriate control agencies for review.
• Handling the annual contracting process between the pilot departments and the Legislature in a consistent manner.
• Implementing a consistent state policy on “gainsharing” (whereby 50 percent of resulting savings are returned to the state) with appropriate controls.

We researched five states that utilize performance measures to identify the scope and responsibilities for monitoring their agencies’ performance and conducting audits. We found that all five states have independent units, within their executive level of government, that oversee performance monitoring and/or conduct performance audits.

The types of entities conducting performance audits varied among the five states. They included state auditor offices, an administrative board, a specialized audit team, and a legislative committee that primarily consists of research analysts, rather than auditors. One state has developed statewide strategic benchmarks and extensive performance measure guidelines that include detailed reporting, monitoring, and review requirements. Two other states perform regular sunset reviews of their agencies or programs to determine if they should continue. Furthermore, one state created a management advisory group that provides consulting services to its agencies in areas such as strategic planning, developing performance measures and statistical sampling and analysis. Details of these states’ oversight, monitoring, and audit functions are provided at Appendix X.

We also reviewed the results from a recent survey of 210 state government auditors that perform various audit functions within 37 states. The survey was sponsored by the Institute of Internal Auditors (IIA) and was conducted jointly by the IIA and two other national audit organizations. Although auditors responsibilities most often included compliance and internal control audits, some performance/operational audits were conducted by 80 percent of the internal auditors surveyed.

Similar to other states and the federal government, then-Governor Wilson and the Legislature attempted to reform California’s budget and management practices with the Strategic Planning and Review Act in 1994. DOF had responsibility for direction and oversight. However, neither DOF nor any other control agency has attempted to monitor or track statewide strategic planning efforts since 1996. In addition to the statewide strategic planning, the state implemented a Performance Budget Pilot Project that involved four agencies. However, the pilot project was not supported past its early implementation and subsequently failed because resources and guidance were not provided.

California’s attempts at strategic planning and experimentation with performance budgeting have shown some positive results. As indicated above, most agencies we surveyed indicated that they use strategic planning to some extent and have developed performance measures in the process. Also, many agencies have integrated performance into their budgeting processes and identified positive results, all using existing resources. However, some agencies with the highest increase in General Fund expenditure did not always use strategic plans or performance measures, as shown in Exhibit K.

Training will be one crucial element in making California successful in implementing statewide strategic planning and performance budgeting. As indicated above, although over half of the surveyed agencies use internal staff to prepare their strategic plans, many hired external consultants to assist them. Neither of these scenarios ensures consistency among the state’s entities. In addition to training, other key obstacles include resources for adequate staffing and for new or enhanced existing systems to track, measure, and report data. Success will also depend on continuous support from the executive levels of government. Furthermore, the state needs to provide better oversight and monitoring of its performance related activities. Specifically, the state should provide agencies with detailed training and guidance and ensure that agencies are complying with strategic planning, performance measuring and performance budgeting requirements.

California state government does not take accountability lightly. What other conclusion could one draw knowing that approximately 60 state agencies employ 4,000 auditors, evaluators and examiners? Moreover, external auditors representing the federal government and the public interest perform audits of specific systems, programs, grants and recipients of state funds in departments statewide. As a case in point, between federal and state, external and internal, 21 different audit teams have reviewed Department of Health Services programs and activities this fiscal year.

As noted above, the state’s internal auditors provide their departments valuable recommendations that are usually implemented. Additionally, OSAE, SCO and BSA auditors serve the Executive Branch, the Controller and the Legislature in fulfilling their responsibilities for monitoring and oversight of the state’s interests. The state’s regulatory and tax auditors carry out their program/function specific audits to maximize revenues, monitor expenditures, and verify compliance as their resources allow. In total, the complementary nature of the audit specialties and the breadth of scope the auditors cover provide reasonable assurance that the state’s operations are carried out in accordance with management’s directives and intent. However, the state’s audit function can and should improve.

In reviewing the varying classifications and requirements placed on the auditors, we determined that many classifications require similar preparation to perform similar duties. Typically, these classifications require a college degree (or equivalent) in a financial related field, with a specific number of accounting/business units as a basic entry-level requirement. In addition, entry-level staff must be able to obtain relevant documentation, analyze data, develop workpapers and present results, all under close supervision. These entry-level requirements help assure a professional and knowledgeable workforce. However, given the similarity of entry-level requirements among the different departments and classifications, the question must be raised as to whether so many different classifications are necessary or useful.

The foregoing notwithstanding, one area that should have a separate classification for all audit organizations to use is the Information Technology auditor. The California Public Employees’ Retirement System and BSA each maintains a specialist classification for IT auditors, but the use of these classifications is restricted to those organizations. Given the reliance we place on our automated systems and the separate, identifiable skills necessary for proficiency in this audit area, this lapse should be addressed.

Again, as discussed in the Financial Review section above, guidance and coordination of the state’s audit function is minimal and not strictly enforced. Without a strong, coordinated effort involving auditors statewide, we believe the audit work completed by the state’s internal auditors may be inefficient and likely results in the inconsistent application of audit standards. To address the need for guidance and coordination, we propose that any reorganization of state government include agency-level audit offices. This would help ensure a consistent application of audit standards, comparable audit results among and between the agencies and a point of responsibility to assess risk throughout the agency. In addition, these audit offices should be staffed with auditors versed in performance auditing and in IT and security reviews.

As mentioned above, the review was performed in accordance with GAGAS applicable to attestation engagements. These standards require the auditor to report on subject matter that is the responsibility of another party. In this case, that subject matter is the agreed-upon procedures set out in the audit proposal to CPR. We note that in accordance with GAGAS, the CPR auditors are organizationally independent from the issues included in the agreed-upon procedures only to the extent that these issues and report are used by CPR and California Executive Branch management. That is, the CPR audit team members are all employees of the departments and organizations that are subject to the engagement. This report is not intended for and should not be used by anyone other than the specified party. However, its distribution is not restricted for any other reason.

Respectfully Submitted,

Richard Bon Smith, CPA, CGFM
CPR Audit Team Leader

Date: June 30, 2004

Audit Team:
Timothy J. Adams, CPA, CIA
Christine W. Berthold, CPA
Steve Castillo, CPA
Edwin W. Dong, CGFM
Doris Jensen, CIA, CGAP
Edward J. Hanson, CPA, CFS
Eric Pfost, CPA
Valerie Varzos, CPA
Michael Wilkening
Valerie Wood
Roberto Zavala, CGFM

Back to Top