Evaluation is only worth doing if it results in more effectively managed protected areas; in other words if the results of an assessment are first interpreted to identify some practical lessons and then acted upon. Although this sounds obvious it is by no means always applied in practice. Many assessments remain as partially completed exercises or locked away in reports or academic papers that have few positive impacts for the protected areas or biodiversity conservation. Unused studies undermine future evaluation because they frustrate the staff members and other stakeholders who have devoted time to help assessors, and make it more difficult to raise funds for monitoring.
Good evaluation should therefore include a period of reflection to work out the implications of what has been learned, leading to some clear recommendations for action. In most cases assessments also require the careful reporting of results. This chapter looks at the final phase (phase 4 in the evaluation process illustrated in Figure 8). It is divided into four sections: looking at the importance of proper analysis of the results, leading to appropriate recommendations and then to communication and implementation of the resulting recommendations (i.e. adaptive management).
Most assessments draw on information that has been collected over time through various forms of monitoring (see Chapter 3). Monitoring is generally expensive in terms of time or money and the data collected only becomes useful once it has been evaluated. Evaluations seek to interpret available information, looking at trends and implications (particularly when assessments can be compared over time) and to find causal links and relationships between context, planning, input, processes, outputs and outcomes. Given the expense involved, monitoring systems should be tailored to fit the needs of assessments to make sure that the most useful information is collected.
Most assessments will draw on and in turn produce a large mass of data that can initially appear quite unrelated. If an assessment is one of a series over time then the quantity of the information increases even more and can produce enough facts and figures to keep researchers working for years. However, assessments are primarily practical tools for managers and results are most useful if they can be produced fairly quickly. A balance is needed between the richness of the information and the speed at which it can be assimilated: a combination of approaches to both presentation and analysis of results is often desirable. Some options are outlined below:
The first level of analysis is a simple compilation of collected data, either for one site or across sites, usually in the form of tables and graphs. Many users will want the raw data for their own analysis; putting this together in an easily usable form means that the survey can be reassessed later by other users who are looking for different information or who are compiling broader-based studies.
Some evaluators find a ‘SWOT’ analysis a useful tool for analysing information further, usually carried out in a workshop with agency staff and/or other stakeholders. SWOT stands for “strengths, weaknesses, opportunities and threats” and involves categorizing data and initial assessments under one or another of these headings. This method can provide a quick summary of management effectiveness in a format that is appropriate for communication with busy upper-level managers and politicians and is also a valuable way of identifying next steps for management.
Many evaluation systems use simple scores, which summarise a lot of data into one number. Scores provide a quick and easy way for an audience to determine comparative conditions. Examples can be seen in the evaluations discussed in the RAPPAM method and World Bank Tracking Tool (see Case Studies II and VI). While protected area managers generally want more detailed reporting, quantitative data and analysis, scores are attractive to policy makers and NGOs as they give an instant overview of relative success and a way of comparing protected areas. However, scores risk over-simplifying complex issues, distorting results and being misinterpreted by those both supportive of or opposed to a particular protected area. Some explanation should therefore accompany “scorecard” reports. Some of the best systems using scoring are accompanied by a large amount of supportive data. For instance, Parks Canada reports its system of ecological monitoring in a very simple diagrammatic form, but this draws on a large amount of information that is also available to those who want to look deeper.53
Results of assessments can also be measured against agreed standards, which might vary from numerical targets (for instance relating to populations of key species) to descriptive conditions. In total, carefully selected targets and standards should provide a reasonably good picture of overall conditions in the protected area. This means that it is important to choose a broad and balanced list of targets; if elements are omitted then assessment is similarly likely to miss important aspects of overall protected area condition. Attainment of standards is also seldom clear-cut and usually requires value judgements by assessors or inspectors. The Nature Conservancy's Conservation Action Planning (CAP) methodology is an example of an approach that employs carefully selected targets both to plan management of a site and as the basis of monitoring progress.54
It may be possible for more advanced statistical analyses to be conducted, looking at trends in data and attempting to draw out broader patterns. However, the resulting statistics will only be as good as the data that they draw upon. Manipulating results through summing and averaging, or assigning weights to different indicators, and through the use of scales and indexes can give misleading results, particularly if the data are limited in either quantity or quality. In particular, any qualitative data that is turned into quantitative data should be treated with care and its limitations fully recognised.
Analysis is often strengthened by looking at changes over time or space; such as by comparing several protected areas within a system or measuring how the effectiveness of a single protected area changes over time.
Comparison between protected areas can be valuable but needs to be treated with caution, particularly if different assessors have been involved (or even different assessment systems). The WWF RAPPAM system is designed to assess all the protected areas within a country or district, in a workshop situation where managers provide a certain amount of peer review for each other (see Case Study II). Comparisons are useful for identifying trends (including for instance common threats or weaknesses) that may need to be addressed at a systems level and also to identify protected areas that are particularly stronger or weaker than average. Comparing between countries also provides interesting data but here the risks of distortion are comparatively greater and results should always be treated with caution.
Comparing individual protected areas over time is probably more valuable. It is usually worth repeating assessments at intervals to check on progress and to identify trends. Several protected area systems are now developing ‘State of the Parks’ evaluations that they intend to apply every few years; Metsähallitus Natural Heritage Services in Finland published the first of what are intended to be five-yearly assessments in 2005 (see Case Study V) and Parks Canada has already published several editions of its ecological monitoring studies. Except in the case of special-purpose single-event evaluations, repeat evaluations are almost always desirable and it is important to adopt an assessment system with low enough costs to allow this. Very simple assessments could be applied annually, while more expensive, time-consuming exercises will probably only be worth undertaking every few years. Assessment does not need to cover all aspects, every single time. For example, most protected area managers will wish to track implementation of management plans and work plans quite regularly, and evaluations are often required at a regular basis for specific projects within protected areas.
The desire to compare between evaluations over time is sometimes in conflict with the opportunity to improve the assessment system. Evaluation is itself a learning experience, and better indicators, changed circumstances, and access to improved technology will all tend to shape evaluation projects over time. Participatory evaluations, by their nature, need to be flexible and respond to people's needs and perceptions. However, changing methodology or indicators will obviously make it much more difficult to compare results over time. This is an inevitable tension and there is no sure way of addressing the problem. In general changes should be minimized to those that are really important, and statistical and other possible adjustments made to help keep results comparable.
An assessment of all six of the elements in the IUCN-WCPA Framework should provide information on both the extent to which management is achieving its own targets and on the effectiveness of the protected area in maintaining biodiversity and other values. An important part of the accompanying analysis should be to identify the extent to which measured outcomes are due to management interventions or to other factors, which may be beyond a manager's control. It is possible to have a well-managed protected area that still loses biodiversity (for instance because of climate change) and conversely even quite inefficiently managed protected areas may in some circumstances still maintain their values. It is important to understand the causes of success or failure of management: without such an analysis attempts to improve performance may be ineffective.
Evaluation can help show the effectiveness of management and indicate trends in biodiversity but does not necessarily explain why certain changes occur. For example, fluctuations in populations of particular species could be due to natural cycling or the result of factors such as encroachment, disease or changing weather patterns. Increased arrests of poachers could be because enforcement has improved or because there are more poachers to catch, and so on. A good evaluation system will provide sufficient data to help explain changes, giving us some ideas as to why outcomes have been achieved or not achieved. Information about context, planning, inputs, processes and outputs help interpret to what extent outcomes are due to particular interventions. Evaluation will often turn up particular questions or problems that require dedicated studies of their own. Explaining results is sometimes easier if evaluators use a simple model. Figure 10 gives an example that links the six elements of the management effectiveness Framework for a marine protected area, showing how different elements are linked, what assumptions have been made and what factors could influence outcomes.55
Figure 10. Assumptions linking the elements of the management cycle
People trained in scientific or other disciplines that put high value on analysis often forget how long it took them to learn the associated skills; they assume that everyone will be able to draw meaningful lessons from masses of data. This is not true, so data interpretation is an important part of a good evaluation. Ideally interpretation should involve staff involved in management but if necessary outside experts can be used; in the latter case they should devote time to explaining not only the lessons learned but also how these have been deduced.
The evaluation should conclude by drawing together some key conclusions. The conclusions in turn should lead on to a series of recommendations for the protected area manager and perhaps also for the protected area agency or managing body. Recommendations should:
ensure that any advice is clear and specific enough to improve conservation practices and realistic enough to ensure feasible solutions are found for priority topics.
include short- and long-term priorities and a timescale and budget (with additional funding needs where required). Short-term actions should be clear, concrete, achievable within time and resource constraints, and prioritized. Long-term recommendations should identify resource and policy changes needed for their implementation.
feed back into management systems to influence future plans, resource allocation and actions. Evaluations that are integrated into the managing agency's culture and processes are more successful and effective in improving management in the long term.
focus primarily on actions for the manager and rangers but where necessary also identify responses needed beyond the park boundaries.
be monitored, through annual work plans and also future assessments, to check whether identified actions have been undertaken and also (not the same thing) whether these have been successful in addressing challenges.
Findings and recommendations of evaluations need to feed back into management systems to influence future plans, resource allocations and management actions. Evaluations that are integrated into the managing agency's culture and processes are more successful and effective in improving management performance in the long term. The NSW NPWS Case Study (VII) illustrates how the IUCN-WCPA Management Effectiveness Evaluation Framework has provided the foundation for development of a broader Park Management Framework, which aims to deliver this integrated approach.
The recommendations from an evaluation will usually be more complex than a simple list of jobs to be done. They may include the need to fill gaps in knowledge, for instance by extra monitoring, research projects or through reference to experience in other protected areas. If the assessment throws up serious gaps in our understanding that need to be filled by monitoring, actions may include adapting the assessment process itself.
Perhaps most fundamentally, assessments sometimes need to address more basic issues such as the management objectives of the protected area or the aims of management. Particularly in the case of long-established protected areas, management priorities may have changed over time or perhaps never been set very clearly. In these cases assessors might suggest that managers step back a little and look at management objectives and accompanying management plans.
Reporting the results of an evaluation can pose considerable challenges for park authorities. The form of the report depends to a large extent on why the assessment was carried out, but it should be remembered that any written report accessible to the public can and probably will be used for a variety of purposes. Planning the assessment should include an early consideration of communication and of the audiences for the evaluation (see Chapter 4). Possible methods of communication include reports in hard copy and on the internet, attractive publications and brochures to increase public interest, presentations to managers, decision makers, interest groups and other stakeholders, field days and special events, media coverage and displays.
The simplest presentation is a verbal report from the assessor to protected area staff or managers, or others who have commissioned the evaluation. A good assessor should be able to include initial analysis within a verbal report. Such an approach is seldom enough in itself – and some more permanent record of the assessment should be available – but is valuable in that it allows immediate questions and feedback, which in turn can influence the final written report. A verbal presentation is often a good way of getting results out to the people who matter as quickly as possible. In cultures where oral presentations are given more weight than their written equivalents, the verbal report may be the form in which most users receive the results and it is therefore important that this also contains key recommendations and suggestions for adaptive management.
Verbal presentations can be greatly strengthened by providing speaking notes, PowerPoint presentations or overheads. These are particularly valuable if some of the audience are not hearing the presentation in their first language as it will help them to follow what is being said. A PowerPoint can be printed as a permanent record, or passed to other people in electronic form. If assessors leave behind a good PowerPoint presentation it can also be used by people who have heard the presentation to summarise results for other audiences.
Written reports are usually an important part of evaluation. Many reports effectively remain as internal documents for protected area managers and local stakeholders. Part of the process of building institutional support for assessment is making the reporting process a standard part of management rather than greeted with particular fanfare. Reports addressing more controversial issues are likely to attract greater interest. Assessments are increasingly likely to be used as the basis for international reporting to institutions such as the Convention on Biological Diversity.
Whether a report is primarily intended for park managers or for the public, it should usually have a number of standard components:
A clear summary including key conclusions and recommendations;
An introduction that lays out the context of the assessment, why it was carried out, the methodology used and people consulted;
A summary of data and analysis (where the data sets are very large this may be in summary form only or accessible in simple form through the web or on request);
Clear analysis including a description of how this was carried out;
Detailed recommendations.
Analysis should include identification of strengths and weaknesses of management, along with recommendations about how these could be improved. But in many cases it is also worth including consideration of limitations and flaws in the assessment process itself and recommendations for improvements in the future.
Several reports or presentations with different levels of detail for different audiences might be appropriate for one evaluation; for instance a summary report or a press release may be more useful to some readers than the full document. Method of presentation, language and terminology used in evaluations should be commonly understandable, though more technical language will be appropriate for selected audiences. Use of electronic publishing and the internet has helped to spread information more widely; for instance in Finland Metsähallitus has set up a dedicated web page for its evaluation report. The internet can be particularly appropriate for regular reporting and for large amounts of information where people are likely to want to see only a fraction at one time. Distribution of some hard copy reports is often needed in addition.
The politics of reporting is complicated and can backfire. Political problems have arisen in several reporting exercises. For instance in Brazil problems arose when agency staff felt they were being openly criticised and in New South Wales, Australia, a generally positive report on the state's protected areas was selectively quoted in a hostile article by a local journalist. Distorted publicity can be used to undermine support for protected areas. However, such challenges face any institution reporting on their performance and careful management can minimize risks of misrepresentation or political backlash. Key steps include:
Involving all participants in an assessment so that they know the contents of a report and have had a chance to comment before it becomes public (which doesn't mean that the assessors should necessarily accept all comments);
Ensuring that the protected area agency has a clearly identified strategy for addressing key concerns and implementing recommendations before the report is released;
Trying to ensure a public release to many people rather than leaking the results gradually over time (although this may conflict with the need to involve stakeholders).
Generally transparency is the best option. But this does not necessarily mean that everything should be public. Careful thought needs to be given to what results should be reported outside a ‘confidential’ audience: for example, scores or comments that relate directly to individuals might be grouped or otherwise reported to avoid potential repercussions on participants and the undermining of future assessments.
Evaluators should also be aware that spending months to conduct detailed analyses and produce attractive reports might be futile if the evaluation is out of date by the time it is disseminated. “Note that if early results show that current management is failing to achieve the objectives, it is essential that decision makers get the facts [in a timely fashion]…and know what needs to be done to improve management. If the results of evaluations don't get back to and influence those who can change ongoing management, the benefits of the evaluation can be lost.”56
The main purpose of protected area evaluations should be to encourage and strengthen adaptive management. Implementation is in the form of actions such as responding to threats, increasing local participation or strengthening financial management. Although we tend to think of adaptive management as something involving primarily protected area managers and their staff, some actions may also be needed at a higher level, for example within a protected area agency or government.
Two key factors determine whether evaluation findings will make a practical difference to management: (1) a high level of commitment to the evaluation by managers and owners of the protected areas and (2) adequate mechanisms, capacity and resources to address the findings and recommendations.
Depending on what aspects of protected area management the evaluation has identified as needing improvement – or as being vital to continue – the cooperation of a range of people and institutions might be needed to improve management effectiveness. This may not be the responsibility of the evaluation team, but the step of identifying who should be responsible for implementing each recommendation is important. There may need to be considerable thought put into how the relevant people can be not only informed, but where necessary persuaded or obliged to do their part in improving management. This is often a ‘missing link’ in ensuring that evaluations lead to better management – for instance if on-ground staff are keen to see improvement but senior staff block changes, or vice versa – little might be achieved, even if the capacity for improvement is present.
A few examples of mechanisms to gain commitment are shown in Table 8 (note that some will be appropriate only for organizations which are free to engage in public and political advocacy).
The success of implementation relies on the support of protected area staff at all levels, who need to see the assessment as a positive tool for them to use rather than a judgemental interference in their work. In “learning organisations” adaptive management is seen as a normal process and people are keen to learn from mistakes or problems. However, such a perception is easier to write about than to achieve and experience suggests that in many situations staff initially react quite negatively to the idea of assessment. Support comes gradually over time and once positive benefits are seen. Reporting correctly and sensitively is a key element in this process of building trust.
Apart from willingness and commitment, capacity is the other primary factor influencing the extent to which the recommendations of evaluations are implemented. Capacity includes knowledge and skills; and availability of resources including time, money, equipment and facilities.
When capacity seems likely to be a limiting factor in improving management effectiveness, several approaches are possible:
Shortcomings in skills and knowledge can be addressed by targetted training; assistance from volunteers; and cooperation with external scientists or other partners;
Additional resources (additional government funding or external funding; donated equipment etc) can be sought from a range of sources. The evaluation findings should assist in identifying the needs and in showing how additional resourcing will improve management;
Existing resources can be reallocated to different tasks or areas according to recommended priorities;
Cheaper or less resource-hungry methods of carrying out some management activities can be sought. For example, sometimes sharing of equipment, facilities or human resources can be very efficient; and
It can be accepted that some recommendations, while worthwhile, may not be implemented immediately, but can be kept in mind if capacity increases.
Table 8. Examples of strategies to ensure recommendations are implemented
An important output from an evaluation process can be a capacity development plan. Directly basing such plans on evaluation results can result in a useful and robust approach to improving actual management effectiveness, by:
Ensuring relevance: The assessment ensures that actions to strengthen capacity focus directly on the most debilitating management weaknesses and urgent threats.
Identifying priorities: Depending on the scale of the evaluation, it may result in a prioritisation of the most vulnerable and threatened protected areas in a system.
Engendering support: By engaging with key stakeholders throughout the process of assessing management effectiveness, prioritising relevant capacities, and developing an inter-institutional plan, this approach encourages broad-based support among multiple actors.
Field monitoring © Robert Ashdown, QPWS
Strengthening capacity to effectively manage national protected area systems Experiences from Mexico
Management effectiveness assessments can provide a means of focussing capacity investments on key conservation issues. Methodologies which focus on assessing national systems of protected areas (such as RAPPAM) link the most urgent threats and relative biological importance of individual protected areas into national prioritisation exercises for conservation actions.
The Mexican Protected Area Agency (Comisión Nacional de Áreas Naturales Protegidas), Pronatura, TNC and WWF, are designing a national strategy to strengthen protected area management based on an integrated assessment of capacity needs and management effectiveness. Pronatura, a Mexican NGO with extensive expertise in protected areas, social issues and evaluation methodologies, is coordinating the effort. Mexico's objectives for designing their national capacity strengthening strategy are to:
Achieve broad consensus on the highest priority capacity-development needs.
Design a plan with broad ownership of different public, NGO, and community actors for strengthening capacity to address those needs.
Fulfil their commitment to the Convention on Biological Diversity's Programme of Work on Protected Areas.
This process includes a series of seven regional workshops with preparatory interviews beforehand. These regional workshops will identify:
Trends in threat, management weaknesses, and relative biological importance.
Key conservation issues that need to be resolved based on these factors.
Capacities needing strengthening to be able to resolves these issues.
Opportunities and activities for addressing these capacity needs.
A national roll up of the seven sub-national workshops will identify common needs that can be best addressed by nationwide activities. The result of this process will be a national and series of sub regional inter-institutional work plans to strengthen effective management of the national protected area system.
53Stephen Woodley, pers.comm. (2004).
54The Nature Conservancy (2000).