Best Practice Evaluation

Evaluation Framework

Indigenous Evaluation Strategy - The Productivity Commission

The Productivity Commission’s Indigenous Evaluation Strategy sets out a comprehensive agenda for the evaluation of programs and services for Indigenous people and communities, consolidating previous thinking in this domain.

Intended for use by government agencies, but with broader relevance in the services sector, the Strategy, “sets out a new approach to evaluating policies and programs affecting Aboriginal and Torres Strait Islander people aimed at improving the quality and usefulness of evaluation. It puts Aboriginal and Torres Strait Islander people at its centre, and emphasises the importance of drawing on the perspectives, priorities and knowledges of Aboriginal and Torres Strait Islander people when deciding what to evaluate and how to conduct an evaluation.”

Aligned with the National Agreement on Closing the Gap, the Strategy calls on Australian Government agencies to disseminate, synthesise and translate evaluation findings in accessible forms that can be used by Aboriginal and Torres Strait Islander people to support their self-determination and their active participation in decision making that affects them.

The Strategy also advocates for a new Office of Indigenous Policy Evaluation (OIPE) to be established in partnership with an Indigenous Evaluation Council with all Aboriginal and Torres Strait Islander members, to oversee the transition to improved Indigenous evaluation.

Health and wellbeing are proposed as policy domains for initial focus under the Strategy, along with education, housing, land and water, justice, culture and languages, family and child safety, and employment.

The Strategy calls for credible, useful, ethical and transparent evaluation of Indigenous programs and services and addresses:

What to evaluate

Indigenous people decide what policies and programs have the greatest impact on their lives and should be subject to evaluation.
Evaluations consider the impacts of policies and programs on Aboriginal and Torres Strait Islander people.

Evaluation planning, design and conduct

Evaluations draw on the perspectives, priorities and knowledges of Aboriginal and Torres Strait Islander people and communities.
Mainstream policies and programs routinely consider impacts on Aboriginal and Torres Strait Islander people and are evaluated accordingly.
Indigenous people decide how to be involved in evaluations, including sufficient time and resources for meaningful, respectful and culturally safe engagement.
Indigenous people are included in data planning, collection and use, and in data governance.
Evaluation reflects the diversity of Aboriginal and Torres Strait Islander people.

Reporting evaluation findings

Indigenous people engage, partner or lead in translating and publishing evaluation findings so they are meaningful, accessible and useful, and publications describe this contribution.
Evaluators share their findings with Indigenous participants.

Building capability and a culture of evaluation

Evaluators incorporate Indigenous knowledges into their thinking.
Evaluation processes strengthen evaluation capability among Aboriginal and Torres Strait Islander people, organisations and communities.
Agencies emphasise cultural capability in their staff, and foster shared understanding with Indigenous people about evaluation priorities.
Agencies encourage feedback about evaluation from Indigenous people.

The Strategy is accompanied by an Evaluation Guide that offers practical advice on evaluation of both Indigenous-specific and mainstream programs and services in Indigenous contexts.

CBPATSISP Evaluation Framework

Background

Rationale

Indigenous Goals, Principles and Rights

Background

Rationale

Indigenous Goals, Principles and Rights

Evaluation Resources

This section includes a range of national and international resources including reports on monitoring and evaluation methodologies and strategies to promote and support best practice in suicide prevention programs and services as well as strengthening community engagement, Indigenous governance and self-determination. It includes a range of culturally appropriate tools and techniques that are applicable to use in various settings including in remote, rural and urban contexts when conducting evaluations with Indigenous community controlled organisations. There are also a range of guidelines and resources to ensure mainstream services and organisations adhere to Indigenous best practice principles and guidelines. The resources encompass a practical, feasible, culturally responsive and user friendly approach to monitoring and evaluation in diverse Indigenous community contexts.

National - Reports

Fisher, S. (2012) Monitoring and evaluation methodologies for remote settings: A literature review conducted in 2010 Report to the Office of Indigenous Policy Coordination, Department of Families, Housing, Community Services and Indigenous Affairs Working Paper

This report was undertaken by Ninti-One and provides useful material for staff working on new initiatives in monitoring and evaluation. Drawing on a review of the literature this report provide an overall analysis and recommendations regarding the conduct of monitoring and evaluation in Indigenous remote communities.

Guenther, J, Galbraith, M 2014, Learning from evaluations of school-family strengthening programs: lessons for all

Abstract: For more than 10 years, a program called Families And Schools Together (FAST) has been run in schools across the Northern Territory. These programs have always had an evaluation component built in. However, over time, the evaluations have changed. Initially, they were summative, built around a quantitative psychometric tool (with a positivist research paradigm). The intent of the summative evaluation was to demonstrate the effectiveness of the program. However, as the program was rolled out in remote contexts, the need for adaptation was recognised. Changes were made but it was soon recognised that other outcomes, not captured or explored in the methodology, were emerging. After six years of working together, the evaluator and the program manager felt that it was time to explore effectiveness in a different way with a more qualitative evaluation process (based on naturalistic and pragmatic paradigms). The purpose of this paper is to share learnings from this experience with other program managers and evaluators. The presentation will be an opportunity for participants to engage in a discussion about monitoring and evaluation from a program evaluation perspective, particularly taking account of the complexities of the northern Australian context. This paper explores the six year learning journey through evaluation that the evaluator and program manager have undertaken. It describes tensions between the need for reliable and generalizable objective quantitative data and the need for authentic and credible data based on participant experience.

Maughan, C 2012, Monitoring and evaluating social impacts in Australia, Ninti One Limited

Abstract: Monitoring and evaluation (M&E) allows people to learn from past experiences, improve service delivery, plan and allocate resources and demonstrate results as part of accountability to stakeholders. M&E also assists in keeping projects on track, providing a basis for reassessing priorities and creating an evidence base for current and future projects There is a growing interest in the measurement of social impact across the business, government and nonprofit sectors. In recognising the role that non-profit organisations play in ‘enhancing the economic, social, cultural and environmental wellbeing of society’, the Australian Government has recently focused on improving the measurements of social impact This report describes the main monitoring and evaluating frameworks and methods used in Australia which include some assessment of social impact. Each of the following are discussed in terms of an overview of how it works (in terms of M&E), the benefits and limitations, examples of organisations using the technique and where to find further resources.

National - Journal Articles

Rogers, Alison, Harrison, Nea, Puruntatameri, Therese, Puruntatameri, Alberta, Meredith, Joan, Dunne, Rachel 2018, Participatory evaluation is the sea eagle looking “long way wide eyed” https://journals.sagepub.com/doi/abs/10.1177/1035719X18778712

Abstract: Participatory evaluation can be embedded in programs to support good governance and facilitate informed decision making in Aboriginal communities in remote and urban contexts. An Aboriginal Elder from the Tiwi Islands in the Northern Territory of Australia described participatory evaluation as a sea eagle looking “long way wide eyed.” The metaphor refers to the long-term and broad approach undertaken when a complex community development program used participatory processes to build evaluation capacity and solve problems. The evaluation approach ensured the program was inclusive, responsive, empowering, and resulted in direct benefits for the communities. This article addresses the lack of literature on applying developmental and empowerment evaluation approaches in practice by describing the methods, tools, and use of evaluation findings. The value of participating for the community members and partner organizations is shared and the benefits and implications for participants and the evaluator are discussed. The authors hope this article inspires practitioners and evaluators to consider participatory ways of working with communities to support community directed action and social change.

Evaluation Tools & Guides

The Evaluation Framework provides Governments and other key stakeholders with tools and a series of guides to assess the effectiveness and cultural responsiveness of proposals and to further progress the implementation and evaluation of programs, services and resources.

The following series of guides outline:

The relevant methodologies to conduct culturally appropriate and comprehensive assessments of programs, services and interventions with regard to rigour, program impact, and community and stakeholder responses to the suicide prevention programs
Culturally relevant indicators to measure the effectiveness of programs and services addressing social and economic determinants on Indigenous social and emotional wellbeing outcomes
The steps and key considerations for
- communities and organisations to plan, implement and evaluate their own community level suicide prevention activities;
- governments and program funders to evaluate suicide prevention proposals;
- health practitioners to evaluate clinical services, protocols and practices.
Tools and resources relevant to each of the guides.

Key Concepts in Conducting Evaluations

Developing Whole of Government Approaches

The most effective suicide prevention involves both whole of community and whole of government approaches. A whole of government approach encourages and facilitates the implementation of comprehensive suicide prevention programs through the development of program linkages and partnerships between government service providers, Indigenous organisations and peak bodies in the delivery of Indigenous programs and through the specification of the goals in program evaluation criteria.

The following guides include key considerations and to assist communities and governments to develop genuine partnerships and dual accountability measures to plan suicide prevention initiatives and to assess the effectiveness, cultural appropriateness and the efficiency of delivery of specific programs and program providers.

Initiating System Level Change

From a government and Indigenous leadership perspective, positive change requires system level change based on evidence of ‘what works’ to promote effective and sustainable suicide prevention. The findings of ATSISPEP literature review confirm that to effect system level change to reduce suicide and self-harm will require:

Genuine engagement with Aboriginal people, communities and organisations to promote sustainable, culturally responsive and effective services and programs
Greater interagency understanding of, and commitment and capacity to address, the variability of human capability and community readiness in Indigenous populations as a consequence of risk factors such as poverty, illness, incarceration, low self-esteem, trans-generational trauma and its relationship to suicide and self-harm
Greater support for families and communities by directly strengthening the protective factors and implementing programs that contribute to pathways to recovery
Resources and strategies that strengthen the commitment, understanding and capacity among mental health, health, education, justice, child protection and social services and their staff
Strategies and policies that are designed to reform and transform systemic, organisational and individual racism and discrimination and contribute to the development and embedding of cultural capabilities and competence across all relevant services and professions (Calma & Dudgeon, 2009; ATSISPEP, 2015; Walker, 2010, Walker et al., 2014)
Workforce development, Indigenous health curriculum and Indigenous governance and leadership
Primary and tertiary services that are culturally responsive, flexible increase access and integration and partnerships

The CBPATSISP Evaluation Framework provides an overarching mechanism to demonstrate the extent to which these elements are implemented, monitored and evaluated by the relevant government and non-government agencies to successfully achieve goals of the Fifth National Mental Health and Suicide Prevention Plan, relevant jurisdictional implementation plans and the renewed Social and Emotional Wellbeing Framework 2017-2023.
This requires State and Commonwealth Governments to genuinely engage with Indigenous Australians to develop local, culturally responsive approaches to identify and respond to individuals, families and groups most at risk within Indigenous communities, and support the evaluation of the cultural responsiveness and effectiveness of programs initially funded through the National Aboriginal and Torres Strait Islander Suicide Prevention Strategy, and now Primary Health Networks (PHNs).

Assessing Effectiveness and Cultural Responsiveness

Government and industry organisations generally specify a range of Key Performance Indicators to assess the effectiveness, (cost effectiveness), efficacy, efficiency and cultural responsiveness and safety of programs they fund. Assessing the effectiveness of implementing suicide prevention programs and strategies is a rigorous approach to ensure that what a community or organisation is doing or plans to do actually works.

In Aboriginal and Torres Strait Islander contexts it requires the following steps or considerations:

Identifying which strategies will be most likely to reduce self-harm and death from suicidal behaviour
Determining the potential effects of cultural, social, historic, legal, ethical, and economic factors on those strategies
Determining the most effective, culturally responsive and culturally safe methods for implementing those strategies
Assessing the effectiveness of the strategy in achieving its intended outcomes as it is developed and implemented

The CBPATSISP Evaluation Framework outlines steps to conduct formative, ongoing, continuous and summative stages of evaluation, as well as process and outcome approaches throughout the development, implementation and evaluation of the project. It includes:

questions and mechanisms to assess the cultural responsiveness and effectiveness of initiatives at each of these phases
a set of questions to be asked in determining the relevance of a proposed program, service or initiative and their alignment with the goals of relevant government strategies
a set of questions to determine the extent to which the project or activity can provide evidence that the longer term benefits of the outcomes are likely to outweigh the costs.

The higher the level of risk of the target population, the more intensive the prevention effort is required and the earlier it needs to begin. The following considerations are crucial for policymakers and program funders in determining both the effectiveness of suicide prevention strategies and policy and funding priorities when making decisions regarding:

Potential to reduce or avoid self-injury or death
Social, legal, and ethical and economic impact
Best methods to ensure cultural safety

Primary Considerations for Funding Bodies

Funding agencies, service providers and communities also need to know ‘what works’ in developing/adapting and implementing suicide prevention programs and services to address community identified needs and priorities outlined in the ATSISPEP. This requires being able to identify the features of effective, culturally responsive programs, services and initiatives in order to achieve successful outcomes.

The most important things funding providers need to consider in the first instance include:

The delivery approach of the program or initiative
The formation of partnerships
Integration and co-ordination of interagency and inter-sectoral services
Capacity building (of staff and of groups /communities intended to benefit)
Targeting approaches (which population group/s is/are involved? and whether universal, indicated or targeted approaches to suicide prevention are required)
The use and quality of evidence and research in establishing the rationale for the proposed initiative

Questions

What works best for whom, why and when?
Does the service, program or initiative help achieve outcomes in other relevant and related program areas? (eg. Are there links with the drug and alcohol strategy and/or through interagency and/or other government programs?)
Does the service, program or initiative include strategies to address the local contextual issues that may hinder the effective implementation of key Aboriginal and Torres Strait Islander priorities in the Fifth Mental Health and Suicide Prevention Plan in achieving its objectives and outcomes?
Does the program or activity include evidenced based ‘what works’ elements that are likely to make a positive difference as outlined in the ATSISPEP Solutions That Work Report?
Is there a sound theory or evidence base (which addresses social determinants and casual pathways) to help explain why some initiatives are most likely to work?
To what extent does the program or activity take account of the key principles underpinning the CBPATSISP Evaluation Framework and key elements identified in community consultations (see table 1)?
Is there an evidence-based rationale to demonstrate how the project goals and objectives will address the risk and protective factors for a specified group, local community or whole population?
Is there evidence of strong community consultation and leadership in determining the issues and priorities being addressed in developing, implementing and evaluating the program?

Assessing the Cultural Appropriateness of Programs

The following considerations are important in assessing the extent to which the goals, aims, purpose and outcomes of suicide prevention programs and services (including specific, mainstream, indicated and targeted) are culturally appropriate for Aboriginal people.

Evaluators need to ask to what extent:

Does this program or service focus on all or some aspects of the physical, spiritual, cultural, SEWB of an individual, their family and community or an individual within specific risk groups (ie LGBTQI+SB)?
Do workforce initiatives encourage and resource frontline workers and practitioners to:
- focus on the physical, spiritual, cultural, SEWB of the individual, family and community?
- recognise trans-generational trauma and align with the SEWB principles and principles of empowerment?
- strengthen cultural values and commitments, family and kinship systems of care, and Indigenous control and responsibility as an intrinsic aspect of healing and facilitating cultural, SEWB?
- acknowledge and work in partnership with the Aboriginal community-controlled sector and facilitate Indigenous people’s right to determine the types of services they receive?
Does the program or service or workforce initiative
- have a statement of commitment and strategies to address racial discrimination?
- support human rights and social justice principles?
- support Indigenous organisations cultural governance/self determination?

Program and service providers or external evaluators undertaking the evaluation need to work with Aboriginal people (as co-designers) to assess the quality of the program plan including the evaluation to embed key principles and culturally appropriate methods and methodologies from the beginning.

Assessing Project/Program Sustainability

A range of risk factors can impact on the sustainability of a program or initiative regardless of its effectiveness. These include:

the short-term nature of most funding programs
difficulties in obtaining and retaining suitable personnel and
poor participation rates due to lack of transport and family constraints
lack of community involvement in the program development
many initiatives that are piloted are not adequately evaluated or further funded.

Community ownership of, and genuine partnership and contribution throughout all stages of the development and implementation process are fundamental to the program and can help offset these issues.

To achieve sustainability organisations delivering a program or initiative are encouraged to consider the following questions:

Does the project or initiative ensure a culturally responsive, localised and stable Indigenous control and governance model?
Is there a long-term vision, leadership and support for the specific initiative or project within the community, region or state?
Has there been broad and ongoing Indigenous community consultation and strategic planning?
Is there evidence of genuine engagement with community groups and the community controlled health sector as well as with relevant stakeholders and institutions?
What collaborative activities have been undertaken to support and strengthen new and existing relationships with partner organisations?
Have program funders and service providers
- contributed to constructive and preventive ‘upstream’ investments to achieve positive goals that contribute to Indigenous wellbeing, promote resilience and reduce suicide?
- incorporated an appropriate set of indicators to evaluate the effectiveness of a program or initiatives’ short, medium and long term outcomes?

Guide to Planning & Evaluating Community Initiatives

This section outlines steps for community groups, community controlled organisations and services to work in partnership to plan, implement and evaluate suicide prevention initiatives using Participatory Action Research (PAR) processes to ensure that programs are responsive to the local culture, context and specific issues.

PAR is a key component in planning, developing and delivering effective community-led suicide prevention programs and services for Indigenous peoples. It involves:

Developing a culturally safe environment for program participants
Fostering trusting relationships between the community and stakeholders

Establishing trust and building respectful relationships and genuine partnerships is an important first step in gaining community support, facilitating community ownership and engagement, within the broader community, and uptake of, suicide prevention programs and services.

PAR provides a rigorous and systematic approach to address issues of cultural and social relevance for communities. The demonstrated benefits of using PAR include:

Strong engagement of the community by way of increased participant recruitment and program retention rates
Validated cultural measures
Reduced reporting bias
Enriched interpretation of research findings
Increased translation of findings into action
Greater ownership of the research and immediate transfer of knowledge into practice

The key considerations in planning and conducting an evaluation include:

Applying Empowering Evaluation Methods and Methodologies

Community development and PAR provide an important information resource and an enriched evidence base for engaging in community-wide planning, identifying relevant programs and initiatives and involving families and communities in developing, implementing and evaluating programs and services. These methods also enable national and state jurisdictions and funding bodies to support local, community-driven suicide prevention strategies and plans.

Key benefits of implementing the CBPATSISP Evaluation Framework using these culturally appropriate and responsive methods and engagement with the CBPATISP website include enabling communities to:

Build a local and national evidence base drawing primarily on Indigenous knowledge, experience and perspectives
Undertake comprehensive assessments of the overall comparative efficiency of initiatives
Produce information about ‘what works for whom and why’ to guide policy development, initiative and project implementation
Provide evidence of existing gaps to reform reporting and accountability requirements
Promote learning by communities /organisations proposing or implementing projects
Document and share with stakeholders emerging understandings of barriers, enablers and effective strategies and good practices in suicide prevention, early intervention and postvention
Provide compelling evidence of what does not work for Indigenous individuals, families and communities as well as what impacts adversely on Indigenous controlled services or training programs.

Evaluations that identify ‘what is working’ and how to enhance and continuously build on program and service outcomes can directly enable greater effectiveness of programs.

Establishing a Culturally Responsive Evaluation Process

The main steps involved in establishing a culturally responsive evaluation process are outlined in Figure 1. These processes provide an opportunity for community groups and community controlled organisations (often in partnership with program and service providers) to use community development and PAR approaches in the planning, implementation, monitoring and evaluation of suicide prevention activities.

These processes enable communities (and partnership agencies and services) to regard their community suicide prevention initiative as a unique opportunity to bring about profound and positive change for Indigenous communities.

The teams planning and conducting evaluations are encouraged to:

Use a qualitative yarning approach involving Indigenous researchers/consultants for data collection, interpretation, analysis and reporting
Give priority to Indigenous perspectives, values, knowledge and experiences using in-depth unstructured conversational interviews and facilitated focus group discussions
Collaborate with existing project partnerships using consultation processes and agreements established with Indigenous communities throughout the project development and implementation

The actual specifics of the evaluation methodology and indicators, specific questions and data collection and communication reporting format need to be negotiated with Indigenous communities at an early stage, taking into account the local community contexts.

Conducting Community Planning and Evaluation

The key tasks for conducting a culturally responsive community planning and evaluation process include:

Establishing a local reference group with relevant community organisations and members who have appropriate knowledge, skills and understanding to provide advice on the evaluation process
Negotiating to include appropriate qualitative and quantitative methodologies and reporting arrangements with all relevant stakeholders groups
Building a comprehensive understanding of the key issues and factors in the specific community contexts for specific populations of concern (eg. children, young people, men, families, perinatal mental health, people in incarceration, young people in out of care, and LGBTQI+SB or other vulnerable groups)
Building an understanding of the risk and protective factors around suicide generally and in the local context by:
- consulting with relevant governmental staff and other stakeholders at the community level
- working collaboratively with groups to identify and conduct appropriate ‘information rich’ case studies using culturally responsive methods (ie storytelling, yarning, focus groups etc)
Referring stakeholders to the website resources and information on what is available to assist communities and services
Reviewing and analysing relevant literature on
- background reports and other documentation regarding the project
- ‘What works’ in suicide prevention
- building strong Indigenous families and communities
Disseminating draft findings and recommendations to community and relevant agencies using appropriate information formats for different audiences (ie community feedback sheets, project newsletters, fact sheets and policy briefs for policy and funding personnel).

Figure 2 provides an overview of the steps involved in developing a community-initiated/led, whole of community suicide prevention initiative.

DEVELOPING, IMPLEMENTING AND EVALUATING COMMUNITY SUICIDE PREVENTION

Using the CBPATSISP Evaluation Framework in developing, implementing and evaluating programs, services or specific initiatives (universal, targeted) teams need to consider the extent to which the community engagement and planning processes are:

Culturally responsive and secure to support Indigenous peoples’ involvement and leadership in all phases of the initiative
Incorporate the key guiding principles
Use local language in framing the evaluation.

The next section includes a set of suggested questions in each of the planning, implementation phases to support and assess the development, implementation and evaluation of intended outcomes.

Building the Picture – Formative Evaluation Phase

The first step for community researchers, groups, organisations (and services) involves:

Bringing community stakeholder together to share their stories;
Defining the issues surrounding suicide and self-harm or suicidal risk in their community;
Identifying existing services or programs and their roles in suicide prevention and/or crisis intervention or postvention.

Considerations/Questions

What are the issues being faced by the community?
Why do community members and stakeholders think these issues are happening?
What do people think needs to happen to address these issues?
What are the potential barriers and enablers for this to happen?
What does the new or existing program or initiative hope to achieve?
What services and agencies need to be involved?

Step 2.

Draw a diagram to show the major relationships between groups and issues in the story

Describe how activities relate to desired outcomes (eg. establish strong and secure early years programs, increase young people’s sense of identity, strengthen the role or fathers, mothers and families, provide hope for the future, and connect people to culture and country).

Step 3.

Review relevant examples of community resources for planning and evaluation in suicide prevention listed in the CBPATSISP resources section and on the ATSISPEP website.

Step 4.

Implement the evaluation plan.

Step 5.

Identify assumptions about relationships between activities and outcomes from an Indigenous community’ perspective, make sure all relevant groups and individuals are included in building the picture using culturally responsive and respectful processes and protocols.

Considerations/Questions

How do these ideas/perspectives match/align with existing government program goals and objectives and expected outcomes?

Develop the key elements the community or organisation wants to explore as the story unfolds in order to develop key questions and/or evaluation/value statements.

Extending the Picture – Developing the Baseline

This phase involves drawing together sound/robust observations and interpretations of individual and communal experiences using a range of approaches to explore issues and begin to gather both qualitative and quantitative information using various methods. It requires identifying the sources and timeframes for collecting data and developing a plan and set of tables to guide the implementation of the evaluation.

Questions might include:

What are the key issues that matter to individuals, families and communities?
How can they be included in the program?
How can they be measured and costed?
Are local issues/ potential barriers taken into account?
Are all groups in the community involved?
Is/are local language/s recognised, are all family groups and language groups represented?
Are key people involved?

Consider issues of cultural and community appropriateness in collecting and interpreting information, using approaches that have been identified as appropriate and supporting Indigenous terms of reference including PAR, as well as assessment tools to measure empowerment and change such as Most Significant Change Technique (MSCT), appreciative inquiry, photovoice, art appreciation; bricolage, Growth and Empowerment Measure (GEM) (Haswell et al., 2013) and other community audit tools.

Understanding the Situation - Measuring the Outcomes

This step involves working with relevant local Aboriginal and Torres Strait Islander groups to assist in interpreting the formative information using appropriate techniques for analysing the qualitative and quantitative data (eg thematic narrative, NVivo).

PAR and Continuous Quality Improvement (CQI) can be applied using the knowledge and findings generated through the ongoing evaluation throughout the implementation to improve programs, services and practices. The same processes are used in establishing the summative evaluation findings to meet the different needs of multiple audiences – including the community, policy makers, program and service providers, other relevant stakeholders and the wider community. In this instance the findings and recommendations can be used to support the continuation of funding and implementation of the programs.

This phase seeks to obtain data related to the following major themes:

Questions

Sharing the Story – Disseminating the Evaluation Findings

An important aspect of any evaluation is to celebrate the story of ‘what works’ with the community, as well as reflecting on and learning from those aspects of the story that don’t work, and using the knowledge created through the evaluation to improve programs and services moving forward.

The evaluation can also be used to leverage more funding and to identify what further research is needed to fill gaps in knowledge about suicide prevention in Indigenous contexts.

Examples of promising programs, initiatives and services identified as demonstrating best practice initially identified through the ATSISPEP have been further reviewed and are available on the CBPATSISP Clearinghouse website.

Building the Capacity of Individuals, Families and Communities

In accordance with the Evaluation Framework principles, the first step in establishing strong healthy communities is to acknowledge and understand the devastating and enduring impacts of colonisation on Indigenous people’s contemporary lives. It also requires understanding the various pathways necessary for healing from historical trauma, using both cultural and contemporary understandings and processes to support social and emotional wellbeing. Establishing culturally responsive community, family and individual support systems and programs to promote pathways to recovery through dealing with loss, grief and disconnection, trauma and helplessness, and powerless and lack of control is essential to reduce mental health issues, high risk behaviours and suicide. This in turn requires:

Self-determination and community governance
Reconnection and community life
Restoration and community resilience

Although there is considerable overlap within the three themes, it is important to address each one specifically as well as all three collectively to promote healing and recovery across generations. The interrelatedness of the ways in which the ongoing history of transgenerational trauma impacts on individual, family and community social and emotional wellbeing, and specific pathways to recovery, are depicted in Figure 24.2 below (Milroy, Dudgeon & Walker, 2014, p. 424).

The diagram shows how each of the pathways to recovery outlines a conceptual framework and theory of change for program development and, in combination, provide a holistic framework to support community life and development, which include both the types of questions required to be addressed in developing, implementing and evaluating programs and services and the types of promising programs that are useful for community groups to consider in pursuing each of the respective pathways in re-establishing individual, family and community social and emotional wellbeing.

The figure depicts how each of the circles encompass the specific pathways to recovery related to each of three trauma themes. In combination they affirm that recognition of cultural strengths and facilitation of Indigenous ways of working, leadership, healing and empowerment is fundamental to promoting sustainable recovery. At the centre, each of the pathways of recovery come together to provide the holistic sense of health and social and emotional wellbeing for individual, family and community (Milroy et al., 2014).

Facilitating Indigenous Governance and Self Determination

It is important that groups, community and organisations have the means to achieve the things that matter to them. Working together to answer the following five questions should help you understand who you are as a group and what kind of governance they want to model to their children. Communities need to work out what their governance strengths and challenges are. One of the best places to start is with community members. The website includes a report outlining an Indigenous Governance Framework and an Indigenous Governance Toolkit. Questions worth considering include:

Considerations/Questions

Who are we? What do we mean by community governance?

These questions are all about identifying which community, region or group you are part of, and who you are representing through your governance. Getting some agreement or resolution about these questions is a fundamental step, right from the start.

Where did we come from? What’s the history of our governance?

This question is about looking back at the history of your governance and considering past practice and protocols and expectations.

What have we got now? What is our current governance like?

This requires mapping all points of decision making in the community, considering issues of representation family and language groups and gender equity.

What do we want? What future goals are we trying to accomplish for the future, how does this.

Answering this question requires thinking about how the community can build on strengths to address the key issues to reduce suicide. It will help develop a strategic vision for your governance.

What kind of governance are we trying to build for our group, our members, our children and the generations to come?
What do we hope will be different or better about our governance arrangements?
What do we want to stay the same?

How do we get it? What is our strategy?

Answering this question is about considering how you can make your future vision happen.

What are our specific concerns and priorities?
What plans and actions should we develop and take?
Who will be responsible for doing it, and by when?
What resources do we need?
What are the risks involved and how can we deal with them?
How we are going to tell if we are making progress and getting the outcomes we want? (see the Indigenous governance toolkit website)

Healing the Trauma

It is now widely recognised that the adverse effects of trauma on individual, family and community wellbeing is contagious and traumatic events effect whole communities because a community is a dynamic system that acts and responds as a whole ‘body’ (Sheehan et al., p.50). Several studies confirm the need to focus on healing to address trauma in order to improve Indigenous health, mental health and wellbeing outcomes and prevent suicide (Atkinson et al., 2014; Milroy et al., 2014; Powell 2014; Anderson, Baum & Bentley 2007). These studies emphasise the need for health promotion or education programs in suicide awareness and prevention to focus on healing and empowerment to help people gain a greater level of control over their lives and circumstances (Atkinson et al., 2014; Dudgeon et al., 2010; Healing Foundation 2014; Powell et al., 2014). What these different programs confirm is the importance of governments recognising Indigenous peoples as capable and efficacious individuals, as resilient, creative, adaptive with cultural knowledge, and wisdom to determine their own solutions.

Resources for Healing, Recovery and Community and Workforce Capacity

Resources for Healing, Recovery and Community and Workforce Capacity
There are a range of validated high-quality resources that are valuable for local mental health and social and emotional wellbeing workers addressing transgenerational trauma available on the CBPATSISP website and the Healing Foundation website:

‘What’s up with My mob?’
Community members, practitioners and service providers engaging in suicide prevention will find a range of culturally responsive strategies, principles and practices to strengthen Indigenous mental health and social and emotional wellbeing in the following resources
Working Together: Aboriginal and Torres Strait Islander Mental Health and Wellbeing Principles and Practice (eds. Dudgeon, Milroy & Walker 2014)

Guide to Evaluating Organisations and Services

This section includes questions and considerations for determining relevant evaluation criteria for assessing the effectiveness and cultural responsiveness of programs and services, applying Continuous Quality Improvement (CQI) and for making decisions about funding and resources in developing programs.

Organisations need to be resourced to develop their capacity for undertaking action research evaluation and/or CQI processes through building an evidence base and promoting and informing best practice in suicide prevention within the public health sector.

It also includes evidence based principles, processes and resources for Aboriginal community controlled organisations, government and non-government organisations and services to use in planning, implementing and evaluating the effectiveness and culturally responsiveness of suicide prevention and early intervention initiatives.

All suicide prevention and early intervention initiatives, program and service delivery models implemented in Indigenous contexts need a sound theoretical base or theory of change, appropriate conceptual frameworks, program logic and social indicators and performance measures to assess the effectiveness of programs, practices, services and delivery systems designed to work with Indigenous children, young people, families and communities to:

Strengthen their capacity, leadership and empowerment
Support their cultural and social and emotional wellbeing
Address the social and economic determinants that are key risk and protective factors

The key considerations in conducting organisation and service evaluations include:

Planning

A clear understanding of the extent of issues

The population they are intending to ‘target’ or work with
A sound theory and logic for the program service or initiative they are evaluating

Step 1: Identify and prioritise issues for action
Step 2: Understand the issue
Step 3: Plan your program or service
Step 4: Implement and adapt your program or service
Step 5: Evaluate your program or service

What follows then is a set of questions or criteria to measure the key elements and principles identified as essential for partner organisations, program and service providers and policy personnel to determine whether programs and services have:

Increased the accessibility of services for people at risk of suicide or self-harm
Improved the cultural responsiveness and effectiveness of service delivery
Improved outcomes and influenced positive changes in the social determinants that contribute to poor mental health and wellbeing outcomes, high risk factors and suicidal behaviours, and suicide.

Broader questions also need to be answered to assess the efficacy, integrity and effectiveness of programs, services and strategies designed to reduce Indigenous suicide and address the associated risk and protective factors in complex and diverse circumstances. The key elements that are operationalised include:

Adopting a social determinants approach
Assessing initiatives to address racism
Embedding cultural responsiveness and cultural safety
Ensuring dual accountability and genuine partnership
Facilitating Indigenous governance and self-determination
Incorporating human rights
Facilitating empowered and socially sustainable communities
Strengthening protective factors
Supporting the healing of trauma, grief and loss
Establishing healthy communities
Developing/identifying resources for individual community healing and recovery
Developing/identifying workforce training and resources
Assessing program sustainability

Evaluating Continuous Quality Improvement Processes

Communities and organisations need to have the capacity to evaluate programs to enhance their understanding of what works, under what conditions and why in particular contexts. Most organisations would benefit from having the support (by way of resources, knowledge and understandings) to undertake cycles of ongoing review and reflection. Continuous Quality Improvement (CQI) allows program and service providers to change things that are not working throughout their implementation, and to embed and document things that are working.

Adopting a Participatory Action Research (PAR) approach, which is closely associated with CQI, enables organisations and services to engage in, and benefit from, action learning processes. This involves a process of critical reflection and a cycle of action, observation, reflection and planning to ensure program and service delivery is continuously improved. The value of measuring performance is in both monitoring progress towards achievement of agreed outcomes and in providing feedback on effective design and implementation of interventions. This continuous cycle of monitoring, review and refinement is illustrated in the figure below extracted from the Australian Health Department website below.

Enhancing Organisational/Community Capacity through CQI

To ensure sustainable, culturally appropriate and effective service and program delivery the principles underpinning the Evaluation Framework need to be observed. These include

community negotiation
community engagement
strengthening community capacity and community control and
self-determination.

These elements are critical to embed organisational cultural competence and cultural safety in a sustained way.

For examples of Best Practice programs and services please see the Yirriman programs on our website.

These programs highlight many of these important effective characteristics of developing and implementing suicide prevention programs through Aboriginal community controlled organisations.

Evaluators need to ask:

How can the service, program or initiative achieve better outcomes?
What strategies and resources are required to ensure processes, practices and policies are effective and culturally appropriate to achieve improved outcomes in the short, medium and long term?
What are the most appropriate outcome measures to demonstrate the effectiveness of the service, program or initiative?

Health Promotion CQI Evaluation Tools

The One21seventy Health promotion CQI tools are an evidence-based resource package for use in Indigenous community settings. Health promotion tools and resources for the prevention of Indigenous suicide, self-harm and high risk behaviours can be successfully used by multidisciplinary teams with limited health promotion experience. The tools are accessible here.

The CQI tools enable communities and service providers to:

Systematically describe and assess how well activities and projects align with good practice
Assess how well organisational systems are functioning to support health promotion
Plan how to improve systems that support good health promotion practice

Incorporating Cultural Indicators in Evaluation

The Evaluation Framework aims to ensure Indigenous cultural values, knowledge and terms of reference are respected, recognised and incorporated into all health, mental health, social and emotional wellbeing and other associated programs and services, including universal, targeted and indicated strategies to reduce suicide.

The use of PAR combined with Aboriginal-led and negotiated CQI in evaluating organisations and services ensures that Indigenous voices are given primacy and authority in determining the most appropriate social and performance indicators to measure improvements in processes and outcomes and restoring/maintaining Indigenous cultural protocols, practices, priorities and aspirations for the future.

Assessing Initiatives to Address Racism

There is growing evidence of the impacts of racism on health and SEWB outcomes and the subsequent association with poor mental health, suicide and self-harm. The experience of racism and social exclusion has been shown to adversely impact on wellbeing outcomes among Aboriginal children at school, adults in the workplace and when accessing health, education and social services contributing to depression, despair and anxiety.

ATSISPEP confirmed that Indigenous people continue to experience institutional racism through the language, practices and processes of social policies and programs which impact upon every aspect of their lives. These findings highlight the need for health and mental health stakeholders to address racism, discrimination and marginalisation and enhance cultural competence in programs, services and initiatives to support suicide prevention and interventions. All communities and community controlled partner organisations have the right to expect state and national government organisations and services to address issues of racism and discrimination as part of their core business.

Skilled, culturally aware leadership and clear positive communication about the importance of addressing race-based discrimination and its impacts is critical to support individual and organisational level change and reduce any resistance.

Institutional racism needs to be addressed at three levels:

Individual (changing individual attitudes and behaviours)
Intra-organisational (fostering a culturally inclusive culture, and positive relationships between staff and organisational policies and procedures)
System-wide (including strategies for services and policymakers to increase the awareness, commitment, willingness and resources to implement system level policy change)

Questions community organisations and service providers may consider include:

Are there recruitment and processes in place to encourage cultural affirmation?
Are there induction practices in place that promote a culturally safe work and service environment?
Is there a Reconciliation Action Plan in place developed with local Aboriginal community advisory groups?
Are there courses to support cultural learning in the workplace and in service provision?
Are the guiding principles outlined in the CBPATSISP Evaluation Framework to support SEWB of Aboriginal individuals, families and communities understood and consciously enacted by staff? How do you intend to demonstrate this?
Are there mechanisms and resources to embed anti-racism and cultural competence in individual and organisational practice?

Strategies for enacting and measuring change

Conducting self-administered organisational audits such as online surveys, interviews and focus groups for use by staff and community groups to monitor and evaluate improvements in cultural safety within service delivery
Encouraging staff to participate in cultural competence and reflective learning courses is also important
Developing resources for workplaces on the basis of their specific needs and contexts of each organisation

Evidence of the effectiveness of strategies might include:

improved awareness, knowledge, skills,
enhanced capacity,
increased networks and partnerships with Aboriginal community groups
reduced racial tension experienced by consumers, clients and community as evidenced in exit interviews and surveys

Approaches to implementing organisational accountability and development, diversity training and measures to increase positive inter-group contact can be found in the resources.

Ensuring Genuine Partnership and Dual Accountability

Ideas of dual accountability recognise that Indigenous organisations are accountable for the efficient use of funds in achieving program goals. At the same time, Indigenous people have fundamental rights to have access to funds and services which can contribute to their social and economic wellbeing and on their terms of reference. An extensive review by Dudgeon, Walker et al., (2014) found compelling evidence that both mainstream and Indigenous-specific programs and services that adhere to the Closing the Gap service-delivery principles of engagement, sustainability, access, integration and accountability are more effective than those that do not. The Closing the Gap service delivery principles are:

Indigenous engagement: Engagement with Indigenous men, women and children and communities should be central to the design and delivery of programs and services.
Sustainability: Programs and services should be directed and resourced over an adequate period of time to meet the COAG targets.
Access: Programs and services should be physically and culturally accessible to Indigenous people recognising the diversity of urban, regional and remote needs.
Integration: There should be collaboration between and within Governments at all levels and their agencies to effectively coordinate programs and services.
Accountability: Programs and services should have regular and transparent performance monitoring, review and evaluation.

In the spirit of genuine partnership, community controlled organisations are encouraged to ensure government and non-government organisations are held accountable to answer the following questions:

Questions that policymakers, funders and program providers need to consider?

Do existing state and national government suicide prevention and intervention policies, programs and services have strategies to:

Protect and enhance the rights of Indigenous Australians to live and develop according to Indigenous terms of reference, values and needs?
Achieve equitable social and economic and wellbeing outcomes?

Such accountability questions need to be included in all evaluations. The Indigenous Evaluation Framework helps to inform government and organisations in responding to these questions.

Facilitating Self-determination

Indigenous self-determination requires genuine partnerships between Indigenous people, governments and non-government agencies, together with a commitment to dual accountability. The notion of partnership along with self-determination is deeply embedded in human rights and is a key principle underlying the Evaluation Framework. Historically Indigenous peoples have been self-determining – defining their own being through control over all aspects of their lives, including ceremonies, spiritual practices, medicine, social relationships, management of land, law, and economic activities (Howitt 2013). There is substantial evidence to confirm that ‘being self-determining’ is a key protective factor in achieving suicide prevention outcomes. Having a sense of control over one’s situation is crucial to Indigenous SEWB.

The extent to which a program, service or initiative supports self-determination may be measured by evidence of how they support:

Indigenous cultural connectedness, language, spirituality and social and emotional wellbeing
Indigenous people contributing equally to decisions about practices and standards throughout all phases of the development, implementation and evaluation of programs and services

Questions that policymakers, funders and program providers need to consider?

What processes and mechanisms are in place to ensure Indigenous people have been able to make a genuine contribution at all phases of the program/service development, implementation and evaluation?
Does the implementation of strategies, programs and services support Indigenous rights and values?
Does the program or initiative enable Indigenous people to have a sense of control over their lives?
Does the program or activity support elements that contribute to an individual, family or community’s sense of being self-determining such cultural ceremonies and spiritual practices, social relationships, and economic activities?
Are there indicators to measure these cultural connectedness, self-determination and rights within the evaluation?

Incorporating Human Rights for Equity and Sustainability

Several studies emphasise the importance of incorporating human and Indigenous rights principles into evaluation and policy frameworks to improve Indigenous health and SEWB and achieve an equitable and sustainable future for Indigenous peoples. Recognition of human rights requires a greater commitment by governments to uphold Indigenous people’s individual rights to health, as well as their collective right to maintain and use their own health systems and practices in pursuit of their right to health.

The recognition of human and Indigenous rights are higher order goals which are fundamental to the achievement of self-determination and equity, and ultimately to strengthening SEWB and reducing poor health outcomes including suicide.

Importantly, a rights-based framework can also serve as a set of standards that can be used to assess the extent to which governments policies, services and programs:

Acknowledge Indigenous people as the traditional owners of the land and their connection with the land
Support and encourage connections and mutual respect between Indigenous and other Australians
Provide services based on understanding and respect for Indigenous culture
Develop and implement policies at the direction of the Indigenous peoples to redress past wrongs and support them to achieve the outcomes they desire
Recognise and work to address the continuing oppression, dispossession and disadvantage perpetrated on Indigenous people
Integrate Indigenous culture into all educational, cultural and ceremonial activities to strengthen relationships, understanding and recognition to provide beneficial outcomes to the whole community and reduce discrimination in the long term
Provide education and information about the Indigenous culture and history of the local area across the whole community to foster understanding and respect and maximise opportunities for Indigenous people to participate in local decisions

Questions that policymakers, funders and program providers need to consider?

Acknowledge Indigenous people as the traditional owners of the land and their connection with the land
Support and encourage connections and mutual respect between Indigenous and other Australians
Provide services based on understanding and respect for Indigenous culture
Develop and implement policies at the direction of the Indigenous peoples to redress past wrongs and support them to achieve the outcomes they desire
Recognise and work to address the continuing oppression, dispossession and disadvantage perpetrated on Indigenous people
Integrate Indigenous culture into all educational, cultural and ceremonial activities to strengthen relationships, understanding and recognition to provide beneficial outcomes to the whole community and reduce discrimination in the long term
Provide education and information about the Indigenous culture and history of the local area across the whole community to foster understanding and respect and maximise opportunities for Indigenous people to participate in local decisions

Strengthening Individual, Family and Community Capacity

While the notion of capacity building is firmly embedded in government policy frameworks, there is a need for a shift in government thinking about program implementation and evaluation approaches that strengthen individual, family and community capacity.

Strengthening capacity is a two-way reciprocal process involving all stakeholders and especially those intended to benefit from the programs and services being planned, implemented and evaluated. All stakeholders need information to understand the different types and purposes of an evaluation, as well as familiarity with the concepts and language of community planning and evaluation.

The CBPATSISP Clearinghouse website has links to a range of useful and culturally relevant resources and toolkits developed to assist evaluators in conducting empowering, participatory community evaluation included on government and community services websites on suicide prevention.

Resources

Cultural Competence – Transforming Policy, Services, Programs and Practice. Walker R, Schultz C & Sonn C. 2014, Chapter 12 in Dudgeon P, Milroy H & Working Together: Aboriginal and Torres Strait Islander Mental Health and Wellbeing Principles and Practice (2014). Canberra: Australian Government Department of the Prime Minister and Cabinet, 2nd edition. pp 195-220.

Vic Health – Building on Our Strengths

This evidence informed framework was developed to guide future practice, programs and policies to address race-based discrimination. It identifies five key factors in which organisations can play a key and important role in reducing race-based discrimination and to promote diversity

Implementing organisational accountability/development
Diversity training
Resource development and provision
Serving as sites for positive inter-group contact (Paradies et al., 2009; VicHealth, 2009)
Serving as ‘role models’ in anti-discrimination and pro-diversity practice for other organisations (Trennery, Franklin & Paradies, 2012)

Guide to Evaluating Clinical Services, Protocols and Practices

This section outlines the key elements that need to be considered in providing clinically safe care, ensuring effective screening, diagnosis and treatment in a range of clinical settings to support Aboriginal and Torres Strait Islander people who have attempted suicide, experienced mental health issues or suicide ideation.

Professional Practice Guidelines

For Psychologists

The Australian Psychological Society – Code of Ethics

The Australian Psychological Society – Ethics and Practice Standards (members only)

For Social Workers

From the Australian Association of Social Workers (AASW)

• AASW Education and Accreditation Standards (2012 V1.4) Revised 2015

• AASW Practice Standards for Mental Health Social Workers 2014

For Occupational Therapists

From the Australian Association of Occupational Therapists

• Code of Ethics (revised 2001)

• Australian Minimum Competency Standards for New Graduate Occupational Therapists 2010. From the Occupational Therapy Board of Australia.
• Code of conduct for registered health practitioners (2012)

For Psychiatrists
From the Royal Australian and New Zealand College of Psychiatrists
• position statements, clinical practice guidelines, ethical guidelines, Code of conduct and Code of ethics are available from:
https://www.ranzcp.org/practice-education/guidelines-and-resources-for-practice

For Nurses
From the Australian College of Mental Health Nurses:
• Standards of Practice for Australian Mental Health Nurses: 2010. Available from:
http://members.acmhn.org/publications/standards-of-practice
From the Nursing and Midwifery Board of Australia:
• Code of Ethics for Nurses in Australia (2008)
• Code of Professional Conduct for Nurses in Australia (2008)
From the National Nursing Competency Standards for the Registered and Enrolled Nurses. Available from:
http://www.nursingmidwiferyboard.gov.au/Codes-Guidelines-Statements/Codes-Guidelines.aspx

From the Congress of Aboriginal and Torres Strait Islander Nursing and Midwives (CATSINaM)
• National Summit on Cultural Safety in Nursing and Midwifery: Background paper for all mental health and health practitioner s/organisations link
From the National Aboriginal Community Controlled Health Organisation 2011,
• Creating the NACCHO Cultural Safety Training Standards and Assessment Process: a background paper, NACCHO, Canberra, http://www.naccho.org.au/promote-health/cultural-safety/
This background paper provides a comprehensive overview of concepts and meanings related to cultural safety and cultural safety training, including: cultural safety and respect as a human right, racism and cultural safety, good practice in cultural safety training. It also outlines a number of organisation strategies to achieve improved cultural safety at an institutional and individual level.

For General Practitioners
Australian Indigenous Doctors Association
• Position paper: Cultural Safety for Aboriginal and Torres Strait Islander doctors, medical students and patients, 2013, AIDA, Canberra.
AIDA’s suggest that strengthening cultural safety is a shared responsibility between educational institutions, national peak bodies and health services, as well as government and individual non-Indigenous medical practitioners

Resources

Ngarngadji! Listen/ understand! Improving Care for Aboriginal and Torres Strait Islander Patients (ICAP) Resource Kit

Clinical and Cultural Safety: Is the program, Service or Initiative Safe in the Real World?

Evaluations need to assess the extent to which appropriate safeguards are in place to minimise the risks of harm. Some questions or issues to consider may include:

Questions/Considerations

Are there adequate safeguards in place in school and community settings to support community awareness and gatekeeping programs? For example is there ready access to culturally competent counsellors in schools or in local community?
Are there culturally responsive, family oriented mental health and wellbeing initiatives with clinical follow-up and referral for people identified with clinical depression or mental health disorder requiring medication?
Is there access to bereavement counsellors?
Have the prevention and postvention strategies and resources been adopted and promoted been formally evaluated and culturally validated?
Have issues of cultural safety /cultural security been addressed in program or service delivery?
Are strategies in place to consider issues of racism as outlined in the section on operationalising the principles in the CBPATSISP Evaluation Framework ?

The Need for Effective Clinical Services

The National Review of Mental Health Programs and Services (2014) recommended bolstering the availability and capacity of both community and culturally responsive primary health care services to support and respond to the mental health and social and emotional wellbeing needs of individuals, families and communities.

In addition, the findings from the Australian Institute of Health and Wellbeing Review (AIHW, 2013) confirm that clinical approaches that have proved to be effective in reducing suicide rates include:

Community programs that focus on the social, emotional, cultural and spiritual underpinnings of community wellbeing
Culturally responsive brief interventions comprising motivational care planning

Programs and services that are not culturally competent and do not have a high level of Indigenous ownership and community support are unlikely to be effective in responding to the diverse and particular needs of Indigenous people in addressing suicide (AIHW, 2013).

Evaluators

The following independent Aboriginal and non-Aboriginal evaluation consultants and organisations that have been recommended to CBPATSISP as experienced and culturally responsive in conducting program and service evaluations with Aboriginal communities and organisations:

Human Capital Alliance (HCA) is a management, workforce planning and development and evaluation research consultancy specializing in aligning human capital to strategic and organisational goals. HCA has particular expertise with health workforces and generally workforces planning, managing and delivering health services. It is comprised of a core group of management and human resources experts and social researchers whose project management and high level content skills are enhanced by an extensive network of associates recognized as experts in their fields. HCA actively listens to clients and stakeholders and applies innovative research methodologies to develop realistic solutions that work.

Debbie Stanford – After commencing her professional career as a social worker in acute health care and community settings, Debbie spent more than two decades working across a broad range of Commonwealth and state government health funding, policy and program development initiatives. The majority of this time was spent working on national health and aged care policy and funding initiatives. This experience has resulted in a comprehensive understanding of national policy development processes and the legislative and funding frameworks that support high quality service delivery to consumers. Debbie has a strong and successful stakeholder consultation record, with particular expertise in identifying and exploring the nexus between government policy requirements and stakeholder perspectives and negotiating pathways forward.

Debbie’s content expertise includes workforce analysis and development strategies, rural and remote health service provision models, consumer-led models for aged care and disability support strategies, Aboriginal and Torres Strait Islander service access strategies, and the development and implementation of quality and safety management frameworks for health systems, services and professions. She has a detailed understanding of the Medicare funding arrangements, legislation and management arrangements and spent over a decade working on various aspects of the role and funding of diagnostic services in health service provision.

Since joining HCA in 2014, Debbie has applied her skills and content knowledge to a range of projects that have included workforce analysis and development activities, evaluation projects and qualitative research. These activities have involved review and analysis of relevant literature, preparation of ethics applications, design and implementation of survey and qualitative research tools, consultation with consumers, professionals, managers and public sector policy advisors and program managers, analysis and reporting of qualitative data, and extensive report-writing and other communication activities. Other critical skills that have been required include detailed and strategic project management, active team participation and leadership when required, meeting or workshop facilitation, client and liaison and relationship building, and risk management across all areas of practice. Like all HCA core staff, Debbie is a member of the Australasian Evaluation Society and a member of the Institute of Public Administration Australia.

In addition, Debbie has postgraduate qualifications in landscape architecture which she has applied in both community development activities and private commissions. This training has honed her design-thinking capabilities, giving her a detailed understanding of the importance of context, relevant detail, process and user input to any design process.

Ninti One work with clients to develop culturally appropriate evaluation approaches and tools, applying a cultural lens to the interpretation of results. They resource communities and organisations to monitor and evaluate their activities to understand how effective their activities are, their impact and success rate and where improvements can be made. Ninti One use a participatory approach to monitoring and evaluation for groups to understand what’s working and what activities can be changed to achieve desired aims.

Their team of Aboriginal Community Researchers ensures a more authentic research outcome for you, thanks to their deeper understanding of location, traditional knowledge, languages and culture that external researchers may lack.

At the other end of the spectrum, Ninti one have a national network of leading universities to call upon to conduct specialised research on any topic, drawing upon cutting edge theories and trends.

Ninti One’s work include Beyond Blue Evaluations – Beyond Blue Inc.

Beyond Blue is a national, independent, not-for-profit organisation working to increase awareness and understanding of depression, anxiety and suicide prevention in Australia and reduce the associated stigma. Ninti is the only Aboriginal company on Beyond Blue’s Evaluation Services Panel, and cultural consultant services panel. They provide evaluation specialists to evaluate Beyond Blue’s programs, products or services, and policies and strategies, with a focus on Aboriginal and Torres Strait Islander people.

Ninti is part of a team with ARTD that is delivering the ‘Design of Beyond Blue Performance Framework’ project, to identify standard measures to demonstrate its impact against the Beyond Blue’s ‘Beyond 2020 Strategic Plan’, provision of a roadmap of key infrastructure dependencies, and the detailing of options for a user-friendly reporting structure. The study also advises on change management for Beyond Blue’s rollout of the performance framework.

Client satisfaction survey for Central Australian Aboriginal Congress

Ninti has been contracted by Congress to conduct a minimum of 1,400 client satisfaction surveys at eight Congress health clinics in Alice Springs and surrounding remote communities. The information and data collected is helping Congress to analyse how it can deliver improved health services for Aboriginal people in Central Australia.

Evaluation panels – various agencies

Ninti also have representatives on a range of evaluation panels. These include:

Australian Federal Police Capability Support Services Panel
Department of Prime Minister & Cabinet (PM&C), Collaborative Research Evaluation and Planning Panel
Australian Department of Jobs and Small Business, Research and Evaluation Services
Australian Department of Social Services (DSS), Research, Evaluation and Data Panel
Australian Department of Human Services. Market and Social Research Services
Australian Department of Health, Remote and Aboriginal and Torres Strait Islander Aged Care Service Development Assistance Panel
Australian Department of Health, Program Reviewers and Evaluators Panel for Health Systems Financing and Evaluation Branch, Health Economics and Research Division, Australian Government
Beyond Blue. Evaluation Services Panel
Beyond Blue. Aboriginal and Torres Strait Islander cultural consultant services panel
Department of Foreign Affairs and Trade, Advisory Services Register for Agriculture, Inclusive Markets and Development Finance