” The Federal Government recently announced it will allocate $10 million a year over four years to strengthen the evaluation of Indigenous programs.
However, given that the average cost of an evaluation is $382,000, the extra $10 million a year for Indigenous program evaluations will not go far.
To make the most of this additional funding the government must change the way it evaluates and monitors programs.”
Sarah Hudson Researcher The Centre for Independent Studies
Download the report HERE
Evaluating Indigenous programs a tool kit for change
” Aboriginal community-controlled organisations treat health not just as a physical problem, but see it as tied in with the social, emotional and cultural wellbeing of the whole community, in which each individual is able to achieve their full potential as a human being.
While this has its roots in Aboriginal cultural norms, she says, it also mirrors well-known social determinants of health.”
” Victoria’s peak Aboriginal health body was recently given two days to respond to a draft family violence plan “the size of a PhD”, its CEO says. It’s another example of governments just not getting how to work with Aboriginal communities.
Co-design with community groups cannot work if government asks for input after the big decisions have already been made or rush consultation, warns the head of Victoria’s peak body for the Aboriginal community health system.
“It’s not an equal partnership. We’re at their whim, and we’ve got to run to their agenda,”
Victorian Aboriginal Community Controlled Health Organisation CEO Jill Gallagher said last week in a speech at the University of Melbourne. See Article 2 Below
” Evaluation at the contract, program and outcome level will ensure we not only know where the money is being spent, but we will know what works and why.
“This is important for the government and taxpayers, but more important for communities in whose name the money is spent.
“It will also mean we will be better able to assess where our investment needs to be focused in the future — and ensure the IAS continues to deliver outcomes for indigenous communities.”
From Indigenous Affairs Minister Nigel Scullion Article 3 below
Although formal evaluations for large government programs are important, evaluation need not involve outside contractors. Government must adopt a learning and developmental approach that embeds evaluation into a program’s design as part of a continuous quality improvement process.
It is not enough just to evaluate, government must actually use the findings from evaluations to improve service delivery. Unfortunately, many government agencies ignore evaluations when making funding decisions or implementing new programs.
Analysis of 49 Indigenous program evaluation reports, found only three used rigorous methodology.
Overall, the evaluations were characterised by a lack of data and the absence of a control group, as well as an overreliance on anecdotal evidence.
Adopting a co-accountability approach to evaluation will ensure that both the government agency funding the program and the program provider delivering the program are held accountable for results.
An overarching evaluation framework could assist with the different levels of outcomes expected over the life of the program and the various indicators needed to measure whether the program is meeting its objectives.
Feedback loops and a process to escalate any concerns will help to ensure government and program providers keep each other honest and lessons are learnt.
Analysis of Indigenous program evaluations
Mapping of total federal, state and territory and non-government/not-for-profit Indigenous programs identified 1082 Indigenous specific programs. Of these:
• 49 were federal government programs;
• 236 were state and territory programs; and
• 797 were programs delivered by non-government organisations.
The largest category of programs were health related programs (n=568) followed by cultural programs (n=145) then early childhood and education programs (n=130) — see Figure 1.
The program category with the highest number of evaluations was health (n=44), followed by early childhood and education (n=16). However, percentage wise, more programs were evaluated under the jobs and economy category (15%) than the other program categories.
Of the 490 programs delivered by Aboriginal organisations, only 20 were evaluated (4%). The small number of businesses delivering a program (n=6) meant that while there were only two evaluations of Indigenous programs provided by a business, this category had the highest percentage of programs evaluated (33%).
Similarly, while only six of the 33 programs delivered by schools and universities were evaluated, this category had the second highest percentage of programs evaluated (23%). Conversely, government and non-Indigenous NGO delivered programs had the highest number of evaluations, n=36 and n=24, but much lower percentages of evaluations as the number of overall programs was higher, n=278 and n=276.
A total of 49 evaluation reports were analysed and assessed against a scale rating the rigour of the methodology. Only three evaluation reports utilised strong methodology (see Figure 4).
In general, Indigenous evaluations are characterised by a lack of data and the absence of a control group, as well as an over-reliance on anecdotal evidence
Suggestions for policy makers and program funders include:
- Embedding evaluation into program design and practice — evaluation should not be viewed as an ‘add on’ but should be built into a program’s design and presented as part of a continuous quality improvement process with funding for self-evaluation provided to organisations.
- Developing an evidence base through an accountability framework with regular feedback loops via an online data management system — to ensure data being collected is used to inform practice and improve program outcomes and there is a process for escalating concerns.
Suggestions for program providers include:
- Embedding evaluation into program practice — evaluation should not be viewed as a negative process, but as an opportunity to learn.
- Developing an evidence base through the regular collection of data via an online data management system to not only provide a stronger evidence base for recurrent funding, but also to improve service delivery and ensure client satisfaction with the program
Article 2 Govt co-design ‘not an equal partnership’: Aboriginal health CEO
Victoria’s peak Aboriginal health body was recently given two days to respond to a draft family violence plan “the size of a PhD”, its CEO says. It’s another example of governments just not getting how to work with Aboriginal communities.
Co-design with community groups cannot work if government asks for input after the big decisions have already been made or rush consultation, warns the head of Victoria’s peak body for the Aboriginal community health system.
“It’s not an equal partnership. We’re at their whim, and we’ve got to run to their agenda,” Victorian Aboriginal Community Controlled Health Organisation CEO Jill Gallagher said last week in a speech at the University of Melbourne.
A particularly vivid example of this is engagement on the establishment of family violence hubs around the state. Gallagher, who is on the family violence industry taskforce, said she was handed a draft plan already outlining the main priorities on Monday, and asked to provide a written response by Wednesday. “A report the size of a PhD,” she added.
“So when they say ‘we want to co-design with you guys’, always ask them what their version of co-designing is,” she told the audience. “Without systematic change in mainstream attitudes and practices, and incorporation of Aboriginal peoples in all stages of policy design, health policies will remain unproductive.”
While Gallagher says she understands the challenges of trying to co-design with a community, government needed to make a more concerted effort to do it properly.
“It doesn’t give us due respect of being part of the beginning right through to the evaluation.”
Culture is strength
Aboriginal culture is often seen in the wider Australian population as a barrier to health, implying that assimilation is the only way forward, Gallagher said.
She rejects this idea. “Cultural differences need to be celebrated and preserved. They are a source of strength and resilience for our peoples, which offer protective factors against traumatic life events.”
Cultural safety and trust can have a big impact on engagement with institutions. She points to the fact that around the country, Aboriginal people are discharged against medical advice or at their own risk at eight times the rate of the rest of the population. This has obvious flow on effects for overall wellbeing.
“When we have a culturally safe place for patients and our people, we improve access to services and improve health for individuals, therefore health for families, therefore health for communities.”
Also in The Mandarin: Indigenous policy evidence, where it exists, over-relies on anecdotal evidence
Creating that environment should not only be up to Aboriginal employees or a good CEO, but come out of an organisation’s systems. This means more than just creating a few identified positions — it’s everyone’s responsibility.
Aboriginal community-controlled organisations treat health not just as a physical problem, but see it as tied in with the social, emotional and cultural wellbeing of the whole community, in which each individual is able to achieve their full potential as a human being. While this has its roots in Aboriginal cultural norms, she says, it also mirrors well-known social determinants of health.
“Possessing a strong sense of cultural identity is also vital for one’s self-esteem. A positive cultural connection not only contributes to better mental health and physical health, but may lessen the consequences of social prejudice against Aboriginal peoples.”
Yet despite plenty of experience to show the importance of culture as a source of resilience, it “remains largely unexplored” as a public health resource, she says.
Funding models that don’t fit
Governments ignoring the role of culture creates other problems, Gallagher explains.
The Commonwealth made a capital investment a few years ago to create a childcare centre and kindergarten in Melbourne’s northern suburbs called Bubup Wilam. Recurrent funding was only given for two years, with the idea that it would become self-sustaining by the end of that short period.
“Bubup Wilam grew and evolved and it’s a beautiful childcare centre and kindergarten for Aboriginal children, where they can learn and express aboriginal culture but also have access to what every other kid has access to.
Despite the success, it’s “struggling to continue that at the moment” and is trying to raise funds in the community, she says, “because it doesn’t just provide a kindergarten like for a mainstream nuclear family.”
“Because a lot of the kids and families that access Bubup Wilam are families that live well under the poverty line, a lot of them are touched by the child protection system. What Bubup Wilam tries to do is work with the children, but also work with the families — the mum or the dad or the caregiver — and that takes a lot of resources.
“So our model there does not fit within the mainstream model of how they fund a nuclear, non-Aboriginal childcare centre. … So that’s an example of how the differences and different needs and funding formulas don’t fit what we need to achieve.”
This comes back to the co-design problem: governments aren’t paying enough attention to what the community says, and end up designing the system to fit what they think the community needs, which is different to what it really needs.
“It’s about involving us from the word go,” says Gallagher.
“What Fitzroy might need is different to what Fitzroy Crossing might need.”
Part 3 : Indigenous Affairs Minister Nigel Scullion
The Turnbull government will spend $40 million evaluating its indigenous affairs programs in an attempt to counter a national audit office report expected to be harshly critical of the way billions of dollars have been allocated.
Sidelined prime ministerial indigenous adviser Warren Mundine said yesterday the report, to be tabled today, was expected to be “damning”, as was the official Closing the Gap report due within days.
The audit office report follows a Senate inquiry last year that blasted the 2014 implementation of the Abbott government’s flagship multi-billion-dollar Indigenous Advancement Strategy.
A 2015 Productivity Commission report found there was insufficient evidence being collected about the outcomes of indigenous programs and that “formal rigorous evaluations of indigenous programs that set the benefits of particular policies for reducing disadvantage against the costs are relatively scarce”.
Spending on mainstream and indigenous-specific programs and services has been estimated by the government to be worth $30 billion. A Centre for Independent Studies report last year found only 8 per cent of 1082 indigenous-specific programs, worth $5.9bn, had been effectively evaluated.
However, Indigenous Affairs Minister Nigel Scullion, who will announce the four-year evaluation program today, said reporting, monitoring and evaluation of the IAS had already been improved, and accounting for how much was being spent in the portfolio was now possible.
“However, we need to continually build on this and further strengthen the evaluation of our investment to ensure that money allocated through the IAS is invested in ways that make the greatest difference for our first Australians,” Senator Scullion said. “By establishing a multi-year funding allocation, we are ensuring there will be a long-term plan for evaluation and a formal strategy to monitor and review how individual contracts and program streams are contributing to our efforts to deliver better outcomes for indigenous Australians.
Senator Scullion said the evaluation would be rolled out in close consultation with Aborigines and Torres Strait Islanders, including indigenous-run firms. “Indigenous-run companies are currently delivering rigorous evaluation for the government and this new framework will continue this partnership,” he said.
Hi Alana,
You might be interested in this.
Kind regards Kylie
Sent from my iPad