Home > Research and publications > Evaluation > Collecting

Collecting

Man juggling
You are now ready to consider your methods for collecting the information outlined in your plan. There are numerous ways that you can do this and being creative can get you the most interesting material.

For instance, Neighbourhood Houses have used a combination of focus groups, a census and an internet based achievement reporting tool to discover that:

3 million people participate in classes annually;
participation can lead to employment, further education and involvement in the community; and
there are cost benefits associated with the program.

Step 8 – Identify or create your data sources

To complete your Evaluation Plan you need to select the best method for each of the information requirements listed in your Evaluation Plan. You should also use a mix of data sources.

When selecting the best method for each item, you need to consider three questions:

The best methods for measuring outcomes are those that measure before and after your intervention, or that compare two similar communities where only one receives the intervention (a randomized control trial). These types of evaluation methods, however, are expensive and you will need the help of a university to run them.

There are alternative methods, though, that can be just as good for collecting the information you need. These include:

Achievement Audits

Key Stakeholder Interviews / Individual Interviews

Focus Groups / Group Interviews

A Census

Literature Reviews

Data Sources / Indicators

Partnership Tools

Surveys

All programs should have a way of regularly recording their achievements – an achievement audit (see Collecting Methods). These can be developed in-house, are cheap to run and you will be amazed at the information you can generate that may otherwise be forgotten over time or lost as your staff change.

Collecting Methods (DOC 143 kb) provides more detail on each of these research methods, including when they are best used, examples of the information that each can generate and further advice on how to use each tool.

The example below shows how you may match up your information requirements with collection methods.

Evaluation Plan
Objectives	Questions	Information Required	Data Source
Better and more timely planning and delivery of community services and infrastructure in community X through partnership model.	What was the model? Who was involved? Was the diversity of the community represented? What was delivered? What was delivered differently because of the model? Has capacity been built to make the model sustainable? What lessons were learnt? Could this model be generalised to other areas?	Description of the model List of partners over time Significant organisations not involved in the partnership Audit of services/ infrastructure delivered Partners assessment of what happened differently because of the model Partners assessment of the capacity that has been built and sustainability Partners assessment of lessons High level strategic opinion about the applicability of the findings to other areas	Project application Bi-monthly reports Stakeholder interviews Achievement audit Stakeholder interviews Stakeholder interviews Stakehoder interviews Workshop on evaluation results
Increased level of community participation in general activities and governance.	Are more people from a range of backgrounds involved in general activities? Are more people from a range of backgrounds involved in governance?	Number of participants, Age groups, CALD, Indigenous etc Number of participants, Age groups, CALD, Indigenous etc	Enrolment forms Achievement audit Pre and post population indicators Bi-monthly reports Achievement tool Pre and post population indicators

In this example, the program would need to establish six new evaluation activities:

A bi-monthly report, six monthly achievement audit and a community survey at the program outset, and

Stakeholder interviews, internal workshops and a community survey at project completion.

Be careful of using Case Studies as a research method. If you draw conclusions from a case study you can make serious errors. For instance, you may have a program that has failed in 10 sites but succeeded in the single site of your case study, leading you to conclude that the program was an overall success. Case studies can be useful communication tools though, helping to highlight your achievements and illustrate your results. For instance, Community ITC has developed a series of case studies to tell a story about their work.

This is the second point where you can seek assistance from a contractor.
Refer to the Contracting page for further advice and then continue to read through the remaining three steps to understand the process.

Once you have chosen appropriate tools, add them to the final column of your Evaluation Plan. Keep in mind that any new tools you develop will need to be piloted to ensure they will generate the information you need. Collecting Worksheet 1 (DOC 93 kb) provides some guidelines on piloting your research tools.

Your Evaluation Plan is now complete.

Step 9 – Overlay your timelines and budget

Once your Evaluation Plan is complete you can overlay your timelines and budget. This involves further developing the timeline and budget you established in the Thinking section, so that you have a table indicating when each evaluation activity occurs and the costs and staff involved.

Collecting Worksheet 2 (DOC 128 kb) will assist you to prepare this information.

In your budget, don’t forget to include the costs involved in producing products such as published reports and workshops.

Step 10 – Consider Privacy and Ethical Issues

Whether you are designing, conducting or funding a piece of evaluation research you have a responsibility (and legal obligation) to ensure that it is conducted in an ethical way. The National Statement on Ethical Conduct in Human Research (NS) outlines Australia’s ethics standards.

There are five important principles to consider when conducting research involving humans:

Respect for individuals and groups

You must respect the autonomy of individuals to make decisions for themselves, including consenting to research (or not), and use of information about them. You must explore and respect the beliefs, customs and cultural practices of relevant parties (in particular note the Guidelines for Ethical Conduct in Aboriginal and Torres Strait Islander Health Research. You must respect privacy and confidentiality and comply with the Information Privacy Act , the Commonwealth Privacy Act and the Health Records Act (if relevant).

Benefits must justify any risk

You must maximise benefits and minimise risks of your research. Benefits include ensuring that the research is used to improve things, while risks include: harm – eg. physical or psychological harm, devaluation in personal worth, social, economic or legal harms; discomfort - eg. anxiety induced by an interview or minor physical discomfort; and inconvenience - eg. filling in a form or completing a research task. Reflecting on risks and benefits requires you to consider the welfare of participants as well as the broader social and cultural implications of the research.

Justice and equity

You must consider who receives the benefits of research and who bears the burden of being researched. For instance, one group may be ‘over-researched’ because they are easy to access, while others may be rarely researched because they are considered ‘too difficult’. Ensure that all people are treated fairly in your research procedures.

Integrity

You must ensure your research is a legitimate search for knowledge that follows recognised principles of conduct. Research should be conducted by qualified and competent professionals and there should be a commitment to disseminating the findings.

Research Merit

You must ensure your research will contribute to knowledge. Make sure that the research question can not be answered using existing data and that the best method is being used. In assessing merit, respect for the well-being and dignity of people take precedence over benefits to knowledge.

Your research proposal may first need to be reviewed by a Human Research Ethics Committee (HREC). This process can take several months and should be factored into your planning.
An evaluation or research activity can proceed without review by an HREC if:

Both

(a) the activity is undertaken with the consent of the participants, providers or institutions involved;

or

is consistent with national Privacy Principle 2.1(a), which states:
‘An organisation must not use or disclose personal information about an individual for a purpose (the secondary purpose [i.e. your evaluation purpose]) other than the primary purpose of collection’… unless the following apply:

(i) the secondary purpose is related to the primary purpose of collection and, if the personal information is sensitive information, directly related to the primary purpose;
(ii) the individual would reasonably expect the organisation to use or disclose the information for the secondary purpose’;

And

(b) it is an activity where participants, including providers or institutions, are unlikely to suffer burden or harm (physical, mental, psychological, spiritual or social).

(National Health & Medical Research Council: Advice to Institutions, Human Research Ethics Committees and Health Care Professionals)

Collecting Worksheet 3 (DOC 116 kb) will help you to consider some of the most significant ethical issues in relation to your evaluation. This advice DOES NOT replace examination of the NS and it is not a complete review of ethical issues. You are advised to seek assistance from a reputable academic or evaluator to ensure your compliance with ethical standards and privacy legislation.

By this stage you will have completed Collecting Methods (DOC 143 kb), Collecting Worksheet 1 (DOC 93 kb) (if you are using any new tools), Collecting Worksheet 2 (DOC 128 kb) and Collecting Worksheet 3 (DOC 116 kb). Your Evaluation Plan should be complete.

Now complete the Collecting Checklist (DOC 92 kb), before moving on to Communicating.

Top