An evaluation is the use of scientific method and rigorous and systematic collection of research data to assess the effectiveness of organisations, services and programmes in achieving predefined objectives. For health services, it is used to see if they fulfill their stated goals, targets or objectives. It is usually based on the collection of data about the structure, inputs, process, outputs and outcomes of the service as well as the appropriateness of the service (e.g. Vitamin D treatment has side effects for the severely deficient). Evaluation is more than audit because it seeks to record not only what changes occur but also what led to those changes.
Evaluation can be formative or summative. A formative evaluation involves the collection of data while the organisation or programme is active. The aim is to improve or develop the programme. A summative evaluation involves collecting data about an active or terminated programme or organisation. The aim here is to decide whether it should be continued or repeated, e.g. a screening programme or re-running a health promotion campaign for smoking if rates have ceased to decline.
The evaluation of health services is usually based on the collection of data about the structure, inputs, process, outputs and outcomes. This method is credited to Donabedian. Structure refers to inputs and resources into the services, e.g. buildings, funding, equipment, staff and hospital beds. It is the organisational framework for the activities that happen within the health service(s). Process on the other hand, is the activities themselves, such as referral patterns, admission procedures and prescribing of drugs. Process is important to the staff as staff are directly involved in it. (Indeed, sometimes staff get so got up in the process that it is difficult for them to see if the outcomes are being met). Outcomes are different to outputs. Outputs are the productivity side of things, such as length of hospital stay, waiting times, discharge rates, patient-professional contact, access, effectiveness and equity). Outcomes refer to the effectiveness of activity/activities. It is measured by mortality and morbidity rates, complication rates, disability, quality of life and patient satisfaction. In other words, the impact upon the patients and communities. A good classification of outcomes is Lohr's (1988) 5 D's of outcome: death, disease, disability, dissatisfaction and discomfort. Often, quantitative (e.g. survey or document analysis) and qualitative data are collected about process and structure in order to investigate how the outcome was caused by the activity. Additionally, data collected about process (often by an audit) can tell us how a service is organized, delivered and used. This can show how accessible the service is and whether scare resources are being used efficiently. So data can be used as performance indicators.
There are two criteria that you can use to measure quality of services.
1. Maxwell (1984) Criteria
The aim is to evaluate all seven criteria but it is impossible to maximize all dimensions. To remember the criteria, think SEE REAL!
Social acceptability (how is it viewed by users?)
Effectiveness (does it work?)
Efficiency (can it produce same outcome with fewer inputs?)
Relevance to Needs (appropriateness)
Equity (fairness)
Accessiblity (can services be reached by the population?)
Locality
2. NHS Performance Assessment Framework
This is a new criteria and can be used with Maxwell's one. To remember the 6 indicators, think FEEPHH!
Fair Access (measured by equity)
Effectiveness (measured by evidence based medicine)
Efficiency (measured by finance)
Patient/carer Experience (measured by complaints/survey)
Health Improvement (measured by public health)
Health Outcome (measured by audit)
A health outcome is defined as the change or lack of change when change is expected - in health, health related status or risk factors affecting health. Outcomes may be attributable to medical interventions or they may be the result of the natural history of the condition. Considerations of case mix, bias and confounding factors are central to the appropriate interpretation of indicator values.
To access an outcome, you will need an adequate description of the service, the underlying level of risk, frequency of outcomes and the data collection needs to be feasible. When measuring outcomes, make sure you include how they are driven (e.g. risk factors, genetics) and case-fatality is influenced by case mix and quality of care.
Indicators of health outcomes
An indicator is an aggregated statistical measure describing a group or whole population, complied from measures on individuals, which provides insights about the functioning of services. Indicators will not necessarily provide definite answers on whether the services are good, adequate or inadequate but when well chosen, they should be capable of providing pointers to where further investigations may be worthwhile. An example of an indicator is in cancer, where screening can be a proxy measure for outcomes - what % are you covering? What % is being picked up?
Use of Health Outcome Indicators should help:
Reduce risk of the condition
Detection of condition early
Assure return to function after an acute episode
Reduce risk of a further acute episode
Reduce adverse impact on well-being
Reduce risk of death
1. Background and state epidemiology of the problem
2. State objectives
3. Select dimensions to be evaluated - (a)Donabedian & (cost)effectiveness, efficiency and acceptability) or (b) Maxwell criteria or (c) NHS Performance Framework
4. Define population to be studied, interventions to be looked at and outcomes to be measured
5. Select study design (experimental/RCT/case control/cohort/ecological/descriptive)
6. Select outcome measures to use: (a) disease measures (objective) - death, morbidity; (b) patient measures (subjective) - functional status, health-related quality of life (SF-36), disease specific (AIMS for arthritis), site specific (e.g. post hip replacement); (c) economic measures - costs (direct: staff, drugs, buildings; indirect: loss of earnings, travel to hospital; intangible: costs of pain and grief); (d) acceptability measures - focus groups, interviews, satisfaction surveys and equity measures
7. Decide when and how to collect the data
8. Calculate sample size
Bowling, A. Research Methods in Health, 2nd Edition. Open University Press, 2002.