SPQS Evaluation
South West Academic Health Science Network Evaluation of Somerset Practice Quality Scheme
Introduction
NHS England engaged with the clinical community in Somerset as part of the Improving General Practice call to action in autumn 2013. The view of GPs, the Clinical Commissioning Group and the Local Medical Committee was that the clinical skills of GPs were not being used to best effect in helping patients with the most complex needs. Although GPs felt that QOF is more clinically relevant in 2014/15 compared with previous years, it does not incentivise integrated multi-disciplinary working and does not align local priorities and incentives across the whole system. NHS England agreed to test a new approach to QOF funding which would allow freedom for clinicians to innovate while continuing to provide assurance of high quality care. A one-year pilot of a local quality scheme called the Somerset Practice Quality Scheme (SPQS) was approved in June 2014.
Conceptual overview
SPQS arose from the assertion by GPs in Somerset that QOF was not incentivising the highest value clinical behaviour which sought to provide person-centred care and work effectively with other elements of the health and care system. The hypothesis is that removing the link between incentives and clinical activity of QOF will improve quality.
The fundamental question it raises is:
“Does increasing the self-determination of general practitioners improve patient care?”Underlying this is a series of subsidiary questions:
- Does removal of the link between specifically defined financial incentives and specifically defined clinical practices have an adverse effect on the quality of care?
- Does removal of the link between specifically defined financial incentives and specifically defined clinical practices accelerate the development of better integrated care for patients?
- Do the combination of removal of incentivised indicators and improvements in patient-centred primary care lead to increased patient activation and improved experience of care for people with multiple long term conditions who may be at risk of emergency admission?
A key concern of stakeholders is that the quality of clinical care may decline. It will be important to measure any change both in recorded interventions currently incentivised by QOF, and in other domains of quality which QOF does not incentivise. The evaluation will measure as many different aspects of quality as possible.
Design of the evaluation
The evaluation design will be guided by a Theories of Change approach. It will focus on patients with long-term conditions as QOF does. It will seek to provide a narrative understanding as well as quantitative data. It will align as far as possible with the evaluation of the Symphony integrated care pilot in South Somerset. It will compare practices participating in SPQS with a control group of similar practices drawn from the South West of England.
Structure of the evaluation
The structure of the final evaluation report will be guided by the findings, but is likely to be as follows:
- Executive summary
- Acknowledgements
- Introduction
- Context: stakeholder views on the role of QOF in promoting quality
- Literature review on quality in general practice and the role of QOF in improving quality
- Description of the SPQS pilot, including participation, costs, development process
- Overview of changes arising from SPQS, including information on patient care, development of integrated and sustainable primary care, and generalisability of results
- Detailed presentation of data, including evaluation methods, patient care measures, organisational measures, participants views, patient views, views of other stakeholders, views of GPs not participating in SPQS, any negative effects, effects on morale
- Discussion of results including summary, positive aspects, negative aspects, theories of professional behaviour change: the place of SPQS, implications for policy both locally and nationally
- References
- Appendices including QOF indicators, SPQS contract documents, interview schedules, data tables.
It is recognised that NHS England needs to have an early understanding of the results of the SPQS pilot, and it is therefore proposed to provide an interim evaluation report in January 2015. Although this will not be able to draw on the full range of data sources, it will provide some early messages about the pilot. The final evaluation report for SPQS will be ready in July 2015, although the process of evaluating the development of integrated care vin Somerset will continue as long-term programme.
Measures
The table below sets out the evaluation measures and data sources. It notes which data sources will be amenable to change within the 6 months to January 2015 and will therefore be included in the interim report, and which are only likely to demonstrate change across a much wider time period of several years. These will be included in the final evaluation report.
| Measure | Source | Included in Final report | Notes |
| Self-Reported ability to manage own Health | Patient Activation Measure questionnaires | Final | Ideally needs to be a before and after measure. Links with CCG LTC programme and the national development programme for proactive care. Has huge potential to link patient and population outcome measures. |
| Patient experience | National GP Patient Survey | Final | July 2014 data collection will be available in Jan 2015. Jan 2015 data collection will be available in July 2015. |
| Patient Experience | Patient Feedback Using Healthwatch Social Media tool | Final | This captures free text comment about health services and subjects it to sophisticated linguistic contextual analysis. |
| Patient Experience | Friends and family Test Free Text | Final | Not due to start until Dec 14so will be a limited data source |
| Patient experience of involvement in and Choices over their care | Symphony Patient Questionnaire | Final | The Symphony patient questionnaire has been developed as part of the Symphony integrated care pilot in South Somerset with input from NHS IQ and South West Academic Health Science Network |
| Patients ability to participate in work or other meaningful activities of their choosing and have links to their local community | Symphony Patient Questionnaire | Final | |
| Quality of life for Patients | Symphony Patient questionnaire | Final | |
| People have one key person who takes ownership for co-ordinating their care and are trusted | Symphony Patient questionnare | Final | |
| Patient experience | Focus groups of groups | Final | A better data source for patients with long-term conditions than the GP patient survey as only one question in the survey relates directly to long-term conditions |
| Small selection of indicators that provide an ‘at a glance’ measure of quality. These were selected not to prioritise some indicators over others, but to provide a feasible group of data that represents a good proxy measure of quality: Diabetes core processes (DM2-4, DM6-9, DM 12 and DM14) Chronic Obstructive Pulmonary Disorders reviews (COPD3) Care plans for patients with severe mental illness (MH2) Quad- medication following Myocardial Infarction (CHD6) Smoking status recorded for patients with long term conditions (SMOK2) |
CQRS via GPES | Interim and | Visible through the year via CQRS and included in the SPQS specification. Not suitable to compare with non-SPQS practices until QOF year-end because of the impact of point-chasing. Year-end data available June 2015. Note the further points on comparability below. |
| QOF-all indicators | CQRS Via GPES | Visible throughout the year via CQRS. Provides one data source to inform discussion of quality. Not suitable to compare with non-SPQS practices until QOF year-end because of the impact of point-chasing. Year-end data available June 2015.There are quite complex methodological issues involved in measuring whether or not there has been a drop in QOF activity. Given the previous high scores of Somerset practices (98.8% in 2012/13), it is proposed to treat only reductions of 10% or more as significant. Of course this would take into account the reduction of number of points from 1000 to 559 for 2014/15. In addition, there is a danger of false positives arising from changes that relate to random chance. A confidence interval of 90% is proposed, and a control group established with a group of practices with similar characteristics | |
| QOF-Patient Subset/s | CQRS via GPES possible Practice Level audit | This will examine the impact of SPQS on particular groups of patients, for example by age group. An example being considered is whether there is a differential impact between age groups in statin prescribing for primary prevention of cardiovascular disease, relating to the indicator CVD-PP001. | |
| Emergency admissions for ambulatory care sensitive admissions | CCG Outcome Indicator/ NHS Outcome Framework 3a | Final | Aligns to Better Care Fund and Symphony measures as a system-wide indicator, although unlikely to show results from which inferences can be made within the timescale available. |
| No Negative impact on Health on inequalities | Linking the 20% of most deprived LSOAs to patient records and then cross- referencing to COPD care as measured by QOF indicator COPD003 (% of patients with COPD who have had a thorough review of their condition in preceding | Interim and Final | This indicator has been proposed by Somerset County Council Public Health Team |
| Measure | Source | Notes | |
| Extent to which integration and sustainability are promoted by the pilot | Analysis of quarterly progress reports and micro-evaluations of local SPQS pilot projects | Interim and Final | Particular emphasis will be placed on improvements in organisational relationships. The theories of change methodology will be used to see what inferences can be drawn about the impact on patient care. |
| Extent to Which integration is furthered by the pilot | Assessment of progress using the taxonomy of integration developed by SWAHSN | Final | Links with SWAHSN evaluation of other integration pilots in the South West |
| Improvements in morale and motivation of GPs, nurses and other practice staff | Sibbald et al selected as measure for GP morale- this could also be used/ adapted for other groups | Final | The Manchester University DH GP morale survey is partly based on Sibbald et al and the methodology may be directly transferrable. Choice of tool and methods of data gathering to be confirmed by SWAHSN following agreement on the set of tools currently being developed in Torbay. |
| Personal experience of the pilot | Focus groups/ individual interviews | Final | The satisfaction, optimism and engagement of clinicians and others engaged in the pilot will be assessed |
Reporting Timescales
| Action | Completion date |
| Progress report with early findings to NHS England | 28 November 2014 |
| Interim Evaluation report to NHS England | 30 January 2015 |
| Final evaluation report to NHS England | 31 July 2015 |
Resources
The evaluation will cost £80,000, of which £40,000 will be met through SWAHSN’s existing evaluation programme of integrated care in Somerset. A funding contribution of £40,000 from NHS England will ensure that the evaluation is as rigorous and comprehensive as possible.
Governance
The evaluation will be undertaken within the ethics governance of SWAHSN, and will be overseen by the SPQS steering group, which has representation from NHS England, Somerset CCG and Somerset LMC. Somerset Health and Wellbeing Board will be asked to comment on the evaluation and to receive the evaluation report.
Glossary of terms
Self-determination: the extent to which people are free to make choices without external influence. A psychological theory concerned with motivation and choice. See for example: Deci, E. L. (1971). Effects of externally mediated rewards on intrinsic motivation. Journal of Personality and Social Psychology, 18, 105–115.
Theory of Change: a specific type of methodology for planning, participation, and evaluation that is used to measure social change. Theory of Change defines long-term goals and then maps backward to identify necessary preconditions. The innovation of Theory of Change lies (1) in making the distinction between desired and actual outcomes, and (2) in requiring stakeholders to model their desired outcomes before they decide on forms of intervention to achieve those outcomes. Theory of Change can begin at any stage of an initiative, depending on the intended use.
See for example: D. Taplin, H. Clark, E. Collins and D. Colby (2013). Technical Papers: A Series of Papers to support Development of Theories of Change Based on Practice in the Field. New York: Actknowledge and The Rockefeller Foundation.
SWAHSN project leads: Dr Alex Mayor, Professor Richard Byng , Richard Blackwell
