During 1997, staff from San Diego County Adult and Older Adult Mental Health Services (AMHS), Community Research Foundation, USD Patient Advocacy Program, and UCSD Department of Psychiatry conducted an analysis of the CSS surveys received by the Patient Advocacy Program between the inception of the project in June 1995, and November 1996. Phase I, as it came to be known, yielded 1272 surveys from 13 inpatient psychiatric hospital programs in San Diego County. A report of preliminary findings was issued on July 23, 1997. Although, in general, levels of reported satisfaction were favorable, findings included several specific areas where improvement was indicated. Wide variation in hospital response rates (the number of surveys received divided by the number of Medi-Cal discharges) was also reported. Only 6 (47%) of the 13 participating hospitals produced a large enough sample of respondents to be included in the analysis. The resulting exclusion of the other seven hospital programs (53%) only reduced the sample by 16% (202 surveys) to 1070 surveys.
Based on these findings, the Client Satisfaction Work Group recommended conducting a Phase II follow-up evaluation using the same CSS instrument to determine whether changes had occurred since the Phase I baseline. Implementation of the Phase II follow-up survey was to be preceded by an intervention designed to produce improvements in the response rate and improvements on specific questions identified as receiving the least favorable responses in the Phase I report. The intervention was convened by AMHS Clinical Director and included: (1) providing each hospital with a copy of the Phase I report, (2) providing each hospital with a letter signed by the Deputy Director and the Clinical Director explaining Phase II, (3) requesting that designated hospital representatives attend one of two meetings regarding Phase II implementation, and (4) providing two orientation and training meetings for representatives of participating programs.
The main objectives of Phase II were: (1) to measure change in the response rate of hospital programs participating in the Medi-Cal reimbursement system, (2) to measure change in client satisfaction, with a focus on those less favorable areas mentioned above, and (3) to include the San Diego County Psychiatric Hospital and the six Short-Term Acute Residential Treatment (START) programs. Phase II was conducted between October 1, 1997 and March 30, 1998, and resulted in the inclusion of fourteen programs (6 START & 8 hospital programs including SDCPH) and 1760 surveys in the Phase II analysis. The analysis comparing individual program change between Phase I and Phase II included surveys from seven hospitals.
Perhaps the most significant finding was the considerable improvement in response rates of hospital programs participating in both Phase I and Phase II. Although increasing the response rate remains an important goal for some programs, the results of Phase II demonstrated dramatic and statistically significant improvements in the response rates of 6 of the 7 programs participating in both phases. The mean response rate of programs participating in both phases improved from 16% in Phase I to 41% in Phase II. The single program that did not appreciably increase its response rate already had nearly a 40% response rate in both Phase I and Phase II.
In general, the findings for the group of programs participating in Phase II were favorable. The overall benchmark of 82.3% favorable responses (mean proportion of favorable responses on all 36 qualitative questions) and the many higher scores on important questions attest to the generally favorable overall findings.
The Phase I Report targeted 9 questions for improvement in Phase II (14a-risks and benefits of medication, 17a-satisfaction with non-psychiatric medical care, 22/22a-language difficulty & staff concern with arrangements for communication, 26-staff concerned more with procedures than clients, 31-family and friends notified about admission, 32-told about Consumer Service Representative, 34-attendance at care planning meeting, 39-informed about community resources, & 40-participation in setting goals for care). We found no significant improvements in any of the 9 targeted questions. However, significant differences were found in 2 of the 9 questions in which proportions of favorable responses were lower at the Phase II measurement (14a & 17a).
Early in the review of individual programs, a theme emerged in which the START programs demonstrated somewhat consistent responses, as well as somewhat higher proportions of favorable responses across a number of questions. To examine these differences, Phase II responses were divided into START or hospital groups and a preliminary analysis was conducted comparing responses on the 36 qualitative questions used to calculate the overall benchmark. There were no significant differences found on 10 of the questions. The remaining 26 questions demonstrated significant differences in which START program respondents reported higher proportions of favorable responses.