For more than ten years the Tobacco Free Project (TFP) has funded a number of community-based projects to implement tobacco advocacy projects using their Community Action Model (CAM). This report describes the methods used by the TFP staff to ensure tobacco funding went to the communities most impacted by tobacco smoke, that funded projects were provided the technical assistance necessary to operate in cultural competent ways, and that the participating agencies self-assess their organizational cultural competency standards.

As part of the comprehensive tobacco control plan for San Francisco, the Tobacco Free Project has funded community based agencies to implement the Community Action Model. The CAM is a five-step model focused on environmental change through policy development or changes in organizational practice, rather than individual behavior change. The intent of the CAM is to work in collaboration with communities and provide a framework for community members to acquire the skills and resources to investigate the health of the place where they live and then plan, implement and evaluate actions that change the environment to promote health. As part of the CAM process, TFP staff provides interactive trainings and technical assistance to community-based organizations to facilitate a sharing of existing skills and community strengths so that the actions are community driven. Between 1995 and 2010, the Tobacco Free Project funded fifty-eight projects to implement the Community Action Model. The CAM has successfully mobilized community members and agencies to change community norms that promote unhealthy behavior such as tobacco advertising, promotion and access for minors.

The Primary Asset (3.7) drawn upon for this Brief Evaluation Report included: The extent to which a tobacco control program implements organizational policies and practices that promote and institutionalize the provision of culturally competent and linguistically appropriate services for diverse populations including organizational values that articulate commitment to cultural competency, participatory collaborative planning, provision of community capacity building, translation policies, staff diversity, and formative research/surveillance within diverse communities.

The objective addressed in this Brief Evaluation Report is: “At least 3-5 culturally and ethnically diverse organizations funded through both Prop 99 and MSA funds to implement the Community Action Model will develop as part of their SOW, a plan to implement cultural competency standards that address organizational values, participatory planning, provision of capacity building, translation policies and staff diversity.”

The Tobacco Free Project chose this objective because San Francisco is becoming an increasingly diverse city with a minority-majority population, meaning that non-Hispanic whites comprise less than half of the population, 41%, down from 92.5% in 1940. There is also mounting evidence that socioeconomic status–gap between rich and poor– contributes to health inequities between the “haves” and “have-nots.” Because race and ethnicity are major determinants of socioeconomic status, communities of color are more likely to have poor health and to die early due to disparities in health. And tobacco related illness is no exception as communities of color and low socioeconomic status groups have a higher prevalence of tobacco use. The objective above was chosen because it strives to improve the cultural competency of all organizations working to reduce the use of tobacco within affected communities.

Overview of the intervention activities 

Intervention activities centered on three different areas related to cultural competence. The first was to ensure that tobacco funding went to organizations that were capable of reaching those populations most at risk of tobacco related illness and providing them with the training and technical assistance necessary to conduct their project activities in a culturally competent manner. Tobacco Free Project staff took into account the relative impact tobacco has had on various communities by incorporating a number of geographic and community indicators into the Request For Application (RFA) itself, and into the review process that determined which applications would be funded. The RFA’s intent was to fund at least three culturally and ethnically diverse organizations to implement the Community Action Model and reduce tobacco’s impact on their respective communities. The RFAs were released to prospective bidders online on September 20, 2010, and paper copies were made available to those without Internet access. (A second RFA was also released in which applicants were asked to mobilize their community to pass a landlord disclosure ordinance using the CAM.)

In order to ensure that tobacco control funding addressed priority populations needs, the following was incorporated into the evaluation criteria of the Require for Applications:

A) Community and geographic priority funding indicators: 1) access to health care; 2) percentage of population aged 5-24 years of age; 3)country of birth; 4) income; 5) language isolation; 6) race/ethnicity, sexual orientation; 7) smoking prevalence;8) exposure to environmental health hazards; 9) availability of clinics/hospitals, 10) availability of health foods; 11) tobacco retail outlet density; and 12) stationary and mobile and health hazards.

B) Extent to which applicant organization has organizational policies and practices that meet culturally competency standards that address organizational values, participatory planning, provision of capacity building, translation policies, and staff diversity.

Released on September 20, 2010, the RFAs required applicants to provide information on how they selected target populations and/or geographic areas with regard to the geographic and community funding indicators. Applicants were also required to provide information on organizational policies and practices that meet culturally competency standards (Appendix C of RFA). The evaluation criteria included ratings in both of the areas (Appendix G of the RFA).

In order to reach a wide array of diverse organizations, a data base of 722 community based organizations and community leaders was compiled. The RFA announcement was emailed to the active list of community-based organizations and community leaders announcing the availability of tobacco funding along with a copy of the two Requests for Proposals on September 20, 2010. The RFAs were also posted on the San Francisco Dept. of Public Health’s Tobacco Free Project website. A total of 12 applications were submitted to the San Francisco Department of Public Health’s Tobacco Free Project in response to the RFAs and were then reviewed by an outside panel of tobacco health educators from nearby counties.

Tobacco Free Project staff answered questions about the RFA up until the bidder’s conference which was held on December 1, 2010. Tobacco Free Project staff covered information about the Community Action Model (CAM), cultural competency standards along with key dates and requirements for the RFA including information about the community and geographic indicators and the cultural competency requirements. Following that, Tobacco Free Project staff could no longer answer RFA-related questions in order to ensure no one received an unfair advantage.

Three tobacco health educators from neighboring counties were recruited to review the funding proposals. One month prior to the review panel meeting, one of the panelists withdrew due to new employment. Another panel member was recruited who subsequently withdrew due to a conflict as the organization she worked for decided to apply for the RFA. At that point, it was too late to find another panelist that met the criteria so the panel consisted of only two members—a Latina and a Native American. The LGBT community was also represented. The review panel convened on November 15, 2010. There were 12 applications submitted of which seven were selected for funding.

 

Contract negotiations were held between December 2010 and January 2011 with the seven projects funded. Two of the seven projects are working on the disclosure policy: the San Francisco Apartment Association and the Dolores Street Community Services/Mission SRO Collaborative. SFAA focused on mobilizing property owners and DSCS focused on mobilizing tenants. The two projects collaborated on the development of the model policy re: landlord disclosure. The five remaining projects selected their own “action” or policy activity based upon the first step in the CAM. Each of the seven funded projects developed a work plan as part of their subcontractor agreement.

The Tobacco Free Project evaluator developed several cultural competency standards checklists that were completed by the funded projects. These checklists were designed to help organizations assess their cultural competence and to document the ways in which they had implemented the Community Action Model using culturally competent and linguistically appropriate methods.

Organizations were asked to rate themselves on the organizational cultural competency standards found in Appendix 1.. 

Brief description of the evaluation design 

A non-experimental evaluation design was used that relied primarily on qualitative research methods. Evaluation of the objective did not include outcome measures, but instead focused on a series of process measures and a series of benchmarks leading toward achievement of the objective. The evaluation design focused on tracking: 1) the extent to which TFP outreached to diverse communities when disseminating the RFA; 2) if the community and geographic indicators and cultural competency standards were used in the selection of funded programs; and 3) whether or not the funded organizations successfully implemented the CAM in a culturally competent manner in order to affect community-level norms change, and 4) where organizations rated themselves with regard to organizational cultural competency.

Record/document review, observation, checklists, training evaluation surveys and key informant interviews were the primary methods used to evaluate this objective. A description of the process measures and the methods used by the evaluator to track each of the process measures follows.

There was no “sample.” Evaluation activities were conducted with both Prop. 99-funded and all the MSA-funded projects…

Instruments and procedures used for data collection 

• Record Review. Record review and observational data collection methods were used to review the data base used to disseminate the RFA, the applications submitted in response to the Tobacco Free Project’s RFA’s, and the Review Panel’s scoring sheets to assess the extent to which use of the parity criteria and adherence to cultural competency standards influenced reviewers’ decisions of the funded groups.

• Evaluation Training Surveys: Two written evaluation training surveys were developed by the evaluator and administered to training participants at the conclusion of both the Project Coordinators and the Advocates’ Training sessions. The first survey targeted the project coordinators and consisted of 10 open and closed-ended questions designed to test the coordinators’ knowledge of tobacco as a social justice issue, the Community Action Model, and research methods, and to assess the cultural competency of the training. The second survey targeted advocates and consisted of 13 closed and open-ended questions. The survey measured whether the CAMs five key concepts were clearly communicated to participants, their overall satisfaction level with the training, cultural relevance of training, and extent to which participants felt prepared to implement the CAM model to complete their advocacy projects. A total of 40 advocates completed the Advocate Evaluation Training Survey. Both surveys were developed by the evaluator with input from Tobacco Free Project staff.

Both the project coordinator and advocate training surveys were pen-to-paper surveys administered immediately following the training. For Spanish speaking advocates the surveys were translated verbally by the translators present at the training so that the forms could be completed by all participants present.

• Checklists: Working together with Tobacco Free Project staff, the evaluator developed four checklists that were used to document the extent to which geographic and community indicators influenced which projects were selected for funding, assessed the extent to which funded organizations met culturally competent standards, and the degree to which the funded projects used culturally competent and linguistically appropriate methods to implement their respective projects. The checklists included:

The Geographic and Community Checklist was developed by the evaluator and, based upon a set of community and geographic indicators, used to identify the priority populations and/or geographic areas to which community-based tobacco control funding will be allocated.

The Organizational Cultural Competency Standards Checklist was developed by the evaluator and based upon a series of indicators that were extracted from research including relevant organizational standards according to the National Standards on Cultural and Linguistically Appropriate Services (National Office of Minority Health). The checklist was provided to organizations to rate their own organizational cultural competence.

• Diagnosis Checklist was designed to collect information about the ways in which staff and advocates conducted surveys, focus groups and key informant interviews in order to try and ensure that cultural competency was part of ways in which these research methods were conducted. The checklist was used by the evaluator when reviewing each project’s written final report.

Action Checklist was designed to encourage organizations to think about the audience they are trying to reach and go through a checklist to help ensure materials and other information provided to them are appropriate in both form and content.

• Key informant Interviews: A total of twelve key informant interviews were conducted. Three “pre” interviews with Executive Directors of three of the funded projects, and six additional “post” interviews were conducted with two advocates from the same three organizations. Two additional interviews were conducted with Tobacco Free Project staff and the one member of the Review Panel. (Initially, the evaluator planned to interview three members of the Review Panel, however due to a last minute cancellation there were only two members of the Review Panel, and one of them was out of the country and unavailable to be interviewed at the time the key informant interviews were conducted.)

The Tobacco Free Project successfully funded a diverse group of applicants. All of the funded projects targeted diverse communities that are adversely affected by tobacco, e.g. African American, Asian, Latino, youth, LGBT.

The Tobacco Free Project made funding decisions using criteria that measured the extent to which applicants responded to geographic and community indicators designed to increase organizations knowledge of their targeted communities. 

• 100% of funded organizations had not only the highest overall scores but also scored “high” with regard to their incorporation of the Geographic and Community Indicators into their applications. 
• Ten of the twelve applicants (7 Neighborhood and 3 Smoking Disclosure) received a “high” rating on the indicators sections of their respective applications but all of them did not receive funding; however, 100% of the funded applications did receive a rating of “high” on those sections.

The Tobacco Free Project’s Request for Application process was viewed as culturally appropriate and not overly burdensome to organizations submitting applications. In interviews, staff from all three organizations felt the RFA circulated by the Tobacco Free Project was clear, not overly burdensome to prepare and that the Community Action Model described was appropriate for their targeted population.

• 100% of organizations indicated the RFA materials provided to them by the Tobacco Free Project helped them develop their proposals and were judged as helpful and were highly useful.
• 66% of the organizations felt it was somewhat easy to respond to the parity indicators section of the application, while the other organization found it challenging to respond – in one case because data was hard to obtain, in the other, because they wanted more clarity on what exactly constituted a ‘highly rated’ cultural competency standard.
• 100% of organizations felt the CAM model was culturally competent and well suited to their targeted populations and could be easily adapted where needed to undertake their advocacy work.

Tobacco Free Project staff successfully trained projects to operate their in a culturally competent manner and carry out their advocacy work. 

• 86% of Project Coordinators felt the TFP training prepared them “extremely well” for the advocacy work ahead of them.
• 100% of Project Coordinators were able to name at least one way they might adapt a step in the Community Action Model (CAM) to ensure that it is culturally appropriate for use with their targeted community including: making materials bilingual or trilingual when needed; using statistics related to tobacco’s impact on a community and being specific about how that particular community is affected; when asking about gender, utilizing a more expansive list of categories for transgender populations; and pilot testing surveys with community members before administering them broadly to ensure they are culturally and linguistically appropriate for the targeted population.
• 100% of advocates were able to identify two locations and two research methods they could use to collect information for step 1 of the CAM (their “diagnosis”).
• 100% of advocates indicated they “learned a lot from the training” and that the trainers were “knowledgeable and well prepared.”

By incorporating checklists into funded projects’ Scopes of Work, TFP staff increased projects’ awareness of and engagement in the use of culturally competent practices while implementing their advocacy projects. 

• 86% of organizations had incorporated or were in the progress of incorporating all of the cultural competency standards outlined in the Organizational Checklist into their organization.
• 100% of Executive Directors indicated their organizations recruit, retain and promote staff at all levels and that leadership is representative of the populations they serve.
• 57% of Executive Directors reported their organization has a strategic plan outlining clear goals, policies, operational plans, and management accountability/oversight mechanisms to provide culturally and linguistically appropriate services. The remaining 43% said it was in progress.
• 86% of Executive Directors but only 66% of advocates reported they felt that board, staff and volunteers reflect the cultural and linguistic characteristics of the populations they serve.
• 57% of organizations reported staff was working to develop identified internal processes to assess the ability of the organization to provide culturally competent services.

Data from the Diagnosis Checklists submitted by each project revealed that:

• 100% of projects submitted their data collection protocols for review by the evaluator.
• 71% of projects pre-tested their data collection tools with members of the target audience before using them on a wider scale.
• 86% of projects included questions in their data collection tool about the preferred language and ways in which target audience members preferred to receive information.
• 100% of the projects included questions in their data collection tools that assessed target members’ support for the project’s proposed action.

Key informants (both staff and advocates) were asked to describe ways in which they had incorporated cultural competency into their advocacy work in new ways. Their responses included:

• Making materials bilingual or trilingual when needed and being responsive to changing populations within their targeted area.
• Using statistics related to tobacco’s impact on a community and being specific about how that particular community is affected.
• When asking about gender, utilizing amore expansive list of categories for transgender populations.
• Pilot testing surveys with community members before administering them broadly to ensure they are culturally and linguistically appropriate for the targeted population.
• Providing ongoing training and education for staff and members of the Board of Directors.

”Staff participates in regular and on-going trainings that are designed to increase staff ability to work ethically and competently in a diverse organization and respond effectively to the needs of the communities served. Trainings have included harm-reduction approaches to substance use, community organizing, immigrant rights, Americans with Disabilities Act, and fair housing laws. Cultural competency trainings for the Board are also planned on an annual basis. Participants in our various leadership programs are also often invited to participate in staff organized trainings, and receive extensive orientation training that involves dialogue about cultural competency.” 

Project Coordinator 

Tobacco Free Project staff efforts to ensure that funded organizations responded to a series of community-identified parity indicators (geographic and community oriented), thereby ensuring that tobacco funding went to those populations and/or areas most adversely impacted by tobacco, were successful.

However, smaller organizations that have not been funded by the Tobacco Free Project previously are at a definite disadvantage when applying for funding. A prior understanding of the CAM and the parity indicators definitely provides an “edge.” If the Tobacco Free Project wants to continue to expand their efforts to fund organizations in the communities most impacted by tobacco, perhaps they might consider holding a separate mini-training on the CAM and the parity indicators following their release of the application and before the deadline for application submission.

The checklists were developed as a way to try to keep funded projects thinking about cultural competence as they implemented their advocacy projects, encouraging them to assess their organization’s cultural competence and to adopt culturally competent practices while implementing their advocacy efforts.

Efforts to assess organizational cultural competence standards were successful, but changing them was not really attainable. Since no funding was provided by TFP for organizations to take on restructuring or making other organizational substantive changes, a self-assessment (Organizational Checklist) was all that could be achieved. The checklist helped organizations to do a self-audit with regard to cultural competency standards, but there was no funding or focus on ensuring organizations addressed any deficiencies they revealed.

Because strategic planning efforts are infrequent and costly in both monetary and staff hours, it is often difficult to ensure organizations’ strategic plans adequately reflect rapidly changing cultural needs and populations.

Diagnosis Checklists and providing funded projects with Final Report outlines ensured that organizations kept cultural competency in mind throughout their efforts to implement their projects.

Increasing the extent to which organizations incorporate cultural competent standards into their advocacy work was easily achieved. By requiring organizations to complete checklists in which they had to document the ways in which they used culturally competent methods when collecting data, presenting information, and seeking policy change from key decision makers made the goal attainable.