Assessment of Knowledge, Skills and Attitudes in the use of Information Technology to Support Hospital Pharmacists Clinical Practice: Development and Validation of a Questionnaire

Rabelo Néri, Woods, and França Fonteles: Assessment of Knowledge, Skills and Attitudes in the use of Information Technology to Support Hospital Pharmacists Clinical Practice: Development and Validation of a Questionnaire



The use of information technology has been growing around the world, especially in health. In this context, knowledge and skills for the optimal use of electronic devices and technological tools are increasingly required in any field of health professional activity, with the objective of effectively managing information, safety and save costs.1-2

The hospitals use of technologies accelerates diagnoses, allows the registration of care in real time; alerts on critical results, among other facilities. Besides that, the application of information technology generated strategic information for the organization. In hospital pharmacy, in the beginning, the information generated was focused on the management and costs, but in the last decades, the practice of clinical pharmacy has brought important information to support safe and effective medication management.3-4

In clinical practice, the hospital pharmacist faces different challenges in selecting the best information in the midst of growing scientific production, not always of good quality. To select properly the best evidence, it is required that the pharmacist have knowledge and skill in the use of electronic devices and technological tools, that allow to meet the challenges of providing the right information and promoting greater security for the user (patient or professional) and to optimize care results.5-6 In addition to this scenario, the communication process among health professionals has undergone profound transformations, incorporating the use of information technology and electronic devices, it is essential for the pharmacist to be prepared to manage this technology to obtain the best possible result in his clinical practice.

In this context, to determine the knowledge, skills and attitudes of the hospital pharmacists in the use of information technology it is strategic. For this purpose, it is necessary to elaborate and validate a questionnaire to ensure the internal validity and properly select the sample to ensure external validity. This process enhances the quality of measurement tools.7-8 The use of a validated tool “permits mapping the gaps, signaling opportunities for a professional development approach to meet the current challenges in the daily routine of clinical pharmacists in hospitals.

This paper aims to present the development and validation process of a questionnaire to evaluate Knowledge, skills and attitudes in the use of information technology to support hospital pharmacists clinical practice.


This study presents the methodological description of the process of internal and external validation of a questionnaire to evaluate Knowledge, skills and attitudes in the use of information technology to support hospital pharmacists clinical practice. The study was carried out by adopting the following steps: questionnaire development, internal validation, receptivity test and external validation.

Questionnaire development

A review of the literature was carried out for the elaboration of the questionnaire.9-16 The questionnaire included openended questions (Questions 1 and 26), Likert scale ratings (Questions 7, 11, 12, 14 and 25) and multiples choice questions.17

Internal Validation

The internal validation of the questionnaire was carried out by a committee of specialists, composed by pharmacists, with clinical practice, chosen for convenience and divided into categories, according to the practical experience in clinical hospital pharmacy. Clinical pharmacy experience time is conceptualized as the time (in years) in which the pharmacist performs one or more clinical activities in hospital routine.15 The number of selected experts can vary from 5 to 20.8

The first contact with the experts was made to invite them to participate in the research, explaining the purpose of their participation; role of the evaluator; time needed to respond and the risks. After they accepted, the Free and Informed Consent Form and the questionnaire were sent by e-mail. The specialists were invited to evaluate the relevance of each question and the clarity of the content of each item.8,17-18

The criterion of relevance considered the importance and appropriateness of the question to achieve the proposed objectives (relevance scale: 1-irrelevant, 2-somewhat relevant, 3-relevant and 4-very relevant),19 and if all necessary dimensions of the objective were included. In relation to clarity, the editing of the items was evaluated, such that the concept expected to be measured was fully understandable and adequately expressed (clarity scale: 1-not clear, 2-somewhat clear, 3-clear and 4-very clear).8-18 The validated questionnaire was structured as an electronic document on the Google Docs® platform.

In each item, there was a space for recording the evaluators’ suggestions to improve the question.18 In addition, the evaluator also analyzed the questionnaire in a global way, for clarity and relevance, using the same scales adopted for individual issues. The principle of methodological triangulation was used, in order to allow a moment of interaction between the researcher and the evaluator, to clarify doubts and validate the adjustments.20

The evaluation was performed independently, by each specialist, from July 12 to 21, 2015, with the Free and Informed Consent Form and the questionnaire returned by electronic means (e-mail) to the researcher. For the validation of the questionnaire, it was determined the Content Validity Index-CVI, which measures the proportion or percentage of evaluators who are in agreement on certain aspects of the instrument,21 and of its items for each question as well as for the questionnaire as a whole (formula used: CVI=No. of responses 3 or 4/No. total responses). The questions with CVI less than 0.9 (1st cycle) were reviewed and adjusted, re-evaluated by the specialists (2nd cycle) and subsequently incorporated if the CVI was 0.9 or higher. To verify the validity of new instruments the CVI recommended is 0.9 or higher.22

To evaluate the degree of agreement in content validation, the agreement percentages-AP were calculated for relevance and clarity using the formula: AP = Number of evaluators who fully agreed with the item (score 4) / Total number of responses x 100). Values greater than 80% were considered as an acceptable rate of agreement.23

Receptivity test

The receptivity test was performed after the validation and adjustment process. In this test a link to the questionnaire was available on two social networks: Facebook® and the website farmaceuticoclinico®, from July 23 to 25, 2015, being self-filled by the pharmacists without the researcher’s intervention.17

The answers obtained in this phase were used to know the receptivity (speed to obtaining a response to the questionnaire) and difficulties in filling it. The answers obtained were not added to the final survey.

External Validation

From July 27 to September 27, 2015, the final questionnaire was sent to Brazilian hospital pharmacists, randomly selected by systematic sampling.

Data analysis

The validation process data were organized in a spreadsheet (Excel® version 1804) to determine the Content Validity Index and the Agreement Percentages. The data are presented as frequencies, percentages, mean and standard deviation.

Ethical approval

The study was approved by the Ethics in Research Committee, Federal University of Ceará - Brazil, CAAE: 44308815.7.0000.5054 and followed the recommendations of the Helsinki Declaration.24


The questionnaire was built with 26 open and closed questions that allow the evaluation of 74 variables (Appendix 1). The variables were divided into 8 dimensions to determine the profile: of the participant (n=9); the access and use of: electronic devices/internet (n=11), databases (n=25) and software (spreadsheets, text editors and statistical data analysis) (n=7). Other variables addressed how the pharmacist works the information resulting from the practice to enable its dissemination (preparation of reports and articles) (n=1); how and for how long they keep files containing records of clinical activities (n=1) and what they expect (functions, reports and features) from software for clinical practice documentation (n=14). We also collected the degree of agreement with some difficulties experienced by the hospital pharmacist in performing clinical activities (n=6). The questionnaire was developed to be used in electronic media (Google Docs®).

After the construction of the questionnaire, the internal validation was performed by specialists (n=10). The specialists had different times of professional experience in Clinical Pharmacy area (1 to 10 years experience) and 70% (n=7) already had publications or research in clinical pharmacy. The majority of specialists were female (80%; n=8) and with age range varying from 25 to 40 years. Among these specialists 30% (n=3) were second-year resident pharmacists, 50% (n=5) were pharmacists with different levels of experience and 20% (n=2) were lectures (50% with master’s degree and 50% with doctorate degree). For all selected experts, the questionnaire was adequate to achieve the proposed objectives and had the necessary dimensions.

The questionnaire evaluation was performed in 2 cycles votes. In the 1st cycle, the initial instrument was evaluated by the experts obtaining the votes, for each question, presented in Figures 1 and 2. The questionnaire was adjusted and the 2nd evaluation cycle was performed (Figures 3 and 4). The CVI was calculated in both cycles for each question (Table 1) and for the questionnaire as a whole. Questions with CVI less than 0.9 (n=2) were analyzed in detail, rewritten and resubmitted to a new analysis by the evaluators in 2nd vote cycle. The overall CVI was calculated by summing all CVI (clarity and relevance) and dividing by the total number of responses in the clarity and relevance assessment (n=52), resulting in CVI (1st cycle) = 0.9635 ± 0.0566 and CVI (2nd cycle) = 1.

Figure 1

First cycle votes for relevance of each question.
Figure 2

First cycle votes for clarity of each question.
Figure 3

Second cycle votes for clarity of each question.
Figure 4

Second cycle votes for relevance of each question.
Table 1

Content Validity Index for relevance and clarity of the questionnaire.

1st vote cycle2nd vote cycle1st vote cycle2nd vote cycle
QuestionNumber of responses 3* and 4**CVIR1Number of responses 3* and 4**CVIR2Number of responses 3# and 4##CVIC1Number of responses 3# and 4##CVIC2

CVIR1: Content Validity Index, first cycle votes, for relevance; CVIR2: Content Validity Index, second cycle votes, for relevance; CVIC1: Content Validity Index, first cycle votes, for clarity; CVIC2: Content Validity Index, second cycle votes, for clarity; *: Relevant; **: Very relevant; # Clear; ##: Very clear. databases, indicating ways to elaborate professional training policies in Brazil.26

The degree of agreement between the experts regarding relevance and clarity was calculated in both vote cycles (Table 2). The global agreement in the first cycle was 82.50% and in the second cycle was 99.61%.

Table 2

Degree of expert agreement, for relevance and clarity, of the questionnaire in 1st and 2nd vote cycles.

Relevance (%)Clarity (%)
1st vote cycle 2nd vote cycle1st vote cycle2nd vote cycle
81.92 99.6183.0799.61

During the 1st cycle validation process, 33 suggestions were received, being 12% (n=4) for grammatical adequacy; 27% (n=9) on inclusion of new options; 39% (n=13) for adjustments in the format of the question-naire and 21% (n=7) suggesting the maintenance of the glossary of terms aggregated to each question.

After the validation, the questionnaire with 26 questions was submitted to verification of acceptability during 3 days, being answered by 14 professionals (1% of the final sample), with 2 suggestions: one word that was misspelled and the inclusion of the option “other” in question “17”. The receptivity test had 4.67 responses per day. After the suggested adjustments, the questionnaire was re-launched on the Google Docs® platform, generating another link that was sent by e-mail to the selected pharmacists.

The final questionnaire with 26 questions was sent to 1373 Brazilian hospital pharmacists, randomly selected by systematic sampling. In this period, it was obtained 348 valid questionnaires resulting in a response rate of 25.34%.


The method used to elaborate a questionnaire, as well as the semantic adequacy of the items are essential for the success of a survey. An important step in the construction of a questionnaire is its prior content validation by individuals who may potentially participate in the research, thus ensuring the language’s and options adjustment and its compatibility with the target audience.8

In our questionnaire, the experts were selected with a range of characteristics that allows us to obtain a better result. The age range of the evaluators (generation X and Y) made possible different views on the proposed questions, due to the differences in the use of information technology and electronic devices.25 Besides that, the specialists with different times of practice were intentionally selected to obtain different degrees of professional maturity, facilitating the identification of fragilities, especially the presence of dubious texts and the absence of options to answer the questionnaire.18

The degree of agreement between the evaluators about relevance and clarity was superior to 80% in both evaluation cycle indicating that the questionnaire was sufficiently clear and relevant to be used to a survey.23 The suggestions received were essential for the improvement of the questionnaire and contributed to the results of the acceptance test and answers obtained in the final collected data.

Similarly, we believe that the electronic format facilitated the process of capturing the answers, because it is possible for the participant to choose the best time to respond. In addition, it reduces the influence of the researcher’s opinion on the participant and the cost of the research.7

Similar to Pedreira, et al..23 we could verify that the panel of evaluators was an essential step to improve the quality of the questionnaire, since it increased the clarity of the questions, contributing to the instrument to be able to evaluate, in fact, what is intended.

The concern about the proper training of the pharmacist for the use of information technology and devices has been discussed in different countries1,5,9,10,14 and has gained relevance in the face of the increasing number of people who own and use electronic devices that produce health information and the growing institutional databases on health. In this context, the pharmacist needs to be prepared to use this information for the benefit of the patient and society and to promote the safe and appropriate use of medicines. Therefore, it is necessary to map the weaknesses.

The validated questionnaire was applied in a survey identifying gaps in knowledge and skills in the use of IT, software and evidence-based

Our study had limitations that could have influenced some of the results. The content validity index used is subjective and could have been supplemented by application of other psychometric measures. In addition, it is possible that the present research did not identify all possible variables related to the knowledge, skills and attitudes in the use of information technology by pharmacists.

As the instrument was constructed and validated in Brazil, it is recommended, for application in other countries, that the technical, linguistic and semantic equivalence be evaluated in advance.


The questionnaire presents measures of the degree of agreement for content validation (percentage of agreement and content validity index) in acceptable standards being the content considered valid to the diagnosis of the knowledge, skills and attitudes of the use of information technology and electronic tools to support the clinical practice of hospital pharmacists. The expert panel contributed decisively to the quality of the questionnaire.

The receptivity test was important to estimate the time of collection and to evaluate the interest of pharmacists in participating in the research, being a useful indicator for the adequate planning of the research.

This questionnaire may lead to knowing the gaps in knowledge and skills in the use of IT, software and evidence-based databases and the information obtained can be used to prepare the pharmacist for better performance in their clinical activities.

For the future, it is strategic to prepare new questionnaire to investigate if and how the pharmacist is being trained to help patients to make the use of the little data (such information that the patients have about his own health state, obtained from everyday objects) and if and how the pharmacist is being trained to help the institutions to use the big data (information about people health).


We thank all pharmacists who participated in this study and Conselho Federal de Farmácia, Conselhos Regionais de Farmácia and Sociedade Brasileira de Farmácia Hospitalar e Serviços de Saúde for the information and mailing list.


The authors declare no conflict of interest.



Agreement percentages


Content Validity Index


Content Validity Index, first cycle votes, for clarity


Content Validity Index, second cycle votes, for clarity


Content Validity Index, first cycle votes, for relevance


Content Validity Index, second cycle votes, for relevance.



Institute of Medicine (US) Committee on the Health Professions Education Summit , author; Greiner AC, Knebel E , editors. Health Professions Education: A Bridge to Quality. Washington (DC): National Academies Press (US); 2003. 10.17226/10681


World Health Organization - WHO. Resolutions and decisions , author. Resolution WHA. 2005; 58 (28): 121 –3


Kohn L, Corrigan J, Donaldson M , authors. To err is human: building a safer health system. Washington DC , author. Committee on Quality of Health Care in America, Institute of Medicine. National Academy of Sciences. 2000


Black AD, Car J, Pagliari C, Anandan C, Cresswell K, Bokun T , authors. et al. The impact of e health on the quality and safety of health care: a systematic overview. PloS Med. 2011; 8 (1): e1000387


Fox BI, Flynn A, Clauson KA, Seaton TL, Breeden E , authors. An Approach for All in Pharmacy Informatics Education. A J Pharm Educ. 2017; 81 (2): 38


Donaldson LJ, Kelley ET, Dhingra KN, Kieny MP, Sheikh A , authors. Medication without harm: WHO’s third global patient safety challenge. Lancet [Internet]. 2017; 389 (10080): 1680 –1.


Hoss M, Caten CS , authors. Processo de Validação Interna de um Questionário em uma Survey Research Sobre ISO 9001:2000. Prod Produção. 2010; 11 (2): 104 –19


Alexandre NMC, Coluci MZO , authors. Content validity in the processes and adaptation of measuring instruments. Cien Saúde Coletiva. 2011; 16 (7): 3061 –8


Balen RM, Jewesson PJ , authors. Pharmacist computer skills and needs assessment survey. J Med Internet Res. 2004; 6 (1): e11


Thomas P, Rutter PM , authors. A computer literacy skills profile of pharmacists residing in two counties of England. Health Info Libr J. 2008; 25 (4): 288 –94


Peterson CG, Aslani P, Williams KA , authors. Pharmacists’ online information literacy: An assessment of their use of Internet-based medicines information. Health Info Libr J. 2010; 27 (3): 208 –16


Fox BI, Flynn AJ, Fortier CR, Clauson KA , authors. Technology in pharmacy education: knowledge, skills and resources for pharmacy informatics education. Am J Pharm Educ. 2011; 75 (5): 93


Pinochet LHC , author. Information technology trends in health management. World Health. 2011; 35 (4): 382 –94


Kostagiolas P, Aggelopoulou V, Niakas D , authors. A study of the information seeking behaviour of hospital pharmacists: empirical evidence from Greece. Heal Info Libr J. 2011; 28 (4): 302 –12


Brazilian Pharmacy Federal Council. Resolution 585: regulates the clinical assignments of the pharmacists. [Internet] , author. 2013 Augst 29– 17– Aug. 13.


Gharbieh EA, Khalidi DAl, Baig MR, Khan SA , authors. Refining knowledge, attitude and practice of evidence-based medicine (EBM) among pharmacy students for professional challenges. Saudi Pharm J. 2015; 23 (2): 162 –6


Mattar FN , author. Marketing research: methodology, planning. 6. São Paulo: Atlas. 2005


Rubio DM, Berg WM, Tebb SS, Lee ES, Rauch S , authors. Objectifying content validity: Conducting a content validity study in social work research. Soc Work Res. 2003; 27 (2): 94 –104


McGilton K , author. Development and psychometric evaluation of supportive leadership scales. Can J Nurs Res. 2003; 35 (4): 72 –86


Hyrkäs K, Appelqvist SK, Oksa L , authors. Validating an instrument for clinical supervision using an expert panel. Int J Nur Stud. 2003; 40 (6): 619 –25


Lynn MR , author. Determination and quantification of content validity. Nurs Res. 1986; 35 (6): 382 –5


Polit DF, Beck CT , authors. The content validity index: are you sure you know what’s being reported?. Critique and recomendations. Res Nurs Health. 2006; 29 (5): 489 –97


Pedreira RBS, Rocha SV, Santos CAD, Vasconcelos LRC, Reis MC , authors. Content validity of the Geriatric Health Assessment Instrument. Einstein (São Paulo). 2016; 14 (2): 158 –77


Venter E , author. Bridging the communication gap between Generation Y and the Baby Boomer generation. Int J Adolesc Youth [Internet]. 2017; 22 (4): 497 –507.


World Medical Association. World Medical declaration of Helsinki: ethical principles for medical research involving human subjects , author. JAMA. 2013; 310 (20): 2191 –4. 10.1001/jama.2013.281053


Néri EDR, Meira AS, Vasconcelos HBDS, Woods DJ, Fonteles MM, de F , authors. Knowledge, skills and attitudes of hospital pharmacists in the use of information technology and electronic tools to support clinical practice: A Brazilian survey. Graetz I , editor. PLoS One [Internet]. 2017 12 (12): e0189918 2018 Feb 18;