Mixed Methods for Study of Gender Issues in Access,Application,and Attitudes Toward ICT in Higher Education Institutions in Papua New Guinea

Abstract

For more than a century, there has been an ongoing paradigm debate between qualitative and quantitative methodological communities that has led to a growing interest in the mixed methods approach. This article seeks to contribute to this ongoing discussion by presenting results of a mixed methods study employed to explore gender issues in the access, application, and attitudes toward information communication technology (ICT) in higher education institutions in Papua New Guinea (PNG). The data presented here involved more than 900 students and faculty members. In each of the studied aspects of staffs’ and students’ interactions with ICT, mixed methods methodology provided a platform for deeper exploration of gender issues that otherwise would not have been discovered. Although quantitative data showed little statistically significant differences in the access to and application of ICT, qualitative data revealed deep inequalities rooted in a PNG male-dominated culture. In terms of participants’ attitudes toward ICT, qualitative analyses not only reaffirmed the statistical results but also enriched understanding of demonstrated attitudes by providing a platform for an in-depth discussion about the positive, negative, and ambivalent perceptions and beliefs held by the staff and students in tertiary institutions.

Keywords

mixed methods gender issues ICT higher education developing country

Introduction

For more than a century, there has been an ongoing paradigm debate between qualitative and quantitative methodological communities (Johnson & Onwuegbuzie, 2004) that has led to a growing interest in a mixed methods approach (Brannen, 2008; Feilzer, 2010; Teddlie & Tashakkori, 2009). Indeed, some authors consider that mixed methods has won its place in the methodological battlefield and has become “a research paradigm whose time has come” (Johnson & Onwuegbuzie, 2004, p. 14). This article aspires to contribute to the popularity of mixed methods by arguing its virtues in terms of providing tools for deeper understanding of investigated phenomena and validation of results (Bazeley, 2004) in the study of gender issues in access, application, and attitudes toward information communication technology (ICT) in higher education institutions in Papua New Guinea (PNG).

Mixed Methods

A number of arguments are posited by proponents of mixed methods research in their attempts to rationalize the use of qualitative and quantitative paradigms in one study. Giannakaki (2005) argued that “combining quantitative and qualitative methods in a single study can help elucidate various aspects of the phenomenon under investigation, providing more holistic understanding” (p. 323). Sale, Lohfeld, and Brazil (2002) maintained that the two approaches could be combined because they share a unified logic of the common goal to understand the world around us. Furthermore, the two study paradigms are united by “shared commitment to understanding and improving the human condition, a common goal of disseminating knowledge for practical use, and a shared commitment for rigor, conscientiousness, and critique in the research process” (Reichardt & Rallis, 1994, as cited in Sale, Lohfeld, & Brazil, 2002, p. 46). Explaining further the aim of combining different methods, Saukko (2003) argued that using mixed methods is like using different lenses to calibrate an optimally clear vision, so the reality is seen more clearly in a less biased and more systematic way. In the same vein, Cresswell and Plano Clark (2007) argued for using mixed methods:

Mixed methods research is a research design with philosophical assumptions as well as methods of inquiry. As a methodology, it involves philosophical assumptions that guide the direction of the collection and analysis of data and the mixture of qualitative and quantitative data in a single study or series of studies. Its central premise is that the use of quantitative and qualitative approaches in combination provides a better understanding of research problems than either approach alone. (As quoted in Cameron, 2011, p. 96)

A growing body of literature demonstrates widespread acceptance of mixed methods, especially in the social, behavioral, and human sciences research that study complex social phenomena (Creswell, Shope, Plano Clark, & Green, 2006; Greene, 2005; Rocco, Bliss, Gallagher, & Perez-Prado, 2003). Harper (2011) used mixed methods in the study of identity among diverse student population in higher education institutions. She concluded that this method provided a deeper understanding of the multiple dimensions of students’ identity and how these perspectives intersect and were renegotiated over time. Museus (2011), in his study of issues of college access and equity among first generation of Asian Americans and South Pacific Islanders, pointed out three important reasons of employing mixed methods: emphasis (both quantitative and qualitative played equally important roles in answering research questions), timing (in his case it was quantitative → qualitative sequential design employed), and purpose (qualitative methods were employed to complement and elucidate findings generated from the quantitative strand). Arguing for the importance of mixed methods in this study, Museus further observed that while quantitative analysis highlighted existing inequities in college access faced by the studied group of students, qualitative analysis complemented the results with more in-depth understanding of factors contributing to the problem and possible ways of addressing the sources of conceived inequalities.

Research Methodology

This article reports research that developed a gender profile of access to, the use of, and attitudes toward ICT among staff and students at tertiary education institutions in PNG. Parallel simultaneous multiple mixed methods was employed in this research. The study used both the between-method triangulation and within-method triangulation in the following manner:

Between-method triangulation—Both quantitative method (survey) and qualitative method (semi-structured interviews) were used to collect data for the project;

Within-method triangulation—Within qualitative strand of the study, the data were collected through semistructured interviews and open questions in survey questionnaires (Williamson, 2005).

It was believed that using both kinds of triangulation would lead to more valid results from this study with each method contributing in a specific way:

Between-method triangulation tested the degree of external validity and

Within-method triangulation ensured internal consistency and reliability (Jick, 1979).

It became apparent in this study that using mixed methods helped to create an extensive gender profile of ICT application in higher education institutions and provided participants with a voice to explain their own attitudes and perceptions of technology. Embedded within constructionist epistemology, the qualitative and quantitative strands of the study provided findings, which, when analyzed in a triangulated manner, illustrated how meanings of technology and gender were constructed within the context of higher education institutions.

The Sampling Design

There is no widely accepted typology of mixed methods sampling strategy (Teddlie & Yu, 2007); thus the sampling was designed separately for both strands of the study. Multistage cluster sampling was used for the quantitative strand to ensure the sample was representative of the population and as far as possible unbiased in any way. As a result, a total number of 898 students and 64 academic staff members from three universities and three other institutions of higher education (OIHEs) participated in the study (see Table 1).

Table 1.

Overview of the Sampling Design in the Quantitative Strand of the Study.

Selected institutions of higher education
Universities
Pacific Adventist University, Port Moresby
University of Goroka
Divine Word University, Madang
OIHEs
Holy Trinity Teachers College, Mt. Hagen
Madang Teachers College
Don Bosco Technological Institute, Port Moresby
Groups selected within the selected institutions of higher education
At faculties In the faculties of arts, business, and education, first-, second-, third-, and fourth-year students, in 36 groups At OIHEs First-, second-, and third-year students, in 9 groups Total number of groups = 45	All faculty members invited to participate in the study
Total number of participants
Female students = 369 Male students = 529 Total = 898	Female lecturers = 26 Male lecturers = 38 Total = 64

Note. OIHEs = other institutions of higher education.

The selection process in qualitative research differs substantially from that used in quantitative. Qualitative research focuses on how the sample clarifies social life. Thus, the main purpose in qualitative sampling is to select people or sites that can best help a researcher to understand the central phenomenon under study (Creswell, 2002). Convenient purposive sampling was a technique utilized in the selection of interviewees. Because ICT was a relatively new phenomenon at the majority of IHEs studied, students had rather limited experience in this area—it was decided that only faculty members were included in the qualitative research. Twenty-three academic staff (13 females and 10 males) were interviewed.

Data Collection Instruments

In the quantitative strand of this study, data were collected with survey questionnaires. They were prepared in two forms, one for students and another for faculty members. Both questionnaires were self-administered to participants in two modes: paper-based questionnaire and web-based questionnaire. Around 2 weeks before distribution of paper-based questionnaires, posters were placed around campuses with an invitation to participate in the study, information about it, and a web link to web-based questionnaire.

Different techniques were used to ensure reliability and validity of quantitative data collection. SPSS was used to assess internal validity of a 32-item scale developed for the survey questionnaire using Cronbach’s alpha technique. The result of the test, an alpha of .708, was acceptable for assuming homogeneity of attitude scale items (Burns & Burns, 2008).

To ensure content validity of the proposed questionnaires, they were pilot tested. In addition, the questions were scrutinized against the following criteria suggested by Thomas (2004):

Does the question clearly link to the objective as intended?

Do all questions as a whole provide thorough coverage of the topic?

To what extent will the questionnaire generate data required?

The validity of results of surveys depends on the response rate and the quality of response. As a result of a number of strategies utilized during the data collection process, the response rate was very high: 98% of questionnaires distributed among students and 77% of questionnaires distributed among staff members.

The qualitative strand was conducted simultaneously with quantitative. Two sources were utilized to collect qualitative data: four open-ended questions in the surveys and a semistructured interview.

The question of validity in the qualitative approach examines a close fit between the data and what people actually say and what they do. By validity, Hammersley meant “truth: interpreted as the extent to which an account accurately represents the social phenomena to which it refers” (as cited in Marvasti, 2004, p. 113). Thus, the term validity means using terminology that participants will interpret consistently, based on their perceptions, attitudes, and behaviors. To ensure validity, the pilot study interviews were conducted and the understanding of significant terms by the researcher and participants was scrutinized.

Data Analysis

Data collected in the qualitative and quantitative strands of the study were analyzed concurrently with appropriate methods. Although this strategy provides a richer understanding of the variables and their relationships, it also limits the researcher to one type of data analysis, qualitative and quantitative, on each subset of the data (Tashakkori & Teddlie, 1998).

Quantitative data were recorded and analyzed with the statistical software package SPSS. Qualitative data were managed with NVivo software; thematic networks was used as an analytical tool. Both types of analysis were followed by concurrent interpretation of data.

A validity claim in this mixed methods research was made with an integrative framework for inference quality as developed by Tashakkori and Teddlie (2008). The framework enhances the process and the outcome of evaluating the quality of conclusions that are made on the basis of the findings in a study. A meta-inference—an overall conclusion and understanding developed through an assimilation of the inferences obtained from the qualitative and quantitative strands of study—addresses both exploratory (e.g., What is a gender profile of access, application, and attitudes toward ICT in IHEs in PNG?) and confirmatory (e.g., Can ICT empower PNG women?) questions of the research and therefore provides perspective that neither the qualitative nor quantitative approaches could do alone.

The two leading criteria in evaluating the quality of inferences are design quality and interpretative rigor (Teddlie & Tashakkori, 2009). To comply with the design quality criterion, every effort was made to ensure that the appropriate procedures were selected to answer the research questions. From the first step of defining the theoretical framework, through selection of research methodology, sampling, data collection instruments, and data analysis and interpretation, each step was informed and justified with appropriate literature. The research results were incorporated into a conceptual framework to answer the research questions. Integration of the inferences drawn from multiple strands of mixed methods does not require an implied or actual agreement (consistency) of inferences. Inconsistency between two (or more) sets of findings, as presented later in the article, provides information that would otherwise be lost (i.e., not uncovered) if only one type of study were used.

Every effort was also made to achieve interpretative rigor. Concurrent interpretation of data followed results of both qualitative and quantitative analysis. The inferences were consistent with the type of evidence demonstrated.

Research Findings and Interpretations

This section reports gender-related issues in access to, application of, and attitudes toward ICT among staff and students of IHEs in PNG. The following discussion combines into one discourse the results of the qualitative and quantitative data sets. Although the quantitative results refer separately to computers and the Internet, the qualitative discussion applies the terms ICT and technology in relation to computers and the Internet. The reason for the use of such terminology is that interviewees used the terms computer, the Internet, and technology interchangeably.

Gender profile of access to ICT

The access to ICT was measured in terms of the degree of access to computers and the Internet, and the amount of time spent using computers and the Internet on an average day. The majority of academic staff and students at participating universities and OIHEs had access to computers only during their time at the institution. Depending on the type of the institution, there were differences of exposure to computers: more computers being available for students’ use at the universities, fewer computers and more restricted access provided at the OIHEs. Similarly, university staff members were provided with a desktop computer in their office, whereas OIHE lecturers had access to computers only in a common staff room.

With the understanding of institutional differences in access to computers, the study explored any gender-related differences in computer accessibility. Statistical analysis revealed no significant difference in access to computers between female and male student participants, χ²(4) = 5.894, p = .207. The majority of them accessed computers every day (females = 38%, males = 43%) or a few times a week (females = 40%, males = 35%). Similarly, there was no statistically significant difference, χ²(3) = 2.634, p = .452, in the time female and male students spent using computers on an average day. Fifty-two percent of female students and 58% of male students spent between 1 and 3 hr per day using computers.

Likewise, there were no statistically significant differences, χ²(3) = 6.239, p = .101, in the access to computers between female and male lecturers—majority of them access computers daily (females = 73%, males = 87%). Academics spent more hours per day using computer than students; however, there was no statistically significant difference between genders, χ²(2) = 0.857, p = .651. Forty-two percent of females and males used computers between 3 and 5 hr or more.

The study revealed a difference in accessibility to the Internet at different IHEs. Generally, universities demonstrated a higher level of exposure to the Internet than OIHEs. All universities provided limited free access to the Internet for their faculty members; two out of three universities provided similar service for their students. Among the OIHEs, only one out of three provided faculty members with free access to the Internet. No OIHE provided free access to the Internet for students.

In the background of institutional exposure to the Internet, the research further examined gender similarities and differences in Internet accessibility. Female and male students’ exposure to the Internet differed significantly (p < .001). Participants who did not answer any survey questions related to the Internet were considered to have had no Internet experience (females = 25%, males = 16%).

The gender disparity between students continued in the frequency of Internet access, χ²(4) = 22.159, p < .001. The majority of them accessed the Internet either a few times a week (females = 47%, males = 30%) or every day (females = 23%, males = 24%). Despite statistically significant differences in access to the Internet, female and male students showed no difference in the amount of time spent using the Internet on an average day, χ²(3) = 3.177, p = .365. Fifty-two percent of female and 59% of male students spent less than 1 hr daily.

There were fewer statistically significant differences in having no experience of the Internet between female and male faculty members, χ²(5) = 5.026, p = .413; females = 19%, males = 6%. Among those who had access to the Internet, 46% of women staff and 56% of men staff used the Internet every day and the majority of them spent every day using the Internet either 1 to 3 hr (females = 48%, males = 50%) or less than 1 hr (females = 29%, males = 28%).

As previously noted, statistical results suggested no significant difference between genders in both groups of participants in the majority of issues related to participants’ access to computers and the Internet. However, qualitative analysis did not confirm the above findings and revealed far deeper differences in access to computers in IHEs. Many interviewees pointed to difficulties that female students faced in terms of computer accessibility. Officially, all research participants at each institution had an equal opportunity to use computer facilities available to them. However, male cultural domination in PNG society, as well as their physical strength, resulted in male students having first access to computers. Female students felt that because of discriminative access to computer facilities, they were treated unfairly. Males perceived their forceful first access to computers as acceptable within PNG cultures.

Papua New Guinea is a male-dominated society. And every male seems to think and believe that they own everything and they have the right to do everything first then the female. . . . That has to do with cultural influence. So, men first, ladies last. It is cultural influence, we can’t help it. (UM11)

Another important reason for females’ unequal access to computers was related to unsafe environments. Because of the high risk of harassment, assault, and rape in PNG, some institutions restrict female students’ night movement within campuses. Although there were no such official restrictions for female staff members, some of them also considered lack of security on the campus as a disadvantage, forcing them to complete their work on computers during day time only.

Some participants, including males, found the uneven access to computers unacceptable and called for administrative action to ensure fair access. Interestingly, although a few participants expected administrative bodies to take action, no one suggested an awareness campaign to influence behavioral change within the community. It seems that, although men’s aggressive behavior was perceived as unfair toward women, it was socially acceptable, even by female participants. Change, if any, was expected to come either by administrative decision or by providing women with other means for computer access.

One of the suggested ways to solve inequities and to give women better access to computers was to assist them to purchase their own personal computer. This proposal seemed to have additional importance considering the difference in the level of computer ownership. Although there was no statistically significant difference between the female and male academic staff members in their ownership of personal computers, females = 73%, males = 76%; χ²(1) = 0.086, p = .769, the level of ownership between female and male students differed significantly, females = 31%, males = 40%; χ²(1) = 7.534, p = .006.

Gender issues in the application of ICT in higher education

The study looked into the use of computers by academic staff and students at tertiary institutions. When invited to summarize their purpose for using computers, both groups showed no statistically significant gender differences. The majority of academic staff indicated using computers for mainly academic purposes, females = 84%, males = 83%; χ²(2) = 0.084, p = .959. However, among students, 46% of males and 44% of females declared using computers for academic purposes while 20% of males and 15% of females admitted using computers primarily for entertaining purposes, χ²(2) = 5.585, p = .061.

The study further explored whether the participants’ own perception of their computer use was consistent with their actual use of computers. All participants were asked to indicate different activities performed with computers on an average day. The majority of the academic staff declared using computers for preparing lecture notes (females = 96%, males = 90%), doing research (females = 73%, males = 76%), attending to administrative tasks (females = 58%, males = 55%); in terms of entertainment, they used computers for listening to music (females = 31%, males = 29%), playing games (females = 15%, males = 24%), and watching videos (females = 15%, males = 8%). None of the computer activities manifested a significant difference between genders of academic staff. However, different results were found among students (p = .011). Female students were more likely (n = 360, $\bar{x}$ = 0.1027, s = 0.9757) to use computers for academic activities than male students (n = 510, $\bar{x}$ = −0.0724, s = 1.0114; Figure 1), whereas male students were more likely to use computers for entertaining purposes (n = 510, $\bar{x}$ = 0.1223, s = 1.0069) than female students (n = 360, $\bar{x}$ = −0.1733, s = 0.9651; Figure 2).

Figure 1.

Box plot showing the distribution of Factor 1 (Computers used mainly for academic purposes) among female and male students at tertiary institutions.

Figure 2.

Box plot showing the distribution of Factor 2 (Computers used mainly for entertaining purposes) among female and male students at tertiary institutions.

To extend the understanding of gender similarities and differences in ICT usage in tertiary institutions, application of the Internet by faculty members and students has been examined. The ICT profile showed that the main purposes for accessing the Internet by academic staff were research (88%), email (79%), news (56%), hobby-related information (26%), banking (18%), and online meeting people (16%). Although further analyses generally revealed no statistically significant differences between female and male lecturers in the purposes listed above for accessing the Internet, differences were noticeable in three instances: using the Internet for news (64% male, 43% female academics), banking (24% female, 14% male academics), and online meeting people (24% female, 12% male academics).

Female and male faculty members did not exhibit significant statistical differences either in their access to email, χ²(4) = 5.930, p = .204, or in the purpose of using email, χ²(2) = 2.744, p = .254. The majority of academics reported accessing email 4 or/and more times on an average day (58% males, 50% females). When asked to summarize the content of their emails, 62% of female and 46% of male academics indicated that the content was equally professional and other than professional. Twenty-four percent of female and 46% of male academics said that their email content was mostly professional.

As was the case for academic staff members, students at tertiary institutions used the Internet in ways similar to the ICT profile previously noted. Among the most popular activities indicated by participating students were research (88%), email (59%), entertainment (41%), and online meeting people (25%). To complete this discussion, students were asked about the kind of information for which they were searching the Internet. The majority of participants (93%) reported searching for information related to their study. The other reported types of information were the news (47%), music and videos (38%), hobby related (30%), and games (16%).

Unlike what was observed with faculty members, some differences between female and male students in obtaining access to the Internet were statistically significant. Both groups differed in email being considered as a reason for using the Internet by 68% of females and 53% of males (p < .001) and in entertainment being considered a main reason by 35% of females and 45% of males (p = .016). There was no statistically significant difference in the type of information searched on the Internet by female and male students, except for games. More males (19%) than females (12%) acknowledged looking for games on the Internet.

As indicated earlier, almost 60% of all participating students indicated using email as a purpose of access to the Internet. The difference was statistically significant, χ²(4) = 17.301, p = .002, between female and male students in their frequency of checking their email on an average day. Forty-two percent of females checked their email 2 and 3 times, and 12% checked it more than 5 times per day, compared with 32% and 6% of males in the same categories.

When invited to summarize their email content, 52% of all participating students indicated it as equally academic and other than academically related (females 53%, males 52%). However, female and male students significantly differed statistically, χ²(2) = 19.322, p < .001, in their use of email for purposes other than academically related (female 52%, male 31%).

Although the qualitative analysis confirmed the statistical results, it also provided additional insights into the gender-related issues in the application of ICT in higher education context. When discussing the issue of application of computers, there were noticeable gender differences in the perception of computer use. Women stressed the exclusive use of computers for academic activities.

I think with ladies, we really use it if we really need to use it. . . . That’s how I see because with females we have a different workload and different roles at home so the time management is something, so the time we spend with on computer particularly on the Internet, would be much more limited than males because of our other commitments particularly family commitments. That’s for men and women. (OF14)

Women’s roles as family providers and family carers restricted the use of computers to what was seen as necessary to fulfill professional obligations. Although women did not complain about having additional duties, and therefore less time to use computers (called it “normal daily life”), the number of references to this issue showed that they felt them rather strongly.

It might be argued that similarities in the application of computers in the educational context were due to very limited use of computers. Academic staff used technology mainly to prepare lecture notes and assignments for students and to keep student records. The majority of academics’ use of software applications was limited to word processing, Excel, and PowerPoint. No one mentioned using any software application in complex ways to integrate ICT into teaching and learning processes. The fact that women discerned the gender difference in the application of computers was related more to their lack of available time and other family obligations than to their actual computer use.

The limitation of the academic staffs’ computer use, as discussed here, was due to their lack of proper training. Both women and men pointed out the lack of proper introduction to computing as an obstacle to effective use of computers for teaching purposes as well as a reason for time wasted trying to figure out how to complete different tasks. They believed the institutions in which they were working should provide courses in computing as well as continuing assistance to ensure that the technology be used in more effective ways. Special attention was given to the use of the Internet. On one hand, it was seen as a vast source of information and an enormous help in preparing a course. On the other, this wonderful tool for academic work could have been better used if academics knew how to do so.

We have never been taught how to surf the Internet and get the information that we wanted. So when we are surfing the Internet we would use up all our credits looking at sites which we were not supposed to and picking up information that we thought were appropriate for us but if someone who really knew how to do research on the Internet would just look at our papers and said “this is crap, you collected information that just not vital to support what are you writing about.” So, when it comes to academic search for information we still don’t know how to do it properly because we pick up anything from anywhere. (UF20)

The important difference between female and male academic staff members in technology use lay in men spending more time on computers for leisure activities. Interestingly, primarily women pointed to this issue.

I think men have more time to sit and surf the net to use this technology. Women use it only when work requires it. At home, they will not use it this much because they are busy with house-work, housekeeping, looking after the children. All these domestic chores would offer little time for women to access, to have access to the computer. But men have all the free time to use it. (UF13)

Academic staff generally did not notice much difference between female and male students, especially in their use of computers for academic activities.

Because it is a requirement that they must learn and it’s a requirement that the work they submit must be printed out so that is why not much of difference. So, I think the use of computers is the same for the males and females . . . (UF07)

The lack of differences between female and male students in study-related computer activities could be ascribed to the fact that, officially, all students had access to the same computer facilities and they all were expected to use them for their studies. The fact that students submitted their work completed with computers could have been understood as an indicator that they used computers in the same way. However, lecturers teaching computer courses found females more focused on the tasks at hand while males were more focused on exploring computers on their own or simply doing other activities besides study.

I see the boys will be a little more tricky and they will go out of boundaries do at least other things and programs. I’ve seen this when I was supervising my students while young teacher was walking away, so to teach in the front, the girls were just following step by step instructions, they were trying to do things, while boys were on to other programs and when went, they put it off and they pretended to listen. . . . In terms of gender equity, I think women will do more things and will be more maybe fruitful in a way to get more results through using computers by using time wisely by doing the right things, you know, they will be more effective in whatever way and whatever things they will do than male. (UM11)

Students themselves admitted using computers not only for academic purposes but also for entertainment. This confirms results obtained from the survey questionnaire where 52% of students reported using computers for both academic and entertainment purposes. The favorite entertainment activities with female students were playing games and chatting with friends on email while male students favored playing games, listening to music, and exploring the Internet.

Gender issues in attitudes toward ICT

A principal components analysis (PCA) technique was applied to derive and describe participants’ attitudes toward computers and the Internet from 32 scale items in the questionnaires. A PCA with subsequent rotation (Varimax) was conducted separately for data sets concerning students’ and staffs’ attitudes and beliefs about ICT. The Keiser–Meyer–Olkin (KMO) and Barlett’s test produced criteria that supported the application of PCA (KMO measured: for students’ data set = 0.806; for staffs’ data set = 0.540). Commonalities varied: in students’ data set from 0.628 to 0.362; in staffs’ data set from 0.860 to 0.570. Applying Kaiser’s Rule and the scree test, a number of factors were deemed important as presented in Table 2.

Table 2.

Summary of Results of PCA Tests Run on the Attitude Scales for Students and Staff in Higher Education Institutions in Papua New Guinea.

Guiding factors in students’ attitudes	Guiding factors in staffs’ attitudes
1. Computers are superior to humans. 2. Technology is a beneficial tool for study and work. 3. The Internet is fascinating. 4. Working with technology is frustrating. 5. Technology becomes an integral part of daily life. 6. Technology changes the world around us. 7. Technology brings a better life. 8. The Internet has a harmful effect on a community. 9. Computers make learning fun.	1. Computers are superior to humans. 2. Technology becomes an integral part of daily life. 3. The Internet is fascinating. 4. The Internet makes life easier but has a negative impact on society. 5. Computers are a beneficial tool for an academic performance. 6. Technology makes work more fun. 7. Technology is frustrating. 8. The Internet becomes an integral part of daily life. 9. People no longer control work done on a computer. 10. Technology is fascinating and easy to learn.

Guiding factors in students’ attitudes

Guiding factors in staffs’ attitudes

1. Computers are superior to humans.
2. Technology is a beneficial tool for study and work.
3. The Internet is fascinating.
4. Working with technology is frustrating.
5. Technology becomes an integral part of daily life.
6. Technology changes the world around us.
7. Technology brings a better life.
8. The Internet has a harmful effect on a community.
9. Computers make learning fun.

1. Computers are superior to humans.
2. Technology becomes an integral part of daily life.
3. The Internet is fascinating.
4. The Internet makes life easier but has a negative impact on society.
5. Computers are a beneficial tool for an academic performance.
6. Technology makes work more fun.
7. Technology is frustrating.
8. The Internet becomes an integral part of daily life.
9. People no longer control work done on a computer.
10. Technology is fascinating and easy to learn.

Note. PCA = principal components analysis.

An independent-samples t test was conducted to evaluate the hypotheses that female and male students did not differ significantly in their attitudes toward technology. The results showed that the mean scores of two out of nine earlier identified factors were statistically significantly different: Factor 3, The Internet is fascinating, and Factor 9, Computers make learning more fun (p < .001). Additional tests were run to explore the gender differences in the two factors.

For Factor 3, The Internet is fascinating, an independent-samples t test revealed that female and male students differed significantly (p = .019) in their fascination with the Internet. Male students were more likely (n = 368, $\bar{x}$ = 0.077, s = 1.036) than female students (n = 246, $\bar{x}$ = −0.115, s = 0.933) to hold the view that the Internet is fascinating (Figure 3). Figure 4 presents additional visual information for the distribution of scores for Factor 3.

Figure 3.

Box plot showing the distribution of Factor 3 (The Internet is fascinating) among female and male students in institutions of higher education.

Figure 4.

Error bar for female and male students showing clear divergence of the 95% confidence interval Factor 3 (The Internet is fascinating) scores for each group.

For Factor 9, Computers make learning more fun, an independent-samples t test demonstrated that female and male students differ significantly in the view that computers make learning more enjoyable (p < .001). Interestingly, male students were more likely (n = 368, $\bar{x}$ = 0.128, s = 0.993) than female students (n = 246, $\bar{x}$ = −0.192, s = 0.980) to view computers as machines that make learning more fun (Figure 5). Figure 6 presents additional visual information for the distribution of scores on Factor 9.

Figure 5.

Box plot showing the distribution of Factor 9 (Computers make learning more fun) among female and male students in institutions of higher education.

Figure 6.

Error bar for female and male students showing clear divergence of the 95% confidence interval Factor 9 (Computers make learning more fun) scores for each group.

The Mann–Whitney U test was conducted to evaluate the hypothesis that female and male academic staff members did not differ significantly in their attitudes toward technology. Test results show that female and male faculty members’ attitudes toward technology differed significantly for Factor 1, Computers are superior to humans, and Factor 10, Technology is fascinating and easy to learn. Additional tests were run to explore gender differences in the two factors.

For Factor 1, Computers are superior to humans, the Mann–Whitney U test revealed a statistically significant difference (U = 186.0, p = .029) between female and male academics in their belief about computers as superior to humans. Female faculty members were more likely (n = 18, $\bar{x}$ = 0.426, s = 1.020) than male faculty members (n = 33, $\bar{x}$ = −0.232, s = 0.922) to view computers as superior to humans (Figure 7). Figure 8 presents additional visual information for the distribution of scores on Factor 1.

Figure 7.

Box plot showing the distribution of Factor 1 (Computers are superior to humans) among female and male faculty members in institutions of higher education.

Figure 8.

Error bar for female and male faculty members showing clear divergence of the 95% confidence interval Factor 1 (Computers are superior to humans) scores for each group.

For Factor 10, Technology is fascinating and easy to learn, the Mann–Whitney U test revealed a statistically significant difference (U = 179.0, p = .020) between female and male academics in their belief about technology as fascinating and easy to learn. Female faculty members (n = 18, $\bar{x}$ = 0.466, s = 0.891) were more likely than male faculty members (n = 33, $\bar{x}$ = −0.254, s = 0.975) to view technology as fascinating and easy to learn (Figure 9). Figure 10 presents additional visual information for the distribution of scores on Factor 10.

Figure 9.

Box plot showing the distribution of Factor 10 (Technology is fascinating and easy to learn) among female and male faculty members in institutions of higher education.

Figure 10.

Error bar for the female and male faculty members showing clear divergence of the 95% confidence interval Factor 10 (Technology is fascinating and easy to learn) scores for each group.

In summary, this section presented statistical analyses of the two attitude scales for female and male students and academic staff in IHEs. Among students, differences were found in two out of nine guiding factors. Male students were more likely to believe that the Internet is fascinating and that computers make learning more enjoyable. Similarly, gender differences were found in only 2 of 10 guiding factors among faculty members. Female academics were more likely to view computers as superior to humans and to view technology as fascinating and easy to learn.

Attitudes toward technology—Concurrent analyses

The quantitative analyses revealed minor gender differences in attitudes toward technology between tertiary staff and students; the majority of attitudes were positive. The only two negative aspects of technology shared by both groups referred to the frustration experienced when working with technology and a belief that the Internet has a harmful effect on a community. The qualitative analyses not only corroborated statistical findings but also deepened understanding of attitudes toward technology by providing more balanced lists of positive and negative views and beliefs about computers and the Internet (Figure 11).

Figure 11.

Attitudes toward technology held by students and staff in higher education institutions as identified in thematic network for “Attitudes towards technology.”

Positive attitudes toward technology

As noted earlier, the majority of attitudes toward technology identified in the quantitative analyses were confirmed by results from the qualitative analyses. One of the attitudes identified in the quantitative analyses was a belief that technology is a beneficial tool for study and work. The perception was also shared by both women and men although they expressed the notion in different forms. The most common expressions that referred to a beneficial aspect of technology in relation to study and work were “handy” and “useful.” The participants saw computers as very helpful in the completion of work tasks. They believed that the computers made work more efficient and of better quality. Also, they found the usefulness of computers enjoyable.

Work has become easier now. Typing is easier now, you just touched type and you get the result. And so, it empowers me in one way, it facilitates my performance. Typing is not tedious anymore, it becomes fun, and it becomes enjoyable. (UF13)

Computers not only contributed to the efficiency of work but also, as discussed by interviewees, changed the way things were done in the workplace.

In the past, our tests, worksheets, all these were typed by a school secretary. You had a draft, gave a copy to the secretary and she did it for you; or you photocopy and you cut, paste, shift into another blank sheet. But, with the computer, with the aid of the scanner, you can scan, compile, and fix it easier. . . . So, it made job a lot of easier now. (OF14)

Both genders also shared their fascination with computer technology. One of the important factors that contributed to it was the perception of technology as a limitless source of information. It provided access to a whole new world of information. Importantly, the access to information was made easy and convenient.

I appreciate the fact that I become computer literate. I use it for almost for everything. It’s limitless, especially with access to Internet. The database is available and kind of information that is there is mind-boggling. So you know, you got a wealth of information literally at your fingertips basically. (UM08)

In addition, technology was seen as powerful because it gave access to knowledge.

Computer knowledge is very powerful because it’s learning. (OF02)

I’ve gotten to realize more about power of what these machines or the computer is got to be actually aiding me, you know, in terms of information now. (UF12)

Negative attitudes toward technology

The quantitative analyses revealed two negative attitudes shared by all participants regardless of gender: “Working with technology seen as frustrating” and “The Internet has a harmful effect on a community.” Both findings were confirmed by the qualitative analyses. The participants shared that using technology may be frustrating, especially when it did not perform to expectations. Other difficulties that contributed to the negative feeling about using computers were problems with unstable power supplies, viruses and the fear of losing data, and computers being valuable commodities thereby attracting thieves. In addition, faculty members and students complained about the lack of proper training. On one hand, participants pointed to a relationship between good training and creative use of technology; on the other, the lack of training experienced by the majority of staff and students caused confusion and lowered their satisfaction with the quality of work completed with computers.

I would like the IS lecturers to offer lectures on how to manage Internet usage. The target groups to be all first years and upwards for those who missed out/interested. This will help individuals to manage well their quota. The current situation shows that we students waste a lot of our quotas browsing everywhere, opening up unwanted information, sometimes we do not understand we are really doing or seeing, Example, Internet promotion games, etc., to win a gift. Explain to us students if there’s any danger registering unnecessarily on webs. (UFS4y)

Among negative aspects of technology, both genders widely discussed harmful effects of the Internet on society. Despite the positive perception that the Internet makes life easier, it was also seen as having a gross negative impact, especially on the young generation. Special attention was given to pornography and its damaging effect on persons and male attitudes to women. Although banned by PNG law, pornography remains easily available through the Internet. Such easy availability of pornographic materials worried the participants who saw them as having bad influences on the Internet users. A lot of time was wasted due to such unlawful searches of Internet content.

Sometimes you can take your attention away to all kinds of things that are on the Internet like pornographic materials that are not good for children, and lecturers. I know that the things are there that you can just spend all your time, waste your time doing things that really take you away from your work, and your ethics, and how you can use things; there are things there that just take you away by. When you have access to the Internet, you have access to every good and bad thing. And some people choose to spend more time on the bad things than the good things. (UM11)

In the discussion, special attention was given to harmful effects of the Internet on younger generations. It worried parents, especially those who were unable to control children’s activities on the Internet, that their children might not follow traditional ways.

Well, already there are illegal sites, pornographic sites; and like you know watching current affairs on TV. Kids as young as 13 and 14 access the information from the internet on how they can commit suicide, so that’s a worry, you know? Papua New Guinea parents, there are some parents that are able to afford the service and kids are smart, once they learnt the use of internet. The negative aspects of it will be behaving like the other kids in the other western societies already behaving. That’s a worry. (UF07)

Damaging influences of the Internet were also seen in the wider perspective of negative changes in the entire society. Unrestricted access to pornographic materials was believed to induce questioning of traditional ethical values.

And I see that it . . . bad influence in the way people see things, and cultural beliefs, such as sex education. Access to the pornographic images, all these things is taboo in our culture and the Internet can break that. And it will have a lot of impact on the lives of people, young people, especially males. And after you access to all kinds of images, it will directly got impact into social problems like rape, all other things, early marriages, and all kinds of experimentations of what they see on the internet which is bad for the young people in PNG. I can see a lot of impacts, bad impacts about the Internet, computers especially the Internet to the younger people more than those in colleges and universities. (UM11)

To deal with negative aspects of the Internet, participants called for personal and social responses. On the personal level, one way to avoid the content that could be damaging was to discipline oneself while working with the Internet. On the social level, to protect society and its traditional values, many suggested introduction of censorship or some other form of control of Internet content.

Another negative attitude identified in the quantitative analyses that was confirmed by the qualitative discussion was “People no longer control work done on a computer.” People become dependent on technology to the point of being controlled by it. In a short period of time since computer proliferation among academia in the country, technology became so important that for many it seemed impossible to complete academic duties without it. Many women and men expressed their dependency with the oft-repeated phrase, “I cannot do without it [the computer],” while others shared more of their experiences:

I use a computer and I get really frustrated when the power goes out because I depend a lot on the computer for all my courses and even mark students’ work on the screen. . . . If there is power blackout, we all just move out of offices because we all rely so heavily on a computer while before, the power went off, you can still work on other things. Sometimes, office is even closed because everything is computerized. (UF21)

For some research participants, their dependency on technology seems to have reached a state of addiction. Although lecturers reported over-reliance on technology, the problem seemed to be more prevalent among students. Some students admitted skipping meals to remain working with a computer or breaking up their relationships as a consequence of all their time spent on computers. Different terms were used to describe the fixation: Apart from addiction, students used terms like computer freak or slave to the Internet. Some lecturers saw students’ addiction to computers as an impediment to their academic progress. While acknowledging that technology has the potential to support learning and teaching, too much time devoted to technology may have adverse effects on students’ expected progress.

Positively, some people realized that being personally disciplined about computer use would mitigate addiction to technology.

In my own time, yes, but during working time, it’s work. If I have time at home, yes, I play games; but now, with the demand of job, it’s either job or a game, otherwise your job is not completed; it requires some discipline too. (OF14)

As discussed earlier, qualitative analyses confirmed quantitative studies in relation to attitudes toward computers. In addition, qualitative results pointed to two other negative attitudes toward technology. First, “Technology replaces social interaction” expressed people’s disillusionment with technology used as a platform of social interaction. Generally, they saw technology as a valuable means of communication with family and friends; however, they realized that technology might replace face-to-face contact with others. They saw this new social interaction model as alien to traditional Melanesian cultures; thus, they had a negative perception of technology used for communication purposes.

The second negative attitude related to computers being seen as luxury items available only to a select group of pri-vileged people. Two important factors contributed to this perception: the high cost of personal computer ownership and huge social disparities as observed by a participant in this research:

It is a very new technology and many Papua New Guineas do not have access to this. A computer first of all, a computer set and the Internet. The Internet is expensive and restricted to only few; you know the kind of people in higher level of country where they can be able to afford a computer and have the Internet linked up and all this. It’s very expensive. (OM22)

The two factors were interconnected. Because computers and the Internet were very expensive, only the relatively wealthy could afford them. Huge disparities in technology’s proliferation in PNG caused very limited access and use of computers and the Internet in rural areas. Thus, computer use and Internet connection in educational institutions was seen as a privilege.

I think that students all across every institution should be given privilege of access and usage of computers and internet to be on equal footing with the academic performance in other modernized countries. Government should look at the policy to introduce internet access and usage to all institutions in the country. By far and foremost most of the disadvantaged ones in this area are the ones in the rural areas. It seems that there is already a trend here where the elite have more access than those unprivileged ones who come from the lower class families. (OMS4y)

Ambivalent attitudes toward technology

One of the two ambivalent attitudes identified in the qualitative analysis was ascribing human characteristics to computers. Participants, women and men, academic staff and students, used different terms to express this attitude. One of them was computer is my partner. When asked to clarify the term, a female lecturer said,

I am using the computer a lot; I am beginning to depend on the computer a lot now. If I don’t have one, I’ll feel a bit inadequate. (OF02)

The computer was also perceived as a friend and companion able to fill emotional and social needs as well as assist in the study.

I love computers; they are like a friend to me. When I’m lonely they are there for me. When I need to complete my school work I turn to them for help. (UFS2y)

I fall in love with my computer in the office every day. It’s my best friend. It plants a seed of curiosity in my mind to pursue knowledge with a sense of determination and enthusiasm. (UML/OQ)

Categorizing people’s attitudes was difficult because the participants’ terms implied positive and negative connotations. On one hand, such terms as workmate or companion insinuated practicality of computers and their positive important role in completion of academic work. On the other hand, such terms as best friend or lifetime partner alluded to exclusiveness of computers in someone’s life and tendency toward computer addiction.

The other ambivalent attitude toward technology identified in the qualitative analyses only was the perception of technology as a new phenomenon alien to PNG cultures. However, although technology was seen as a foreign concept that required a new set of skills, the realization of its importance initiated a call that all Papua New Guineans should advance in knowledge and use of computers. There seemed to be no way back to pre-computer times.

Computer skills are new skills and knowledge to Papua New Guineans. It requires time and effort to help all Papua New Guineans to get access to these facilities. Many things are still new for Papua New Guineans in terms of using it. (UML/OQ)

No one wants to go back to biro and paper now. It has become so useful now, I think there is no going back so, there will be more and more inventions. Next time computers might do away with thinking. (OM04)

Ambivalent also was the perception of computers as a foreign concept alien to PNG cultures. On one hand, computer technology was seen as an outlandish concept that has no connection with traditional PNG cultures. This perception brought the negative connotation and uncomfortable feelings about the contact of foreign computer-related ideas with traditional values, which were discussed in the previous section. On the other hand, the attitude also had positive connotation because people realized that technology has become a part of their own life and has effected positive changes in private and social lives.

Mixed methods for study of gender issues in access to, use of, and attitudes toward ICT

This article presented results of mixed methods methodology employed in the study of gender issues in access to, use of, and attitudes toward ICT in higher education institutions in PNG. Presented analyses contribute to the discussion on mixed methods methodology and position of the author as a proponent of mixed methods approach to the study of gender issues in the context of technology and education. In each of the studied aspects of staffs’ and students’ interactions with ICT, using mixed methods methodology provided a platform for deeper exploration of gender issues, which otherwise would not be discovered.

In terms of computer and Internet accessibility, quantitative results suggested no statistically significant difference between genders in majority of issues related to participants’ access to computers and the Internet. The only revealed differences were found among students where females were more likely to have no experience with the Internet. However, qualitative analysis revealed far deeper disparities in access to ICT. The study identified numerous factors that contributed to female students’ difficulties in computer access. Male cultural domination and their physical strength resulted in male students having first access to computers. Gendered social roles, and consequently social expectations for women to take care of their families first, limited female academics’ time available for computer use. Lack of security on campuses restricted female participants’ work with computers to day hours only. In the study of gender issues in access to ICT within the context of higher education institutions in PNG, the qualitative strand has played complementary role to the findings of quantitative strand of the study. The evidence presented earlier supports the notion raised by Creswell et al. (2006) that the qualitative research “is not always in a supportive, auxiliary role” (p. 8) to quantitative research. Using both methods made possible in this research to not only present the “big picture” of who and how often they accessed computers and the Internet but also listen to the story of the participants’ daily experiences in access to ICT.

In the two other studied aspects of staff and students interactions with ICT—their use of and attitudes toward computer technology, the presented results confirmed the generative power of mixed methods inquiry, which offers “potential for better, enriched, more insightful understanding” (Greene, 2005) of the issues under investigation. Attitudes toward ICT identified in the quantitative analyses were not only confirmed by qualitative analyses but also provided further thoughtful consideration and understanding. Men and women participating in this study share the notion that technology is a beneficial tool for study and work. They were fascinated with technology perceived as a limitless source of information easily and conveniently accessible. Both genders also shared a number of negative attitudes. All participants expressed frustration using technology, especially if it did not perform to expectations. Confusion and a low satisfaction level with the quality of work completed with computers were associated with the lack of training, unstable power supply, viruses and fear of losing data, and computers being valuable commodities that attracted thieves. Despite the positive perception that the Internet makes life easier, it was also seen as having a gross negative impact on the younger generation. Pornography, banned by PNG law but easily available through the Internet, was seen as making traditional values ambiguous and thus having a damaging effect on the entire society. The study also revealed two ambivalent attitudes toward ICT that will need further examination in the future. First, participants were ascribing human characteristics to computers. They saw computers as a friend and companion to fill emotional and social needs as well as to assist in study. However, they also used terms that alluded to the exclusiveness of computers in someone’s life and tendency toward computer addiction. The second ambivalent attitude was the perception of technology as a new phenomenon alien to PNG culture. On one hand, the stress put on the computers as alien, and thus having no connection with traditional PNG cultures, brought the negative connotation and uncomfortable feelings discussed earlier. On the other hand, the attitude also had positive connotations because people realized that technology has become a part of their own lives and has effected positive changes in their private, social, and professional lives.

Although quantitative data showed little statistically significant differences in the access to and application of ICT, qualitative data revealed deep inequalities rooted in PNG male-dominated culture. In terms of participants’ attitudes toward ICT, qualitative analyses not only reaffirmed the statistical results but also enriched understanding of demonstrated attitudes by providing a platform for an in-depth discussion about the positive, negative, and ambivalent perceptions and beliefs held by the staff and students in tertiary institutions. The evidence presented in this article supports Green, Caracelli, and Graham’s (1989) notion that “mixed-methods study, qualitative and quantitative methods are used to measure overlapping but also different facets of a phenomenon, yielding an enriched, elaborated understanding of that phenomenon” (p. 258). While the quantitative strand of the study was more concerned with the actions and behaviors of study participants, the qualitative strand explored par-ticipants’ attitudes and nuanced meanings women and men ascribed to their daily interactions with technology. Indeed, the use of mixed methods allowed enriched interpretation of gender issues involved in access, application, and attitudes toward ICT among staff and students of tertiary education institutions in PNG.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research and/or authorship of this article.

Author Biography

Iwona Kolodziejczyk is an associate professor in the Faculty of Arts and Social Sciences at Divine Word University. Her research interests include mixed methods and research ethics; gender, higher education, technology, culture, postcolonialism and behavioral interventions.

References

Bazeley

(2004). Issues in mixing qualitative and quantitative approaches to research. In Ruber

Gadner

Richards

(Eds.), Applying qualitative methods to marketing management research (pp. 141-156). UK: Palgrave Macmillan. Retrieved from http://www.researchsupport.com.au/mmissues.pdf

Brannen

(2008). The practice of a mixed methods research strategy: Personal, professional and project considerations In Bergman

M. M.

(Ed.), Advances in mixed methods research (pp. 53-65). London, England: SAGE.

Burns

R. B.

Burns

R. A.

(2008). Business research methods and statistics using SPSS. Los Angeles, CA: SAGE.

Cameron

(2011). Mixed methods research: The five Ps framework. The Electronic Journal of Business Research Methods, 9, 96-108.

Creswell

J. W.

(2002). Educational research planning, conducting, and evaluating quantitative and qualitative research. Upper Saddle River, NJ: Merrill Prentice Hall.

Creswell

J. W.

Shope

Plano Clark

V. L.

Green

D. O.

(2006). How interpretive qualitative research extends mixed methods research. Research in the Schools, 13(1), 1-11.

Feilzer

M. Y.

(2010). Doing mixed methods research pragmatically: Implications for the rediscovery of pragmatism as a research paradigm. Journal of Mixed Methods Research, 4, 6-16.

Giannakaki

M.-K.

(2005). Using mixed-methods to examine teacher’s attitude to educational change: The case of the skills for life strategy for improving adult literacy and numeracy skills in England. Educational Research and Evaluation, 11, 323-348.

Green

J. C.

Caracelli

V. J.

Graham

W. F.

(1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11, 255-274.

10.

Greene

J. C.

(2005). The generative potential of mixed methods inquiry. International Journal of Research & Method in Education, 28, 207-211.

11.

Harper

C. E.

(2011). Identity, intersectionality, and mixed-methods approaches. New Directions for Institutional Research, 2011(151), 103-115. doi:10.1002/ir.402

12.

Jick

T. D.

(1979). Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly, 24, 602-611.

13.

Johnson

R. B.

Onwuegbuzie

A. J.

(2004). Mixed-methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14-26.

14.

Marvasti

A. B.

(2004). Qualitative research in sociology: An introduction. London, England: SAGE.

15.

Museus

S. D.

(2011). An introductory mixed-methods intersectionality analysis of college access and equity: An examination of first-generation Asian Americans and Pacific Islanders. New Directions for Institutional Research, 2011(151), 63-75. doi:10.1002/ir.399

16.

Rocco

T. S.

Bliss

L. A.

Gallagher

Perez-Prado

(2003). Taking the next step: Mixed methods research in organizational systems. Information Technology, Learning, and Performance Journal, 21(1), 19-29.

17.

Sale

J. E. M.

Lohfeld

L. H.

Brazil

(2002). Revisiting the quantitative-qualitative debate: Implications for mixed-methods research. Quality & Quantity, 6, 43-53.

18.

Saukko

(2003). Doing research in cultural studies: An introduction to classical and new methodology approaches. London, England: SAGE.

19.

Tashakkori

Teddlie

(1998). Mixed methodology combining qualitative and quantitative approaches. Thousand Oaks, CA: SAGE.

20.

Tashakkori

Teddlie

(2008). Quality of inferences in mixed-methods research: Calling for an integrative framework. In Bergman

M. M.

(Ed.), Advances in mixed methods research (pp. 101-119). London, England: SAGE.

21.

Teddlie

Tashakkori

(2009). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Thousand Oaks, CA: SAGE.

22.

Teddlie

(2007). Mixed methods sampling: A typology with examples. Journal of Mixed Methods Research, 1, 77-100.

23.

Thomas

S. J.

(2004). Using web and paper questionnaires for data-based decision making: From design to interpretation of the results. Thousand Oaks, CA: Corwin Press.

24.

Williamson

G. R.

(2005). Illustrating triangulation in mixed-methods nursing research. Nurse Researcher, 12(4), 7-18.