Big ideas,small data: Opportunities and challenges for data science and the social services sector

The dangers of extrapolation in small data

DS can be an invaluable tool for creating and supporting more effective and efficient practices for social services. However, even the most advanced methods may struggle to detect and assess the individualistic and humanistic characteristics of social service data. The difficulty in collecting and quantifying nuanced information, such as dimensions of a client's social interactions, feelings, and life experiences, means that some data that impact social service programs are not operationalized. The inability to capture this type of information limits inferences that can be made using DS methods. Through careful consideration of the characteristics of small data, generalization, and extrapolation can be mitigated, and informative analysis for decision makers produced.

The performance and depth of insights gleaned from DS methods are linked to both the size and quality of data. Data quality can be characterized by how clean, objective, and consistent data are (Kitchin and Lauriault, 2015). Compared to big data, small data requires a higher level of quality as its characteristics may exacerbate and amplify erroneous conclusions. For example, in the author's experience working with an NGO intercepting human trafficking, the completeness of collected data was found to be inconsistent (i.e. substantial missing or incomplete data). Although missing data is secondary to an organization's humanitarian mission, for the data scientist, understanding blank data fields presents invaluable information and impacts possible conclusions. While research interest in improving DS to applications on small data is growing (Chahal et al., 2021), we caution against the use of these methods in the social services context, as there are critical considerations regarding how these data should be managed and utilized. For example, algorithmic decision-making in child welfare systems has been criticized as inappropriate use of DS (Ho and Burke, 2022).

Small data are prone to extrapolation. Failing to recognize this may result in generalized information used to make decisions that can negatively impact those the programs are designed to serve. In addition to data extrapolation, DS experts should carefully consider any algorithmic bias that may be produced and both the short and long-term tractability of models and results.

As data scientists, we need to be careful not to generalize analysis before considering its implications. For these reasons, decisions related to operational and program changes cannot solely rely on the results of a DS model but need stakeholder guidance and input early and throughout the modeling process. Through careful selection of DS methods, we can better reduce the unintended consequences of applying these methods in the social services domain.

Considerations moving forward

There are multiple ways to use and apply small data appropriately: predictively, prescriptively, and descriptively. Each offers a contribution to social services. While these data strategies offer exciting possibilities, ethical considerations can balance the possible unintended consequences. It is important for practitioners to remember that sometimes the simplest solutions are the best solutions. DS applications offer unique insights but need to be tempered by the socio-cultural context and societal structures that often impact those in social services.

It is vital for social services professionals in these contexts to develop a basic understanding of the methods being used. It is equally important for data scientists working on these problems to have a firm understanding of the domain context. This collaborative partnership of domain experts, social scientists, and data scientists ensures the appropriate use of methodologies for social services.

Refining and leveraging available data

The volume, quality, and storage of data in social services are generally reflective of the size and resources available to each organization. The barriers to effective data management and data use are significant. However, striving for improvement across the field of social services is a useful goal. The collection of more data is an obvious request by many data scientists; however, in a resource-constrained environment such as social services, this is typically difficult for budgetary reasons, as has been noted (Abazajian, 2022).

Despite the challenges related to increasing the volume of data collected, existing data can be leveraged. One way is to refine and improve data documentation. Data documentation in the most basic sense, focuses on describing information such as variable units. However, providing more detailed information regarding why a data point is collected can benefit analysis (Rasmussen and Blank, 2007). Although data documentation likely exists in an organization, data elements that are beneficial to data scientists are often not considered. For example, social services programs make frequent refinements to their programmatic models, changes in their service delivery systems, data collection processes, or definitions used. Carefully documenting these changes and when they occur can help improve analysis and create opportunities to explore new research questions and directions, as previously discussed. Existing data can further be improved through ongoing feedback loops between the data generated, those collecting the data, and doing the data work (e.g. those in administrative, clerical, and data entry positions) (Møller et al., 2020). In addition, while organizations do collect similar data, such datasets are often disparate. Nuances of the data and its collection processes may cause bias when comparing data across organizations. Improving the data documentation process and elements included could help to mitigate bias and allow for a more equitable comparison across organizations.

Improving collaborations

To prescribe actionable insights, it may take time for the volume of data to accumulate sufficiently. However, this doesn’t mean data scientists cannot play an early role. Assuredly there is much value in the continued partnership between social service professionals, social scientists, and data scientists. Data collected by organizations are often limited to collecting data specified by their funder(s) and current goals rather than an awareness of details that could be relevant for future analysis. Programmatic delivery models evolve over time, and building commensurate data analytics capacity from the beginning by involving data people can improve the overall data life cycle and ensuing analysis. As is true for any DS application, considering the domain context is crucial in social service applications. Working alongside social service practitioners and social science researchers enhances the integration of domain knowledge and theory, allowing for improved data and domain-specific insights.

Understanding the population of interest can be informed locally as well as on a larger scale through data sharing and collaboration agreements. DS researchers could leverage collaborations within and between organizations to enhance the already existing, practitioner-collected data and help operationalize it. While considering privacy, pooling data with other agencies serving the same population gives more data and increases the information gained. However, the analysis must be carefully interpreted as variations across definitions and collection practices can impact the interpretation of results. As discussed earlier, data documentation can be a catalyst for improving an organization's analysis. Data documentation can also improve analysis within collaborations as it helps alleviate some of the data variation and privacy concerns.

Collaborations play an integral role within the social services sector and can occur in multiple ways to address the needs present. Together, data scientists, social scientists, and domain experts can (i) support advocacy for greater investments in data infrastructure in the social services sector; (ii) build more accessible data-sharing tools so that this sector can pool data resources, and (iii) implement cross-training so that social science and humanities researchers become more conversant in computational thinking and data science researchers become more aware of the possibilities of complimenting their methods with stories and qualitative data and methods.

Respecting data limitations

Regardless of the quality and volume of data, an understanding is needed to recognize that data is limited in what it can reveal. Patterns may be weak, and caution is needed when drawing conclusions. Limitations of the analysis (such as bias, weak signal, etc.) should be made known to social service practitioners so they can decide how to best proceed within their organization's context. Therefore, it is vital for social services professionals and data scientists to consider the appropriate use of DS, its related implications, and the possibility of unintended consequences.

The many complexities of social services can limit the interpretation of results, such as variability across service delivery models, client populations, and governing policies and regulations. Analysis and considerations for behavioral interactions of these aspects in social services, both within a program as well as between other entities working in this space should be considered as these elements influence results. For example, while data from a local provider may gather insights regarding a certain program aspect, data may be limited in understanding and capturing the larger systemic influences of a program such as policy, program perceptions, and agency behaviors. Model selection criteria should be case-by-case dependent and need to critically examine the different social elements and objectives when selecting a model. Data scientists should be equipped to offer alternative social-centered approaches when limitations are present such as qualitative methods, human-centered design, and participatory groups.