Fast-Track Your Abstract Screening: Mastering ASReview for Accelerating Abstract Screening and Evaluating Decisions From Automatic-Screening Methods

Review and simulation modes

The left panel of ASReview offers two primary modes: review and simulation. Review mode is designed for real screening tasks and is the primary focus of this tutorial. In this mode, users can import nonlabeled, partially labeled, or fully labeled data sets and use ASReview to screen abstracts, either randomly or systematically, employing AI-aided screening.

Simulation mode, in contrast, enables users to explore ASReview using only fully labeled data sets. Users can upload previously screened abstracts or select data sets from ASReview’s Synergy collection, a collection of data sets from different fields used to benchmark ASReview (van de Schoot et al., 2021). This mode allows testing different model configurations in a simulation environment to evaluate model performance and identification rates for specific cutoff values of the data-driven heuristic. In essence, simulation mode mimics the screening of unlabeled data by using preexisting labels to simulate user decisions during screening. Consequently, this mode supports the examination of the theoretical performance of various screening configurations and was instrumental in the large-scale simulation studies that informed the recommendations presented in this tutorial (Campos et al., 2024; de Bruin et al., 2025; König, Zitzmann, Fütterer, et al., 2024; König, Zitzmann, & Hecht, 2024).

Data upload

The data set should be a tabular list of references, with each reference containing an abstract in a column explicitly named “abstract” and a column named “data_id” that provides a unique number for each record. The data must be stored in one of the following file formats: .csv, .tsv, .tab, .ris, .txt, or .xlsx. Before initiating AI-aided screening, the data set should be deduplicated.

To initiate a new project in ASReview, a data set must be uploaded using the options in the center of the landing page (see Fig. 2). By default, the “File” tab is selected. In this view, users can upload data files by either dragging and dropping them into the upload area or selecting a directory after clicking the upload area. Alternatively, data sets can be uploaded via the URL option by providing a URL or DOI. The “Search” option, designed to search OpenAlex (Priem et al., 2022) for data sets, is inactive in the version used for this tutorial (Version 2.1.1). The “Discover” option, in contrast, is active and provides access to ASReview’s built-in data sets from the Synergy collection (van de Schoot et al., 2021), which are particularly useful for using simulation mode.

The “Import” option allows loading existing ASReview project files. Each ASReview project is stored in an .asreview file that contains all data, model configurations, and screening decisions for that project. Project files are stored in the default installation directory (C:/user/[username]/.asreview), persist through software upgrades or reinstallation, and are automatically detected and reloaded when ASReview is reopened. These files can be shared among researchers, providing a straightforward way to ensure reproducibility and facilitate collaborative screening.

All active review projects appear under “Current Reviews” on the landing page once a data set has been uploaded. All completed review projects are listed below this section under “Finished Reviews.”

General settings

Additional configuration options are available in the lower left panel of the landing page under “Settings” and “Community” (see Fig. 2). In “Settings,” users can customize ASReview’s appearance, including switching between “System,” “Always light,” and “Always dark” display modes. Font size can be adjusted, the screening layout can be set to “Landscape” view, and the interface can be configured to always display model information. We recommend enabling “Show model information” because this feature provides useful context during screening and is referenced multiple times in this tutorial. However, all of these settings can be modified at any stage of the screening process.

Tags

We recommend creating specific tags before initializing the screening (see Fig. 5). For example, users can create a tag group named “Exclusion Reason” and add specific exclusion categories within it—such as those based on the population, intervention, comparison, outcome, and study design (PICO-S) criteria or other criteria (Methley et al., 2014)—allowing reasons to be assigned during screening by simply clicking the corresponding tag. In the exported data, each tag is represented in a dedicated column with values of 1 (“tag applies”) or 0 (“default”; “tag does not apply”) to indicate its presence. Therefore, tags provide valuable support for reproducibility by streamlining documentation of screening decisions. Unfortunately, tags can only be altered and added but not removed after they have been created.

OPEN IN VIEWER

Fig. 5.

Defining tags.

Model selection

ASReview allows users to select from several preconfigured AI models—called “Learners”—or to build a custom configuration by selecting individual components (see Fig. 6). The default learner (“ELAS u4”) in this version uses TF-IDF with bigrams as the feature extractor, an SVM classifier, a maximum (certainty-based) querier, and a balancer. In addition, the multilingual (“ELAS l2”) and high-capacity (“ELAS h3”) variants, available via the Dory extension, replace TF-IDF with transformer-based embeddings and require greater computational resources. Although the default learner has performed well across the Synergy data sets used to benchmark ASReview (https://github.com/asreview/synergy-dataset), studies in the psychological and educational literature suggest that alternative settings may be more effective in these research areas (de Bruin et al., 2025). In this tutorial, we demonstrate how to adjust these settings to suit psychological reviews as well.

OPEN IN VIEWER

Fig. 6.

Selecting the machine-learning model.

To build a custom learner, four components must be selected: feature extractor, classifier, querier, and balancer (see Fig. 7). The feature extractor converts abstracts into numerical representations. Built-in options include One-Hot and TF-IDF, whereas installing the Dory extension provides access to additional extractors, including various BERT-based models. Although ASReview recommends the default “ELAS u4” model (see the official ASReview documentation: https://asreview.readthedocs.io/en/latest/index.html) because it is lightweight and fast, we advise selecting a custom configuration that combines the logistic regression classifier with the SBERT feature extractor (default: “all-mpnet-base-v2”) for the AI-aided screening phase (see below). This combination not only achieved the shortest screening time to identify 95% of the relevant literature but also showed the least variability across data sets and across different prevalence rates of relevant abstracts. In addition, it consistently ranked first or second in performance when the data-driven heuristic was applied to psychologically oriented literature (Campos et al., 2024; König, Zitzmann, Fütterer, et al., 2024). Other transformer-based extractors include “LaBSE” and “multilingual-e5-large,” which support more than 100 languages, and “mxbai-embed-large-v1,” a high-capacity model designed for nuanced semantic matching.

OPEN IN VIEWER

Fig. 7.

Customizing one’s own machine-learning model.

The classifier predicts whether an abstract is relevant based on the extracted features. ASReview includes naive Bayes, SVM, RF, and LR as built-in options. The Dory extension adds more advanced algorithms, such as AdaBoost, gradient boosting (e.g., XGBoost), and neural networks. However, previous evaluation studies found that the SBERT feature extractor performed best when paired with the LR classifier; this combination is therefore recommended (Campos et al., 2024; König, Zitzmann, Fütterer, et al., 2024).

The querier controls the order in which abstracts are presented for screening. Options include maximum (showing the most likely relevant abstracts first), uncertainty (showing abstracts for which the model is least certain), random, and top-down. Mixed strategies, which interleave maximum selection with a small proportion of random or uncertainty-based choices, can help balance exploration and exploitation. In most evaluation studies, the maximum querying was used. Therefore, we recommend this option.

Finally, the balancer determines how training data are sampled during model updates. The “Balanced” option is recommended for most reviews because it increases the proportional influence of relevant studies, thereby counteracting the dominance of irrelevant studies in the data set. Whereas pre-2.x versions of ASReview supported multiple balancing methods, the version used here supports balancing only sample weights. Although users can disable the “Balanced” option, doing so is not recommended. In data sets with substantial class imbalance—a common situation in abstract screening—disabling it may reduce models’ capability to identify relevant abstracts because the algorithm is more likely to focus on features associated with irrelevance given the much larger proportion of irrelevant abstracts.

Prior knowledge

Before initiating screening, users may search for specific studies they already know to be relevant or irrelevant. By selecting “Search,” users can enter keywords or phrases, and ASReview will return matching abstracts using pattern matching (see Fig. 8). Any identified prior knowledge can be used to train the model during its first iteration.

OPEN IN VIEWER

Fig. 8.

Manual prior-knowledge selection.

This step is optional, and we generally recommend not using this functionality. Although it might be beneficial in cases in which prevalence is anticipated to be really low, randomly identifying prior knowledge can reduce the risk of a selection bias that influences the ranking of abstracts. When no prior knowledge is provided, screening begins with the random presentation of records until one relevant abstract and one irrelevant abstract have been identified. Reaching this threshold activates the selected model, which is then trained on the labeled abstracts and orders the remaining records according to predicted relevance. However, in our recommended workflow, we do not use this functionality either.

Navigation pane

The navigation pane is displayed or hidden depending on whether landscape view is enabled or the user’s screen is too small. When hidden, it can be opened by clicking the three-line icon in the top left corner. At the top of the navigation pane, users can return to the landing page, where all projects are displayed. Below this, the pane contains the options “Dashboard,” “Reviewer,” “Collection,” and “Customize,” with “Reviewer” serving as the default screening view (see Fig. 9).

Reviewing

The default page displayed after project creation is the “Review” mode, which enables users to classify abstracts. Alongside the abstract, ASReview displays the title and when available, a link to the study via its URL or DOI. When “Show model information” is enabled in the “Settings” menu, the active screening model is displayed above the title. When screening randomly, this area reads as follows: “This record is presented in a random manner” (see Fig. 9). When screening with an ML model, it instead shows the number of labeled abstracts, the feature extractor, the classifier, and the balancing method. When a selected model cannot be fitted, a warning appears—for example: “Model training error: TypeError: sparse array length is ambiguous; use getnnz() or shape[0]. Change model on the Customize tab.”

Below the abstract, any predefined user tags are displayed. During screening, users can add tags to individual abstracts by clicking on them—an especially useful feature for documenting exclusion reasons or flagging abstracts for reconsideration or team discussion. Tags can also be modified or added later in the project interface via the “Customize” page, as outlined below. In addition, users can add notes to an abstract by clicking the icon in the bottom right corner (depicted as a sheet with a pen).

Dashboard

The “Dashboard” allows users to monitor the screening process. It displays the number of relevant, irrelevant, and unlabeled abstracts and a pie chart showing their proportions (Fig. 10). A stopping rule can also be defined here. By clicking “Set Threshold,” users can specify the cutoff value for a data-driven heuristic as either an integer or a percentage. This feature streamlines the screening experience by eliminating the need to manually check whether the threshold has been reached.

OPEN IN VIEWER

Fig. 10.

Dashboard: screening progress, stopping, and history chart.

In addition, the “Dashboard” provides automatically generated and continuously updated visualizations (Fig. 10). These include “History,” which shows the sequence in which abstracts were labeled; “Density,” which displays the proportion of relevant abstracts within the last 10 screened abstracts (see Fig. 11); “Wave,” which shows the number of consecutive irrelevant abstracts in relation to the number of screened abstracts including any defined cutoff for the data-driven heuristic; and “Recall,” which visualizes screening progress over time. Among these, the “Wave” and “Recall” charts are the most informative for assessing model performance and will be explained in the AI-aided screening phase for semiautomatic abstract screening.

OPEN IN VIEWER

Fig. 11.

Dashboard: density visualization.

The “Dashboard” also offers a “Words of Importance” view that lists the terms the AI model considers most influential during classification, grouped into relevant and irrelevant categories. Although these cannot be altered, they may provide users with some insight into how the model is functioning. For the purposes of this tutorial, however, only the “Progress” statistics (“n_relevant,” “n_irrelevant,” “n_unlabeled”) and the “Stopping Rule” are of primary interest. For detailed descriptions of all dashboard charts and visualizations, we refer users to the official ASReview documentation.

Collection

Navigating to “Collection” allows users to reexamine screening decisions. ASReview provides information on which model was used to screen a specific abstract and allows filtering by studies labeled relevant or irrelevant. In addition, users can search for notes and for abstracts used as prior knowledge (Fig. 12). All abstracts of studies that were selected or identified by ASReview as prior knowledge can thus be reviewed. In addition, looking for abstracts that contain a note can be useful for identifying key studies when marked accordingly or for identifying any other abstracts that contain a note. Furthermore, by clicking the three dots in the bottom left corner of an abstract, notes can be added, screening decisions can be reverted, and labels can be removed; the latter functionality is displayed but not yet available. This feature is likely to be fully supported in future versions. Finally, users can export their results by clicking “Export” in the top right corner of the screen.

OPEN IN VIEWER

Fig. 12.

Collection.

Customize

Finally, users can customize their screening configurations. These options provide the same functionality as during project creation (see Fig. 4). Users can create or modify tags, add prior knowledge, and define or change the screening model. When a model is changed, it will be trained using all previously labeled abstracts and the selected prior knowledge. The new model will be available as soon as its training is complete. However, users can continue screening with the old configuration until the new configuration is set up.

Project setup

To initialize random prescreening, users must first upload the data set as described above. In the following example, we use the data set “Tang_et_al_2022_unlabeled_pre.csv,” which is available exclusively through our materials (https://osf.io/xrb9z/overview). To follow our tutorial, users must ensure the data set includes a column named “data_id” that stores a unique identifier for each abstract. When this information is stored in a differently named column, ASReview may delete it.

After uploading the data, ASReview will evaluate the completeness and uniqueness of the abstracts. When duplicates are detected, we strongly recommend removing them using a deduplication tool of your choice and then uploading the cleaned data set as a new project. When the data set contains no duplicates and includes 100% of abstracts and titles, users can proceed to create tags and define the screening model. For tagging, we recommend that all exclusion criteria—or at least exclusion categories (i.e., PICOS)—are represented as tags.

For model selection, a custom learner (AI model) should be defined by setting the query strategy to “random” (see Fig. 13). This ensures that references are presented in a purely random order, thereby disabling AI-aided screening. Although ASReview defaults to random presentation when no prior knowledge is provided, explicitly setting the query strategy to “random” prevents the system from switching to systematic querying once the first relevant and irrelevant abstracts have been identified. All other model settings can remain at their default values during this screening phase. We do not recommend searching for prior knowledge at this step because it will be identified through random screening of a subset of abstracts. Once tags are specified and the model is defined, users can continue by clicking “Screen” in the bottom right corner (see Fig. 13).

OPEN IN VIEWER

Fig. 13.

Model setup for random prescreening for semiautomated abstract screening.

Screening

After the project setup is complete, screening can begin. As noted above, we recommend screening at least 100 abstracts and continuing until at least one relevant abstract and one irrelevant abstract have been identified. All relevant abstracts should be annotated with notes to mark them as key studies, which facilitates the later application of the key-study stopping rule.

The “Dashboard” can be accessed at any time to monitor screening progress. Note, however, that dashboard visualizations are not informative during the prescreening stage. Users should nonetheless document the number of screened abstracts, the number labeled as relevant, and the number labeled as irrelevant because this information is used to estimate the prevalence of relevant abstracts in the data set (see metrics above).

To apply the key-study stopping rule, future versions of ASReview will allow users to prepare the data directly in the “Collection” tab. There, relevant abstracts with attached notes can be filtered and labels removed by clicking the three dots in the upper left corner of an abstract. In the version of ASReview used here, label removal is not yet supported, even though the option is displayed. As a temporary workaround, we recommend exporting the data, including all relevant, irrelevant, and yet unlabeled abstracts. Note that during screening, ASReview automatically adds several columns to the data set: “asreview_label” stores the screening decision (1 = relevant, 0 = not relevant, missing = not screened), “asreview_time” abstracts the date and time of each decision, and “asreview_note” contains user-added notes. For each user-defined tag, ASReview also generates a column indicating whether the tag was applied (1) or not applied (0).

When opening the data, all relevant labeled abstracts will be placed at the top, and all irrelevant labeled abstracts will be placed at the bottom. Thus, users can remove the labels in “asreview_labels” column from all but one relevant abstract and document the “data_ids” of these abstracts. The manipulated data set can then be uploaded to a new project. When labels remain unchanged, all labeled abstracts will be treated as prior knowledge. When only one relevant article is identified during prescreening or when the key-study stopping rule is not applied, the data set is already prepared for AI-aided screening.

Finally, the information gathered during this phase should be documented (see Table 1). In our example, we screened 100 abstracts, of which four were relevant, resulting in an estimated prevalence of 4%. We used three relevant abstracts as key studies. In our repository, the exported file is labeled “Tang_et_al_2022_unlabeled_pre_completed.csv,” and the file with removed labels for key studies is named “Tang_et_al_2022_unlabeled_AI.csv.”

Evidence

When selecting stopping rules, both logical reasoning and empirical evidence underscore the importance of aligning thresholds with the estimated prevalence of relevant literature. From a logical perspective, higher prevalence reduces the intervals between relevant and irrelevant abstracts, meaning that even under random screening, any cutoff for the data-driven heuristic will largely depend on prevalence. In addition, because ML algorithms improve with larger training data sets, lower-prevalence conditions—by providing fewer relevant examples when the overall sample size is held constant—slow learning and delay the identification of relevant studies. This ultimately influences the performance of the time-based heuristic.

Empirical findings support these mechanisms. König, Zitzmann, Fütterer, et al. (2024) found that a 2.5% data-driven cutoff produced median identification rates above 95% when prevalence was 5% to 10% but only around 60% when prevalence was 0.5% to 1%. They also observed that larger data sets improved performance and prevalence effects were nonlinear: Increasing prevalence from 0.5% to 5% reduced screening time to reach 95% recall, but further increasing prevalence to 10% increased screening time again because of the larger number of relevant abstracts to retrieve. Campos et al. (2024) reported similar findings: Higher prevalence reduced performance, whereas larger data sets improved it. In their sample of data sets with a median prevalence of 18% (ranging from 2% to 50%), conservative thresholds were required—at least 70% for the time-based heuristic and 7% for the data-driven heuristic—to reach 95% recall. Importantly, they showed that combining heuristics could achieve the same goal at lower cost; the most efficient combination was 20% of abstracts screened and a 5% irrelevant-abstract threshold. They also observed that all combinations using a 10% data-driven cutoff or the 10% cutoff alone consistently achieved the 95% threshold, whereas a 5% cutoff required higher time-based thresholds to perform reliably.

Other studies reinforce these patterns. Research on medical- and health-economics data sets has shown that thresholds of 100 to 200 consecutive irrelevant abstracts generally improved performance and reduced variability across replications, although at the cost of additional screening (Callaghan & Müller-Hansen, 2020; Oude Wolcherink et al., 2023; Scherhag & Burgard, 2023). For example, raising the cutoff from 50 to 100 markedly increased recall, and further increases did not improve median sensitivity. Even though such thresholds missed roughly 20% of potentially relevant abstracts, they still identified nearly 100% of studies deemed relevant after full-text screening. In another study, Campos et al. (2024) evaluated an adaptive data-driven heuristic in which cutoffs were defined as percentages of the data set. Setting the cutoff to 7% consistently achieved 95% recall across multiple educational and psychology data sets when using algorithms such as LR+SBERT. In contrast, other algorithms required a 10% cutoff to achieve a similar identification rate.

Taken together, this evidence highlights both the potential and variability of stopping rules across domains. Performance is strongly shaped by prevalence, data-set size, and algorithm stability. Conservative thresholds tend to improve recall but may require additional screening, reducing efficiency. Conversely, more liberal thresholds may save time but risk missing relevant studies that appear late. For meta-analytical research, comprehensiveness should be prioritized over efficiency, making conservative thresholds preferable. For exploratory work, efficiency may be prioritized instead. In both cases, we recommend aligning cutoffs with prevalence estimates obtained from an initial random subsample.

Recommendation

Following König, Zitzmann, Fütterer, et al. (2024) and the SAFE method (Boetje & van de Schoot, 2024), we endorse combining the key-study, data-driven, and time-based heuristics as primary rules and the data-driven heuristic also serving as a breakout rule. To further encourage conservative practice, we recommend adjusting the prevalence categories suggested by König, Zitzmann, Fütterer, et al. from < 2.5%, 2.5% to 7.5%, and > 7.5% to < 5%, < 10%, and > 10%. As prevalence increases, both data-driven and time-based cutoffs should be reduced. Building on these principles, we translate them into concrete guidelines for different prevalence levels, specifying how each stopping rule can be applied in practice and how breakout strategies can be incorporated. Based on findings from the prescreening phase, a prevalence estimate can be calculated. Corresponding to this estimate, we suggest the following guidelines:

Evidence of the effectiveness of the recommendations

Although in this tutorial we focus on implementing ASReview, we also conducted a simulation study using 36 data sets from Campos et al. (2024) and König, Zitzmann and Hecht (2024) to verify the outcomes of our recommendations. In each simulation, we first sampled studies until at least 100 abstracts had been screened and at least one relevant study had been identified. We then unlabeled the remaining relevant studies, marked them as key studies, and used one relevant abstract together with all irrelevant abstracts from prescreening as training data. This process was repeated 100 times for each data set. Screening was simulated with ASReview’s built-in simulation mode via the Python API, which mirrors reviewer decisions by accessing the column containing the original labels. For example, when the algorithm recommended screening an abstract marked as irrelevant, the system classified it as such before updating its predictions (see above). We then used the screening order of the labeled abstracts to apply the stopping rules.

The time-based heuristic was established as a reference point in the screening process. Before this point was reached, screening generally continued unless the breakout condition or the key-study rule was met. The data-driven heuristic was applied once the time threshold had been reached. Specifically, we counted the number of consecutive irrelevant abstracts since the last relevant abstract before the time-based heuristic was applied. When the proportion of this sequence exceeded the cutoff value, the key-study rule was applied. When all key studies had already been identified, screening was stopped; when not, screening continued until all key studies were found. Each rule (Rules A–C) was applied to every data set. We then calculated sensitivity and screening cost and documented whether the breakout and key-study rules had been triggered.

Next, we grouped the data sets by prevalence, estimated from random prescreening, to examine whether the rules performed differently across prevalence levels. We documented the average sensitivity and screening cost for each rule in each category (Table 2). In addition, we recorded the percentage of simulation runs in which each rule achieved a sensitivity above 95% and 100%. We provide the full documentation of this simulation, including code and data, on our OSF project page (https://osf.io/xrb9z/overview).

OPEN IN VIEWER

Table 2.

Simulated Performance of the Stopping Rules Across Prevalence Categories

Category	Estimate	N_s	Rule	M_c	sd_c	m_s	sd_s	≥95	=100	n_b	n_key
≤ 5%	2.97%	1,174	A	46.01	11.42	99.04	1.42	100.00	46.85	79	26
			B	38.91	12.24	98.50	1.55	92.42	25.38	5	35
			C	31.15	12.64	96.00	4.74	75.89	3.49	0	103
5% to 10%	7.76%	910	A	59.35	16.07	99.53	0.73	100.00	43.85	46	30
			B	54.98	16.55	99.33	0.82	99.56	29.12	1	41
			C	47.10	15.82	97.74	3.24	90.88	15.71	0	169
> 10%	19.54%	1,100	A	79.02	16.81	99.74	0.47	100.00	58.55	0	57
			B	75.06	16.67	99.55	0.61	99.91	34.55	0	106
			C	69.22	17.21	98.92	1.65	97.09	28.36	0	267

Note: Estimate = average prevalence estimate based on the training sample; N_s = number of simulation runs within that category; rule = stopping procedure (Rules A, B, or C, as described in text); c = screening cost; s = sensitivity; ≥ 95 = percentage of simulation runs in which the rule achieved a sensitivity of at least 95%; ≥ 100 = percentage of simulation runs in which the rule achieved a sensitivity of 100%; n_b = number of times the breakout rule was triggered; n_key = number of times not all key studies were identified after the rules were met.

As shown in Table 2, all rules achieved a mean identification rate above 95% across prevalence levels. However, not all rules consistently reached this threshold in at least 95% of simulation runs. The most conservative approach, Rule A, achieved this threshold across all prevalence categories, Rule B did so only in the 5% to 10% and > 10% prevalence categories, and Rule C did so only in the latter. When screening costs are also considered, the most efficient rule, which achieved sensitivity above 95% in at least 95% of cases, was consistently the one tailored to the respective prevalence category.

Another important observation is that none of the rules succeeded in identifying 100% of relevant abstracts in 95% of the cases even when applying the most conservative option. This finding is not unexpected because the rules are derived from the literature, in which a 95% sensitivity threshold is typically the goal. Nonetheless, researchers seeking to identify all relevant abstracts should consider raising our recommended cutoff values.

Finally, the results highlight the contribution of the key-study stopping rule, which was triggered up to 292 times depending on the method and prevalence category. Likewise, the breakout rule was occasionally triggered even under the most conservative condition, underscoring its usefulness in reducing screening effort once most relevant abstracts had already been identified.

All in all, our rules performed as expected. Nonetheless, we wish to emphasize that although we view these recommendations as conservative, they do not guarantee that at least 95% of the relevant literature will be identified, and our sample was too small to be generalizable. For instance, some findings suggest that the performance of these rules depends not only on prevalence but also on data-set size (Campos et al., 2024; König, Zitzmann, Fütterer, et al., 2024). Researchers may therefore adjust the cutoff values upward to increase the likelihood of identifying all relevant articles or downward to improve screening efficiency when minimizing time is a priority. Moreover, when applying stopping rules other than those included here, alternative algorithms may be more appropriate (see König, Zitzmann, & Hecht, 2024).

Implementation

Considering the prescreening results displayed on the “Dashboard,” screening 100 studies yielded four relevant abstracts and 96 irrelevant abstracts, corresponding to an estimated prevalence of 4%. Using the cutoff values for stopping-procedure selection, this outcome would align with Rule A. For guidance on selecting appropriate procedures across different prevalence levels of relevant abstracts, see the Recommendations section. With this strategy, screening would stop once 35% of the abstracts have been reviewed, 7% have been consecutively labeled as irrelevant, and all key studies have been identified. Alternatively, the breakout strategy could be applied, which, in this case, would require 15% consecutive irrelevant abstracts. Note that ASReview does not display relative screening progress in the “Analytics” tab; instead, it reports the absolute number of screened abstracts and the number of consecutively irrelevant abstracts. Consequently, users must calculate the required thresholds manually. For example, in our data set containing 2,035 abstracts with an estimated prevalence of 4%, the time-based heuristic with a 35% cutoff would require screening at least 2,035 × 0.35 = 712 abstracts. The threshold for consecutively irrelevant abstracts—after this point—would be 2,035 × 0.07 = 142 abstracts in a row (data-driven heuristic). For the breakout strategy, the corresponding threshold would be 2,035 × 0.15 = 305 consecutive irrelevant abstracts.

Project setup

Model configuration is critical during the AI-aided screening phase. Because our recommendations are based on the LR+SBERT configuration, we encourage users to select the SBERT feature extractor (“all-mpnet-base-v2”) in combination with the LR classifier (for results on other algorithms in psychology- and education-related literature, see Campos et al., 2024; König, Zitzmann, & Hecht, 2024). We further recommend using the “Maximum” (certainty-based) query strategy and activating the “Balancer” (Fig. 14). To the best of our knowledge, most evaluation studies on learning algorithms and stopping rules in ASReview have employed this setup (e.g., Campos et al., 2024; König, Zitzmann, Fütterer, et al., 2024; König, Zitzmann, & Hecht, 2024; Teijema et al., 2023).

OPEN IN VIEWER

Fig. 14.

Model setup for artificial-intelligence-aided semiautomated abstract screening.

Once the model is defined, ASReview proceeds with initialization, which involves training the model and generating the first ranking of abstracts by predicted relevance. Because the SBERT feature extractor is computationally intensive, this process may take up to 20 min, depending on data-set size and hardware specifications. During this time, users can either wait or continue random screening by clicking on “I can’t wait.” When initialization is complete, the model configuration is displayed above the abstract provided that “Show model information” is enabled in the “Settings” menu (Fig. 15).

OPEN IN VIEWER

Fig. 15.

Settings.

Before starting active learning, users should enable ASReview’s built-in stopping rule functionality via the “Dashboard.” They can specify either the number or the percentage of consecutive irrelevant abstracts to activate the data-driven heuristic. Initially, we recommend using the value defined by the breakout strategy, which in our example corresponds to 305 abstracts (see above). Once this is set, screening can begin. During this phase, careful and accurate classification is essential because misclassifications can reduce the model’s predictive accuracy.

Stopping

To determine whether the time-based heuristic has been met, users should check the number of labeled abstracts displayed above each abstract during screening when “Show model information” is enabled in “Settings.” In our example, this rule would be satisfied once 0.35 × 2,035 = 712 abstracts had been screened. Once this threshold is reached, the cutoff for the data-driven heuristic should be adjusted according to the selected strategy. In our example, this corresponded to a percentage of 7%, which meant stopping after encountering 142 consecutive irrelevant abstracts.

Once these heuristic thresholds have been applied, ASReview will automatically stop (see Fig. 16). ASReview also asks whether users wish to take additional measures to further improve screening quality, such as increasing the cutoff value or selecting a different algorithm. Additional measures to increase sensitivity are described below under Additional Screening Measures. We do not recommend marking the project as finished until the results are verified. However, users can always change their project’s status back to “unfinished” if they decide to continue screening.

OPEN IN VIEWER

Fig. 16.

Automatic stopping of ASReview and additional screening measures.

To evaluate screening success, users can, for example, examine the “Dashboard” visualizations. The “Recall” chart (Fig. 17a) plots the cumulative number of relevant abstracts (y-axis) against the total number of screened abstracts (x-axis), providing a clear representation of progress over time. Researchers can focus on identifying the “knee” of the curve—the inflection point at which the model’s ability to find relevant abstracts begins to decline, leading to longer sequences of irrelevant abstracts between relevant ones, which we show in the AI-aided screening section. This reflection point suggests that a large portion of the relevant abstracts have been identified. In our example, this inflection point is anywhere around 40 relevant abstracts, which roughly corresponds to 80% sensitivity. By hovering over the “Recall” chart with the mouse, users can also see how many more relevant abstracts have been identified compared with random screening. The “Wave” chart (Fig. 17b) can complement the “Recall” analysis by showing whether sequences of consecutive irrelevant abstracts are increasing or decreasing and whether the data-driven heuristic has been met. In our example, the intervals increase, clearly illustrating how the data-driven heuristic operates.

OPEN IN VIEWER

Fig. 17.

Dashboard: (a) “Recall” and (b) “Wave” visualizations.

However, before concluding the screening, it is essential to confirm that all key studies have been identified by searching for their titles or notes. This can be done most efficiently by exporting the relevant labeled abstracts and then searching for their titles or “data_ids.” When any key studies remain unidentified, screening should be extended until this condition is met. In our example, all key studies had already been identified. After a project is marked as finished in ASReview, the software automatically estimates the number of hours saved, which in our example corresponded to 25 working hours (see Fig. 18).

OPEN IN VIEWER

Fig. 18.

Estimated time saved.

Results

With these conditions satisfied, the AI-aided screening process achieved a sensitivity of 98% and a screening cost of 39%. All but one relevant abstract were identified without the need to screen 61% of the irrelevant abstracts.

Finally, users can document their results as suggested in Table 1. On our OSF project page (https://osf.io/xrb9z/overview), the data set after screening is named “Tang_et_al_2022_unlabeled_AI_completed.csv.”

Screening

To screen the subset of prelabeled abstracts, the model does not matter because all abstracts of the subset require screening. The rest of the project-setup process follows the same procedure described in the prescreening for unlabeled data (see above). During screening, users can monitor whether their screening decisions align with the preassigned labels, enabling the calculation of true positives, false positives, true negatives, and false negatives. In addition, they can define the cutoff of the time-based heuristic for the breakout strategy and combined strategy, respectively. However, manually recording each decision can be time-consuming, and ASReview does not provide built-in functionality to directly extract this information. Thus, we recommend exporting the data set after all abstracts have been screened and using R to prepare the data for AI-aided screening and to estimate the sensitivity, specificity, and prevalence of relevant abstracts.

To retrieve the necessary information—frequencies of true positives, true negatives, false positives, and false negatives—the table() function in R can be used:

OPEN IN VIEWER

Output:

OPEN IN VIEWER

In this function, the rows correspond to the first variable entered, and the columns correspond to the second. As shown in the output, the automatic screening correctly labeled 40 relevant abstracts and 100 irrelevant abstracts from the subset. However, it also misclassified 198 irrelevant abstracts as relevant. This screening would have resulted in an observed interrater agreement (Equation 4) of 41% and a specificity of 100%. Although these estimates might not be entirely accurate because they are based on only a small sample, they inform users whether they need to correct the prevalence estimate. When specificity is below 100%, users can adjust the prevalence estimate by accounting for relevant abstracts that the ML algorithm labeled as irrelevant. One way to do this is to estimate how often abstracts classified by the ML algorithm as irrelevant are relevant in the checked subset and then apply this rate to the remaining abstracts that have not been evaluated by the human reviewer (see Equation 5). However, because stopping rules for lower prevalences are more conservative, ignoring potentially relevant abstracts that might have been missed in prevalence estimation also yields conservative stopping rules. In our example, we estimated a specificity of 100%. Thus, we would estimate the prevalence from the 40 relevant abstracts, yielding a prevalence of roughly 2%. This information should be documented (see Table 3) for use when selecting the stopping strategy.

Before proceeding, users should merge the prescreening data with the data produced by the automated-screening method that was not used for prescreening. ASReview automatically adds or modifies some columns in the data set, such as the new “asreview_label” column storing the user’s decision, columns for all tags, “asreview_time,” and “asreview_note.”

Because both data sets use the previously created “data_id” column to store the same identifier for each abstract, this column can now be used to remove all abstracts screened during prescreening from the unscreened data set. This process ensures that no duplicates are introduced when merging the prescreened and unscreened data sets to create the final data set for AI-aided screening. To remove the prescreened data entries from the unscreened data set, we used indexing in R. To merge the data sets, we recommend using the bind_rows() function from the dplyr package (Wickham et al., 2020), which allows for row-wise merging of data frames even when they contain different columns. Once these steps are completed, the data set for AI-aided screening can be merged and used for AI-aided screening:

OPEN IN VIEWER

Evidence

Although empirical evaluations of combined semiautomated- and automated-screening approaches have shown high rates of correctly classified abstracts, generalizable evidence for their combination remains limited. Consequently, reliable recommendations for stopping rules are also lacking. Nonetheless, our proposed procedure allows existing evidence from semiautomated screening to serve as a guideline. For instance, recent findings suggest that screening approximately 30% of abstracts often yields 80% of the relevant studies (Campos et al., 2024). Thus, applying a time-based heuristic may no longer be necessary when the automated tool has already identified more than 80% of relevant abstracts. In such cases, the cutoff values for the time-based heuristic can be aligned with the estimated percentage of already identified relevant abstracts. This proportion can be calculated using the sensitivity and specificity values obtained during prescreening. Likewise, these values can inform prevalence estimates, which can then be used to align the cutoff values for the data-driven heuristic, as described for semiautomated screening. However, given the lack of robust evidence, we propose adopting more conservative estimates regardless of prevalence.

Recommendation

Based on these considerations, we suggest the following strategies while emphasizing that they lack direct empirical support beyond what is available for semiautomated screening:

Implementation

In our example, we observed a specificity of 100% and a prevalence of 2%. Thus, we would use Strategy A, applying only the data-driven heuristic with a cutoff of 7%. However, because we screened 338 abstracts to evaluate the automated screening, we had already screened about 15% of the abstracts. Thus, a 15% cutoff for the time-based heuristic would have been fulfilled regardless. However, when accuracy was below 50%, we would stop at 15% with an additional 5% and then apply the data-driven heuristic, which corresponds to Strategy B.

Because ASReview does not support selecting a cutoff of 7%, we would need to calculate the corresponding number of abstracts, which is 142.

Project setup

After completing the prescreening and selecting a stopping-rule procedure, the AI-aided screening process can be initialized. However, in contrast to prescreening, the AI-aided screening process is the same for both unlabeled and prelabeled references, and the model is essential. For the example presented here, we use the data set “Tang_et_al_2022_prelabeled_AI.csv,” available on our OSF project page (https://osf.io/xrb9z/overview).

Screening

For the AI-aided screening of prelabeled references, we recommend the same model configuration as for semiautomated screening (see Fig. 14). The model may take longer to initialize because of the larger volume of training data. However, when users want to start screening while the model is trained, they can click on “I can’t wait” to start screening randomly. Before screening, we recommend creating tags for the inclusion and exclusion criteria and defining the cutoff for the data-driven heuristic in the “Dashboard” section before screening the first abstract.

In this example, we met the stopping threshold after screening 825 abstracts (see Fig. A2 in Appendix A), which corresponds to 41% of the abstracts. As a result, all relevant articles were identified. Thus, we would have saved 59% of the screening time without any loss. A cutoff value of 5% for the time-based heuristic would have resulted in missing one relevant article. The semiautomated screening took 5% longer to achieve the same identification rate of relevant articles. However, we want to emphasize that our example is not generalizable, and we simulated a worse-performing automated-screening method than state-of-the-art research suggests (e.g., Li et al., 2024; Vembye et al., 2024). However, before concluding the screening, users should export the data (e.g., “Tang_et_al_2022_prelabeled_AI_completed.csv”) and document their results. Moreover, they can take additional steps to improve the quality of their screening, as summarized below.