Rapid Whole-Genome Identification of High Quality CRISPR Guide RNAs with the Crackling Method

Abstract

The design of CRISPR-Cas9 guide RNAs is not trivial and is a computationally demanding task. Design tools need to identify target sequences that will maximize the likelihood of obtaining the desired cut, while minimizing off-target risk. There is a need for a tool that can meet both objectives while remaining practical to use on large genomes. In this study, we present Crackling, a new method that is more suitable for meeting these objectives. We test its performance on 12 genomes and on data from validation studies. Crackling maximizes guide efficiency by combining multiple scoring approaches. On experimental data, the guides it selects are better than those selected by others. It also incorporates Inverted Signature Slice Lists (ISSL) for faster off-target scoring. ISSL provides a gain of an order of magnitude in speed compared with other popular tools, such as Cas-OFFinder, Crisflash, and FlashFry, while preserving the same level of accuracy. Overall, this makes Crackling a faster and better method to design guide RNAs at scale. Crackling is available at https://github.com/bmds-lab/Crackling under the Berkeley Software Distribution (BSD) 3-Clause license.

Introduction

The design of guide RNA is a crucial step for any CRISPR experiment. However, guide design is not trivial, as the efficiency and specificity of guides are crucial properties. Efficiency broadly refers to the guide correctly binding to the targeted region and to the endonuclease. This is influenced by a number of factors that depend both on the guide itself (e.g., secondary structure) and on the targeted region (e.g., chromatin accessibility). Specificity refers to the ability of the guide RNA to have minimal off-target modifications. This means not only that the targeted sequence must be unique but also that closely related sequences (typically less than four mismatches) elsewhere in the genome must be carefully considered. There exist many computational pipelines to assess the properties of CRISPR guides. However, they are sometimes inaccurate and often slow.¹ In this study, we present Crackling, a pipeline designed to rapidly analyze large-sized genomes, without sacrificing accuracy.

We recently benchmarked existing guide design tools.¹ Our two main findings were as follows:

When considering efficiency, for any given genomic sequence, there is a limited overlap between the set of guides that each tool produces.

Several tools have inadequate filtering on specificity, and when that is considered carefully, it tends to be a computationally expensive task.

The limited overlap between the tools can be exploited for guide selection: when a guide is recommended by multiple tools, there is a higher chance that it will actually be efficient. We explored consensus approaches in detail and found that they provide better guides.² However, it is not practical to run several different tools. A recent review suggested the use of a tool that would offer more than one scoring algorithm, and while these tools exist, none of them have explored the possibility of combining scores to achieve an overall gain in precision.³ Our Crackling method directly integrates multiple scoring approaches and combines their result for improved precision when predicting guide efficiency.

While guide efficiency is critical, the success of a CRISPR experiment is also determined by the specificity of a guide RNA. Several methods have been implemented to tackle evaluating specificity. One of the earliest methods, published in 2014, is Cas-OFFinder,⁴ which provides a count of the number of mismatches between each candidate guide and off-target site if the count is less than a predetermined threshold (e.g., four). The results can then be used to get a sense of the off-target risk for each candidate guide.

While identifying off-target sites that are within a fixed number of mismatches is useful, it is not necessarily enough. It has been experimentally shown that the position of the mismatches also matters.⁵ A score for this exists; sometimes referred to as the Zhang score, from the corresponding author of that article. To calculate the score, a candidate guide is compared against all potential CRISPR sites in the genome of interest. When a site is at most four mismatches away, a local score is calculated. The sum of local scores provides a measure for the off-target risk of some candidate guide. Methods such as Cas-OFFinder,⁴ CRISPRseek⁶ and CRISPOR,^7,8 and mm10db⁹ each use a custom implementation of this score.

In mm10db, the multithreaded function implemented for calculating the score, called findMismatches, terminates early as soon as the running sum of local scores guarantees that the global score will drop below some fixed threshold.

The initial implementation of findMismatches relies on string comparisons. For this article, we have reimplemented it using a binary encoding of the sequences, so that mismatches can be identified more efficiently.

An alternative to the Zhang score is the Cutting Frequency Determination (CFD) score, which accounts for position- and identity-specific mismatches between the guide and off-target sequences.¹⁰ Doench et al. assessed the ability of the score to predict the off-target activity for an independent set of 89 guide sequences. Compared with other popular scores, including the Zhang score, CFD continually performed better when measured using the area under the receiver operating characteristic curve (AUROC), as shown by Doench et al.¹⁰

The Zhang and CFD scores have been used in an array of ensemble learning algorithms, including using simple consensus approaches, and the machine learning algorithms AdaBoost, Random Forest, Support Vector Machine, and Decision Tree.¹¹ The CFD score was found to be a driver in predictive performance. The simple pairwise combination of the Zhang and CFD scores performed best compared with other techniques and any score taken individually.

Similarly to the Zhang score, instead of calculating an exact global score, it is possible to keep track of the running sum of local CFD scores and terminate early as soon as a guide is guaranteed to end up being rejected.

The recently published Crisflash relies on the Zhang score to evaluate guides, but without the early termination.¹² The authors report that Crisflash can outperform Cas-OFFinder by an order of magnitude on the human genome when more than a couple of 100 guides are supplied. We benchmark the performance of these tools in this study.

Crisflash also provides support for variants, which are not used here. It is also noted that Crisflash relies on a volume of memory that would not be available on a modern workstation (90 Gb), but rather a server cluster.

FlashFry presents a discovery approach that utilizes a table of bit-encoded off-target prefixes.¹³ Similar to other methods, a candidate guide is only scored on a given off-target if the number of mismatches is fewer than the specified threshold. The authors claim that FlashFry is able to run two to three orders of magnitude faster than Cas-OFFinder. FlashFry does not report the Zhang score but uses the CFD score.

Other tools such as CRISPRseek, CasOT, and CRISPRitz exist.^6,14,15 However, we do not consider them here as published results show that they perform poorly compared with others, which we have included.¹²

Materials and Methods

In this article, we present Crackling, a method aimed at addressing the two challenges of efficiency and specificity when designing CRISPR-Cas9 guide RNAs. To increase the efficiency of the set of guides we produce, we have combined three scoring strategies, and to speed up the off-target scoring, we have introduced a solution for constant-time lookup of sequence neighborhoods.

The tool is implemented in Python (version 3) for most steps, with the high-performance off-target scoring in C++. It can run on any platform with minimal dependencies: Bowtie2 for guide realignment onto the input genome and RNAfold for secondary structure prediction.^16,17 Where possible, components of the tool are parallelized for improved performance. Preprocessing is minimal, only requiring the input genome to be indexed by Bowtie2 and off-target sites to be indexed for Inverted Signature Slice Lists (ISSL).¹⁸ The ISSL index is simple to build and can be reused at any time for the genome it is constructed for.

Crackling is configured via an initialization file, which we have provided thorough documentation for. Parameters include:

Providing a name to prefix output file names.

An optimization level that dictates how restrictive Crackling is to be when calculating results. A low optimization instructs Crackling to calculate results for all tests, whereas a high optimization progressively filters out tests, on an individual guide basis. The optimization level will influence the run-time.

The consensus approach: selecting which rules, derived from other tools, should be included.

Paths to input and output files.

Program binaries, threading, and batch sizes to avoid saturating memory.

Once configured, the pipeline is called via any command line terminal using the Python v3.7 interpreter. Postprocessing, which provides the ability to annotate guides with any genomic features and to count the number of targeted gene transcripts, is an optional final step, for which we also provide code.

Multiapproach efficiency evaluation

We previously showed that when different tools actively filter or score guides based on their predicted efficiency, they only rarely agree with each other.¹ We also showed that this can be leveraged for consensus-based approaches that combine the output of multiple tools.² However, having to install and run multiple tools may not always be practical.

Based on the results of this consensus study and the need for computational performance, Crackling directly incorporates three scoring approaches and only recommends candidate guides that are accepted by at least two of the approaches:

As in CHOPCHOP, we also look for the presence of a guanine at position 20.¹⁹

We select for positively scored guides based on the model from sgRNAScorer 2.0.²⁰

We use the filtering steps from mm10db (GC content, secondary structure, etc.), and only accept candidate guides that pass all steps.⁹

Other consensus-based approaches can be easily incorporated by modifying the default configuration. Table 5 describes the precision (the proportion of selected guides that are truly efficient), recall (the proportion of efficient guides that are selected), and F1 score (the harmonic mean of precision and recall) for each configuration.

For each configuration, we explored the number of guides that Crackling reported per gene (for the Oryza sativa genome), before considering off-target effects. For the strictest configuration, where all three scoring approaches must agree, Crackling designed guides for 93.9% of genes. This configuration also reported at least three guides for 90% of genes, which allows for a triple-target strategy delivering almost perfect knockout efficiency.⁹ The default configuration designed guides for 96.4% of the genes. Figure 1 describes the number of guides per gene for all configurations.

FIG. 1.

Number of guides per gene, for each consensus configuration, shown as proportions across the Oryza sativa genome. Range boundaries are inclusive.

The performance of Crackling may change depending on the configuration chosen. See Supplementary Figure S1 for the run-time of Crackling when ran in each of the configurations.

The sgRNAScorer 2.0 model is included in the Crackling code repository, in addition to the raw data and a script to retrain the model for version compatibility reasons.

Assessing off-target risk using the Zhang and CFD scores

The Zhang score assesses the off-target risk by considering position-specific mismatches between the candidate guide and CRISPR sites that differ by no more than four mismatches. For each CRISPR site meeting this criterion, a local score is calculated using Equation (1).

where W is an array of position-specific weights, d is the pairwise distance between mismatches, n is the number of mismatches between target and sequence, and M is the list of mismatch positions.

The sum of local scores provides a measure for the off-target risk of some candidate guide. If we denote Q as the set of all off-target sites at most four mismatches away from the candidate sgRNA, then Equation (2) combines all the $S (M)$ local scores into a global score for the guide. $S_{g l o b a l} = \frac{100}{100 + \sum_{M \in Q} S (M)}$ (2)

The objective is to reject all guides for which the global score is below a fixed threshold $τ = 0.75$ . Based on Equation (2), $S_{g l o b a l} \leq τ$ is equivalent to Equation (3).

The CFD score is stricter than the Zhang score, as in addition to the position of the mismatch, it also considers the type of mismatch. A global CFD score can be calculated in the same way as the Zhang score—first by calculating local scores for CRISPR sites that are no more than four mismatches away from the candidate guide, as in Equation (4).

where P is a matrix of percent-active values for positions n to the length of the guide sequence l and mismatch type m. Greater values of $P_{n m}$ indicate higher risk of activity. If there is no mismatch in position n, then a percent-activity of 1.0 is used.

By again using Equation (2) to combine the local scores, a global CFD score can be calculated for the guide. When the score is below a fixed threshold $τ = 0.75$ , as in Equation (3), the guide is rejected.

Multiapproach specificity evaluation

A plethora of tools have been proposed for predicting off-target activities of CRISPR-Cas9. Each has been critically evaluated using data that include measured editing outcomes. Zhang et al. assessed an array of techniques for combining the tools, including using simple consensus approaches, and machine learning algorithms.¹¹ The simple pairwise combination of the Zhang and CFD scores was found to be better than other techniques and any score taken individually. Using the already curated data of multiple DNA double-stranded break detection methods (such as CIRCLE-seq, GUIDE-seq, and others), we assessed five approaches for determining whether a guide has too high of an off-target risk: (1) Zhang score alone, (2) CFD score alone, (3) Zhang and CFD scores, (4) Zhang or CFD score, and (5) the average of the Zhang and CFD scores.²¹

The AUROC for the Zhang and CFD scores were 0.86 and 0.89, respectively. When calculating the average of these two scores, the resulting measure outperformed both scores individually, with an AUROC of 0.92. We also assessed the performance of the machine learning algorithms, AdaBoost and Random Forest, when the Zhang and CFD scores were provided as model features. The machine learning models did not outperform the simple approach of calculating the average of the two scores. See Supplementary Figure S2 for the Receiver Operating Characteristic (ROC) curve, including Area Under the Curve (AUC) measures. As the average is faster to calculate and easier to interpret, this is the method implemented as the default option in Crackling. The four other approaches are also available as options in the configuration file.

Off-target scoring using ISSL

ISSL can be used to rapidly perform approximate nearest neighborhood searches in collections of locality-sensitive signatures.¹⁸ By using fixed-length signatures as search keys, items in a neighborhood can be found in constant time. This approach was initially proposed as a high-performance method for searching web-scale collections of data. Given that genomic data are at a comparable scale, it provided a feasible method for evaluating CRISPR guide specificity.

The approach utilizes a bit-encoded index to reduce memory and storage requirements and to use CPU instructions most effectively. The four letters of the genomic alphabet (A, T, C, G) are 2-bit encoded; hence, a 20-bp genomic sequence requires 40-bits but is stored in a 64-bit word. In the context of Crackling, this 20-bp sequence represents an off-target site. For each bit-encoded signature, the critical 40-bit segment is portioned into $n + 1$ slices, where n is the maximum number of mismatches (here, $n = 4$ to match other methods). This slice, in combination with its position n, is used as key within the key-value structured index. For each key, a list of off-target identifiers exists. Each identifier is also prefixed with a count of how many times the off-target site is seen across the genome. This allows the scoring method to preemptively consider all duplicates, rather than each individually, as when this would be needed with a sequential-styled approach.

Separately, when considering guide specificity, the bit-encoding process is repeated for each candidate guide. Given that n mismatches are allowed and there exist $n + 1$ slices, there will always be one slice shared between a candidate guide and a potential off-target site that is at most n mismatches away from the candidate guide. The time complexity of this discovery step is constant given the structure of the index, and the number of potential off-target sites is less than that in other approaches (such as using a sequential search where all off-target sites are considered). Similar to the findMismatches approach, the candidate guide is scored against off-target sites using the Zhang score until a specified threshold is reached [Eq. (3)]. The approach is visualized in Figure 2.

FIG. 2.

(A) A list of off-target sites, each depicted by a colored shape. The adjacent number indicates how many occurrences of the 20 base pair sequence exist in the genome. Vertical lines indicate how the sequences are sliced, which in (B), form the index. (B) The ISSL off-targets index is a dual-key tabular data structure, constructed using locality-sensitive hashes. The key of the first dimension is the slice number. The key of the second dimension is the bit-encoded genomic sequence of the slice. The content is a reference to the bit-encoded genomic sequence of the off-target site (depicted by the colored shapes). (C) Scoring a candidate guide against off-target sites. An x denotes a mismatch between the candidate guide and the off-target site. An arrow indicates that the given slice is common for both sequences. Each candidate guide is portioned into $n + 1$ slices ( $n = 4$ in this figure). Each matching slice is used a locality-sensitive lookup key into the index. For each off-target site, the positions of mismatches are used in calculating the specificity score (off-targets #1, #2, and #3). If no slice exists in both sequences, then the sequence is not deemed as an off-target site (off-target #4). If more than n mismatches exist, then the sequence is not deemed as an off-target site (off-target #5). An off-target may be identified for multiple slices; however, ISSL strictly considers all distinct off-targets per candidate guide (e.g., off-target #3 has been identified via three slices but only considered for scoring once). (D) The global score for the sample candidate guide is calculated using the expression given here. See Equation (2). The positions of mismatches and the number of occurrences for the off-target sites are considered. (E) Algorithm for ISSL. ISSL, Inverted Signature Slice Lists.

Off-target scoring performance

We timed each of the off-target tools on genomes of increasing size, which were obtained from the National Center for Biotechnology Information (NCBI) (Table 1). For each genome, candidate guides were extracted from the forward and reverse strands using a regular expression: [ACG][ATCG]²⁰GG (and the complement for the reverse strand). Two editions of these target sites were created: one including the PAM sequence and one without, as this was required by the tools. This process was repeated for off-target sites, however, using the following expression: [ACG][ATCG]²⁰[AG]G (again, with a complementing expression for the reverse strand).

Table 1.

The genomes used in this study

Genome	Accession number	Size, Mb	Off-target sites count
Oryza sativa	GCF_001433935.1	374.4	61.7 × 10⁶
Panicum hallii	GCF_002211085.1	507.4	91.9 × 10⁶
Xiphophorus couchianus	GCF_001444195.1	685.5	100.6 × 10⁶
Parus major	GCF_001522545.3	998.3	173.0 × 10⁶
Crotalus viridis	GCA_003400415.2	1297.0	193.8 × 10⁶
Danio rerio	GCF_000002035.6	1679.4	214.8 × 10⁶
Oncorhynchus mykiss	GCF_002163495.1	1950.0	295.9 × 10⁶
Mus musculus	GCF_000001635.24	2730.7	499.7 × 10⁶
Monodelphis domestica	GCF_000002295.2	3502.4	570.9 × 10⁶
Triticum urartu	GCA_003073215.1	4661.6	829.4 × 10⁶
Rhinatrema bivittatum^*	GCF_901001135.1	5178.6	984.7 × 10⁶
Triticum turgidum^*	GCA_900231445.1	9964.3	1763.2 × 10⁶

Asterisk indicates that these genomes were only used on the high-performance machine due to memory limitations.

Mb, megabases.

Using the candidate guides extracted for each genome, we further extracted 5 distinct sets of 10,000 candidate guides. For O. sativa, we repeated this but for sets of size 5000 to 25,000, incrementing by 5000. For each of these, we tailored copies for Crisflash and Cas-OFFinder, which required custom formats. ISSL required an index of the off-target sites. Crisflash required a single genome file, as opposed to a file for each chromosome. For this, we separated each chromosome with sufficient Ns to prevent false sites at the junction between chromosomes. FlashFry required a custom index for each genome but could not produce one for any genome larger than that of O. sativa.

For findMismatches, findMismatchesBit, and Crisflash, we modified the source code to time the guide specificity evaluation method in each tool. Additionally, for Crisflash, we made modifications to time the construction of its tree data structure. For Cas-OFFinder, we timed it with the time application with microsecond precision. For ISSL, we measured using the time library within Python. All preliminary tests were performed on a high-performance Linux workstation, with two 18-core Intel Xeon E5-2699 v3 (2.3 GHz) CPUs, 512 GB RAM, 4.2 GB allocated swap space, and Hewlett-Packard Enterprise MB4000GEFNA 4TB HDD. Further testing was performed on a different Linux workstation with one 8-core Intel Core i7-5960X (3.0 GHz) CPU, 32 GB RAM, 32 GB allocated swap space, and Samsung PM87 SSD.

For all tests, we used a strict 30-h wall time, as tools slower than this would not scale to the analysis of entire genomes.

Benchmarking guide design pipelines

We previously benchmarked leading CRISPR guide design methods using custom data sets derived from the mouse genome.¹ These data sets were of size 500,000 bp (500k), 1,000,000 bp (1m), 5,000,000 bp (5m), and the full 61m bp chromosome 19 (full). Some tools require an annotation file, and only consider guides in coding regions. To account for this, we calculated the effective base pairs per second (EFPS), which uses the length of the regions used for candidate extraction. Crackling considers the entire input genome; thus, the genome length was used when calculating the EFPS. We constructed the ISSL index for these data sets and ran identical experiments on Crackling from the original article.

Results

All preliminary tests were executed on a high-performance machine and followed up on a machine where the available memory was more limited. Cas-OFFinder could not be tested on the high-performance machine due to compatibility issues. For FlashFry, it was not possible to generate the required indexes for genomes larger than Oncorhynchus mykiss, and the tool was therefore not tested past this point. Generating the index for the larger genomes was attempted on both machines, with no success.

The ISSL approach powering Crackling is the fastest off-target scoring method

Three tools (findMismatches, findMismatchesBit, and ISSL) were capable of completing tests on all 12 genomes on the high-performance machine. They also completed tests on the 10 genomes considered on the workstation (with the 2 largest being excluded due to the memory limitation of this machine). On the high-performance machine, Crisflash failed on genomes larger than O. mykiss due to segmentation faults. On the workstation, it saturated physical memory, causing swapping to occur.

For both machines, the performance for findMismatchesBit was greater than findMismatches by a factor of up to 6 for all tests, confirming that bit-encoding alone provides a performance benefit. The results for the Triticum urartu data set on the workstation are an outlier due to memory being swapped to disk.

ISSL was able to evaluate 10,000 candidate guides for O. sativa in 12 s on average; whereas findMismatches completed analysis of the same data sets in an average of ∼7.5 min. This is a 38 × speedup. ISSL also significantly outperforms Cas-OFFinder. For instance, it completed the T. urartu test in ∼4 min on the workstation compared with 10 h for Cas-OFFinder. This is a 161 × speedup.

The mean run-times for each test are given in Table 2 and visualized in Figure 3A and B. ISSL is the fastest method for off-target scoring. For large genomes on low-memory machines, findMismatchesBit can also be a useful alternative, due to its lower memory footprint.

FIG. 3.

Run-time for an increase in genome size when using (A) high-performance machine and (B) workstation. The memory requirements of Crisflash on the workstation exceeded that available and caused swapping to occur; these tests were stopped. Run-time for generating off-targets index when using: (C) high-performance machine and (D) workstation.

Table 2.

Mean run-time, of five tests, when genome size is increased (min:s)

Genome	High-performance machine					Workstation
Genome	ISSL	fMb	fM	FlashFry	Crisflash	ISSL	fMb	fM	FlashFry	Crisflash	Cas-OFFinder
Oryza sativa	0:12	1:12	7:39	1:31	11:59	0:28	2:24	10:46	1:26	13:04	46:17
Panicum hallii	0:13	1:16	8:02	1:29	12:56	0:43	2:30	11:12	1:24	13:43	140:49
Xiphophorus couchianus	0:26	2:27	15:37	1:39	15:35	1:17	4:56	22:24	1:33	16:09	315:20
Parus major	0:46	3:38	22:52	2:15	19:36	2:25	7:19	33:05	2:06	—	134:29
Crotalus viridis	0:44	2:56	19:02	2:00	19:50	2:13	5:52	26:42	1:53	—	309:51
Danio rerio	0:37	2:39	17:01	1:43	17:01	1:31	5:12	23:37	1:45	—	178:37
Oncorhynchus mykiss	0:48	3:25	21:50	1:58	21:08	2:25	6:50	30:16	2:00	—	240:19
Mus musculus	1:40	4:49	30:41	—	—	5:13	9:18	41:52	—	—	383:51
Monodelphis domestica	2:01	4:20	26:37	—	—	6:10	7:58	36:04	—	—	500:43
Triticum urartu	0:56	2:46	17:37	—	—	3:48	5:03	306:15	—	—	612:32
Rhinatrema bivittatum	2:37	4:42	27:54	—	—	—
Triticum turgidum	2:18	4:19	21:49	—	—	—

— Indicates the test failed due to memory limitations.

fM, findMismatches; fMb, findMismatchesBit; ISSL, Inverted Signature Slice Lists.

For ISSL and FlashFry, the supporting indexes containing off-target sites are generated in a preprocessing step that is required only once. For Crisflash, the tree data structure is constructed at run-time so we modified the source code to include a timer for this. We timed the construction of the indexes for each of these tools and report the results in Table 3 and Figure 3C and D.

Table 3.

Mean run-time for generating the off-targets index, of five tests, when genome size is increased (min:s)

Genome	High-performance machine			Workstation
Genome	FlashFry	Crisflash	ISSL	FlashFry	Crisflash	ISSL
Oryza sativa	5:13	1:21	0:03	4:31	1:06	0:14
Panicum hallii	8:11	2:06	0:04	7:02	1:45	0:21
Xiphophorus couchianus	7:32	2:13	0:07	6:20	1:49	0:25
Parus major	12:42	4:07	0:13	11:06	—	0:43
Crotalus viridis	16:01	5:35	0:13	13:30	—	0:47
Danio rerio	13:41	4:20	0:11	12:10	—	0:49
Oncorhynchus mykiss	28:03	12:24	0:17	47:42	—	1:06
Mus musculus	—	—	0:32	—	—	1:55
Monodelphis domestica	—	—	0:38	—	—	2:12
Triticum urartu	—	—	0:27	—	—	2:53
Rhinatrema bivittatum	—	—	0:54	—	—	—
Triticum turgidum	—	—	1:02	—	—	—

— Indicates the test failed due to memory limitations.

Impact of the number of targets

We ran each tool on the O. sativa genome and varied the number of candidate guides to be evaluated. The results are shown in Table 4 for mean run-times over five tests.

Table 4.

Run-time when increasing the number of candidate guides to evaluate (min:s)

Data set	ISSL	fMb	fM	Cas-OFFinder	Crisflash
5k	0:08	1:12	5:22	23:11	6:33
10k	0:14	2:24	10:46	46:17	13:04
15k	0:20	3:34	16:02	68:51	19:35
20k	0:27	4:47	21:23	92:10	26:10
25k	0:33	5:58	26:44	115:11	32:48

Tested on Oryza sativa.

ISSL completed all tests in less than 35 s, whereas Cas-OFFinder completed the test on the smallest sized data set in 23 min and largest sized data set in 2 h. Here, ISSL performed up to 300 × faster than Cas-OFFinder and approximately an order of magnitude faster than the next tool.

Crackling is a fast method to produce efficient guides

We constructed the ISSL index for the data sets generated in a previous article where we benchmarked CRISPR guide design pipelines. We repeated these experiments using Crackling, using its default settings. The consensus approach, by default, considers a guide as efficient when at least two of the three approaches agree. The optimization level, by default, is set to high, which, for each guide, progressively excludes tests to reduce run-time. The average EFPS over five tests for the data sets were:

500k: 11,012 EFPS

1m: 12,406 EFPS

5m: 22,487 EFPS

full: 34,440 EFPS

We also explored the effect on run-time when changing the consensus approach or optimization level, from their respective defaults to other options. Without changing the optimization level but adjusting the consensus approach, run-time can be reduced by making decisions early. For example, when accepting a guide when at least one of the three tools agrees, run-time is reduced, on average, by 16%. The strictest consensus approach, where all methods need to agree, meaning guides are rejected early, decreases run-time, on average, by 57%. Without changing the consensus approach, but adjusting the optimization level from high to low, run-time is increased, on average, by 84%. The low optimization level evaluates all tests for those guides that are seen only once during the initial discovery step. We recommend carefully considering the effects of changing the consensus approach as the precision of Crackling may change. If the user wishes to improve run-time, or to retrieve more detailed results, then modifying the optimization level is how this should be achieved.

The EFPS for Crackling, using the default configuration, is visualized in Figure 4. In the benchmark, we used this figure to define low-, medium-, and high-performance groups. Crackling enters the high-performance group and is the only method in that group to correctly filter guides. Two other tools in this group, CasFinder and CRISPR-ERA, reported scores. However, for both tools, the score was identical across all guides, and as such, it did not carry useful information. Compared with other tools that filter guides, Crackling scales to larger sized data sets the best.

FIG. 4.

Run-time on mouse chromosome 19 data sets. This plot is extended from Bradford and Perrin¹ to include Crackling when ran with the default settings.

On the desktop workstation, we measured how fast Crackling was able to process guides for all genes in the human genome (for both on-target and off-target evaluation). It completed in 4 h and 36 min, without needing swap space for additional memory. This demonstrates the ability for Crackling to run on a standard workstation that does not have an unusually large volume of memory. In contrast, Crisflash is reported to need 90 GB of memory to process the human genome for off-target evaluation alone.¹² This volume of memory is usually available only in specialist computers. Even when we ran Crisflash on a machine with ample memory, it could not complete any of the experiments where the size of the input genome was close to that of the human (Table 2).

The quality of the guides produced by Crackling is also the highest among the tools reviewed. Table 5 describes the precision of each tool on two experimentally validated data sets, Wang and Doench.^22,23 As shown in Figure 5A for the Wang data set, Crackling correctly selected many guides that are efficient and very few that are inefficient. The same result can be observed for the Doench data set in Figure 5B. Crackling, by default, requires a candidate guide to be accepted by two of the three scoring approaches to be selected. When reconfigured this to require all three approaches to agree, the precision of Crackling increases (to 90% and 48.5% on the Wang and Doench data sets, respectively), but the recall drops to values that are only acceptable when a low number of candidates is sufficient (7.4% and 8.9%, respectively).

FIG. 5.

Results on the experimental data sets. (A) Wang data set. An ideal precision is when all accepted guides (those points in blue) appear in the efficient region. (B) Doench data set. As per Bradford and Perrin,² “The blue distribution shows the number of guides accepted, and the grey distribution shows the number of guides rejected. The vertical marker at 0.8 shows the threshold used to determine efficiency; guides with a gene rank score greater than this were deemed experimentally efficient.”

Table 5.

Precision of tools on experimentally validated data sets

Tool name				Wang			Doench
Tool name				Precision, %	Recall, %	F1, %	Precision, %	Recall, %	F1, %
CRISPR-DO				—	—	—	30.4	57.7	39.8
CHOPCHOP				84.3	31.5	45.8	29.4	42.6	34.5
SSC^a				—	—	—	27.7	78.7	40.8
sgRNAScorer2				83.3	55.1	66.3	27.0	65.0	38.4
PhytoCRISP-Ex^a				76.4	36.4	49.3	23.5	33.2	27.4
CRISTA				57.1	4.4	8.1	22.9	8.6	12.5
mm10db				65.2	29.4	40.5	22.7	23.5	23.1
Cas-Designer				61.2	17.2	26.9	21.0	37.7	26.0
FlashFry				84.4	16.3	27.3	—	—	—
WU-CRISPR				81.8	32.0	46.0	—	—	—
TUSCAN^a				71.5	72.1	71.8	—	—	—
Crackling configurations
mm10db	sgRNAScorer2	CHOPCHOP	Consensus
		✓	≥1	84.2	31.5	45.8	29.4	42.6	34.8
	✓		≥1	82.8	56.5	67.2	27.0	66.0	38.3
	✓	✓	≥1	81.3	59.4	68.6	26.7	68.7	38.5
	✓	✓	≥2	87.8	28.6	43.1	30.2	39.9	34.4
✓			≥1	64.1	30.5	41.3	22.7	23.5	23.1
✓		✓	≥1	71.7	54.3	61.8	25.0	56.6	34.7
✓		✓	≥2	83.6	7.7	14.1	43.8	9.4	15.5
✓	✓		≥1	73.2	71.7	72.4	24.3	74.7	36.7
✓	✓		≥2	85.5	15.3	26.0	35.9	14.8	21.0
✓	✓	✓	≥1	73.0	74.3	73.6	24.3	76.8	36.9
✓	✓	✓	≥2	85.1	36.8	51.4	29.3	46.4	35.9
✓	✓	✓	≥3	90.0	7.4	13.7	48.5	8.9	15.0

The bolded configuration for Crackling is its default. Table was adapted from Bradford and Perrin.¹

— Indicates the tool was not tested on this data set as it was used for training.

Indicates the tool was not able to complete all benchmarking tests from Bradford and Perrin.¹

These data sets contained activity measures for guides that were expressed using U6 promoters. To explore the adaptability of Crackling to experimental conditions where T7 promoters are used, we used a data set, as prepared by the authors of the CRISPOR pipeline, that contains mutation rates for 107 guides expressed using T7 promoters.⁷ The distribution of mutation rates suggested that a threshold of 0.2 is suitable to classify guides as being efficient or not. Using the default configuration of Crackling, a similar F1 score was obtained (57% compared with 51.4% for the Wang data set). However, this is with a lower precision (65%) and higher recall (52%). These results are preliminary, as publicly available data using T7 promoters appear to be sparse, but they suggest a possible impact. We will continue to investigate this as more data become available. The modularity and consensus-based approach that characterizes Crackling allows the future inclusion of additional scoring methods to address this if needed.

Discussion

One of the main challenges of experiments using CRISPR-Cas9 systems is the design of suitable guide RNAs. It is crucial, but not trivial, to ensure that these guides will be efficient but also specific enough not to lead to off-target modifications. Currently, there are no tools that meet both objectives while remaining practical to use on large genomes. In this article, we presented Crackling, a new tool for whole-genome identification of suitable CRISPR targets. Crackling is powered by a new high-performance off-target scoring approach based on ISSL and utilizes our previously published consensus approach for improving the precision when predicting guide efficiency.

When executing ISSL on a set of genomes using a high-performance machine, ISSL performs approximately an order of magnitude faster than the next best performing tool (findMismatchesBit), and up to two orders of magnitude faster than the worst performing tool (findMismatches).

We ran a number of off-target scoring tools on the O. sativa genome and varied the number of candidate guides to be evaluated. It would be expected that run-time is a linear function of input size, seeing that each candidate guide is evaluated individually. For all tools, we find that this is true.

Overall, our findings are that Cas-OFFinder is the poorest performing tool; findMismatches and findMismatchesBit are proportional; ISSL is the best performing, followed by Crisflash and finally FlashFry.

The consensus approach, which by default only accepts guides that have been predicted as efficient by at least two of the three scoring methods, provides gains in precision while preserving a reasonable recall. Changing the configuration of the consensus approach can further improve the precision through sacrificing recall. The precision, recall, and F1 score for all configurations are shown in Table 5.

Taken together, all these results show that Crackling can produce high-quality results in a relatively short period. It is among the best performing tools with respect to both critical properties (specificity and efficiency) that are considered when evaluating CRISPR guides.

We show that it provides the fastest way to calculate off-target risk, by using binary encoding and the ISSL method to identify closely related sites throughout the genome. This is confirmed on 12 genomes of increasing length. We also show that by combining the results of multiple scoring approaches, we can maximize the efficiency of the guides being designed. On experimental data, the set of guides it selects is better than those produced by existing tools.

Overall, this makes Crackling a faster and better method to design guide RNAs at scale.

Crackling is available as a standalone pipeline via GitHub (https://github.com/bmds-lab/Crackling) and will run on any modern workstation. For unusually large genomes, a machine with extra memory may be required. Installation and usage instructions are provided in the repository. Support is available by submitting a report using GitHub Issues.

Footnotes

Acknowledgments

We would like to thank the Big Data laboratory at the Queensland University of Technology for providing access to their high-performance machine, and the authors of the other tools for making their source code available.

Authors’ Contributions

J.B. and D.P. designed the study, analysed the results and wrote the manuscript. J.B. and D.P. implemented the on-target evaluation. J.B. and T.C. implemented the off-target scoring. All authors have reviewed and approved the article before submission. This article has been submitted solely to The CRISPR Journal, is not published, nor in press.

Author Disclosure Statement

No competing financial interests exist.

Funding Information

J.B. was supported by the Australian Government Research Training Program Scholarship. D.P. is supported by the Australian Research Council (ARC Discovery Project DP210103401).

Supplementary Material

References

Bradford

, Perrin

. A benchmark of computational CRISPR-Cas9 guide design methods. PLoS Comput Biol. 2019; 15:8. DOI: 10.1371/journal.pcbi.1007274.

Bradford

, Perrin

. Improving CRISPR guide design with consensus approaches. BMC Genomics. 2019; 20:1–11. DOI: 10.1186/s12864-019-6291-z.

Sledzinski

, Nowaczyk

, Olejniczak

. Computational tools and resources supporting CRISPR-Cas experiments. Cells. 2020; 9:5. DOI: 10.3390/cells9051288.

Bae

, Park

, Kim

. Cas-OFFinder: A fast and versatile algorithm that searches for potential offtarget sites of Cas9 RNA-guided endonucleases. Bioinformatics. 2014; 30:1473–1475. DOI: 10.1093/bioinformatics/btu048.

Hsu

, Scott

, Weinstein

, et al. DNA targeting specificity of RNA-guided Cas9 nucleases. Nat Biotechnol. 2013; 21:827–832. DOI: 10.1038/nbt.2647.

Zhu

, Holmes

, Aronin

, et al. CRISPRseek: A bioconductor package to identify target-specific guide RNAs for CRISPR-Cas9 genome-editing systems. PLoS One. 2014; 9:e108424. DOI: 10.1371/journal.pone.0108424.

Haeussler

, Schönig

, Eckert

, et al. Evaluation of off-target and on-target scoring algorithms and integration into the guide RNA selection tool CRISPOR. Genome Biol. 2016; 17:1. DOI: 10.1186/s13059-016-1012-2.

Concordet

, Haeussler

. CRISPOR: Intuitive guide selection for CRISPR/Cas9 genome editing experiments and screens. Nucleic Acids Res. 2018; 46:W242–W245. DOI: 10.1093/nar/gky354.

Sunagawa

, Sumiyama

, Ukai-Tadenuma

, et al. Mammalian reverse genetics without crossing reveals Nr3a as a short-sleeper gene. Cell Rep. 2016; 14:662–677. DOI: 10.1016/j.celrep.2015.12.052.

10.

Doench

, Fusi

, Sullender

, et al. Optimized sgRNA design to maximize activity and minimize off-target effects of CRISPR-Cas9. Nat Biotechnol. 2016; 34:184. DOI: 10.1038/nbt.3437.

11.

Zhang

, Xiangtao

, Qiuzhen

, et al. Synergizing CRISPR/Cas9 off-target predictions for ensemble insights and practical applications. Bioinformatics. 2019; 35:1108–1115. DOI: 10.1093/bioinformatics/bty748.

12.

Jacquin

ALS

, Odom

, Lukk

. Crisflash: Open-source software to generate CRISPR guide RNAs against genomes annotated with individual variation. Bioinformatics. 2019; 35:3146–3147. DOI: 10.1093/bioinformatics/btz019.

13.

McKenna

, Shendure

. Flashfry: A fast and flexible tool for large-scale CRISPR target design. BMC Biol. 2018; 16:74. DOI: 10.1186/s12915-018-0545-0.

14.

Xiao

, Cheng

, Kong

, et al. CasOT: A genome-wide Cas9/gRNA off-target searching tool. Bioinformatics. 2014; 30:1180–1182. DOI: 10.1093/bioinformatics/btt764.

15.

Cancellieri

, Canver

, Bombieri

, et al. CRISPRitz: Rapid, high-throughput and variant-aware in silico off-target site identification for CRISPR genome editing. Bioinformatics. 2020; 36:2001–2008. DOI: 10.1093/bioinformatics/btz867.

16.

Langmead

, Salzberg

. Fast gapped-read alignment with Bowtie 2. Nat Methods. 2012; 9:357–359. DOI: 10.1038/nmeth.1923.

17.

Lorenz

, Bernhart

, Zu Siederdissen

, et al. Viennarna package 2.0. Algorithms Mol Biol. 2011; 6:26. DOI: 10.1186/1748-7188-6-26.

18.

Chappell

, Geva

, Zuccon

. Approximate nearest-neighbour search with inverted signature slice lists. Adv Inform Retrieval. 2015; 9022:147–158. DOI: 10.1007/978-3-319-16354-3_16.

19.

Montague

, Cruz

, Gagnon

, et al. CHOPCHOP: A CRISPR/Cas9 and TALEN web tool for genome editing. Nucleic Acids Res. 2014; 42:401–407. DOI: 10.1093/nar/gku410.

20.

Chari

, Yeo

, Chavez

, et al. sgRNA Scorer 2.0: A species-independent model to predict CRISPR/Cas9 activity. ACS Synth Biol. 2017; 6:902–904. DOI: 10.1021/acssynbio.6b00343.

21.

Yan

, Dongyu

, Guphui

, et al. Benchmarking and integrating genome-wide CRISPR off-target detection and prediction. Nucleic Acids Res. 2020; 48:11370–11379: DOI: 10.1093/nar/gkaa930.

22.

Wang

, Wei

, Sabatini

, et al. Genetic screens in human cells using the CRISPR-Cas9 System. Science. 2014; 343:80–84. DOI: 10.1126/science.1246981.

23.

Doench

, Hartenian

, Graham

, et al. Rational design of highly active sgRNAs for CRISPR-Cas9-mediated gene inactivation. Nat Biotechnol. 2014; 32:1262–1267. DOI: 10.1038/nbt.3026.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

2.55 MB

1.67 MB