LncRNA-BERT: an RNA language model for classifying coding and long non

Abstract Understanding (novel) RNA transcripts generated in next generation sequencing experiments requires accurate classification, given the increasing evidence that long non-coding RNAs (lncRNAs) play crucial regulatory roles. Recent developments in Large Language Models present opportunities for classifying RNA coding potential with sequence-based algorithms that can overcome the limitations of classical approaches that assess coding potential based on a set of predefined features. We present lncRNA-BERT, an RNA language model pre-trained and fine-tuned on human RNAs collected from the GENCODE, RefSeq, and NONCODE databases to classify lncRNAs. LncRNA-BERT matches and outperforms state-of-the-art classifiers on three test datasets, including the cross-species RNAChallenge benchmark. The pre-trained lncRNA-BERT model distinguishes coding from long non-coding RNA without supervised learning which confirms that coding potential is a sequence-intrinsic characteristic. LncRNA-BERT has been shown to benefit from pre-training on human data from GENCODE, RefSeq, and NONCODE, improving upon configurations pre-trained on the commonly used RNAcentral dataset. In addition, we propose a novel Convolutional Sequence Encoding method which demonstrated superior efficiency and performance than K-mer Tokenization and Byte Pair Encoding for training with long RNA sequences that are otherwise above the common context window size. lncRNA-BERT is available at https://github.com/luukromeijn/lncRNA-Py. 1. Introduction Owing to advancements in RNA library extraction and sequencing technologies, bulk RNA and single-cell RNA sequencing have been regularly applied in research projects and routine clinical diagnostics. Bioinformatics analysis of these RNAseq datasets leverages the well-maintained annotations from gene annotation databases, such as RefSeq and Ensembl, to examine the expression profiles of genes or transcripts. Although earlier research has focused on protein-coding transcripts, more recent work has investigated noncoding RNAs (ncRNAs) for their important regulatory functions (Cesana et al., Citation2011; Fan et al., Citation2023; Li et al., Citation2021). Non-coding RNAs are often classified into two large groups according to their transcript length: short ncRNAs (including biotypes such as miRNA, piRNA, snoRNA, and tRNA) and long non-coding RNAs (lncRNAs) with length 200 nt. Specific databases, such as NONCODE and RNAcentral, have been implemented to track newly reported ncRNAs in different species (Sweeney et al., Citation2020; Zhao et al., Citation2020). In fact, lncRNAs are a highly prevalent type of RNA, underlined by the 173,112 human lncRNA transcripts stored in the NONCODE (v6) database versus the 197,151 mRNAs in RefSeq (r225). To maintain high-quality curation, it is important to classify novel transcripts into a proper biotype and, in particular, distinguish lncRNAs from protein-coding mRNAs, as their length ranges overlap and they are often detected together. This will benefit several downstream application domains that require high-quality lncRNA classification, such as the detection of novel cancer biomarkers (Beylerli et al., Citation2022; Guo et al., Citation2024; Hussain et al., Citation2023), the modelling of disease mechanisms (Hussain et al., Citation2024; Zhu et al., Citation2025), and the understanding of cell and tissue development (Bao et al., Citation2018; Zhu et al., Citation2024). Currently, researchers can choose from more than 40 algorithms to classify whether a new transcript sequence is an lncRNA. Most existing classifiers are machine learning models trained on annotated RNA data from RefSeq or GENCODE (e.g. CPC, CNCI, CPAT, and CPC2 (Kang et al., Citation2017; Kong et al., Citation2007; Sun et al., Citation2013; Wang et al., Citation2013)). These models rely on features such as open reading frame (ORF) length, protein database alignment hits, and k-mer frequencies to predict coding potential. CPC, CPC2, and CNCI are based on support vector machines (SVMs), whereas CPAT employs a logistic regression framework. Notably, CPAT also provides a widely used benchmarking dataset consisting of 4,000 human mRNAs and 4,000 lncRNAs, which has been adopted in numerous follow-up studies to evaluate the performance of novel lncRNA classifiers. The challenge of the classification problem lies in its inherent ambiguity: some lncRNAs contain short ORFs that translate into small peptide chains (Pang et al., Citation2018), and mRNA genes can have non-coding isoforms (Li et al., Citation2020; Mustoe et al., Citation2018). This has stimulated the continuous development of novel classifiers, leveraging the latest available annotations and methodological advancements, such as feature selection algorithms (Han et al., Citation2018; Liu et al., Citation2019), boosting models (Feng et al., Citation2023), and neural networks (Camargo et al., Citation2020; Hill et al., Citation2018; Yang et al., Citation2018). Although feature-based algorithms can learn the relationship between the target and a predetermined set of predictors, they may fail to capture the true underlying signal. Purely sequence-based deep learning methods for lncRNA classification have been proposed (Baek et al., Citation2018; Hill et al., Citation2018; Meng et al., Citation2021), but these were outperformed by feature-based or hybrid methods in a recent benchmark (Singh & Roy, Citation2022). These approaches utilise convolutional and/or recurrent neural networks (CNN/RNN), which suffer from limited receptive fields and exploding/vanishing gradients, respectively. The transformer architecture improves upon these designs by employing an attention mechanism and is nonrecurrent (Vaswani et al., Citation2017). We propose using a transformer-based Nucleotide Language Model (NLM) to overcome the limitations of previous classifiers. NLMs are Large Language Models (LLMs) that have been pre-trained and fine-tuned on genomic data. DNA NLMs have demonstrated their potential to predict chromatin features, identify promoter regions, and perform variant prioritisation (Dalla-Torre et al., Citation2024; Fishman et al., Citation2023; Zhou et al., Citation2023). RNA NLMs have been shown to be capable of predicting splice sites, secondary structures, and interactions and modification of RNAs (Akiyama & Sakakibara, Citation2022; Chen et al., Citation2022; Li et al., Citation2024; Penić et al., Citation2024; Wang et al., Citation2024; Yin et al., Citation2024). Despite being presented for general purposes, previous RNA language models may not be properly adapted for detecting coding potential due to utilising RNAcentral (which excludes mRNA) as the main data source and having a limited context length (< nt). Furthermore, classical coding potential analyses are highly dependent on the reading frame, as this parameter is used to compute features such as open reading frame length and coverage. In contrast, for NLM-based approaches, important questions remain regarding which sequence encoding strategies are least sensitive to potential frameshifts in the input sequences. In this work, we present a new lncRNA classification method called lncRNA-BERT (long non-coding RNA Bidirectional Encoder Representations from Transfomers), an RNA language model specifically adapted to classifying RNA as coding or long non-coding. The main contributions of our study are threefold. First, lncRNA-BERT obtained state-of-the-art classification performance on three different test sets and significantly outperformed all other methods in the most difficult cross-species RNAChallenge test set. Second, lncRNA-BERT effectively distinguished between mRNA and lncRNA without reliance on target labels by being pre-trained on human RNA sequences from GENCODE, RefSeq, and NONCODE, in contrast to other RNA NLMs that were trained on multi-species ncRNA from RNAcentral. Third, an in-depth comparison of four sequence encoding methods demonstrates the effectiveness of a novel Convolutional Sequence Encoding (CSE) method for pre-training on long RNA sequences and identifies CSE and 3-mer tokenization as the most suitable methods for classifying lncRNAs. 2. Methods lncRNA-BERT was implemented in the Python package lncRNA-Py, which is available from GitHub https://github.com/luukromeijn/lncRNA-Py). Additional experiments, such as architecture and hyperparameter tuning, have also been reported. 2.1. Data An overview of the datasets utilised for the different tasks is presented in . The human pre-training dataset was constructed by combining all RNA sequences from GENCODE (v46) (Frankish et al., Citation2022), NONCODE (v6) (Zhao et al., Citation2020), and RefSeq (v255) (O'Leary et al., Citation2015), comprising 297,724 mRNA and 238,470 lncRNA sequences. A randomly selected 5% subset of GENCODE RNA sequences was held out for validation. Additionally, we experimented with RNAcentral (v24) (Sweeney et al., Citation2020), which contains 37 million ncRNAs, as an alternative multi-species pre-training data source, reserving 2,500 sequences for validation. For fine-tuning where all parameters in the BERT model are optimised, it is important to ensure non-redundancy and test set independence. Thus, we applied the CD-HIT algorithm (Fu et al., Citation2012) similar to Feng et al. (Citation2023), Liu et al. (Citation2019) to eliminate duplicated and highly similar annotations across databases. CD-HIT algorithm was executed with a 90% sequence identity threshold on the combined human mRNA/lncRNA datasets from GENCODE and RefSeq to generate representative, non-redundant RNA sequences. These representative sequences were then randomly divided into 90% for training, 5% for validation, and 5% for testing. NONCODE was excluded at this stage since it contains sequences of varying degrees of annotation reliability, while those with strong support are already incorporated in GENCODE and RefSeq. In addition to the GENCODE/RefSeq-derived test set (5,650 mRNAs and 2,686 lncRNAs), two independent public test sets were employed to further evaluate lncRNA-BERT and other classifiers. The CPAT dataset (Wang et al., Citation2013) (4,000 RefSeq mRNAs and 4,000 human lncRNAs) is a widely used benchmark, although some overlap with the training sets of the evaluated classifiers is expected. The RNAChallenge dataset (Singh & Roy, Citation2022), consisting of 27,283 difficult-to-classify RNA sequences, was further used to assess the generalisability of our model to ambiguous RNAs from animal, plant, and fungal species. 2.2. Encoding methods We compared four methods for converting nucleotide sequences into a numerical format: Nucleotide-Level Tokenization (NUC), K-mer Tokenization (K-mer), Byte Pair Encoding (BPE), and a novel Convolutional Sequence Encoding (CSE) method. Efficient encoding is required for long sequences, as the transformer's attention mechanism only accepts a limited number of input positions (512-1024 on standard GPUs) due to its quadratic memory complexity. The effects of different encoding methods on the encoded sequence length are illustrated in . For most of these encoding methods, a medium-sized context window length of 768 input tokens is sufficient to completely encode the majority of transcripts in the human pre-training dataset. 2.2.1. Nucleotide-level tokenization Most RNA NLMs apply NUC as the sequence encoding method using a vocabulary of four nucleotide tokens (A,C,T,G). This technique allows attention to nucleotide resolution and works well for short sequences, such as most RNAs in RNAcentral. However, NUC is insufficient for long RNAs in our human pre-training set, as only 25% would fit into a medium-sized context length of 768 when NUC-encoded (). 2.2.2. K-mer tokenization With a vocabulary comprised of all nucleotide combinations of length , K-mer Tokenization tokenizes the input as non-overlapping consecutive k-mers (Dalla-Torre et al., Citation2024). This reduces the sequence length by a factor of and yields a vocabulary of size . For a large , the exponentially large vocabulary size introduces token sampling efficiency problems (Zhou et al., Citation2023) and results in an explosion of parameters in the transformer's embedding layer. This complicates the training and may not reflect the actual complexity of the data, as two k-mers are learned independently even when they are highly similar. Moreover, the model must learn and recognise each data signal in alternative reading frames, taking up parameters/dimensions that would preferably be dedicated to other patterns. 2.2.3. Byte pair encoding BPE considers the most frequently occurring combinations of characters as tokens (Sennrich et al., Citation2016) and was first applied to nucleotide sequences in Zhou et al. (Citation2023). During training, BPE iteratively expands its vocabulary by merging the most frequent token pairs in the input corpus. This process is repeated until a prespecified vocabulary size () is reached. During tokenization, BPE merges the subwords in the same order. BPE yields a highly efficient, fixed-size vocabulary of variable-length tokens. This method has three advantages over K-mer Tokenization: (1) a larger sequence length reduction for the same vocabulary size (); (2) a higher token sampling efficiency; and (3) a better robustness against frameshifts. BPE has been shown to improve upon K-mer Tokenization in multiple NLM applications (Romeijn et al., Citation2024; Zhou et al., Citation2023), but still requires a large vocabulary to obtain strong compression (Fishman et al., Citation2023). 2.2.4. Convolutional sequence encoding With CSE, nucleotide sequences are directly embedded into a high-dimensional space by means of a convolution (). First, we convert an input sequence of length to a Position Weight Matrix (PWM), for example and . A one-dimensional convolution layer (ReLU activated) with four input channels then transforms the PWM into dimensions for the transformer using learnable kernels. The stride is set to kernel size , such that the sequence length is reduced times, analogous to K-mer Tokenization. We add zero-padding to allow mini-batch training, and mask these positions during the attention operation. CSE effectively reduces the embedded sequence length with an efficient number of trainable parameters while maintaining nucleotide-level resolution. The method is inspired by the Vision Transformer (Dosovitskiy et al., Citation2020), a similar approach for short DNA sequences was taken in He et al. (Citation2023). Through the use of convolutions, CSE extracts important patterns from the data, seeing the input positions as combinations of nucleotides instead of independent tokens. 2.3. Language model architecture We adapted the transformer encoder architecture as used by the (Devlin et al., Citation2018; Vaswani et al., Citation2017), with transformer blocks, a dimensionality of , nodes in the feed-forward layers, and attention heads. lncRNA-BERT uses a medium-sized context length of input positions. The transformed embedding of the CLS token was used as input to the lncRNA classification output head, which is a sigmoid-activated linear layer containing a single node. The model has 85 M trainable parameters. Our CSE-based models use a modified version of BERT (). For classification, we replaced the CLS token with a learnable CLS embedding, as in ViT (Dosovitskiy et al., Citation2020). For the Masked Language Modelling (MLM) pre-training task, we enable nucleotide-level predictions by applying a softmax-activated, transposed convolution with stride/kernel size on linearly transformed embeddings. 2.4. Training We pre-train lncRNA-BERT for 7 days on an A100 MIG 4g.40GB GPU in the ALICE compute resources provided by Leiden University. Fine-tuning can take about 20 hours on the same A100 MIG 4g.40GB GPU or 30 hours on a smaller L4 24 GB GPU. Classification using the models published on the Hugging Face platform can be performed on both CPU and GPU with 6 GB memory. The optimal model configurations were determined based on pre-training, fine-tuning, and probing. The optimal model checkpoints were stored based on the validation set performance. 2.4.1. Pre-training MLM was used as a pre-training task, where tokens or nucleotides were selected with a probability of , of which 80% is masked and 10% is randomly replaced. The model's task is to predict which tokens or nucleotides occur at the selected positions. With CSE, we use the IUPAC symbol “N” to mask out nucleotides, as it indicates an equal probability for any of the four bases. The model is pre-trained using a cross entropy loss function, a batch size of 8, the Adam optimiser, and the learning rate schedule proposed in Vaswani et al. (Citation2017), with 32,000 warmup steps. 2.4.2. Fine-tuning LncRNA-BERT was fine-tuned with all parameters for coding potential classification on 101,270 coding and 48,785 long non-coding RNAs from GENCODE and RefSeq. We optimised 100 epochs of 10,000 randomly selected samples using Adam, a fixed learning rate of , and a batch size of 8. A smaller epoch size of 10,000 instead of a full pass through the complete training dataset allows us to have more frequent insights on how the model is performing during fine-tuning. A binary cross entropy loss function was used, with the reciprocal class sizes as weights to counteract the class imbalance. 2.4.3. Probing Probing is used to evaluate the extent to which a pre-trained model encodes the coding potential in its sequence embeddings, without fine-tuning the weights of the network itself (Dalla-Torre et al., Citation2024). Hereto, we trained a small Multi-Layer Perceptron with a single hidden layer of 256 nodes on the mean-pooled output embeddings of a model. Optimisation follows the same settings as during fine-tuning, except for an increased learning rate of . 2.5. Experimental setup A wide range of sequence-derived features have been proposed as possible coding-potential predictors in previous works. Similar to Han et al. (Citation2018), Li et al. (Citation2020), Zheng et al. (Citation2021), we provide a categorisation of these features into five main feature types as listed in . Note that a sequence's Open Reading Frame (ORF) and other patterns are purely sequence-intrinsic, and most physicochemical and secondary structure features can be indirectly inferred from sequence data. Hence, these features motivate the use of an NLM based approach for lncRNA classification. Nevertheless, numerous methods enrich these sequence-intrinsic features with extrinsic data, for example through database alignments. This provides feature-based algorithms with data that cannot be learned by NLMs. We compared lncRNA-BERT to six published lncRNA classification methods (). These include three classical machine learning approaches (CPAT (Wang et al., Citation2013), LncFinder (Han et al., Citation2018), PredLnc-GFStack (Liu et al., Citation2019)), and three deep learning methods with different model type (feature-based LncADeep (Yang et al., Citation2018), sequence-based mRNN (Hill et al., Citation2018), and hybrid model RNAsamba (Camargo et al., Citation2020). LncADeep, RNAsamba, and mRNN are the best-performing, publicly available, human models in a recent benchmark (Singh & Roy, Citation2022). For the main performance comparison, out-of-the-box models were used without retraining. To provide a fairer and more comprehensive evaluation, we also retrained three methods that support model retraining, CPAT, RNAsamba, and mRNN, using our labelled fine-tuning dataset. We experimented with two pre-training datasets (human, RNAcentral) and four encoding methods (NUC, K-mer, BPE, CSE) to determine an optimal lncRNA-BERT configuration. These comparisons were based on 500 pre-training epochs. The macro-averaged F1-score was used as the main performance metric, balancing precision and recall while equally weighing mRNA/ncRNA to address class imbalance. 3. Results 3.1. LncRNA-BERT outperforms and matches state-of-the-art performance in lncRNA classification and demonstrates a better generalisability We selected 3-mer tokenization and CSE with as the optimal encoding methods (Section 3.3), and benchmarked these configurations of lncRNA-BERT against six existing methods. As illustrated in , lncRNA-BERT matched state-of-the-art lncRNA classifiers in the two test sets and significantly outperformed all other methods in the most difficult RNAChallenge test set. This result proves the applicability of NLM for distinguishing coding from long non-coding RNA. In particular, we improved upon a previous sequence-based model (mRNN) by using a more advanced architecture and training procedure (BERT instead of RNN). Our BERT-based lncRNA classification method did not result in a large performance gain in comparison to the existing methods for two human test sets. Previous studies have set a high standard for lncRNA classification and a performance plateau is likely within reach (Section 4). In particular, for the CPAT test set, all methods performed almost equally well with a macro F1-score between 0.950 and 0.970 (lncRNA-BERT's score is 0.963). The fact that the CPAT test set has been a well-known set for many years and contains more distinguishable mRNAs and lncRNAs also makes true performance distinction difficult. Furthermore, some methods also benefit from sequence-extrinsic features, which cannot be learned by NLMs. For example, LncADeep performs alignments against a protein reference database to which lncRNA-BERT does not have access to. Nevertheless, we showed that coding potential is mostly a sequence-intrinsic characteristic, as lncRNA-BERT already distinguishes between mRNA and lncRNA after self-supervised pre-training (Section 3.2). In contrast to conventional feature-based approaches, both mRNN and lncRNA-BERT are purely sequence-based methods without the need for explicit feature selection or engineering. Notably, lncRNA-BERT outperformed mRNN across evaluation metrics, indicating that NLM based architectures may have advantages over traditional deep learning frameworks when trained exclusively on sequence data. Overfitting is a difficult problem in machine learning, which can cause some models to perform well on a particular test set but fail to generalise to unseen data. This potential bias is mostly caused by the similarity between the training and test sets, which we mitigated for lncRNA-BERT by generating an independent train/test split after redundancy removal with CD-HIT (Section 2.1). To address similar issue in the CPAT test test, we applied the CD-HIT method to construct a refined CPAT subset consisting of 2,321 mRNAs and 527 lncRNAs that differ by more than 10% from any sequence in our lncRNA-BERT fine-tuning training set. On this non-overlapping CPAT test set, lncRNA-BERT achieved comparable performance, with a weighted average F1 score of (3-mer) and (CSE). The consistent performance of lncRNA-BERT on an independent test set shows proper generalisation within human data competitive with established methods. The performance on the cross-species RNAChallenge benchmark also showed improvements in generalisation across species compared to established methods. Different than feature based machine learning methods, the explainability of NLM methods is limited. In order to interpret the decision made by lncRNA-BERT, we have checked one of most important feature of ORF coverage used in earlier methods (Han et al., Citation2018; Wang et al., Citation2013). As expected, correctly classified mRNAs (TP) by lncRNA-BERT have significantly larger ORF coverages than those misclassified (FN): with compared to with . Three existing methods, CPAT (Wang et al., Citation2013), RNAsamba (Camargo et al., Citation2020), and mRNN (Hill et al., Citation2018), support model retraining. We therefore retrained their models using our labelled fine-tuning data set, and the retrained performance are reported in (shown in brackets). While the retrained RNAsamba model maintained robust and comparable performance, it is slightly surprising that both CPAT and mRNN performed substantially worse on the GENCODE/RefSeq and CPAT test sets, where their macro-averaged F1 scores decreased significantly from to . In contrast, the retrained CPAT and mRNN models achieved improved results on the RNAChallenge test set, with macro-averaged F1 scores of and , respectively. This finding aligns with observations reported in the RNAChallenge study (Singh & Roy, Citation2022), which noted that tools with generally weaker overall performance may exhibit relatively stronger performance on the RNAChallenge benchmark. 3.2. Pre-training on human mRNA/lncRNA captures coding potential without access to target labels The MLM pre-training task enabled our model to differentiate coding from long non-coding RNA without depending on target labels, given an appropriate pre-training dataset, as seen in the embedding spaces in . Quantitatively, indicates that pre-training on human data leads to faster convergence and higher F1-scores (0.01–0.08 performance gain) compared to RNAcentral or no pre-training. This affirms that the coding potential is a prominent sequence-intrinsic signal. Although lncRNA-BERT is not the first method to solely base its predictions on sequence patterns (Hill et al., Citation2018; Li et al., Citation2014; Sun et al., Citation2013), it is the first to be able to partially discriminate between mRNA and lncRNA using a pre-training step in a self-supervised manner. Pre-training on RNAcentral biases the model towards ncRNA types other than human mRNA and lncRNA. This is reflected in the per-sequence MLM accuracy (% of correctly predicted tokens) in , which shows that the RNAcentral model performs well on sequences from RNAcentral (mean: 70%), but does not achieve the same accuracy on human lncRNAs (mean: 15%). The same can be concluded from the embedding spaces generated by both models. The RNAcentral model embeds mRNA and ncRNA less distinctly than the human model. We also generated embedding spaces for the ArchiveII dataset, containing multi-species ncRNAs from 10 structural families (Sloma & Mathews, Citation2016). Here, the RNAcentral model successfully separates the different classes, in contrast to the human model. Hence, the patterns learned from RNAcentral do not generalise well to human mRNA/lncRNA, and vice versa. This may be caused by the abundance of specific ncRNAs in RNAcentral, as well as the complete lack of mRNA. Moreover, lncRNAs are less well-conserved across evolution (Li et al., Citation2020), complicating the detection of coding potential in a cross-species dataset. Our results highlight a potential limitation of existing general-purpose RNA language models (Penić et al., Citation2024; Wang et al., Citation2024; Yin et al., Citation2024), which primarily rely on RNAcentral as their training data source, thereby reducing their suitability for lncRNA classification. Consequently, it is advisable to compare general-purpose NLMs with models pre-trained on domain- and species-specific datasets when selecting an appropriate model for a given downstream task. 3.3. Convolutional sequence encoding improves pre-training on longer RNA sequences We pre-trained, probed, and fine-tuned BERT models with different encoding methods (), and identified CSE () and 3-mer tokenization as the most suitable for coding potential classification. Of all encoding methods that achieve a large sequence length reduction ( times), CSE leads to the most effective models. This is reflected in the embedding spaces () as well as in the obtained F1-scores after probing (). Specifically, CSE and K-mer Tokenization obtain a probing F1-score of 0.93 versus 0.91 for , and 0.93 versus 0.79 for , respectively, on the validation dataset. CSE's fine-tuning scores are also higher for these configurations (). The reduced performance of K-mer Tokenization can be explained by the limitations of using a large token vocabulary (e.g. for 9-mers), which include reduced sampling efficiency (Zhou et al., Citation2023) and a high embedding layer parameter count (e.g. for 9-mers). By contrast, CSE sees k-mers as combinations of nucleotides and maintains a fixed number of parameters () by learning important patterns from the data. BPE encodes sequences into fully independent tokens, such as the k-mer approach, contributing to why it is outperformed by CSE for when probed. Although superior for long sequences, CSE is outperformed by K-mer and BPE when the token size is small and fine-tuning is allowed. In , CSE with results in a fine-tuning F1-score of 0.93, while the 3-mer and BPE for and , achieve 0.94, 0.93, and 0.94 respectively. Shorter tokens enable attention at a higher resolution, thereby improving contextualised embeddings. Seeing input positions as predefined, independent tokens also helps the model discriminate between sequences. In comparison, CSE must learn important patterns before BERT can condition on them, making the model less stable than when using a tokenizer. Issues specific to tokenizers, such as sampling efficiency and parameter count blow-ups, are less prevalent for smaller vocabularies. This specifically applies to fine-tuning, during which the model can prioritise important tokens over less important tokens. Consequently, BPE and K-mer tokenization show a larger performance gain from probing to fine-tuning compared to CSE. Another factor that affects classification performance is the sequence length coverage of each encoding method (). A longer coverage allows the model to consider a larger part of the sequence in its predictions, explaining why 3-mer tokenization is superior to NUC, as shown in . An extended context length may also hinder the model from learning important local signals such as CSE . The context length of a model with 3-mer tokenization () was shown to be sufficient for classifying coding potential. The biological implication here is that the coding potential of long RNA transcripts is usually inferrable from the first 2304 nucleotides. This, in combination with its small vocabulary, explains why a model with 3-mer tokenization achieves a similar fine-tuning performance to CSE with (0.944 versus 0.941), despite having a smaller context length. 3.4. Three-base periodicity in sequence encoding method benefits performance and affects embedding space Encoding methods that align with the three-base periodicity of coding RNA are shown to better distinguish mRNAs from lncRNAs, owing to their sensitivity to biological reading frames. For example, CSE with improves upon (). However, there is also a negative effect of the reading frame sensitivity of the three-base periodic encoding methods as illustrated in , in which the same RNA sequence could jump in the embedding space between the three groups representing different reading frames. While input position embeddings are reading frame dependent (particularly for K-mer and CSE), the aggregated (mean) embedding at the sequence level should remain the same, as the sequence's meaning is preserved. shows that this applies to BPE or CSE with , but not for K-mer or CSE with as a multiple of 3. Moreover, the figure shows that the Nucleotide Transformer, which uses 6-mer tokenization, suffers from the same issue. This effect was not observed in the non-coding sequences. These observations demonstrate how biological reading frames in the data affect the behaviour of three-base periodic models, which appear to favour in-frame signals. These signals are easier to learn, because their periodic organisation aligns with the input positions of the model. For example, from “ACT TGA ACT” it is easier to learn that “TGA” follows “ACT”, compared to learning that “GAA” follows “CTT”. We expect the presence/absence of such easily recognisable signals to cause the observed shifts between the embeddings in . Three-base periodicity is less evident in lncRNA, causing a lower MLM accuracy () but a better embedding consistency than mRNA. The frameshift sensitivity is mitigated by setting to a value not divisible by 3, which breaks up the three-base periodicity. 3.5. Embedding spaces of mRNA/lncRNA data reveal differences between NLMs We compared lncRNA-BERT to previously released NLMs by visualising the embedding spaces of the validation set in . lncRNA-BERT is pre-trained on human mRNA and lncRNA, in contrast to other RNA NLMs, which use RNAcentral as the main data source. Excluded from are ERNIE-RNA, RNABERT, RNAErnie, and RNA-FM (Akiyama & Sakakibara, Citation2022; Chen et al., Citation2022; Wang et al., Citation2024; Yin et al., Citation2024). These methods were only trained on non-coding RNAcentral data, making them less suitable for inference on mRNAs and lncRNAs (). BiRNA-BERT (Tahmid et al., Citation2024) and RiNALMo (Penić et al., Citation2024) also utilise the RNAcentral dataset, but augment it with data from RefSeq and Rfam/Ensembl. Nevertheless, the embedding space of BiRNA-BERT did not separate mRNA from ncRNA as clearly as lncRNA-BERT. Only the RiNALMo model (650 M parameters) generated a clearer distinction, but was 7.6 larger than lncRNA-BERT (85 M parameters) and was pre-trained with a limited context length (< nt). shows the competitiveness of lncRNA-BERT with other RNA NLMs on this dataset and task, while also being the most parameter-efficient. Interestingly, DNA NLMs such as DNABERT-2, Nucleotide Transformer (v2) and HyenaDNA (Nguyen et al., Citation2023) show some potential for RNA classification, suggesting that DNA NLMs may be generalised to RNA tasks. In particular, HyenaDNA, with its extended context length capability, achieves performance comparable to lncRNA-BERT and represents a promising direction for future research in RNA applications that demand longer contextual windows. 4. Discussion The human RNA language model proposed in this work, lncRNA-BERT, demonstrates state-of-the-art performance in classifying RNAs as coding or long non-coding (). Pre-training on human mRNA/lncRNA from GENCODE, RefSeq, and NONCODE leads to a clear distinction between the two classes, showing that an NLM can learn coding potential from sequence data without relying on target labels (). LncRNA-BERT stands apart from other RNA language models through its longer context length, smaller model size, and the use of human pre-training data, which enhances classification performance compared to models pre-trained with RNAcentral (). A performance plateau may have been reached for binary coding potential classification, which could explain why lncRNA-BERT did not consistently outperform existing methods. Even a simple logistic regression algorithm (CPAT) can reach an F1-score of 0.89 on GENCODE/RefSeq (), indicating the triviality of this task. At the same time, the problem is complicated by the false assumption of an unambiguous separation between mRNAs and lncRNAs. Some lncRNAs are known to contain short ORFs that encode functional micro-peptides (Pang et al., Citation2018), whereas some mRNAs can have lncRNA-like regulatory functions (Li et al., Citation2020; Mustoe et al., Citation2018). Hence, it is impossible to obtain 100% accuracy without overfitting to the human annotation system. Here, a method such as LncADeep benefits from incorporating protein alignment data in its predictions, which is also dependent on currently available knowledge. In contrast, lncRNA-BERT distinguishes coding from non-coding RNA without access to annotations, paving the way for a more nuanced view of the two RNA classes in future work. We demonstrated that the choice of encoding method and pre-training data is critical for lncRNA-BERT's performance. This highlights two key areas for potential improvements in NLMs in future studies. Regarding encoding methods, each have a different effect on the obtained sequence coverage, resolution, efficiency, and information retention. NUC represents nucleotides individually rather than combining them into single input positions, resulting in a demanding attention operation and a limited context length. While context length may be extended through architectural advancements, such as Attention with Linear Biases (Tahmid et al., Citation2024; Zhou et al., Citation2023) and Flash Attention (Penić et al., Citation2024; Zhou et al., Citation2023), adopting a more compressive encoding method is still advisable. BPE reduces the sequence length more efficiently than NUC and K-mer, but leads to reduced probing performance (). This indicates that BPE models are less capable of recognising coding potential after pre-training, in comparison to using K-mer or CSE. We attribute this to BPE's frequency-based vocabulary, which may not be biologically relevant and breaks up the three-base periodicity. Inconsistent token size also introduces problems for other fine-tuning tasks that require nucleotide resolution. Among the evaluated encoding methods, CSE achieves the strongest compression by employing learnable convolutions to capture complex patterns in RNA sequences while preserving nucleotide-level resolution. Nevertheless, 3-mer tokenization attains comparable performance to CSE (k = 9) (), suggesting that a context length of 2,304 nucleotides is sufficient for recognising the coding potential of most RNAs. We anticipate, however, that lncRNA-BERT with CSE may yield superior performance when fine-tuned for tasks that demand longer context lengths and nucleotide-level resolution, such as RNA secondary structure prediction and RNA-protein binding analysis (Penić et al., Citation2024). Ultimately, the optimal choice of encoding method depends on the characteristics of the specific problem and dataset, and identifying a universal solution will require systematic evaluation across diverse fine-tuning tasks. Regarding pre-training data, an NLM requires human mRNA/lncRNA to achieve maximum performance in distinguishing the two classes, as shown in and . This demonstrates the necessity of a sufficient amount of task-specific pre-training data. Nevertheless, we anticipate that lncRNA-BERT and other RNA language models may be improved by increasing the dataset size when a proper class balance and sequence diversity are ensured. Adding genetic variation has helped DNA language models generalise over subtle signals between different individuals and phylogenetic signals between species. For example, pre-training the Nucleotide Transformer with data from the 1000 Genomes Project enables it to perform variant prioritisation (Dalla-Torre et al., Citation2024), and GENA-LM can be used for taxonomic classification after pre-training on multi-species data (Fishman et al., Citation2023). In contrast, our results show that using the 37 M multi-species RNAcentral dataset as the only pre-training source does not improve lncRNA-BERT, but results in a bias towards RNA types other than mRNA and lncRNA (). Algorithms such as CD-HIT and MMSeqs2 can remove redundant sequences from datasets like RNAcentral, which may be the key to ensuring proper data balance and diversity. RiNALMo implements this approach, but excludes mRNA (Penić et al., Citation2024). Hence, pre-training a model with an efficient encoding method on a well-balanced dataset encompassing all types of RNA may yield the next generation of RNA NLMs. Training and fine-tuning NLMs require substantial computational infrastructure and access to advanced GPUs, as the process is both resource-intensive and time-consuming. These constraints limit the number of training runs and reduce flexibility in exploring different validation and testing splits for our lncRNA-BERT model, as well as for larger NLMs such as RiNALMo and HyenaDNA. Consequently, we were unable to conduct repeated training-evaluation cycles to obtain more robust statistical estimates of model performance. Moreover, due to the blackbox nature of NLMs, the decision-making process of lncRNA-BERT remains poorly understood and it is not yet clear which sequence patterns are most critical for classification. Future research leveraging explainable AI tools such as Attention Map and Layer-Wise Relevance Propagation (LRP) may provide deeper insights into the mechanisms underlying NLM predictions. Given the substantial computational resources and expertise required to train large NLMs, we anticipate that this responsibility will primarily fall to large organisations and genomic centres. These institutions could leverage approaches such as federated learning to incorporate sensitive genomic data (Beutel et al., Citation2022) while also advancing efforts in NLM explainability. Smaller institutions would then be well positioned to fine-tune these foundational models for domain specific applications, such as patient splicing variant analysis, alternative splicing prediction, or RNA interaction modelling, either locally to address data sensitivity concerns or through cloud based platforms. Disclosure statement None declared. Funding None declared. Data availability statement The source data used in this study are available at the public gene annotation databases namely GENCODE (v46) at https://www.gencodegenes.org/human/release_46.html, RefSeq (v225) at https://ftp.ncbi.nlm.nih.gov/refseq/release/release-catalog/archive/RefSeq-release225.catalog.gz, NONCODE (v6) at http://www.noncode.org/datadownload/NONCODEv6_human.fa.gz, and RNAcentral (v24) at https://ftp.ebi.ac.uk/pub/databases/RNAcentral/releases/24.0/sequences/. The scripts supporting the findings of this study are openly available from GitHub at https://github.com/luukromeijn/lncRNA-Py. Any additional underlying data with respect to the manuscript will be made available upon reasonable request. A pre-print of this article is available at Romeijn et al. (Citation2025). References - Akiyama, M., & Sakakibara, Y. (2022). Informative rna base embedding for rna structural alignment and clustering by deep representation learning. NAR Genomics and Bioinformatics, 4(1). https://doi.org/10.1093/nargab/lqac012 - Baek, J., Lee, B., Kwon, S., Yoon, S., & Valencia, A. (2018). Lncrnanet: Long non-coding rna identification using deep learning. Bioinformatics (Oxford, England), 34(22), 3889–3897. https://doi.org/10.1093/bioinformatics/bty418 - Bao, M. H., Li, G. Y., Huang, X. S., Tang, L., & Dong, L. (2018). Long noncoding rna linc00657 acting as a mir-590-3p sponge to facilitate low concentration oxidized low-density lipoprotein-induced angiogenesis. Molecular Pharmacology, 93(4), 368–375. https://doi.org/10.1124/mol.117.110650 - Beutel, D. J., Topal, T., Mathur, A., Qiu, X., Fernandez-Marques, J., Gao, Y., Sani, L., Li, K. H., Parcollet, T., de Gusmão, P. P. B., & D. Lane, N. (2022). Flower: A friendly federated learning research framework. Available from: https://arxiv.org/abs/2007.14390 - Beylerli, O., Gareev, I., Sufianov, A., Ilyasova, T., & Guang, Y. (2022). Long noncoding rnas as promising biomarkers in cancer. Non-coding RNA Research, 7(2), 66–70. https://doi.org/10.1016/j.ncrna.2022.02.004 - Camargo, A. P., Sourkov, V., Pereira, G., & Carazzolle, M. F. (2020). Rnasamba: Neural network-based assessment of the protein-coding potential of rna sequences. NAR Genomics and Bioinformatics, 2(1). https://doi.org/10.1093/nargab/lqz024 - Cesana, M., Cacchiarelli, D., Legnini, I., Santini, T., Sthandier, O., Chinappi, M., Tramontano, A., & Bozzoni, I. (2011). A long noncoding rna controls muscle differentiation by functioning as a competing endogenous rna. Cell (Cambridge, MA), 147(2), 358–369. https://doi.org/10.1016/j.cell.2011.09.028 - Chen, J., Hu, Z., Sun, S., Tan, Q., Wang, Y., Yu, Q., Zong, L., Hong, L., Xiao, J., Shen, T., King, I., & Li, Y. (2022). Interpretable rna foundation model from unannotated data for highly accurate rna structure and function predictions. Available from: https://arxiv.org/abs/2204.00300 - Dalla-Torre, H., Gonzalez, L., Mendoza-Revilla, J., Lopez Carranza, N., Grzywaczewski, A. H., Oteri, F., Dallago, C., Trop, E., de Almeida, B. P., Sirelkhatim, H., Richard, G., Skwark, M., Beguir, K., Lopez, M., & Pierrot, T. (2024). Nucleotide transformer: building and evaluating robust foundation models for human genomics. Nature Methods, 22, 287–297. https://doi.org/10.1038/s41592-024-02523-z - Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. Available from: https://arxiv.org/abs/1810.04805 - Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. Available from: https://arxiv.org/abs/2010.11929 - Fan, C., González-Prieto, R., Kuipers, T. B., Vertegaal, A. C. O., van Veelen, P. A., Mei, H., & ten Dijke, P. (2023). The lncrna lets1 promotes tgf-β-induced emt and cancer cell migration by transcriptionally activating a tβr1-stabilizing mechanism. Science Signaling, 16(790). https://doi.org/10.1126/scisignal.adf1947 - Feng, H., Wang, S., Wang, Y., Ni, X., Yang, Z., Hu, X., & Yang, S. (2023). Lnccat: An orf attention model to identify lncrna based on ensemble learning strategy and fused sequence information. Computational and Structural Biotechnology Journal, 21, 1433–1447. https://doi.org/10.1016/j.csbj.2023.02.012 - Fishman, V., Kuratov, Y., Shmelev, A., Petrov, M., Penzar, D., Shepelin, D., Chekanov, N., Kardymon, O., & Burtsev, M. (2023). Gena-lm: A family of open-source foundational dna language models for long sequences. https://doi.org/10.1101/2023.06.12.544594 - Frankish, A., Carbonell-Sala, S., Diekhans, M., Jungreis, I., Loveland, J. E., Mudge, J. M., Sisu, C., Wright, J. C., Arnan, C., Barnes, I., Banerjee, A., Bennett, R., Berry, A., Bignell, A., Boix, C., Calvet, F., Cerdán-Vélez, D., Cunningham, F., Davidson, C., … Flicek, P. (2022). Gencode: Reference annotation for the human and mouse genomes in 2023. Nucleic Acids Research, 51(D1), D942–D949. https://doi.org/10.1093/nar/gkac1071 - Fu, L., Niu, B., Zhu, Z., Wu, S., & Li, W. (2012). Cd-hit: Accelerated for clustering the next-generation sequencing data. Bioinformatics (Oxford, England), 28(23), 3150–3152. https://doi.org/10.1093/bioinformatics/bts565 - Guo, Z., Guan, K., Bao, M., He, B., & Lu, J. (2024). Linc-pint plays an anti-tumor role in nasopharyngeal carcinoma by binding to xrcc6 and affecting its function. Pathology - Research and Practice, 260, 155460. https://doi.org/10.1016/j.prp.2024.155460 - Han, S., Liang, Y., Ma, Q., Xu, Y., Zhang, Y., Du, W., Wang, C., & Li, Y. (2018). Lncfinder: An integrated platform for long non-coding rna identification utilizing sequence intrinsic composition, structural information and physicochemical property. Briefings in Bioinformatics, 20(6), 2009–2027. https://doi.org/10.1093/bib/bby065 - He, S., Gao, B., Sabnis, R., & Sun, Q. (2023). Nucleic transformer: Classifying dna sequences with self-attention and convolutions. ACS Synthetic Biology, 12(11), 3205–3214. https://doi.org/10.1021/acssynbio.3c00154 - Hill, S. T., Kuintzle, R., Teegarden, A., Merrill, E., Danaee, P., & Hendrix, D. A. (2018). A deep recurrent neural network discovers complex biological rules to decipher rna protein-coding potential. Nucleic Acids Research, 46(16), 8105–8113. https://doi.org/10.1093/nar/gky567 - Hussain, M. S., Sharma, S., Kumari, A., Kamran, A., Bahl, G., Bisht, A. S., Sultana, A., Ashique, S., Ramalingam, P. S., & Arumugam, S. (2024). Role of long non-coding rnas in neurofibromatosis and schwannomatosis: Pathogenesis and therapeutic potential. Epigenomics, 16(23-24), 1453–1464. https://doi.org/10.1080/17501911.2024.2430170 - Hussain, M. S., Afzal, O., Gupta, G., Altamimi, A. S. A., Almalki, W. H., Alzarea, S. I., Kazmi, I., Fuloria, N. K., Sekar, M., Meenakshi, D. U., Thangavelu, L., & Sharma, A. (2023). Long non-coding rnas in lung cancer: Unraveling the molecular modulators of mapk signaling. Pathology - Research and Practice, 249, 154738. https://doi.org/10.1016/j.prp.2023.154738 - Kang, Y. J., Yang, D. C., Kong, L., Hou, M., Meng, Y., Wei, L., & Gao, G. (2017). Cpc2: A fast and accurate coding potential calculator based on sequence intrinsic features. Nucleic Acids Research, 45(W1), W12–W16. https://doi.org/10.1093/nar/gkx428 - Kong, L., Zhang, Y., Ye, Z. Q., Liu, X., Zhao, S., Wei, L., & Gao, G. (2007). Cpc: Assess the protein-coding potential of transcripts using sequence features and support vector machine. Nucleic Acids Research, 35(suppl_2), W345–W349. https://doi.org/10.1093/nar/gkm391. - Li, A., Zhang, J., & Zhou, Z. (2014). Plek: A tool for predicting long non-coding rnas and messenger rnas based on an improved k-mer scheme. BMC Bioinformatics, 15(1), 311. https://doi.org/10.1186/1471-2105-15-311 - Li, J., Zhang, X., & Liu, C. (2020). The computational approaches of lncrna identification based on coding potential: Status quo and challenges. Computational and Structural Biotechnology Journal, 18, 3666–3677. https://doi.org/10.1016/j.csbj.2020.11.030 - Li, C., Wang, H., Wen, Y., Yin, R., & Zeng, X. (2024). Genom7gnet: An efficient n7-methylguanosine site prediction approach based on a nucleotide language model. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 21(6), 2258–2268. https://doi.org/10.1109/TCBB.2024.3459870 - Li, S., Xiong, Q., Chen, M., Wang, B., Yang, X., Cui, Z., & Ge, F. (2021). Long noncoding rna hotair interacts with y-box protein-1 (ybx1) to regulate cell proliferation. Life Science Alliance, 4(9), e202101139. https://doi.org/10.26508/lsa.202101139 - Liu, S., Zhao, X., Zhang, G., & Li, W. (2019). Predlnc-gfstack: A global sequence feature based on a stacked ensemble learning method for predicting lncrnas from transcripts. Genes, 10(9), 672. https://doi.org/10.3390/genes10090672 - Meng, J., Kang, Q., Chang, Z., & Luan, Y. (2021). Plncrna-hdeep: plant long noncoding rna prediction using hybrid deep learning based on two encoding styles. BMC Bioinformatics, 22(S3), 242. https://doi.org/10.1186/s12859-020-03870-2 - Mustoe, A. M., Busan, S., Rice, G. M., Hajdin, C. E., Peterson, B. K., Ruda, V. M., Kubica, N., Nutiu, R., Baryza, J. L., & Weeks, K. M. (2018). Pervasive regulatory functions of mrna structure revealed by high-resolution shape probing. Cell (Cambridge, MA), 173(1), 181–195.e18. https://doi.org/10.1016/j.cell.2018.02.034 - Nguyen, E., Poli, M., Faizi, M., Thomas, A., Birch-Sykes, C., Wornow, M., Patel, A., Rabideau, C., Massaroli, S., Bengio, Y., Ermon, S., Baccus, S. A., & Ré, C. (2023). Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Available from: https://arxiv.org/abs/2306.15794 - O'Leary, N. A., Wright, M. W., Brister, J. R., O'Leary, N. A., Ciufo, S., Haddad, D., McVeigh, R., Rajput, B., Robbertse, B., Smith-White, B., Ako-Adjei, D., Astashyn, A., Badretdin, A., Bao, Y., Blinkova, O., Brover, V., Chetvernin, V., Choi, J., Cox, E., … Pruitt, K. D. (2015). Reference sequence (refseq) database at ncbi: Current status, taxonomic expansion, and functional annotation. Nucleic Acids Research, 44(D1), D733–D745. https://doi.org/10.1093/nar/gkv1189 - Pang, Y., Mao, C., & Liu, S. (2018). Encoding activities of non-coding rnas. Theranostics, 8(9), 2496–2507. https://doi.org/10.7150/thno.24677 - Penić, R. J., Vlašić, T., Huber, R. G., Wan, Y., & Šikić, M. (2024). General-purpose rna language models can generalize well on structure prediction tasks. Available from: https://arxiv.org/abs/2403.00043 - Romeijn, L., Bernatavicius, A., & Vu, D. (2024). Mycoai: Fast and accurate taxonomic classification for fungal its sequences. Molecular Ecology Resources, 24(8). https://doi.org/10.1111/1755-0998.14006 - Romeijn, L., Cats, D., Wolstencroft, K., & Mei, H. (2025). Lncrna-bert: An rna language model for classifying coding and long non-coding rna. bioRxiv. Available from: https://www.bioRxiv.org/content/early/2025/01/17/2025.01.09.632168 - Sennrich, R., Haddow, B., & Birch, A. (2016). Neural machine translation of rare words with subword units. In K. Erk, & N. A. Smith (Eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Vol. 1, Long Papers, pp. 1715–1725). Berlin, Germany: Association for Computational Linguistics. https://doi.org/10.18653/v1/P16-1162 - Singh, D., & Roy, J. (2022). A large-scale benchmark study of tools for the classification of protein-coding and non-coding rnas. Nucleic Acids Research, 50(21), 12094–12111. https://doi.org/10.1093/nar/gkac1092 - Sloma, M. F., & Mathews, D. H. (2016). Exact calculation of loop formation probability identifies folding motifs in rna secondary structures. RNA, 22(12), 1808–1818. https://doi.org/10.1261/rna.053694.115 - Sun, L., Luo, H., Bu, D., Zhao, G., Yu, K., Zhang, C., Liu, Y., & Chen, R. (2013). Utilizing sequence intrinsic composition to classify protein-coding and long non-coding transcripts. Nucleic Acids Research, 41(17), e166–e166. https://doi.org/10.1093/nar/gkt646 - Sweeney, B. A., Petrov, A. I., Ribas, C. E., Finn, R. D., Bateman, A., Szymanski, M., Karlowski, W. M., Seemann, S. E., Gorodkin, J., Cannone, J. J., Gutell, R. R., Kay, S., Marygold, S., dos Santos, G., Frankish, A., Mudge, J. M., Barshir, R., Fishilevich, S., Chan, P. P., … Weinberg, Z. (2020). Rnacentral 2021: Secondary structure integration, improved sequence search and new member databases. Nucleic Acids Research, 49(D1), D212–D220. https://doi.org/10.1093/nar/gkaa921 - Tahmid, M. T., Shahgir, H. S., Mahbub, S., Dong, Y., & Bayzid, Md. S. (2024). Birna-bert allows efficient rna language modeling with adaptive tokenization. https://doi.org/10.1101/2024.07.02.601703 - Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, L. (2017). Attention is all you need. CoRR. abs/1706.03762. Available from: http://arxiv.org/abs/1706.03762 - Wang, L., Park, H. J., Dasari, S., Kocher, J., & Li, W. (2013). Cpat: coding-potential assessment tool using an alignment-free logistic regression model. Nucleic Acids Research, 41(6), e74–e74. https://doi.org/10.1093/nar/gkt006 - Wang, N., Bian, J., Li, Y., Mumtaz, S., Kong, L., & Xiong, H. (2024). Multi-purpose rna language modelling with motif-aware pretraining and type-guided fine-tuning. Nature Machine Intelligence, 6(5), 548–557. https://doi.org/10.1038/s42256-024-00836-4 - Yang, C., Yang, L., Zhou, M., Xie, H., Zhang, C., Wang, M. D., Zhu, H., & Birol, I. (2018). Lncadeep: Anab initiolncrna identification and functional annotation tool based on deep learning. Bioinformatics (Oxford, England), 34(22), 3825–3834. https://doi.org/10.1093/bioinformatics/bty428 - Yin, W., Zhang, Z., He, L., Jiang, R., Zhang, S., Liu, G., Zhang, X., Qin, T., & Xie, Z. (2024). Ernie-rna: An rna language model with structure-enhanced representations. https://doi.org/10.1101/2024.03.17.585376 - Zhao, L., Wang, J., Li, Y., Song, T., Wu, Y., Fang, S., Bu, D., Sun, L., Pei, D., Zheng, Y., Huang, J., Xu, M., Chen, R., & He, S. (2020). Noncodev6: An updated database dedicated to long non-coding rna annotation in both animals and plants. Nucleic Acids Research, 49(D1), D165–D171. https://doi.org/10.1093/nar/gkaa1046 - Zheng, H., Talukder, A., Li, X., & Hu, H. (2021). A systematic evaluation of the computational tools for lncrna identification. Briefings in Bioinformatics, 22(6). https://doi.org/10.1093/bib/bbab285 - Zhou, Z., Ji, Y., Li, W., Dutta, P., Davuluri, R., & Liu, H. (2023). Dnabert-2: Efficient foundation model and benchmark for multi-species genome. Available from: https://arxiv.org/abs/2306.15006 - Zhu, L., Liu, Y., Gong, Z., Zhang, C., Hui, S., Duan, S., Bing, P., & Yao, Z. (2025). Noncoding rnas in atopic dermatitis: Insight into inflammation and immune regulation. Dermatologic Therapy, 2025(1), 5568546. https://doi.org/10.1155/dth/5568546 - Zhu, Q., Sun, J., An, C., Li, X., Xu, S., He, Y., Zhang, X., Liu, L., Hu, K., & Liang, M. (2024). Mechanism of lncrna gm2044 in germ cell development. Frontiers in Cell and Developmental Biology, 12. https://doi.org/10.3389/fcell.2024.1410914