Enhancing Arabic Handwritten Recognition System-Based CNN-BLSTM Using Generative Adversarial Networks

Arabic Handwritten Recognition (AHR) presents unique challenges due to the complexity of Arabic script and the limited availability of training data. This paper proposes an approach that integrates generative adversarial networks (GANs) for data augmentation within a robust CNN-BLSTM architecture, aiming to significantly improve AHR performance. We employ a CNN-BLSTM network coupled with connectionist temporal classification (CTC) for accurate sequence modeling and recognition. To address data limitations, we incorporate a GANs based data augmentation module trained on the IFN-ENIT Arabic handwriting dataset to generate realistic and diverse synthetic samples, effectively augmenting the original training corpus. Extensive evaluations on the IFN-ENIT benchmark demonstrate the efficacy of adopted approach. We achieve a recognition rate of 95.23%, surpassing the baseline model by 3.54%. This research presents a promising approach to data augmentation in AHR and demonstrates a significant improvement in word recognition accuracy, paving the way for more robust and accurate AHR systems.

Introduction

Pattern recognition is a broad field dedicated to extracting and interpreting meaningful information from intricate data patterns. Among its various subdomains, text recognition is pivotal in numerous applications, such as data storage, mobile and desktop applications, historical manuscript processing and preservation, among other fields [1]–[5]. Arabic Handwritten Recognition (AHR) is particularly challenging due to the inherent complexities of Arabic script, including diverse writing styles, ligatures, and diacritics. The intricate character connections and vowel marks in Arabic script make this a difficult task. Despite significant advancements, achieving high accuracy in AHR remains an ongoing pursuit [6]–[9].

A primary obstacle in AHR research is the limited availability of large-scale, diverse datasets. Training deep learning models, especially those involving Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) [10]–[13], requires ample data to learn robust representations and generalize effectively. The scarcity of labeled Arabic handwritten datasets hinders model performance and limits practical applicability.

Generative Adversarial Networks (GANs) have revolutionized pattern recognition and computer vision with their ability to generate realistic synthetic data. They have found applications in diverse areas such as neuroimaging and clinical neuroscience, remote sensing and computer vision for audio-visual speech recognition [14]–[17]. These models consist of two competing neural networks: a generator that creates synthetic data resembling the real distribution and a discriminator that attempts to distinguish real from generated data. This adversarial training process incentivizes the generator to produce increasingly realistic samples, potentially alleviating data scarcity issues.

This paper proposes an approach to AHR by leveraging GANs for data augmentation. We employ a CNN-BLSTM model, a well-established architecture for sequence modeling in AHR. By using GANs as a data augmentation technique, we can generate a large amount of synthetic Arabic handwriting data. This not only helps in improving the robustness of the handwriting recognition model but also helps in tackling the problem of data scarcity, which is a common issue in handwriting recognition tasks. Thus, GANs hold great potential in advancing the field of Arabic handwriting recognition. Our key contributions include:

• A GAN-based data augmentation technique for AHR: GAN architecture that captures the intricacies of Arabic handwriting while preserving the essential characteristics of real data.
• Improving recognition accuracy using the augmented dataset: We demonstrate that training the CNN-BLSTM model on augmented data effectively enhances its performance, surpassing the baseline model trained on real data alone.
• Paving the way for more robust and accurate AHR systems: This research presents a promising avenue for addressing data limitations in AHR and contributes to the development of more robust and accurate recognition systems.

The structure of this paper is organized as follows: Section 2 introduces the Arabic handwriting recognition approach and the application of Generative Adversarial Networks as a data augmentation technique. Section 3 delineates the GAN architecture implemented in this study and a description of our baseline CNN-BLSTM model. Section 4 details the experimental setup and presents the obtained results. The paper concludes with Section 5, which summarizes the key findings, discusses potential avenues for future research, and underscores the broader implications of this study for the AHR domain.

Literature Review

The field of Arabic Handwritten Text Recognition has seen significant progress in recent years. Before the deep learning revolution, offline handwritten text recognition relied heavily on Hidden Markov Models (HMMs) [18]–[21]. These statistical models examined sequences of features but struggled to consider the broader context of surrounding characters. To overcome this limitation, researchers experimented with hybrid approaches, combining HMMs with neural networks models [22]–[24]. The paradigm shift emerged with the rise of deep learning, specifically Convolutional Neural Networks and Recurrent Neural Networks. CNNs automatically extract robust features directly from images, while RNNs excel at capturing the sequential nature of characters and words. Combining their strengths, Convolutional Recurrent Neural Networks (CRNNs) became the reigning champions of offline handwriting recognition. CRNNs first use a CNN to generate a sequence of features from the image, then feed those features into an RNN to decode the final text.

Despite considerable advancements in AHR, it continues to face several challenges. One of the primary obstacles is the limited availability of large-scale, diverse datasets. This scarcity of data impedes the training of robust deep learning models, especially those with complex architectures that necessitate substantial data for effective generalization [25]–[27]. Data scarcity has long been a major obstacle in deep learning; this challenge necessitates the use of data augmentation techniques, which artificially expand the training data by modifying existing samples or generating new ones [28]–[31].

In handwriting recognition, acquiring large and diverse datasets can be extremely difficult. Traditional data augmentation techniques involve geometric transformations and image manipulations such as rotation, scaling, skewing, and shifting of images, introducing variability in orientation, size and position, thereby mimicking the natural variations observed in handwritten text [32]–[38]. Elastic deformations, achieved through random warping or jittering of the image, simulate distortions and imperfections typically found in real-world documents. Furthermore, changes in intensity and the injection of noise, which involve modifying the brightness and contrast and adding noise, respectively, can simulate varying lighting conditions and imperfections encountered during the scanning or capturing of handwritten text. Despite their straightforward implementation and computational efficiency, these methods have limited effectiveness in AHR. They often fall short in capturing the specific complexities of the Arabic script, such as complex ligatures, diacritics, and diverse writing styles.

The limitations of traditional data augmentation techniques in capturing the specific complexities of Arabic handwriting necessitate exploring more deeped approaches. Generative adversarial networks have demonstrated significant potential in enhancing performances by offering the ability to generate realistic and diverse synthetic data, making them widely applied in various fields. In medicine, the data augmentation GAN has been used to enhance the performance of basic Convolutional Neural Network classifiers on datasets EMNIST, VGG-Face, and Omniglot [39]. Deep learning techniques have been used to analyze neuroimaging data for diagnosing neurodegenerative diseases. Due to the limited availability of neuroimaging data, Deep Convolutional Adversarial Networks (DCGANs) have been used to generate synthetic images, thereby increasing the size and variety of the dataset [40]. Similarly, in a study on functional near-infrared spectroscopy (fNIRS), Wasserstein GANs were used to generate artificial fNIRS data, which improved the accuracy of classifying different task types [41].

Recent studies on handwritten recognition have focused on addressing the challenges of data scarcity and class imbalance through the use of GANs. Alwaqfi et al. [42] proposed a model that uses a GAN and a CNN to recognize Arabic handwritten characters. The GAN consists of a generator and a discriminator, both of which are CNNs with 10 and 9 layers, respectively. The model was trained on the Arabic Handwritten Characters Dataset (AHCD) and achieved an accuracy of 99.78% when the GAN was used and 96.28% when it was not. Eltay et al. [43] developed a GAN model based on the ScrabbleGAN model to address the imbalance in the frequency of generated handwritten Arabic characters. The model achieved better results when evaluated using generated images in addition to the originals. Mustapha et al. [44] proposed a deep convolutional generative adversarial network (CDCGAN) model to generate isolated handwritten Arabic characters; the model was trained and tested using a CNN model (LeNet5), showing a gap of 10% between the accuracy obtained while using the generated set and the real set. Jemni et al. [45] presented a GAN-based model to enhance the process of the binarization of degraded handwritten documents; the model was tested on the KHATT and IAM datasets. The results showed that the model achieved the best results among all the baseline models for Arabic and English script.

The works in AHR, as presented in this section, demonstrate the significant strides that have been made in this field. The use of Generative Adversarial Networks as a data augmentation technique has notably enhanced the performance and robustness of various systems. However, despite these advancements, there remain challenges and opportunities for further research and development.

Materials and Methods

In this study, we propose a framework aimed at enhancing the accuracy of Arabic handwritten recognition. By integrating the power of deep learning and GANs data augmentation, our approach is grounded on the robust architecture of Convolutional Neural Networks (CNNs), Bidirectional Long Short-Term Memory (BLSTMs), and Connectionist Temporal Classification (CTC) for the purpose of feature extraction and sequence decoding. To tackle the challenges of data scarcity and diversity, we employ GANs to generate synthetic data that closely resembles real Arabic handwriting. An overview of the developed model is provided in Fig. 1.

Fig. 1. Model architecture.

GANs

Generative Adversarial Networks is an innovative type of neural network architecture first proposed by Goodfellow et al. [46]. The basic GAN framework consists of two competing neural networks-a generator model and a discriminator model. The generator’s role is to synthesize new data instances that mimic real data, while the discriminator aims to differentiate between real data examples and those created by the generator. The two networks are pitted against each other in an adversarial game where the generator tries to better fool the discriminator over time. Through this competitive process, both models improve-the generator gets better at creating realistic synthetic data, and the discriminator gets better at detecting generated data.

The generator output is connected directly to the discriminator input. Through backpropagation, the discriminator’s classification provides a signal that the generator uses to update its weights. The generator model is trained using the loss signals received from the discriminator’s classifications to iteratively enhance its outputs (Fig. 2).

Fig. 2. GANs architecture.

GANs have rapidly gained popularity in recent years due to their ability to produce high-quality synthetic data across various problem domains. They have proven especially adept at image-to-image translation tasks, such as turning semantic layouts into photo-realistic scenes or sketches into rendered faces [46]. In [47] incorporated recurrent neural network recognizer into a GAN to control the generated output. Jha et al. [48] introduced a GAN structure that is grounded on the Convolutional Neural Network, the sigmoid function was employed at the generator’s output layer, and binary cross-entropy was selected as the loss function. Fogel et al. [49] presented ScrabbleGAN which generates each character independently using CNNs. This accounts for influence of neighboring letters. ScrabbleGAN enhancements produce more realistic images compared to previous GAN models. Our GANs layer follows paradigm [49], an approach to synthesize handwritten text images that are versatile both in style and lexicon. ScrabbleGAN relies on a novel generative model which can generate images of words with an arbitrary length.

Baseline Model

Our baseline model is a fusion of three critical elements. Firstly, it employs a Convolutional Neural Network to serve as a feature extractor, pulling out significant features from a text image. Secondly, it uses recurrent layers that leverage Bidimensional Long Short-Term Memory to predict a pre-frame from an input sequence. Lastly, it incorporates a transcription layer that integrates Connectionist Temporal Classification (CTC). This combination of components enables the model to interpret and analyze the input data effectively (Fig. 3) [50]–[52].

Fig. 3. Baseline model architecture [52].

CNN

The Convolutional Neural Network is typically employed for image classification tasks. It comprises several layers that are adept at learning high-level features from labeled training data. The fundamental concept of CNN is the utilization of convolutional and pooling layers to progressively extract more abstract patterns that can be executed locally. The convolution layer applies a set of convolution filters to the input images, with each filter extracting a specific feature from the image. The pooling layer simplifies the output through nonlinear down-sampling, thereby reducing the number of parameters that the network needs to learn. This operation enables the identification of different characteristics at each layer.

As depicted in Fig. 4, the architecture embodies a Convolutional Neural Network designed for handwritten recognition, drawing inspiration from LeNet-5 [53]. It comprises several layers. Initially, the input is processed by a convolutional layer that convolves it with a set of learnable filters or weights, each generating one feature map. Following this, a pooling layer (sub-sampling) is employed to gradually reduce the spatial size of the feature map by averaging the features in the neighborhood or pooling for a maximum value, thereby reducing the number of parameters and computation in the network. Each convolution layer is followed by a sub-sampling layer. The alternating sequence of convolutional and pooling layers forms the feature extractor, which extracts distinguishing features from the raw images. Fully connected layers are utilized at the end of the network for high-level reasoning after the feature extraction and consolidation have been carried out by the convolutional and pooling layers. Our CNN architecture, derived from LeNet-5 with modifications, is further detailed in [52]–[54].

Fig. 4. CNN architecture for proposed CNN for offline Arabic handwriting recognition [54].

BLSTM

The Bi-directional Long Short-Term Memory architecture, introduced by Graves et al. [55], was designed to address the limitations of LSTM, particularly its reliance on processing information based solely on previous context. This is a significant drawback of both RNN and LSTM.

BLSTM has found favor in numerous applications, including speech recognition, due to its ability to be trained using all available input information from both the past and future within a specific time frame. The structure of BLSTM includes two separate hidden layers. One layer processes the input sequence in a forward direction and the second in a backward direction. These two hidden layers are connected to the same output layer. This connection provides the output layer with access to the past and future context for every point in the sequence. As a result, BLSTM outperforms unidirectional LSTMs and standard RNNs in terms of speed and accuracy. The architecture of a BLSTM network is illustrated in Fig. 5.

Fig. 5. The architecture of a bi-directional recurrent neural network.

CTC-WBS

Connectionist Temporal Classification (CTC) plays a pivotal role in enabling recognition without the need for prior segmentation. CTC was initially conceived for speech recognition and later extended to handwriting recognition. The CTC layer transcribes data and predicts transcriptions for test images during the recognition phase, which are then compared with the ground truth of the line image using the Levenshtein edit distance. CTC interprets the network output as a distributed probability over all potential label sequences for a given input sequence, facilitating a direct alignment between input variables and the target label. The final layer of the network, with N outputs where N represents the total number of labels, indicates the probability of each label being observed at a given time. This approach allows for the generation of a set of probabilities that potentially match outputs to a given input.

The Word Beam Search (WBS) decoding algorithm is utilized to decode the output matrix of the Connectionist Temporal Classification layer, positioned immediately after the CTC layers for output decoding. The WBS decoder offers several advantages over the token passing decoder, including faster processing speed, the ability to allow an arbitrary number of non-word characters between words such as numbers and punctuation marks, and the constraint of words by a dictionary.

Results and Discussion

The following subsections explore the influence of Generative Adversarial Networks (GANs) as a data augmentation method for Arabic handwriting recognition, specifically on the IFN-ENIT dataset. We conduct a thorough comparison of our system’s performance under diverse conditions, thereby shedding light on the nuanced impact of GAN-generated data on accuracy and robustness.

Dataset

In the field of handwritten Arabic text recognition research, the IFN/ENIT dataset, published by [56], is the most commonly utilized and recognized dataset. It comprises over 32,492 handwritten Arabic names of Tunisian cities, with the lexicon encompassing 937 names. For each word in the dataset, a Ground Truth (GT) file has been assembled, providing information about the word, including its baseline position and the specific characters employed.

The training process commences with the generator receiving a set of random seeds and transcriptions from a randomly selected word list. We selected random words from the dataset along with their transcriptions to generate a set of synthetic images. By altering the noise vector input into the network, we were able to generate various handwriting styles. Fig. 6 presents examples of synthesized words from the IFN/ENIT dataset showcasing the diversity of generated handwriting styles. For each style corresponding to the words in the lexicon, we produced images and consequently generated a total of 65,590 samples to train our baseline deep learning recognition network.

Fig. 6. GANs generated images.

Results

To assess the outcomes, our model underwent three primary phases on IFN-ENIT database: initial training on the original data, application of traditional data augmentation methods in the second phase, and utilization of expanded dataset via GANs in the final phase. This process allowed us to compare the performance and draw conclusions to confirm the effectiveness of our approach. Table I presents the results.

Table I. Results Under Three Primary Phases

Model	Data augmentation	RR (%)
	None [52]	91.69
CNN-BLSTM/CTC	Traditional [52]	94.58
	GANs	95.23

The comparative results reveal the impact of data augmentation techniques on the performance of the Arabic handwriting recognition system. The system’s performance without data augmentation serves as a baseline for comparison. The results are measured in terms of recognition rate (RR), which is the percentage of words that were correctly recognized. The GANS model achieved the highest RR of 95.23%. This is an improvement of 0.65% over the traditional model and 3.54% over the baseline model. The GANs model was able to generate more realistic training data, which led to better generalization on the test data.

To ascertain the impact of data augmentation using the previously described method, we executed a series of classification evaluation experiments. The effectiveness of GAN’s data augmentation and its fundamental principle were evaluated by integrating the generated data with the original training data. The classification process encompassed five variants, each one explored a different combination of training and testing data. Table II summarize the comparative results.

Table II. Comparative Results Under Different Scenarios

Variant	Training set	Test set	RR (%)
–	Original	Original	91.96
1	Original	Generated	89.12
2	Generated	Generated	92.54
3	Generated	Original	87.17
4	Original + Generated	Original	95.23
5	Original + Generated × 2	Original	94.55

The synthesis of the results offers a holistic understanding of the advantages and constraints of employing Generative Adversarial Networks for data augmentation in the context of the Arabic Handwriting Recognition System. It also underscores potential areas for enhancement in the system’s approach and modeling.

The significant improvement in the model’s accuracy on real IFN-ENIT images when combining original and GAN-generated data (scenario 4) validates the efficacy of GANs in mitigating data scarcity and overfitting. This implies an enhancement in recognition capabilities for real-world scenarios. The experiments-based training on real data and testing on GAN-generated images (scenario 1), demonstrate that GANs can generate images that are accurately classified by the model. However, this similarity also introduces the possibility of mistaking GAN-generated images for real ones in certain contexts. Similarly, Performance Dip when Testing GAN-trained Models on Real Data (scenario 3), the decrease in performance when testing GAN-trained models on real data exposed the inevitable differences between generated and real distributions. This implies that the prototype images produced by the GANs may not be adequate for training a robust classifier, which is expected to handle the variability present in the real images of the test dataset. Training and Testing Solely on GAN-generated Images (scenario 2), this experiment showcased the model’s capacity to learn the underlying patterns captured by the GAN, and how GAN images can be used for training a classifier. After adding GAN-generated data to the original training set, the classifier’s performance improved. However, when we added more GAN data (scenario 5), the accuracy decreased slightly. This shows a divergence in the model’s effectiveness after a certain saturation point. The optimal distribution between GAN-generated and real data can be ascertained through cross-validation.

This comprehensive analysis provides valuable insights into the potential of GANs data augmentation in the field of Arabic handwriting recognition and suggests areas for future research and development. GANs offer a powerful tool for Arabic handwritten recognition, expanding training data and boosting model performance. Future research should focus on approach to unlock the full potential of GAN-based data augmentation.

Conclusion and Perspectives

The research presented in this paper has demonstrated a significant advancement in the field of Arabic handwritten recognition. By integrating Generative Adversarial Networks for data augmentation within a robust CNN-BLSTM architecture. The proposed approach addresses the challenges of AHR, including the complexity of Arabic script and the limited availability of training data. The use of GANs for data augmentation has proven effective in generating realistic and diverse synthetic samples, thereby augmenting the original training corpus. The CNN-BLSTM network, coupled with connectionist temporal classification for accurate sequence modeling and recognition, has demonstrated its efficacy in improving AHR performance. The results of this research pave the way for more robust and accurate AHR systems. While the results of this study are promising, there are several potential avenues for future research. The GAN-based data augmentation technique could be further refined to generate even more realistic and diverse synthetic samples. Additionally, the CNN-BLSTM architecture could be optimized to further improve sequence modeling and recognition. The research also opens up the possibility of applying similar techniques to other languages and scripts that present similar challenges in terms of complexity and limited availability of training data. The integration of GANs for data augmentation within robust deep learning architectures could potentially revolutionize the field of handwritten recognition. Finally, the research underscores the broader implications of this study for the AHR domain. By addressing data limitations in AHR and contributing to the development of more robust and accurate recognition systems, this research presents a promising approach to data augmentation in AHR.

Looking ahead, the application of Natural Language Processing (NLP) techniques presents a promising direction for further improving AHR systems. NLP can contribute to the refinement of AHR in several ways:

• Semantic Understanding: NLP can help in understanding the context and meaning behind the handwritten text, which can be particularly useful for recognizing words that are visually similar but have different meanings.
• Language Modeling: Incorporating NLP-based language models can improve the prediction of word sequences, thereby enhancing the overall recognition process.
• Error Correction: Post-recognition, NLP can be employed to correct errors that are common in handwritten texts, such as misspellings or grammatical mistakes, by using context-aware algorithms.

By exploring these NLP techniques, future research can lead to the development of more sophisticated AHR systems that not only recognize the text with high accuracy but also understand and process the content effectively.