Adaptive Anomaly Detection in Database Transactions: Bridging Security Gaps with Reinforcement Learning

Anomaly detection in database transactions is critical for safeguarding sensitive information and ensuring the integrity of operations in industries like finance, healthcare, and e-commerce. Existing techniques, including rule-based, machine learning, and deep learning methods, face challenges such as high false positive rates, poor adaptability to evolving patterns, and limited scalability in imbalanced datasets. This research proposes a novel Reinforcement Learning (RL)-based anomaly detection system to address these limitations. The model employs a dynamic reward mechanism and anomaly scoring system to classify transactions accurately while reducing false positives. It leverages the Kaggle Anomaly Detection in Transactions Dataset and a synthetically generated dataset for training and evaluation. Experimental results show that the RL-based model outperforms traditional methods, achieving a precision of 95.2%, recall of 92.4%, and an AUC-ROC score of 97.2%, significantly higher than Autoencoders, Isolation Forest, and Support Vector Machines. The proposed model’s adaptability and robustness make it a scalable solution for real-time anomaly detection, addressing critical gaps in existing techniques. This study advances database security by offering a highly accurate, adaptive, and efficient system for detecting anomalies in complex transactional environments.

Introduction

Background

Database systems form the backbone of critical industries such as banking, healthcare, and e-commerce, where secure, reliable, and efficient data handling is paramount. Database transactions ensure data integrity, consistency, and availability through the properties of ACID (Atomicity, Consistency, Isolation, and Durability). Transactions facilitate essential operations like processing customer payments, maintaining patient health records, and managing inventory systems. However, as these systems become increasingly interconnected and operate in real-time, they are also becoming prime targets for security breaches, fraud, and data corruption [1], [2]. Security incidents such as unauthorized access, SQL injection attacks, and insider threats can compromise sensitive data, resulting in significant financial and reputational losses. The global financial industry alone has faced billions of dollars in fraud-related damages, emphasizing the need for robust detection mechanisms [3], [4]. Though essential, traditional preventive methods like access controls, encryption, and firewalls often do not detect sophisticated and evolving anomalies [5], [6]. As modern applications generate massive volumes of dynamic transactional data, traditional methods struggle to maintain efficacy, leaving systems vulnerable [7], [8]. Traditional anomaly detection approaches often rely on predefined rules or statistical thresholds to flag unusual activity [9], [10]. While these methods are computationally efficient, they exhibit significant limitations:

• High False Positives: Rigid thresholds fail to adapt to legitimate yet rare transaction patterns, leading to false alarms [8], [11].

• Scalability Issues: With the exponential growth in transaction volumes, traditional techniques become computationally prohibitive [12], [13].

• Evolving Attack Patterns: Cybercriminals continuously adapt their techniques, rendering static detection systems ineffective against novel threats [1], [14]. Furthermore, anomaly detection in database transactions is challenging due to the interdependencies between transactions. A series of benign transactions may collectively exhibit anomalous behavior, which simple rule-based systems cannot detect [2], [15]. Advanced machine learning techniques like deep learning have shown promise but often require extensive labelled data, which is scarce in real-world scenarios. Additionally, these models lack adaptability to new transaction patterns, making them suboptimal for evolving systems [16], [17].

This landscape necessitates the development of adaptive, scalable, and intelligent solutions that can evolve alongside the dynamic nature of database transactions. RL is a paradigm that learns optimal actions by interacting with an environment and offers a promising alternative for detecting anomalies by continually adapting to new data and optimizing detection strategies [9], [10]. This research addresses the abovementioned challenges by leveraging RL to develop an advanced anomaly detection system for database transactions. Specifically, the objectives are:

• Developing an RL-Based Model: To detect unusual patterns in database transactions that may indicate security breaches or data corruption [10], [18].

• Reducing False Positives: Implementing a dynamic reward mechanism to optimize anomaly classification, thereby minimizing false alarms [1], [8].

• Adapting to Dynamic Behaviors: Designing the system to evolve with changing transaction patterns, ensuring sustained effectiveness [19], [20].

• Handling Imbalanced Datasets: Employing techniques to address the skewed distribution of normal and anomalous transactions, a common challenge in real-world datasets [4], [8].

The primary contributions of this research are as follows: the study introduces the innovative application of RL for anomaly detection in database transactions, providing a self-adaptive and scalable solution to address evolving threats [1], [21]. A dynamic anomaly scoring mechanism is integrated into the RL agent, enhancing detection accuracy while minimizing false positives and ensuring more reliable classifications [9], [13]. The model is rigorously evaluated using a combination of the publicly available Kaggle Anomaly Detection in Transactions Dataset [22] and synthetically generated datasets replicating real-world transaction scenarios. This dual-dataset strategy ensures robust and comprehensive performance assessment [12], [23]. Experimental results reveal significant performance improvements over traditional and machine learning-based methods, particularly in key metrics such as precision, recall, and F1-score [16], [18]. This study advances state-of-the-art anomaly detection for database transactions and provides a scalable framework applicable to various industries, including finance, healthcare, and e-commerce [3], [24], [25]. The results highlight the potential of RL to redefine how database systems detect and respond to security threats.

Related Work

Traditional anomaly detection methods rely heavily on rule-based and statistical techniques to flag suspicious transactions. These approaches define static thresholds or rules to identify deviations from normal behavior. Statistical models like Gaussian distributions and Principal Component Analysis (PCA) have been widely used to monitor transaction anomalies [5], [9]. While effective in some scenarios, these techniques often suffer from rigidity and high false favourable rates, as they fail to adapt to evolving transaction patterns [2], [3]. Rule-based systems also require manual configuration and frequent updates, which limit their scalability and applicability in dynamic environments such as modern databases [15], [16]. Machine learning (ML) techniques introduced automation and adaptability to anomaly detection. Supervised models, such as Random Forests, Support Vector Machines (SVMs), and k-Nearest Neighbors (kNN), leverage labeled datasets to classify normal and anomalous transactions [6], [20]. Unsupervised methods, including Isolation Forest and k-means clustering, eliminate the need for labelled data and focus on finding patterns that deviate from the norm [18], [19]. Despite their advantages, ML models face challenges when applied to large-scale, imbalanced datasets typical of database transactions. Supervised models require extensive labelled data, often scarce in anomaly detection. On the other hand, unsupervised methods frequently produce false positives due to the complexity of real-world transaction patterns [4], [11].

Deep learning models, including Autoencoders and Long Short-Term Memory (LSTM) networks, have demonstrated superior performance in modelling complex data patterns. Autoencoders are widely used to reconstruct normal transaction behaviours and identify deviations as anomalies [12], [21]. LSTMs, known for capturing sequential dependencies, effectively model time-series data such as transaction logs [1], [23]. However, these methods are computationally intensive and often lack interpretability, making them challenging to deploy in real-time systems. Additionally, their reliance on large, labelled datasets and sensitivity to hyper-parameter tuning limit their practicality in dynamic environments like database systems [14], [25]. RL has emerged as a powerful tool in security applications due to its adaptive learning capabilities. RL models learn optimal policies by interacting with an environment, making them suitable for scenarios where transaction patterns evolve [8], [10]. Existing RL applications in cybersecurity have focused on areas such as intrusion detection and fraud prevention [13], [17]. Despite its promise, RL has seen limited application in database transaction anomaly detection. Challenges include designing appropriate state representations, defining practical reward functions, and handling the complexity of multi-agent environments [1], [2]. Furthermore, RL models in security often require extensive computational resources and lack explainability, which hinders their widespread adoption [15], [16]. The study summary of the existing literature is shown in Table I.

Table I. Consolidated Study on the Existing Literature

Technique	Advantages	Challenges	References
Rule-based methods	Simple and interpretable	High false positive rates; rigidity; poor scalability	[2], [5], [9]
Statistical models	Efficient in stable environments	Ineffective in dynamic and evolving transaction patterns	[3], [15], [16]
Supervised ML models	High accuracy with sufficient labelled data	Reliance on labelled data; limited scalability; sensitivity to imbalanced datasets	[4], [6], [20]
Unsupervised ML models	No need for labelled data	High false positives; poor adaptability to complex patterns	[11], [18], [19]
Autoencoders	Effective in detecting complex anomalies	Computationally expensive; lack of interpretability	[12], [21], [25]
LSTMs	Captures sequential dependencies	Requires large datasets; difficult to tune; limited real-time applicability	[1], [14], [23]
Reinforcement learning	Adaptive to evolving patterns; dynamic policies	Requires extensive computational resources; challenges in reward function design and explainability	[10], [13], [17]

Existing research on anomaly detection in database transactions faces several critical gaps. Imbalanced data remains a significant challenge, as most datasets are heavily skewed, with rare anomalies leading to biased and ineffective models [4], [11]. Many traditional and machine learning-based methods exhibit low precision, resulting in high false positive rates undermining their reliability [1], [18]. The absence of dynamic and context-aware anomaly scoring mechanisms further limits classification accuracy [2], [9]. Poorly designed reward functions in reinforcement learning often lead to suboptimal policies, reducing the model’s effectiveness [8], [10]. Additionally, many approaches struggle with adaptability to real-world scenarios, as they fail to accommodate evolving transaction behaviours, restricting their practical applicability [6], [14].

Proposed Work

The proposed work introduces a Reinforcement Learning (RL)-based anomaly detection system tailored for database transactions. Unlike traditional methods, this model learns optimal policies through interaction with transactional data, enabling it to adapt to evolving transaction patterns dynamically. The RL agent operates by observing a series of database transactions, analyzing their features, and assigning anomaly scores based on predefined reward mechanisms, as depicted in Fig. 1. The system emphasizes reducing false positives and improving detection accuracy, especially in environments with imbalanced data. The high-level workflow involves four primary stages:

Fig. 1. Architecture of the proposed system.

1. Data Collection and Pre-processing: Preparing raw transaction data for analysis.

2. RL Agent Training: Training the agent to detect anomalies using state-action-reward feedback loops.

3. Dynamic Anomaly Scoring: Generating scores to classify transactions as normal or anomalous.

4. Evaluation and Feedback: Testing the model on real-world and synthetic datasets to refine its performance.

The system is designed to address challenges in scalability, adaptability, and imbalanced datasets, offering a robust solution for detecting anomalies in database transactions.

Model Architecture

The RL-based model consists of an agent interacting with an environment (the database transactions) and learning a policy to maximize rewards, as shown in Fig. 2. The policy network is implemented using deep neural networks, ensuring the model can handle high-dimensional transaction data effectively.

Fig. 2. Reinforcement learning-based anomaly detection model.

The proposed model leverages diverse input features, including transaction metadata (timestamp, transaction ID, user ID), attributes (amount, volume, account type), and derived metrics (transaction frequency, historical trends), to capture comprehensive transactional behavior. Its core is a policy network structured as a multi-layer perceptron (MLP) with input, hidden, and output layers. The input layer normalizes transaction features, while hidden layers use ReLU activation to identify complex patterns. The output layer generates probabilities for anomaly classification, enabling precise detection. This design ensures adaptability and accuracy in distinguishing normal transactions from anomalies, addressing the complexities of real-world database systems.

In the proposed model, a state represents the current context of a transaction, encompassing its features and recent transactional history. This includes details such as the transaction amount, user activity patterns, and the time of day, providing a comprehensive snapshot of the transaction environment. The RL agent’s action is chosen from a discrete set, where it either classifies the transaction as normal or anomalous. A reward function that encourages accurate classifications guides the agent’s decisions. Positive rewards are granted for correct classifications, while negative rewards are applied for misclassifications, with higher penalties for false positives to minimize disruption caused by false alarms. Additionally, the reward mechanism dynamically adjusts based on the rarity of anomalies, ensuring the agent prioritizes detecting rare, high-risk events. This adaptive approach enables the RL agent to effectively identify anomalies in complex transactional environments.

Dynamic Anomaly Scoring

The proposed model integrates a dynamic anomaly scoring mechanism to enhance adaptability and precision in detecting anomalies. This mechanism begins with transaction embedding, where each transaction is encoded as a feature vector using the policy network, capturing essential characteristics and patterns. The next step is score calculation, where the anomaly score is derived from the RL agent’s action probabilities. For instance, the anomaly score is calculated as

$$\begin{array}{r}AnomalyScore=1-P\left(Normal|Features\right)\end{array}$$

where $P(Normal|Features)$ is the probability assigned to the transaction being normal.

This probability-based approach ensures a nuanced evaluation of transactions. The model further employs threshold adjustment, dynamically modifying thresholds based on the transaction context, such as user profiles or peak activity periods, to enhance flexibility and accuracy. Finally, a feedback loop refines the anomaly scores as the model learns from its environment, enabling improved differentiation between normal and anomalous transactions over time. By moving away from static thresholds, this mechanism significantly reduces false positives and negatives while adapting seamlessly to evolving transaction patterns. This dynamic approach ensures the model remains robust and effective in diverse and complex transactional environments.

Experimentation and Result Analysis

Experimental Setup

The proposed RL-based anomaly detection model uses Python with frameworks such as TensorFlow and PyTorch for deep learning and Scikit-learn for preprocessing and evaluation. The experiments are conducted on a machine equipped with an Intel Core i7 processor, 16GB RAM, and an NVIDIA RTX 3060 GPU, running Ubuntu 20.04. The Kaggle Anomaly Detection in Transactions [22] Dataset and synthetic datasets undergo extensive pre-processing. Missing values are handled using mean or mode imputation, while categorical variables are encoded using one-hot encoding. Numerical features are normalized to a range of [0, 1] to ensure uniform scaling. Feature extraction includes deriving temporal and statistical features like transaction frequency and deviations from user norms to enhance model input. The dataset is split into training, validation, and test sets in a 70:20:10 ratio, ensuring that the class imbalance is preserved during training.

The model’s performance is benchmarked against widely used anomaly detection techniques. The proposed study compares the performance of the RL-based anomaly detection model against several baseline techniques. Autoencoders, unsupervised deep learning models, identify anomalies by reconstructing transaction patterns and detecting deviations through reconstruction errors. Isolation Forest, another unsupervised method, isolates anomalies by randomly partitioning the data, making it practical for identifying outliers. Additionally, traditional machine learning models such as SVM and Random Forest are utilized as supervised classifiers, trained using undersampled and SMOTE-balanced datasets to address class imbalance. These models serve as benchmarks to evaluate the RL-based approach regarding key metrics like precision, recall, F1-score, and AUC-ROC.

Evaluation Metrics and Scenarios

The evaluation of the proposed RL-based anomaly detection model is conducted using the following metrics:

• Precision: Measures the proportion of correctly identified anomalies out of all flagged anomalies, indicating the model’s ability to avoid false positives.

• Recall: Assesses the model’s ability to correctly identify all actual anomalies, addressing sensitivity to rare events.

• F1-Score: The harmonic mean of precision and recall, balancing both metrics for a comprehensive performance measure.

• Accuracy: Measures the overall correctness of predictions but can be misleading in imbalanced datasets.

• AUC-ROC Curve: Evaluates the trade-off between true positive and false positive rates across different thresholds, providing a robust measure of classification effectiveness.

The RL-based model is benchmarked against Autoencoders, Isolation Forest, and traditional machine learning models (e.g., SVM, Random Forest). The comparison focuses on precision, recall, and F1-score to highlight the proposed model’s ability to outperform these techniques in detecting anomalies. The model’s performance is tested on datasets with severe class imbalance, such as the Kaggle Anomaly Detection in Transactions Dataset. Techniques like Synthetic Minority Oversampling (SMOTE) and dynamic reward adjustments are evaluated to handle imbalance effectively. Metrics like the F1-score and AUC-ROC are emphasized to ensure robust detection of rare anomalies while minimizing false positives.

Result Analysis

This section illustrates the significant performance improvement of the proposed RL-based model compared to traditional and machine learning-based anomaly detection techniques. Table II compares the proposed system’s performance with baseline models, and the corresponding discussion is in the following sections. Fig. 3 depicts the performance comparison of different techniques grouped by factors.

Table II. Assessment Table for the Proposed RL-Based Model and Baseline Methods

Model	Precision (%)	Recall (%)	F1-Score (%)	Accuracy (%)	AUC-ROC (%)
Proposed RL-based model	95.2	92.4	93.8	94.7	97.2
Autoencoder	87.6	85.3	86.4	88.9	91.8
Isolation forest	84.5	81.7	83.1	85.6	89.5
SVM	78.9	76.4	77.6	80.3	83.6
Random forest	82.3	79.8	81.0	83.7	87.4

Fig. 3. Performance comparison of different techniques grouped by factors.

Precision

Precision measures the proportion of correctly identified anomalies among all flagged anomalies. The Proposed RL-Based Model achieves a precision of 95.2%, outperforming all baseline methods. This improvement can be attributed to the dynamic reward mechanism in the RL framework, which penalizes false positives, ensuring the model focuses on accurately flagging anomalies. In contrast, Autoencoders (87.6%) and Isolation Forest (84.5%) show reduced precision due to their reliance on reconstruction errors and statistical thresholds, which are less effective at differentiating subtle anomalies. SVM (78.9%) and Random Forest (82.3%) perform even worse, primarily due to their sensitivity to imbalanced datasets. The RL model’s ability to adapt its decision-making process dynamically, even in the presence of noise and outliers, demonstrates its superiority. High precision ensures fewer false alarms, which is crucial for maintaining operational efficiency and reducing the workload on human analysts.

Recall

Recall evaluates the model’s ability to detect actual anomalies. The Proposed RL-based model achieves a recall of 92.4%, significantly higher than the baselines. Autoencoders (85.3%) and Isolation Forest (81.7%) struggle to identify rare anomalies effectively due to their dependence on predefined patterns and static thresholds. The RL model excels in recall by leveraging a feedback-driven learning mechanism, allowing it to explore rare anomaly patterns during training. The lower recall of SVM (76.4%) and Random Forest (79.8%) indicates their limited capability in handling the complexity of database transactions. The proposed RL model’s high recall ensures robust anomaly detection, particularly in scenarios where missing even a single anomaly can have catastrophic consequences, such as financial fraud or cybersecurity breaches.

F1-Score

The F1-score, the harmonic mean of precision and recall, balances these two metrics’ trade-offs. The Proposed RL-based model achieves an F1-score of 93.8%, outperforming all other methods. Autoencoders (86.4%) and Isolation Forest (83.1%) deliver lower F1 scores due to their inconsistent performance in either precision or recall. SVM (77.6%) and Random Forest (81.0%) suffer significantly from their inability to handle imbalanced datasets effectively. The RL model’s dynamic learning process and anomaly scoring mechanism enable it to achieve this high F1 score, proving its ability to detect anomalies while minimizing false positives and negatives. This balance makes the RL model highly reliable for real-world applications requiring robust performance across diverse scenarios.

Accuracy

The accuracy of the Proposed RL-based model is 94.7%, demonstrating its capability to classify transactions correctly across normal and anomalous classes. Accuracy for Autoencoders (88.9%) and Isolation Forest (85.6%) is relatively lower, primarily due to their higher false positive and false negative rates. SVM (80.3%) and Random Forest (83.7%) lag significantly, indicating their inefficiency in dealing with high-dimensional and complex transaction data. While accuracy is general, the RL model’s dynamic adaptation ensures a higher proportion of correct classifications, even in highly imbalanced datasets.

AUC-ROC

The AUC-ROC curve measures the model’s ability to distinguish between normal and anomalous transactions across varying thresholds. The Proposed RL-based model achieves the highest AUC-ROC score of 97.2%, indicating its exceptional performance in separating the two classes. Autoencoders (91.8%) and Isolation Forest (89.5%) deliver lower scores, reflecting their reliance on static patterns, which limits their ability to adapt to evolving anomalies. SVM (83.6%) and Random Forest (87.4%) further highlight traditional models’ challenges in achieving robustness across diverse transaction scenarios. The RL model’s AUC-ROC score underscores its versatility and robustness, making it well-suited for real-world applications that demand high-class discrimination.

Discussion

The overall performance of the proposed RL-based model demonstrates its superiority in anomaly detection for database transactions, significantly outperforming baseline methods across all evaluation metrics. The model achieves precision (95.2%), recall (92.4%), and an F1-score (93.8%), highlighting its ability to detect anomalies while minimizing false positives and negatives accurately. This is particularly crucial in real-world scenarios, where false alarms can strain resources, and undetected anomalies can lead to security breaches or data corruption. The accuracy (94.7%) further underscores the model’s robustness in classifying transactions correctly, even in highly imbalanced datasets. The RL-based dynamic anomaly scoring mechanism enables the model to adapt to evolving transaction patterns, a critical advantage over traditional machine learning models like SVM (80.3% accuracy) and Random Forest (83.7% accuracy). The RL model’s highest AUC-ROC score (97.2%) highlights its exceptional capability to distinguish between normal and anomalous transactions across varied thresholds. The discussion reveals that the RL-based model’s ability to leverage feedback-driven learning, handle high-dimensional data, and dynamically adjust anomaly thresholds ensures superior performance. These capabilities position the proposed model as a reliable and scalable solution for anomaly detection in database transactions, addressing critical gaps in existing methods and advancing database security.

Conclusion

The proposed Reinforcement Learning-based anomaly detection system effectively addresses the challenges of detecting anomalies in database transactions, including high false positives, imbalanced datasets, and evolving attack patterns. By integrating dynamic anomaly scoring and adaptive learning mechanisms, the model achieves superior performance across all evaluation metrics, including a precision of 95.2% and an AUC-ROC of 97.2%, outperforming traditional and machine learning-based approaches. Its ability to dynamically adapt to changing transaction behaviors ensures scalability and reliability in real-world applications. This research highlights the potential of reinforcement learning in advancing database security by providing a robust, real-time, and efficient anomaly detection framework. Future work may focus on integrating explainable AI techniques for greater transparency and extending the model to handle multi-agent environments and diverse datasets, further enhancing its applicability across industries.