FakeReveal: Multi-modal_Audio-Visual_feature-based dataset
Description
This dataset is a multimodal, feature-based dataset developed for an audio-visual deepfake detection system. It provides numerical feature representations for 7,602 samples, including 3,396 real and 4,206 fake samples. The AVLips dataset was used as the source of the video samples from which this feature-based dataset was derived. Its video samples were computationally processed through a multimodal pipeline combining visual speech and audio speech recognition. The audio was processed using Whisper (base) to generate an audio speech transcription. In parallel, the mouth region was extracted using MediaPipe Face Mesh and processed using the pre-trained AV-HuBERT base_vox_433h model to generate a visual speech transcription based on lip movements. These two textual representations were then compared to capture inconsistencies between the spoken audio and the lip movements. Three cross-modal textual comparison features were extracted from the Whisper and AV-HuBERT outputs. Before comparison, both transcripts were normalised by expanding contractions, converting text to lowercase, removing punctuation, and standardising whitespace. The obtained features include Word Error Rate (WER), text similarity score, and correct word count. Together, these features provide complementary measurements of word-level error, overall textual similarity, and exact lexical agreement between audio and visual speech. In a parallel audio-based pipeline, MFCC and spectral features were extracted using the Librosa library. The resulting 50-dimensional acoustic representation consists of 40 Mel-Frequency Cepstral Coefficients (MFCCs), 1 Spectral Centroid, 1 Spectral Roll-off, 1 Spectral Bandwidth, and 7 Spectral Contrast features. A linear Support Vector Machine (SVM) classifier was then applied to the extracted MFCC and spectral features. Three model-derived outputs were retained for each sample: the prediction label, the prediction probability and the decision score. The dataset contains two complementary files: FakeReveal_Master_Features.csv, which provides the consolidated feature table and FakeReveal_MFCC_Spectral_Features.pkl, which preserves the 50-dimensional MFCC and spectral feature vectors. The FakeReveal_Master_Features.csv contains the following columns: Video_ID: Identifier. label: class: label, where 0 represents real and 1 represents fake. similarity: textual similarity between the audio transcription and the visual speech transcription. WER: measuring the word-level discrepancy between the audio and visual speech transcriptions. correct_words_#: Number of words correctly matched between the audio and visual speech transcriptions. mfcc_prediction: Binary prediction produced by the MFCC-based SVM classifier mfcc_probability: Prediction probability produced by the MFCC-based SVM classifier. mfcc_svm_decision_score: decision function score representing the sample's position relative to the classifier's decision boundary.
Files
Steps to reproduce
The dataset was derived from video samples obtained from the AVLips dataset. No new video recordings were collected for this dataset. The original video samples were computationally processed to generate numerical audio, visual speech representations, and cross-modal feature representations. The processing workflow consisted of the following steps: 1. Source Video Samples: Video samples were obtained from the AVLips dataset. The samples were identified where 0_xxxxx.mp4 represents real videos (label = 0) and 1_xxxxx.mp4 represents fake videos (label = 1). 2. Audio Speech Processing: The audio stream was extracted from each video and processed using Whisper (base) to generate an audio speech transcription. 3. Visual Speech Processing: The mouth region was extracted from each video using MediaPipe Face Mesh. The extracted mouth-region sequences were then processed using the pre-trained AV-HuBERT base_vox_433h model to generate a visual speech transcription based on lip movements. 4. Cross-Modal Comparison: The audio and visual speech transcriptions were normalized before comparison by expanding contractions, converting text to lowercase, removing punctuation, and standardizing whitespace. Three features were then calculated: Word Error Rate (WER) using the werpy library, text similarity using Python's SequenceMatcher from the difflib library, and correct word count using dynamic-programming-based word sequence alignment. 5. Audio Feature Extraction: In parallel, acoustic features were extracted from the audio using the Librosa library. A 50-dimensional feature vector was generated for each sample, consisting of 40 MFCCs, 1 spectral centroid, 1 spectral roll-off, 1 spectral bandwidth, and 7 spectral contrast features. The mean value of each feature was calculated to obtain fixed-length representations. 6. SVM-Based Processing: The extracted acoustic features were normalized using StandardScaler and used with a linear Support Vector Machine (SVM) classifier. The resulting prediction label, prediction probability, and SVM decision score were retained as additional features. 7. Final Dataset Construction: The extracted features were consolidated into FakeReveal_Master_Features.csv, while the 50-dimensional MFCC and spectral feature vectors were preserved in FakeReveal_MFCC_Spectral_Features.pkl. The resulting dataset consists entirely of computationally derived numerical feature representations rather than the original video files. The workflow can be reproduced by obtaining the corresponding AVLips video samples and applying the described audio representation extraction, speech recognition, visual speech processing, feature extraction, normalization, and classification.
Institutions
- University of JeddahMecca Region, Jeddah