A Multimodal Indian Sign Language Dataset for English and Marathi Alphabets Featuring Pose and Skin Segmentation Representations
Description
This paper presents a multimodal dataset for static Indian Sign Language (ISL) hand signs encompassing English and Marathi alphabets as well as numeric digits. Each sign is represented in three index-aligned formats: (1) original RGB hand gesture images, (2) pose-extracted skeletal keypoints generated using MediaPipe [1], and (3) skin-segmented binary masks produced with OpenCV [2]. The dataset covers 79 sign classes — 26 English letters (A–Z), 43 Marathi letters (अ–ज्ञ), and 10 digits (0–9) — collected from four participants against a uniform background under consistent indoor lighting using an iPhone 15 in square format, with all images resized to 1280 × 1280 pixels. Each class contains exactly 20 images per modality, yielding 1,580 images per modality and 4,740 images in total, derived from 1,425 unique photographic captures; for classes with fewer than 20 unique captures, existing images were duplicated to equalize class sizes, and all such entries are explicitly flagged in the accompanying manifest. To our knowledge, this is the first ISL dataset to include bilingual English–Marathi alphabet gestures with pose and skin-segmented modalities. The dataset is designed for training and benchmarking machine learning models for ISL recognition, comparative studies across image modalities, and the development of assistive technologies, with the aim of facilitating inclusive gesture recognition research and multilingual accessibility.
Files
Steps to reproduce
Data collection: Static ISL hand signs for 79 classes (26 English letters A–Z, 43 Marathi letters अ–ज्ञ, 10 digits 0–9) were performed by four participants against a neutral black background under consistent indoor lighting (natural light supplemented with a ring light). Images were captured with an iPhone 15 rear camera in square (1:1) format. Each participant performed each sign multiple times, yielding up to 20 unique captures per class. Preprocessing: All images were EXIF-orientation corrected and resized to 1280 × 1280 pixels (JPEG, quality 95). Duplicate files introduced during collection and transfer were identified by MD5 hash comparison and removed, retaining 1,425 unique captures. To provide uniform class sizes of 20 images per class, classes with fewer than 20 unique captures were topped up by duplicating existing images in cyclic order; all 155 such entries are flagged in manifest.csv (is_padded_duplicate, duplicate_of_index columns). Pose extraction: Hand landmarks were detected with MediaPipe Hands (Python, static_image_mode=True, max_num_hands=2, minimum detection confidence 0.5 with a 0.25 fallback pass for initially undetected images). The 21-point skeleton and connections were rendered onto the RGB photograph. Two-hand detection was enabled because many ISL letter signs are two-handed. Images in which no hand was detected retain the plain RGB photograph in the pose modality and are flagged in the manifest. Skin segmentation: Images were converted to HSV color space and thresholded with two hue bands covering the skin-tone range, followed by morphological opening (noise removal) and closing (hole filling), connected-component filtering retaining components ≥3% of the largest component's area, and light dilation, producing a binary mask (white hand/forearm on black background) via OpenCV. Organization: Images were sorted into Language/Modality/Class folders and renumbered 1–20 per class, with the three modalities index-aligned (identical index = same source photograph). Python scripts implementing the full pipeline (dedupe_rebuild.ipynb, build_isl_dataset.py) are included in the deposit.
Institutions
- Vishwakarma Institute of TechnologyMaharashtra, Pune