From_Shloka_to_Shobda_Corpus
Description
From Shloka to Shobda Corpus is a manually curated Sanskrit–Bengali parallel corpus for low-resource neural machine translation (NMT), containing 38,605 sentence-aligned Sanskrit–Bengali pairs compiled from five sources: Bhagavad Gita (5,885 pairs), Bible (7,838 pairs), Spoken Tutorial (23,835 pairs, derived from the Saamayik Sanskrit–English repository via an English-intermediate Bengali translation pipeline), Buddhacharitam (709 pairs), and Ramacharitam (338 pairs), the latter two digitized from printed editions via OCR followed by manual expert correction. This release (v1.1) provides, for each source, both the raw collected text files and cleaned text files (with embedded URLs and formatting artifacts removed), a reproducible processing script (code.py) that regenerates the cleaned files and a per-source parallel_corpus.xlsx, and a corpus-wide metadata.xlsx containing a unique sentence_id, domain label (Religious / Literary / Educational), collection method, Bengali translation method, license/reuse status, word and character counts, and a train/validation/test split assignment for every sentence pair. A fixed-seed (42), source-stratified 80:10:10 train/validation/test split is provided as both a metadata column and standalone ID list files (split_train_ids.txt, split_validation_ids.txt, split_test_ids.txt). A cleaning_report.csv documents, per source, how many pairs contained URLs or triple-quotation-mark formatting artifacts prior to cleaning. All Sanskrit and Bengali sentence pairs are aligned at the sentence level (line n in the Sanskrit file corresponds to line n in the Bengali file, and to the matching sentence_id in metadata.xlsx). Sentence-level quality was validated on a stratified sample (first 100 pairs of every 1,000) by a domain expert in Sanskrit and Bengali linguistics. The dataset supports research in low-resource NMT, multilingual transfer learning, OCR-assisted NLP for Indic languages, and Sanskrit–Bengali cross-lingual studies. License: CC BY 4.0 for the original compilation; the Spoken Tutorial subset's underlying Saamayik source should be checked independently for its own license terms.
Files
Steps to reproduce
1. Source data was collected from five sources: digital public-domain text (Bhagavad Gita, Bible), the publicly available Saamayik Sanskrit–English repository (Spoken Tutorial), and printed classical editions digitized via OCR (Buddhacharitam, Ramacharitam). 2. For the Spoken Tutorial subset, Bengali translations were produced from the English side of each Sanskrit–English pair by human translators, using Google Translate only as an assistive aid for syntactically complex sentences, followed by full human review and post-editing. 3. For Buddhacharitam and Ramacharitam, OCR output from printed editions was manually reviewed and corrected character-by-character by a language expert before inclusion. 4. To regenerate the cleaned files and per-source spreadsheet from the raw text files, run the provided code.py inside each source folder (e.g. cleaned_sources/bible/code.py): python3 code.py This script (a) normalizes line endings and removes blank/junk lines, (b) removes embedded URLs and literal triple-quotation-mark formatting artifacts while preserving legitimate punctuation, (c) verifies that the Sanskrit and Bengali line counts match (raising an error on misalignment), (d) assigns a deterministic 80:10:10 train/validation/test split using a fixed random seed (42), and (e) writes <source>.clean.txt files and parallel_corpus.xlsx. 5. Sentence-level alignment was verified by confirming equal line counts between the Sanskrit and Bengali files for each source; a character-length ratio filter (0.5 ≤ |Sanskrit|/|Bengali| ≤ 2.0) was applied during original corpus assembly to flag likely misaligned pairs. 6. Quality validation was performed by a domain expert (Sanskrit and Bengali linguistics) on a stratified sample: the first 100 sentence pairs of every 1,000 consecutive pairs, evaluated for semantic adequacy, grammatical correctness, and alignment validity. 7. The corpus-wide metadata.xlsx and split ID files were generated by running the per-source code.py scripts across all five sources and concatenating the resulting sentence_id, domain, provenance, and split fields; the same fixed seed (42) reproduces an identical split assignment on re-run.
Institutions
- University of Frontier Technology, BangladeshDhaka Division, Gazipur