A Newly Constructed Multilingual Digital Capability Model Dataset for 1,496 Listed Firms Across 20 Asian and Middle Eastern Markets, 2019–2024

Description

This repository contains a newly constructed, model-derived dataset of firm-level digital capability. The dataset did not previously exist in the underlying corporate disclosures or in an existing structured database. Public annual and sustainability reports serve only as source documents. Through the Digital Capability Model (DCM) protocol, the authors created structured paragraph records, multilingual term mappings, digital-context classifications, dimension-level measurements, normalized firm-year scores, provenance records, and validation indicators. Quantified outputs: - 1,496 listed firms across 20 Asian and Middle Eastern markets, 2019–2024 - 7,696 validated firm-year observations - 2,914,037 structured paragraph records - 733,702 digital-context classifications - 622 multilingual dictionary rows - five DCM components plus the composite DCM score Primary analytical files (start here): - firm_year_DCM_scores_VALIDATED.csv (definitive 7,696-row panel) - SOURCE_DOCUMENT_REGISTRY.csv - DATA_DICTIONARY.csv - DCM_multilingual_dictionary.csv - digital_context_paragraphs.csv.gz - VALIDATION_REPORT.md, LINEAGE_RECONCILIATION_REPORT.md, LICENSE_AND_THIRD_PARTY_NOTICE.md The deposit also preserves the Step 1–5 Python pipeline, processing logs, corpora, and notebooks: Input company-year workbook | v Step 1 — Mapping and link audit | v Step 2 — URL resolution | v Step 3 — PDF download and extraction input | v Step 4 — PDF text extraction | +--> report_pages | +--> report_paragraphs | v Step 4B — Language audit and multilingual dictionary | v Step 5 — Paragraph scoring and firm-year aggregation | v Validated Digital Capability firm-year dataset Intended uses: research transparency, reproducibility, methodological review, source-document compliance, and subsequent empirical analysis. *) For more details, please check README.md and DATASET_OVERVIEW.md.

Steps to reproduce

Recommended environment: Linux or another Unix-like system, Python 3.10 or 3.11, sufficient disk space, stable internet access for report acquisition, and adequate memory. A GPU is optional for faster dictionary translation. 1) Environment setup (deposit root): python3 -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip python -m pip install -r code/requirements.txt python -m pip install torch transformers sentencepiece jupyterlab matplotlib openpyxl 2) Verify integrity: sha256sum -c CHECKSUMS_sha256.txt 3) For analysis, use these deposited files first (no pipeline rerun required): - firm_year_DCM_scores_VALIDATED.csv (7,696 validated firm-years) - SOURCE_DOCUMENT_REGISTRY.csv - DATA_DICTIONARY.csv - VALIDATION_REPORT.md Do not report the raw 7,719-row intermediate file as the analytical sample. 4) To reproduce the full pipeline from code/ (review config.yaml; write to output/): cd code Step 1: python run_01_mapping_and_link_audit.py --input-excel "../data/input/Input data_company list and report link.xlsx" --source-sheet "Table Per Year" --trial-sheet "Data" --result-sheet "Data - Mapped All" Step 2: python run_02_resolve_links.py Step 3: python run_03_download_pdfs.py Step 4: python run_04_extract_pdf_text_v5.py --input-csv output/extraction_input.csv --path-col saved_pdf_path --timeout 180 --batch-size 10 cp ../logs/stage4b_document_language_audit.csv output/document_language_audit.csv Step 4B: python run_04b_prepare_multilingual_dictionary.py --mode audit --doc-language-csv output/document_language_audit.csv python run_04b_prepare_multilingual_dictionary.py --mode build --doc-language-csv output/document_language_audit.csv --output-csv output/dcm_dictionary_multilingual.csv --gaps-csv output/dcm_dictionary_translation_gaps.csv --cache-csv output/dcm_dictionary_translation_cache.csv --model facebook/nllb-200-distilled-600M --batch-size 32 Step 5: python run_05_score_digital_capability.py --input-csv output/report_paragraphs.csv --language-audit-csv output/document_language_audit.csv --dictionary-csv output/dcm_dictionary_multilingual.csv --output-dir output --chunksize 100000 --reset-output Or score the deposited corpus directly with: --input-csv ../data/report_paragraphs/report_paragraphs.csv.gz 5) Optionally regenerate top-level artifacts: python code/build_deposit_artifacts.py 6) Run notebooks/DCM_End_to_End_Data_Processing_and_Analysis_Validated_Final.ipynb (Restart Kernel and Run All). Use the validated 7,696 observations. Intermediate doc_id differences are documented in LINEAGE_RECONCILIATION_REPORT.md and do not alter the validated panel; merge ESG/controls before inferential analysis.

Institutions

• Binus University

  Jakarta, Jakarta

Categories

Licence