MCC

Published: 8 March 2024| Version 1 | DOI: 10.17632/68dkhk8gch.1
Contributors:
,
,
,
,

Description

Despite recent advancements showcasing the impressive capabilities of Large Language Models (LLMs) in conversational systems, we show that even state-of-the-art LLMs are morally inconsistent in their generations, questioning their reliability (and trustworthiness in general). Prior works in LLM evaluation focus on developing ground-truth data to measure accuracy on specific tasks. However, for moral scenarios that often lack universally agreed-upon answers, consistency in model responses becomes crucial for their reliability. To this extent, we construct the Moral Consistency Corpus (MCC), containing 50K moral questions, responses to them by LLMs, and the RoTs that these models followed.

Files

Steps to reproduce

Simply run the notebooks / scipts present in the github repo (https://github.com/vnnm404/SaGE) to reproduce results.

Institutions

International Institute of Information Technology Hyderabad

Categories

Machine Learning, Morality

Licence