PsyProxy
Datasets·Social science·03__fact_checking__liar__truth_ordinal

LIAR political-claim truth ratings

LIAR is a public corpus of 12,836 fact-checked political statements labeled on a six-level truth scale (pants-on-fire to true). The texts predominantly address a range of political and social issues, often focusing on tax relief , healthcare reform , and education funding . Recurring themes include the financial implications of government policies , such as the impact of budget cuts on school districts and the consequences of tax structures on low-income populations. Additionally, the texts highlight concerns regarding gun control and border security , as well as the socioeconomic challenges faced by specific demographics, including unemployed veterans and minority communities . The statements reflect a critical examination of political figures and their decisions, often questioning the accuracy of claims made in public discourse. [Summary on 50 random texts by ChatGPT 4o Mini].

Distribution of Truth (1=pants-on-fire to 6=true)
1
2
3
4
5
6
2,063 at floor1,050 at ceiling
12,836
items
2,568
holdout n
Truth (1=pants-on-fire to 6=true)
target
Ordinal
kind
25
systems compared
Criterion validity

Reported holdout systems from the verified card

Ordinal prediction uses Quad. κ as the task-primary metric. Secondary columns keep the companion metrics visible so binary, ordinal, regression, and multiclass cards are not compared through one flattened score.

Source podium · Quad. κ · 10 families
Gold
PsyProxy
0.269
Silver
LIWC
0.208
Bronze
TextDescriptives
0.183
Model-family mix
PsyProxy · 4Lexicon · 2Baseline · 15OpenAI / LLM · 3Topic model · 1
SystemFamilyVariantQuad. κWithin-oneMAEPrimary scale
psyproxyPsyProxy — Social Economics Lens v0.5 · 1000d
PsyProxypermissive0.2690.5851.42
psyproxyPsyProxy — Health Lens v0.9 · 1100d
PsyProxypermissive0.2660.5941.40
psyproxyPsyProxy — Technology Lens v0.5 · 800d
PsyProxypermissive0.2580.5981.42
psyproxyPsyProxy — Behavioral Sciences Lens v0.5 · 1000d
PsyProxypermissive0.2440.5901.42
lexLinguistic Inquiry and Word Count (LIWC)
Lexiconpermissive0.1920.5671.48
baselineTextDescriptives
Baselinepermissive0.1810.5751.44
baselineTool for the Automatic Analysis of Syntactic Sophistication and Complexity (TAASSC)
Baselinepermissive0.1660.5631.47
baselineTool for the Automatic Analysis of Cohesion (TAACO)
Baselinepermissive0.1360.5601.44
llmOpenAI Model gpt-4.1-nano
OpenAI / LLMpermissive0.1350.5601.48
baselineTool for the Automatic Analysis of Lexical Sophistication (TAALES)
Baselinepermissive0.1040.5601.48
topicBERTopic
Topic modelpermissive0.0990.5761.32
llmOpenAI Model gpt-5-nano
OpenAI / LLMpermissive0.0920.5381.50
llmOpenAI Model gpt-4o-mini
OpenAI / LLMpermissive0.0640.5501.46
lexValence Aware Dictionary and sEntiment Reasoner (VADER)
Lexiconpermissive0.0370.5541.38
baselineEmpath
Baselinepermissive0.0030.5521.36
baselineDisneyland TripAdvisor reviews binary · via best lens
Baselinepermissive
baselineIMDB movie reviews (ACL) binary · via best lens
Baselinepermissive
baselineAmazon Video-Games reviews ordinal · via best lens
Baselinepermissive
baselineAmazon Video-Games reviews continuous · via best lens
Baselinepermissive
baselineDouban movie reviews (Chinese) ordinal · via best lens
Baselinepermissive
baselineAmazon Video-Games reviews binary · via best lens
Baselinepermissive
baselineDruglib drug reviews regression · via best lens
Baselinepermissive
baselineDruglib drug reviews ordinal · via best lens
Baselinepermissive
baselineDisneyland TripAdvisor reviews continuous · via best lens
Baselinepermissive
baselineSentiment140 tweets binary · via best lens
Baselinepermissive