Reasoners or Translators?

This study asks whether large language models reason over tax statute or reproduce answers seen during training. It compares models that answer directly in natural language with a neuro-symbolic pipeline, in which the model translates a case into Prolog and a solver carries out the inference.

How it works

The work builds on SARA, the StAtutory Reasoning Assessment dataset. SARA pairs natural-language statutes and cases with human-coded Prolog, for deciding entailment and for computing tax owed.

In the neuro-symbolic pipeline, the LLM converts a case's fact pattern into Prolog facts. Those facts join SARA's statute knowledge base along with a query such as owes_tax(alice, 2015, X), and the solver binds X to 14,000.

Two additions separate reasoning from memorization. Contamination quizzes check whether a model can pick the verbatim SARA instance out of meaning-preserving perturbations. SARA+ adds new splits that change the numbers in the rules and cases and paraphrase the text, so each problem keeps its structure while its answer changes.

Why it's interesting

When both the rules and the cases are perturbed, GPT-5's direct answers on numerical entailment fall from 98% to 73%, while the Prolog pipeline with the same model stays at 95%. This traces much of the direct models' earlier high scores to benchmark exposure, and it shows that statutory tax reasoning favors neuro-symbolic methods over standalone LLMs.

Status

The paper was accepted at SURGeLLM 2026, the ACL 2026 Workshop on Structured Understanding, Retrieval, and Generation in the LLM Era, and the preprint is on arXiv.