Benchmarking LLMs Against TPTP: A Vision for the Near Future

4 pages•Published: October 24, 2026

Andrija Urošević, Gordan Đurić and Sana Stojanović-Đurđević

Abstract

We propose the TPTP World as a benchmark for evaluating LLMs' reasoning in which success is a TPTP derivation verified by independent verifiers. We describe the design of Proof Assistant for Agents (PAFA), an interface in which an AI agent states claims and premises, justified by a primitive rule or by automation of configurable strength. A finished proof is serialized directly as the TPTP derivation. We give the operation set for PAFA, a pipeline, and an evaluation protocol, and leave the underlying proof system, the automation boundary, and the proof state's retrieval mechanism open.

Keyphrases: automated theorem proving, benchmarking, large language models, proof verification, tptp

In: Stephan Schulz, Konstantin Korovin and Michael Rawson (editors). Proceedings of 16th International Workshop on the Implementation on Logics. Proceedings of the 26th Conference on Logic for Programming, Artificial Intelligence, and Reasoning (short papers), vol 23, pages 129-132.

Links:	https://easychair.org/publications/paper/PMbN
	https://doi.org/10.29007/fsmp

BibTeX entry

@inproceedings{IWIL-2026:Benchmarking_LLMs_Against_TPTP,
  author    = {Andrija Urošević and Gordan Đurić and Sana Stojanović-Đurđević},
  title     = {Benchmarking LLMs Against TPTP: A Vision for the Near Future},
  booktitle = {Proceedings of 16th International Workshop on the Implementation on Logics. Proceedings of the 26th Conference on Logic for Programming, Artificial Intelligence, and Reasoning (short papers)},
  editor    = {Stephan Schulz and Konstantin Korovin and Michael Rawson},
  series    = {Kalpa Publications in Computing},
  volume    = {23},
  publisher = {EasyChair},
  bibsource = {EasyChair, https://easychair.org},
  issn      = {2515-1762},
  url       = {/publications/paper/PMbN},
  doi       = {10.29007/fsmp},
  pages     = {129-132},
  year      = {2026}}

Download PDF Open PDF in browser