Large Language Models (LLMs) frequently produce confident but incorrect answers, commonly called hallucinations, which limit their reliability in high stakes domains. We study this problem in the setting of multiple choice question answering, where it becomes a selective prediction task: the goal is to recognise when an answer is likely wrong and to replace it with an explicit abstention. Across four benchmarks (MMLU Pro, ARC Challenge, MedQA, TruthfulQA) and the gpt-4o-mini model, we first show that no single confidence signal, namely self reported confidence, answer token log probabilities, or multi sample consistency, reliably separates correct from incorrect answers on its own. Combining the signals in a multi signal detector reaches a cross validated AUROC of 0.79-0.81 on MMLU Pro and 0.78-0.88 when transferred to an unseen benchmark, and reduces the error rate among the retained half of the answers by 33-41%. The log probability features carry almost all of this performance and the five sample consistency features add nothing. We then analyse cost based abstention policies and the effect of prompting style on calibration and detector transfer. Finally, for the open weight Qwen 2.5 7B model, we train lightweight classifiers on hidden state activations and attention entropy collected in a single forward pass, reaching an AUROC of 0.72 from a single layer of the 4 bit model and 0.76 from all layers of the uncompressed model, on par with a SAPLMA style probe and with the model's own answer token probability (0.74), and far above its verbalised confidence (0.63) and sampling consistency (0.60), without any additional generation. Within the scope of multiple choice question answering, the probabilities and hidden states a model computes are thus more informative about the correctness of its answers than what it says about them. Open ended generation is left for future work.