Memory-Efficient Adaptation and Unanswerable-Aware Supervision for Reliable Private Email Question Answering

Abstract

Private document question answering (QA) requires language models to answer from local evidence while abstaining when the supplied context is insufficient. This study evaluates quantized parameter-efficient adaptation for private email QA using EnronQA. A Qwen2.5-7B-Instruct backbone is adapted with QLoRA and QPiSSA under matched rank, target modules, quantization, and training budget. QPiSSA combines PiSSA initialization with residual quantization, whereas QLoRA quantizes the original frozen weight matrix and trains standard low-rank adapters. In addition to answerable-only fine-tuning, an unanswerable-aware supervision variant augments training with synthetic missing-evidence examples whose target response is I don't know. Experiments are conducted on a fixed EnronQA subset with 20,000 training, 2000 development, and 2000 test examples. Fine-tuning substantially improves oracle-email answer quality over zero-shot prompting, but answerable-only tuning sharply reduces abstention under missing evidence. Adding unanswerable examples restores abstention accuracy to approximately 0.998 for both QLoRA and QPiSSA while preserving most answer-generation performance. Bootstrap confidence intervals over the fixed test set, multi-seed evaluation, BERTScore evaluation, leakage diagnostics, strict filtered unanswerable evaluation, retrieval-augmented testing, quantization-error diagnostics, and a small diagnostic ablation support a cautious interpretation: QLoRA and QPiSSA perform similarly under this matched configuration, and the main reliability gain comes from explicit missing-evidence supervision rather than from the adapter choice alone.