LLM Wars: A Benchmark of Large Language Models for Pain Point Extraction from Software Feedback

Script:

Hi, my name is Om Patel. I'm a Grade 11 student at NCIVS in Sarnia, Ontario, and my project is called LLM Wars.

Every industry generates massive amounts of unstructured feedback containing valuable signals like frustrations and unmet needs, but it is too large to analyze manually. AI models can process thousands of records in minutes, but nobody has scientifically tested which one is the best at this.

If we can extract these signals accurately, we can understand what millions of people are struggling with and use that to solve problems through software that actually matters.

So I built the first benchmark to find out. I tested ten AI models head to head on thousands of real software reviews on various platforms. Almost a thousand developers validated the results over 70 days.

The results were surprising. The most accurate model wasn't the best overall. Every model collapsed on informal unstructured data. And the cheapest model delivered 93 percent of the top model's quality at a fraction of the cost.

But this goes beyond software. Anywhere humans describe problems in natural language, this framework can turn feedback into real-world solutions. Thank you.

Background

Every industry generates massive amounts of unstructured user feedback. Platforms like G2, Capterra, the App Store, and Reddit contain millions of reviews describing real frustrations with software products. These complaints, called pain points, represent validated signals of unmet needs. But this data is too large, inconsistent, and messy to analyze manually. After spending hours reading reviews and finding only one useful insight, it became clear that this process needed to be automated.

The Problem

Large Language Models can process thousands of reviews in minutes. But a critical question remained unanswered: which model is actually best at this task and does it the most accurately? Existing benchmarks like MMLU and HELM test models on clean, controlled tasks. None evaluate performance on real-world feedback that is noisy, ambiguous, and context-dependent. Without task-specific evaluation, there is no scientific basis for choosing which model to deploy.

Purpose

This study introduces the first multi-dimensional benchmark for evaluating LLMs on structured pain point extraction from real user feedback. Ten models were tested on 12,000 records under identical conditions, scored on accuracy, consistency, and cost. The goal is to provide a scientific basis for model selection in real-world deployment.

Beyond Software

While software feedback was the testing domain, the framework is designed to generalize. Healthcare organizations could detect recurring symptom patterns from patient feedback. Public health agencies could spot emerging health trends earlier. Financial institutions could catch fraud patterns in consumer complaints. The framework applies to any domain where humans describe problems in natural language.

Data Collection

Data was collected from four platforms using ScraperAPI: G2, Capterra, App Store/Google Play, and Reddit. The raw dataset totalled approximately 50,000 records across hundreds of software categories and subreddits. A stratified random sample of 12,000 records was drawn using seed 0.42, maintaining proportional representation across all sources and preventing any single category from dominating.

See Fig. 1 (end-to-end pipeline).

Gold Standard

Rather than using AI to generate labels, real human validation was used. Over 70 days, 958 software founders and developers swiped on extracted pain point cards, producing 7,920 judgments. Each user saw 20 unique cards per day, swiping right for valid and left for not useful, with a required skip reason on left swipes. Approximately 890 cards received five or more independent swipes, forming the 900-record gold standard with 84.7% inter-validator agreement.

See Fig. 2 (gold standard validation process).

Model Testing

All 10 models ran on the same 12,000 records under identical conditions with the same system prompt. Each returned structured JSON with up to 2 pain points per record. All scripts were idempotent and ran in parallel with provider-specific rate limits.

Evaluation

F1 accuracy was measured by comparing model extractions against the gold standard using cosine similarity matching with a domain tag check. Consistency was measured using inter-model semantic agreement across all 12,000 records. The final composite score weighted F1 at 60% and agreement at 40%.

See Fig. 3 (matching algorithm), Fig. 4 (agreement computation), Fig. 5 (scoring framework).

Controlled Variables

All parameters were fixed: seed 0.42, prompt v1, text-embedding-3-small for all embeddings, cosine threshold 0.60, max 2 pain points per record. All data stored in PostgreSQL with separate schemas per pipeline stage. Every script is idempotent and the entire study is fully reproducible.

Results

A total of 10 Large Language Models were evaluated on 12,000 software feedback records and scored against a 900-record human-validated gold standard. Performance was measured using F1 score (60% weight) and inter-model semantic agreement (40% weight), combined into a final composite score.

Finding 1: The most accurate model was not the best overall.

Claude Sonnet 4.6 achieved the highest F1 score (0.8293) but ranked second overall due to lower consistency (A = 0.6891). Claude Haiku 4.5 ranked first by combining strong accuracy (F1 = 0.8176) with significantly higher agreement (A = 0.7693). This is the central finding: accuracy and consistency are two separate dimensions, and evaluating only one gives an incomplete picture.

See Fig. 6 (final leaderboard).

Finding 2: A clear precision-recall tradeoff exists.

Claude Sonnet had the highest precision (P = 0.9115) but missed 23.9% of real pain points. GPT-5.4-mini had the highest recall (R = 0.8421) but a 40.5% false positive rate. Claude Haiku was the only balanced model. A system prioritizing quality needs a different model than one prioritizing coverage.

See Fig. 7 (precision-recall comparison).

Finding 3: Every single model scored lowest on Reddit.

Average F1 dropped from 0.8083 on Capterra to 0.5990 on Reddit, a gap of 20.9 percentage points. This held across every model type, provider, and price point, confirming that informal conversational language is the primary challenge for current LLMs, not any individual model's weakness.

See Fig. 8 (F1 by data source).

Finding 4: Agreement and accuracy are empirically independent.

GPT-5.4-mini had the highest agreement (A = 0.8762) but ranked 7th on F1. Mixtral 8x7B ranked 2nd on agreement but produced the lowest F1 (0.6018). A model can look consistent without being accurate, which is why the composite framework uses both metrics together.

See Fig. 10 (F1 vs agreement).

Finding 5: Cost efficiency varied by more than 15x.

Grok-4-fast delivered 93.3% of Claude Sonnet's composite score at just 6.1% of the cost. GPT-5.4-nano ranked 4th overall at $0.48 per 1,000 records, outranking five more expensive models. The most expensive model is not always worth the cost.

See Fig. 11 (cost vs performance).

Hypothesis Outcomes

The primary hypothesis was partially supported. Anthropic models ranked first and second, but the frontier advantage did not hold uniformly. GPT-5.4-mini and GPT-5.4-nano ranked 3rd and 4th overall but their F1 scores were lower than non-frontier models like Grok-4-fast and DeepSeek-v3. The supporting hypotheses were partially to fully supported: models performed worse on Reddit than structured platforms, and models producing more structured outputs scored higher. The null hypothesis was rejected as the F1 range of 0.2275 points confirms model choice has a measurable impact. A system using the lowest-ranked model would find roughly 27 percentage points fewer valid pain points than the top-ranked model.

Conclusion

Large Language Models can handle production pain point extraction, but how well they perform depends heavily on which model is chosen and what type of data it is processing. The highest-accuracy model is not always the best choice, the most expensive model is not always worth the cost, and no single metric tells the full story.

What These Results Mean

Accuracy and consistency are independent dimensions. Claude Sonnet had the highest F1 but ranked second because its outputs were inconsistent at scale. The Reddit performance drop held across every model without exception, confirming informal language is a fundamental challenge and not a model-specific weakness. Finally, the cheapest model delivered 93.3% of the top model's performance at 6.1% of the cost, challenging the assumption that expensive models deliver proportionally better results.

Why This Matters Beyond Software

Although software feedback was the testing domain, the evaluation framework is designed to generalize. Important signals are hidden inside messy, large-scale data everywhere, from patient complaints and support tickets to public health discussions and financial service feedback. If these signals can be extracted accurately, organizations can respond to problems faster, make better decisions, and uncover opportunities that would otherwise stay hidden. A multi-dimensional benchmarking approach, as introduced in this study, gives a more complete and honest picture of model performance than any single metric can. For anyone building a real system to analyze user feedback at scale, this study provides the first scientific basis for making that model selection decision.

Future Work

Test multiple prompt strategies (few-shot, chain-of-thought) to measure ranking stability.

Fine-tune open-source models on validated data to close the gap with frontier models.

Re-run the benchmark as new models are released.

Extend the framework into healthcare, finance, and other domains to test generalization.

What Could Be Improved

Expand validator engagement through incentives and gamification to increase gold standard coverage.

Include more advanced frontier models that were excluded due to budget constraints.

Test across multiple languages to validate findings beyond English-only content.

Learn more about the project here!

Demo of the benchmark

Acknowledgements

I would like to thank Aryan Patel, B.Eng. graduate in Software Engineering from McMaster University, for his guidance throughout this project. He helped refine the research question, advised on evaluation methodology, and provided feedback on the IEEE research paper.

I would also like to thank NCIVS for supporting this project and the 958 software founders and developers who participated in the 70-day human validation study, without whom the gold standard would not have been possible.

Data collection was performed using ScraperAPI. The API providers whose models were evaluated include Anthropic, OpenAI, Google, xAI, DeepSeek, Groq, and Together AI. Total API spending for the complete benchmark was approximately $225 USD. Finally, I would like to thank my family for all of their support and encouragement.

Journal Articles:

Artstein, R., & Poesio, M. (2008). Inter-Coder Agreement for Computational Linguistics. Computational Linguistics, 34(4), 555-596.

Pang, B., & Lee, L. (2008). Opinion Mining and Sentiment Analysis. Foundations and Trends in Information Retrieval, 2(1-2), 1-135.

Vaughan, J. W. (2017). Making Better Use of the Crowd: How Crowdsourcing Can Advance Machine Learning Research. Journal of Machine Learning Research, 18(1), 7026-7071.

Books:

Liu, B. (2012). Sentiment Analysis and Opinion Mining. Morgan & Claypool.

Manning, C. D., Raghavan, P., & Schutze, H. (2008). Introduction to Information Retrieval. Cambridge University Press.

Jurafsky, D., & Martin, J. H. (2023). Speech and Language Processing, 3rd ed. (draft). Stanford University. Retrieved from https://web.stanford.edu/~jurafsky/slp3/

Proceedings & Conference Papers:

Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008). Cheap and Fast-But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. In Proc. EMNLP, 254-263.

Kazai, G., Kamps, J., & Milic-Frayling, N. (2011). Worker Types and Personality Traits in Crowdsourcing Relevance Labels. In Proc. ACM Int. Conf. Information and Knowledge Management (CIKM), 1941-1944.

Hsueh, P., Melville, P., & Sindhwani, V. (2009). Data Quality from Crowdsourcing: A Study of Annotation Selection Criteria. In Proc. NAACL HLT Workshop on Active Learning for NLP, 27-35.

Kittur, A., Chi, E. H., & Suh, B. (2008). Crowdsourcing User Studies with Mechanical Turk. In Proc. ACM SIGCHI Conf. Human Factors in Computing Systems, 453-456.

Hendrycks, D., et al. (2021). Measuring Massive Multitask Language Understanding. In Proc. Int. Conf. Learning Representations (ICLR).

Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Proc. Advances in Neural Information Processing Systems (NeurIPS), vol. 35.

Kojima, T., et al. (2022). Large Language Models are Zero-Shot Reasoners. In Proc. Advances in Neural Information Processing Systems (NeurIPS), vol. 35.

Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proc. EMNLP, 3982-3992.

Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020). BERTScore: Evaluating Text Generation with BERT. In Proc. Int. Conf. Learning Representations (ICLR).

Wang, S., et al. (2021). Want To Reduce Labeling Cost? GPT-3 Can Help. In Findings of EMNLP, 4195-4205.

Ding, N., et al. (2022). Is GPT-3 a Good Data Annotator? In Proc. Annual Meeting of the Association for Computational Linguistics (ACL), 26-38.

McAuley, J., Targett, C., Shi, Q., & van den Hengel, A. (2015). Image-Based Recommendations on Styles and Substitutes. In Proc. ACM SIGIR Conf. Research and Development in Information Retrieval, 43-52.

Sarvari, P. (2025). Rapidly Benchmarking Large Language Models for Diagnosing Comorbid Patients. JMIRx Med, 6, e67661. doi: 10.2196/67661.

Preprints:

OpenAI. (2023). GPT-4 Technical Report. arXiv preprint arXiv:2303.08774.

Bubeck, S., et al. (2023). Sparks of Artificial General Intelligence: Early Experiments with GPT-4. arXiv preprint arXiv:2303.12712.

Liang, P., et al. (2022). Holistic Evaluation of Language Models. arXiv preprint arXiv:2211.09110.

Srivastava, A., et al. (2022). Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models. arXiv preprint arXiv:2206.04615.

Google Team, et al. (2023). Gemini: A Family of Highly Capable Multimodal Models. arXiv preprint arXiv:2312.11805.

Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv preprint arXiv:1301.3781.

Webpages & Technical Reports:

Anthropic. (2024). The Claude 3 Model Family: Opus, Sonnet, Haiku. Retrieved from https://www.anthropic.com/claude-3-model-card

Anthropic. (2024). Evaluation and Safety Methods. Retrieved from https://www.anthropic.com/research

OpenAI. (2024). GPT-4o System Card. Retrieved from https://openai.com/index/gpt-4o-system-card

OpenAI. (2022). Text Embeddings and Semantic Search. Retrieved from https://platform.openai.com/docs/guides/embeddings

OpenAI. (2024). Evals Framework Documentation. Retrieved from https://github.com/openai/evals

OpenAI. (2024). Model Evaluation Best Practices. Retrieved from https://platform.openai.com/docs/guides/evals

DeepSeek AI. (2024). DeepSeek Technical Overview. Retrieved from https://www.deepseek.com

Meta AI. (2024). Introducing Meta Llama 3: The Most Capable Openly Available LLM. Retrieved from https://ai.meta.com/blog/meta-llama-3

Mistral AI. (2024). Mistral Large 2. Retrieved from https://mistral.ai/news/mistral-large-2407

Stanford CRFM. (2022). HELM: Holistic Evaluation of Language Models. Retrieved from https://crfm.stanford.edu/helm/latest/

Hugging Face. (2024). Open LLM Leaderboard. Retrieved from https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

Data Sources:

G2. G2 Software Reviews Platform. Retrieved from https://www.g2.com. Accessed: March 2026.

Capterra. Capterra Software Reviews Platform. Retrieved from https://www.capterra.com. Accessed: March 2026.

Apple Inc. App Store Review Guidelines. Retrieved from https://developer.apple.com/app-store/review/guidelines/

Google LLC. Google Play Store Reviews System. Retrieved from https://play.google.com/about/developer-content-policy/

Reddit Inc. Reddit API Documentation. Retrieved from https://www.reddit.com/dev/api/

ScraperAPI. Web Scraping API Documentation. Retrieved from https://www.scraperapi.com. Accessed: March 2026.