The One With GenAI Vendor Selection
Generative AI Benchmarking & Vendor Advisory
An empirical evaluation of Claude, ChatGPT, and Perplexity against human analyst baselines for enterprise IT vendor assessment.
Context
Enterprise teams increasingly rely on large language models to accelerate vendor research, but model-generated recommendations may vary in consistency, compliance coverage, analytical quality, and reliability.
Challenge
Enterprise teams increasingly use LLMs to accelerate vendor research, but model-generated recommendations may vary in consistency, compliance coverage, analytical quality, and reliability.
Objective
Determine whether GenAI tools can reliably support enterprise IT vendor selection when evaluated against human analyst baselines and compliance requirements.
Approach
- Designed and executed an empirical benchmarking framework.
- Compared Claude, ChatGPT, and Perplexity with human analyst baselines.
- Evaluated 6 IT helpdesk vendors.
- Assessed vendors against SOC 2, ISO 27001, PDPA, and GDPR.
- Evaluated model performance across consistency, analytical quality, compliance coverage, and recommendation reliability.
My Role
Designed the benchmarking framework, conducted vendor and model evaluations, and synthesised findings into governance-oriented recommendations.
Analysis
- Consistency of model-generated vendor assessments
- Analytical quality relative to human analyst baselines
- Compliance coverage across SOC 2, ISO 27001, PDPA, and GDPR
- Recommendation reliability for enterprise vendor selection
Outcome
Identified important limitations in using LLM-generated outputs for enterprise vendor selection and demonstrated the need for human validation, transparent evaluation criteria, and governance controls.
Key Takeaways
- GenAI can accelerate vendor research but requires structured validation.
- Compliance coverage and analytical consistency vary significantly across models.
- Human oversight and governance controls are essential for enterprise vendor selection.