A Confidence-Aware Hybrid Vision-Language Framework for Food Recognition and Nutritional Monitoring.

Background/

Objectives

The objective evaluation of regional dietary intake remains a core challenge in personalized health management due to complex plate presentations and a lack of culturally specific dataset benchmarks.

Methods

This study introduces a confidence-aware hybrid vision-language framework engineered for traditional Turkish food recognition and structured nutritional assessment.

Results

We curate a balanced dataset containing 14,711 verified images spanning 40 representative Turkish culinary classes to train and evaluate seven deep learning architectures. Among the visual models, EfficientNet V2-L achieved the highest standalone performance with an accuracy of 93.47%, 0.92 macro-precision, 0.92 macro-recall, and a 0.92 F1 score. To overcome visual ambiguity and automate content analysis, a confidence-aware routing strategy escalates uncertain predictions (τ<0.70) or user-rejected classifications to the Google Gemini 2.5 Flash multimodal large language model (MLLM).

Conclusions

This hybrid paradigm yields a combined classification accuracy of 95.50% while validating portion weight estimations within a mean absolute error (MAE) of 18.42 g and total energy within 36.75 kcal. Fully realized as a cross-platform Flutter mobile application, the end-to-end pipeline demonstrates localized plate detection, adaptive portion analysis, and structured nutrient tracking, providing a scalable design for consumer-facing digital nutrition platforms.