Добавить новость
ru24.net
News in English
Октябрь
2024

Probing the link between vision and language in material perception using psychophysics and unsupervised learning

0

by Chenxi Liao, Masataka Sawayama, Bei Xiao

We can visually discriminate and recognize a wide range of materials. Meanwhile, we use language to describe what we see and communicate relevant information about the materials. Here, we investigate the relationship between visual judgment and language expression to understand how visual features relate to semantic representations in human cognition. We use deep generative models to generate images of realistic materials. Interpolating between the generative models enables us to systematically create material appearances in both well-defined and ambiguous categories. Using these stimuli, we compared the representations of materials from two behavioral tasks: visual material similarity judgments and free-form verbal descriptions. Our findings reveal a moderate but significant correlation between vision and language on a categorical level. However, analyzing the representations with an unsupervised alignment method, we discover structural differences that arise at the image-to-image level, especially among ambiguous materials morphed between known categories. Moreover, visual judgments exhibit more individual differences compared to verbal descriptions. Our results show that while verbal descriptions capture material qualities on the coarse level, they may not fully convey the visual nuances of material appearances. Analyzing the image representation of materials obtained from various pre-trained deep neural networks, we find that similarity structures in human visual judgments align more closely with those of the vision-language models than purely vision-based models. Our work illustrates the need to consider the vision-language relationship in building a comprehensive model for material perception. Moreover, we propose a novel framework for evaluating the alignment and misalignment between representations from different modalities, leveraging information from human behaviors and computational models.



Moscow.media
Частные объявления сегодня





Rss.plus




Спорт в России и мире

Новости спорта


Новости тенниса
ATP

Надаль приехал на молодежный итоговый турнир ATP в Джидде






Путин сообщил о вылете в Актау самолета МЧС РФ с врачами и оборудованием

Косметолог Анчикова: рынок серой косметологии в РФ растет – это очень опасно

Ефимов: два участка в СВАО и САО реорганизуют по программе КРТ

Собянин анонсировал открытие более 40 новых станций метро и МЦД к 2030 году