Abstract analysis that is trivial for humans often stymies GPT-4o, Gemini, and Sonnet.
Enlarge/ Whatever you do, don't ask the AI how many horizontal lines are in this image. (credit: Getty Images)
In the last couple of years, we've seen amazingadvancements in AI systems when it comes to recognizing and analyzing the contents of complicated images. But a new paper highlights how many state-of-the-art "vision learning Models" (VLMs) often fail at simple, low-level visual analysis tasks that are trivially easy for a human.
In the provocatively titled pre-print paper "Vision language models are blind" (which has a PDF version that includes a dark sunglasses emoji in the title), researchers from Auburn University and the University of Alberta create eight simple visual acuity tests with objectively correct answers. These range from identifying how often two colored lines intersect to identifying which letter in a long word has been circled to counting how many nested shapes exist in an image (representative examples and results can be viewed on the research team's webpage).
Crucially, these tests are generated by custom code and don't rely on pre-existing images or tests that could be found on the public Internet, thereby "minimiz[ing] the chance that VLMs can solve by memorization," according to the researchers. The tests also "require minimal to zero world knowledge" beyond basic 2D shapes, making it difficult for the answer to be inferred from "textual question and choices alone" (which has been identified as an issue for some other visual AI benchmarks).
Saint Seiya: Meteor Shine это новая мобильная игра по аниме «Рыцари Зодиака»
Blizzard politely tells Hearthstone players their game isn't dead just because it's not getting a new cosmetic board this expansion
Activision secretly experimented on 50% of Call of Duty players by 'decreasing' skill-based matchmaking, and determined players like SBMM even if they don't know it
Co-op survival game Enshrouded now lets you make things way tougher, or way easier, with an update that adds more than 30 difficulty sliders
Филиал № 4 ОСФР по Москве и Московской области информирует:
С 1 августа Соцфонд увеличит страховые пенсии россиян
Филиал № 4 ОСФР по Москве и Московской области информирует:
С 1 августа Соцфонд увеличит страховые пенсии россиян
Филиал № 4 ОСФР по Москве и Московской области информирует:
Пенсии работающих пенсионеров начнут индексироваться с 2025 года
Филиал № 4 ОСФР по Москве и Московской области информирует:
В Московском регионе свыше 11,3 тыс. неработающих родителей получают пособие по уходу за ребенком до 1,5 лет
Основательница Wildberries подала иск о разделе имущества с мужем
Филиал № 4 ОСФР по Москве и Московской области информирует:
За полгода 14,9 тысячи жителей Московского региона оформили страховую пенсию в автоматическом режиме на портале госуслуг
Уроки анимации в Екатеринбурге
Отделение СФР по Москве и Московской области проактивно открыло свыше 32 тысяч СНИЛС новорожденным