This is the third volume.

  • Mobile-MMLU: A Mobile Intelligence Language Understanding Benchmark
    Sondos Mahmoud Bsharat, Mukul Ranjan, Aidar Myrzakhan, Jiacheng Liu, Bowei Guo, Shengkun Tang, Zhuang Liu, Yuanzhi Li, Zhiqiang Shen, (1):1−48, 2026.
    Abstract

    Rapid advancements in large language models (LLMs) have increased interest in deploying them on mobile devices for on-device AI applications. Mobile users interact differently with LLMs compared to desktop users, creating unique expectations and data biases. Current benchmark datasets primarily target server and desktop environments, and there is a notable lack of extensive datasets specifically designed for mobile contexts. Additionally, mobile devices face strict limitations in storage and computing resources, constraining model size and capabilities, thus requiring optimized efficiency and prioritized knowledge. To address these challenges, we introduce Mobile-MMLU, a large-scale benchmark dataset tailored for mobile intelligence. It consists of 16,186 questions across 80 mobile-related fields, designed to evaluate LLM performance in realistic mobile scenarios. A challenging subset, Mobile-MMLU-Pro, provides advanced evaluation similar in size to MMLU-Pro but significantly more difficult than our standard full set. Both benchmarks use multiple-choice, order-invariant questions focused on practical mobile interactions, such as recipe suggestions, travel planning, and essential daily tasks. The dataset emphasizes critical mobile-specific metrics like inference latency, energy consumption, memory usage, and response quality, offering comprehensive insights into model performance under mobile constraints. Moreover, it prioritizes privacy and adaptability, assessing models’ ability to perform on-device processing, maintain user privacy, and adapt to personalized usage patterns. The Mobile-MMLU family offers a standardized framework for developing and comparing mobile-optimized LLMs, enabling advancements in productivity and decision-making within mobile computing environments. Our data is available at https://huggingface.co/datasets/MBZUAI-LLM/Mobile-MMLU.
    [PDF] [bib]

  • Automated Data Preparation for Machine Learning: A Survey
    Sasa Mladenovic, Marius Lindauer, Carola Doerr, (2):1−72, 2026.
    Abstract

    Data preparation is essential for effective machine learning (ML), yet typically remains a manual, time-consuming process. While automated machine learning (AutoML) has successfully addressed modeling aspects of the ML workflow, data preparation has largely been overlooked, leading to challenges with real-world, imperfect data. Conversely, a rising paradigm in the world of artificial intelligence (AI) and ML is that of data-centric AI, shifting focus from just refining models, to enhancing data in order to advance performance boundaries. This survey motivates the need for automated solutions regarding data preparation, offering a fundamental understanding of the benefits of data transformations and establishing the complexity of data pipeline optimization, while highlighting the importance of data quality. We provide a comprehensive overview and categorization of existing automation approaches, both in AutoML and as standalone fully or semi-automated systems. We discuss underlying methodologies, their advantages, and limitations. Our work explores the prospects of expanding automation to cover a broader data preparation process, aiming to bridge the gap between data-centric AI and AutoML. It paves the way to a wholly automated pipeline from raw real-world data to quality model predictions, and outlines future research directions towards that goal.
    [PDF] [bib]

  • ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learning
    Jannis Becktepe, Julian Dierkes, Carolin Benjamins, Aditya Mohan, David Salinas, Raghu Rajan, Frank Hutter, Holger Hoos, Marius Lindauer, Theresa Eimer, (3):1−41, 2026.
    Abstract

    Hyperparameters are a critical factor in reliably training well-performing reinforcement learning (RL) agents. Unfortunately, developing and evaluating automated approaches for tuning such hyperparameters is both costly and time-consuming. As a result, such approaches are often only evaluated on a single domain or algorithm, making comparisons difficult and limiting insights into their generalizability. We propose ARLBench, a benchmark for hyperparameter optimization (HPO) in RL that allows comparisons of diverse HPO approaches while being highly efficient in evaluation. To enable research into HPO in RL, even in settings with low compute resources, we select a representative subset of HPO tasks spanning a variety of algorithm and environment combinations. This selection allows for generating a performance profile of an automated RL (AutoRL) method using only a fraction of the compute previously necessary, enabling a broader range of researchers to work on HPO in RL. With the extensive and large-scale dataset on hyperparameter landscapes that our selection is based on, ARLBench is an efficient, flexible, and future-oriented foundation for research on AutoRL. Both the benchmark and the dataset are available at https://github.com/automl/arlbench.
    [PDF] [bib]

  • SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Models
    Justus Westerhoff, Erblina Purelku, Jakob Hackstein, Jonas Loos, Leo Pinetzki, Erik Rodner, Lorenz Hufe, (4):1−30, 2026.
    Abstract

    Typographic attacks exploit the interplay between text and visual content in multimodal foundation models, causing misclassifications when misleading text is embedded within images. Existing datasets are limited in size and diversity, making it difficult to study such vulnerabilities. In this paper, we introduce SCAM, the largest and most diverse dataset of real-world typographic attack images to date, containing 1162 images across hundreds of object categories and attack words. Through extensive benchmarking of Vision-Language Models on SCAM, we demonstrate that typographic attacks significantly degrade performance, and identify that training data and model architecture influence the susceptibility to these attacks. Our findings indicate that typographic attacks remain effective against state-of-the-art Large Vision-Language Models, especially those employing vision encoders inherently vulnerable to such attacks. However, employing larger Large Language Model backbones reduces this vulnerability while simultaneously enhancing typographic understanding. Additionally, we demonstrate that synthetic attacks closely resemble real-world (handwritten) attacks, validating their use in research. Our work provides a comprehensive resource and empirical insights to facilitate future research toward robust and trustworthy multimodal AI systems. Finally, we publicly release the datasets introduced in this paper, along with the code for evaluations under https://www.bliss.berlin/research/scam
    [PDF] [bib]

  • Hierarchical and Multimodal Data for Daily Activity Understanding
    Ghazal Kaviani, Yavuz Yarici, Seulgi Kim, Mohit Prabhushankar, Ghassan AlRegib, Mashhour Solh, Ameya Patil, (5):1−30, 2026.
    Abstract

    Daily Activity Recordings for artificial intelligence (DARai, pronounced /Dahr-ree/), is a multimodal, hierarchically annotated dataset constructed to understand human activities in real-world settings. DARai consists of continuous scripted and unscripted recordings of 50 participants in 10 different environments, totaling over 200 hours of data from 20 sensors including multiple camera views, depth and radar sensors, wearable inertial measurement units (IMUs), electromyography (EMG), insole pressure sensors, biomonitor sensors, and gaze tracker. To capture the complexity in human activities, DARai is annotated at three levels of hierarchy: (i) high-level activities (L1) that are independent tasks, (ii) lower-level actions (L2) that are patterns shared between activities, and (iii) fine-grained procedures (L3) that detail the exact execution steps for actions. The unscripted nature of DARai enables the collection of action counterfactuals, defined as observed alternative executions of the same activity under different conditions (e.g., lifting a heavy versus a light object). Experiments with various machine learning models showcase the value of DARai in uncovering important challenges in human-centered applications. Specifically, we conduct unimodal and multimodal sensor fusion experiments for recognition, temporal localization, and future action anticipation across all hierarchical annotation levels. To showcase the shortcomings of individual sensors, we conduct domain-variant experiments that are possible because of DARai’s multi-sensor setup and its inclusion of action counterfactuals, i.e., observed alternative executions of the same activity. The code, documentation, and dataset is available at the dedicated DARai website.
    [PDF] [bib]

  • IntelliGraphs: Datasets for Benchmarking Knowledge Graph Generation
    Thiviyan Thanapalasingam, Emile van Krieken, Peter Bloem, Paul Groth, (6):1−34, 2026.
    Abstract

    Knowledge Graph Embedding (KGE) models are used to learn continuous representations of entities and relations, commonly trained to predict missing links between entities. However, Knowledge Graphs are not just sets of links but also have complex semantics underlying their structure. Semantics plays a crucial role in several downstream tasks, such as query answering and reasoning. Recognizing this, our work goes beyond simple link prediction to focus on inferred knowledge that adheres to rich semantics. Specifically, 1) we introduce the subgraph inference task, where a model is required to generate novel subgraphs that are logically consistent with background knowledge; 2) we propose IntelliGraphs, a set of five new datasets that contain subgraphs with logical rules that express complex semantics for evaluating subgraph inference models, and 3) we design four baseline models, which include three models based on traditional KGEs, and show empirically that the KGE-based baselines can not capture complex semantics. We believe that IntelliGraphs will encourage the development of machine learning models that focus on semantic understanding.
    [PDF] [bib]

  • FinSurvival: A Suite of Large Scale Survival Modeling Tasks from Finance
    Aaron Green, Zihan Nie, Hanzhen Qin, Oshani Seneviratne, Kristin Bennett, (7):1−32, 2026.
    Abstract

    Survival modeling predicts the time until an event occurs and is widely used in risk analysis; for example, it’s used in medicine to predict the survival of a patient based on censored data. There is a need for large-scale, realistic, and freely available datasets for benchmarking artificial intelligence (AI) survival models. In this paper, we derive a suite of 16 survival modeling tasks from publicly available transaction data generated by lending of cryptocurrencies in Decentralized Finance (DeFi). Each task was constructed using an automated pipeline based on choices of index and outcome events. For example, the model predicts the time from when a user borrows cryptocurrency coins (index event) until their first repayment (outcome event). We formulate a survival benchmark consisting of a suite of 16 survival-time prediction tasks (FinSurvival). We also automatically create 16 corresponding classification problems for each task by thresholding the survival time using the restricted mean survival time. With over 7.5 million records, FinSurvival provides a suite of realistic financial modeling tasks that will spur future AI survival modeling research. Our evaluation indicated that these are challenging tasks that are not well addressed by existing methods. FinSurvival enables the evaluation of AI survival models applicable to traditional finance, industry, medicine, and commerce, which is currently hindered by the lack of large public datasets. Our benchmark demonstrates how AI models could assess opportunities and risks in DeFi. In the future, the FinSurvival benchmark pipeline can be used to create new benchmarks by incorporating more DeFi transactions and protocols as the use of cryptocurrency grows.
    [PDF] [bib]

  • Time Series Machine Learning for Classifying Electroencephalograms
    Aiden Rushbrooke, Saber Sami, Matthew Middlehurst, Tony Bagnall, (8):1−26, 2026.
    Abstract

    Electroencephalography (EEG) is a crucial tool across neuroscience domains, including medical diagnostics, psychological research, and brain-computer interfacing (BCI). Its popularity is due to its non-invasiveness, high temporal resolution, and cost-effectiveness. The task of EEG classification, which involves learning to predict class labels associated with EEG segments based on previously observed data. This task is fundamental yet complex, given the high dimensionality, variability, and subject-specific nuances inherent in EEG data. We systematically evaluate recent advances in general-purpose time series machine learning (TSML) approaches to EEG classification. We present an EEG classification archive of 30 benchmark datasets, spanning diverse applications from clinical diagnostics to cognitive and BCI tasks. Our empirical evaluation compares traditional EEG approaches, deep learning models, Riemannian geometry-based classifiers, and state-of-the-art time series machine learning algorithms on this new benchmark. We find that one algorithm, a meta ensemble called HIVE-COTE v2.0, consistently outperforms alternative classifiers.
    [PDF] [bib]

  • LOCKED: A Dataset of Sociodemographic, Economic, Health and Living Features to Assess Mental Health Impact of the Spanish Lockdown during COVID-19
    Alberto Nogales, Alfredo Guitian, Blanca Mellor-Marsá, Alvaro J. García-Tejedor, (9):1−37, 2026.
    Abstract

    The COVID-19 pandemic, which began in late 2019 in Wuhan, China, quickly escalated into a global crisis that affected nearly every aspect of life. Governments around the world implemented stringent public health measures to control the spread, including quarantines, social distancing, and lockdowns. In Spain, where the first cases emerged in January 2020, a nationwide lockdown was imposed on 14 March after the number of infections exceeded 5,000. Although these interventions were crucial for public health, they also presented significant social challenges, particularly for vulnerable groups. The primary goal of this study is to describe a dataset that combines psychological assessments of nine mental health conditions with sociodemographic, economic, living, and general health features. The data collected could potentially be used to identify the key factors that influenced mental health outcomes during lockdown. By analysing these data, the research seeks to shed light on the broader psychological effects of the pandemic and the factors that can exacerbate or mitigate these impacts. As an added value and to demonstrate the quality and potential of the dataset for mental health research, baseline machine learning models were developed, achieving performance metrics that exceed 80%. LOCKED is publicly available at https://zenodo.org/uploads/14203988.
    [PDF] [bib]

  • A Benchmark Dataset for Graph Regression with Homogeneous and Multi-Relational Variants
    Peter Samoaa, Marcus Vukojevic, Morteza Haghir Chehreghani, Antonio Longa, (10):1−41, 2026.
    Abstract

    Graph-level regression underpins many real-world applications, yet public benchmarks remain heavily skewed toward molecular graphs and citation networks. This limited diversity hinders progress on models that must generalize across both homogeneous and heterogeneous graph structures. We introduce RelSC, a new graph-regression dataset built from program graphs that combine syntactic and semantic information extracted from source code. Each graph is labelled with the execution-time cost of the corresponding program, providing a continuous target variable that differs markedly from those found in existing benchmarks. RelSC is released in two complementary variants. TypeOne supplies rich node features under a single (homogeneous) edge type, while TypeTwo preserves the original multi-relational structure, connecting nodes through multiple edge types that encode distinct semantic relationships. Together, these variants let researchers probe how representation choice influences model behaviour. We evaluate a diverse set of graph neural network architectures on both variants of RelSC. The results reveal consistent performance differences between the homogeneous and multi-relational settings, emphasising the importance of structural representation. These findings demonstrate RelSC’s value as a challenging and versatile benchmark for advancing graph regression methods.
    [PDF] [bib]

  • Putting Data at the Centre of Offline Multi-Agent Reinforcement Learning
    Juan Claude Formanek, Louise Beyers, Callum Rhys Tilbury, Jonathan Phillip Shock, Arnu Pretorius, (11):1−24, 2026.
    Abstract

    Offline multi-agent reinforcement learning (MARL) is an exciting direction of research that uses static datasets to find optimal control policies for multi-agent systems. Though the field is by definition data-driven, efforts have thus far neglected data in their drive to achieve state-of-the-art results. We first substantiate this claim by surveying the literature, showing how the majority of works generate their own datasets without consistent methodology and provide sparse information about the characteristics of these datasets. We then show why neglecting the nature of the data is problematic, through salient examples of how tightly algorithmic performance is coupled to the dataset used, necessitating a common foundation for experiments in the field. In response, we take a big step towards improving data usage and data awareness in offline MARL, with three key contributions: (1) a clear guideline for generating novel datasets; (2) a standardisation of over 80 existing datasets, hosted in a publicly available repository, using a consistent storage format and easy-to-use API; and (3) a suite of analysis tools that allow us to understand these datasets better, aiding further development.
    [PDF] [bib]

  • AI Competitions and Benchmarks book - Practical issues and open problems
    Andrey Ustyuzhanin, Harald Carlens, (12):1−68, 2026.
    Abstract

    The ecosystem of artificial intelligence contests is diverse and multifaceted, encompassing several platforms that each host numerous competitions and challenges annually, alongside many specialized websites dedicated to individual contests. These platforms manage the overarching administrative responsibilities inherent in orchestrating contests, thus allowing organizers to allocate greater attention to other aspects of their contests. Notably, these platforms exhibit considerable variety in their features, economic models, and communities. This chapter conducts an extensive review of the leading services in this space and explores alternative methods facilitating the independent hosting of such contests. We provide hints and tips on choosing the right platform for your challenge at the end.
    [PDF] [bib]

  • CFDLLMBench: A Benchmark Suite for Evaluating Large Language Models in Computational Fluid Dynamics
    Nithin Somasekharan, Ling Yue, Yadi Cao, Weichao LI, Patrick Emami, Pochinapeddi Sai Bhargav, Anurag Acharya, Xingyu Xie, Shaowu Pan, (13):1−40, 2026.
    Abstract

    Large Language Models (LLMs) have demonstrated strong performance across general NLP tasks, but their utility in automating numerical experiments of complex physical systems and labor-intensive components remains underexplored. As the major workhorse of computational science over the past decades, Computational Fluid Dynamics (CFD) offers a uniquely challenging testbed for evaluating the scientific capabilities of LLMs. We introduce CFDLLMBench, a benchmark suite comprising three complementary components: CFDQuery, CFDCodeBench, and FoamBench, designed to holistically evaluate LLM performance across three key competencies: graduate-level CFD knowledge, numerical and physical reasoning of CFD, and context-dependent implementation of CFD workflows. Grounded in real-world CFD practices, our benchmark combines a detailed task taxonomy with a rigorous evaluation framework to deliver reproducible results and quantify LLM performance across code executability, solution accuracy, and numerical convergence behavior. CFDLLMBench establishes a solid foundation for the development and evaluation of LLM-driven automation of numerical experiments for complex physical systems. Code and data are available at https://github.com/NLR-Theseus/cfdllmbench/.
    [PDF] [bib]

  • AIDOVECL: AI-generated Dataset of Outpainted Vehicles for Eye-level Classification and Localization
    Amir Kazemi, Qurat ul ain Fatima, Volodymyr Kindratenko, Christopher W Tessum, (14):1−34, 2026.
    Abstract

    Image labeling is a critical bottleneck in the development of computer vision technologies, often constraining machine learning performance due to the time-intensive nature of manual annotations. This work introduces a novel approach that leverages outpainting to mitigate annotated data scarcity by generating artificial contexts and annotations, significantly reducing labeling efforts. We apply this technique to a particularly acute challenge in autonomous driving, urban planning, and environmental monitoring: the lack of diverse, eye-level vehicle images from desired classes. Our dataset comprises AI-generated vehicle images obtained by detecting and cropping vehicles from manually selected seed images, which are then outpainted onto larger canvases to simulate varied real-world conditions. The outpainted images include detailed annotations, providing high-quality ground truth data. Advanced outpainting techniques and image quality assessments ensure visual fidelity and contextual relevance. Ablation results show that incorporating AIDOVECL improves overall detection performance by up to about 10%, and delivers gains of up to about 40% in settings with greater diversity of context, object scale, and placement, with underrepresented classes achieving up to about 50% higher true positives. AIDOVECL enhances vehicle detection by augmenting real training data and supporting evaluation across diverse scenarios. By demonstrating outpainting as an automatic annotation paradigm, it offers a practical and versatile solution for building fine-grained datasets with reduced labeling effort across multiple machine learning domains. The code and links to datasets are available for further research and replication at https://github.com/amir-kazemi/aidovecl.
    [PDF] [bib]

  • MOMAland: A Set of Benchmarks for Multi-Objective Multi-Agent Reinforcement Learning
    Florian Felten, Umut Ucak, Hicham Azmani, Gao Peng, Willem Röpke, Hendrik Baier, Patrick Mannion, Diederik M Roijers, J K Terry, El Ghazali Talbi, Grégoire Danoy, Ann Nowe, Roxana Rădulescu, (15):1−49, 2026.
    Abstract

    Many challenging tasks such as managing traffic systems, electricity grids, or supply chains involve complex decision-making processes that must balance multiple conflicting objectives and coordinate the actions of various independent decision-makers (DMs). One perspective for formalising and addressing such tasks is multi-objective multi-agent reinforcement learning (MOMARL). MOMARL broadens reinforcement learning (RL) to problems with multiple agents each needing to consider multiple objectives in their learning process. In reinforcement learning research, benchmarks are crucial in facilitating progress, evaluation, and reproducibility. The significance of benchmarks is underscored by the existence of numerous benchmark frameworks developed for various RL paradigms, including single-agent RL (e.g., Gymnasium), multi-agent RL (e.g., PettingZoo), and single-agent multi-objective RL (e.g., MO-Gymnasium). To support the advancement of the MOMARL field, we introduce MOMAland, the first collection of standardised environments for multi-objective multi-agent reinforcement learning. MOMAland addresses the need for comprehensive benchmarking in this emerging field, offering over 10 diverse environments that vary in the number of agents, state representations, reward structures, and utility considerations. To provide strong baselines for future research, MOMAland also includes algorithms capable of learning policies in such settings.
    [PDF] [bib]

  • Wake Vision: A Tailored Dataset and Benchmark Suite for TinyML Computer Vision Applications
    Colby Banbury, Emil Njor, Andrea Mattia Garavagno, Mark Mazumder, Pete Warden, Manjunath Kudlur, Nat Jeffries, Xenofon Fafoutis, Vijay Janapa Reddi, (16):1−48, 2026.
    Abstract

    Tiny machine learning (TinyML) co-locates models with sensors on microcontrollers, where small models (which are disproportionately sensitive to label noise) and bespoke binary tasks (which lack standard benchmarks) make general-purpose dataset practices a poor fit. Visual Wake Words (VWW), the prior standard TinyML person detection benchmark, contains roughly 123K images and has an estimated label error rate of 7.8%, which limits its usefulness for production-grade systems. Manual labeling, however, is prohibitively expensive for the scale and diversity of TinyML use cases. We address this gap with the Wake Vision pipeline, an automated method for generating and curating large-scale binary classification datasets for TinyML. We use data-centric TinyML for the dataset construction, curation, and lifecycle methods that produce the large, well-curated datasets these systems require. The pipeline combines label fusion across image-level and bounding-box sources, confidence-, area-, and depiction-aware filtering, label correction on the evaluation splits, and automatic generation of fine-grained benchmark subsets. Applying it to person detection, we release Wake Vision, a dataset of almost 6M images (close to 100× more person images than VWW) with a manually relabeled validation and test set at a 2.2% label error rate. Models trained on Wake Vision improve test accuracy by up to 6.6% over VWW across MobileNetV2, MCUNet, MicroNets, and ColabNAS architectures, and match or exceed VWW-trained models on 13 of 16 fine-grained subsets covering perceived gender, perceived age, distance, lighting, and depictions. The advantage holds under distribution shift on three out-of-distribution datasets covering driving and overhead-surveillance imagery. We additionally uncover two TinyML-specific insights: small models are more sensitive to label errors than large models, and two-stage training, which pretrains on the noisier large set and fine-tunes on the cleaner small set, is a viable strategy even for tiny, low-capacity models. Beyond person detection, the Wake Vision pipeline applies to the 9.6K trainable classes of Open Images v7; on bird detection it produces a dataset 27× larger than a VWW-style baseline with label error reduced from 6.6% to 0.6%. All artifacts are released under CC-BY 4.0 through TensorFlow Datasets and Hugging Face. To continue improving the dataset over time, we partner with the Edge AI Foundation to host community competitions; the first round contributed a label-correction technique that reduced the Wake Vision (Large) label error rate from 15.2% to 9.8%, at a cost orders of magnitude below the $600,000 implied by manual relabeling at this scale.
    [PDF] [bib]

  • Academic Competitions
    Hugo Jair Escalante, Aleksandra Kruchinina, (17):1−39, 2026.
    Abstract

    Competitions comprise effective means for (i) advancing the state of the art, (ii) putting in the spotlight of a scientific community specific topics and problems, as well as (iii) closing the gap for under represented communities in terms of accessing and participating in the shaping of research fields. Competitions can be traced back for centuries and their achievements have had great influence in our modern world. Recently, they (re)gained popularity, with the overwhelming amounts of data that is being generated in different domains, as well as the need of pushing the barriers of existing methods, and available tools to handle such data. This chapter provides a survey of academic challenges in the context of machine learning and related fields. We review the most influential competitions in the last few years and analyze challenges per area of knowledge. The aims of scientific challenges, their goals, major achievements and expectations for the next few years are reviewed. An associated repository is available here: https://hugojair.github.io/challenges-survey/
    [PDF] [bib]

  • How to judge a competition: Fairly judging a competition or assessing benchmark results
    Adrien Pavão, (18):1−42, 2026.
    Abstract

    This chapter addresses how to minimize randomness in competition or benchmark judging. We discuss scoring metrics, size of test data, error bars, splitting into phases, and score aggregation methods. Our approach blends theoretical insights with practical guidelines, aiming to provide a clear framework for effective decision making and reduced uncertainty.
    [PDF] [bib]

  • The CLRS-Text Algorithmic Reasoning Language Benchmark
    Larisa Markeeva, Sean Michael McLeish, Borja Ibarz, Wilfried Bounsi, Olga Kozlova, Alex Vitvitskyi, Charles Blundell, Tom Goldstein, Avi Schwarzschild, Petar Veličković, (19):1−21, 2026.
    Abstract

    Eliciting reasoning capabilities from language models (LMs) is a critical direction on the path towards building intelligent systems. Many foundational studies dedicated to reasoning focus on out-of-distribution performance on procedurally-generated synthetic benchmarks, bespoke-built to evaluate specific skills only. This trend makes results hard to transfer across publications, slowing down progress. Several years ago, a similar issue was identified and rectified in the field of neural algorithmic reasoning, with the advent of the CLRS benchmark. CLRS is a dataset generator comprising graph execution traces of classical algorithms from the Introduction to Algorithms textbook. Inspired by this, we propose CLRS-Text, a textual version of these algorithmic traces. Out of the box, CLRS-Text is capable of procedurally generating trace data for thirty diverse, challenging algorithmic tasks across any desirable input distribution, while offering a standard pipeline in which any additional algorithmic tasks may be created in the benchmark. We fine-tune and evaluate various LMs as generalist executors on this benchmark, validating prior work and revealing a novel, interesting challenge for the LM reasoning community. Our code is available at https://github.com/google-deepmind/clrs/tree/master/clrs/_src/clrs_text.
    [PDF] [bib]

  • CleanPatrick: A Benchmark for Image Data Cleaning
    Fabian Gröger, Simone Lionetti, Philippe Gottfrois, Alvaro Gonzalez-Jimenez, Ludovic Amruthalingam, Elisabeth Victoria Goessinger, Hanna Lindemann, Marie Bargiela, Marie Hofbauer, Omar Badri, Philipp Tschandl, Arash Koochek, Matthew Groh, Alexander A. Navarini, Marc Pouly, (20):1−31, 2026.
    Abstract

    Robust machine learning depends on clean data, yet current image data cleaning benchmarks rely on synthetic noise or narrow human studies, limiting comparison and real-world relevance. We introduce CleanPatrick, the first large-scale benchmark for data cleaning in the image domain, built upon the publicly available Fitzpatrick17k dermatology dataset. We collect 496,377 binary annotations from 933 medical crowd workers, identify off-topic samples (4%), near-duplicates (21%), and label errors (32%), and employ an aggregation model inspired by item-response theory followed by expert review to derive high-quality ground truth. CleanPatrick formalizes issue detection as a ranking task and employs standard ranking metrics that mirror real audit workflows. We benchmark classical anomaly detectors, perceptual hashing, SSIM, Confident Learning, NoiseRank, FINE, BHN, and SelfClean. On CleanPatrick, self-supervised representations excel at near-duplicate detection, classical methods achieve competitive off-topic detection under constrained review budgets, and detecting implausible labels under conservative human judgment remains challenging for fine-grained medical classification. By releasing both the dataset and the evaluation framework, CleanPatrick enables a systematic comparison of image-cleaning strategies.
    [PDF] [bib]

  • AutoML to Advance Earth Observation
    Julia Wąsala, Joannes D. Maasakkers, Ilse Aben, Rochelle Schneider, Holger H. Hoos, Mitra Baratchi, (21):1−58, 2026.
    Abstract

    Machine learning (ML) techniques have become crucial enablers of progress in Earth observation (EO) research by providing data-driven solutions to effectively leverage the large volumes of available satellite imagery. However, configuring ML pipelines demands substantial time and expertise, making it challenging to consistently develop robust, high-performing ML pipelines for EO applications. Automated machine learning (AutoML), the study of automatic configuration and design of methods and tools for ML systems, has emerged as a solution to this problem. Key AutoML techniques, such as neural architecture search (NAS) and hyperparameter optimisation (HPO), are increasingly applied to EO-related problems. Developing AutoML for processing EO data is challenging, as automation removes human interventions and domain knowledge that can ensure the development of robust Machine Learning for Earth observation (ML4EO) models for new applications. While both AutoML and ML for EO (ML4EO) techniques have been widely studied and reviewed independently, no existing survey provides a systematic overview of their intersection, including the domain-specific challenges that arise when applying AutoML in an EO context. To address this gap, we present a survey of AutoML4EO through both data-centric and model-centric perspectives, covering the application of existing AutoML systems to EO problems and the design of entirely new systems. Finally, we highlight specific characteristics of EO data that are relevant to the development and evaluation of AutoML4EO systems and provide a perspective on challenges that must be addressed to create the next generation of AutoML4EO systems.
    [PDF] [bib]

  • Decomposing Complex Visual Comprehension into Atomic Visual Skills for Vision Language Models
    Hyunsik Chae, Seungwoo Yoon, Jaden Park, Chloe Yewon Chun, Yongin Cho, Mu Cai, Yong Jae Lee, Ernest K. Ryu, (22):1−93, 2026.
    Abstract

    Recent Vision-Language Models (VLMs) have demonstrated impressive multimodal comprehension and reasoning capabilities, yet they often struggle with trivially simple visual tasks. In this work, we focus on the domain of basic 2D Euclidean geometry and systematically categorize the fundamental, indivisible visual perception skills, which we refer to as atomic visual skills. We then introduce the Atomic Visual Skills Dataset (AVSD) for evaluating VLMs on the atomic visual skills. Using AVSD, we benchmark state-of-the-art VLMs and find that they struggle with these tasks, despite being trivial for adult humans. Our findings highlight the need for purpose-built datasets to train and evaluate VLMs on atomic, rather than composite, visual perception tasks.
    [PDF] [bib]

  • European Space Agency Dataset and Benchmark for Real-World Anomaly Detection in Spacecraft Time Series
    Krzysztof Kotowski, Christoph Haskamp, Jacek Andrzejewski, Bogdan Ruszczak, Jakub Nalepa, Daniel Lakey, Peter Collins, Jose Martinez Heras, Gabriele De Canio, (23):1−76, 2026.
    Abstract

    Time series from spacecraft sensors are high-dimensional, nonstationary, nonlinear, irregularly sampled, and exhibit both spatial and temporal dependencies. Detecting anomalies in such signals is critical for both on-ground and in-orbit space operations. The potential of machine learning in this task is currently hampered by a lack of comprehensive datasets and benchmarks that capture its real-world complexity. The European Space Agency Anomaly Detection Benchmark (ESA-ADB) addresses this issue and establishes a new standard in the domain. It is a result of close cooperation between engineers from the European Space Operations Center and machine learning experts from industry and academia. Our newly introduced dataset (doi.org/10.5281/zenodo.12528695) contains several years of real-life raw data from 3 large spacecraft, including 224 channels, 821 control signals, and 1430 annotated events, which makes it the biggest dataset of its kind in the literature. The associated benchmark defines 9 specific requirements and 5 evaluation metrics for assessing anomaly detection algorithms in operational practice. The results indicate that widely used anomaly detection algorithms, even with our proposed adaptations, are not yet suitable for effective deployment. Thus, ESA-ADB remains an open challenge and continues to be explored through dedicated competitions and related projects listed on the official project webpage (ai4gs.space-codev.org/esa-adb).
    [PDF] [bib]

  • CycliST: A Video Language Model Benchmark for Reasoning on Cyclical State Transitions
    Simon Kohaut, Daniel Ochs, Shun Zhang, Benedict Flade, Julian Eggert, Kristian Kersting, Devendra Singh Dhami, (24):1−31, 2026.
    Abstract

    We present CycliST, a novel benchmark dataset designed to evaluate Video Language Models (VLM) on their reasoning over cyclical state transitions. CycliST captures fundamental aspects of real-world processes in synthetic, richly structured video sequences featuring periodic visual patterns. Furthermore, CycliST offers a tiered evaluation, providing increasing difficulty levels by varying the number of cyclic objects, scene clutter, and lighting conditions. We conduct extensive experiments with current open-source and proprietary state-of-the-art VLMs and reveal their limitations in generalizing to cyclical dynamics, such as linear and orbital motion, as well as to time-dependent changes in visual attributes like color and scale. Our results demonstrate that present-day VLMs struggle to reliably detect and exploit cyclic patterns, lack a notion of temporal understanding, and are unable to extract quantitative insights from scenes, such as the number of objects in motion, highlighting a significant technical gap that needs to be addressed. More specifically, we find no single model consistently outperforms others: neither size nor architecture correlates strongly with outcomes, and no model performs equally well across all tasks. By providing a targeted challenge and a comprehensive evaluation framework, CycliST paves the way for visual reasoning models that surpass the state-of-the-art in understanding periodic patterns.
    [PDF] [bib]

  • The Unreasonable Benchmark
    Fernando Perez-Cruz, Briland Hitaj, Giovanna Maria Dimitri, Oishi Deb, Amine M'Charrak, Shikhar Srivastava, Christopher Kanan, Anup Kumar Gupta, Zefang Liu, Zihan Zhu, Tarun Kumar, Álvaro Galisteo Bermúdez, Vasilios Mavroudis, Yuyang Xue, Brian Matejek, Swati Rajwal, Avinash Kumar Pandey, Ayyüce Begüm Bektaş, Anil B Murthy, Feng Chen, Hamed Babaei Giglou, Jennifer D'Souza, Huascar Sanchez, Michael Kraus, Isamu Lautenschläger, Ziyang Wang, Xuanli He, Hyun Song Shin, (25):1−17, 2026.
    Abstract

    Effective benchmarking is instrumental for the systematic evaluation of Large Language Models (LLMs) performance, providing vital insights in the path towards advancements in Artificial Intelligence (AI). Existing benchmarks, such as Humanity’s Last Exam and FrontierMath, stress-test the boundaries of the model’s performance. However, they provide limited understanding of a model’s reliability in routine, low-difficulty scenarios. It is, however, these contexts, which, despite their simplicity, frequently expose persistent model weaknesses. To address this, we present the Dataset, a complementary benchmark aiming to address this oversight via systematic (and continuous) evaluation of LLMs’ performance in basic reasoning and everyday tasks. It is the end result of a crowdsourcing effort seeking to ensure a diversity of perspectives and topic coverage. Moreover, the dataset is designed to be dynamic in nature; it incorporates new items as emerging failure modes are identified while retiring resolved items, thereby maintaining relevance over time. Similar to previous efforts, such as TruthfulQA, the primary objective of this new benchmark is to identify persistent shortcomings and support the development of robust and reliable LLMs. Ultimately, this process should converge upon a set of unreasonable issues, at which point we can be confident that the LLMs will indeed exhibit impressive and resilient capabilities. The set of (multimodal) questions is available at https://huggingface.co/datasets/unreasonablebenchmark/unreasonable-benchmark, under a Creative Commons Attribution-ShareAlike (CC BY-SA) license.
    [PDF] [bib]

  • Dataset Development
    Romain Egele, Julio Cezar Silveira Jacques Junior, Jan N. van Rijn, Isabelle Guyon, Xavier Baro, Albert Clapés, Prasanna Balaprakash, Sergio Escalera, Thomas B. Moeslund, Jun Wan, Walter Reade, (26):1−46, 2026.
    Abstract

    Machine learning is now used in many applications due to its ability to predict, generate, or discover patterns from large quantities of data. However, the process of collecting and transforming data for practical use is intricate. Even in today’s digital era, where substantial data is generated daily, it is uncommon for it to be readily usable; most often, it necessitates meticulous manual data preparation. The haste in developing new models can frequently result in various shortcomings, potentially posing risks when deployed in real-world scenarios (e.g., social discrimination, critical failures), leading to the failure or substantial escalation of costs in AI-based projects. In this chapter, we propose a comprehensive framework for dataset development. The framework consists of several stages (i.e., requirements, design, implementation, evaluation, distribution, and maintenance), each consisting of a set of possible operators (e.g., data cleaning or data reduction). We describe the various operators in detail. Finally, we address practical considerations regarding dataset distribution and maintenance. While the framework is partially based on our experience, we aim to substantiate the steps with scientific references where possible.
    [PDF] [bib]

  • VIBE: Vector Index Benchmark for Embeddings
    Elias Jääsaari, Ville Hyvönen, Matteo Ceccarello, Teemu Roos, Martin Aumüller, (27):1−42, 2026.
    Abstract

    Approximate nearest neighbor (ANN) search is a performance-critical component of many machine learning pipelines, and rigorous benchmarking is essential for assessing the performance of vector indexes for ANN search. However, the datasets of existing benchmarks no longer represent modern ANN applications, creating a need for an up-to-date benchmark. To address this gap, we introduce Vector Index Benchmark for Embeddings (VIBE), an open-source framework for benchmarking ANN algorithms. VIBE provides a pipeline for generating benchmark datasets with dense embedding models representative of modern applications, including retrieval-augmented generation (RAG). To represent real-world workloads, we also include out-of-distribution (OOD) datasets where the queries and the corpus are drawn from different distributions. These include multimodal retrieval datasets and maximum inner product search (MIPS) datasets covering two recent use cases: approximate attention computation and reductions of multi-vector retrieval to single-vector MIPS. We use VIBE to conduct a comprehensive evaluation of 22 open-source vector-index implementations across 11 in-distribution and 8 out-of-distribution datasets. The benchmark is available at https://github.com/vector-index-bench/vibe
    [PDF] [bib]

  • SloMoDeblur: A Large-Scale Smartphone Image Deblurring Dataset
    Syed Mumtahin Mahmud, Mahdi Mohd Hossain Noki, Prothito Shovon Majumder, Abdul Mohaimen Al Radi, Sudipto Das Sukanto, Afia Lubaina, Md Mosaddek Khan, (28):1−18, 2026.
    Abstract

    Motion blur remains one of the most common and visually disruptive degradations in real-world smartphone imaging, yet existing deblurring benchmarks are often limited in scale, resolution, or domain relevance. This gap is especially pronounced for smartphones, where rolling shutter, small sensors, and ISP processing produce blur statistics that differ from GoPro/DSLR-based benchmarks. We introduce a large-scale smartphone-oriented deblurring dataset constructed from 240 fps slow-motion video. To approximate exposure-time radiance integration, we synthesize blur by temporally averaging a fixed window of N = 30 consecutive frames, which corresponds to an effective exposure of T = 1/8 second, and we select the temporally centered frame as the sharp ground truth. The resulting benchmark contains 42,045 paired blur–sharp images at 1920×1080 resolution spanning 843 distinct scenes, with a train/test split of 37,841/4,204 pairs. We benchmark multiple state-of-the-art deblurring models using PSNR and SSIM and observe consistent performance degradation relative to the baseline similarity between the input blurry images and ground truth, underscoring the realism and difficulty of the proposed data. We release the dataset and generation scripts via HuggingFace to facilitate the development and evaluation of robust, deployment-oriented deblurring methods. The dataset is available at https://huggingface.co/datasets/masterda/SloMoBlur.
    [PDF] [bib]

  • PowerBench: A Benchmark Suite for Monitoring and Security in Distribution Systems
    Md Joshem Uddin, Roshni Anna Jacob, Damilola R. Olojede, Nanhong Liu, Baris Coskunuzer, Jie Zhang, (29):1−29, 2026.
    Abstract

    Modern power distribution networks face growing risks from physical faults and cyber intrusions, demanding robust data-driven monitoring for operational resilience. Yet, progress in this area is constrained by the lack of standardized, openly accessible datasets that reflect realistic grid behaviors. This paper presents PowerBench, a comprehensive benchmark suite designed to advance machine learning research for distribution network monitoring and security. PowerBench provides high-fidelity simulations of IEEE 34-, 123-, and 8500-bus feeders under diverse operating and attack scenarios, annotated with line status, cyber events, and system states. Each dataset includes fixed training–validation–test splits, task-specific metrics, and baseline implementations for outage detection and localization, cyberattack identification, and state estimation. Comparative experiments with representative graph neural networks and graph transformer models are reported to establish performance baselines. All datasets, simulation scripts, and evaluation tools are released as open-source resources to ensure transparency and facilitate community adoption. By unifying multiple diagnostic and estimation tasks within a realistic framework, PowerBench provides a common foundation for developing and benchmarking intelligent, trustworthy methods for smart grid applications.
    [PDF] [bib]

  • QuAnTS: Question Answering on Time Series
    Felix Divo, Maurice Kraus, Quoc Anh Nguyen, Hao Xue, Imran Razzak, Flora D. Salim, Kristian Kersting, Devendra Singh Dhami, (30):1−46, 2026.
    Abstract

    Text offers intuitive access to information. This can, in particular, complement the density of numerical time series, thereby enabling improved interactions with time series models to enhance accessibility and decision-making. While the creation of question-answering datasets and models has recently seen remarkable growth, most research focuses on question answering (QA) on vision and text, with time series receiving minute attention. To bridge this gap, we propose a challenging novel time series QA (TSQA) dataset, QuAnTS, for Question Answering on Time Series data. Specifically, we pose a wide variety of questions and answers about human motion in the form of tracked skeleton trajectories. We verify that the large-scale QuAnTS dataset is well-formed and comprehensive through extensive experiments. Thoroughly evaluating existing and newly proposed baselines then lays the groundwork for a deeper exploration of TSQA using QuAnTS. Additionally, we provide human performances as a key reference for gauging the practical usability of such models. We hope to encourage future research on interacting with time series models through text, enabling better decision-making and more transparent systems.
    [PDF] [bib]

  • Vox-Profile: A Speech Foundation Model Benchmark for Characterizing Diverse Speaker Traits and Speech Properties
    Tiantian Feng, Jihwan Lee, Anfeng Xu, Yoonjeong Lee, Thanathai Lertpetchpun, Xuan Shi, Helin Wang, Thomas Thebaud, Laureano Moro-Velazquez, Dani Byrd, Najim Dehak, Shrikanth Narayanan, (31):1−22, 2026.
    Abstract

    We introduce Vox-Profile, a comprehensive benchmark to characterize rich speaker traits and speech properties using speech foundation models. Unlike prior work that targets a single dimension of speaker traits, Vox-Profile provides holistic and multi-dimensional profiles spanning both static speaker traits (e.g., age, sex, accent, voice quality) and dynamic speech traits (e.g., emotion, speech flow). This benchmark is grounded in speech science and linguistics, developed with domain experts to accurately index speaker and speech characteristics. We report benchmark experiments using over 15 publicly available speech datasets and several widely used speech foundation models that target various static and dynamic speaker and speech properties. In addition to benchmark experiments, we showcase several downstream applications supported by Vox-Profile. First, we show that Vox-Profile can augment existing speech recognition datasets to analyze ASR performance variability. Vox-Profile is also used as a tool to evaluate the performance of speech generation systems. Finally, we assess the quality of our automated profiles through comparison with human evaluation and show convergent validity. Vox-Profile is publicly available at: https://github.com/tiantiaf0627/vox-profile-release.
    [PDF] [bib]

  • CIRCLED: A Multi-turn CIR Dataset with Consistent Dialogues across Domains
    Tomohisa Takeda, Yu-Chieh Lin, Yuji Nozawa, Youyang Ng, Osamu Torii, Yusuke Matsui, (32):1−33, 2026.
    Abstract

    Existing Multi-Turn Composed Image Retrieval (MTCIR) datasets lack dialogue-history consistency and are restricted to the fashion domain. To address these limitations, we construct CIRCLED by extending FashionIQ, CIRR, and CIRCO. In CIRCLED, the query at each turn progressively approaches the target image. Data are generated via a CIReVL-based retrieval pipeline and curated with multiple filters on retrieval success, turn length, consistency, and information redundancy to ensure quality. In total, we collect 22,608 multi-turn sessions across nine subsets, substantially exceeding Multi-turn FashionIQ (11,505 sessions) in both scale and generality. We further apply multiple baseline methods and quantitatively assess retrieval accuracy on CIRCLED. Our work provides a practical, high-quality benchmark to facilitate future research on multi-turn CIR. The dataset is publicly available at https://huggingface.co/datasets/tk1441/CIRCLED, and the code at https://github.com/mti-lab/circled.
    [PDF] [bib]

  • Defining and Benchmarking a Data-Centric Design Space for Brain Graph Construction
    Qinwen Ge, Roza G Bayrak, Anwar Said, Catie Chang, Xenofon D. Koutsoukos, Tyler Derr, (33):1−26, 2026.
    Abstract

    The construction of brain graphs from functional Magnetic Resonance Imaging (fMRI) data plays a crucial role in enabling graph machine learning for neuroimaging. However, current practices often rely on rigid pipelines that overlook critical data-centric choices in how brain graphs are constructed. In this work, we adopt a Data-Centric AI perspective and systematically define and benchmark a data-centric design space for brain graph construction, constrasting with primarily model-centric prior work. We organize this design space into three stages: temporal signal processing, topology extraction, and graph featurization. Our contributions lie less in novel components and more in evaluating how combinations of existing and modified techniques influence downstream performance. Specifically, we study high-amplitude BOLD signal filtering, sparsification and unification strategies for connectivity, alternative correlation metrics, and multi-view node and edge features, such as incorporating lagged dynamics. Experiments on the HCP1200 and ABIDE datasets show that systematic, validation-guided exploration of data-centric configurations yields substantial accuracy gains over a standard fixed pipeline, while the best-performing construction varies across datasets and settings. These findings highlight the critical role of upstream data decisions and underscore the importance of systematically exploring the data-centric design space for graph-based neuroimaging. Our code is available at https://github.com/GeQinwen/DataCentricBrainGraphs.
    [PDF] [bib]

  • MObyGaze: a film dataset of multimodal objectification densely annotated by experts
    Elisa Ancarani, Julie Tores, Lucile Sassatelli, Hui-Yin Wu, Rémy Sun, Paul Mouret, Clement Bergman, Léa Andolfi, Victor Ecrement, Thierry Devars, Magali GUARESI, Virginie Julliard, Sarah Lécossais, Frederic Precioso, (34):1−63, 2026.
    Abstract

    Characterizing and quantifying gender representation disparities in audiovisual storytelling contents is necessary to grasp how stereotypes may perpetuate on screen. In this article, we consider the high-level construct of objectification and introduce a new AI task to the ML community: characterize and quantify complex multimodal (visual, speech, audio) temporal patterns producing objectification in films. Building on film studies and psychology, we define the construct of objectification in a structured thesaurus involving 5 sub-constructs manifesting through 11 concepts spanning 3 modalities. We introduce the Multimodal Objectifying Gaze (MObyGaze) dataset, made of 20 movies annotated densely by experts for objectification levels and concepts over freely delimited segments: it amounts to 6072 segments over 43 hours of video with fine-grained localization and categorization. We formulate new video interpretation tasks, show the feasibility of multimodal objectification detection, and analyze data and model bias to propose improvements. We exemplify two applications of MObyGaze, showing how rich concept annotation can improve model reliability and explainability. We make our code and our dataset available to the community and described in the Croissant format: https://github.com/husky-helen/MObyGaze
    [PDF] [bib]

  • Beyond Borders: Quantifying Geographic Domain Shift in Automotive Perception
    Joe Murtagh, Reenu Mohandas, Tim Brophy, Ganesh Sistu, Ciarán Eising, (35):1−41, 2026.
    Abstract

    Automated driving systems rely on diverse training data, yet increasing privacy regulations restrict cross-border data sharing. This study shows that geographic separation is associated with a continuous, distance-based domain shift and progressively lower model performance, with distance acting as a proxy for latent environmental factors, challenging the assumption of binary in-domain versus out-of-domain behaviour. Using two European driving datasets, EuroCity Persons Dataset (ECP) and the Zenseact Open Dataset (ZOD), and one global dash-camera dataset OpenStreetView-5M (OSV-5M), we provide converging evidence at representation and task levels. Encoder semantic embeddings diverge systematically with geographic distance, yielding Pearson correlations of 0.476–0.632 for ECP, 0.570–0.711 for ZOD, and 0.404–0.568 for OSV-5M across three distributional metrics (Maximum Mean Discrepancy (MMD), Sliced Wasserstein Distance (SWD), Energy Distance (ED)), strengthening to 0.665–0.762 when controlling for confounding variables on ZOD. Downstream perception models exhibit corresponding performance degradation: object detection correlates at -0.512 to -0.585 on ECP and -0.563 to -0.659 on ZOD, while semantic segmentation shows effects of -0.503 to -0.726 across models. While this study is observational and does not establish causality, controlled analyses reduce confounding effects, and annotation spatial distributions show no statistically significant correlation with geographic distance, suggesting that annotation distribution differences are unlikely to be the primary driver of the observed trends. These findings suggest that geographically fragmented datasets may contribute to performance gaps when models are deployed substantially outside their training regions. We characterise geographic domain shift as continuous, with perception performance degrading progressively with increasing spatial separation rather than at a simple in-domain/out-of-domain boundary.
    [PDF] [bib]