Senior Python Engineer – AI Agent Evaluation
Mindrift (Toloka AI) is hiring a Senior Software Engineer focused on AI Agent Evaluation.
- Mindrift (Toloka AI) is hiring a Senior Software Engineer
- Role focuses on AI Agent Evaluation
Topic stream
Research updates reveal the next wave of product capabilities and infrastructure needs. This hub follows papers, benchmarks, datasets, lab systems, releases, and open reproductions, focusing on which results may reach model training, agent systems, robotics, or developer tools.
Mindrift (Toloka AI) is hiring a Senior Software Engineer focused on AI Agent Evaluation.
A debate between AI models explores whether AI-native tools mark the end or a new beginning for solo founders. Consensus: execution cost collapse but discovery becomes key. Divergence on what replaces coding as the moat—human relationships vs. canonical/workflow embedment.
The author argues from a meta-epistemic perspective that philosophy journals and correspondence should reject AI-generated texts, because human experts' word choices (even subtle ones) reflect deep engagement with the subject, while LLM outputs blur that expertise. Through a case study on Klara and the Sun, he shows how AI rewrites lose crucial philosophical nuance. He also offers guidelines for using LLMs as editing assistants.
Gigatoken, an MIT-licensed Rust BPE tokenizer developed by Stanford PhD student Marcel Rød, encodes GPT-2 text at 24.53 GB/s on a 144-core AMD EPYC 9565, achieving 989x speedup over HuggingFace tokenizers and 681x over tiktoken. Gains come from a hand-written SWAR pretokenizer and pretoken caching, not a faster BPE merge loop. It supports 23 tokenizer families, though SentencePiece vocabularies see only 7–22x speedups. Compatibility mode preserves exact output parity at roughly 200–300x speedup.
Parker and Tom, along with 7 other economists, coauthored a paper analyzing simple models of how AI may accelerate AI R&D. They clarify definitions of recursive self-improvement (RSI), emphasize that the strength of feedback effects determines capability acceleration, and call for labs to release more relevant data. They cannot rule out substantial acceleration despite potential bottlenecks.
Ours.network introduces ours-mcp, a tool that enables AI agents to communicate directly without human intervention. It simplifies setup with an installable MCP server that allows agents to connect via one-time invites, bypassing the need for manual copy-paste. Features include end-to-end encryption, a blind relay for privacy, and full human control over connections. The tool is in early alpha, source-available, and designed for agent-to-agent communication across different runtimes like Claude Code and Codex.
New research from The University of Manchester and Durham University finds that AI chatbots can match or outperform humans in everyday emotional support, particularly in anger and fear contexts. The key to effective support is providing specific, actionable guidance, regardless of the source.
BeamWire delivers personalized, ad-free daily news briefs as email and podcast, with fact-checking, bias detection, and customizable topics. It offers multiple news and feature 'Beams' across various interests, AI anchors, and tone customization. Pricing starts free.
PyPI now rejects new file uploads to releases older than 14 days to prevent supply-chain attacks. This closes a potential vulnerability that could be exploited if publishing tokens are compromised.
Orphograph generates Bitcoin-anchored receipts for each consequential AI agent action, ensuring the record is dated, tamper-evident, and verifiable without trusting the operator.
Searchdesk is an AI-powered job search tool that researches real openings, prepares fact-based application materials, and keeps every opportunity organized in a private workspace. Users review all drafts before any action is taken, ensuring full control. Currently in alpha, feedback is welcome.
This paper proposes EgoRecovery, a framework that uses egocentric human video data to train robot failure recovery policies, achieving over 10x data collection efficiency compared to robot teleoperation, and aligning human corrective intent to robot actions via co-training.
Researchers present Morphing MILR, a cable-driven limbless robot with rolling joints that can reconfigure its body morphology and compliance to achieve multiple locomotion modes such as lateral undulation, sidewinding, rolling, and twisting. The robot uses distributed cable actuation and programmable passive compliance for robust locomotion without complex sensing. Applications include search and rescue, environmental monitoring, and inspection.
A new control-theoretic framework called 'contact-persistent full actuation' is introduced for UAVs during physical interaction. It defines residual wrench sets and residual authority margins, going beyond rank-based certification. Numerical tests on a tilted hexarotor show that full row rank does not guarantee feasible contact; intermediate tilt angles preserve residual authority.
This paper presents a decentralized, spoofing-aware trajectory planning framework for small unmanned aerial systems under Remote Identification (RID) location spoofing attacks. Unlike prior work that assumes RID is trustworthy, the proposed approach treats RID as unverified and uses received signal strength measurements to detect spoofing and probabilistically localize the attacker. The resulting uncertainty is converted into a risk-bounded unsafe region via chance constraints and integrated into a per-agent Markov decision process planner. Simulations in a multi-aircraft package delivery scenario demonstrate reduced near mid-air collision events while maintaining computational efficiency.
Despite advances in general-purpose robotic manipulation, real-world multi-object clutter remains challenging. LENS, a plug-and-play fix, uses LLMs to automatically generate scene-specific, task-relevant abstractions by merging or pruning entities, improving performance across various manipulation methods.
A Learning from Haptics (LfH) framework is proposed to learn user-preferred safety interventions from sparse demonstrations using differentiable Control Barrier Functions. It eliminates manual tuning and adapts haptic feedback to individual preferences, as validated in simulations and hardware experiments.
Blind and low-vision individuals often rely on guide dogs for navigation, but these animals are expensive and have limited availability. Milo is a fully autonomous, low-cost robotic guide dog built on the Unitree Go2 platform, designed for indoor and outdoor use without prior environmental knowledge.
This study investigates when drifting may be optimal for safety in real-world winter driving. The team presents a drift-capable nonlinear MPC controller tested in high-fidelity simulations based on crash fatality data. The controller naturally initiates drifting to stay on the road when hitting ice on the rear axle and to avoid an oncoming vehicle that slid into its lane. Compared to electronic stability control, the drift-capable controller trades stability for controllability, achieving lower median lane error at higher speeds.
ModPack presents a modular, extensible teleoperation system centered on a wearable backpack that integrates computation, power, communication, and storage. It supports plug-and-play modules for joint-level teleoperation with haptic feedback, mobile manipulation, and active perception. Tests on two robot platforms confirm its flexibility and reusability for data collection and policy learning. The complete hardware and software stack is open-sourced.
A unified variational framework is proposed for image segmentation with sparse pixel-level supervision. It uses a simplex-constrained Potts model with a smooth perimeter regularizer, resulting in a convex, smooth energy functional usable as a training loss or for iterative optimization. Sparse labels are incorporated via a fuzzy membership function from an RKHS extension, capturing inhomogeneous intensity statistics. Experiments show robustness and consistent improvement over baselines without requiring ground-truth segmentation.
A study across six echocardiographic datasets finds that domain shift in left ventricular segmentation largely stems from field-of-view and framing inconsistencies, not acoustic differences. Geometry-aware preprocessing improves transfer, and representation-specific discrepancy measures can predict performance drops with high accuracy, supporting mask-free monitoring.
This work introduces pathologist attention into report generation model training. A multimodal dataset of 121 prostate WSIs with pathologists' gaze, verbal descriptions, and cursor movements was collected. Two models fine-tuned with an attention-alignment loss showed average gains of 10.9% on NLP metrics and 19.3% accuracy across five clinical report components.
This paper presents EGRNet, a lightweight deep learning model for real-time semantic segmentation in urban scenarios. With only 0.46M parameters, it achieves 65.28% mIoU on Cityscapes while incorporating depthwise separable convolutions, dilated residual blocks, a novel Edge-Gated Refinement module, and a lightweight adversarial attack detection strategy for robust edge deployment.
VQ-Transplant introduces a lightweight framework for plug-and-play integration of new vector quantization modules into frozen pre-trained tokenizers without costly end-to-end retraining. A lightweight decoder adaptation trained for only 5 epochs on ImageNet-1k mitigates quantization mismatch, achieving near state-of-the-art reconstruction fidelity on industry-level models like VAR while reducing training cost by 95%. This democratizes quantization research, enabling resource-efficient exploration of novel VQ techniques.
This paper introduces ChronoStitch, a training-free method for composing independently stored visual key-value (KV) memories to enable long-horizon temporal reasoning in video question answering. By re-basing stored post-rotary keys onto a global three-axis multimodal RoPE coordinate system and selectively recomputing high-deviation visual tokens, it overcomes temporal phase collisions and content gaps from naive concatenation. Experiments on Qwen2.5-VL-3B and the temporal split of TempCompass show improved event-ordering accuracy and 3.3x speedup over full joint re-prefilling.
This study evaluates whether synthetic and derived images improve a YOLOv8n detector for campus waste detection. Using a real dataset of 148 campus photographs, experiments showed that all synthetic augmentation configurations failed to exceed the real-only baseline (mean [email protected] of 0.691). A hand-and-forearm composite experiment was invalidated due to test set contamination and corrected, showing no significant effect. The small test set limits conclusions.
This paper introduces D3VL, a novel multimodal large language model framework that integrates 2D and 3D time-series data for autonomous driving scene understanding, achieving 11% improvement on the KITTI QA dataset and introducing a new Waymo QA extension.
Crowd4D is the first scene-aware 4D crowd reconstruction framework that jointly optimizes crowd and scene from monocular RGB video. It introduces Human-Scene Interaction Proxy (HSIP) to resolve scale and position alignment, and Crowd Structural Coherence Regularization (CSCR) for temporal stability under occlusions, outperforming existing methods in complex large-scale scenes.
The paper proposes a novel approach for detecting AI-generated videos in real-time by analyzing the compressed bitstream instead of decoding to pixels. It introduces a streaming perception framework that uses motion field data from the codec, enabling anytime detection with a single calibrated threshold. The method achieves 0.64 AUC on GenVidBench with five orders of magnitude less compute than pixel-based CNNs, and a deferral strategy improves accuracy from 0.75 to 0.78 while reducing compute by 7x.
The HIPE-2026 shared task introduces person-place relation extraction from multilingual historical newspapers. The DS@GT HIPE team investigates a lightweight, interpretable system without any pretrained language model, using dependency graphs, proximity and POS features, and small ensembles or compact GATs (under 847K parameters). Best run achieved macro recall 0.5142, 3rd in efficiency, mid-table in accuracy. Key findings: minimum character distance captures most signal; document-grouped cross-validation prevents data leakage.
Reinforcement learning has enabled test-time scaling in explicit Chain-of-Thought reasoners but is computationally expensive. Latent reasoning uses continuous vectors for intermediate computation, matching explicit CoT efficiency but lacking RL training. This paper introduces Surrogate Latent Policy Optimization (SLPO) to apply outcome-reward RL to autoregressive latent reasoners via a surrogate policy density for trajectory-level credit assignment and a correctness-supervised stopping head for variable-horizon policy. SLPO improves Pass@k and allocates longer computation to harder instances.
Proposes Multi-Mask Diffusion Model (MultiMDM) that addresses the terminal entropy issue in masked diffusion models for few-step generation by introducing multiple mask states, enabling high-quality text generation with few steps.
A new framework decomposes LLM reasoning traces into segments, uses NLI and hypergraphs to audit reasoning, offering a more reliable reference-free evaluation than LLM-as-judge, validated on math and medical benchmarks.
A new paper identifies a failure mode called 'adaptive capitulation' where LLMs first validate the user's perceived social injustice and then pivot to facilitating the very acquisition they nominally discouraged. The study tests three commercial LLMs across 900 sessions and proposes Minimal Reattributive Sufficiency (MRS) as a design principle.
Small language models are often competent at tasks but fail to follow instructions when they conflict with standard behavior. Larger models show a clearer gap between standard and non-standard instruction accuracy. The study demonstrates that task ability and instruction following are distinct capabilities.
Researchers explore using hypernetworks for train-time knowledge injection into LLMs, and conduct the first systematic study of scaling behavior for hypernetwork architectures. Results show power-law scaling along all axes and reliable OOD generalization at scale, outperforming LoRA and full fine-tuning. They create the MegaWikiQA dataset with tens of millions of multi-hop QA examples.
This paper formally defines structural generalization, and proves that, under standard complexity assumptions, pure Transformers cannot learn it, while neuro-symbolic systems achieve high scores by hardcoding semantic projections.
A study finds that supervised fine-tuning (SFT) significantly reduces behavioral diversity in large language models when adapted to downstream tasks, especially in sequential decision-making. Using controlled experiments on deterministic board games like tic-tac-toe variants, the authors show that reasoning-mode generation often suppresses action diversity, and standard SFT induces premature diversity collapse beyond what is necessary for accuracy. Action augmentation (training on all optimal actions per state) partially mitigates this effect.
Existing safety guardrails for LLMs evaluate each prompt-response pair in isolation, missing failures that arise from benign turns composing into harm over a dialogue. This paper introduces Conversational Risk Accumulation (CRA) and a session-layer framework tracking semantic drift, sensitivity-weighted information accumulation, and compliance gradient. It releases CRA-Bench benchmarks and evaluation protocols.
This work extends one-dimensional single-sweep neural-operator studies to two dimensions, using Fourier neural operators (FNOs) and U-shaped neural operators (UNOs) to approximate high-fidelity scalar flux. Three surrogates are investigated: direct mapping with FNO, direct mapping with UNO, and an FNO that takes the single-sweep approximation as an additional input. Training over three random seeds assesses variability. The study explores whether single-sweep input and log-flux training improve accuracy.
This study extends binned spectral loss functions to unstructured meshes for surrogate modeling of chaotic dynamical systems. By replacing Fourier bands with graph-Laplacian frequency bands and introducing scalable Chebyshev and multilevel approximations, the method improves long-horizon rollout fidelity. Results show superior performance in forecasting turbulent flows on unstructured meshes compared to deterministic baselines.
Research shows that autointerpretability scores for sparse autoencoders are dominated by evaluation pipeline choices rather than architectural differences, undermining cross-paper comparisons.
This paper proposes STN-TGAT, a model combining temporal Transformer and graph attention network with NMI prior graph and soft-threshold sparsification for stock ranking and portfolio construction under realistic settings, outperforming benchmarks in accuracy and returns.
This paper proposes SUM, a server-side framework that performs geometric surgery on adaptation vectors during aggregation to mitigate both spatial and temporal interference in Federated Class Incremental Learning, achieving up to 22% improvement without extra client-side computation or communication.
This research investigates explainability as a key tool for understanding continual learning in adaptive time series forecasting. Using experience replay strategies, it studies neural architectures including PatchMixer, PatchTST, and DLinear, enhanced with attention-based sampling. Explainability methods such as attention rollout and Grad-CAM are employed to analyze predictive behavior and sampling strategies. Experiments on real-world piezometric time series reveal challenges and opportunities for leveraging explainability in non-stationary forecasting scenarios.
Air pollution causes an estimated 7.9 million premature deaths annually, making accurate forecasting a critical public health priority. Machine learning is increasingly being applied to forecast air pollution levels, yet existing benchmarks remain narrow in both geographic scope and pollutant coverage, and fail to evaluate the latest generation of time series foundation models (TSFMs) on real world, large scale data. We present Air Quality Arena (AQA), a large scale multi-country and multi-pollutant dataset (AQA-Data) and benchmark (AQA-Bench) to address this gap. AQA covers 6 major pollutants over a three year period across 7 diverse countries and 4 continents, with more than 14,000 station-pollutant series, aiming to provide a comprehensive benchmark for air quality tasks. We benchmark this dataset across 11 leading time series foundation models and classical baselines to assess performance on short-term air quality forecasting. Our results demonstrate that TSFMs are effective zero-shot forecasters and consistently outperform classical baselines, with our top-performing model employing a cross-modal architecture that leverages a vision foundation model for time series forecasting. AQA is publicly released at AirQualityArena.github.io
CruiseBench is a cruise-stage RUL benchmark derived from N-CMAPSS, designed to enable reproducible and controlled comparison of remaining useful life prediction models for aircraft engines.
This paper investigates whether transformers can perform Bayesian model selection—identifying the correct hypothesis class from data. Using controlled 'Bayesian wind tunnels' with ground-truth posteriors, a small transformer achieves near-optimal performance on relational tasks but fails completely on arithmetic tasks with opaque symbols, a limitation that persists even after 112x scaling. Frontier LLMs show qualitative Bayesian behavior but with a large calibration gap.
The paper introduces HyenaND, a subquadratic, global, input-dependent operator that acts directly on the native geometry of multidimensional data through convolutions with implicitly parametrized global, input-dependent multi-dimensional convolutional kernels. Its CUDA implementation, nSubQ, fuses the FFT-convolution path for wall-clock speedups. HyenaND matches attention baselines in genomics, vision, medical imaging, and PDE modeling, and hybrid configurations outperform both pure attention and recurrence-based hybrids.
Foundation models, multimodal models, open weights, and capability evaluations.
Agent products, workflows, automation platforms, and enterprise adoption.
AI chips, compute supply, infrastructure, and supply chains.
Regulation, copyright, safety governance, international policy, and industry rules.
Papers, benchmarks, experimental systems, and academic research updates.
Funding, acquisitions, product launches, and commercialization updates.
Robotics models, embodied AI, autonomous driving, and hardware systems.
Developer tools, productivity software, plugins, and engineering practices.