Source-led article
IBM Research Introduces ScarfBench to Evaluate AI Agents in Enterprise Java Migration
Modernizing enterprise applications is a significant and costly endeavor for organizations globally, including those in India. A key aspect of this modernization involves migrating applications across different frameworks to enhance maintainability, cloud readiness, developer productivity, and access to modern functionalities. While recent advancements in AI coding agents have generated considerable excitement for AI-assisted modernization, a crucial question remains: can these AI agents reliably modernize real-world enterprise applications?
IBM Research has addressed this by introducing ScarfBench (Self-Contained Application Refactoring Benchmark), an open benchmark specifically designed to evaluate AI agents on cross-framework migration tasks within Enterprise Java. This benchmark aims to fill a gap in existing software engineering evaluations, which often focus on bug fixing and code generation but overlook the complexities of framework migration.
Key facts:
| Feature | Description |
|---|---|
| Name | ScarfBench (Self-Contained Application Refactoring Benchmark) |
| Focus | Evaluating AI agents on cross-framework migration tasks in Enterprise Java applications |
| Developers | IBM Research |
| Availability | Open resource for researchers and practitioners, with GitHub repository, Hugging Face dataset, and Space |
The Challenge of Framework Migration
Unlike traditional benchmarks that compare generated code against reference implementations, ScarfBench assesses whether migrated applications successfully build, deploy, and preserve their intended behavior. Framework migration is far more intricate than simply replacing annotations or translating source code. It demands a deep understanding of framework semantics and requires changes across dependency injection, persistence configuration, queries, and framework descriptors. Even minor errors in these areas can prevent successful deployment.
ScarfBench provides a systematic approach to evaluate AI agents on these complex tasks. Applications are required to build, deploy, and pass behavioral tests, offering a more realistic measure of modernization quality. The benchmark includes both focused migration tasks and comprehensive whole-application migrations across three major Java ecosystems: Spring, Jakarta EE, and Quarkus.
Agent Performance and Limitations
Initial evaluations of several state-of-the-art coding agents on ScarfBench revealed that despite strong performances on traditional software engineering benchmarks, framework migration remains a difficult challenge. Success rates varied significantly across different framework pairs, with whole-application migrations proving particularly demanding.
A notable observation was the discrepancy between agent-reported outcomes and actual build verification. For instance, Claude Code reported successful builds for a high percentage of applications, but independent validation showed a lower success rate. This highlights that agent self-assessment cannot be solely relied upon as a definitive signal of migration completion, emphasizing the need for independent build and test validation.
Insights into Migration Complexity
ScarfBench also sheds light on the iterative and dependency-driven nature of framework migration. Changes often cascade across various layers of an application, including configuration, services, databases, and web components. The most frequently revisited layers during migration efforts involved debugging, dependency resolution, and framework adaptation. This indicates that migration is an iterative process of resolving dependencies rather than a simple, linear code transformation.
Furthermore, agents frequently encountered environmental issues, such as build system and deployment environment problems, which often delayed validation even when source-code migration was largely complete. Modernization failures span build systems, deployment environments, dependency injection, databases, endpoints, assertions, and infrastructure.
The biggest hurdle in framework modernization is not merely translating Java code, but effectively managing the complex web of dependencies across configuration, infrastructure, and runtime environments. While advanced AI agents can automate substantial portions of the migration process, reliable validation and robust architectural reasoning remain crucial for achieving successful outcomes. For Indian enterprises considering AI-driven modernization, ScarfBench underscores the importance of thorough testing and validation beyond initial code generation.
ScarfBench is an open resource for both researchers and practitioners. It offers resources such as a GitHub repository, Hugging Face dataset, and a dedicated space for evaluation. Researchers can use it to compare agent architectures and techniques, while practitioners can leverage it to evaluate modernization solutions before implementing them in production environments.