O Java , Java ! Wherefore Art Thou Java ?
This paper demonstrates and recommends how to perform meaningful experiments with managed languages and contrasts them with methodologies in use for C or C++ and recommends some approaches for encouraging community benchmark and infrastructure development.
Abstract
The experimental systems community relies on benchmarks to evaluate proposed innovations. Therefore, the choice of benchmarks and the programming language in which they are written is a gating function for innovation in our field. While application developers are increasingly embracing managed languages such as Java and C#, systems researchers have not. This disconnect between researchers and application developers may lead to misdirected research. This problem is partially due to a dearth of appropriate benchmarks and appropriate evaluation methodologies. This paper demonstrates and recommends how to perform meaningful experiments with managed languages and contrasts them with methodologies in use for C or C++. The paper also recommends some approaches for encouraging community benchmark and infrastructure development. To further inform systems research, it may behoove some researchers to follow language implementers who “eat their own dog food”, and build systems in managed languages.