Workflows and e-Science: An overview of workflow system features and capabilities
Future Generation Computer SystemsPublished 11 July 2008
Ewa Deelman, Dennis Gannon, Matt Shields, Ian Taylor
Citations818
SJR quartileQ1
SJR score1.55
SNIP2.23
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The taxonomy provides end users with a mechanism by which they can assess the suitability of workflow in general and how they might use these features to make an informed choice about which workflow system would be a good choice for their particular application.
Abstract
Scientific workflow systems have become a necessary tool for many applications, enabling the composition and execution of complex analysis on distributed resources. Today there are many workflow systems, often with overlapping functionality. A key issue for potential users of workflow systems is the need to be able to compare the capabilities of the various available tools. There can be confusion about system functionality and the tools are often selected without a proper functional analysis. In this paper we extract a taxonomy of features from the way scientists make use of existing workflow systems and we illustrate this feature set by providing some examples taken from existing workflow systems. The taxonomy provides end users with a mechanism by which they can assess the suitability of workflow in general and how they might use these features to make an informed choice about which workflow system would be a good choice for their particular application.
Keywords
Computer ScienceDecision Sciences
The International Journal of High Performance Computing ApplicationsThe Anatomy of the Grid: Enabling Scalable Virtual Organizations
6,583 Citations2001Ian Foster, Carl Kesselman +1 more
The authors present an extensible and open Grid architecture, in which protocols, services, application programming interfaces, and software development kits are categorized according to their roles in enabling resource sharing.
Architectural styles and the design of network-based software architectures
5,103 Citations2000Roy T. Fielding, Richard N. Taylor
IEEE Transactions on Parallel and Distributed SystemsPerformance-effective and low-complexity task scheduling for heterogeneous computing
3,308 Citations2002Haluk Rahmi Topcuoglu, Salim Hariri +1 more
Two novel scheduling algorithms for a bounded number of heterogeneous processors with an objective to simultaneously meet high performance and fast scheduling time are presented, called the Heterogeneous Earliest-Finish-Time (HEFT) algorithm and the Critical-Path-on-a-Processor (CPOP) algorithm.
Condor-a hunter of idle workstations
2,272 Citations2003Michael Litzkow, Miron Livny +1 more
The design, implementation, and performance of the Condor scheduling system, which operates in a workstation environment, are presented and a performance profile of the system is presented that is based on data accumulated from 23 stations during one month.
Journal of Parallel and Distributed ComputingA Comparison of Eleven Static Heuristics for Mapping a Class of Independent Tasks onto Heterogeneous Distributed Computing Systems
1,715 Citations2001Tracy D. Braun, Howard Jay Siegel +9 more
It is shown that for the cases studied here, the relatively simple Min?min heuristic performs well in comparison to the other techniques, and one even basis for comparison and insights into circumstances where one technique will out-perform another.
BioinformaticsTaverna: a tool for the composition and enactment of bioinformatics workflows
1,623 Citations2004Tom Oinn, Matthew Addis +9 more
Grid information services for distributed resource sharing
1,553 Citations2002Karl Czajkowski, S. Fitzgerald +2 more
This work presents an information services architecture that addresses performance, security, scalability, and robustness requirements of Grid software infrastructure and has been implemented as MDS-2, which forms part of the Globus Grid toolkit and has be widely deployed and applied.
Scientific ProgrammingPegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems
1,229 Citations2005Ewa Deelman, Gurmeet Singh +11 more
The results of improving application performance through workflow restructuring which clusters multiple tasks in a workflow into single entities are presented.
Journal of Grid ComputingA Taxonomy of Workflow Management Systems for Grid Computing
832 Citations2005Jia Yu, Rajkumar Buyya
A taxonomy that characterizes and classifies various approaches for building and executing workflows on Grids is proposed that highlights the design and engineering similarities and differences of state-of-the-art in Grid workflow systems, but also identifies the areas that need further research.
Lecture notes in computer scienceA resource management architecture for metacomputing systems
787 Citations1998Karl Czajkowski, Ian Foster +5 more
This work describes a resource management architecture that distributes the resource management problem among distinct local manager, resource broker, and resource co-allocator components and defines an extensible resource specification language to exchange information about requirements.
Chimera: a virtual data system for representing, querying, and automating data derivation
608 Citations2002Ian Foster, Jens Vöckler +2 more
The Chimera virtual data system is developed, which combines avirtual data catalog for representing data derivation procedures and derived data, with a virtual data language interpreter that translates user requests into data definition and query operations on the database.
Security for Grid services
506 Citations2004Von Welch, F. Siebenlist +8 more
This work describes new approaches developed to support the Globus Toolkit version 3 (GT3) implementation of the Open Grid Services Architecture, an initiative that is recasting Grid concepts within a service-oriented framework based on Web services.
VisTrails
485 Citations2006Steven P. Callahan, Juliana Freire +4 more
The VisTrails system represents the initial attempt to improve the scientific discovery process and reduce the time to insight, and is presented by presenting actual scenarios in which scientific visualization is used and showing how the system improves usability, enables reproducibility, and greatly reduces the time required to create scientific visualizations.
Workflows for e-Science
424 Citations2007
Future Generation Computer SystemsA worldwide flock of Condors: Load sharing among workstation clusters
290 Citations1996Dick Epema, Miron Livny +3 more
The design and implementation of a distributed, layered Condor flocking mechanism that represents in each pool idle machines from other pools in the flock and allows job transfers across pool boundaries is described.
Estimated-regression planning for interactions with web services
209 Citations2002Drew McDermott
A preliminary implementation of the proposed Estimated-regression planners for web-services domain requires extending classical notations in various ways, and further tests are underway.
ASKALON: a Grid application development and computing environment
189 Citations2005Thomas Fahringer, Radu Prodan +8 more
The ASKALON environment is centered around a set of high-level services for transparent and effective Grid access, including a Scheduler for optimized mapping of workflows onto the Grid, an Enactment Engine for reliable application execution, and a Resource Manager covering both computers and application components.
Journal of Grid ComputingGrid Service Orchestration Using the Business Process Execution Language (BPEL)
157 Citations2005Wolfgang Emmerich, Ben Butchart +3 more
The extent with which the BPEL language supports the definition of scientific workflows is explained and the reliability, performance and scalability that can be achieved by executing a complex scientific workflow with ActiveBPEL, an industrial strength but freely available BPEL engine are described.
The International Journal of High Performance Computing ApplicationsEnabling Applications on the Grid: A Gridlab Overview
154 Citations2003Gabrielle Allen, Tom Goodale +12 more
An overview of GridLab, a large-scale, EU-funded Grid project spanning over a dozen groups in Europe and the US, is presented and the vision of Grid-empowered applications is outlined, as well as how applications can be Grid-enabled with the GAT and GridLab’s scheduler as an example of GAT services.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIEMontage: a grid-enabled engine for delivering custom science-grade mosaics on demand
146 Citations2004G. Bruce Berriman, Ewa Deelman +8 more
The design of a grid-enabled version of Montage, an astronomical image mosaic service, suitable for large scale processing of the sky, is described, including its application to science product generation by Spitzer Legacy Program teams and large-scale, all-sky image processing projects.
Journal of Grid ComputingVisual Grid Workflow in Triana
143 Citations2005Ian Taylor, Matt Shields +2 more
This approach, which the authors refer to as the Visual GAT, differs substantially from other approaches because it base its implementation on application level requirements and model such primitives from a user's perspective by hiding as much complexity as possible without undermining the core capabilities required.
The International Journal of High Performance Computing ApplicationsThe Cactus Worm: Experiments with Dynamic Resource Discovery and Allocation in a Grid Environment
141 Citations2001Gabrielle Allen, David Angulo +6 more
The authors describe the adaptive resource selection mechanisms and describe how they are used to achieve automatic application migration to “better” resources following performance degradation, and the results provide insights into the architectural structures required to support Adaptive resource selection.
Workflow Management in Condor
137 Citations2007P. Couvares, Tevfik Kosar +3 more
The Condor project began in 1988 and has evolved into a feature-rich batch system that targets high-throughput computing; that is, Condor focuses on providing reliable access to computing over long periods of time instead of highly tuned, high-performance computing for short period of time or a small number of applications.
Journal of Grid ComputingGEMLCA: Running Legacy Code Applications as Grid Services
133 Citations2005Thierry Delaitre, Tamás Kiss +4 more
The paper introduces the GEMLCA concept, its life cycle, design and implementation, and presents as an example a legacy simulation code that has been successfully transformed into a Grid service using GEM LCA.
Scheduling Data-IntensiveWorkflows onto Storage-Constrained Distributed Resources
129 Citations2007Arun Ramakrishnan, Gurmeet Singh +7 more
This paper examines the issue of optimizing disk usage and of scheduling large-scale scientific workflows onto distributed resources where the workflows are data- intensive, requiring large amounts of data storage, and where the resources have limited storage resources and designed an algorithm that can improve the overall workflow performance.
Innovative Applications of Artificial IntelligenceWings for Pegasus: creating large-scale scientific applications using semantic representations of computational workflows
125 Citations2007Yolanda Gil, Varun Ratnakar +3 more
A new approach to workflow creation that uses semantic representations to describe compactly complex scientific applications in a dataindependent manner, then automatically generates workflows of computations for given data sets, and finally maps them to available computing resources.
Specification of grid workflow applications with AGWL: an Abstract Grid Workflow Language
122 Citations2005Thomas Fahringer, Jun Qin +1 more
This paper presents an abstract grid workflow language (AGWL) for describing grid workflow applications at a high level of abstraction and describes a material science workflow that has been successfully ported to a grid infrastructure based on an AGWL specification.
Journal of Grid ComputingTriana Applications within Grid Computing and Peer to Peer Environments
118 Citations2003Ian Taylor, Matt Shields +2 more
The motivation behind the development of P2PS is given – emphasising its minimal, but effective Peer-to-Peer mechanisms that allow scalable, decentralized discovery and communication amongst cooperating P2 PS peers within highly unstable environments.
The Caltech Institute Archives (California Institute of Technology)Montage: A grid enabled engine for delivering custom science-grade mosaics on demand
114 Citations2004G. Bruce Berriman, Ewa Deelman +8 more
Proceedings of the IEEEGrid Computing and Beyond: The Context of Dynamic Data Driven Applications Systems
113 Citations2005Frederica Darema
DDDAS environments extend the current notion of Grid infrastructure to also include the measurement systems in an integrated and synergistic way, and require support and services that go beyond the current Grid services in terms of services and software.
Managing the Evolution of Dataflows with VisTrails
104 Citations2006Steven P. Callahan, Juliana Freire +4 more
To successfully analyze and validate various hypotheses, it is necessary to pose several queries, correlate disparate data, and create insightful visualizations of both the simulated processes and observed phenomena.
Computing in Science & EngineeringService-oriented environments for dynamically interacting with mesoscale weather
100 Citations2005
Research that is enabling a major shift toward dynamically adaptive responses to rapidly changing environmental conditions is described.
A Taxonomy of Adaptive Workflow Management
100 Citations2002Yanbo Han, Amit Sheth +1 more
The need for adaptive workflow management is highlighted, various types of workflow adaptation are classified, and potential mechanisms for achieving adaptive workflowmanagement are discussed.
An intelligent assistant for interactive workflow composition
89 Citations2004Jihie Kim, Marc Spraragen +1 more
This work presents an approach that combines knowledge bases that have rich representations of components together with planning techniques (that can track the relations and constraints among individual steps) together with an implemented system that analyzes workflows and generates error messages and suggestions in order to help users compose complete and consistent workflows.
A provisioning model and its comparison with best-effort for performance-cost optimization in grids
88 Citations2007Gurmeet Singh, Carl Kesselman +1 more
This work uses trace-based simulations to compare the application performance and cost using the provisioned and the best effort approach with a number of artificially generated workflow-structured applications and a seismic hazard application from the earthquake science community.
Astronomical Data Analysis Software and Systems (ADASS) XIIIMontage: A Grid Enabled Image Mosaic Service for the National Virtual Observatory
78 Citations2004G. Bruce Berriman, John Good +9 more
The design of a grid-enabled version of Montage, suitable for large scale processing of the sky, exploits to the maximum the parallelization inherent in the Montage architecture, whereby image reprojections are performed in parallel.
An integrated problem solving environment: the SCIRun computational steering system
74 Citations2002S. Parker, Michelle Miller +2 more
Why a tightly integrated problem solving environment, such as SCIRun, simplifies the design and debugging phases of computational science applications and how such an environment aids in the scientific discovery process is discussed.
Concurrency and Computation Practice and ExperienceGrid‐Flow: a Grid‐enabled scientific workflow system with a Petri‐net‐based interface
70 Citations2005Zhijie Guan, Francisco Hernández +5 more
Grid‐Flow assists researchers in specifying scientific experiments using a Petri‐net‐based interface and a program integration component of the Grid‐Flow system is presented to integrate all possible programs into the system.
Enabling ScientificWorkflow Reuse through Structured Composition of Dataflow and Control-Flow
67 Citations2006Shawn Bowers, Bertram Ludäscher +2 more
A generic framework, based on scientific workflow templates and frames, for embedding control-flow intensive subtasks within dataflow process networks, which can enable scientific workflows that are more robust and at the same time more reusable, since the embedding of frames and templates yields more structured and modular workflow designs.
Lecture notes in computer sciencePerformance Evaluation of the Karma Provenance Framework for Scientific Workflows
67 Citations2006Yogesh Simmhan, Beth Plale +2 more
This study finds that Karma scales exceedingly well for collecting and queryingprovenance records, showing linear or sub-linear scaling with increasing number of provenance records and clients when tested against workloads in the order of 10,000 application-service invocations and over 36 concurrent clients.
INCOSE International Symposium8.3.1 A New Graphical Systems Engineering International Standard: ISO/IEC 15909 for High‐level Petri Nets
63 Citations2001Jonathan Billington
The background, scope, justification, current status and future directions of this project are provided.
Java CoG Kit Workflow
57 Citations2007Gregor von Laszewski, Mihael Hategan +1 more
In order to satisfy the need for sophisticated experiment and simulation management solutions for the scientific user community, various frameworks must be provided.
ACM SIGMOD RecordSimplifying construction of complex workflows for non-expert users of the Southern California Earthquake Center Community Modeling Environment
57 Citations2005P. J. Maechling, Hans Chalupsky +14 more
A set of tools and interfaces that simplify the construction of workflows that supports both domain experts as well as non expert users.
Future Generation Computer SystemsSolving the grid interoperability problem by P-GRADE portal at workflow level
57 Citations2008Péter Kacsuk, Tamás Kiss +1 more
The paper describes those features and techniques that are provided and used by the P-GRADE portal to solve the Grid interoperability problem.
Adapting Golog for Programming the Semantic Web
55 Citations2001Sheila A. McIlraith, Tran Cao Son
It is argued that an augmented version of the logic programming language Golog provides a natural formalism for programming Web services and adapt and extend the Golog language to enable programs that are generic, customizable and usable in the context of the Web.
Towards optimising distributed data streaming graphs using parallel streams
55 Citations2010Chee Sun Liew, Malcolm Atkinson +2 more
This paper looks into the implementation of fine-grained data-flow between computational elements in a scientific workflow as streams and argues that this use of data-streaming to enable both overlapped pipeline and parallelised enactment is a generally applicable optimisation strategy.
Web Services Composition for Distributed Data Mining
52 Citations2005Akhtar Ali, Omer Rana +1 more
A Web services-based toolkit for supporting distributed data mining is presented and a workflow engine is provided within the toolkit to enable a user to compose Web services to implement particular point solutions.
The VLDB JournalComposing, optimizing, and executing plans for bioinformatics web services
52 Citations2005Snehal Thakkar, José Luis Ambite +1 more
This paper uses real-world bioinformatics web services to show that automatic composition techniques can efficiently generate parameterized plans that integrate data from large numbers of existing services and that optimization techniques can significantly reduce the response time of the generated integration plans.
Scientific ProgrammingDynamic Grid Scheduling with Job Migration and Rescheduling in the GridLab Resource Management System
48 Citations2004Krzysztof Kurowski, Bogdan Ludwiczak +3 more
It is shown how resource matching techniques used within GRMS can be improved by the use of a job migration based rescheduling policy, which is to shorten job pending times and reduce machine overloads.
Advanced data flow support for scientific grid workflow applications
39 Citations2007Jun Qin, Thomas Fahringer
This paper introduces a data collection concept and the corresponding collection distribution constructs, which are inspired by HPF, however applied to Grid workflow applications, and extended AGWL with these concepts and implemented the corresponding runtime support in ASKALON.
Computing in Science & EngineeringProvenance: The Bridge Between Experiments and Data
37 Citations2008Simon Miles, Paul Groth +4 more
The authors investigate how provenance helps scientists connect their results with the original experiment's formulation by automating the compilation step of abstract experiment designs.
Connecting Scientific Data to Scientific Experiments with Provenance
37 Citations2007Simon Miles, Ewa Deelman +4 more
It is argued that scientists should have access to the full provenance of their data, including not only parameters, inputs and intermediary data, but also the abstract experiment, refined into a concrete execution by the "workflow compiler".
UvA-DARE (University of Amsterdam)VLE-WFBus: A Scientific Workflow Bus for Multi e-Science Domains
37 Citations2006Zhiming Zhao, Suresh Booms +3 more
Different options to derive a generic workflow management system from domain specific implementations are discussed, and a workflow bus based solution is proposed, called VLE-WFBus, which promotes a generic framework for different e-Science domains.
Machine Translation in the Year 2004
35 Citations2006Kai D. Knight, Daniel Marcu
The basic elements of state-of-the-art, statistical MT, including modeling, decoding, evaluation, and data preparation are covered.
ACM SIGMOD RecordReport on the International Provenance and Annotation Workshop
27 Citations2006Rajendra Bose, Ian Foster +1 more
The provenance of a data item refers to its origins and processing history, while annotation is a term that refers to the process of adding notes or data to an existing structure.
Workflow Composition: Semantic Representations for Flexible Automation
26 Citations2007Yolanda Gil
This chapter focuses on the requirements and challenges of scientific workflow composition, and outlines the approaches that have found effective, including separating levels of abstraction in workflow descriptions, using semantic representations of workflows and their components, and supporting flexible automation through reuse and automatic completion of user specifications of partial workflows.
ACM SIGGRAPH Computer GraphicsImproving the application development process with modular visualization environments
25 Citations1995Hambleton D. Lord
The strengths and weaknesses of Modular Visualization Environments in certain aspects of the application development process are discussed and an increased level of developer productivity is witnessed.
Scientific ProgrammingAn Approach for the High‐Level Specification of QoS‐Aware Grid Workflows Considering Location Affinity
23 Citations2006Ivona Brandić, Sabri Pllana +1 more
This paper presents an approach for high level workflow specification that considers a comprehensive set of QoS requirements, which includes economical, legal and security aspects, and introduces a QoS-aware workflow reduction technique.
Future Generation Computer SystemsVLAM-G: a grid-based virtual laboratory
23 Citations2003Adam Belloum, David Groep +5 more
The design and an implementation prototype of the VLAMG platform are described and the feasibility of the system is demonstrated by a generalized sample scenario from the chemo-physical analysis domain.
Workflow-Level Parametric Study Support by MOTEUR and the P-GRADE Portal
22 Citations2007Tristan Glatard, Gergely Sipos +3 more
Workflow managers can help in exploiting the infrastructure parallelism, given that they are able to handle the data flow resulting from the application’s execution.
Enabling Parallel Scientific Applications with Workflow Tools
22 Citations2006A. Lathers, Mei-Hui Su +6 more
The architecture of the Telescience project is described, specifically the use of layered workflow technologies to parallelize and execute scientific codes across a distributed and heterogeneous computational resource pool without the need for the application developer to understand the intricacies of the grid.
Adaptive pricing for resource reservations in Shared environments
20 Citations2007Gurmeet Singh, Carl Kesselman +1 more
ePrints Soton (University of Southampton)A Proof of Concept: Provenance in a Service Oriented Architecture
17 Citations2005Liming Chen, Victor Tan +6 more
The cake baking scenario is described, some provenance-type questions that can be asked about the baking process are presented and an implementation of the cake baking process within a service-oriented architecture framework that can answer these questions using collated provenance data is provided.
Lecture notes in computer scienceBrokering Multi-grid Workflows in the P-GRADE Portal
16 Citations2007Attila Kertész, Gergely Sipos +1 more
A brokering solution that enables the interoperability of various Globus and LCG-2 based grids during the execution of workflow applications, and supports users to utilize computing and storage resources from multiple production grids by a single application.
A new grid workflow description language
15 Citations2005Yong Wang, Chunming Hu +1 more
A new grid workflow description language called GPEL (grid process execution language) is proposed based on the Web service composition language BPEL4WS, which is being extended according to the characteristics of grid environment and applications.
Spectrum Research Repository (Concordia University)Triana Generations
15 Citations2006Ian Taylor
The Triana workflow system is discussed within the context of the workflow community at large and a description of its key features, including its user interface and its ability to work simultaneously in heterogeneous distributed environments is given.
Triana Generations
8 Citations2006Ian Taylor
Faculty publicationsDARRT: Distributed Audio Rendering & Retrieval using Triana - Experiments in applying Grid Computing technologies for audio processing
3 Citations2006Ian Taylor, Eddie Al-Shakarchi +1 more
The DARRT project, “Distributed Audio Rendering & Retrieval using Triana,” explores the potential uses of Gridcomputing technologies towards sound computation and music synthesis.