Applying Twister to Scientific Applications
Published 1 November 2010
Bingjing Zhang, Yang Ruan, Tak‐Lon Wu, Judy Qiu, Adam L. Hughes, Geoffrey Fox
Citations32
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper describes how Twister is applied to several kinds of scientific applications such as BLAST, MDS interpolation and GTM Interpolation in a non-iterative style and to MDS without interpolation in an iterative style, and shows the applicability of Twister to data parallel and EM algorithms with small overhead and increased efficiency.
Abstract
Many scientific applications suffer from the lack of a unified approach to support the management and efficient processing of large-scale data. The Twister MapReduce Framework, which not only supports the traditional MapReduce programming model but also extends it by allowing iterations, addresses these problems. This paper describes how Twister is applied to several kinds of scientific applications such as BLAST, MDS Interpolation and GTM Interpolation in a non-iterative style and to MDS without interpolation in an iterative style. The results show the applicability of Twister to data parallel and EM algorithms with small overhead and increased efficiency.
Keywords
Computer Science
BMC BioinformaticsBLAST+: architecture and applications
23,184 Citations2009Christiam Camacho, George Coulouris +5 more
The new BLAST command-line applications, compared to the current BLAST tools, demonstrate substantial speed improvements for long queries as well as chromosome length database sequences.
Munich Personal RePEc Archive (Ludwig Maximilian University of Munich)Some methods for classification and analysis of multivariate observations
22,787 Citations1967James B. MacQueen
Communications of the ACMMapReduce
18,538 Citations2008Jay B. Dean, Sanjay Ghemawat
This presentation explains how the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks.
Computer Networks and ISDN SystemsThe anatomy of a large-scale hypertextual Web search engine
15,828 Citations1998Sergey Brin, Lawrence M. Page
This paper provides an in-depth description of Google, a prototype of a large-scale search engine which makes heavy use of the structure present in hypertext and looks at the problem of how to effectively deal with uncontrolled hypertext collections where anyone can publish anything they want.
PsychometrikaMultidimensional Scaling by Optimizing Goodness of Fit to a Nonmetric Hypothesis
7,398 Citations1964Joseph B. Kruskal
The fundamental hypothesis is that dissimilarities and distances are monotonically related, and a quantitative, intuitively satisfying measure of goodness of fit is defined to this hypothesis.
Dryad
2,460 Citations2007Michael Isard, Mihai Budiu +3 more
The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.
Neural ComputationGTM: The Generative Topographic Mapping
1,389 Citations1998Chris Bishop, Markus Svensén +1 more
A form of nonlinear latent variable model called the generative topographic mapping, for which the parameters of the model can be determined using the expectation-maximization algorithm, is introduced.
Twister
777 Citations2010Jaliya Ekanayake, Hui Li +5 more
This paper presents the programming model and the architecture of Twister an enhanced MapReduce runtime that supports iterative Map Reduce computations efficiently and shows performance comparisons of Twisters with other similar runtimes such as Hadoop and DryadLINQ for large scale data parallel applications.
The Design, Implementation, and Evaluation of mpiBLAST
377 Citations2009Aaron E. Darling, Lucas B. Carey +1 more
This work presents the software architecture of mpiBLAST, an open-source parallelization of BLAST that achieves superlinear speed-up by segmenting a BLAST database and then having each node in a computational cluster search a unique portion of the database.
CloudBLAST: Combining MapReduce and Virtualization on Distributed Resources for Bioinformatics Applications
307 Citations2008Andréa Matsunaga, Maurício Tsugawa +1 more
The proposed approach uses the MapReduce paradigm to parallelize tools and manage their execution, machine virtualization to encapsulate their execution environments and commonly used data sets into flexibly deployable virtual machines, and networkvirtualization to connect resources behind firewalls/NATs while preserving the necessary performance and the communication environment.
PLANT PHYSIOLOGYThe Complement of Protein Phosphatase Catalytic Subunits Encoded in the Genome of Arabidopsis
269 Citations2002David Kerk, Joshua Bulgrien +4 more
Eight sequences were identified as new protein phosphatase candidates: five dual-specificity phosphatases and three PP2Cs, and phylogenetic analyses were used to infer clustering patterns reflecting sequence similarity and evolutionary ancestry.
AzureBlast
125 Citations2010Wei Lü, Jared Jackson +1 more
This paper investigates the applicability of cloud to the sciences by investigating an implementation of a well known and computationally intensive algorithm called BLAST, a very popular life sciences algorithm used commonly in bioinformatics research.
Cloud computing paradigms for pleasingly parallel biomedical applications
67 Citations2010Thilina Gunarathne, Tak‐Lon Wu +2 more
Dimension reduction and visualization of large high-dimensional data via interpolation
55 Citations2010Seunghee Bae, Jong Youl Choi +2 more
The authors propose an interpolated approach to utilizing the mapping of only a subset of the given data, which effectively reduces computational complexity and demonstrates that the quality of interpolated mapping results are comparable to the mapping results of original algorithm only.
BMC BioinformaticsFastGroup: A program to dereplicate libraries of 16S rDNA sequences
52 Citations2001Victor Seguritan, Forest Rohwer
The FastGroup program simplifies the dereplication of 16S rDNA sequence libraries and prepares the raw sequences for subsequent analyses.
Computational Statistics & Data AnalysisThe out-of-sample problem for classical multidimensional scaling
51 Citations2008Michael W. Trosset, Carey E. Priebe
Out-of-sample embedding techniques insert additional points into previously constructed configurations and the objective function is a fourth-order polynomial, easily minimized by standard gradient-based methods for numerical optimization.
Knowledge and Information SystemsEmbedding new data points for manifold learning via coordinate propagation
43 Citations2008Shiming Xiang, Feiping Nie +3 more
This work first formulate this task as a quadratic programming, and then develops an iterative algorithm for coordinate propagation, to propagate the known coordinates to each of the new data points.
Massively Parallel Genomic Sequence Search on the Blue Gene/P Architecture
41 Citations2009Heshan Lin, Pavan Balaji +4 more
Nucleic Acids ResearchFELINES: a utility for extracting and examining EST-defined introns and exons
17 Citations2003Scott D. Drabenstot
FELINES was shown to be useful for characterizing branchsites, polypyrimidine tracts and 5' and 3' splice sites in the intron databases and exonic splicing enhancers in S.pombe exons.
A scalable generative topographic mapping for sparse data sequences
16 Citations2005Ata Kabán
A novel, computationally efficient generative topographic model for inferring low dimensional representations of high dimensional data sets, designed to exploit data sparseness, and its associated parameter estimation algorithm scales linearly with the number of nonzero entries in the observations.