A survey of the applications of text mining in financial domain
Knowledge-Based SystemsPublished 9 October 2016
B. Shravan Kumar, Vadlamani Ravi
Citations261
SJR quartileQ1
SJR score1.93
SNIP2.34
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A state-of-the-art survey of various applications of Text mining to finance, categorized broadly into FOREX rate prediction, stock market prediction, customer relationship management (CRM) and cyber security.
Abstract
Text mining has found a variety of applications in diverse domains. Of late, prolific work is reported in using text mining techniques to solve problems in financial domain. The objective of this paper is to provide a state-of-the-art survey of various applications of Text mining to finance. These applications are categorized broadly into FOREX rate prediction, stock market prediction, customer relationship management (CRM) and cyber security. Since finance is a service industry, these problems are paramount in operational and customer growth aspects. We reviewed 89 research papers that appeared during the period 2000–2016, highlighted some of the issues, gaps, key challenges in this area and proposed some future research directions. Finally, this review can be extremely useful to budding researchers in this area, as many open problems are highlighted.
Keywords
Computer ScienceDecision Sciences
Bell System Technical JournalA Mathematical Theory of Communication
80,675 Citations1948Claude E. Shannon
Machine LearningSupport-vector networks
40,463 Citations1995Corinna Cortes, Vladimir Vapnik
High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated and the performance of the support- vector network is compared to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
Machine LearningSupport-Vector Networks
33,035 Citations1995Corinna Cortes, Vladimir Vapnik
High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated and the performance of the support- vector network is compared to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
NatureLearning representations by back-propagating errors
30,885 Citations1986David E. Rumelhart, Geoffrey E. Hinton +1 more
Back-propagation repeatedly adjusts the weights of the connections in the network so as to minimize a measure of the difference between the actual output vector of the net and the desired output vector, which helps to represent important features of the task domain.
TechnometricsStatistical Learning Theory
26,913 Citations1999Yuhai Wu, Vladimir Vapnik
Presenting a method for determining the necessary and sufficient conditions for consistency of learning process, the author covers function estimates from small data pools, applying these estimations to real-life problems, and much more.
IEEE Transactions on Information TheoryNearest neighbor pattern classification
16,134 Citations1967Thomas M. Cover, Peter E. Hart
The nearest neighbor decision rule assigns to an unclassified sample point the classification of the nearest of a set of previously classified points, so it may be said that half the classification information in an infinite sample set is contained in the nearest neighbor.
Machine LearningInduction of Decision Trees
14,815 Citations1986J. R. Quinlan
This paper summarizes an approach to synthesizing decision trees that has been used in a variety of systems, and it describes one such system, ID3, in detail, which is described in detail.
ACM Computing SurveysData clustering
13,184 Citations1999Anil K. Jain, M. Narasimha Murty +1 more
An overview of pattern clustering methods from a statistical pattern recognition perspective is presented, with a goal of providing useful advice and references to fundamental concepts accessible to the broad community of clustering practitioners.
Machine LearningInduction of decision trees
12,383 Citations1986J. R. Quinlan
This paper summarizes an approach to synthesizing decision trees that has been used in a variety of systems, and it describes one such system, ID3, in detail, which is described in detail.
Information Processing & ManagementTerm-weighting approaches in automatic text retrieval
9,532 Citations1988Gerard Salton, Chris Buckley
This paper summarizes the insights gained in automatic term weighting, and provides baseline single term indexing models with which other more elaborate content analysis procedures can be compared.
ACM Computing SurveysMachine learning in automated text categorization
7,899 Citations2002Fabrizio Sebastiani
This survey discusses the main approaches to text categorization that fall within the machine learning paradigm and discusses in detail issues pertaining to three different problems, namely, document representation, classifier construction, and classifier evaluation.
Mining and summarizing customer reviews
7,714 Citations2004Minqing Hu, Bing Liu
This research aims to mine and to summarize all the customer reviews of a product, and proposes several novel techniques to perform these tasks.
Thumbs up?
6,987 Citations2002Bo Pang, Lillian Lee +1 more
This work considers the problem of classifying documents not by topic, but by overall sentiment, e.g., determining whether a review is positive or negative, and concludes by examining factors that make the sentiment classification problem more challenging.
now publishers, Inc. eBooksOpinion Mining and Sentiment Analysis
6,745 Citations2008Bo Pang, Lillian Lee
Journal of Computational ScienceTwitter mood predicts the stock market
4,995 Citations2011Johan Bollen, Huina Mao +1 more
This work investigates whether measurements of collective mood states derived from large-scale Twitter feeds are correlated to the value of the Dow Jones Industrial Average (DJIA) over time and indicates that the accuracy of DJIA predictions can be significantly improved by the inclusion of specific public mood dimensions but not others.
A Comparative Study on Feature Selection in Text Categorization
4,766 Citations1997Yiming Yang, Jan Pedersen
DF thresholding, the simplest method with the lowest cost in computation, can be reliably used instead of IG or CHI when the computation of these measures are too expensive, and strong correlations between the DF, IG and CHI values of a term are found.
IEEE Transactions on Software EngineeringAn Intrusion-Detection Model
3,358 Citations1987Dorothy E. Denning
Machine LearningOn the Optimality of the Simple Bayesian Classifier under Zero-One Loss
3,072 Citations1997Pedro Domingos, Michael J. Pazzani
The Bayesian classifier is shown to be optimal for learning conjunctions and disjunctions, even though they violate the independence assumption, and will often outperform more powerful classifiers for common training set sizes and numbers of attributes, even if its bias is a priori much less appropriate to the domain.
International Journal of Man-Machine StudiesSimplifying decision trees
2,502 Citations1987J. R. Quinlan
Techniques for simplifying decision trees while retaining their accuracy are discussed, described, illustrated, and compared on a test-bed of decision trees from a variety of domains.
The Journal of FinanceIs All That Talk Just Noise? The Information Content of Internet Stock Message Boards
2,400 Citations2004Werner Antweiler, Murray Z. Frank
The Journal of FinanceMore Than Words: Quantifying Language to Measure Firms' Fundamentals
2,245 Citations2008Paul C. Tetlock, Maytal Saar‐Tsechansky +1 more
Extracting product features and opinions from reviews
1,720 Citations2005Ana-Maria Popescu, Oren Etzioni
Opine is introduced, an unsupervised information-extraction system which mines reviews in order to build a model of important product features, their evaluation by reviewers, and their relative quality across products.
Data Mining and Knowledge DiscoveryE-Commerce Recommendation Applications
1,518 Citations2001J. Ben Schafer, Joseph A. Konstan +1 more
An explanation of how recommender systems are related to some traditional database analysis techniques is presented, and a taxonomy ofRecommender systems is created, including the inputs required from the consumers, the additional knowledge required from a database, the ways the recommendations are presented to consumers,The technologies used to create the recommendations, and the level of personalization of the recommendations.
Management ScienceYahoo! for Amazon: Sentiment Extraction from Small Talk on the Web
1,442 Citations2007Sanjiv Ranjan Das, Mike Y. Chen
A methodology for extracting small investor sentiment from stock message boards is developed, which comprises different classifier algorithms coupled together by a voting scheme.
ACM SIGKDD Explorations NewsletterWeb mining research
1,439 Citations2000Raymond Kosala, Hendrik Blockeel
This paper surveys the research in the area of Web mining, point out some confusions regarded the usage of the term Web mining and suggest three Web mining categories, which are then situate some of the research with respect to these three categories.
A simple rule-based part of speech tagger
1,418 Citations1992Eric Brill
This work presents a simple rule-based part of speech tagger which automatically acquires its rules and tags with accuracy comparable to stochastic taggers, demonstrating that the stochastics method is not the only viable method for part ofspeech tagging.
Journal of Accounting ResearchThe Information Content of Forward‐Looking Statements in Corporate Filings—A Naïve Bayesian Machine Learning Approach
1,371 Citations2010Feng Li
It is found that firms with better current performance, lower accruals, smaller size, lower market-to-book ratio, less return volatility, lower MD&A Fog index, and longer history tend to have more positive FLSs.
A Survey of Opinion Mining and Sentiment Analysis
1,331 Citations2012Bing Liu, Lei Zhang
Sentiment analysis or opinion mining is the computational study of people’s opinions, appraisals, attitudes, and emotions toward entities, individuals, issues, events, topics and their attributes.
Why phishing works
1,286 Citations2006Rachna Dhamija, J. D. Tygar +1 more
This paper provides the first empirical evidence about which malicious strategies are successful at deceiving general users by analyzing a large set of captured phishing attacks and developing a set of hypotheses about why these strategies might work.
Knowledge-Based SystemsA survey on opinion mining and sentiment analysis: Tasks, approaches and applications
1,274 Citations2015Kumar Ravi, Vadlamani Ravi
A rigorous survey on sentiment analysis is presented, which portrays views presented by over one hundred articles published in the last decade regarding necessary tasks, approaches, and applications of sentiment analysis.
A Probabilistic Analysis of the Rocchio Algorithm with TFIDF for Text Categorization
1,264 Citations1997Thorsten Joachims
A Probabilistic analysis of the Rocchio relevance feedback algorithm, one of the most popular learning methods from information retrieval, is presented in a text categorization framework and suggests that the probabilistic algorithms are preferable to the heuristic Rocchio classifier.
Lecture notes in computer scienceThe Enron Corpus: A New Dataset for Email Classification Research
1,141 Citations2004Bryan Klimt, Yiming Yang
The Enron corpus is introduced as a new test bed for email folder prediction, and the baseline results of a state-of-the-art classifier (Support Vector Machines) are provided under various conditions.
Knowledge-Based SystemsTransfer learning using computational intelligence: A survey
963 Citations2015Jie Lü, Vahid Behbood +4 more
This paper systematically examines computational intelligence-based transfer learning techniques and clusters related technique developments into four main categories and provides state-of-the-art knowledge that will directly support researchers and practice-based professionals to understand the developments in computational Intelligence- based transfer learning research and applications.
International Journal of Information ManagementSocial media competitive analysis and text mining: A case study in the pizza industry
876 Citations2013Wu He, Shenghua Zha +1 more
An in-depth case study which applies text mining to analyze unstructured text content on Facebook and Twitter sites of the three largest pizza chains: Pizza Hut, Domino's Pizza and Papa John's Pizza reveals the value of social media competitive analysis and the power of text mining as an effective technique to extract business value from the vast amount of available social media data.
Lecture notes in computer scienceAnomalous Payload-Based Network Intrusion Detection
845 Citations2004Ke Wang, Salvatore J. Stolfo
A payload-based anomaly detector, called PAYL, for intrusion detection that demonstrates the surprising effectiveness of the method on the 1999 DARPA IDS dataset and a live dataset the authors collected on the Columbia CS department network.
ACM Transactions on Information SystemsTextual analysis of stock market prediction using breaking financial news
770 Citations2009Robert P. Schumaker, Hsinchun Chen
This research examines a predictive machine learning approach for financial news articles analysis using several different textual representations: bag of words, noun phrases, and named entities and found that a Proper Noun scheme performs better than the de facto standard of Bag of Words in all three metrics.
Information Processing & ManagementThe impact of preprocessing on text classification
653 Citations2013Alper Kürşat Uysal, Serkan Günal
Experimental analysis on benchmark datasets reveals that choosing appropriate combinations of preprocessing tasks, rather than enabling or disabling them all, may provide significant improvement on classification accuracy depending on the domain and language studied on.
Expert Systems with ApplicationsText mining for market prediction: A systematic review
591 Citations2014Arman Khadjeh Nassirtoussi, Saeed Aghabozorgi +2 more
A comparative analysis of the systems based on market prediction based on online-text-mining expands onto the theoretical and technical foundations behind each and should help the research community to structure this emerging field and identify the exact aspects which require further research and are of special significance.
Lecture notes in computer scienceSentiment Knowledge Discovery in Twitter Streaming Data
580 Citations2010Albert Bifet, Eibe Frank
To deal with streaming unbalanced classes, a sliding window Kappa statistic is proposed for evaluation in time-changing data streams, and a study on Twitter data is performed using learning algorithms for data streams.
IEEE Communications Surveys & TutorialsPhishing Detection: A Literature Survey
576 Citations2013Mahmoud Khonji, Youssef Iraqi +1 more
A high-level overview of various categories of phishing mitigation techniques is presented, such as: detection, offensive defense, correction, and prevention, which it is believed is critical to present where the phishing detection techniques fit in the overall mitigation process.
A Survey of Text Clustering Algorithms
562 Citations2012Charų C. Aggarwal, ChengXiang Zhai
This chapter will study the key challenges of the clustering problem, as it applies to the text domain, and discuss the key methods used for text clustering, and their relative advantages.
Journal of the ACMAutomatic Indexing: An Experimental Inquiry
552 Citations1961M. E. Maron
The design, execution and evaluation of a modest experimental study aimed at testing empirically one statistical technique for automatic indexed documents according to their subject content are described.
Expert Systems with ApplicationsA review of machine learning approaches to Spam filtering
531 Citations2009Thiago Guzella, Walmir M. Caminhas
A comprehensive review of recent developments in the application of machine learning algorithms to Spam filtering, focusing on both textual- and image-based approaches concludes that while important advancements have been made in the last years, several aspects remain to be explored, especially under more realistic evaluation settings.
arXiv (Cornell University)An evaluation of Naive Bayesian anti-spam filtering
527 Citations2000Ion Androutsopoulos, John Koutsias +3 more
It is reached that additional safety nets are needed for the Naive Bayesian anti-spam filter to be viable in practice.
A comparison of machine learning techniques for phishing detection
498 Citations2007Saeed Abu‐Nimeh, Dario Nappa +2 more
This study compares the predictive accuracy of several machine learning methods including Logistic Regression (LR), Classification and Regression Trees (CART), Bayesian Additive Regression trees (BART), Support Vector Machines (SVM), Random Forests (RF), and Neural Networks (NNet) for predicting phishing emails.
Text Classification Using Machine Learning Techniques
488 Citations2005M. Ikonomakis, Sotiris Kotsiantis +1 more
This paper illustrates the text classification process using machine learning techniques to manage and process a vast amount of documents in digital forms that are widespread and continuously increasing.
Decision Support SystemsThe impact of social and conventional media on firm equity value: A sentiment analysis approach
487 Citations2012Yang Yu, Wenjing Duan +1 more
The findings suggest that overall social media has a stronger relationship with firm stock performance than conventional media while social and conventional media have a strong interaction effect on stock performance.
Conference on Email and Anti-SpamSpam Filtering with Naive Bayes - Which Naive Bayes?
480 Citations2006Vangelis Metsis, Ion Androutsopoulos +1 more
An experimental procedure that emulates the incremental training of person- alized spam filters is adopted, and roc curves that allow us to compare the dierent versions of nb over the entire tradeo between true positives and true negatives are plotted.
Expert Systems with ApplicationsTwitter brand sentiment analysis: A hybrid system using n-gram analysis and dynamic artificial neural network
478 Citations2013M. Ghiassi, J.F. Skinner +1 more
This research introduces an approach to supervised feature reduction using n-grams and statistical analysis to develop a Twitter-specific lexicon for sentiment analysis, and develops sentiment classification models using this reduced lexicon and the DAN2 machine learning approach, which has demonstrated success in other text classification problems.
A framework for detection and measurement of phishing attacks
468 Citations2007Sujata Garera, Niels Provos +2 more
It is found that it is often possible to tell whether or not a URL belongs to a phishing attack without requiring any knowledge of the corresponding page data.
An experimental comparison of naive Bayesian and keyword-based anti-spam filtering with personal e-mail messages
451 Citations2000Ion Androutsopoulos, John Koutsias +2 more
This work introduces appropriate cost-sensitive measures, and investigates at the same time the effect of attribute-set size, training-corpus size, lemmatization, and stop lists, issues that have not been explored in previous experiments.
The ghost in the browser analysis of web-based malware
422 Citations2007Niels Provos, Dean McNamee +3 more
This work identifies the four prevalent mechanisms used to inject malicious content on popular web sites: web server security, user contributed content, advertising and third-party widgets, and presents examples of abuse found on the Internet.
Computers & Operations ResearchApplication of SVM and ANN for intrusion detection
409 Citations2004Wun-Hwa Chen, Sheng-Hsun Hsu +1 more
Two data mining methodologies-Artificial Neural Networks and Support Vector Machine and two encoding methods-simple frequency-based scheme and tfi?idf scheme are used to detect potential system intrusions in this study.
Artificial Intelligence ReviewA survey of learning-based techniques of email spam filtering
401 Citations2008Enrico Blanzieri, Anton Bryl
An overview of the state of the art of machine learning applications for spam filtering, and of the ways of evaluation and comparison of different filtering methods.
A Survey of Malware Detection Techniques
398 Citations2007Nwokedi Idika, Aditya P. Mathur
This paper presents a meta-modelling system that automates and automates the very labor-intensive and therefore time-heavy and expensive and expensive process of manually cataloging and annotating Malware.
Journal of Financial EconomicsHow is macro news transmitted to exchange rates?
378 Citations2008Martin D.D. Evans, Richard K. Lyons
Ontologies improve text document clustering
353 Citations2004Andreas Hotho, Steffen Staab +1 more
This work integrates core ontologies as background knowledge into the process of clustering text documents and compares clustering techniques based on pre-categorizations of texts from Reuters newsfeeds and on a smaller domain of an eLearning course about Java.
Expert Systems with ApplicationsPhishing detection based Associative Classification data mining
345 Citations2014Neda Abdelhamid, Aladdin Ayesh +1 more
Experimental results show that AC particularly MCAC detects phishing websites with higher accuracy than other intelligent algorithms and generates new hidden knowledge (rules) that other algorithms are unable to find and this has improved its classifiers predictive performance.
Decision Support SystemsAutomated news reading: Stock price prediction based on financial news using context-capturing features
343 Citations2013Michael Hagenau, Michael Liebmann +1 more
It is shown that a robust feature selection allows lifting classification accuracies significantly above previous approaches when combined with complex feature types and reduces the problem of over-fitting when applying a machine learning approach.
Research Showcase @ Carnegie Mellon University (Carnegie Mellon University)An Empirical Analysis of Phishing Blacklists
340 Citations2018Steve Sheng, Brad Wardman +4 more
This paper used 191 fresh phish that were less than 30 minutes old to conduct two tests on eight anti-phishing toolbars and found that two tools using heuristics to complement blacklists caught signicantly more phish initially than those using only blacklists.
ACM Transactions on Asian Language Information ProcessingAn evaluation of statistical spam filtering techniques
317 Citations2004Le Zhang, Jingbo Zhu +1 more
Experiments show that classifiers using features from message header alone can achieve comparable or better performance than filters utilizing body features only, which implies that message headers can be reliable and powerfully discriminative feature sources for spam filtering.
Decision Support SystemsEvaluating sentiment in financial news articles
311 Citations2012Robert P. Schumaker, Yulei Zhang +2 more
Investigation of the Arizona Financial Text system, a financial news article prediction system, and a sentiment analysis tool found that subjective news articles were easier to predict in price direction and using a simple trading engine, subjective articles garnered a 3.30% return.
Proceedings of the International AAAI Conference on Web and Social MediaWidespread Worry and the Stock Market
288 Citations2010Éric Gilbert, Karrie Karahalios
It is demonstrated that estimating emotions from weblogs provides novel information about future stock market prices, and how the mood of millions in a large online community, even one that primarily discusses daily life, can anticipate changes in a seemingly unrelated system.
Trinity's Access to Research Output (TARA) (Trinity College Dublin)Sentiment Polarity Identification in Financial News: A Cohesion-based Approach
275 Citations2007Ann Devitt, Khurshid Ahmad
This paper aims to explore a computable metric of positive or negative polarity in financial news text which is consistent with human judgments and can be used in a quantitative analysis of news sentiment impact on financial markets.
Expert Systems with ApplicationsText mining of news-headlines for FOREX market prediction: A Multi-layer Dimension Reduction Algorithm with semantics and sentiment
272 Citations2014Arman Khadjeh Nassirtoussi, Saeed Aghabozorgi +2 more
A novel approach is proposed to predict intraday directional-movements of a currency-pair in the foreign exchange market based on the text of breaking financial news-headlines and produces a multi-layer algorithm that tackles each of the mentioned aspects of the text-mining problem at a designated layer.
Forecasting Intraday stock price trends with text mining techniques
265 Citations2004Marc-André Mittermayer
The findings indicate that a categorization of press releases is able to provide additional information that can be used to forecast stock price trends, but that an adequate trading strategy is essential for the results of the categorization to be fully exploited.
ACM SIGKDD Explorations NewsletterText mining for product attribute extraction
241 Citations2006Rayid Ghani, Katharina Probst +3 more
This work describes the work on extracting attribute and value pairs from textual product descriptions, and presents promising results on apparel and sporting goods products and shows that the system can accurately extract attribute-value pairs from product descriptions.
Decision Support SystemsMaking words work: Using financial text as a predictor of financial events
240 Citations2010Mark Cecchini, Haldun Aytuğ +2 more
A methodology for automatically analyzing text to aid in discriminating firms that encounter catastrophic financial events and achieves best prediction results for both bankruptcy and fraud with the combined data, showing that that the text of the MD&A complements the quantitative financial information.
Expert Systems with ApplicationsDendroid: A text mining approach to analyzing and classifying code structures in Android malware families
232 Citations2013Guillermo Suárez‐Tangil, Juan Tapiador +2 more
Dendroid, a system based on text mining and information retrieval techniques for malware analysis, is introduced, suggesting that the approach is remarkably accurate and deals efficiently with large databases of malware instances.
Studies in fuzziness and soft computingDetection of Phishing Attacks: A Machine Learning Approach
229 Citations2008Ram B. Basnet, Srinivas Mukkamala +1 more
Though there are several anti-phishing software and techniques for detecting potential phishing attempts in emails and detecting phishing contents on websites, phishers come up with new and hybrid techniques to circumvent the availableSoftware and techniques.
Anomaly Based Web Phishing Page Detection
211 Citations2006Ying Pan, Xuhua Ding
The idea is to examine the anomalies in Web pages, in particular, the discrepancy between a Web site's identity and its structural features and HTTP transactions, which demands neither user expertise nor prior knowledge of the Web site.
ACM Transactions on Management Information SystemsBusiness Intelligence and Analytics
209 Citations2013Ee‐Peng Lim, Hsinchun Chen +1 more
The article aims to review the state-of-the-art techniques and models and to summarize their use in BIA applications to categorize BIA research activities into three broad research directions: (a) big data analytics, (b) text analytics, and (c) network analytics.
Expert Systems with ApplicationsIntelligent phishing detection system for e-banking using fuzzy data mining
195 Citations2010Maher Aburrous, Mohammad Alamgir Hossain +2 more
The proposed model is based on fuzzy logic combined with data mining algorithms to characterize the e-banking phishing website factors and to investigate its techniques by classifying the phishing types and defining six e-banking phishing website attack criteria's with a layer structure.
IMDS
195 Citations2007Yanfang Ye, Dingding Wang +2 more
Promising experimental results demonstrate that the accuracy and efficiency of the IMDS system out perform popular anti-virus software such as Norton AntiVirus and McAfee VirusScan, as well as previous data mining based detection systems which employed Naive Bayes, Support Vector Machine and Decision Tree techniques.
Journal of the ACMAutomatic Document Classification
188 Citations1963Harold Borko, Myrna Bernick
Of the ninety documents in the Validation Group which contained two or more clue words, and which could be automatically classified, 44 documents, or 48.9%, were placed into their correct categories by use of a computer formula.
Decision Support SystemsA computational model for financial reporting fraud detection
186 Citations2010Fletcher H. Glancy, Surya B. Yadav
The proposed computational fraud detection model (CFDM) was tested on the Management's Discussion and Analysis from 10-K filings and was able to distinguish fraudulent filings from non-fraudulent ones.
Lecture notes in computer scienceOn the Effectiveness of Techniques to Detect Phishing Sites
170 Citations2007Christian Ludl, Sean Mcallister +2 more
Over a period of three weeks, the effectiveness of the blacklists maintained by Google and Microsoft with 10,000 phishing URLs was tested, and the existence of page properties that can be used to identify phishing pages were explored.
Decision Support SystemsAn intraday market risk management approach based on textual analysis
168 Citations2010Sven S. Groth, Jan Muntermann
Evaluation results provide strong evidence that unstructured (textual) data represents a valuable source of information also for financial risk management - a domain in which, in the past, little attention has been paid to unstructuring data.
Using Text Categorization Techniques for Intrusion Detection
165 Citations2002Yi‐Hua Liao, V. Rao Vemuri
A new approach, based on the k-Nearest Neighbor (kNN) classifier, is used to classify program behavior as normal or intrusive, and preliminary experiments with 1998 DARPA BSM audit data show that the kNN classifier can effectively detect intrusive attacks and achieve a low false positive rate.
Journal of Computer SecurityNew filtering approaches for phishing email
163 Citations2010André Bergholz, Jan De Beer +4 more
A number of novel features that are particularly well-suited to identify phishing emails are described, including statistical models for the low-dimensional descriptions of email topics, sequential analysis of email text and external links, the detection of embedded logos as well as indicators for hidden salting.
Using spatio-temporal information in API calls with machine learning algorithms for malware detection
155 Citations2009Faraz Ahmed, Haider Hameed +2 more
The key novelty of this proposed tool is the use of statistical features which are extracted from both spatial arguments and temporal information available in Windows API calls, and provided as an input to standard machine learning algorithms to raise the final alarm.
Decision Support SystemsImproving customer complaint management by automatic email classification using linguistic style features as predictors
150 Citations2007Kristof Coussement, Dirk Van den Poel
The empirical results show that adding linguistic style information into a classification model with conventional text-classification variables results in a significant increase in predictive performance and linguistic style differences between complaint emails and others.
Intelligent systems in accounting, finance and management/Intelligent systems in accounting, finance & managementCombining data and text mining techniques for analysing financial reports
142 Citations2004Antonina Kloptchenko, Tomas Eklund +4 more
This paper combines data mining methods for analyzing quantitative and qualitative data from financial reports, in order to see if the textual part of the report contains some indication about future financial performance.
Knowledge-Based SystemsTwitter user profiling based on text and community mining for market analysis
138 Citations2013Kazushi Ikeda, Gen Hattori +3 more
A hybrid text-based and community-based method for the demographic estimation of Twitter users, where these demographics are estimated by tracking the tweet history and clustering of followers/followees is proposed.
Expert Systems with ApplicationsMalicious web content detection by machine learning
138 Citations2009Yung-Tsung Hou, Yimeng Chang +3 more
This study analyzes the characteristic of a malicious webpage systematically and presents important features for machine learning, and demonstrates that the proposed method is resilient to code obfuscations and can correctly determine whether a webpage is malicious or not.
Text mining
133 Citations1999Jochen Dörre, Peter Gerstl +1 more
This work defines the notion of “text mining”, focuses on the differences between text and data mining, and describes in some more detail the unique technologies that are key to successful text mining.
Lecture notes in computer scienceNews Sensitive Stock Trend Prediction
133 Citations2002Gabriel Pui Cheong Fung, Jeffrey Xu Yu +1 more
This paper investigates the immediate impact of news articles on the time series based on the Efficient Markets Hypothesis and proposes an algorithm, which is called guided clustering, to filter news articles with the help of the clusters that have been obtained from trends.
Computers in IndustryText analytics in industry: Challenges, desiderata and trends
127 Citations2015Ashwin Ittoo, Le-Minh Nguyen +1 more
A systematic review of the current state of the art in the application of text analytics in industry and a set of desiderata that text analytics techniques should satisfy in order to alleviate challenges and to ensure their successful deployment in industry are provided.
Journal of International Money and FinanceCurrency jumps, cojumps and the role of macro news
122 Citations2013Arjun Chatrath, Hong Miao +2 more
Expert Systems with ApplicationsAn efficient phishing webpage detector
115 Citations2011Mingxing He, Shi‐Jinn Horng +6 more
A heuristic method to determine whether a webpage is a legitimate or a phishing page which could detect new phishing pages which black list based anti-phishing tools could not.
Stock prediction: Integrating text mining approach using real-time news
111 Citations2003Gabriel Pui Cheong Fung, Jeffrey Xu Yu +1 more
This paper proposes a systematic framework for mining multiple time series based on Efficient Market Hypothesis and explores the opportunity for mining textual documents and time series concurrently to predict stock prices based on news articles.
Automatic malware categorization using cluster ensemble
110 Citations2010Yanfang Ye, Tao Li +2 more
An Automatic Malware Categorization System (AMCS) for automatically grouping malware samples into families that share some common characteristics using a cluster ensemble by aggregating the clustering solutions generated by different base clustering algorithms is developed.
Lecture notes in computer scienceAn Evaluation of Machine Learning-Based Methods for Detection of Phishing Sites
110 Citations2009Daisuke Miyamoto, Hiroaki Hazeyama +1 more
9 machine learning techniques including AdaBoost, Bagging, Support Vector Machines, Classification and Regression Trees, Logistic Regression, Random Forests, Neural Networks, Naive Bayes, and Bayesian Additive Regression trees are employed for detection of phishing sites.
Expert Systems with ApplicationsDesign and implementation of fuzzy expert system for Tehran Stock Exchange portfolio recommendation
109 Citations2010Mehdi Fasanghari, Gholam Ali Montazer
A fuzzy expert system for selecting superior stocks in order to encounter the uncertainty of stock portfolio recommendation is developed and the recommendation rules which experts at Tehran Stock Exchange (TSE) use for portfolio recommendation are modeled.
Mining associations in text in the presence of background knowledge
107 Citations1996Ronen Feldman, Haym Hirsh
FACT takes a query-centered view of knowledge discovery, in which a discovery request is viewed as a query over the implicit set of possible results supported by a collection of documents, and where background knowledge is used to specify constraints on the desired results of this query process.
ACM Computing SurveysA survey of emerging approaches to spam filtering
96 Citations2008Godwin Caruana, Maozhen Li
This survey focuses on emerging approaches to spam filtering built on recent developments in computing technologies, which include peer-to-peer computing, grid computing, semantic Web, and social networks.
Open Research Online (The Open University)An Experiment in Integrating Sentiment Features for Tech Stock Prediction in Twitter
90 Citations2012Tien Thanh Vu, Shu Chang +2 more
A novel model combining features namely positive and negative sentiment, consumer confidence in the product with respect to ‘bullish’ or ‘bearish” lexicon and three previous stock market movement days is proposed.
Decision Support SystemsA text-based decision support system for financial sequence prediction
86 Citations2011Samuel W. K. Chan, James Franklin
A novel text-based decision support system (DSS) that extracts event sequences from shallow text patterns, and predicts the likelihood of the occurrence of events using a classifier-based inference engine, while preserving robustness and without indulging in formalism is proposed.
Decision Support SystemsAssessing the severity of phishing attacks: A hybrid data mining approach
76 Citations2010Xi Chen, Indranil Bose +2 more
This research analyzes 1030 phishing alerts released on a public database as well as financial data related to the targeted firms using a hybrid method that predicts the severity of the attack with up to 89% accuracy.
…