Multi-Modal Learning over User-Contributed Content from Cross-Domain Social Media

Proceedings of the AAAI Conference on Artificial IntelligencePublished 5 March 2016Open access

Wen-Yu Lee

DOI10.1609/aaai.v30i1.9813

Source URL

Citations1

SJR score0.13

SNIP0.00

View PDF

TL;DR

This research investigates modern challenges of image annotation, image retrieval, and cross-media mining, followed by presenting promising ways to conquer the challenges.

Abstract

The goal of the research is to discover and summarize data from the emerging social media into information of interests. Specifically, leveraging user-contributed data from cross-domain social media, the idea is to perform multi-modal learning for a given photo, aiming to present people’s description or comments, geographical information, and events of interest, closely related to the photo. These information then can be used for various purposes, such as being a real-time guide for the tourists to improve the quality of tourism. As a result, this research investigates modern challenges of image annotation, image retrieval, and cross-media mining, followed by presenting promising ways to conquer the challenges.

Keywords

Computer Science

MPG.PuRe (Max Planck Society)Learning with Local and Global Consistency

3,750 Citations2003Dengyong Zhou, Olivier Bousquet +3 more

A principled approach to semi-supervised learning is to design a classifying function which is sufficiently smooth with respect to the intrinsic structure collectively revealed by known labeled and unlabeled points.

View PDF

Earthquake shakes Twitter users

3,667 Citations2010Takeshi Sakaki, Makoto Okazaki +1 more

This paper investigates the real-time interaction of events such as earthquakes in Twitter and proposes an algorithm to monitor tweets and to detect a target event and produces a probabilistic spatiotemporal model for the target event that can find the center and the trajectory of the event location.

Lost in binarization

45 Citations2011Yu–Gang Jiang, Jun Wang +1 more

A novel approach that facilitates query-adaptive ranking for the images with equal Hamming distance by firstly offline learning bit weights of the binary codes for a diverse set of predefined semantic concept classes and then rapidly computed by evaluating the proximity between a query and the concept categories.

Journal of Visual Communication and Image RepresentationGraph-based semi-supervised learning with multi-modality propagation for large-scale image datasets

17 Citations2012Wen-Yu Lee, Liang-Chi Hsieh +2 more

This paper proposes a new graph-based SSL technique with multi-label propagation, leveraging the distributed computing power of the MapReduce programming model and presents both a multi-layer learning structure and a tag refinement approach, where the former unifies both visual and textual information of image data during learning, while the latter simultaneously suppresses noisy tags and emphasizes the other tags after learning.

Search-based relevance association with auxiliary contextual cues

13 Citations2013Chun-Che Wu, Kuan-Yu Chu +4 more

This work proposes a relevance association by investigating the effectiveness of different auxiliary contextual cues (i.e., face, click logs, visual similarity) and shows that the proposed method can have 16% relative improvement compared to the original ranking results.

Discovering the City by Mining Diverse and Multimodal Data Streams

11 Citations2014Yin-Hsi Kuo, Yanying Chen +10 more

This work attempts to tackle the IBM grand challenge - seeing the daily life of New York City (NYC) in various perspectives by exploring rich and diverse social media content by mining a huge amount of diverse and freely available media.

Augmenting mobile city-view image retrieval with context-rich user-contributed photos

8 Citations2011Yin-Hsi Kuo, Wen-Yu Lee +2 more

This work proposes to augment the visual diversity of database images by integrating the context-rich user-contributed photos from social media by combining both visual and GPS constraints in the retrieval process on inverted indexing so that the system can achieve a real-time retrieval system.

City-view image retrieval leveraging check-in data

2 Citations2013Wen-Yu Lee, Yin-Hsi Kuo +1 more

A simple approach that unifies visual features, geo-tags, and check-in data for city-view image retrieval is presented and sparse coding is applied for a high-performance and memory-efficient retrieval implementation, where a graph-based dictionary learning approach is proposed.

Unsupervised Latent Aspect Discovery for Diverse Event Summarization

2 Citations2015Wen-Yu Lee, Yin-Hsi Kuo +8 more

This paper proposes effective methods on detecting events with human attribute discovery, such as human pose and clothes, scene analysis, image aspect discovery, and temporal and semantic analysis, to provide people different perspectives for the events they are interested in.