TLDR - Data Augmentation involves generating more labelled data to train deep learning models.Large Language Models can generate large amounts of realistic text data. This survey paper discusses the positive impact of LLMs on DA, including various strategies for using LLMs to generate new training data.

--> For video tutorials on top LLM papers, check Kalyan KS YouTube channel

--> For top LLM papers of the week, check the newsletter.

Introduction

• Data-centric AI is vital for moving towards Artificial General Intelligence (AGI). This approach emphasizes high-quality data to ensure AI systems learn effectively.

• Acquiring and annotating quality data is a major bottleneck due to cost, time, and the potential for human errors during labeling.

• Data augmentation (DA) expands training data by modifying existing data or creating synthetic samples, a valuable solution to limited datasets.

Data Augmentation with Large Language Models (LLMs)

• The critical role of data becomes even more pronounced with LLMs - their success is tied to the availability of massive, high-quality datasets.

• Because human-generated data may be depleted in the near future, synthetic data from LLMs offers a way to continue scaling models.

• LLMs can produce high-quality synthetic data, often exceeding what humans can create, while also reducing data collection costs and energy usage.

Learning Paradigms Shift

• LLMs used for DA have opened up new learning approaches beyond traditional tasks (translation, sentiment analysis, etc.).

• New paradigms include:

  • Instruction Tuning: LLMs learn to refine instructions to improve data generation results.

  • In-Context Learning: LLMs learn from examples within a prompt to better modify or create data samples.

  • Alignment Learning: LLMs align with specific data requirements or formats.

The Importance of a Systematic Review

• Given the growing interest in LLMs for DA, a comprehensive paper on the topic is needed. This paper aims to:

  • Explore DA with LLMs from a data-focused perspective.

  • Investigate learning paradigms where LLMs train on LLM-generated data.

  • Outline key challenges in the field to guide future research.

Paper Structure

The paper delves into various sections, including:

• A comparison to other related surveys

• Data perspective analysis of DA using LLMs

• Exploration of generative and discriminative learning paradigms

• Discussions of challenges and future research potential

• Appendices listing existing DA methods organized by task and domain

For complete details, refer to the survey paper.