Today’s paper introduces Generalized Instruction Tuning (GLAN), a novel method for instruction tuning of Large Language Models (LLMs) using a pre-curated taxonomy of human knowledge and capabilities to generate synthetic instruction data across multiple disciplines. This method aims to improve LLMs’ ability to follow human instructions without relying on specific task training data. The main idea is to address the limitations of current instruction tuning methods by proposing a scalable and general approach that leverages a systematic structure inspired by the human education system.

Method Overview

GLAN constructs a taxonomy of human knowledge, breaks it down into disciplines, subjects, and key concepts, and then uses this structured information to generate diverse and comprehensive synthetic instruction data for tuning LLMs. Below you can find a comparison between GLAN and other methods from literature:

GLAN utilizes LLMs like GPT-4 to create a comprehensive taxonomy of human knowledge, organizing it into fields, sub-fields, and disciplines. This taxonomy guides the autonomous generation of subjects for each discipline, then a syllabus for each subject, outlining key concepts and class sessions. Then, GPT-4 is prompted to generate homework questions based on these elements. This methodical breakdown from broad disciplines to atomic-level components like class sessions and key concepts helps generating a comprehensive set of synthetic instruction data that is used for training/fine-tuning of other models. The process is formalized below:

Results

Extensive experiments demonstrate that GLAN significantly enhances the base LLMs’ performance across various tasks, including mathematical reasoning, coding, logical reasoning, and general instruction following, without using task-specific training data. In the experiments, the authors use a Mistral-7B model. Here are the results:

Conclusion

GLAN proposes an interesting way to generate synthetic instruction tuning data for LLMs, showing promising results on multiple tasks. For more informations, please consult the full paper.

Congrats to the authors for their work!

Li, Haoran et al. “Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models.” (2024).