Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Colin Raffel, Noam Shazeer, Adam Roberts · and 6 others (Google)

Published 23 October 2019 · Journal of Machine Learning Research (JMLR) · Journal article

Summary

This paper introduces T5 (Text-to-Text Transfer Transformer), a framework that casts every NLP problem—translation, classification, question answering, summarization—as a text-to-text task with a unified model, objective, and decoding procedure. The authors conduct a large-scale empirical study comparing pre-training objectives, architectures, datasets, and transfer strategies, and release the C4 corpus. Scaling the model up to 11 billion parameters achieved state-of-the-art results on many benchmarks.

Key findings

A unified text-to-text format allows a single model and training objective to be applied across diverse NLP tasks.
Systematic comparison identified effective choices for pre-training objective (span corruption), architecture (encoder-decoder), and data scale.
Combining the framework with scale and the new C4 dataset yielded state-of-the-art performance on benchmarks including GLUE, SuperGLUE, and SQuAD.

Subjects & keywords

Artificial Intelligence llm transfer learning text-to-text t5 pretraining

Cite this paper

APA

Colin Raffel, Noam Shazeer, & Adam Roberts [and 6 others (Google)] (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research (JMLR). https://doi.org/10.48550/arXiv.1910.10683

BibTeX

@article{raffel2020exploring,
  author    = {Colin Raffel and Noam Shazeer and Adam Roberts and {and 6 others (Google)}},
  title     = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
  journal   = {Journal of Machine Learning Research (JMLR)},
  year      = {2020},
  doi       = {10.48550/arXiv.1910.10683},
  url       = {https://arxiv.org/abs/1910.10683}
}

Related in Artificial Intelligence

AI2023

Segment Anything

Alexander Kirillov, Eric Mintun and Nikhila Ravi

This paper introduces the Segment Anything project: a promptable image segmentation task, the Segment Anything Model (SAM), and the SA-1B dataset. SAM combines an image encoder, a flexible prompt encoder (points, boxes, masks, text), and a fast mask decoder to produce valid segmentation masks from arbitrary prompts. Trained on over 1 billion masks across 11 million images, SAM shows strong zero-shot transfer to many segmentation tasks without additional training.

IEEE/CVF International Conference on Computer Vision (ICCV) Open access

AI2023

GPT-4 Technical Report

OpenAI

This technical report describes GPT-4, a large-scale multimodal Transformer model that accepts image and text inputs and produces text outputs. The report emphasizes that GPT-4 achieves human-level performance on a range of professional and academic benchmarks, and details infrastructure and optimization methods that allowed performance to be predicted from much smaller models. For competitive and safety reasons, the report withholds architecture, dataset, and training details.

arXiv Open access

AI2023

LLaMA: Open and Efficient Foundation Language Models

Hugo Touvron, Thibaut Lavril and Gautier Izacard

The paper presents LLaMA, a family of foundation language models ranging from 7B to 65B parameters trained exclusively on publicly available datasets. It argues that strong performance can be reached without proprietary data and at smaller parameter counts than prior models. LLaMA-13B outperforms the much larger GPT-3 175B on most benchmarks, and LLaMA-65B is competitive with the best contemporary models such as Chinchilla-70B and PaLM-540B.

arXiv preprint (arXiv:2302.13971) Open access