BigScience Workshop

architecture-objective

bigscience

Central place for the engineering/scaling WG: documentation, SLURM scripts and logs, compute environment and data.

biomedical

Tools for curating biomedical training data for large-scale language modeling

data-preparation

Code used for sourcing and cleaning the BigScience ROOTS corpus

lm-evaluation-harness

A framework for few-shot evaluation of autoregressive language models.

Megatron-DeepSpeed

Ongoing research training transformer language models at scale, including: BERT & GPT-2

petals

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

promptsource

Toolkit for creating, sharing and using natural language prompts.

t-zero

Reproduce results and replicate training fo T0 (Multitask Prompted Training Enables Zero-Shot Task Generalization)

xmtf

Crosslingual Generalization through Multitask Finetuning

bigscience-workshop/petals

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

bigscience-workshop/promptsource

Toolkit for creating, sharing and using natural language prompts.

bigscience-workshop/Megatron-DeepSpeed

Ongoing research training transformer language models at scale, including: BERT & GPT-2

bigscience-workshop/bigscience

Central place for the engineering/scaling WG: documentation, SLURM scripts and logs, compute environment and data.

bigscience-workshop/xmtf

Crosslingual Generalization through Multitask Finetuning

bigscience-workshop/t-zero

Reproduce results and replicate training fo T0 (Multitask Prompted Training Enables Zero-Shot Task Generalization)

bigscience-workshop/biomedical

Tools for curating biomedical training data for large-scale language modeling

bigscience-workshop/data-preparation

Code used for sourcing and cleaning the BigScience ROOTS corpus

bigscience-workshop/lm-evaluation-harness

A framework for few-shot evaluation of autoregressive language models.

bigscience-workshop/architecture-objective

bigscience-workshop/lam

Libraries, Archives and Museums (LAM)

bigscience-workshop/data_tooling

Tools for managing datasets for governance and training.

bigscience-workshop/multilingual-modeling

BLOOM+1: Adapting BLOOM model to support a new unseen language

bigscience-workshop/evaluation

Code and Data for Evaluation WG

bigscience-workshop/metadata

Experiments on including metadata such as URLs, timestamps, website descriptions and HTML tags during pretraining.

bigscience-workshop/model_card

bigscience-workshop/tokenization

bigscience-workshop/carbon-footprint

A repository for `codecarbon` logs.

bigscience-workshop/bloom-dechonk

A repo for running model shrinking experiments

bigscience-workshop/catalogue_data

Scripts to prepare catalogue data

bigscience-workshop/historical_texts

BigScience working group on language models for historical texts

bigscience-workshop/pii_processing

PII Processing code to detect and remediate PII in BigScience datasets. Reference implementation for the PII Hackathon

bigscience-workshop/massive-probing-framework

Framework for BLOOM probing

bigscience-workshop/training_dynamics

bigscience-workshop/transformers

🤗 Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX.

bigscience-workshop/bibliography

A list of BigScience publications

bigscience-workshop/datasets_stats

Generate statistics over datasets used in the context of BS

bigscience-workshop/evaluation-robustness-consistency

Tools for evaluating model robustness and consistency

bigscience-workshop/interpretability-ideas

bigscience-workshop/multilingual-modeling-1