8 Advanced parallelization - Deep Learning with JAX

Using easy-to-revise parallelism with xmap() · Compiling and automatically partitioning functions with pjit() · Using tensor sharding to achieve parallelization with XLA · Running code in multi-host configurations

High-Performance LLM Training at 1000 GPU Scale With Alpa & Ray

Efficiently Scale LLM Training Across a Large GPU Cluster with

Breaking Up with NumPy: Why JAX is Your New Favorite Tool

Breaking Up with NumPy: Why JAX is Your New Favorite Tool

Running a deep learning workload with JAX on multinode multi-GPU

Lecture 2: Development Infrastructure & Tooling - The Full Stack

Model Parallelism

7 Parallelizing your computations - Deep Learning with JAX

Fully Sharded Data Parallel: faster AI training with fewer GPUs

Why You Should (or Shouldn't) be Using Google's JAX in 2023

Compiler Technologies in Deep Learning Co-Design: A Survey

Compiler Technologies in Deep Learning Co-Design: A Survey

Machine Learning in Python: Main developments and technology