Serving Quantized LLMs on NVIDIA H100 Tensor Core GPUs
Quantization is a technique for making machine learning models smaller and faster. We quantize Llama2-70B-Chat, producing an equivalent-quality model that generates 2.2x more...
Page 1
LLM Inference Performance Engineering: Best Practices
In this blog post, the MosaicML engineering team shares best practices for how to capitalize on popular open source large language models (LLMs)...
MosaicML Delivers Leading NLP Performance in MLPerf v2.1
MosaicML leads the MLPerf NLP results, delivering a score of 7.9 minutes on 8x NVIDIA A100 GPUs in the Open Division, thanks to...