Latest advances in massive language fashions (LLMs) like GPT-4,  PaLM have led to transformative capabilities in pure language duties. LLMs are being integrated into numerous purposes comparable to chatbots, search engines like google and yahoo, and programming assistants. Nevertheless, serving LLMs at scale stays difficult as a consequence of their substantial GPU and reminiscence necessities.…