I built this little example of how to use www.robbie.run to perform batch LLM inference on customer service conversations to identify topics and sentiment.

Here's what I used:

• meta-llama/Llama-3.1-8B-Instruct
• Open source instruct prompts from https://developers.endeavorcx.com/docs/topic-level-sentiment-analysis
• vLLM Inference framework
• Ubuntu Dialog Corpus for the chat conversations (Kaggle) - that i cleaned up with Pandas but the file is 1.39 GB
• 2 A100 40G GPUs from Robbie (to demonstrated distributed inference)

Thanks to Chris Crosby for the prompts.

Example Customer Conversations in the .csv:

Inference running on the remote machine!

Json Output with topics and sentiment.

Here's the repo link: https://github.com/Positron-Networks/robbie-examples/tree/main/command_runner/llama3-batch-inference