Use KeyLLM, KeyBERT, and Mistral 7B to extract keywords
Large Language Models (LLMs) are becoming smaller, faster, and more efficient. Up to the point where I started to consider them for iterative tasks, like keyword extraction.
Having created KeyBERT, I felt that it was time to extend the package to also include LLMs. They are quite powerful and I wanted to prepare the package for when these models can be run on smaller GPUs.
As such, introducing KeyLLM, an extension to KeyBERT that allows you to use any LLM to extract, create, or even fine-tune the keywords!
In this tutorial, we will go through keyword extraction with KeyLLM using the recently released Mistral 7B model.
We will start by installing a number of packages that we are going to use throughout this example:
pip install --upgrade git+https://github.com/UKPLab/sentence-transformers
pip install keybert ctransformers[cuda]
pip install --upgrade git+https://github.com/huggingface/transformers
We are installing sentence-transformers from its main branch since it has a fix for community detection which we will use in the last few use cases. We do the same for transformers since it does not yet support the Mistral architecture.
You can also follow along with the Google Colab Notebook to make sure everything works as intended.
In previous tutorials, we demonstrated how we could quantize the original model’s weight to make it run without running into memory problems.
Over the course of the last few months, TheBloke has been working hard on doing the quantization for hundreds of models for us.
This way, we can download the model directly which will speed things up quite a bit.
We’ll start with loading the model itself. We will offload 50 layers to the GPU. This will reduce RAM usage and use VRAM instead. If you are running into memory errors, reducing this parameter (gpu_layers) might help!
from ctransformers import AutoModelForCausalLM# Set gpu_layers to the number of layers to offload…
