Local ML

Running some sort of ML model locally is actually easy if you know Elixir and Phoenix.

The only dependencies you need to add to a Phoenix app are bumblebee and exla. After adding those, add the backend config for nx in config.exs:

config :nx, default_backend: EXLA.Backend

Loading a Model

defmodule MyApp.Model do
  def sentiment_serving() do
    {:ok, bertweet} =
      Bumblebee.load_model({:hf, "finiteautomata/bertweet-base-sentiment-analysis"})

    {:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, "vinai/bertweet-base"})

    Bumblebee.Text.text_classification(bertweet, tokenizer)
  end
end

Loading a model from huggingface is done with Bumblebee. Here we’re loading a simple sentiment analysis model and a matching tokenizer. We then give those to Bumblebee.Text.text_classification/3 to get our Nx.Serving.

The serving is what does the work so let’s add it as a child process of our Phoenix application. Over in application.ex, add this child to the list of children:

{Nx.Serving, name: SentimentServing, serving: MyApp.Model.sentiment_serving()}

Using the Model

To use the serving process, add this function to MyApp.Model:

def get_sentiment(text) do
  Nx.Serving.batched_run(SentimentServing, text)
end

It simply passes the arg(assuming it’s a string) to Nx.Serving.batched_run/3 along with the name of the Nx.Serving child process.

With that function in place, the loaded model can now be used within the Phoenix app. For example, a POST route can be added for this action:

  def sentiment(conn, %{"q" => q}) do
    result = MyApp.Model.get_sentiment(q)
    render(conn, :sentiment, result: result)
  end

It takes a q param, passes it to MyApp.Model.get_sentiment/1, and renders the resulting map as JSON.

Something to keep in mind

Multiple models can be loaded like this as long as the hardware allows.