Do Interesting Things with Livebook and Your Production App

Livebook is an exciting advancement in the Elixir community. It was created for the Elixir machine learning library nx. You can think of it as Elixir’s version of Jupyter Notebooks. Livebook ends up being very flexible and powerful. Because Elixir nodes can easily be clustered together, you can run Livebook on your local machine, connect it to your Elixir servers, and do some really interesting things. Once you have Livebook connected to your app on Fly, you are ready to start playing!

Getting Started with Livebook

Once you have Livebook running locally, visit the “Learn” section for examples of things you can do. Also, check out the Livebook News to find interesting developments and to spark new ideas. Here are a few ideas of things you can do when Livebook is connected to your server.

Query your users grouping by an interesting attribute and graph it.
Chart user sign-ups over time.
Analyze customer orders in a table.
Create scripts in the form of notebooks to run common administrative commands.

Business Intelligence Through Livebook

Let’s walk through using Livebook to get Business Intelligence (BI) data from your Elixir app.

Visualization Setup

With Livebook v0.2, support was added for visualizations through the Elixir packages VegaLite and Kino. In this example, we’re connecting a locally running Livebook to our app running on Fly. In order for the visualization tools to work, these libraries need to be installed on our server. You may have seen Livebook examples showing how we can Mix.install the needed graphics libraries. That doesn’t work when you are attaching to a remote mix project. It’s actually a good thing that we can’t just Mix.install like that on a remote project. You wouldn’t want anyone with access to attach to your live system and install random packages! That just means we need to install the dependencies first. Add the vega_lite and kino libraries to your project’s mix.exs file and deploy your app.

  defp deps do
    [
      {:phoenix, "~> 1.5.6"},
      # ...

      # Graphing/charting - for Livebook or analysis
      {:vega_lite, "~> 0.1.0"},
      {:kino, "~> 0.1.0"}
    ]
  end

With the updated app deployed to Fly, Livebook can connect and render charts and visualizations!

Starting Your Analysis Notebook

While you are figuring out what you want to examine in your app, you can explore it locally on your dev machine. Once you start, you may find you want to update your project code. Also, while playing locally, you reduce the chance of a badly structured query negatively impacting your live system. I like starting Livebook using the EScript method. Once installed, in a terminal you just go to the directory where your project’s source code lives and start Livebook from there.

livebook server

Create a Notebook

Open Livebook in a browser and create a “New notebook”. For the first notebook, let’s keep it really generic because I don’t know what makes sense for your project. Name the notebook something like “Exploring Data”. Create a section for “Setup” and one for “Analysis”. Save the notebook with your project. I suggest creating a “notebook” folder to hold them so they can be checked in with your project and shared with the team.

Connect to Your Project

At this point, we haven’t connected Livebook to our project so we can’t access the project’s code yet. To do this, start the project with something similar to the following command:

iex --sname my_app --cookie secret -S mix phx.server

This starts our local project with a short name of “my_app” and a cookie of “secret”. Back in Livebook, click the “Runtime settings” button.

From here, click “Configure” then “Attached Node”. Now, enter the name “my_app” and the cookie “secret” (or whatever values were used when starting the server) and click “Connect”. If everything went well, Livebook is connected to your locally running application! We can verify it’s connected by typing a module from the project into a code cell in the notebook and getting auto-complete help that’s specific to our project.

Your First Analysis

This is where everything gets specific to the project. I’ll walk through an example so we can see how to pull data, aggregate it, and visualize it. Then we’ll turn you loose on your project! My project contains survey-style questions. Some questions use a “personality” type field. My first goal is just to get the data and some counts to see how it’s distributed. Under Setup, I add an Elixir cell and the following code:

alias VegaLite, as: Vl
alias Core.Schemas.Question
alias Core.Repo
import Ecto.Query

The important one for visualizations is:

alias VegaLite, as: Vl

My first query groups the questions by the personality type value and returns the count.

query =
  from(q in Question,
    group_by: q.personality,
    select: {q.personality, count(q.id)}
  )

personality_counts =
  query
  |> Repo.all()
  |> Enum.into(%{})

The results look like this:

%{any: 77, extrovert: 10, introvert: 13}

That’s helpful already, but it’s still just data. I’d like to see what that looks like. I add another Elixir cell and write the following code:

# Convert the raw data into a Pie Chart friendly format
data =
  Enum.map(personality_counts, fn {type, count} ->
    %{"personality" => Atom.to_string(type), "value" => count}
  end)

Vl.new()
|> Vl.data_from_values(data)
|> Vl.mark(:arc)
|> Vl.encode_field(:theta, "value", type: :quantitative)
|> Vl.encode_field(:color, "personality", type: :nominal)
|> Vl.config(view: [stroke: nil])

The Vl is the VegaLite alias. Here’s the result:

Nice! I get a better sense for the personality type distribution in the question data. What if I want to render the data as bar charts?

# Convert the raw data into a Bar Chart friendly format
data =
  Enum.map(personality_counts, fn {type, count} ->
    %{"Personality" => Atom.to_string(type), "Question Count" => count}
  end)

# Sort the data by counts to order the results
data = Enum.sort_by(data, & &1["Question Count"], :desc)

Vl.new(width: 300, height: 300)
|> Vl.data_from_values(data)
|> Vl.mark(:bar)
|> Vl.encode_field(:x, "Personality",
  type: :nominal,
  axis: [label_angle: 0],
  sort: [field: "Question Count", order: "descending"]
)
|> Vl.encode_field(:y, "Question Count", type: :quantitative)

Here’s the result:

Getting Started with VegaLite

If the VegaLite code looks scary, don’t worry! I’m actually a total VegaLite noob! Livebook includes some built-in notebooks you should check out for code examples on how to structure data and configure VegaLite for the visualization you want.

Then I customized a little using the VegaLite docs and examples.

Summary

Livebook is an awesome tool for BI on your Elixir applications. Because it’s going directly to your code and not just your data, you can use your code to query integrated services like Stripe and do awesome things!

​Getting Started with Livebook

​Business Intelligence Through Livebook

​Visualization Setup

​Starting Your Analysis Notebook

​Create a Notebook

​Connect to Your Project

​Your First Analysis

​Getting Started with VegaLite

​Summary