> ## Documentation Index
> Fetch the complete documentation index at: https://docs.fly.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Add Ollama

You can get an `ollama` image running on a GPU in a few minutes. Get started by adapting the following `fly.toml` file:

```toml theme={null}
app = '<your-app>'
primary_region = 'ams'

[build]
  image = 'ollama/ollama'

[[mounts]]
  source = 'models'
  destination = '/root/.ollama'
  initial_size = '10gb'

[http_service]
  internal_port = 11434
  force_https = false
  auto_stop_machines = 'stop'
  auto_start_machines = true
  min_machines_running = 0
  processes = ['app']

[[vm]]
  size = 'a100-80gb'
```

Modify the app name and region to suit your preferences and save the file as `fly.ollama.toml`.
Then you can launch the app:

```bash theme={null}
fly launch -c fly.ollama.toml --flycast
```

There are a couple of things to note here:

* The `ollama` image is a GPU image, so you need to specify a GPU size in the `[[vm]]` section.
* Not all regions have GPUs available, refer to [region GPU availability](/gpus#regions-with-gpus) for more info
* The `[[mounts]]` section is used to mount a volume to store the models. This is a good practice to keep the models separate from the app code.
* The volume size in this config file is 10gb, that's enough for small models. Change this value if you need more.
* The `--flycast` flag creates a private IPv6 address for the app.

Finally, this *only* starts the `ollama` server; at this point you cannot interact with any models yet.
To do so, you will have to pull in a model with this one easy, short, intuitive command:

```bash theme={null}
fly m run -e OLLAMA_HOST=http://<your-app>.flycast --shell --command "ollama pull llama3.1" ollama/ollama
```

This command will pull in the `llama3.1` model. You can change the model name to suit your needs.
At this point this model is now available to the internal network of the organization it is deployed in.
You can access it using Flycast from this URL: `http://<your-app>.flycast`.

Now that we have a functioning `ollama` with a model, we have to expose the ollama host to our app. One way to do this is to set the host as a secret:

```bash theme={null}
fly secrets set OLLAMA_HOST=http://<your-app>.flycast
```

To interact with our new AI friend, we will have to install the `ollama` package:

```bash theme={null}
poetry add ollama
```

Now we can initialize the client:

```python theme={null}
import os
from ollama import AsyncClient

OLLAMA_HOST = os.getenv('OLLAMA_HOST')
ollama_client = AsyncClient(OLLAMA_HOST)
```

From here we can start integrating it into our app:

```python theme={null}
@app.get("/")
async def read_root():
    resp = await ollama_client.generate(
        model="llama3.1",
        prompt="Why is the sky not green?",
    )
    return resp["response"]
```

When you re-deploy your app you should see llama's answer:

```bash theme={null}
fly deploy
```

You can check out [this gist](https://gist.github.com/fliepeltje/8a8ce9e364097ce61db67cd713799b24) for the complete example app.