Setup remote Minikube instance

Things to do or know before getting started

If you are tired of running Minikube on your working PC / laptop you can (with some effort) move it to another device which only runs Minikube.

Once you have installed Minikube on the remote device, I also want to show you how to setup your working PC to point to your remote device’s running instance of Minikube.

This post is based on the work of this article.

But… I am going to make it more streamlined and provide more detail to make it super clear how one does it.

For this post one needs to understand the basics of Linux and TCP/IP but I assume you would already 🙂

First off, you would need to decide which remote device you want to install Minikube on and that it’s reachable from your working PC over your local network.

Before installing Minikube , have a look at the minimum hardware requirements to see if your remote device would be suitable to run Minikube.

In this post I am using an Intel NUC to install Minikube on. I would recommend getting a similar mini PC with at least 16 GB RAM and 4 CPU cores or more.

If you have such a mini PC to utilize then let’s get started!

Prepping the NUC

For the NUC I am going to use Ubuntu Desktop Linux as the host OS. You can use another OS as well, but that is your choice. This post is based on using Ubuntu Linux.

If you have a remote device with Ubuntu already on it, that’s great! Otherwise you would need to reformat your device. I am not going to show the steps to reformat your device in this post but rather in another post.

At this point in time, you would need to perform the following steps:

Step 1: Figure out the IP address of the NUC. This can be done by executing ip a in Ubuntu terminal. Mine is 192.168.0.104

Step 2: Create a host entry on your working PC in the /etc/hosts file. For this post I am going to map the minikube.local host alias to 192.168.0.104 as follows:

$ echo '192.168.0.104 minikube.local' | sudo tee -a /etc/hosts

Step 3: Setup SSH access to the NUC. By default SSH access is disabled on Ubuntu but it can be setup by executing the following commands:

# apt-get update
# apt-get install ssh
# ufw allow 22

Step 4: Install the following software on the NUC:

Software	Install instructions
Docker	Click here
VirtualBox	# apt-get install virtualbox
NGINX	# apt-get install nginx
kubectl	Click here

Setting up Minikube on the NUC

Downloading and installing Minikube on the NUC is pretty easy, just execute the following commands within a terminal on the NUC:

$ curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64

$ chmod +x minikube
# install minikube /usr/local/bin/

Now we are going to start up Minikube for the first time. I have create a little script here to set some flags beforehand. Essentially the format of the command is as follows:

$ minikube start \
    --driver=<driver> \
    --apiserver-names=<apiserver-names> \
    --kubernetes-version=<kubernetes-version> \
    --addons=<addons> \
    --memory=<memory> \
    --cpus=<cpus>

Here is a breakdown of what these flags mean (See full list of flags here):

Flag	What does it mean?
driver	Type of underlying driver to use. See all options here
apiserver-names	Alternative hostnames to allow access to your cluster’s API server (as well as the Docker Daemon within Minikube)
kubernetes-version	Defaults to latest, but you can specify it if your deployment is not compatible to the latest Kubernetes version yet
addons	List of addons to enable when creating Minikube. Addons can be enabled after creating a Minikube cluster as well. See all available addons here
memory	Total amount of megabytes to allocate to your cluster (e.g. 4096, 8192, etc.)
cpus	Total amount of CPU’s to allocate to your cluster. If not specified it takes the max amount available

NOTE: For this exercise it is necessary to use the virtualbox driver as we are going to use VirtualBox to host Minikube in it. VirtualBox makes it easier to trash Minikube if you want to start from scratch again.

Example command which I am going to execute:

$ minikube start \
    --driver=virtualbox \
    --apiserver-names=minikube.local \
    --kubernetes-version=v1.23.3 \
    --addons=ingress \
    --memory=8192 \
    --cpus=4

Using the mk-start.sh script it should start Minikube on the NUC with an auto-generated IP. For me it was 192.168.59.101 but yours could differ.

To check if everything is running on the NUC, just execute:

$ kubectl get nodes

To obtain your Minikube’s IP execute this:

$ minikube ip

Setting up port forwarding on the NUC

As Minikube contains a Kubernetes cluster and its own Docker Daemon within we need to expose well-known ports to be accessible from the host OS.

To achieve this I have used NGINX to make the port forwarding configuration external to the Minikube instance. This way you don’t have to worry about losing configuration if you destroy a MInikube instance.

Before configuring NGINX, it is a good idea to make a host entry on the NUC to map the generated Minikube IP to a host alias such as minikube.internal. This way if your Minikube’s IP changes you only need to update the host entry.

To do this, you can take the Minikube IP from the previous section and add it on the NUC as follows:

$ echo '192.168.59.101 minikube.internal' | sudo tee -a /etc/hosts

Then add the following config to /etc/nginx/nginx.conf located on the NUC:

stream {
   
   # Docker Daemon within Minikube
   server {
       listen 0.0.0.0:2376;
       proxy_pass minikube.internal:2376;
   } 
   
   # HTTP
   server {
       listen 0.0.0.0:80;
       proxy_pass minikube.internal:80;
   }   
   
   # HTTPS
   server {
       listen 0.0.0.0:443;
       proxy_pass minikube.internal:443;
   }
   
   # K8s API Server
   server {
       listen 0.0.0.0:8443;
       proxy_pass minikube.internal:8443;
   }    
   
   # Minikube SSH
   server {
       listen 0.0.0.0:2222;
       proxy_pass minikube.internal:22;
   } 

}

After you have added the config you should restart NGINX to take the config into effect.

# systemctl restart nginx

Sorting out the certificates

When Minikube starts for the first time it generates X.509 encoded certificates for the Docker Daemon within Minikube and other certificate for accessing the Kubernetes API Server. The certificates would usually be located at ~/.minikube on the NUC with the following structure:

devnull@nuc:~/.minikube$ ls
addons  ca.crt  ca.pem    certs   files    logs machines  proxy-client-ca.crt
cache   ca.key  cert.pem  config  key.pem  machine_client.lock  profiles  proxy-client-ca.key

The following files and folder are required to be copied to your working PC:

FIle/folder name	What does it contain?
ca.crt	This file is the CA certificate of your Kubernetes cluster running on Minikube
certs	This folder contains the certificates for the Docker Daemon running within Minikube
profiles	This folder contains the keys to access your Kubernetes cluster using the Minikube user

Not going to more further detail about what the purpose of each of these certificates are and will just supply the commands to copy the necessary files over to your working PC using scp

At this point you must have a terminal open on your working PC and have created a folder to contain the files being copied from the NUC to your working PC.

I have created a .minikube folder in my home directory on my working PC.

$ mkdir ~/.minikube

Using scp makes it quite simple to retrieve files from remote devices. The format of the command is as follows:

scp [-r] <username>@<remote-host>:<remote-path> <local-path>

Now we are going to copy files from the NUC to your working PC, follow these steps:

Step 1: Copy the ca.crt as follows:

$ scp devnull@minikube.local:/home/devnull/.minikube/ca.crt ~/.minikube/ca.crt

Step 2: Copy over the Docker Daemon certificates:

$ scp -r devnull@minikube.local:/home/devnull/.minikube/certs ~/.minikube/certs

Step 3: Copy over the Kubernetes certificates for accessing the cluster’s API server using the minikube user:

$ scp -r devnull@minikube.local:/home/devnull/.minikube/profiles ~/.minikube/profiles

If you are familiar with Kubernetes, you would be familiar with the Kubernetes config file that is usually located at ~/.kube/config .

There are two ways to create this config file:

First method: Copy the one from the NUC by doing the following:

$ scp -r devnull@minikube.local:/home/devnull/.kube/config ~/.kube/config

But now you need to fix the file paths in the config file to point to your home directory and change the IP of the cluster server to point to your NUC or in my case minikube

Second method: Using a YAML template and envsubst:

Create an environmental variable called KUBE_HOST (which points to the NUC’s hostname), for example:

$ export KUBE_HOST=minikube.local

And then create ~/.kube/config.template file and copy the following into that file:

apiVersion: v1
clusters:
- cluster:
    certificate-authority: ${HOME}/.minikube/ca.crt
    server: https://${KUBE_HOST}:8443
  name: minikube
contexts: 
- context:
    cluster: minikube
    namespace: default  
    user: minikube
  name: minikube  
current-context: minikube
kind: Config   
preferences: {}
users:
- name: minikube
  user:
    client-certificate: ${HOME}/.minikube/profiles/minikube/client.crt
    client-key: ${HOME}/.minikube/profiles/minikube/client.key

And then execute the following to replace the environment variables:

$ envsubst < config.template > config

After you have done it you should be able to see the pods running within Minikube by executing the following on your working PC:

$ kubectl get pods -A

Next we need to configure our working PC’s Docker client to point to your remote Minikube’s Docker instance.

There are two methods to set it up, just depends which you prefer:

First method: Create an bash script to set specific environment variables. Using a template would make it suitable for your environment, See template below:

export DOCKER_TLS_VERIFY=1
export DOCKER_HOST="tcp://${KUBE_HOST}:2376"
export DOCKER_CERT_PATH="${HOME}/.minikube/certs"

Create a new file (e.g. docker-env.template) and copy above code block.

Then execute the following to fill in the placeholders:

$ envsubst < docker-env.template > docker-env

To apply these environmental variables, you simply source them as follows:

$ source docker-env

Now you would be able to execute docker ps and see all the containers running within Minikube.

Second method: Creating a Docker Context. This is more suitable if you are working with different Docker daemons on the same machine and need to switch between them. To create a context execute the following command:

docker context create mk-nuc \
  --docker "host=tcp://minikube.local:2376,ca=$HOME/.minikube/certs/ca.pem,cert=$HOME/.minikube/certs/cert.pem,key=$HOME/.minikube/certs/key.pem"

I have chosen to name for the context as mk-nuc but it could be anything really.

To activate context you simply execute:

$ docker context use mk-nuc

I prefer this method as it persists the Docker context between terminal sessions.

If you want to switch back to your local Docker daemon, then execute the following:

$ docker context use default

NOTE: The Docker Context is not used when using Docker Compose. If you want to use the Minikube’s Docker with Docker Compose, then I would advise to use the first method by using the environmental variables.

You can use Helm by now as it reads the Kube config file to know where to deploy your application to.

That should be it, you are all set up now!

Final note: This may not be the best way to setup a remote instance of Minikube, but it suits my needs thus far. I will update this post if there are ways that I find would improve the configuration.

Errata

13 Jul 2022: Changed hostname from minikube to minikube.local due to issues when DNS lookups are being performed