Howto: K3s Test Cluster with Vagrant

Vagrant

Note that following example has been tested on Linux using Vagrant Libvirt ¹ as provider. Some statements in the VagrantFile are Libvirt specific, and need adjustment in case another provider is used.

Name	IP-address	Descriptor
server1	192.168.124.201	K3s Server2 …control-plane and datastore components
agent1	192.168.124.211	K3s Agent …kubelet, container runtime, and CNI
agent2	192.168.124.212	…second K3s agent

Note that port 6443 from the K3s server if forwarded³ to the Vagrant host, in order to enable access from outside kubectl clients.

VagrantFile

Vagrant.configure("2") do |config|

  config.vm.provider :libvirt do |libvirt|
    libvirt.memory = 8192
    libvirt.cpus = 10
    libvirt.qemu_use_session = false
  end
    
  config.vm.box = "almalinux/9"


  config.vm.define "server1" do |node|
    node.vm.hostname = "server1"
    node.vm.network "private_network", ip: "192.168.124.201"
    config.vm.network "forwarded_port", id: "k3s", host: 6443, guest: 6443

    # TODO: ...add configurations to install K3s server
  end

  config.vm.define "agent1" do |node|
    node.vm.hostname = "agent1"
    node.vm.network "private_network", ip: "192.168.124.211"
    
    # TODO: ...add configurations to install K3s agent
  end
  
  config.vm.define "agent2" do |node|
    node.vm.hostname = "agent2"
    node.vm.network "private_network", ip: "192.168.124.212"

    # TODO: ...add configurations to install K3s agent
  end

end

In case an HTTP proxy is required reach the Internet, add the following configurations. Note that this reads the HTTP_PROXY environment variables to configure the services in the Vagrant instance:

VagrantFile

http_proxy=ENV['HTTP_PROXY']
https_proxy=ENV['HTTPS_PROXY']

#  ...https://minikube.sigs.k8s.io/docs/handbook/vpn_and_proxy
etc_environment = %Q{
HTTP_PROXY="#{http_proxy}"
HTTPS_PROXY="#{https_proxy}"
NO_PROXY="localhost,127.0.0.1,10.96.0.0/12,172.16.0.0/12,192.168.0.0/16"
}

config.vm.provision "shell", privileged: true , inline: <<-SHELL
  echo "proxy=#{http_proxy}" >> /etc/dnf/dnf.conf
  echo "#{etc_environment}" >> /etc/environment
SHELL

Configuration

K3s installation documentation⁴

• …installation script available at https://get.k3s.io
• …command flags to pass configuration to the service configuration⁵
• Option --token
  • …provides a shared secret for agents to join the cluster
  • …required to both K3s server & agent during installation
• …configuring an HTTP proxy⁶ with drop-ins {k3s,k3s-agwnt}.service.env

Configure K3s on server1…

• …note that the server argument to the install script is optional
• …deploy the kubeconfig for the vagrant user for debugging

VagrantFile

k3s_token='secret12345'

config.vm.define "server1" do |node|
#…
  node.vm.provision "shell", inline: <<-SHELL
    cp /etc/environment /etc/systemd/system/k3s.service.env
    curl -sfL https://get.k3s.io | sh -s - server\
          --write-kubeconfig-mode 644 --token #{k3s_token}
    sudo mkdir -p /home/vagrant/.kube
    sudo cp /etc/rancher/k3s/k3s.yaml /home/vagrant/.kube/config
    sudo chown -R vagrant:vagrant /home/vagrant/.kube/config
  SHELL
#…

Configure K3s on an agent…

VagrantFile

config.vm.define "agent1" do |node|
#…
  node.vm.provision "shell", inline: <<-SHELL
    cp /etc/environment /etc/systemd/system/k3s-agent.service.env
    curl -sfL https://get.k3s.io | sh -s - agent \
          --server https://192.168.124.201:6443 --token #{k3s_token}
  SHELL
#…

In case debugging is required …check Systemd units and logs

# server systemd unit configuration
systemctl cat {k3s,k3s-agent}

# log information
journalctl -f -u {k3s,k3s-agent}

Usage

Cluster access⁷…

• KUBECONFIG path is /etc/rancher/k3s/k3s.yaml
• …from outside …adjust the server field with the IP or name of your K3s server

vagrant ssh server1 -- cat /etc/rancher/k3s/k3s.yaml > kubeconfig
export KUBECONFIG=$PWD/kubeconfig
kubectl get nodes

Footnotes

1. Vagrant Libvirt Documentation
   https://vagrant-libvirt.github.io/vagrant-libvirt

2. Architecture, K3s Documentation
   https://docs.k3s.io/architecture

3. Forwarded Port, Vagrant Documentation
   https://developer.hashicorp.com/vagrant/docs/networking/forwarded_ports

4. Installation, K3s Documentation
   https://docs.k3s.io/installation

5. Install Configuration, K3s Documentation
   https://docs.k3s.io/installation/configuration

6. Configuring an HTTP proxy, K3s Documentation
   https://docs.k3s.io/advanced?_highlight=proxy#configuring-an-http-proxy

7. Cluster Access, K3s Documentation
   https://docs.k3s.io/cluster-access