New to Guard? Please start here.

Kubespray Installation Guide

Kubespray (originally called Kargo) is a preferred choice for deploying customized production grade Kubernetes clusters, particularly for those who are comfortable with Ansible. Using Kubespray, you can deploy a cluster on GCE, Azure, OpenStack, AWS, or Baremetal, along with a choice of various network plugins.

• This document shows how to configure GitHub as Kubernetes authenticator, the configuration for other authenticators should be similar.
• The document can be followed on a running cluster or before creating a new Kubernetes Cluster.
• All files referenced in this documents are in guard branch of appscode/kubespray repository

Before You Begin

• Guard installed on OS X or Linux
• GitHub Acccount with an organization and team(s)
• GitHub Token with public_repo and read:org access

Guard Setup

Initialize PKI

This command creates a self-signed certificate authority (CA) and a key.

$ guard init ca
$ tree .guard/
.guard/
└── pki
    ├── ca.crt
    └── ca.key

1 directory, 2 files

Initialize Guard Server

To initialize a Guard server, you just need a free cluster IP. Choose an IP from the IP range while installing Kubernetes Cluster through Kubespray. This can be found in k8s-cluster.yml. We have chosen IP 10.233.0.27 in this document.

$ guard init server --ips=10.233.0.27
$ tree .guard/
.guard/
└── pki
    ├── ca.crt
    ├── ca.key
    ├── server.crt
    └── server.key

1 directory, 4 files

It is possible to use guard server DNS address instead of a cluster IP. Please visit here to learn the steps and its implications.

Initialize Guard Clients

Here, we are using GitHub as a Guard auth provider.

$ guard init client <your_github_org> -o github
$ tree .guard/
.guard/
└── pki
    ├── ca.crt
    ├── ca.key
    ├── server.crt
    ├── server.key
    ├── <your_github_org>@github.crt
    └── <your_github_org>@github.key
1 directory, 6 files

Modifications in Kubespray Repository

Clone the Kubespray git repository or the already clones repository which you have used to install Kubernetes Cluster.

Note: All the following files and commands are relative to the root of the git repository. Hyperlinks are also used a lot to make things easier.

• Enable Guard and optionally RBAC policies in k8s-cluster.yml

kube_guard: true
kube_guard_rbac_policies: false

Note: kube_guard_rbac_policies if set to true assumes that you have teams named admins and developers in your GitHub Organization. Feel free to edit the files to match your environment. GitHub users in admins group get full access to Kubernetes Cluster and users in developers get full access to only default namespace.

• kubernetes Ansible role: Include Guard Auth Token File

$ mkdir roles/kubernetes/master/files
$ guard get webhook-config <your_github_org> -o github --addr=10.233.0.27:443 > roles/kubernetes/master/files/guard_auth_token_file

• kubernetes Ansible role: Create guard-file.yml

$ curl -sL https://raw.githubusercontent.com/appscode/kubespray/guard/roles/kubernetes/master/tasks/guard-file.yml > roles/kubernetes/master/tasks/guard-file.yml

• kubernetes Ansible role: Import guard-file.yml as a task in main.yml

- import_tasks: guard-file.yml
  when: kube_guard|default(false)

• kubernetes Ansible role: Update Kube API Server Deployment template with Guard Auth Token

{% if kube_guard|default(false) %}
    - --authentication-token-webhook-config-file={{ kube_token_dir }}/guard
{% endif %}

• guard-auth-webook Ansible Role: Add an ansible role to configure Guard

This role enables Guard server on Kubernetes Master nodes.

$ mkdir -pv roles/guard-auth-webook/{files,tasks}

• guard-auth-webook Ansible Role: Copy all files and directories from guard-auth-webook role

At the time of writing this document, the structure looks like this.

$ tree roles/guard-auth-webook
roles/guard-auth-webook
├── files
│   ├── cluster-admin-role.yml
│   ├── default-namespace-admin-role.yml
│   └── guard-installer.yml
└── tasks
    ├── main.yml
    ├── rbac-guard.yml
    └── setup-guard.yml

2 directories, 6 files

• guard-auth-webook Ansible Role: Create Guard Deployment File

$ guard get installer \
  --auth-providers=github \
  --namespace="kube-system" \
  --addr="10.233.0.27:443" > roles/guard-auth-webook/files/guard-installer.yml

The generated file creates these Kubernetes assets packaged in a yaml file

• serviceaccount
• clusterrole
• clusterrolebinding
• deployment
• secret
• service

By default, Guard installer yamls will run it on master instances. Kubespray does not label the master nodes correctly. See this issue for details. You can fix by running the below command.

$ sed -i "s/node-role.kubernetes.io\/master: \"\"/app: guard/" roles/guard-auth-webook/files/guard-installer.yml

• Add the new Ansible Role

Update cluster.yml with the guard-auth-webook role

- hosts: kube-master[0]
  roles:
    - { role: guard-auth-webook, when: kube_guard }

Run Ansible Playbook to update or install Kubernetes Cluster

$ ansible-playbook -i inventory/hosts cluster.yml

Add a GitHub User

Add a Github user which is in your organization.

$ kubectl config set-credentials <github-user> --token=<your-github-token>

Check if everything went as desired

• When RBAC policies are disabled.

$ kubectl get pods --user <github-user>
Error from server (Forbidden): pods is forbidden: User "<github-user>" cannot list pods in the namespace "default"

The Forbidden message here is a good sign, this means you have been authenticated but you do not have proper authorization. Now, you can start creating RBAC policies.

• When RBAC policies are enabled (your output may slightly vary)

$ kubectl --user <github-user> get nodes
NAME                    STATUS    ROLES     AGE       VERSION
kubernetes-master0      Ready     master    1d        v1.9.5
kubernetes-master1      Ready     master    1d        v1.9.5
kubernetes-worker0      Ready     node      1d        v1.9.5
kubernetes-worker1      Ready     node      1d        v1.9.5

Troubleshooting

Ensure guard file is present in Kubernetes API Server

$ kubectl -n kube-system exec -it kube-apiserver-kubernetes-lab-master0 -- ls /etc/kubernetes/tokens/guard
/etc/kubernetes/tokens/guard
$

$ grep authentication-token-webhook-config-file /etc/kubernetes/manifests/kube-apiserver.manifest
    - --authentication-token-webhook-config-file=/etc/kubernetes/tokens/guard

Check Logs of API Server

$ kubectl -n kube-system logs kube-apiserver-kubernetes-lab-master0

Support

We use Slack for public discussions. To chit chat with us or the rest of the community, join us in the AppsCode Slack team channel #guard. To sign up, use our Slack inviter.

If you have found a bug with Guard or want to request for new features, please file an issue.