1. kubespray?
– kubespray 는 ansible 기반의 kubernetes 설치 프로젝트.
– 다중 마스터 클러스터를 손쉽게 구축 가능.
– 아래는 kubespray의 github의 설명. ( https://github.com/kubernetes-incubator/kubespray )
- Can be deployed on AWS, GCE, Azure, OpenStack, vSphere or Baremetal
- Highly available cluster
- Composable (Choice of the network plugin for instance)
- Supports most popular Linux distributions
- Continuous integration tests
– 해당 글은 CentOS 7.5( 2018,07,06 최신 ) 에서 설치 되었음.
2. kubespray 사전 준비
2.1. SELinux Disable
|
|
# setenforce 0
# sed -i –follow-symlinks ‘s/SELINUX=enforcing/SELINUX=disabled/g’ /etc/sysconfig/selinux
|
cs |
2.2 Master 서버의 firewall 설정
|
|
# firewall-cmd –permanent –add-port=6443/tcp
# firewall-cmd –permanent –add-port=2379-2380/tcp
# firewall-cmd –permanent –add-port=10250/tcp
# firewall-cmd –permanent –add-port=10251/tcp
# firewall-cmd –permanent –add-port=10252/tcp
# firewall-cmd –permanent –add-port=10255/tcp
# firewall-cmd –reload
# modprobe br_netfilter
# echo ‘1’ > /proc/sys/net/bridge/bridge-nf-call-iptables
# sysctl -w net.ipv4.ip_forward=1
|
cs |
2.3 node ( minion ) 서버의 firewall 설정
|
|
# firewall-cmd –permanent –add-port=10250/tcp
# firewall-cmd –permanent –add-port=10255/tcp
# firewall-cmd –permanent –add-port=30000-32767/tcp
# firewall-cmd –permanent –add-port=6783/tcp
# firewall–cmd —permanent —add–port=2380/tcp
# firewall-cmd –reload
# modprobe br_netfilter
# echo ‘1’ > /proc/sys/net/bridge/bridge-nf-call-iptables
# sysctl -w net.ipv4.ip_forward=1
|
cs |
2.4 Master 서버에 필수 패키지 설치
2.4.0 pip 설치
|
|
# yum install epel-release
# yum install python-pip
# pip –version
pip 8.1.2 from /usr/lib/python2.7/site-packages (python 2.7)
# pip install –upgrade pip
# pip –version
pip 10.0.1 from /usr/lib/python2.7/site-packages/pip (python 2.7)
|
cs |
2.4.1 Ansible 설치
|
|
# pip install ansible
Collecting ansible
.
.
. [ 중략 ]
Successfully installed MarkupSafe-1.0 PyYAML-3.13 ansible-2.6.1 asn1crypto-0.24.0 bcrypt-3.1.4 cffi-1.11.5 cryptography-2.2.2 enum34-1.1.6 idna-2.7 jinja2-2.10 paramiko-2.4.1 pyasn1-0.4.3 pycparser-2.18 pynacl-1.2.1
You are using pip version 8.1.2, however version 10.0.1 is available.
You should consider upgrading via the ‘pip install –upgrade pip’ command.
|
cs |
2.4.2 Jinja2 설치
|
|
# pip install jinja2
Collecting jinja2
.
.
. [ 중략 ]
Successfully installed jinja-1.2
You are using pip version 8.1.2, however version 10.0.1 is available.
You should consider upgrading via the ‘pip install –upgrade pip’ command.
|
cs |
2.4.3 Python 설치
|
|
# yum install python36 –y
Installed:
python36.x86_64 0:3.6.3-7.el7
|
cs |
2.4.4 ansible-modules-hashivault 설치
|
|
# pip install ansible-modules-hashivault==3.8.0
Collecting ansible-modules-hashivault==3.8.0
# pip show ansible-modules-hashivault
Name: ansible-modules-hashivault
Version: 3.8.0
|
cs |
2.4.5 hvac 설치
|
1
2
3
4
5
6
|
# pip install hvac==0.3.0
Collecting hvac==0.3.0
# pip show hvac
Name: hvac
Version: 0.3.0
|
cs |
2.4.6 requests 설치
|
|
# cd /usr/local/src/
# wget https://files.pythonhosted.org/packages/54/1f/782a5734931ddf2e1494e4cd615a51ff98e1879cbe9eecbdfeaf09aa75e9/requests-2.19.1.tar.gz
# tar zxvf requests-2.19.1.tar.gz
# cd requests-2.19.1/
# python setup.py install
——————————————-
Installed /usr/lib/python2.7/site-packages/requests-2.19.1-py2.7.egg
Processing dependencies for requests==2.19.1
Searching for certifi==2018.4.16
Best match: certifi 2018.4.16
Adding certifi 2018.4.16 to easy-install.pth file
Using /usr/lib/python2.7/site-packages
Searching for urllib3==1.23
Best match: urllib3 1.23
Adding urllib3 1.23 to easy-install.pth file
Using /usr/lib/python2.7/site-packages
Searching for idna==2.7
Best match: idna 2.7
Adding idna 2.7 to easy-install.pth file
Using /usr/lib/python2.7/site-packages
Searching for chardet==3.0.4
Best match: chardet 3.0.4
Adding chardet 3.0.4 to easy-install.pth file
Installing chardetect script to /usr/bin
Using /usr/lib/python2.7/site-packages
Finished processing dependencies for requests==2.19.1
|
cs |
2.5 Master 서버의 ssh keygen 생성 및 배포
|
|
# ssh-keygen -t rsa -b 4096 -C “root@192.168.0.11”
# ssh–copy–id root@192.168.0.11
# ssh-copy-id root@192.168.0.12
# ssh-copy-id root@192.168.0.13
# ssh-copy-id root@192.168.0.14
|
cs |
3. kubespray 소스 다운로드
|
|
# cd /usr/local/src
# git clone https://github.com/kubernetes-incubator/kubespray.git
|
cs |
4. kubespray 의존성 페키지 설치 ( requirements.txt 이용 )
|
|
# cd kubespray/
# pip install -r requirements.txt
Installing collected packages: netaddr, pbr, ansible-modules-hashivault
Found existing installation: ansible-modules-hashivault 3.8.0
Uninstalling ansible-modules-hashivault-3.8.0:
Successfully uninstalled ansible-modules-hashivault-3.8.0
Running setup.py install for ansible-modules-hashivault … done
Successfully installed ansible-modules-hashivault-3.9.6 netaddr-0.7.19 pbr-4.1.0
|
cs |
5. inventory에 클러스터 구성 Ansible 업데이트
|
|
# pwd
/usr/local/src/kubespray
# cp -arp inventory/sample inventory/kubecluster
# declare –a IPS=(192.168.0.11 192.168.0.12 192.168.0.13 192.168.0.14)
# CONFIG_FILE=inventory/kubecluster/hosts.ini python36 contrib/inventory_builder/inventory.py ${IPS[@]}
DEBUG: Adding group all
DEBUG: Adding group kube-master
DEBUG: Adding group kube-node
DEBUG: Adding group etcd
DEBUG: Adding group k8s-cluster:children
DEBUG: Adding group calico-rr
DEBUG: Adding group vault
DEBUG: adding host node1 to group all
DEBUG: adding host node2 to group all
DEBUG: adding host node3 to group all
DEBUG: adding host node4 to group all
DEBUG: adding host kube-node to group k8s-cluster:children
DEBUG: adding host kube-master to group k8s-cluster:children
DEBUG: adding host node1 to group etcd
DEBUG: adding host node1 to group vault
DEBUG: adding host node2 to group etcd
DEBUG: adding host node2 to group vault
DEBUG: adding host node3 to group etcd
DEBUG: adding host node3 to group vault
DEBUG: adding host node1 to group kube-master
DEBUG: adding host node2 to group kube-master
DEBUG: adding host node1 to group kube-node
DEBUG: adding host node2 to group kube-node
DEBUG: adding host node3 to group kube-node
DEBUG: adding host node4 to group kube-node
|
cs |
6. 클러스터 구성에 대한 inventory/kubecluster/hosts.ini 설정 ( 위의 5번 단계 진행시 자동 입력됨, 그러나 다른구성을 원한다면 수정 진행 )
|
|
# vi inventory/kubecluster/hosts.ini
[all]
node1 ansible_host=192.168.0.11 ip=192.168.0.11
node2 ansible_host=192.168.0.12 ip=192.168.0.12
node3 ansible_host=192.168.0.13 ip=192.168.0.13
node4 ansible_host=192.168.0.14 ip=192.168.0.14
[kube-master]
node1
node2
[kube-node]
node1
node2
node3
node4
[etcd]
node1
node2
node3
[k8s-cluster:children]
kube-node
kube-master
[calico-rr]
[vault]
node1
node2
node3
|
cs |
7. 네트워크 플러그인 선택 ( 규모에 맞게 설정 진행 )
|
|
vi inventory/kubecluster/group_vars/k8s-cluster.yml
67 # Can also be set to ‘cloud’, which lets the cloud provider setup appropriate routing
68 kube_network_plugin: calico
|
cs |
8. 메트릭스가 클러스터 리소스 사용 데이터 ( nodes, pods ) 를 가저올 수 있도록 주석 제거
|
|
# vi inventory/kubecluster/group_vars/all.yml
132 # The read-only port for the Kubelet to serve on with no authentication/authorization. Uncomment to enable.
133 kube_read_only_port: 10255
|
cs |
9. Ansible Playbook를 이용하여 Kubespray 배포
|
|
# pwd
/usr/local/src/kubespray
# ansible-playbook -i inventory/kubecluster/hosts.ini cluster.yml
.
.
.[중략]
PLAY RECAP *************************************************************************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0
node1 : ok=343 changed=111 unreachable=0 failed=0
node2 : ok=314 changed=107 unreachable=0 failed=0
node3 : ok=281 changed=88 unreachable=0 failed=0
node4 : ok=230 changed=68 unreachable=0 failed=0
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
node1 Ready master,node 3m v1.10.4
node2 Ready master,node 3m v1.10.4
node3 Ready node 3m v1.10.4
node4 Ready node 3m v1.10.4
|
cs |