首页 > 代码库 > k8s1.5.4挂载volume之glusterfs
k8s1.5.4挂载volume之glusterfs
k8s1.5.4挂载volume之glusterfs
volume的例子集合
https://github.com/kubernetes/kubernetes/tree/master/examples/volumes
http://www.dockerinfo.net/2926.html
http://dockone.io/article/2087
https://www.kubernetes.org.cn/1146.html
https://kubernetes.io/docs/user-guide/volumes/
k8s集群安装部署
http://jerrymin.blog.51cto.com/3002256/1898243
k8s集群RC、SVC、POD部署
http://jerrymin.blog.51cto.com/3002256/1900260
k8s集群组件kubernetes-dashboard和kube-dns部署
http://jerrymin.blog.51cto.com/3002256/1900508
k8s集群监控组件heapster部署
http://jerrymin.blog.51cto.com/3002256/1904460
k8s集群反向代理负载均衡组件部署
http://jerrymin.blog.51cto.com/3002256/1904463
k8s集群挂载volume之nfs
http://jerrymin.blog.51cto.com/3002256/1906778
glusterfs参考文档
https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/glusterfs
http://gluster.readthedocs.io/en/latest/Administrator%20Guide/
http://blog.gluster.org/2016/03/persistent-volume-and-claim-in-openshift-and-kubernetes-using-glusterfs-volume-plugin/
https://docs.openshift.org/latest/install_config/persistent_storage/persistent_storage_glusterfs.html
The example assumes that you have already set up a Glusterfs server cluster and the Glusterfs client package is installed on all Kubernetes nodes.
需要先在node节点安装部署gluster集群环境，可以参考http://www.linuxidc.com/Linux/2017-02/140517.htm
1，glusterfs服务端和客户端环境部署：
CentOS 安装 glusterfs 非常的简单
在三个节点都安装glusterfs
yum install centos-release-gluster
yum install -y glusterfs glusterfs-server glusterfs-fuse glusterfs-rdma
配置 GlusterFS 集群：
启动 glusterFS
systemctl start glusterd.service
systemctl enable glusterd.service
配置
[root@k8s-master glusterfs]# cd /usr/local/kubernetes/examples/volumes/glusterfs
[root@k8s-master glusterfs]# ls
glusterfs-endpoints.json glusterfs-pod.json glusterfs-service.json README.md
[root@k8s-master glusterfs]# gluster peer probe k8s-master
peer probe: success. Probe on localhost not needed
[root@k8s-master glusterfs]# gluster peer probe k8s-node1
peer probe: success.
[root@k8s-master glusterfs]# gluster peer probe k8s-node2
peer probe: success.
查看集群状态
[root@k8s-master glusterfs]# gluster peer status
Number of Peers: 2
Hostname: k8s-node1
Uuid: 4853baab-e8fb-41ad-9a93-bfb5f0d55692
State: Peer in Cluster (Connected)
Hostname: k8s-node2
Uuid: 2c9dea85-2305-4989-a74a-970f7eb08093
State: Peer in Cluster (Connected)
创建数据存储目录
[root@k8s-master glusterfs]# mkdir -p /data/gluster/data
[root@k8s-node1 ~]# mkdir -p /data/gluster/data
[root@k8s-node2 ~]# mkdir -p /data/gluster/data
创建volume使用默认DHT也叫分布卷: 将文件已hash算法随机分布到 一台服务器节点中存储。
[root@k8s-master glusterfs]# gluster volume create glusterfsdata replica 3 k8s-master:/data/gluster/data k8s-node1:/data/gluster/data k8s-node2:/data/gluster/data force
volume create: glusterfsdata: success: please start the volume to access data
查看volume信息
[root@k8s-master glusterfs]# gluster volume info
Volume Name: glusterfsdata
Type: Replicate
Volume ID: 100d1f33-fb0d-48c3-9a93-d08c2e2dabb3
Status: Created
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: k8s-master:/data/gluster/data
Brick2: k8s-node1:/data/gluster/data
Brick3: k8s-node2:/data/gluster/data
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
启动volume
[root@k8s-master glusterfs]# gluster volume start glusterfsdata
volume start: glusterfsdata: success
测试客户端
[root@k8s-master glusterfs]# mount -t glusterfs k8s-master:glusterfsdata /mnt
[root@k8s-master glusterfs]# df -h|grep gluster
k8s-master:glusterfsdata 422G 934M 421G 1% /mnt
[root@k8s-master mnt]# echo glusterfs > glusterfs
[root@k8s-master mnt]# cat glusterfs
glusterfs
2，k8s集群挂载glusterfs
注意endpoint改成glusterfs服务集群的一个节点的IP，配置文件是2个IP所以填2个节点IP
[root@k8s-master glusterfs]# vim glusterfs-endpoints.json
[root@k8s-master glusterfs]# kubectl create -f glusterfs-endpoints.json
endpoints "glusterfs-cluster" created
[root@k8s-master glusterfs]# kubectl get ep |grep glusterfs
glusterfs-cluster 172.17.3.7:1,172.17.3.8:1 1m
[root@k8s-master glusterfs]# kubectl create -f glusterfs-service.json
service "glusterfs-cluster" created
注意volumeMounts挂载name改成上面新建的glusterfs volume
"volumes": [
{
"name": "glusterfsvol",
"glusterfs": {
"endpoints": "glusterfs-cluster",
"path": "glusterfsdata",
"readOnly": true
}
}
]
[root@k8s-master glusterfs]# kubectl create -f glusterfs-pod.json
pod "glusterfs" created
[root@k8s-master glusterfs]# kubectl get pod -o wide |grep glus
glusterfs 1/1 Running 0 4m 10.1.39.8 k8s-node1
[root@k8s-node1 ~]# mount | grep gluster
172.17.3.7:glusterfsdata on /var/lib/kubelet/pods/61cd4cec-0955-11e7-a8c3-c81f66d97bc3/volumes/kubernetes.io~glusterfs/glusterfsvol type fuse.glusterfs (ro,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
可见github上面例子不太直观，版本比较老，但是过程比较清楚。这里也可以参考nfs挂载把glusterfs挂载到nginx站点，那样测试更直观。下面是升级测试
创建PV/PVC
[root@k8s-master glusterfs]# cat glusterfs-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: gluster-default-volume
spec:
capacity:
storage: 8Gi
accessModes:
- ReadWriteMany
glusterfs:
endpoints: "glusterfs-cluster"
path: "glusterfsdata"
readOnly: false
[root@k8s-master glusterfs]# cat glusterfs-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: glusterfs-claim
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 8Gi
[root@k8s-master glusterfs]# kubectl create -f glusterfs-pv.yaml
persistentvolume "gluster-default-volume" created
[root@k8s-master glusterfs]# kubectl create -f glusterfs-pvc.yaml
persistentvolumeclaim "glusterfs-claim" created
[root@k8s-master glusterfs]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
glusterfs-claim Bound gluster-default-volume 8Gi RWX 2m
创建nginx站点挂载glusterfs-claim
[root@k8s-master glusterfs]# kubectl create -f glusterfs-web-rc.yaml
replicationcontroller "glusterfs-web" created
[root@k8s-master glusterfs]# kubectl create -f glusterfs-web-service.yaml
service "glusterfs-web" created
配置文件如下
[root@k8s-master glusterfs]# cat glusterfs-web-rc.yaml
# This pod mounts the nfs volume claim into /usr/share/nginx/html and
# serves a simple web page.
apiVersion: v1
kind: ReplicationController
metadata:
name: glusterfs-web
spec:
replicas: 2
selector:
role: glusterfs-web-frontend
template:
metadata:
labels:
role: glusterfs-web-frontend
spec:
containers:
- name: glusterfsweb
image: nginx
imagePullPolicy: IfNotPresent
ports:
- name: glusterfsweb
containerPort: 80
volumeMounts:
# name must match the volume name below
- name: gluster-default-volume
mountPath: "/usr/share/nginx/html"
volumes:
- name: gluster-default-volume
persistentVolumeClaim:
claimName: glusterfs-claim
[root@k8s-master glusterfs]# cat glusterfs-web-service.yaml
kind: Service
apiVersion: v1
metadata:
name: glusterfs-web
spec:
ports:
- port: 80
selector:
role: glusterfs-web-frontend
验证
[root@k8s-master glusterfs]# kubectl get pods -o wide |grep glusterfs-web
glusterfs-web-280mz 1/1 Running 0 1m 10.1.39.12 k8s-node1
glusterfs-web-f952d 1/1 Running 0 1m 10.1.15.10 k8s-node2
[root@k8s-master glusterfs]# kubectl exec -ti glusterfs-web-280mz -- bash
root@glusterfs-web-280mz:/# df -h |grep glusterfs
172.17.3.7:glusterfsdata 422G 954M 421G 1% /usr/share/nginx/html
root@glusterfs-web-280mz:/# cd /usr/share/nginx/html/
root@glusterfs-web-280mz:/usr/share/nginx/html# ls
glusterfs
root@glusterfs-web-280mz:/usr/share/nginx/html# cat glusterfs
glusterfs
本文出自 “jerrymin” 博客，请务必保留此出处http://jerrymin.blog.51cto.com/3002256/1907274
k8s1.5.4挂载volume之glusterfs