加载中…
个人资料
微博
  • 博客等级：
  • 博客积分：
  • 博客访问：
  • 关注人气：
  • 获赠金笔：0支
  • 赠出金笔：0支
  • 荣誉徽章：
正文 字体大小：大 中 小

kubectl get nodes NotReady

(2017-08-09 15:23:36)
分类： linux


检查节点情况竟然notready

[root@compile ~]# kubectl get nodes

NAME      STATUS     AGE       VERSION

compile   NotReady   1h        v1.7.3

rtdock    NotReady   47m       v1.7.3

查看日志：

[root@compile ~]# kubectl describe node compile

Name:                   compile

Role:

Labels:                 beta.kubernetes.io/arch=ppc64le

                        beta.kubernetes.io/os=linux

                        kubernetes.io/hostname=compile

                        node-role.kubernetes.io/master=

Annotations:            node.alpha.kubernetes.io/ttl=0

                        volumes.kubernetes.io/controller-managed-attach-detach=true

Taints:                 node-role.kubernetes.io/master:NoSchedule

CreationTimestamp:      Wed, 09 Aug 2017 13:56:02 +0800

Conditions:

  Type                  Status  LastHeartbeatTime                       LastTransitionTime                      Reason                          Message

  ----                  ------  -----------------                       ------------------                      ------                          -------

  OutOfDisk             False   Wed, 09 Aug 2017 15:14:24 +0800         Wed, 09 Aug 2017 13:55:58 +0800         KubeletHasSufficientDisk        kubelet has sufficient disk space available

  MemoryPressure        False   Wed, 09 Aug 2017 15:14:24 +0800         Wed, 09 Aug 2017 13:55:58 +0800         KubeletHasSufficientMemory      kubelet has sufficient memory available

  DiskPressure          False   Wed, 09 Aug 2017 15:14:24 +0800         Wed, 09 Aug 2017 13:55:58 +0800         KubeletHasNoDiskPressure        kubelet has no disk pressure

  Ready                 False   Wed, 09 Aug 2017 15:14:24 +0800         Wed, 09 Aug 2017 13:55:58 +0800         KubeletNotReady                 runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Addresses:

  InternalIP:   192.168.207.32

  Hostname:     compile

Capacity:

 cpu:           128

 memory:        129693760Ki

 pods:          110

Allocatable:

 cpu:           128

 memory:        129591360Ki

 pods:          110

System Info:

 Machine ID:                    34954115202248b4b3067e2fe008cbfe

 System UUID:                   780B14A        

 Boot ID:                       dbb1443a-87cb-4d7a-b086-5b20a68a3699

 Kernel Version:                3.10.0-514.26.2.el7.ppc64le

 OS Image:                      CentOS Linux 7 (AltArch)

 Operating System:              linux

 Architecture:                  ppc64le

 Container Runtime Version:     docker://Unknown

 Kubelet Version:               v1.7.3

 Kube-Proxy Version:            v1.7.3

PodCIDR:                        10.244.0.0/24

ExternalID:                     compile

Non-terminated Pods:            (5 in total)

  Namespace                     Name                                    CPU Requests    CPU Limits      Memory Requests Memory Limits

  ---------                     ----                                    ------------    ----------      --------------- -------------

  kube-system                   etcd-compile                            0 (0%)          0 (0%)          0 (0%)          0 (0%)

  kube-system                   kube-apiserver-compile                  250m (0%)       0 (0%)          0 (0%)          0 (0%)

  kube-system                   kube-controller-manager-compile         200m (0%)       0 (0%)          0 (0%)          0 (0%)

  kube-system                   kube-proxy-lz6dj                        0 (0%)          0 (0%)          0 (0%)          0 (0%)

  kube-system                   kube-scheduler-compile                  100m (0%)       0 (0%)          0 (0%)          0 (0%)

Allocated resources:

  (Total limits may be over 100 percent, i.e., overcommitted.)

  CPU Requests  CPU Limits      Memory Requests Memory Limits

  ------------  ----------      --------------- -------------

  550m (0%)     0 (0%)          0 (0%)          0 (0%)

Events:

  FirstSeen     LastSeen        Count   From                    SubObjectPath   Type            Reason                  Message

  ---------     --------        -----   ----                    -------------   --------        ------                  -------

  1h            1h              1       kubelet, compile                        Normal          NodeAllocatableEnforced Updated Node Allocatable limit across pods

  1h            1h              42      kubelet, compile                        Normal          NodeHasSufficientDisk   Node compile status is now: NodeHasSufficientDisk

  1h            1h              42      kubelet, compile                        Normal          NodeHasSufficientMemory Node compile status is now: NodeHasSufficientMemory

  1h            1h              42      kubelet, compile                        Normal          NodeHasNoDiskPressure   Node compile status is now: NodeHasNoDiskPressure

  1h            1h              1       kube-proxy, compile                     Normal          Starting                Starting kube-proxy.

  17m           17m             1       kubelet, compile                        Normal          Starting                Starting kubelet.

  17m           17m             1       kubelet, compile                        Normal          NodeAllocatableEnforced Updated Node Allocatable limit across pods

  17m           17m             27      kubelet, compile                        Normal          NodeHasSufficientDisk   Node compile status is now: NodeHasSufficientDisk

  17m           17m             27      kubelet, compile                        Normal          NodeHasSufficientMemory Node compile status is now: NodeHasSufficientMemory

  17m           17m             27      kubelet, compile                        Normal          NodeHasNoDiskPressure   Node compile status is now: NodeHasNoDiskPressure

  17m           17m             1       kubelet, compile                        Warning         Rebooted                Node compile has been rebooted, boot id: dbb1443a-87cb-4d7a-b086-5b20a68a3699

  17m           17m             1       kube-proxy, compile                     Normal          Starting                Starting kube-proxy.

 

 

[root@compile ~]# kubectl get pods --all-namespaces -o wide

NAMESPACE     NAME                              READY     STATUS    RESTARTS   AGE       IP               NODE

kube-system   etcd-compile                      1/1       Running   1          1h        192.168.207.32   compile

kube-system   kube-apiserver-compile            1/1       Running   1          1h        192.168.207.32   compile

kube-system   kube-controller-manager-compile   1/1       Running   1          1h        192.168.207.32   compile

kube-system   kube-dns-3424415278-qc1wj         0/3       Pending   0          1h                  

kube-system   kube-proxy-lz6dj                  1/1       Running   1          1h        192.168.207.32   compile

kube-system   kube-proxy-z6w6w                  1/1       Running   1          44m       192.168.207.33   rtdock

kube-system   kube-scheduler-compile            1/1       Running   1          1h        192.168.207.32   compile

发现kube-dns-3424415278-qc1wj一直处于pending状态。

 

处理方法：

1.检查防火墙是否关闭：

systemctl status firewalld.service

 systemctl stop firewalld.service

 systemctl disable firewalld.service

2.编辑vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

删掉KUBELET_NETWORK_ARGS这行

Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"

然后重启服务

[root@compile ~]# systemctl daemon-reload

[root@compile ~]# systemctl restart kubelet

[root@compile ~]# systemctl status kubelet

[root@compile ~]# kubectl get nodes

NAME      STATUS    AGE       VERSION

compile   Ready     1h        v1.7.3

rtdock    Ready     54m       v1.7.3

[root@compile ~]# kubectl get pods --all-namespaces -o wide

 

如下为操作日志记录：

======================================

[root@compile ~]# kubectl get nodes

NAME      STATUS     AGE       VERSION

compile   NotReady   1h        v1.7.3

rtdock    NotReady   47m       v1.7.3

 [root@compile ~]# vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

删掉KUBELET_NETWORK_ARGS这行

Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"

[Service]

Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"

Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"

Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"

Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"

Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"

Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"

ExecStart=

ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_EXTRA_ARGS

~

~

~

~

~

"/etc/systemd/system/kubelet.service.d/10-kubeadm.conf" 9L, 744C written

[root@compile ~]# systemctl daemon-reload

[root@compile ~]# systemctl restart kubelet

[root@compile ~]# systemctl status kubelet

* kubelet.service - kubelet: The Kubernetes Node Agent

   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)

  Drop-In: /etc/systemd/system/kubelet.service.d

           `-10-kubeadm.conf

   Active: active (running) since Wed 2017-08-09 15:21:22 CST; 4s ago

     Docs: http://kubernetes.io/docs/

 Main PID: 8053 (kubelet)

   Memory: 83.9M

   CGroup: /system.slice/kubelet.service

           |-8053 /usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --pod-manifest-path=/etc/kubernetes/manifests --allow-privile...

           `-8150 journalctl -k -f

 

Aug 09 15:21:23 compile kubelet[8053]: E0809 15:21:23.116484    8053 kubelet.go:1737] Failed to check if disk space is available on the root partition: fa...ontainer /

Aug 09 15:21:23 compile kubelet[8053]: I0809 15:21:23.116648    8053 kubelet_node_status.go:82] Attempting to register node compile

Aug 09 15:21:23 compile kubelet[8053]: I0809 15:21:23.140815    8053 kubelet_node_status.go:133] Node compile was previously registered

Aug 09 15:21:23 compile kubelet[8053]: I0809 15:21:23.140832    8053 kubelet_node_status.go:85] Successfully registered node compile

Aug 09 15:21:23 compile kubelet[8053]: I0809 15:21:23.143714    8053 kuberuntime_manager.go:901] updating runtime config through cri with podcidr 10.244.0.0/24

Aug 09 15:21:23 compile kubelet[8053]: I0809 15:21:23.143963    8053 docker_service.go:308] docker cri received runtime config &RuntimeConfig{NetworkConfi...0.0/24,},}

Aug 09 15:21:23 compile kubelet[8053]: I0809 15:21:23.144207    8053 kubelet_network.go:326] Setting Pod CIDR:  -> 10.244.0.0/24

Aug 09 15:21:23 compile kubelet[8053]: I0809 15:21:23.288668    8053 manager.go:293] Recovery completed

Aug 09 15:21:23 compile kubelet[8053]: E0809 15:21:23.509918    8053 helpers.go:771] Could not find capacity information for resource storage.kubernetes.io/scratch

Aug 09 15:21:23 compile kubelet[8053]: W0809 15:21:23.509968    8053 helpers.go:782] eviction manager: no observation found for eviction signal allocatabl....available

Hint: Some lines were ellipsized, use -l to show in full.

[root@compile ~]# kubectl get nodes

NAME      STATUS    AGE       VERSION

compile   Ready     1h        v1.7.3

rtdock    Ready     54m       v1.7.3

[root@compile ~]#

[root@compile ~]# kubectl get pods --all-namespaces -o wide

NAMESPACE     NAME                              READY     STATUS    RESTARTS   AGE       IP               NODE

kube-system   etcd-compile                      1/1       Running   1          1h        192.168.207.32   compile

kube-system   kube-apiserver-compile            1/1       Running   1          1h        192.168.207.32   compile

kube-system   kube-controller-manager-compile   1/1       Running   1          1h        192.168.207.32   compile

kube-system   kube-dns-3424415278-qc1wj         3/3       Running   0          1h        172.17.0.2       rtdock

kube-system   kube-proxy-lz6dj                  1/1       Running   1          1h        192.168.207.32   compile

kube-system   kube-proxy-z6w6w                  1/1       Running   1          58m       192.168.207.33   rtdock

kube-system   kube-scheduler-compile            1/1       Running   1          1h        192.168.207.32   compile

 

喜欢

0

阅读┊ 收藏 ┊ 喜欢▼ ┊打印┊举报/Report
  

新浪BLOG意见反馈留言板　欢迎批评指正

新浪简介 | About Sina | 广告服务 | 联系我们 | 招聘信息 | 网站律师 | SINA English | 产品答疑

Copyright © 1996 - 2022 SINA Corporation, All Rights Reserved

新浪公司 版权所有