xiaoming728

xiaoming728

请求elasticsearch报403forbidden排查过程

请求elasticsearch报403forbidden排查过程

1、es报错:

{"type": "server", "timestamp": "2021-04-14T02:18:39,905Z", "level": "WARN", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "docker-cluster", "node.name": "c9721b7cf6e9", "message": "flood stage disk watermark [95%] exceeded on [L5lpjyRxQY6lIFn8F0g0xA][c9721b7cf6e9][/usr/share/elasticsearch/data/nodes/0] free: 12.9gb[4.3%], all indices on this node will be marked read-only", "cluster.uuid": "Me3hE70hSYSBLY9mdi3PHw", "node.id": "L5lpjyRxQY6lIFn8F0g0xA"  }

显示磁盘空间占用95以上,es开启了只读保护。

原因参考: https://zhuanlan.zhihu.com/p/181671838

es修改:

#es属性查看
GET _settings?pretty
#手动关闭只读保护
PUT _settings
{
  "index": {
    "blocks": {
      "read_only_allow_delete": "false"
    }
  }
}

2、去服务器查看磁盘占用情况

使用df -h查看磁盘空间
linux查询文件占用硬盘大小:
du -hs ./*

3、发现是logstash容器日志打的日志太多了

解决方法:config配置里删除output中的stdout{}

4、清理Docker容器日志

Docker容器日志查看与配置

https://www.yuque.com/ygkdsh/newbie/hwo8vi