作者：上犹日期：2019-08-18 06:51:19

返回目录：文件问题

在这之前呢,我不得不吐槽一下网上写的关于ELK stack集群的部署文章,真是垃圾,首先是文章标题跟内容不符,二是文章写的前言不搭后语(我这个人脾气平时很好的呦!)

好了继之前的ELK 我们结合官方文档继续谈谈stack集群高可用部署篇

下面是我的主机IP组件列表:

192.168.111.3 elasticsearch-node1

192.168.111.4 elasticsearch-node2

192.168.111.5 elasticsearch-node3

192.168.111.6 elasticsearch-node0，kibana，logstash

192.168.111.16 nginx，filebeat

简述：

其中node1，node2，node3作为数据节点，而node0则作为集群当中与kibana连接的节点。
nginx主机通过filebeat将日志转给logstash，logstash发给es集群，然后再由kibana展示

1，安装配置。

1，安装依赖。

yum -y install lrzsz vim curl wget java

2，配置yum源。

cat > /etc/yum.repos.d/elk.repo << EOF
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

3，安装。

在四台主机安装elasticsearch。

yum -y install elasticsearch

接下来进行配置：

elk-node1

[root@localhost ~]$egrep -v "^#|^$" /etc/elasticsearch/elasticsearch.yml
cluster.name: ebei-elk
node.name: elk-node1
path.data: /logs/elasticsearch6
path.logs: /logs/elasticsearch6/log
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.111.3:9300","192.168.111.4:9300","192.168.111.5:9300","192.168.111.6:9300"]
discovery.zen.minimum_master_nodes: 2
xpack.security.enabled: false

说明：

cluster.name：自定义集群名称
node.name：自定义节点名，不同主机名称不能一致。
path.data：data存储路径，这里更改成自定义以应对日志的big。
path.logs：log存储路径，是为es自己的日志。
注意创建上边两项定义的两个文件目录。否则会启动失败。

mkdir -p /logs/elasticsearch6/log
cd /logs
chown -R elasticsearch.elasticsearch elasticsearch6/

network.host：es监听地址，采用"0.0.0.0"，表示允许所有设备访问。
http.port：es监听端口，可不取消注释，默认即此端口。
discovery.zen.ping.unicast.hosts：集群节点发现列表
discovery.zen.minimum_master_nodes：定义可成为master的节点数量。为避免脑裂的情况，建议设置成2。
xpack.security.enabled：添加这条，这条是配置kibana的安全机制，暂时关闭。

4，分发配置。

接着将配置分发给其他几台主机。

[root@localhost ~]$scp /etc/elasticsearch/elasticsearch.yml 192.168.111.4:/etc/elasticsearch/elasticsearch.yml
[root@localhost ~]$scp /etc/elasticsearch/elasticsearch.yml 192.168.111.5:/etc/elasticsearch/elasticsearch.yml
[root@localhost ~]$scp /etc/elasticsearch/elasticsearch.yml 192.168.111.6:/etc/elasticsearch/elasticsearch.yml

然后更改对应三台的配置，只需更改node-name即可。

列出另外三台的配置：

elk-node2

[root@localhost ~]$egrep -v "^#|^$" /etc/elasticsearch/elasticsearch.yml
cluster.name: ebei-elk
node.name: elk-node2
path.data: /logs/elasticsearch6
path.logs: /logs/elasticsearch6/log
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.111.3:9300","192.168.111.4:9300","192.168.111.5:9300","192.168.111.6:9300"]
discovery.zen.minimum_master_nodes: 2
xpack.security.enabled: false

elk-node3

[root@localhost ~]$egrep -v "^#|^$" /etc/elasticsearch/elasticsearch.yml
cluster.name: ebei-elk
node.name: elk-node3
path.data: /logs/elasticsearch6
path.logs: /logs/elasticsearch6/log
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.111.3:9300","192.168.111.4:9300","192.168.111.5:9300","192.168.111.6:9300"]
discovery.zen.minimum_master_nodes: 2
xpack.security.enabled: false

elk-node0

[root@localhost ~]$egrep -v "^#|^$" /etc/elasticsearch/elasticsearch.yml
cluster.name: ebei-elk
node.name: elk-node0
path.data: /logs/elasticsearch6
path.logs: /logs/elasticsearch6/log
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.111.3:9300","192.168.111.4:9300","192.168.111.5:9300","192.168.111.6:9300"]
discovery.zen.minimum_master_nodes: 2
xpack.security.enabled: false
node.master: false
node.data: false
node.ingest: false

注意最后多了三条配置。

一开始我也并没有想到这一点，而且也并没有node0这一节点的，但当我开始配置kibana的配置时发现，kibana只能写一条es的连接信息，于是乎，才发现有这样一个问题，后来了解到，官方给了一种解决办法[www.elastic.co/guide/en/kibana/6.4/production.html]。

下面是官方英文:

Install Elasticsearch on the same machine as Kibana.
Configure the node as a Coordinating only node. In elasticsearch.yml, set node.data, node.master and node.ingest to false:

# 3. You want this node to be neither master nor data node nor ingest node, but
# to act as a "search load balancer" (fetching data from nodes,
# aggregating results, etc.)
#
node.master: false
node.data: false
node.ingest: false

Configure the client node to join your Elasticsearch cluster. In elasticsearch.yml, set the cluster.name to the name of your cluster.

cluster.name: "my_cluster"

Check your transport and HTTP host configs in elasticsearch.yml under network.host and transport.host. The transport.host needs to be on the network reachable to the cluster members, the network.host is the network for the HTTP connection for Kibana (localhost:9200 by default).

network.host: localhost
http.port: 9200
# by default transport.host refers to network.host
transport.host: <external ip>
transport.tcp.port: 9300 - 9400

Make sure Kibana is configured to point to your local client node. In kibana.yml, the elasticsearch.url should be set to localhost:9200.

# The Elasticsearch instance to use for all your queries.
elasticsearch.url: "http://localhost:9200"

方式就是添加如上三条配置，然后将这一节点变成与kibana对接的桥梁，但是并不参与数据方面的处理。

最后就是启动几台主机上的es服务。

systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl start elasticsearch.service
systemctl status elasticsearch.service

2，查看集群信息。

查看es是否正常启动。

[root@localhost ~]$curl -X GET "127.0.0.1:9200/"
{
 "name" : "elk-node1",
 "cluster_name" : "ebei-elk",
 "cluster_uuid" : "53LLexx8RSW16nE4lsJMQQ",
 "version" : {
 "number" : "6.5.3",
 "build_flavor" : "default",
 "build_type" : "rpm",
 "build_hash" : "159a78a",
 "build_date" : "2018-12-06T20:11:28.826501Z",
 "build_snapshot" : false,
 "lucene_version" : "7.5.0",
 "minimum_wire_compatibility_version" : "5.6.0",
 "minimum_index_compatibility_version" : "5.0.0"
 },
 "tagline" : "You Know, for Search"
}

另一种方式查看。

[root@localhost ~]$curl -XGET 'http://127.0.0.1:9200/_cluster/health?pretty'
{
 "cluster_name" : "ebei-elk",
 "status" : "green",
 "timed_out" : false,
 "number_of_nodes" : 4,
 "number_of_data_nodes" : 3,
 "active_primary_shards" : 11,
 "active_shards" : 22,
 "relocating_shards" : 0,
 "initializing_shards" : 0,
 "unassigned_shards" : 0,
 "delayed_unassigned_shards" : 0,
 "number_of_pending_tasks" : 0,
 "number_of_in_flight_fetch" : 0,
 "task_max_waiting_in_queue_millis" : 0,
 "active_shards_percent_as_number" : 100.0
}

查看集群状态。

[root@localhost ~]$curl -XGET 'http://127.0.0.1:9200/_cat/nodes'
192.168.111.3 48 39 0 0.22 0.07 0.06 mdi * elk-node1 #带*号的表示master
192.168.111.6 41 98 0 0.03 0.06 0.05 - - elk-node0 #注意这个节点并不参与数据处理。
192.168.111.4 44 57 0 0.00 0.01 0.05 mdi - elk-node2
192.168.111.5 27 62 0 0.00 0.01 0.05 mdi - elk-node3

另一种方式查看。

[root@localhost ~]$curl -XGET 'http://127.0.0.1:9200/_cat/nodes?v'
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.111.3 34 39 0 0.13 0.10 0.07 mdi * elk-node1
192.168.111.6 20 98 0 0.01 0.05 0.05 - - elk-node0
192.168.111.4 32 57 0 0.00 0.01 0.05 mdi - elk-node2
192.168.111.5 38 62 0 0.00 0.01 0.05 mdi - elk-node3

查看集群详细信息。

[root@localhost ~]$curl -XGET 'http://127.0.0.1:9200/_cluster/state/nodes?pretty'
{
 "cluster_name" : "ebei-elk", #集群名称
 "compressed_size_in_bytes" : 14291,
 "cluster_uuid" : "53LLexx8RSW16nE4lsJMQQ", #集群id
 "nodes" : {
 "oCNARqdUT5KXLZTlcS22GA" : { #node的ID值
 "name" : "elk-node1", #node名称
 "ephemeral_id" : "y-NCFJULTEmWdWjNjCPS2A",
 "transport_address" : "192.168.111.3:9300", #集群通讯地址
 "attributes" : {
 "ml.machine_memory" : "8202727424",
 "xpack.installed" : "true",
 "ml.max_open_jobs" : "20",
 "ml.enabled" : "true"
 }
 },
 "8F_rZuR1TByEb6bXz0EgzA" : {
 "name" : "elk-node0",
 "ephemeral_id" : "b3CtPKpyRUahT4njpRqjlQ",
 "transport_address" : "192.168.111.6:9300",
 "attributes" : {
 "ml.machine_memory" : "8202039296",
 "ml.max_open_jobs" : "20",
 "xpack.installed" : "true",
 "ml.enabled" : "true"
 }
 },
 "ptEOHzaPTgmlqW3NRhd7SQ" : {
 "name" : "elk-node2",
 "ephemeral_id" : "YgypZZNcTfWcIYDhOlUAzw",
 "transport_address" : "192.168.111.4:9300",
 "attributes" : {
 "ml.machine_memory" : "8202039296",
 "ml.max_open_jobs" : "20",
 "xpack.installed" : "true",
 "ml.enabled" : "true"
 }
 },
 "PJYNTw5nTeS2kjjft4UcrA" : {
 "name" : "elk-node3",
 "ephemeral_id" : "C2nnREt0TOGKeDw9OQRJUA",
 "transport_address" : "192.168.111.5:9300",
 "attributes" : {
 "ml.machine_memory" : "8202031104",
 "ml.max_open_jobs" : "20",
 "xpack.installed" : "true",
 "ml.enabled" : "true"
 }
 }
 }
}

查询master。

[root@localhost ~]$curl -XGET 'http://127.0.0.1:9200/_cluster/state/master_node?pretty'
{
 "cluster_name" : "ebei-elk",
 "compressed_size_in_bytes" : 14291,
 "cluster_uuid" : "53LLexx8RSW16nE4lsJMQQ",
 "master_node" : "oCNARqdUT5KXLZTlcS22GA"
}
[root@localhost ~]$curl -XGET 'http://127.0.0.1:9200/_cat/master?v'
id host ip node
oCNARqdUT5KXLZTlcS22GA 192.168.111.3 192.168.111.3 elk-node1

查询集群健康状态。

[root@localhost ~]$curl -XGET 'http://127.0.0.1:9200/_cat/health?v'
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1545386516 10:01:56 ebei-elk green 4 3 22 11 0 0 0 0 - 100.0%

3，高可用验证。

通过刚刚的测试，已经看出，master在node1，现在将node1的es停掉，看看是否会自动漂移。

[root@localhost ~]$curl -XGET 'http://127.0.0.1:9200/_cat/nodes?v'
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.111.3 36 39 0 0.03 0.04 0.05 mdi * elk-node1
192.168.111.6 22 98 0 0.00 0.02 0.05 - - elk-node0
192.168.111.4 38 57 0 0.00 0.01 0.05 mdi - elk-node2
192.168.111.5 44 62 0 0.00 0.01 0.05 mdi - elk-node3
[root@localhost ~]$systemctl stop elasticsearch

然后到另外一个节点查看一下：

[root@localhost ~]$curl -XGET 'http://127.0.0.1:9200/_cat/nodes?v'
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.111.4 45 58 0 0.00 0.01 0.05 mdi - elk-node2
192.168.111.5 30 62 0 0.00 0.01 0.05 mdi * elk-node3
192.168.111.6 24 98 0 0.00 0.02 0.05 - - elk-node0

可以看到master已经跑到了node3。

4，对接配置。

filebeat往logstash转发日志的配置还是原来的样子，只不过logstash的配置文件要更改一下output了，如下：

input {
 beats {
 host => 192.168.111.16
 port => 5044
 }
}
filter {
 if [fields][logtype] == "nginx-access" {
 json {
 source => "message"
 target => "data"
 }
 }
 if [fields][logtype] == "nginx-error" {
 json {
 source => "message"
 target => "data"
 }
 }
}
output {
 if [fields][logtype] == "nginx-access" {
 elasticsearch {
 hosts => ["192.168.111.3:9200","192.168.111.4:9200","192.168.111.5:9200"]
 index => "logstash-nginx-access-%{+YYYY.MM.dd}"
 }
 }
 if [fields][logtype] == "nginx-error" {
 elasticsearch {
 hosts => ["192.168.111.3:9200","192.168.111.4:9200","192.168.111.5:9200"]
 index => "logstash-nginx-error-%{+YYYY.MM.dd}"
 }
 }
}

再看kibana的配置。

[root@localhost logs]$egrep -v "^$|^#" /etc/kibana/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://127.0.0.1:9200"
kibana.index: ".kibana"
xpack.security.enabled: false

然后访问即可，可以看到日志能够源源不断的打进来，如果es当中某个节点挂了，并不会影响kibana当中日志的展示，但是，此时可以看到与kibana对接的es还是单个，所以这种还不能算是绝对高可用。

5，高可用。

所以正确的做法应该是，撤掉刚刚的node0，然后将集群当中的三台es通过前端nginx做一个代理，然后让kibana连接nginx配置的地址即可实现高可用。

现在就在192.168.111.16的nginx添加配置：

upstream elasticsearch {
 zone elasticsearch 64K;
 server 192.168.111.3:9200;
 server 192.168.111.4:9200;
 server 192.168.111.5:9200;
}
server {
 listen 9200;
 server_name 192.168.111.16;
 location / {
 proxy_pass http://elasticsearch;
 proxy_redirect off;
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 }
 access_log logs/es_access.log;
}

然后加载配置。

nginx -t
nginx -s reload

接着更改kibana当中的连接地址：

[root@localhost logs]$!egr
egrep -v "^$|^#" /etc/kibana/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.111.16:9200"
kibana.index: ".kibana"
xpack.security.enabled: false

这样以来，数据还都正常流通，即便nginx后端的某个es挂掉，还不会影响整个链路的正常。当然目前nginx是单节点，其实问题不大，如果量实在非常大，可以在nginx前端再添加高可用组件即可

本文标签：ElasticSearch(10)EOF(9)root @ localhost ~(7)ELK(3)hosts文件修改(25)elasticsearch.yml(1)

ELK stack集群部署之-初级篇-hosts文件修改

返回目录：文件问题

1，安装配置。

2，查看集群信息。

3，高可用验证。

4，对接配置。

5，高可用。

相关阅读

Linux ACL权限设置（setfacl和getfacl）-权限设置

分布式系统中日志如何查看？难道一个一个机器查看？ELK了解一下-电脑日志怎么看

Linux系统之firewalld防火墙基础配置-防火墙设置在哪里

设置小精灵无人区！法国要求Pokemon GO滚出校园-比特精灵设置

黑法术高端无线路由器劫持漏洞攻击技术原理（原创）-贝尔金路由器设置

贝尔金解释路由器无法联网的原因-贝尔金路由器设置

不知道选哪款隔离？那你就得戳进来看看！-360隔离的文件在哪

热门文章

移动电竞职业化探访：保持盈利的皇室战争STK战队-stk文件

用Java实现对电脑文件的压缩和解压（上）（附图解和源码注解）-文件解压

可以对征地拆迁红头文件说“不”吗？“你有权提意见！”-什么是红头文件

C语言文件操作之——文件的读写（详解，附学习资料）-文件名长度

如何在Linux中轻松删除源安装的软件包？-linux删除文件夹

如何给文件批量命名-文件批量更名

聚合标签

电脑蓝屏_电脑怎么了_win7问题_win10问题_设置问题_文件问题_上犹电脑信息网