人人IT網

人人IT網

當前位置: 主頁 > JAVA編程 > ANT >

ELK日志分析平台搭建全過程

時間:2016-12-03 01:04來源:Internet 作者:Internet 點擊:
ELK日志分析平台搭建全過程   <iframe id="iframe_0.21720791777080461" style="margin: 0px; padding: 0px; bor

ELK日志分析平台搭建全過程

 
<iframe id="iframe_0.21720791777080461" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 686px; height: 350px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22http://static.oschina.net/uploads/space/2016/1110/114329_AWy9_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.21720791777080461',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

一、使用背景

    當生產環境有很多服務器、很多業務模塊的日志需要每時每刻查看時

二、環境

系統:centos 6.5

JDK:1.8

Elasticsearch-5.0.0

Logstash-5.0.0

kibana-5.0.0

三、安裝

1、安裝JDK

下載JDK:http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

本環境下載的是64位tar.gz包,將安裝包拷貝至安裝服務器/usr/local目錄

[root@localhost ~]# cd /usr/local/ 
[root@localhost local]# tar -xzvf jdk-8u111-linux-x64.tar.gz

配置環境變量

[root@localhost local]# vim /etc/profile

將下面的內容添加至文件末尾(假如服務器需要多個JDK版本,为了ELK不影響其它系統,也可以將環境變量的內容稍後添加到ELK的启動腳本中)

JAVA_HOME=/usr/local/jdk1.8.0_111
JRE_HOME=/usr/local/jdk1.8.0_111/jre
CLASSPATH=.:$JAVA_HOME/lib:/dt.jar:$JAVA_HOME/lib/tools.jar
PATH=$PATH:$JAVA_HOME/bin
export  JAVA_HOME
export  JRE_HOME

ulimit -u 4096

[root@localhost local]# source /etc/profile

配置limit相關参數

[root@localhost local]# vim /etc/security/limits.conf
添加以下內容

* soft nproc 65536
* hard nproc 65536
* soft nofile 65536
* hard nofile 65536

創建運行ELK的用戶

[root@localhost local]# groupadd elk

[root@localhost local]# useradd -g elk elk

<iframe id="iframe_0.5598784179658625" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 367px; height: 135px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22http://static.oschina.net/uploads/space/2016/1110/105046_pQqO_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.5598784179658625',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

創建ELK運行目錄

[root@localhost local]# mkdir /elk
[root@localhost local]# chown -R elk:elk /elk

關閉防火牆:

[root@localhost ~]# iptables -F

以上全部是root用戶完成

2、安裝ELK

以下由elk用戶操作

以elk用戶登錄服務器

下載ELK安裝包:https://www.elastic.co/downloads,並上傳到服務器且解壓,解壓命令:tar -xzvf 包名

<iframe id="iframe_0.0815892494236754" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 599px; height: 186px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22https://static.oschina.net/uploads/space/2016/1110/105703_Cv7F_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.0815892494236754',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

配置Elasticsearch

<iframe id="iframe_0.10374600164018122" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 462px; height: 72px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22https://static.oschina.net/uploads/space/2016/1110/105827_8GZs_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.10374600164018122',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

修改如下內容:

<iframe id="iframe_0.07410717744910933" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 227px; height: 126px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22https://static.oschina.net/uploads/space/2016/1110/105906_zUJ2_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.07410717744910933',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

<iframe id="iframe_0.33557386976636194" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 259px; height: 96px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22http://static.oschina.net/uploads/space/2016/1110/105923_AuR5_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.33557386976636194',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

保存退出

启動Elasticsearch

<iframe id="iframe_0.40192665314822595" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 570px; height: 24px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22https://static.oschina.net/uploads/space/2016/1110/110110_ZgnH_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.40192665314822595',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

查看是否启動成功

<iframe id="iframe_0.9350422771105631" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 629px; height: 519px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22http://static.oschina.net/uploads/space/2016/1110/110158_he0l_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.9350422771105631',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

用瀏覽器訪問:http://192.168.10.169:9200

<iframe id="iframe_0.9338390194613313" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 602px; height: 284px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22https://static.oschina.net/uploads/space/2016/1110/110253_sJnh_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.9338390194613313',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

Elasticsearch安裝完畢

安裝logstash

logstash是ELK中負責收集和過滤日志的

<iframe id="iframe_0.5476563389939548" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 410px; height: 53px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22http://static.oschina.net/uploads/space/2016/1110/110555_5ckN_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.5476563389939548',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

編寫配置文件如下:

<iframe id="iframe_0.3824549073131953" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 686px; height: 458px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22http://static.oschina.net/uploads/space/2016/1110/110650_4SLy_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.3824549073131953',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

解釋:

logstash的配置文件須包含三個內容:

input{}:此模塊是負責收集日志,可以從文件讀取、從redis讀取或者開启端口讓產生日志的業務系統直接寫入到logstash

filter{}:此模塊是負責過滤收集到的日志,並根據過滤後對日志定義顯示字段

output{}:此模塊是負責將過滤後的日志輸出到elasticsearch或者文件、redis等

本環境采用從文件讀取日志,業務系統產生日志的格式如下:

[2016-11-05 00:00:03,731  INFO] [http-nio-8094-exec-10] [filter.LogRequestFilter] - /merchant/get-supply-detail.shtml, IP: 121.35.185.117, [device-dpi = 414*736, version = 3.6, device-os = iOS8.4.1, timestamp = 1478275204, bundle = APYQ9WATKK98V2EC, device-network = WiFi, token = 393E38694471483CB3686EC77BABB496, device-model = iPhone, device-cpu = , sequence = 1478275204980, device-uuid = C52FF568-A447-4AFE-8AE8-4C9A54CED10C, sign = 0966a15c090fa6725d8e3a14e9ef98dc, request = {
  "supply-id" : 192
}]
[2016-11-05 00:00:03,731 DEBUG] [http-nio-8094-exec-10] [filter.ValidateRequestFilter] - Unsigned: bundle=APYQ9WATKK98V2EC&device-cpu=&device-dpi=414*736&device-model=iPhone&device-network=WiFi&device-os=iOS8.4.1&device-uuid=C52FF568-A447-4AFE-8AE8-4C9A54CED10C&request={
  "supply-id" : 192

output直接輸出到Elasticsearch

本環境需處理兩套業務系統的日志

<iframe id="iframe_0.943884629889175" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 472px; height: 354px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22http://static.oschina.net/uploads/space/2016/1110/111658_5Uuu_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.943884629889175',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

type:代表類型,其實就是將這個類型推送到Elasticsearch,方便後面的kibana進行分類搜索,一般直接命名業務系統的項目名

path:讀取文件的路徑

<iframe id="iframe_0.6482273387038273" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 224px; height: 85px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22http://static.oschina.net/uploads/space/2016/1110/111953_4YUe_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.6482273387038273',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

這個是代表日志報錯時,將報錯的換行歸屬於上一條message內容

start_position => "beginning"是代表從文件頭部開始讀取

<iframe id="iframe_0.051983671744315174" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 686px; height: 112px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22http://static.oschina.net/uploads/space/2016/1110/112220_hFBY_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.051983671744315174',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

filter{}中的grok是采用正則表達式來過滤日志,其中%{TIMESTAMP_ISO8601}代表一個內置獲取2016-11-05 00:00:03,731時間的正則表達式的函數,%{TIMESTAMP_ISO8601:date1}代表將獲取的值賦给date1,在kibana中可以體現出來

本環境有兩條grok是代表,第一條不符合將執行第二條

<iframe id="iframe_0.2657880905321013" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 391px; height: 149px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22http://static.oschina.net/uploads/space/2016/1110/112742_n814_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.2657880905321013',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

其中index是定義將過滤後的日志推送到Elasticsearch後存儲的名字

%{type}是調用input中的type變量(函數)

启動logstash

<iframe id="iframe_0.640560640535079" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 686px; height: 16px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22https://static.oschina.net/uploads/space/2016/1110/113125_RDU9_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.640560640535079',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

<iframe id="iframe_0.7989725293287735" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 686px; height: 94px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22http://static.oschina.net/uploads/space/2016/1110/113232_XDky_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.7989725293287735',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

代表启動成功

安裝kibana

<iframe id="iframe_0.7789425109757211" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 474px; height: 58px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22http://static.oschina.net/uploads/space/2016/1110/113318_OzBV_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.7789425109757211',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

<iframe id="iframe_0.653312015653114" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 686px; height: 295px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22https://static.oschina.net/uploads/space/2016/1110/113341_w95f_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.653312015653114',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

保存退出

启動kibana

<iframe id="iframe_0.49168528510009146" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 686px; height: 107px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22http://static.oschina.net/uploads/space/2016/1110/113457_At8S_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.49168528510009146',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

<iframe id="iframe_0.14360340804598515" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 686px; height: 352px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22http://static.oschina.net/uploads/space/2016/1110/113614_4JLN_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.14360340804598515',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

<iframe id="iframe_0.5319714407383598" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 686px; height: 362px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22https://static.oschina.net/uploads/space/2016/1110/113653_vanl_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.5319714407383598',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

其中api-app-*和api-cxb-*從<iframe id="iframe_0.03820734213560706" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 0px; height: 0px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22http://static.oschina.net/uploads/space/2016/1110/112742_n814_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.03820734213560706',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>來的,*代表所有

<iframe id="iframe_0.09740154046682292" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 686px; height: 294px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22http://static.oschina.net/uploads/space/2016/1110/114021_cGJH_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.09740154046682292',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

代表實時收集的日志條數

<iframe id="iframe_0.31039741402983934" style="margin: 0px; padding: 0px; border-width: initial; border-style: none; width: 0px; height: 0px;" src="data:text/html;charset=utf8,%3Cimg%20id=%22img%22%20src=%22http://static.oschina.net/uploads/space/2016/1110/114329_AWy9_3030681.png?_=6059231%22%20style=%22border:none;max-width:686px%22%3E%3Cscript%3Ewindow.onload%20=%20function%20()%20%7Bvar%20img%20=%20document.getElementById('img');%20window.parent.postMessage(%7BiframeId:'iframe_0.31039741402983934',width:img.width,height:img.height%7D,%20'http://www.cnblogs.com');%7D%3C/script%3E" frameborder="0" scrolling="no"></iframe>

紅色框內的就是在剛才filter過滤規則中定義的

每天進步一點點
 
http://go.rritw.com/www.cnblogs.com/onetwo/p/6059231.html

From:ITEYE
頂一下
(0)
0%
踩一下
(0)
0%
------分隔線----------------------------
發表評論
請自覺遵守互聯網相關的政策法規,嚴禁發布色情、暴力、反動的言論。
評價:
表情:
驗證碼:點擊我更換圖片
欄目列表
推薦內容