Visualize Nginx web Log with GoAccess

Install GoAccess (Ubuntu 18.04)

Using apt

$ sudo apt install goaccess

For the latest version:

$ echo "deb http://deb.goaccess.io/ $(lsb_release -cs) main" | sudo tee -a /etc/apt/sources.list.d/goaccess.list
$ wget -O - https://deb.goaccess.io/gnugpg.key | sudo apt-key add -
$ sudo apt-get update
$ sudo apt-get install goaccess

From source

Install the dependencies:

$ sudo apt-get install libgeoip-dev libncursesw5-dev

Download, configure and install goaccess:

$ wget https://tar.goaccess.io/goaccess-1.4.tar.gz
$ tar -xzvf goaccess-1.4.tar.gz
$ cd goaccess-1.4/
$ ./configure --enable-utf8 --enable-geoip=legacy
$ make
$ sudo make install

Determine Log Format

Once you have GoAccess installed, you should be ready to start using it. However, first you need to determine the log format of your access log. GoAccess comes with several predefined log format options that you can use. Either you can set them permanently in your configuration file or simply passing it through the command line.

Custom log format

goaccess config for custom log format: https://goaccess.io/man#custom-log

Run GoAccess

Terminal Output

The following prompts a log configuration dialog with predefined log formats for you to choose from and then displays the stats in real-time.

goaccess access.log -c

Static HTML Output

The following parses the access log and displays the stats in a static HTML report.

goaccess access.log -o report.html --log-format=COMBINED

Real-Time HTML Output

The following parses the access log and displays the stats in a real-time HTML report.

goaccess access.log -o /var/www/html/report.html --log-format=COMBINED --real-time-html

From nginx log format to goaccess

Small shell script to convert Nginx log_format to goaccess config can be found here:

stockrt/nginx2goaccess

Usage:

$ ./nginx2goaccess.sh '<log_format>'

For example:

$ ./nginx2goaccess.sh '"$time_local" client=$remote_addr method=$request_method request="$request" request_length=$request_length status=$status bytes_sent=$bytes_sent body_bytes_sent=$body_bytes_sent referer=$http_referer user_agent="$http_user_agent" upstream_addr=$upstream_addr upstream_status=$upstream_status request_time=$request_time upstream_response_time=$upstream_response_time upstream_connect_time=$upstream_connect_time upstream_header_time=$upstream_header_time'

The output will be something like this:

time-format %T
date-format %d/%b/%Y
log_format "%d:%t %^" client=%h method=%m request="%r" request_length=%r_length status=%s bytes_sent=%b body_bytes_sent=%b referer=%R user_agent="%u" upstream_addr=%^ upstream_status=%^ request_time=%T upstream_response_time=%^ upstream_connect_time=%^ upstream_header_time=%^

Can be used with -p and read from a config file or placed into ~/.goaccessrc.

References

GoAccess - Visual Web Log Analyzer