Registration is open - Live, Instructor-led Online Classes - Elasticsearch in March - Solr in April - OpenSearch in May. See all classes

Sematext

Solr / Elasticsearch Experts – Search & Big Data Analytics

Menu

Automatic Geo-IP Enrichment for Docker Logs

Posted on by Stefan Thies

In “Innovative Docker Log Management” we wrote about the alternative (and better?) method for Docker logging compared to log drivers, which do only log forwarding. Getting logs from Docker Containers collected, shipped and parsed out of the box is already a big time saver, but some application logs need additional enrichment with information from other data sources. A common use case is to enrich web server logs (or really any logs with IP addresses) with geographical information derived from those IP addresses. Over the last few weeks, we’ve added Geo-IP support to logagent-js (blog post), which is used by the Sematext Docker Agent.

Use Sematext Docker Agent for out of the box Geo-IP support!

Here’s how to enable GeoIP lookups for your logs:

  1. Enable the feature with -e GEOIP_ENABLED=true in the docker run command for sematext/sematext-agent-docker
  2. Geo-IP lookups are enabled for web server logs out of the box (SDA v1.29.32 and above)
  3. Any new pattern in the rich pattern library could use Geo-IP lookup just by adding the setting
    geoIP: fieldName in the pattern. See for example the web server patterns here.

Things you do not need to think about at all:

  1. Maxmind Geo-IP lite database is downloaded automatically (on each start of the agent)

  2. Automatic updates for the GeoIP database is integrated too (update check runs every hour)

  3. Elasticsearch mapping for the Geo-Coordinates in Logsene for geographic queries and map displays

So if you install Sematext Docker Agent on Docker, Docker Cloud, Docker UCP or Docker Swarm all your web server logs will automatically get new fields geoip.location (longitude/latitude), geoip.info.country, geoip.info.city, geoip.info.region, …

The new Geo-IP lookup feature for web server logs needs ZERO configuration for Docker users. Getting Geo-IP information into logs traditionally required administrative/IP work like downloading the Geo-IP database and setup cron jobs to keep it up to date, then addition of configurations for web servers to add this information to logs or configure log shippers like Logstash to do so …. This is not the case when you use Sematext Agent for Docker and the setup is easy, here is a complete example to check it out:

  1. Run Sematext Docker Agent agent as usual:
    docker run -d --name sematext-agent --restart=always \
    -e SPM_TOKEN=    YOUR_SPM_APP_TOKEN_HERE \
    -e LOGSENE_TOKEN=    YOUR_LOGSENE_APP_TOKEN_HERE\
    -e GEOIP_ENABLED=true \
    -v /var/run/docker.sock sematext/sematext-agent-docker

  2. Start nginx (or jwilder/nginx-proxy or Apache if you like … )
    docker run -p 80:80 -v $PWD/content:/usr/share/nginx/html:ro -d nginx

  3. Open a web browser to access nginx http://your-docker-host/

Sematext Docker Agent collects, parses and enriches the nginx logs and then ships them to Logsene. We made a little dashboard in Logsene’s integrated Kibana showing logs, image name of the Docker containers and a map with locations of the clients:

The example above needs no configuration for web server logs or the Geo-IP lookups! It’s never been this simple to get a web server setup including web analytics and performance metrics: One command to run a web server and another one to get structured logs, metrics and events!

We think such an easy setup is a good reason to run web servers on Docker, Docker-Swarm or Docker Cloud!

Do you need support for Geo-IP in other applications running on Docker? Please let us know and get in touch with on Twitter @sematext or via Github for feature requests. If you like what you have seen here give SPM for Docker and Logsene a go!

Start Free Trial