The graylog-ctl script¶

Configuration commands¶

The following commands are changing the configuration of Graylog:

Commands for multi node setups:

General commands:

Multi VM setup¶

At some point it make sense to not run all services on a single VM anymore. For performance reasons you might want to add more Elasticsearch nodes to the cluster or even add a second Graylog server. This can be achieved by changing IP addresses in the Graylog configuration files by hand or use our canned configurations which come with the graylog-ctl command.

The idea is to have one VM which is a central point for other VMs to fetch all needed configuration settings to join the cluster. Typically the first VM you spin up is used for this task. Automatically an instance of etcd is started and filled with the necessary settings for other hosts.

For example, to create a small cluster with a dedicated Graylog server node and another for Elasticsearch, spin up two VMs from the same Graylog image. On the first one start only Graylog and MongoDB:

vm1> sudo graylog-ctl set-admin-password sEcReT
vm1> sudo graylog-ctl reconfigure-as-server

On the second VM start only Elasticsearch. Before doing so set the IP of the first VM to fetch the configuration data from there:

vm2> sudo graylog-ctl set-cluster-master <ip-of-vm1>
vm2> sudo graylog-ctl reconfigure-as-datanode

vm1> sudo graylog-ctl reconfigure-as-server

This results in a perfectly fine dual VM setup. However if you want to scale this setup out by adding an additional Elasticsearch node, you can proceed in the same way:

vm3> sudo graylog-ctl set-cluster-master <ip-of-vm1>
vm3> sudo graylog-ctl reconfigure-as-datanode

vm1> sudo graylog-ctl reconfigure-as-server
vm2> sudo graylog-ctl reconfigure-as-datanode

Verify that all nodes are working as a cluster by going to the Kopf plugin on one of the Elasticsearch nodes open http://vm2:9200/_plugin/kopf/#!/nodes.

Important: In case you want to add a second Graylog server you have to set the same server secret on all machines. The secret is stored in the file /etc/graylog/graylog-secrets and can be applied to other hosts with the set-server-secret sub-command.

The following configuration modes do exist:

A server with only the web interface running is not supported as of Graylog 2.0. The web interface is now included in the server process. But you can create your own service combinations by editing the file /etc/graylog/graylog-services.json by hand and enable or disable single services. Just run graylog-ctl reconfigure afterwards.

Extend disk space¶

All data of the appliance setup is stored in /var/opt/graylog/data. In order to extend the disk space mount a second (virtual) hard drive into this directory.

Install Graylog plugins¶

The Graylog plugin directory is located in /opt/graylog/plugin/. Just drop a JAR file there and restart the server with sudo graylog-ctl restart graylog-server to load the plugin.

Install Elasticsearch plugins¶

Elasticsearch comes with a helper program to install additional plugins you can call it like this sudo JAVA_HOME=/opt/graylog/embedded/jre /opt/graylog/elasticsearch/bin/plugin

Install custom SSL certificates¶

During the first reconfigure run self signed SSL certificates are generated. You can replace this certificate with your own to prevent security warnings in your browser. Just drop the key and combined certificate file here: /opt/graylog/conf/nginx/ca/graylog.crt respectively /opt/graylog/conf/nginx/ca/graylog.key. Afterwards restart nginx with sudo graylog-ctl restart nginx.

Assign a static IP¶

Per default the appliance make use of DHCP to setup the network. If you want to access Graylog under a static IP please follow these instructions:

$ sudo ifdown eth0

Edit the file /etc/network/interfaces like this (just the important lines):

auto eth0
  iface eth0 inet static
  address <static IP address>
  netmask <netmask>
  gateway <default gateway>
  pre-up sleep 2

Activate the new IP and reconfigure Graylog to make use of it:

$ sudo ifup eth0
$ sudo graylog-ctl reconfigure

Wait some time until all services are restarted and running again. Afterwards you should be able to access Graylog with the new IP.

Upgrade Graylog¶

Always perform a full backup or snapshot of the appliance before proceeding. Only upgrade if the release notes say the next version is a drop-in replacement. Choose the Graylog version you want to install from the list of Omnibus packages . graylog_latest.deb always links to the newest version:

$ wget https://packages.graylog2.org/releases/graylog-omnibus/ubuntu/graylog_latest.deb
$ sudo graylog-ctl stop
$ sudo dpkg -G -i graylog_latest.deb
$ sudo graylog-ctl backup-etcd
$ sudo graylog-ctl reconfigure
$ sudo reboot

Errno::ECONNREFUSED
-------------------
Connection refused - connect(2) for "127.0.0.1" port 4001``

Migrate manually from 1.x to 2.x¶

To update a 1.x appliance to 2.x the administrator has to purge the Graylog installation, migrate the stored log data and install the new version as Omnibus package. Before upgrading read the upgrade notes. This procedure can potentially delete log data or configuration settings. So it’s absolutely necessary to perform a backup or a snapshot before!

Stop all services but Elasticsearch:

$ sudo -s
$ graylog-ctl stop graylog-web
$ graylog-ctl stop graylog-server
$ graylog-ctl stop mongodb
$ graylog-ctl stop nginx
$ graylog-ctl stop etcd

Check for index range types. The output of this command should be {}, if not read these notes for how to fix this:

$ curl -XGET <appliance_IP>:9200/_all/_mapping/index_range; echo
{}

Delete the Graylog index template:

$ curl -X DELETE <appliance_IP>:9200/_template/graylog-internal

Migrate appliance configuration:

$ cd /etc
$ mv graylog graylog2.2
$ vi graylog2.2/graylog-secrets.json

# Remove the graylog_web section
},  << don't forget the comma!
"graylog_web": {
  "secret_token": "3552c87f3e3..."
}

$ vi graylog2.2/graylog-services.json

# Remove the graylog_web section
}, << don't forget the comma!
"graylog_web": {
  "enabled": true
}

$ vi graylog2.2/graylog-settings.json

# Remove "rotation_size", "rotation_time", "indices"
"enforce_ssl": false,
"rotation_size": 1073741824,
"rotation_time": 0,
"indices": 10,
"journal_size": 1,

Migrate appliance data:

$ cd /var/opt
$ mv graylog graylog2.2
$ mv graylog2.2/data/elasticsearch/graylog2 graylog2.2/data/elasticsearch/graylog

Delete old Graylog version and install new Omnibus package:

$ wget http://packages.graylog2.org/releases/graylog-omnibus/ubuntu/graylog_2.2.1-1_amd64.deb
$ apt-get purge graylog
$ dpkg -i graylog_2.2.1-1_amd64.deb

Move directories back:

$ cd /etc
$ mv graylog2.2 graylog
$ cd /var/opt/
$ mv graylog2.2 graylog

Reconfigure and Reboot:

$ graylog-ctl reconfigure
$ reboot

Graylog should now be updated and old data still available.

Advanced Settings¶

To change certain parameters used by graylog-ctl during a reconfigure run you can override all default parameters found in the attributes file.

If you want to change the username used by Graylog for example, edit the file /etc/graylog/graylog-settings.json like this:

"custom_attributes": {
  "user": {
    "username": "log-user"
  }
}

Afterwards run sudo graylog-ctl reconfigure and sudo graylog-ctl restart. The first command renders all changed configuration files and the later makes sure that all services restart to activate the change.

There are a couple of other use cases of this, e.g. change the default data directories used by Graylog to /data (make sure this is writeable by the graylog user):

"custom_attributes": {
    "elasticsearch": {
      "data_directory": "/data/elasticsearch"
    },
    "mongodb": {
      "data_directory": "/data/mongodb"
    },
    "etcd": {
      "data_directory": "/data/etcd"
    },
    "graylog-server": {
      "journal_directory": "/data/journal"
    }
  }

Or change the default memory settings used by Graylog or Elasticsearch:

"custom_attributes": {
     "graylog-server": {
       "memory": "1700m"
     },
     "elasticsearch": {
       "memory": "2200m"
     }
   }

Again, run reconfigure and restart afterwards to activate the changes.

Securing an appliance¶

Even though the Graylog appliances are not meant for production use there are still two commands you can use to increase the security of an installation. With graylog-ctl local-connect only the web interface is reachable from the outside. All other services are listening on the local loopback device. This is only useful when you run the appliance as a single node. Clustered setups are not possible anymore. But data stored in MongoDB or Elastcsearch are protected from direct external access.

The other one is graylog-ctl set-mongodb-password. This command enables authentication for MongoDB and creates or updates a database user. First an admin user should be created. This user is needed for database maintenance and future password changes. Afterwards an unprivileged service user can be created for Graylog. The procedure works like this:

$ graylog-ctl set-mongodb-password -a -u admin -p someAdminPassword123
$ graylog-ctl set-mongodb-password -g -u graylog -p someGraylogServicePassword
$ graylog-ctl reconfigure

MongoDB and the Graylog server will be restarted with activated authentication. The username and password needs to be set on every Graylog node to make a cluster work. Login to another Graylog server and only set the service user:

$ graylog-ctl set-cluster-master 1.1.1.2
$ graylog-ctl set-mongodb-password -g -u graylog -p someGraylogServicePassword
$ graylog-ctl reconfigure-as-server

Since the pre-build appliances are based on standard Ubuntu-Linux, tools like iptables/SELinux/AppArmor can be used additionally. But to explain all available countermeasurements would go beyond this documentation.

Restore cluster configuration¶

With graylog-ctl backup-etcd a backup of the cluster configuration of a multi node setup can be created. In order to restore this backup copy the wal-file back to the data directory:

$ graylog-ctl stop etcd
$ rm -r /var/opt/graylog/data/etcd/member/*
$ cp /var/opt/graylog/backup/etcd/<timestamp>/member/wal /var/opt/graylog/data/etcd/member/
$ chown -R graylog.graylog /var/opt/graylog/data/etcd/member/wal
$ su -c '/opt/graylog/embedded/sbin/etcd -data-dir=/var/opt/graylog/data/etcd -force-new-cluster' graylog
<Ctrl-C>
$ graylog-ctl start etcd