Logging HTTP Requests with Apache: A Quick Tutorial

Fundamentals of Apache Logging

The Apache log records events have information about the Apache responses, requests from other computers, and internal actions to the server. In this guide, we will discuss the fundamentals of Logging, and specifically log types, their storage location, and even their interpretation. In addition to this, we will share some information about module configuration and setting of the custom log formats.

Categories of Apache Logs

Apache logs are of 2 categories such as error logs and access logs.

Access Logs

The access logs include data about requests that are accessing the server. Such kind of log usually contains data about the webpage that was checked, whether this specific request was successful, and how long it took for the request to get results.

Error Logs

As obvious error logs include the data about errors that occurred when the server was processing a specific request. As with access logs, they also include the exact time when the request occurred and the IP address. With error logs, you will be notified about the issues that happened. For instance, if the file is missing,

Locations of Log Files

According to the default setting, the storage for both error and access logs is done separately on the server. Depending on the used OS, the location can vary a little.

Adjusting Apache Logging Settings

Apache offers extremely adjustable logging settings with the help of which it is possible to configure the logging behavior for every virtual host or globally. The logging behavior can be changed via several directives. So, let’s discuss the most common directives which are log format and log level.  

Using the LogLevel Directive

LogLevel is needed for determining of minimum severity level of events. The severity level specifies the importance of a specific event and can be varied. For instance, the severity level can be determined as Trace8 or Emerg. Emerg specifies the events that can lead to some sort of instability, and Trace8 specifies the trace messages. For example, LogLevel crit ignores all the levels except for Emerg, Alert, and Crit.

Defining Custom Log Formats

The Log formats directive is responsible for formatting and layouts of the log events. According to the default characteristics, CLF or Common Format is used, but you can also use a specific format string.

The CustomLog directive is also needed for changing of log files’ location. In most of the Linux systems, the logs are usually written in var/log/httpd or var/log/apache2. Also, you can change the string after the filename so that specific changes will be made only with the specific file.

The full list of the fields can be checked in the log documentation. In order to monitor the server activity and immediately solve the immerging issues, you should use the next 5 fields:

1. %T – specifies the time spent on the request processing. This is a really useful field as you can measure the site speed. To get the same results, but in the microseconds, instead of T, you should use D.
2. %>s – with this option you can check the request status after some internal redirection.
3. %a – it shows the IP from which a request has been made. This means you can identify the traffic activity way more easily.
4. %{ID}e – ID stands for the request ID, which means that every request has its unique identifier. This can be used for tracing requests between the web app server and Apache.
5. %U – URL path, but with the exclusion of the additional parameters for instance query string.

Assigning Log Nicknames

Assigning Log nicknames is extremely helpful, especially in case various log files are used for several virtual hosts. The assigned nickname can be used with CustomLog directive so that logs will be written in the specified format. This means you can specific log formats for different log files and at the same time, you would not waste time and redefine the format every time.

Logging in JSON Format

The storage of the logs in the usual text format makes it way easier to check the files in case there is a necessity. Of course, such a format is easy for usual reading, but it will be challenging to process such data via log management tools because these solutions have a specific format. Sure, most management tools use the default Apache format, but if not, you should consider JSON.

JSON is an abbreviation for JavaScript Object Notation and is a great format for structure data storage. JSON is quite useful because it can store almost any data structure/type. Moreover, this format is great for self-documentation of the information. Among the basic types that are supported by this format, we can specify arrays, numbers, strings, null value, and Boolean operators.

Implementing Log Rotation

The issues that can occur with access logs or other logs are overconsuming of the space or challenges with the management. With the implementation of the log rotation, it is possible to deal with this trouble.

There is a specific rotatelogs program that can assist with rotating logs based on their size and time. The standard logrotate command is also a good variant that you can start from. On most Linux systems Apache has a config file.

The default config file on Debian 11 is located in /etc/logrotate.d/apache2. You can change this file in order to modify the Apache log rotation. Let’s explain the meaning of each section of this config file:

-        Missingok doesn’t show any issues in case the log file is specified.

-        /var/log/apache2/*.log – means that config will relate to all the files that end on .log at the specified location. After { there are specified all the directives for logrotate and after then } is used.

-        Compress means it is made on rotated log files.

-        If daily is mentioned that means, the rotations take place daily, also you can change this value to yearly, weekly, or monthly.

-        Rotate 14- means that 14 log files are kept while others are deleted. In case, the daily rotation is used, then you have approximately 2 weeks of saved logs.

-        Delaycompress means that recent files are uncompressed while the oldest ones will be compressed.

-        Notifempty – means that all the empty files are not rotated.

-        Sharedscripts – postrotate scripts won’t function until all logs are rotated.

-        Create 640 root adm – means that the creation of files with 640 permissions will be created after rotation.

Default Log Settings for Virtual Hosts

Virtual hosts are needed for running several websites on one server. For each vhost, you can specify a separate logging config. That means you can make each site to be logged into a separate directory. Or you can also decide to leave the default global config.

Here is a sample where the logs are written in the separate access.log file:

<VirtualHost *:80>
 ​ServerName skills.com (sample domain)
 ​ServerAdmin webmaster@skills.com
 ​DocumentRoot /var/www/skills.com
 ​LogLevel info ssl:warn
 ​ErrorLog /var/www/skills.com/logs/error.log
 ​CustomLog /var/www/skills.com/logs/access.log example
</VirtualHost>

Debian, Ubuntu, and Linux Mint

On the Debian systems, the default website encryption with SSL/TLS is located at /etc/apache2/sites-available/default-ssl.conf, and vhost config for unencrypted websites is located at /etc/apache2/sites-available/000-default.conf.

Default Directive Table

Red Hat, Fedora, and CentOS

On Red Hat and similar systems, the vhost config files are located in /etc/httpd/conf.d and the main config file is located at /etc/httpd/conf/httpd.conf.

Default Directive Table

OpenSUSE

In the OpenSUSE system, the config for websites with SSL/TLS is at /etc/apache2/default-vhost-ssl.conf and for unencrypted at /etc/apache2/default-vhost.conf.

Default Directive Table

Apache Modules for Logging

There are several modules that can significantly help with changing logging behavior. Based on our knowledge gained from the Apache server configuration, we will share important information about the useful Apache modules for logging.

mod_log_config

This is probably the most basic module and one that we have already discussed in this article.

mod_log_debug

This module might not be included in all the Apache distributions, but it is helpful for logging debug messages. For instance, this module can be used to log messages from certain clients, log events to a certain URL path, and more.

For instance:

<Location /path/to/some/directory>
​LogMessage "/path/to/directory has been requested by"
​${REMOTE_ADDR}
</Location>

mod_log_forensic

With this module, it is possible to logging before/after the process of request. After the enabling of this module, the keywords may be used to determine which log file is forensic. It is also possible to use %{forensic-id} in various strings to add forensic data to usual logs.

Here is a sample:

<VirtualHost *:80>
​ServerName skills.com (it is a sample domain)
   ServerAdmin webmaster@skills.com
   DocumentRoot /var/www/skills.com
   LogLevel info ssl:warn
   ErrorLog /var/www/skills.com/logs/error.log
   CustomLog /var/www/skills.com/logs/access.log example
   CustomLog /var/www/skills.com/logs/forensic.log forensic
</VirtualHost>

mod_logio

Such a module offers the possibility to log the number of bytes received/sent per request. This module also accounts for changes in size because of TLS/SSL encryption and is usually included based on the default characteristics.  

mod_filter

This module helps with extracting certain requests based on the given filter. The context-sensitive filter which is provided by this module contains vhost config, apache config, .htaccess files, and more.

This module is usually a default option, but you might need to enable it.

mod_unique_id

This module is needed for the construction of the unique identifier for every request. The identifier is written to the access log. It is passed to the app handler with the help of the environment variable UNIQUE_ID. So, it makes it much more easier for the developers to trace requests.