Getting visibility into your application is crucial when running your code in production. What do we mean by visibility? Primarily things like application performance via metrics, application health, and availability, its logs should you need to troubleshoot it, or its traces if you need to figure out what makes it slow and how to make it faster.
Metrics give you information about the performance of each of the elements of your infrastructure. Traces will show you a broader view of the code execution and flow along with code execution metrics. Finally, well-crafted logs will provide an invaluable insight into the code logic execution and what was happening in your code. Each of the mentioned pieces is crucial to your application and indeed the overall system observability. Today, we will focus only on a single piece – the logs. To be more precise – on Java application logs. If you’re interested in metrics, check out our article on key JVM metrics you should monitor.
However, before we get into it, let’s address an issue that impacted the community using this framework. On December 9, 2021, a critical vulnerability nicknamed Log4Shell was reported. Identified as CVE-2021-44228, it lets an attacker take complete control of a machine running Apache Log4j 2 version 2.14.1 or lower, allowing them to execute arbitrary code on the vulnerable server. In our recent blog post about the Log4jShell vulnerability, we detailed how to determine whether you are affected, how to resolve the issue, and what we, at Sematext, have done to protect our system and users.
Log4j 1.x End Of Life
Keep in mind that on August 5th, 2015 the Logging Services Project Management Committee announced that the Log4j 1.x had reached its end of life. All users are advised to migrate to Log4j 2.x. In this blog post, we will help you understand your current Log4j setup – specifically, the log4j 2.x version – and after that, I’ll help you migrate to the latest and greatest Log4j version.
“I’m using Log4j 1.x, what should I do?”. Don’t panic, there is nothing wrong with it. Make a plan to transition to Log4j 2.x. I’ll show you how just keep reading :). Your application will thank you for that. You will get the security fixes, performance improvements, and far more features after migration.
“I’m starting a new project, what should I do?”. Just use the Log4j 2.x right away, don’t even think about Log4j 1.x. If you need help with that, check out this Java logging tutorial where I explain everything you need.
Logging in Java
There is no magic behind logging in Java. It all comes down to using an appropriate Java class and its methods to generate log events. As we discussed in the Java logging guide there are multiple ways you can start
Of course, the most naive one and not the best path to follow is just using the System.out and System.err classes. Yes, you can do that and your messages will just go to the standard output and standard error. Usually, that means that it will be printed to the console or written to a file of some kind or even sent to /dev/null and forever be forgotten. An example of such code could look like this:
public class SystemExample { public static void main(String[] args) { System.out.println("Starting my awesome application"); // some work to be done System.out.println( String.format("My application %s started successfully", SystemExample.class) ); } }
The output of the above code execution would be as follows:
Starting my awesome application My application class com.sematext.logging.log4jsystem.SystemExample started successfully
That ain’t perfect, right? I don’t have any information about which class generated the message and many, many more “small” things that are crucial and important during debugging.
There are other things that are missing that are not really connected to debugging. Think about the execution environment, multiple applications or microservices, and the need to unify logging to simplify the log centralization pipeline configuration. Using the System.out, or/and System.err in our code for logging purposes would force us to redo all the places where we use it whenever we need to adjust the logging format. I know it is extreme, but believe me, we’ve seen the use of System.out in production code in “traditional” application deployment models! Of course, logging to System.out is a proper solution for containerized applications and you should use the output that fits your environment. Remember about that!
Because of all the mentioned reasons and many more that we do not even think of, you should look into one of the possible logging libraries, like Log4 j, Log4j 2, Logback, or even the java.util.logging that is a part of the Java Development Kit. For this blog post, we will use Log4j.
The Abstraction Layer – SLF4J
The topic of choosing the right logging solution for your Java application is something that we already discussed in our tutorial about logging in Java. We highly recommend reading at least the mentioned section.
We will be using SLF4J, an abstraction layer between our Java code and Log4j – the logging library of our choice. The Simple Logging Facade provides bindings for common logging frameworks like Log4j, Logback, and java.util.logging. Imagine the process of writing a log message in the following, simplified way:
You may ask why use an abstraction layer at all? Well, the answer is quite simple – eventually, you may want to change the logging framework, upgrade it, unify it with the rest of your technology stack. When using an abstraction layer such operation is fairly simple – you just exchange the logging framework dependencies and provide a new package. If we were not to use an abstraction layer – we would have to change the code, potentially lots of code. Each class that logs something. Not a very nice development experience.
[sematext_banner type=”logs”]
The Logger
The code of your Java application will be interacting with a standard set of key elements that allow log event creation and manipulation. We’ve covered the crucial ones in our Java logging tutorial, but let me remind you about one of the classes that we will constantly use – the Logger.
The Logger is the main entity that an application uses to make logging calls – create log events. The Logger object is usually used for a single class or a single component to provide context-bound to a specific use case. It provides methods to create log events with an appropriate log level and pass it on for further processing. You will usually create a static object that you will interact with, for example like this:
... Logger LOGGER = LoggerFactory.getLogger(MyAwesomeClass.class);
And that’s all. Now that we know what we can expect, let’s look at the Log4j library.
Log4j
The simplest way to start with Log4j is to include the library in the classpath of your Java application. To do that we include the newest available log4j library, which means version 1.2.17 in our build file.
We use Gradle and in our simple application and the dependencies section for Gradle build file looks as follows:
dependencies { implementation 'log4j:log4j:1.2.17' }
We can start developing the code and include logging using Log4j:
package com.sematext.blog; import org.apache.log4j.Logger; public class ExampleLog4j { private static final Logger LOGGER = Logger.getLogger(ExampleLog4j.class); public static void main(String[] args) { LOGGER.info("Initializing ExampleLog4j application"); } }
As you can see in the above code we initialized the Logger object by using the static getLogger method and we provided the name of the class. After doing that we can easily access the static Logger object and use it to produce log events. We can do that in the main method.
One side note – the getLogger method can be also called with a String as an argument, for example:
private static final Logger LOGGER = Logger.getLogger("com.sematext.blog");
It would mean that we want to create a logger and associate the name of com.sematext.blog with it. If we will use the same name anywhere else in the code Log4j will return the same Logger instance. That is useful if we wish to combine logging from multiple different classes in a single place. For example, logs related to payment in a single, dedicated log file.
Log4j provides a list of methods allowing the creation of new log events using an appropriate log level. Those are:
- public void trace(Object message)
- public void debug(Object message)
- public void info(Object message)
- public void warn(Object message)
- public void error(Object message)
- public void fatal(Object message)
And one generic method:
- public void log(Level level, Object message)
We talked about Java logging levels in our Java logging tutorial blog post. If you are not aware of them, please take a few minutes to get used to them as the log levels are crucial for logging. If you’re just getting started with logging levels, though, we recommend you go over our log levels guide as well. We explain everything from what they are to how to choose the right one and how to make use of them to get meaningful insights.
If you like videos, we’ve also gone over log levels in a comprehensive but short video. You can check that out below:
Back to the tutorial, if we were to run the above code the output we’d get on the standard console would be as follows:
log4j:WARN No appenders could be found for logger (com.sematext.blog.ExampleLog4j). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
We didn’t see the log message that we expected. Log4j informed us that there is no configuration present. Oooops, let’s talk about how to configure Log4j…
Log4j Configuration
There are multiple ways we can configure our Log4j logging. We can do it programmatically – for example by including a static initialization block:
static { BasicConfigurator.configure(); }
The above code configures Log4j to output the logs to the console in the default format. The output of running our example application would look as follows:
0 [main] INFO com.sematext.blog.ExampleLog4jProgrammaticConfig - Initializing ExampleLog4j application
However, setting up Log4j programmatically isn’t very common. The most common way would be to either use a properties file or an XML file. We can change our code and include the log4j.properties file with the following content:
log4j.rootLogger=DEBUG, MAIN log4j.appender.MAIN=org.apache.log4j.ConsoleAppender log4j.appender.MAIN.layout=org.apache.log4j.PatternLayout log4j.appender.MAIN.layout.ConversionPattern=%r [%t] %-5p %c %x - %m%n
That way we told Log4j that we create the root logger, that will be used by default. Its default logging level is set to DEBUG, which means that log events with severity DEBUG or higher will be included. So DEBUG, INFO, WARN, ERROR, and FATAL. We also gave our logger a name – MAIN. Next, we configure the logger, by setting its output to console and by using the pattern layout. We will talk about it more later in the blog post. The output of running the above code would be as follows:
0 [main] INFO com.sematext.blog.ExampleLog4jProperties - Initializing ExampleLog4j application
If we wish, we can also change the log4j.properties file and use one called log4j.xml. The same configuration using XML format would look as follows:
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd"> <log4j:configuration> <appender name="MAIN" class="org.apache.log4j.ConsoleAppender"> <param name="Target" value="System.out"/> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%r [%t] %-5p %c %x - %m%n" /> </layout> </appender> <root> <priority value ="debug"></priority> <appender-ref ref="MAIN" /> </root> </log4j:configuration>
If we would now change the log4j.properties for log4j.xml one and keep it in the classpath the execution of our example application would be as follows:
0 [main] INFO com.sematext.blog.ExampleLog4jXML - Initializing ExampleLog4j application
So how does Log4j know which file to use? Let’s look into that.
Initialization Process
It is crucial to know that Log4j doesn’t make any assumptions regarding the environment it is running in. Log4j doesn’t assume any kind of default log events destinations. When it starts it looks for the log4j.configuration property and tries to load the specified file as its configuration. If the location of the file can’t be converted to a URL or the file is not present it tries to load the file from the classpath.
That means that we can overwrite the Log4j configuration from the classpath by providing the -Dlog4j.configuration during startup and pointing it to the correct location. For example, if we include a file called other.xml with the following content:
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd"> <log4j:configuration> <appender name="MAIN" class="org.apache.log4j.ConsoleAppender"> <param name="Target" value="System.out"/> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%r %-5p %c %x - %m%n" /> </layout> </appender> <root> <priority value ="debug"></priority> <appender-ref ref="MAIN" /> </root> </log4j:configuration>
And then run out code with -Dlog4j.configuration=/opt/sematext/other.xml the output from our code will be as follows:
0 INFO com.sematext.blog.ExampleLog4jXML - Initializing ExampleLog4j application
Log4j Appenders
We already used appenders in our examples… well, really just one – the ConsoleAppender. Its sole purpose is to write the log events to the console. Of course, with a large number of log events and systems running in different environments writing pure text data to the standard output may not be the best idea, unless you are running in containers. That’s why Log4j supports multiple types of Appenders. Here are a few common examples of Log4j appenders:
- ConsoleAppender – the appender that appends the log events to System.out or System.err with the default being System.out. When using this appender you will see your logs in the console of your application.
- FileAppender – the appender that appends the log events to a defined file storing them on the file system.
- RollingFileAppender – the appender that extends the FileAppender and rotates the file when it reaches a defined size. The use of RollingFileAppender prevents the log files from becoming very big and hard to maintain.
- SyslogAppender – the appender sending the log events to a remote Syslog daemon.
- JDBCAppender – the appender that stores the log events to the database. Keep in mind that this appender will not store errors and it is generally not the best idea to store the log events in a database.
- SocketAppender – the appender that sends the serialized log events to a remote socket. Keep in mind that this appender doesn’t use layouts because it sends the serialized, raw log events.
- NullAppender – the appender that just discards the log events.
What’s more, you can have multiple Appenders configured for a single application. For example, you can send logs to the console and to a file. The following log4j.properties file contents would do exactly that:
log4j.rootLogger=DEBUG, MAIN, ROLLING log4j.appender.MAIN=org.apache.log4j.ConsoleAppender log4j.appender.MAIN.layout=org.apache.log4j.PatternLayout log4j.appender.MAIN.layout.ConversionPattern=%r %-5p %c %x - %m%n log4j.appender.ROLLING=org.apache.log4j.RollingFileAppender log4j.appender.ROLLING.File=/var/log/sematext/awesome.log log4j.appender.ROLLING.MaxFileSize=1024KB log4j.appender.ROLLING.MaxBackupIndex=10 log4j.appender.ROLLING.layout=org.apache.log4j.PatternLayout log4j.appender.ROLLING.layout.ConversionPattern=%r [%t] %-5p %c %x - %m%n
Our root logger is configured to log everything starting from the DEBUG severity and to send the logs to two Appenders – the MAIN and the ROLLING. The MAIN logger is the one that we already saw – the one that sends the data to the console.
The second logger, the one called ROLLING is the more interesting one in this example. It uses the RollingFileAppender which writes the data to the file and let’s define how large the file can be and how many files to keep. In our case, the log files should be called awesome.log and write the data to /var/log/sematext/ directory. Each file should be a maximum of 1024KB and there shouldn’t be more than 10 files stored. If there are more files they will be removed from the file system as soon as log4j sees them.
After running the code with the above configuration the console would print the following content:
0 INFO com.sematext.blog.ExampleAppenders - Starting ExampleAppenders application 1 WARN com.sematext.blog.ExampleAppenders - Ending ExampleAppenders application
In the /var/log/sematext/awesome.log file we would see:
0 [main] INFO com.sematext.blog.ExampleAppenders - Starting ExampleAppenders application 1 [main] WARN com.sematext.blog.ExampleAppenders - Ending ExampleAppenders application
Appender Log Level
The nice thing about Appenders is that they can have their level which should be taken into consideration when logging. All the examples that we’ve seen so far logged every message that had the severity of DEBUG or higher. What if we wanted to change that for all the classes in the com.sematext.blog package? We would only have to modify our log4j.properties file:
log4j.rootLogger=DEBUG, MAIN log4j.appender.MAIN=org.apache.log4j.ConsoleAppender log4j.appender.MAIN.layout=org.apache.log4j.PatternLayout log4j.appender.MAIN.layout.ConversionPattern=%r %-5p %c %x - %m%n log4j.logger.com.sematext.blog=WARN
Look at the last line in the above configuration file. We’ve used the log4j.logger prefix and said that the logger called com.sematext.blog should only be used for severity levels WARN and above, so ERROR and FATAL.
Our example application code looks like this:
public static void main(String[] args) { LOGGER.info("Starting ExampleAppenderLevel application"); LOGGER.warn("Ending ExampleAppenderLevel application"); }
With the above Log4j configuration the output of the logging looks as follows:
0 WARN com.sematext.blog.ExampleAppenderLevel - Ending ExampleAppenderLevel application
As you can see, only the WARN level log was included. That is exactly what we wanted.
Log4j Layouts
Finally, the part of the Log4j logging framework that controls the way our data is structured in our log file – the layout. Log4j provides a few default implementations like PatternLayout, SimpleLayout, XMLLayout, HTMLLayout, EnchancedPatternLayout, and the DateLayout.
In most cases, you will encounter the PatternLayout. The idea behind this layout is that you can provide a variety of formatting options to define the log structure. Some of the examples are:
- d – date and time of the log event,
- m – message associated with the log event,
- t – thread name,
- n – platform dependent line separator,
- p – log level.
For more information on the available options head over to the official Log4j Javadocs for the PatternLayout.
When using the PatternLayout we can configure which option we would like to use. Let’s assume we would like to write the date, the severity of the log event, the thread surrounded by square brackets, and the message of the log event. We could use a pattern like this:
%d %-5p [%t] - %m%n
The full log4j.properties file in this case could look as follows:
log4j.rootLogger=DEBUG, MAIN log4j.appender.MAIN=org.apache.log4j.ConsoleAppender log4j.appender.MAIN.layout=org.apache.log4j.PatternLayout log4j.appender.MAIN.layout.ConversionPattern=%d %-5p [%t] - %m%n
We use the %d to display the date, the %-5p to display the severity using 5 characters, %t for thread, %m for the message, and the %n for line separator. The output that is written to the console after running our example code looks as follows:
2021-02-02 11:49:49,003 INFO [main] - Initializing ExampleLog4jFormatter application
Nested Diagnostic Context
In most real-world applications the log event doesn’t exist on its own. It is surrounded by a certain context. To provide such context, per-thread, Log4j provides the so-called Nested Diagnostic Context. That way we can bound a given thread with additional information, for example, a session identifier, just like in our example application:
NDC.push(String.format("Session ID: %s", "1234-5678-1234-0987")); LOGGER.info("Initializing ExampleLog4jNDC application");
When using a pattern that includes x variable additional information will be included in each logline for the given thread. In our case the output will look like this:
0 [main] INFO com.sematext.blog.ExampleLog4jNDC Session ID: 1234-5678-1234-0987 - Initializing ExampleLog4jNDC application
You can see that the information about the session identifier is in the logline. Just for reference, the log4j.properties file that we used in this example looks as follows:
log4j.rootLogger=DEBUG, MAIN log4j.appender.MAIN=org.apache.log4j.ConsoleAppender log4j.appender.MAIN.layout=org.apache.log4j.PatternLayout log4j.appender.MAIN.layout.ConversionPattern=%r [%t] %-5p %c %x - %m%n
Mapped Diagnostic Context
The second type of contextual information that we can include in our log events is mapped diagnostic context. Using the MDC class we can provide additional key-value related information. Similar to nested diagnostic context, the mapped diagnostic context is thread-bound.
Let’s look at our example application code:
MDC.put("user", "rafal.kuc@sematext.com"); MDC.put("step", "initial"); LOGGER.info("Initializing ExampleLog4jNDC application"); MDC.put("step", "launch"); LOGGER.info("Starting ExampleLog4jNDC application");
We have two context fields – the user and the step. To display all of the mapped diagnostic context information associated with the log event we just use the X variable in our pattern definition. For example:
log4j.rootLogger=DEBUG, MAIN log4j.appender.MAIN=org.apache.log4j.ConsoleAppender log4j.appender.MAIN.layout=org.apache.log4j.PatternLayout log4j.appender.MAIN.layout.ConversionPattern=%r [%t] %-5p %c %X - %m%n
Launching the above code along with the configuration would result in the following output:
0 [main] INFO com.sematext.blog.ExampleLog4jMDC {{step,initial}{user,rafal.kuc@sematext.com}} - Initializing ExampleLog4jNDC application 1 [main] INFO com.sematext.blog.ExampleLog4jMDC {{step,launch}{user,rafal.kuc@sematext.com}} - Starting ExampleLog4jNDC application
We can also choose which information to use by changing the pattern. For example, to include the user from the mapped diagnostic context we could write a pattern like this:
%r [%t] %-5p %c %X{user} - %m%n
This time the output would look as follows:
0 [main] INFO com.sematext.blog.ExampleLog4jMDC rafal.kuc@sematext.com - Initializing ExampleLog4jNDC application 0 [main] INFO com.sematext.blog.ExampleLog4jMDC rafal.kuc@sematext.com - Starting ExampleLog4jNDC application
You can see that instead of the general %X we’ve used the %X{user}. That means that we are interested in the user variable from the mapped diagnostic context associated with a given log event.
Migration to Log4j 2
Migration from Log4j 1.x to Log4j 2.x is not hard and, in some cases, it may be very easy. If you didn’t use any internal Log4j 1.x classes, you’ve used configuration files over programmatically setting up loggers and you didn’t use the DOMConfigurator and PropertyConfigurator classes the migration should be as simple as including the log4j-1.2-api.jar jar file instead of the Log4j 1.x jar files. That would allow Log4j 2.x to work with your code. You would need to add the Log4j 2.x jar files, adjust the configuration, and voilà – you’re done.
If you would like to learn more about Log4j 2.x check out our Java logging tutorial and its Log4j 2.x dedicated section.
However, if you did use internal Log4j 1.x classes, the official migration guide on how to move from Log4j 1.x to Log4j 2.x will be very helpful. It discusses the needed code and configuration changes and will be invaluable when in doubt.
Centralized Logging with Log Management Tools
Sending log events to a console or a file may be good for a single application, but handling multiple instances of your application and correlating the logs from multiple sources is no fun when the log events are in text files on different machines. In such cases, the amount of data quickly becomes unmanageable and requires dedicated solutions – either self-hosted or coming from one of the vendors. And what about containers where you typically don’t even write logs to files? How do you troubleshoot and debug an application whose logs were emitted to standard output, or whose container has been killed?
This is where log management services, log analysis tools, and cloud logging services come into play. It’s an unwritten Java logging best practice among engineers to use such solutions when you’re serious about managing your logs and getting the most out of them. For example, Sematext Logs, our log monitoring and management software, solves all the problems mentioned above, and more.
With a fully managed solution like Sematext Logs, you don’t have to manage another piece of the environment – your DIY logging solution, typically built using pieces of the Elastic Stack. Such setups may start small and cheap, however, they often grow big and expensive. Not just in terms of infrastructure costs, but also management costs. You know, time and payroll. We explain more about the advantages of using a managed service in our blog post about logging best practices.
Alerting and log aggregation are also crucial when dealing with problems. Eventually, for Java applications, you may want to have garbage collection logs once you turn garbage collection logging on and start analyzing the logs. Such logs correlated with metrics are an invaluable source of information for troubleshooting garbage collection-related problems.
Summary
Even though Log4j 1.x reached its end of life a long time ago it is still present in a large number of legacy applications used all over the world. The migration to its younger version is fairly simple, but may require substantial resources and time and is usually not a top priority. Especially in large enterprises where the procedures, legal requirements, or both require audits followed by long and expensive testing before anything can be changed in an already running system. But for those of us who are just starting or thinking about migration – remember, Log4j 2.x is there, it is already mature, fast, secure, and very capable.
But regardless of the framework you’re using for logging your Java applications, we definitely recommend marrying your efforts with a fully-managed log management solution, such as Sematext Logs. If you’re interested in learning more, check out this short video below:
Give Sematext a try! There’s a 14-day free trial available for you to test drive it.
Happy logging!