Secure, Flexible and Scalable Build Configuration with Maven

Many of the projects I'm involved in use Maven as build system due to its reliability, widespread use, and flexibility. However, I often find myself hurdling around bad choices with regard to the build process.

In this article I'd like to illustrate some of the most useful techniques to implement flexible, adaptable, and secure builds with Maven.

Download the example project from GitHub

Key requirements of an enterprise build

Enterprise projects have demanding requirements when it comes to configuration and build management. The main reasons are that they often last years, they employ many people from different teams (developers, release managers, configuration managers, system administrators) and... they change A LOT during their lifetime.

In short, there's a need for a build system that is:

• easy to use
• flexible
• secure

Maven builds, if properly used, can fulfill all of these requirements. Let's see how.

First version: fixed configurations

In this first version we're adding a simple application.properties configuration file. We will put it in src/main/resources folder, which is the default location of application resources for Maven builds.

src/main/resources/application.properties

# JDBC Configuration

jdbc.driverClassName=com.mysql.jdbc.Driver
jdbc.url=jdbc:mysql://localhost:3306/maven-flexiconf
jdbc.username=root
jdbc.password=password

# Logging

log.path=C:/logs/my-application
log.rolled.path=C:/logs/my-application/rolled

During the process-resources phase Maven will process all resources under src/main/resources, copying them to the target/classes folder.

Try it yourself:

mvn process-resources

The output will be something similar:

...
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ maven-flexiconf ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
...

The problem: shared, inflexible configuration

This first example suffers from various problems.

The most obvious, and the source of all problems, is that each build has the same configuration. Developers, Continuous Integration, even QA and Production (!) share the same exact application.properties file! This scenario, as crazy as it seems, is not so uncommon as you may think. Unfortunately, I've been lucky enough to see projects using this "style". Development was a pain as it was hindered by these huge, complicated, unflexible configurations files.

There are so many problems with this approach that I really don't know where to start.

1) Developer on-boarding is a pain

Remember the "Logging" section of out configuration file?

log.path=C:/logs/my-application
log.rolled.path=C:/logs/my-application/rolled

It looks like the original developer used Windows, but maybe you are on Linux, Mac OSX, o maybe you are using Windows but just cant't use the same directories for whatever reason. Farewell build portability!

Some astute team members would tell you:

"Oh, well, just modify them locally. But remember not to commit them, or you will ruin our QA Environment!"

Well, yeah, sure. Am I the only one who hears a bomb ticking?

2) Shared Database = territorial pissing

While not always an easy option, each developer should have a separate database, otherwise development will be hindered by territorial disputes over who has the right to do schema changes or the ownership of test data.

Good luck trying to implement a new feature that requires significant schema changes, launching integration tests that involves a lot of data, all while other developers are in a rush for a bugfix release. Total. Chaos.

3) Manual or semi-manual builds

In this scenario you will get to know a "Build / Release Manager" who, despite the pompous name, is in charge of the most tedious job in the world:

• download the latest commits
• modify configuration files (locally only)
• manually build the artifacts (usually skipping tests): JARs, WARs, EARs
• deploy on DEV/QA/PRODUCTION environments
• fan out emails to the whole team complaining that the latest deploy "did not work"

A Layered Approach

To solve those problems, over the years I've developed my own "recipes" for Maven builds, which combine several Maven mechanisms to handle resource files, namely Resource Filtering and Build Profiles.

Resource Filtering

The process-resources phase of a Maven build can process resources before copying them to the target folder, replacing placeholders with actual values. This process is called "Filtering".

Activating resources can be done using <resources> elements inside <build> section of the POM (pom.xml - Project Object Model):

<build>
    <resources>
        <resource>
            <directory>src/main/resources</directory>
            <filtering>true</filtering>
        </resource>
    </resources>
</build>

The lines above tell Maven to replace placeholders with their values.

Now, our application.properties can be modified to contain only placeholders:

# JDBC Configuration

jdbc.driverClassName=${jdbc.driverClassName}
jdbc.url=${jdbc.url}
jdbc.username=${jdbc.username}
jdbc.password=${jdbc.password}

# Logging

log.path=${log.path}
log.rolled.path=${log.rolled.path}

How to set properties

The values of the placeholders can be specified in a myriad of ways:

• Properties:
• pre-defined variables defined by Maven and project-related properties (i.e. ${project.version})
• <property> elements in pom.xml, usually in conjunction with Build Profiles (we'll see that later on)
• environment variables
• values passed in the command line using the "-D" switch (for example, "-Dname=value")
• Filter Files

From Command Line ("-Dproperty=name")

The most basic way of specifying properties values without For example, with the application.properties above you can already invoke the build in a environment-specific way just by specifying all the required properties (the whole command must be in a single line):

mvn package "-Djdbc.driverClassName=com.mysql.jdbc.Driver" \
  "-Djdbc.url=jdbc:mysql://localhost:3306/maven-flexiconf" \
  "-Djdbc.username=root" \
  "-Djdbc.password=password"
  "-Dlog.path=C:/logs/my-application" \
  "-Dlog.rolled.path=C:/logs/my-application/rolled"

While this is a very annoying way of doing builds on a local developer machine, it is very well suited for Continuous Integration (CI) servers, as they can specify required properties very easily.

Filter Files

Filter files are regular property files containing the variable values that Maven will use to replace placeholders in resources in src/main/resources.

You must place filter files outside of src/main/resources. I usually put them in a /configuration folder, in directly underneath the project's root.

Let's create a configuration/default.properties file:

# JDBC Configuration

jdbc.driverClassName=com.mysql.jdbc.Driver
jdbc.url=jdbc:mysql://localhost:3306/maven-flexiconf
jdbc.username=root
jdbc.password=password

# Logging

log.path=C:/logs/my-application
log.rolled.path=C:/logs/my-application/rolled

To apply filtering on resources we must instruct pom.xml to do so:

<build>
    <resources>
        <resource>
            <directory>src/main/resources</directory>
            <filtering>true</filtering>
        </resource>
    </resources>

    <filters>
        <filter>configuration/default.properties</filter>
    </filters>
</build>

Build Profiles

Build Profiles let you customize the build process for a particular environment up to the finest detail. We will use profiles to define a custom variable named "env" that will drive how filter files will work in our build.

Putting it all together

Example Project

You can find a complete example project on GitHub:

maven-flexiconf

If you look at commit history you can also follow the evolution of the project.

File Structure

Here is how files are organized:

src
  \-- main
    \-- resources                   <== configuration files *with* placeholders
          +-- application.properties
          +-- logback.xml           <== a simple LogBack configuration that just logs to CONSOLE
    \-- resources-override          <== override configuration files (*with* placeholders) for specific profiles
          \-- local
                +-- logback.xml     <== local LogBack configuration that logs to local file system
          \-- prod
                +-- logback.xml     <== prod LogBack configuration with different loggers and appenders
  \-- configuration
        +-- default.properties      <== properties that don't change between environments
        +-- SAMPLE-local.properties <== template to use for the "local" filter file
        +-- prod.properties         <== prod properties
pom.xml
.gitignore

Here is pom.xml file:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>it.megadix</groupId>
    <artifactId>maven-flexiconf</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>maven-flexiconf</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-classic</artifactId>
            <version>1.2.3</version>
        </dependency>
    </dependencies>

    <build>
        <resources>
            <resource>
                <directory>src/main/resources</directory>
                <filtering>true</filtering>
            </resource>
            <resource>
                <directory>src/main/resources-override/${env}</directory>
                <filtering>true</filtering>
            </resource>
        </resources>

        <filters>
            <filter>configuration/default.properties</filter>
            <filter>configuration/${env}.properties</filter>
        </filters>
    </build>

    <profiles>
        <profile>
            <id>local</id>
            <properties>
                <env>local</env>
            </properties>
        </profile>
        <profile>
            <id>production</id>
            <properties>
                <env>production</env>
            </properties>
        </profile>
    </profiles>

</project>

Let's examine how everything works.

Developer on-boarding & Setup of "local" build profile

When a new developer joins the team he/she will have two possibilities:

1. setup a default environment: a MySql database containing a maven-flexiconf, accesible with user / password root / password
2. setup a local environment:
3. copy SAMPLE-local.properties to local.properties
4. modify properties according to his/her environment (JDBC URl, log paths, etc.)

Since local build profile is always active, Maven will know how to compose the application correctly without any further intervention.

Moreover, .gitignore contains a rule that excludes configuration/local.properties, so you can safely modify it and be sure it won't be pushed to Git repository.

Production build

You certainly don't want production database password to be stored in Git (I MEAN IT!), so to specify passwords and other secrets (encryption keys, etc.) you can pass them through the command line as seen above ("-Dproperty=value" switches).

Here's an example (notice the -Pprod switch):

mvn package -Pprod "-Djdbc.url=jdbc:mysql://prod.example.org:3306/maven-flexiconf" \
  "-Djdbc.username=prod-user" \
  "-Djdbc.password=prod-password"
  "-Dlog.path=/var/logs/my-application" \
  "-Dlog.rolled.path=/var/logs/my-application/rolled"