Spring log only to file

Logging is one of the most fuzzy area in Java. The standard JUL, java.util.logging, entered the arena late, and it has to compete against well respected libraries like Log4J2, SLF4J and Logback (usually SLF4J with Logback). The Spring guys decided to go for JCL, the Apache Commons Logging, that wraps SLF4J for the actual logging library of your choice, here being Logback.

If you don't have any special requirement, you can happily ignore which actual logger is used, just write your code for the JCL interface, leaving out of your scope any low level dependency.

However, if you want your log going to a file, and not to console, as often is the case, you have to deal with the actual logger. Not a big deal, if Logback is your choice.

Let's modify the function greeting in my GreetingController to log some (un)useful comments:

public String greeting() {
 log.trace("trace hello");
 log.debug("debug hello");
 log.info("info hello");
 log.warn("warn hello");
 log.error("error hello");
 log.fatal("fatal hello");
 return "Hello!";
}

Where log is private static final object of type org.apache.commons.logging.Log initialized through the JCL LogFactory.

This could be enough. Still you should have a mildly surprising output, something like:

2016-05-30 22:14:00.888  INFO (...)  : info hello
2016-05-30 22:14:00.888  WARN (...)  : warn hello
2016-05-30 22:14:00.888 ERROR (...)  : error hello
2016-05-30 22:14:00.888 ERROR (...)  : fatal hello

I edited out details in the middle of the lines, I want to focus on the fact that we miss trace and debug messages, and the fatal one became an error one. If you really want fatal log messages, logback is not your choice, since it does not have this log level, and so they are mapped as simple errors.

The first problem could be easily solved adding an entry in the Spring application.properties file (in source/main/resources). Say that I want to log all messages, from trace up to fatal, generated in my packages rooted in dd.manny. I'll add this line:

logging.level.dd.manny=trace

Good. Now I want Spring to log to a file. By default the file will have name spring.log, and I can decided in which folder to be placed like this:

logging.path=/tmp

Nice and easy. Just one thing. I wanted the log to go exclusively to file. To get this effect I have to configure the actual logger.

For this reason I added a logback configuration file in the src/main/resources folder that is mimic of the default Spring one, but it has no appender for console. The key point is that I keep the log level to INFO and I specify FILE as appender, that is going to be set through the property specified above.

The full Spring Boot project is on github. The relevant files are GreetingController.java, application.properties, and logback.xml.

Go to the full post

Learning Apache Maven 3

I have just finished watching to this Pack video course about Maven, on youtube there is a preview that shows what you can expect from it.

It is designed to follow a Java programmer who knows nothing about Maven from the absolute beginning to writing a multi module project.

The major emphasis is on Maven for Windows plus Eclipse, just at the beginning it is shown how to install Maven on Linux and Mac, and how to integrate it also in Intellij and NetBeans. This is not a big issue, since we are in the Java world, and the platform differences are usually not too harsh.

The course is structured in four parts. After the installation/integration done in the introduction, we are guided to write a first "hello world" application. The third block is about creating a Web App by Maven that uses features from Struts2, Hibernate, and Spring. The last part shows how to develop a client/sever multi module project.

I found the standard price for this course a bit excessive, but in this bargains season, at less than five bucks, I wouldn't see how to complain.

It works fine as an introduction to the matter, in a couple of hours you can't expect to become a Maven master, but I'd say it succeeds in giving a good overview.

It has a few weak spots, too. Mainly, the audio comment is not lively at all, and there is a curious alternation in voices that I found distracting. Hear this, for instance, at around 1.30:

Go to the full post

Maven 3 video course

I have got a pointer to this video course on Apache Maven 3, you could also find a preview on YouTube that gives the gist of it.

It is a couple of hours long, it looks to be designed as an introduction to Maven for a Java developer who has no (or little) previous knowledge of it.

Since I use Maven in an unstructured way, it would probably good for me to find the way to spend time watching it. I plan to do it in the near future, and writing something more about it.

Go to the full post

Introduction to ActiveMQ

Sort of mouthful of a title for this book, Instant Apache ActiveMQ messaging application development how-to, but don't let this taking you aback. It is a slim (sixty-sh pages) tutorial on ActiveMQ written by Timothy Bish, who actually is an active contributor to that Apache project.

You can't expect it to go too deep in the matter, after all it's in a series that has as headline "Learn in an instant, short, fast, focused", still it I think it is good as a first tutorial. It refers ActiveMQ version 5.8 (the latest release, currently), and it is thought for a Java developer who needs to get introduced to this message broker. You'd better to have some experience on Java, but you'd probably get the most of it even if you don't know much about MOM (Message-Oriented Middleware) and in particular about JMS, the Java EE messaging standard defined by the JSR 914 specification and implemented by ActiveMQ.

After installing ActiveMQ and setting up a working environment (no IDE is used, we are showed instead how to do it "by hand", via Maven), it is showed how to create a first simple application that sends and receive a message. Then we are introduced to JMS queues, topics, selectors. All this stuff is marked as "simple", and it is followed by a chapter where we are introduced to the request-response pattern (the only part of the book that is considered "intermediate").

The third part of the book consists in half a dozen "advanced" chapters where we can read about scheduling, monitoring, testing, pooling connections, using virtual destinations and failover transport.

Lot of examples help to make clearer the concepts explained by Bish in a plain and readable prose. There is no room to enter in much details, still many "There's more" sections offer pointers to find more information in the net.

Go to the full post

Hello Maven

As a primer on Maven, there is nothing better than the Five Minutes Getting Started Page on its Apache site. Here you could find just a few extra notes I jotted down when I was reading it.

Once you have downloaded and unzipped the Maven package on your machine, you could run this basic command:

mvn --version

to ensure it is up and ready.

The most common issues you could have at this time are:

• you can't see Maven from your current directory
• Maven can't see Java

Ensure that the environment variables PATH and JAVA_HOME are properly set to avoid this problems. PATH should include also the Maven bin folder, and JAVA_HOME should refer to your preferred base JDK directory (notice that JDK is required, JRE is not enough).

Create a project

You can create a Java project from scratch running this Maven command (it is a single line, even if I split it to make it more readable):

mvn archetype:generate -DgroupId=package.name -DartifactId=app.name
-DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

I have used "helloMaven" as artifact id, and "hello" as group id. The artifact id is the root directory where all the stuff related to this you application will be stored, and the group id is the name of a package that would be created, and where a simple hallo App would be created by Maven.

Proxy issue?

Maven tries to connect to its remote repository to get what it needs and couldn't be found locally (or for a newer version of it). If you get an error here, it is often due to a proxy issue.

If you do have such problem, you should have a look at the settings.xml file in the conf directory in you Maven installation.

In the "proxies" section you should create a "proxy" block containing all the information required to go through your proxy. In its minimal form it looks something like this:

<proxy>
  <protocol>http</protocol>
  <host>your.proxy</host>
  <port>80</port>
</proxy>

When Maven works fine, you get a directory structure based in the specified artifactId, that would contain in its first level a src folder and a pom.xml

The Project Object Model (if you wonder what pom stands for) XML that I got is:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>hello</groupId>
  <artifactId>helloMaven</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>helloMaven</name>
  <url>http://maven.apache.org</url>
  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
</project>

Project build

To build our project we run:

mvn package

The result is published in the "target" folder. And, if we had no error, we can run our application in this way:

java -cp target/helloMaven-1.0-SNAPSHOT.jar hello.App

Notice that the JAR name is built putting together the artifact id and the version field, as stored in the POM.

Go to the full post

JMS Request-Reply

Implementing the Request-Reply messaging pattern with ActiveMQ is not complicated. The client component sends a request to the server on the dedicated queue, it attaches to it, as properties, the destination queue where it expects to find the reply, and a correlation id to uniquely identify the request-reply couple. The server uses these information to generate a consistent reply.

The complete Java source code is on github, here I comment just what I think are the most interesting part of the code.

This it the client, after establishing a connection to the broker:

Destination destination = session.createQueue(REQUEST_QUEUE_NAME); // 1
MessageProducer producer = session.createProducer(destination);
Destination dRep = session.createTemporaryQueue(); // 2
TextMessage message = session.createTextMessage(arg);
message.setJMSCorrelationID(UUID.randomUUID().toString()); // 3
message.setJMSReplyTo(dRep); // 4
producer.send(message);
// ...
MessageConsumer consumer = session.createConsumer(dRep); // 5
Message reply = consumer.receive(); // 6
// ...

1. The queue where the requests are sent.
2. A second queue, for the replies. We could have used a "normal" queue, but it is more common to create a cheaper temporary queue for this task.
3. We could keep track by hand of the unique message ids generated by the application, or we could delegate to Java the generation of an unique id, as I did here.
4. The queue to be used as destination for the reply is stored in the JMSReplyTo message property.
5. A consumer is created on the reply queue.
6. And the client patiently waits for an answer from the server.

The interesting part of the server is where it replies to the client:

if(message instanceof TextMessage) {
    TextMessage answer = session.createTextMessage(); // 1
    answer.setText("Reply to " + ((TextMessage) message).getText());
    answer.setJMSCorrelationID(message.getJMSCorrelationID()); // 2
    MessageProducer producer = session.createProducer(null); // 3
    producer.send(message.getJMSReplyTo(), answer);
}

1. The message to be sent as answer.
2. The correlation id is sent back to the client.
3. A producer with no associated queue is created, we are going to explicitly set the destination in its send() method using the JMSReplyTo message property.

Go to the full post

Embedding a broker by BrokerService

Instead of having the ActiveMQ broker as a standalone process, we could decide to have it embedded in one of our Java processes. The other processes would access it in the same way as before, but from other threads within, we could send and receive messages to the broker using the faster internal vm protocol.

A possible advantage of this solution is that the broker could be programmatically configured before it starts, and we could have a logic to determine when to stop it:

BrokerService broker = new BrokerService();
broker.setBrokerName("myBroker");
broker.setDataDirectory("data/"); // 1
// ...
broker.addConnector("tcp://localhost:61616"); // 2
broker.start(); // 3
// ...
broker.stop(); // 4
// ...

1. For a default setup, in the data directory it is created a folder for the broker (using its name, in this case we set it to "myBroker") containing a folder where the KahaDB files for message persistence are stored.
2. Let's our broker to be mimic of a standard broker.
3. When the broker setup is completed, we start it.
4. Shutting down the broker.

The complete Java source code for this example is on github.

Go to the full post

Selecting messages by custom properties

A JMS message comes with a number of standard properties, providing information on the associated message. For example, each message comes with a message id that could be fetched calling getJMSMessageID(). Besides, we can use custom properties to enrich a message in any way we think make sense for our specific case. An interesting aspect of custom properties is that we can use them to discriminate on which message a consumer should receive.

Think to an application where the producer generates messages for the best offers we have in our hardware store. The consumer could be interested in all or only in a subset of them. For this reason, it could be useful to put in the message payload just a generic description, and use custom properties to store structured information, as the product name and its price.

In this way the consumer could easily specify if it wants to get all the messages, or create a selector on the custom properties. The rules to create a selector are coming by simplification from the SQL conditional expressions.

The complete Java source code of this example is on github, it is written for ActiveMQ, so you should have this MOM installed and working on your development environment.

The producer delegates the most interesting part of its job to the send() method, that creates a message, fills it with specific data, and then send it to its associated queue:

TextMessage message = session.createTextMessage();
message.setText(txt);
message.setStringProperty(PROP_DES, des); // 1
message.setDoubleProperty(PROP_PRICE, price); // 2
producer.send(message);

1. Create a String property named as defined in the first parameter ("Description"), and containing the value stored in the second one ("Pins" and "Nails" in my test example).
2. The PROP_PRICE (actually, "Price") property is of type double.
In this test it is handy using a MessageListener, as discussed in a previous post, so that all the relevant messages are consumed by its onMessage() method:

logger.info("{}: {}", PROP_DES, message.getStringProperty(PROP_DES)); // 1
logger.info("{}: {}", PROP_PRICE, message.getDoubleProperty(PROP_PRICE)); // 2
logger.info("Message id: {}", message.getJMSMessageID()); // 3
logger.info("Message: {} ", ((TextMessage) message).getText());

1. The PROP_DES property is retrieved from the message, as the String that it is.
2. PROP_PRICE is a double, but we could have managed it as a String, calling getStringProperty(). ActiveMQ knows how to convert the internal double value to a String, so this cast works fine. If we tried to do the other way round, extracting a double from (1), we would have got a NumberFormatException.
3. There is not much use for this line here, but it is just to show a standard JMS property at work.

We need just one fundamental step, telling the JMS Session that the consumer has to get only some message:

switch (arg) {
    case "Nails":
    case "Pins":
        filter = PROP_DES + " = '" + arg + "'"; // 1
        consumer = session.createConsumer(destination, filter);
        break;
    case "Cheap":
        filter = PROP_PRICE + " < 15"; // 2
        consumer = session.createConsumer(destination, filter);
        break;
    default:
        consumer = session.createConsumer(destination); // 3
        break;
}

1. The string "Description = Nails" (or "Description = Pins") is passed in the next line to the session as a message selector for the consumer that has to be created. Only the messages true under that condition are delivered to this consumer.
2. Same as above, but the condition is now "Price < 15".
3. A "normal" consumer is created, with no associated selector.

Go to the full post

Asynchronous Hello ActiveMQ consumer

It is simple to modify the hello ActiveMQ example to change the consumer from syncronous to asyncronous.

If we want to asynchronously consume a message, we have to create a class that implements the JMS MessageListener interface, something like:

private class MyMessageListener implements MessageListener {
    @Override
    public void onMessage(Message message) {
        if(message instanceof TextMessage) {
            try {
                logger.info("Received {} ", ((TextMessage) message).getText());
            } catch (JMSException e) {
                logger.error("Can't extract text from received message", e);
            }
        }
        else
            logger.info("Unexpected non-text message received.");
    }
}

Then the consumer code changes only slightly:

MessageConsumer consumer = session.createConsumer(destination);
consumer.setMessageListener(new MyMessageListener()); // 1
try{ Thread.sleep(1000); } catch(InterruptedException e) {} // 2

1. Associate to the consumer an instance of our listener.
2. Usually we should implement a more sophisticated way of keeping alive the consumer till all the relevant messages are consumed. But for this simple example sleeping for a second should be enough.

The Java source code for the hello producer-consumer example, both in the synchronous and asynchronous flavor, is on github.

Go to the full post

Hello ActiveMQ

Writing an hello application with ZeroMQ is very easy, RabbitMQ requires a bigger effort, and ActiveMQ is not complicated, but probably the less immediate in the company. This escalation in complication is reflected also in the system's responsiveness (ZeroMQ is blazing fast and, compared to it, ActiveMQ could seem slow) but is well repaid by a higher level of offered services.

It is impossible to say in abstract which Message Queue structure is "better". It is more a matter of selecting the adequate one for specific environment requirements. A reason to decide for ActiveMQ is often that it implements the JMS specification.

In any case, say that we have already decided, and we have installed ActiveMQ on on our machine. Now it is time to write an hello application.

ActiveMQ is bundle with SLF4J as logger. I have talked of the trickiness in their relation in a previous post, so let's assume this is not an issue anymore. If you don't even know what SLF4J is, you could have a look at another post where I have written some notes on its installation.

Producer

The following code is all you need to establish a connection to the ActiveMQ broker and put a text message on a queue:

ConnectionFactory factory = new ActiveMQConnectionFactory(); // 1
Connection connection = null;
try {
    connection = factory.createConnection(); // 2
    connection.start();
    Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE); // 3
    Destination destination = session.createQueue(QUEUE_NAME); // 4
    MessageProducer producer = session.createProducer(destination); // 5
    TextMessage message = session.createTextMessage(arg); // 6
    producer.send(message); // 7
    logger.info("Sending: {}", message.getText()); // 8
} catch(JMSException ex) {
    ex.printStackTrace();
} finally { try { if(connection != null) connection.close(); } catch (JMSException e) {} } // 9

1. Since no address is specified, ActiveMQ assumes we want to connect to default broker, on localhost:61616.
2. A new connection to the browser is created and, in the next line, started.
3. On a connection we create a session, that could be transactional. Here we don't need it, so we pass "false" as first parameter. The second parameter specify if we want to acknowledge explicitly the message consume or not. Here we let ActiveMQ to take care of this.
4. Where the message is going to be sent. If the queue specified does not exist, ActiveMQ creates it for us.
5. The producer is an object created by the session for a specified destination that takes care of the message move from this client to the broker.
6. A text message is created from a String (in this case "arg" is the variable containing it).
7. The producer sends the message.
8. Remember that we are using SLF4J as a logger, if you wander about the strange syntax in the string, you may be interested in the post I have written on the efficiency reason for it.
9. Whatever happens in the try block, we want the connection to be closed.

Consumer

Consuming a message on ActiveMQ is not much different than producing it:

MessageConsumer consumer = session.createConsumer(destination); // 1
Message message = consumer.receive(1000); // 2
if(message == null)
    logger.info("No pending message on {} queue.", QUEUE_NAME);
else if(message instanceof TextMessage) // 3
    logger.info("Received: {}", ((TextMessage)message).getText());
else // 4
    logger.info("Unexpected non-text message consumed.");

1. First difference, we create a consumer on the session.
2. A message is consumed. The passed parameter is a timeout. If no message is waiting for us on the queue, and nothing arrives in a second, the consumer returns a null.
3. In the producer we created a text message, so we expect it to be of the same type here. If this is what we have, we work with it.
4. If we get here we are perplexed: who put a non-textual message on our queue?

You you want to play with the complete source Java code, you can find it on github.

Go to the full post

Installing ActiveMQ

What you need to run ActiveMQ on your machine is a recent Java development kit, and ActiveMQ itself. You can find the latter in the official Apache ActiveMQ download page, and you can get more information on the process in the related Getting Started page.

The ActiveMQ broker is ready to run out of the box, on its default port 61616, simply running a batch file in its binary folder. In my case, I have installed ActiveMQ on Windows XP (I know, we are in 2012, but this ancient operating system is still alive and kicking) in a folder named C:\dev\apache-activemq-5.5.1, and I run its broker executing from a shell in that folder the command

bin\activemq

As a feedback I get some log information including a notification of the JVM used, and on the started ActiveMQ components and subsystems. For the sake of testing the installation, the two most interesting lines are:

INFO | Listening for connections at: (your machine):61616
...
INFO | ActiveMQ Console at http://0.0.0.0:8161/admin

Whenever you want to check if the connection is still alive, you can run netstat, as I did here:

netstat -na|find "61616"
 TCP    0.0.0.0:61616          0.0.0.0:0              LISTENING

Meaning: the port 61616 on localhost is listening for a TCP connection. All is running as expected.

Besides, we can access the ActiveMQ console from our favorite browser at the http://localhost:8161/admin address, and see there a number of administrative information on the broker.

Go to the full post

ActiveMQ 5.5 and SLF4J 1.6

Working on a simple Hello World example for ActiveMQ, I bumped in a conflict between this Apache Message-Oriented-Middleware (MOM) and the SLF4J logging system. It is nothing serious, but I guess it is worth to enter in some detail on this issue.

SLF4J is the used as logger by ActiveMQ. Unfortunately, still in version 5.5, the SLF4J bundled version is 1.5, that's a pity, because starting from SLF4J version 1.6 if you place no binder in your application classpath, the NOP logger is assumed. The effect is that all your log goes to a logical respective of /dev/null, as to say, it disappears in thin air. That's no fun, but better than the behavior of 1.5: crashing miserably with an error like this:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
 further details.
Exception in thread "main" java.lang.NoClassDefFoundError: 
    org/slf4j/impl/StaticLoggerBinder
        at org.slf4j.LoggerFactory.getSingleton(LoggerFactory.java:230)
...

So, you really have to plug in a binder. I already had at hand a 1.6 plugin, I tried to used it, but I got another error:

SLF4J: The requested version 1.6 by your slf4j binding is not compatible
 with [1.5.5, 1.5.6, 1.5.7, 1.5.8, 1.5.9, 1.5.10, 1.5.11]

No compatibily.

There are a couple of solution to this issue. Or you use a plugin for that specific version, or you put the SLF4J API jar of your choice in your classpath before the ActiveMQ full jar.

I opted for the second choice, and my classpath is now referring to these jar - in this order:

• slf4j-api-1.6.4.jar
• activemq-all-5.5.1.jar
• slf4j-simple-1.6.4.jar

Actually, the SLF4J plugin could be in any place, but it is crucial that the API SLF4J jar is before the ActiveMQ one.

Go to the full post

Solving systems of linear equations

In the Java Apache Commons Math library there is a package, linear, that is about Linear Algebra. There we could find support to manage systems of linear equation.

A very simple linear system, as shown in the above quoted wikipedia page, features two equations and two variables:

2x + 3y = 6
4x + 9y = 15

Let's solve it through Apache:

RealMatrix coeffs = new Array2DRowRealMatrix(
        new double[][] {{2, 3}, {4, 9}}, false ); // 1
DecompositionSolver ds = new
        LUDecompositionImpl(coeffs).getSolver(); // 2

RealVector consts = new ArrayRealVector(
        new double[] {6, 15}, false ); // 3
RealVector sol = ds.solve(consts); // 4
System.out.println(sol);

1. We create an Array2DRowRealMatrix object passing to it our coefficients as a bidimensional array of double. The second parameter, set to false, says to the ctor not to bother making a local copy of the array, but use it directly. We won't care of the actual matrix type, so we store the result in the interface RealMatrix, root of a hierarchy of matrices including also an implementation for sparse matrix (OpenMapRealMatrix).
2. We should decide which algorithm use to solve the system. Here the chosen one is the LU decomposition. See the Apache documentation for the available alternatives (Cholesky, QR ...). Notice that here we need just to get a solver from the decompositor, so it is created and then forgotten at the end of the same line. But its solver survives and it is kept as a DecompositionSolver interface.
3. Very similarly to (1), we create an ArrayRealVector object for the constants, without copying the double raw array created on the fly, and keep it as a RealVector interface.
4. Calling DecompositionSolver.solve() for the constants vector we get a vector containing the solution. If there is no solution we get an exception instead. So it would be better to try/catch all this code (also the Array2DRowRealMatrix ctor throws unchecked exceptions) to avoid unpleasant surprises. It is easy to make this code crashing. Just change the coefficients and constants so that the system has no solution - just think to the geometrical interpretation of this system, remember that there is no intersection between parallel lines, and try to solve a system where:

RealMatrix coeffs = new Array2DRowRealMatrix(
        new double[][] {{2, 3}, {4, 6}}, false );
RealVector consts = new ArrayRealVector(
        new double[] {6, 12}, false );

You will get instead of a solution a SingularMatrixException.

We can extract from a RealVector the underlying raw double array:

double[] dv = sol.getData();
for(double d : dv)
    System.out.print(d + " ");
System.out.println();

But we can even get rid of all the RealVector objects above, and use directly just raw arrays:

// ...
double[] consts = new double[] {6, 15};
double[] sol = ds.solve(consts);

for(double d : sol)
    System.out.print(d + " ");
System.out.println();

Go to the full post

More random generators

If we need a sequence of pseudo-random numbers in Java we could use the java.util.Random class. For simple requirements this would suffice, but when the game gets tougher it should be worthy have a look at the generators provided by the Apache Commons Math library.

If we want to fill up an array of integer with a random sequence of one digit numbers, we could write:

Random rg = new Random();
rg.setSeed(System.currentTimeMillis()); // 1

for (int i = 0; i < ia.length; ++i) // 2
    ia[i] = rg.nextInt(10); // 3

1. The generator is initialized using the current timestamp. 2. "ia" is an array of integer, supposed to be properly initialized. 3. Each element in the array is set to a random number extracted from the interval [0,9]. For testing purpose it is useful to generate always the same sequence, to do that we can rewrite (1) to set the seed to a constant value:

rg.setSeed(42);

Apache Commons Math makes available a wrapper class that makes handier working with the standard random generator, letting us to easily change the generator to more advanced ones - provided by the library itself or created by ourselves.

RandomData rd = new RandomDataImpl(); // 1
for (int i = 0; i < ia.length; ++i) {
    ia[i] = rd.nextInt(0, 9); // 2

1. The current time is used to seed the generator, if we want to force using a specific seed we should call RandomDataImpl.reSeed(). Unfortunately this method is not part of the RandomData interface, so if we want this functionality be available we are forced to work with the concrete type. 2. Here nextInt() is more flexible, we specify both limit of the interval of values generated (besides, here the last one is included, in the java.util.Random it is excluded). Moreover, this method throws an exception if the second parameter is not bigger than the first one. One last notation, there is no check on the positiveness of the passed parameters (the standard Random throws a IllegalArgumentException if we pass a negative value), but the algorithm is not expected to works correctly for negative values. If we write the code using the Apache wrapper class, we can easily swap the random generator, using an overload of the constructor. So, to use the generator based on the Mersenne Twister instead of the JDK standard, we just have to rewrite (1) in this way:

RandomData rd = new RandomDataImpl(new MersenneTwister());

The Mersenne twister is one of the handful of generator based on the WELL family provided out of the (Apache) box.

Go to the full post

OLSMultipleLinearRegression parabola

We know how to infer a first degree curve (i.e. a straight line) from a bunch of observations using the Ordinary Least Squares estimator provided in the Java Apache Commons Math package.

Things are getting a bit more complicated here, as we try to get as estimation a second degree curve (a parabola).

Assuming that "ols" is a previously defined OLSMultipleLinearRegression object, here is the code that sets its sample data and then extract the relative coefficent estimation:

int vars = 2; // 1
int obs = 3; // 2
double[] data = { 4, 1, 1, /**/ 8, 2, 4, /**/ 14, 3, 9, }; // 3

ols.newSampleData(data, obs, vars); // 4

double[] coe = ols.estimateRegressionParameters();
dumpEstimation(coe); // 5

1. The number of independent variables for a parabola should be 2.
2. As before, we should provide at least one observation more than the vars.
3. Here is the input data. First component is y, than we have x, and then x square. These observations are lazily taken calculating y = x^2 + x + 2, as you could have guessed, so we would expect to get back as coefficients values close to (2, 1, 1).
4. I didn't try/catch, assuming that the caller of this code would do that for me. Actually, all the exceptions thrown by this package are unchecked (derived from RuntimeException), so we are not forced to try/catch or declare them in the method signature - and we can simply accept the risk of a sudden death of our application.
5. We have seen this little testing function in the previous post, it works fine here too.

Does the flattened data array bother you? Maybe not in this so simple example, but in a more real scenario could be cumbersome to organize data accordingly to this model. It could be useful to place y's in a unidimensional array, and x's in a separate bidimensional one. There is a OLSMultipleLinearRegression.newSampleData() overload that works right in this way:

double[] ys = { 4, 8, 14 };
double[][] xs = new double[][] { {1, 1}, {2, 4}, {3, 9} };
ols.newSampleData(ys, xs);

double[] coe = ols.estimateRegressionParameters();
dumpEstimation(coe);

This piece of code should be equivalent to what we have seen above.

Go to the full post

The simplest OLSMultipleLinearRegression example ever

Assuming that the reader knows what Multivariate Linear Regression by Ordinary Least Squares is, and that he would like to use it in Java by means of OLSMultipleLinearRegression, class that is part of the Apache Commons Math library, let's see the simplest example of its usage I could think of.

I have two observations, one variable, and I want to get the description of an interpolating curve:

OLSMultipleLinearRegression ols = new OLSMultipleLinearRegression();

double[] data = { 2, 1, 4, 2 }; // 1
int obs = 2;
int vars = 1; // 2
try {
    ols.newSampleData(data, obs, vars); // 3
}
catch(IllegalArgumentException e) {
    System.out.print("Can't sample data: ");
    e.printStackTrace();
    return;
}

double[] coe = null;
try {
    coe = ols.estimateRegressionParameters(); // 4
}
catch(IllegalArgumentException | InvalidMatrixException e) { // 5
    System.out.print("Can't estimate parameters: ");
    e.printStackTrace();
    return;
}

dumpEstimation(coe);

1. The input data is flattened in an array, where all the observations are stored one after the other, respecting the convention of having first the y and then the x component.
2. We should have more observations than variables, so this is the minimal case. Having just one variable, we'll get a straight line as a result.
3. The functions performs a few check on the passed data before working with them, if the number of variables is not bigger than the number of observations, or if there are less values than expected in the array, an IllegalArgumentException is thrown.
4. The resulting curve parameters are returned in a double array. In case of failure an exception is thrown.
5. Cool Java 7 feature, we can group all the exceptions that requires the same management in a single block. In previous Java version, a specific catch is required for each exception type.

The last line is a call to a testing function that gives some feedback to the user. It calls another short function that actually calculates the estimated y value given a specific x:

private void dumpEstimation(double[] coe) {
    if(coe == null)
        return;

    for(double d : coe)
        System.out.print(d + " ");
    System.out.println();

    System.out.println("Estimations:");
    System.out.println("x = 1, y = " + calculateEstimation(1, coe));
    System.out.println("x = 2, y = " + calculateEstimation(2, coe));
    System.out.println("x = 3, y = " + calculateEstimation(3, coe));
    System.out.println("x = 4, y = " + calculateEstimation(4, coe));
}

private double calculateEstimation(double x, double[] coe) {
    double result = 0;
    for(int i = 0; i < coe.length; ++i)
        result += coe[i] * Math.pow(x, i); // 1
    return result;
}

1. The most interesting line in this testing code. It shows how the coefficients are stored in the array returned from OLSMultipleLinearRegression.estimateRegressionParameters(). As we see, the coefficient 'i' is relative to the 'i'-th power of x.

The expected output is:

-0.0 2.0 
Estimations:
x = 1, y = 2.0
x = 2, y = 4.0
x = 3, y = 6.0
x = 4, y = 8.0

Go to the full post

SummaryStatistics vs. DescriptiveStatistics

Apache Commons Math makes available the classes DescriptiveStatistics and SummaryStatistics, both derived from the interface StatisticalSummary (there are a few more classes in the same hierarchy, that are not in this post spotlight). Both of them are used to get basic univariate statistics, but there are a few reason that should help us to decide which one is the most suitable for a specific task.

Let's start talking about commonality. Both of them are not synchronized, if you really need a thread-safe implementation, check out for SynchronizedDescriptiveStatistics and SynchronizedSummaryStatistics, and both of them implement all the basic functionality defined in StatisticalSummary.

It is a handful of methods that should look pretty straightforward to the reader having just some basic statistics knowledge:

double getMean(); // arithmetic mean
double getVariance();
double getStandardDeviation();
double getMax();
double getMin();
long getN(); // number of available values
double getSum(); // sum of the values

All this methods return NaN when called if no values have been added to the object, with the obvious exception of getN() that returns 0.

The substantial difference is that SummaryStatistics does not store data values in memory, resulting being sleeker and leaner than DescriptiveStatistics. On the other hand, DescriptiveStatistics makes available some more functionality to the user. So, if what you need is in StatisticalSummary, you can manage huge collection of data with SummaryStatistics and happily avoid to pay a large price in terms of memory usage.

There are then a few common methods that are defined for both SummaryStatistics and DescriptiveStatistics, even though they are not part of the commonly implemented interface StatisticalSummary.

To load the data we use public void addValue(double value), that could be called like this, where generator is a Random object previously initialized:

for(int i = 0; i < 1000; ++i) {
    stats.addValue(generator.nextDouble());
}

From object of both classes we can get the sum of the squares, getSumsq(), and the geometric mean, getGeometricMean(). Sometimes it is useful to reset the values on which we are working, and this is done by calling clear().

Only for SummaryStatistics are defined getSumOfLogs() and getSecondMoment().

Only for DescriptiveStatistics are available:

void removeMostRecentValue(): discards just the last value inserted in the underlying dataset or throws an exception.
double replaceMostRecentValue(double v): replaces the last inserted value or throws an exception.
double getSkewness(): the skewness is a measure of the current distribution asymmetry.
double getKurtosis(): the kurtosis is a measure of the current distribution protrusion.
double[] getValues(): creates a copy of the current data set.
double[] getSortedValues(): creates a sorted copy of the current data set.
double getElement(int index): gets a specific element or throws an exception.
double getPercentile(double p): an estimation of the requested percentile, or throws an exception.

Window size

When we have no idea of how many values could be entered, it could be dangerous using DescriptiveStatistics in its default mode, that let the underlying data collection growing without any limit. Better to define the dimension of the "window" we want to work with using setWindowSize(int windowSize). What happens when we reach the limit is that the oldest value is discarded to let room for the new entry. If you wonder what is the current size, you can check it through getWindowSize() that returns, as an int, its current value. The "no window" value is represented by DescriptiveStatistics.INFINITE_WINDOW, defined as -1.

Go to the full post