This is the fifth part of the Playing With args4j series. For your convenience you can find other parts in the table of contents in Part 1 – Mixins

There is one more problem with our setters — they are not type safe. This means that we may have integer parameter:

@Option(name="--parameter", usage = "Some parameter")
default int parameter(){
    return get(0);
}

And then we can call this:

parameters.set(parameters::parameter, "1.23");

This compiles and crashes in runtime:

Exception in thread "main" java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer
	at IJobParameters.parameter(IJobParameters.java:16)
	at Program.run(Program.java:35)
	at Program.main(Program.java:24)

Can we do something about it? Currently our setter signature is:

default < T> void set(Supplier<T> getter, T value)

This means that the line parameters.set(parameters::parameter, "1.23") is actually treated as parameters.< Object>set(parameters::parameter3, "1.23") thanks to target typing and covariant return type in method references. See more in http://cr.openjdk.java.net/~briangoetz/lambda/lambda-state-final.html.

To fix that we need to stop covariance in some way. We can do this by enforcing the type T to be invariant by putting it in contravariant position. First trick is:

default < T> Consumer< T> set(Supplier< T> getter) {
    return value -> {
        Object target = getTarget(getter);
        Method method = MethodReferenceUtils.getReferencedMethod(getter, target);
        set(method.getAnnotation(Option.class).name(), value);
    };
}

Then we set the parameter in this way:

parameters.set(parameters::parameter).accept(123);

Because we return the Consumer< T> where T is in both covariant and contravariant position, the compiler cannot use Object as T and we are safe again. However, this is cumbersome as we need to use accept method.

There is yet another trick. We need to separate type parameters:

default < T, U extends Supplier< T>> void set(U getter, T value) {
    Object target = getTarget(getter);
    Method method = MethodReferenceUtils.getReferencedMethod(getter, target);
    set(method.getAnnotation(Option.class).name(), value);
}

Thanks to that we can still call:

parameters.set(parameters::parameter, 123);

And when we pass String we get:

Error:(34, 23) java: incompatible types: cannot infer type-variable(s) T,U
    (argument mismatch; bad return type in method reference
      int cannot be converted to java.lang.String)

So the final code is:

Base type for mixin:

import org.kohsuke.args4j.Option;

import java.lang.reflect.Method;
import java.util.Map;
import java.util.function.Supplier;

public interface ParametersMixin {
    Map<String, Object> parameters();

    default < T> T get(String name, T defaultValue){
        return (T)parameters().getOrDefault(name, defaultValue);
    }

    default < T> void set(String name, T value){
        parameters().put(name, value);
    }

    default < T> T get(T defaultValue){
        for(StackTraceElement e : Thread.currentThread().getStackTrace()){
            try {
                Option o = Class.forName(e.getClassName()).getMethod(e.getMethodName(), new Class[0]).getAnnotation(Option.class);
                if(o != null){
                    return get(o.name(), defaultValue);
                }
            } catch (NoSuchMethodException | ClassNotFoundException e1) {
            }
        }

        return defaultValue;
    }

    default < T, U extends Supplier< T>> void set(U getter, T value) {
        Method method = MethodReferenceUtils.getReferencedMethod(getter);
        set(method.getAnnotation(Option.class).name(), value);
    }
}

Actual interface with parameters:

import org.kohsuke.args4j.Option;

public interface IJobParameters extends ParametersMixin {
    @Option(name="--parameter", usage = "Some parameter")
    default String parameter(){
        return get("Some default value");
    }

    @Option(name="--parameter2", usage = "Some parameter 2")
    default String parameter2(){
        return get("Some default value 2");
    }

    @Option(name="--parameter3", usage = "Some parameter 3")
    default int parameter3(){
        return get(123);
    }
}

Concrete parameters implementation:

import java.util.HashMap;
import java.util.Map;

public class ParametersMixinImpl implements ParametersMixin{
    private Map< String, Object> parameters = new HashMap<>();

    @Override
    public Map<String, Object> parameters() {
        return parameters;
    }
}

public class JobParameters extends ParametersMixinImpl implements IJobParameters {}

Helper methods for accessing the lambda instance:

import net.bramp.unsafe.UnsafeHelper;
import net.sf.cglib.proxy.Enhancer;
import net.sf.cglib.proxy.MethodInterceptor;
import sun.misc.Unsafe;

import java.lang.reflect.Method;
import java.util.concurrent.atomic.AtomicReference;
import java.util.function.Supplier;

public class MethodReferenceUtils {
    @SuppressWarnings("unchecked")
    public static < T, U> Method getReferencedMethod(Supplier< T> getter) {
        Object target = MethodReferenceUtils.getTarget(getter);
        Class< ?> klass = target.getClass();
        AtomicReference<Method> ref = new AtomicReference<>();
        Enhancer enhancer = new Enhancer();

        enhancer.setSuperclass(klass);
        enhancer.setCallback((MethodInterceptor) (obj, method, args, proxy) -> {
            ref.set(method);
            return null;
        });
        try {
            setTarget(getter, enhancer.create());
            getter.get();
        } catch (ClassCastException e) {
            throw new IllegalArgumentException(String.format("Invalid method reference on class [%s]", klass));
        }

        Method method = ref.get();
        if (method == null) {
            throw new IllegalArgumentException(String.format("Invalid method reference on class [%s]", klass));
        }

        return method;
    }

    private static void setTarget(Object lambda, Object target){
        long targetAddress = UnsafeHelper.toAddress(target);
        long lambdaAddress = UnsafeHelper.toAddress(lambda);
        Unsafe unsafe = UnsafeHelper.getUnsafe();
        unsafe.putByte(lambdaAddress + 15, (byte) (targetAddress >> 24 & 0xFF));
        unsafe.putByte(lambdaAddress + 14, (byte) (targetAddress >> 16 & 0xFF));
        unsafe.putByte(lambdaAddress + 13, (byte) (targetAddress >> 8 & 0xFF));
        unsafe.putByte(lambdaAddress + 12, (byte) (targetAddress & 0xFF));
    }

    private static  Object getTarget(Object lambda){
        byte[] bytes = UnsafeHelper.toByteArray(lambda);
        int offset = bytes.length - 1;
        int address = bytes[offset] << 24 | (bytes[offset - 1] & 0xFF) << 16 | (bytes[offset - 2] & 0xFF) << 8 | (bytes[offset - 3] & 0xFF);
        return UnsafeHelper.fromAddress(address);
    }
}

Parser for custom setters:

private void parseInterfaces(Object bean, CmdLineParser parser){
    // args4j doesn't support methods on interfaces, we need to parse them manually
    Set< Class> interfaces = new HashSet< >();
    Deque< Class> classes = new ArrayDeque< >();
    classes.add(bean.getClass());

    while(!classes.isEmpty()){
        Class c = classes.getFirst();
        classes.removeFirst();
        if(c.isInterface()){
            interfaces.add(c);
        }

        if(c.getSuperclass() != null){
            classes.addLast(c.getSuperclass());
        }

        classes.addAll(Arrays.asList(c.getInterfaces()));
    }

    for(Class c : interfaces) {
        parseInterfaceMethods(c, bean, parser);
    }
}

private void parseInterfaceMethods(Class c, Object bean, CmdLineParser parser){
    for(Method m : c.getDeclaredMethods()){
        Option o = m.getAnnotation(Option.class);
        if(o != null){
            parser.addOption(new MixinSetter((ParametersMixin) bean, o.name(), m), o);
        }
    }
}

Custom setter:

import org.kohsuke.args4j.CmdLineException;
import org.kohsuke.args4j.spi.FieldSetter;
import org.kohsuke.args4j.spi.Setter;

import java.lang.reflect.AnnotatedElement;
import java.lang.reflect.Method;

public class MixinSetter implements Setter {
    private ParametersMixin bean;
    private String name;
    private Method method;

    public MixinSetter(ParametersMixin bean, String name, Method method) {
        this.bean = bean;
        this.name = name;
        this.method = method;
    }

    @Override
    public void addValue(Object o) throws CmdLineException {
        bean.set(name, o);
    }

    @Override
    public Class getType() {
        return this.method.getReturnType();
    }

    @Override
    public boolean isMultiValued() {
        return false;
    }

    @Override
    public FieldSetter asFieldSetter() {
        return null;
    }

    @Override
    public AnnotatedElement asAnnotatedElement() {
        return this.method;
    }
}

And finally, execution:

JobParameters parameters = new JobParameters();
CmdLineParser parser = new CmdLineParser(parameters);
parseInterfaces(parameters, parser);
parser.parseArgument(args);

System.out.println("parameter value from command line: " + parameters.parameter());
System.out.println("parameter2 default value: " + parameters.parameter2());

parameters.set(parameters::parameter3, 123);
System.out.println("parameter3 value set using setter: " + parameters.parameter3());

When you execute this as:

java Program --parameter "Value from command line"

You get the following output:

parameter value from command line: Value from command line
parameter2 default value: Some default value 2
parameter3 value set using setter: 123

This is the fourth part of the Playing With args4j series. For your convenience you can find other parts in the table of contents in Part 1 – Mixins

Last time we ended with the setter of a form

parameters.set(IJobParameters::parameter, "Value", JobParameters.class);

. What we don’t like here is the last parameters — why would we want to pass the class type if we pass the method reference? Can we extract it somehow?

In theory we cannot but with some help of unsafe methods we will be able to do so. Beware — this is not production ready, this is just for fun.

We will need the Unsafe library to make it a little easier. We could go with using sun.misc.unsafe directly but this library does that and gives nice interface so we can use it.

First, we need to change the lambda type:

default < T> void set(Supplier<T> getter, T value) {
    Object target = getTarget(getter);
    Method method = MethodReferenceUtils.getReferencedMethod(getter, target);
    set(method.getAnnotation(Option.class).name(), value);
}

So we get the Supplier which points to the method directly. Next, we get the target from the lambda. This is how we do it:

default Object getTarget(Object lambda){
    byte[] bytes = UnsafeHelper.toByteArray(lambda);
    int offset = bytes.length - 1;
    int address = bytes[offset] << 24 | (bytes[offset - 1] & 0xFF) << 16 | (bytes[offset - 2] & 0xFF) << 8 | (bytes[offset - 3] & 0xFF);
    return UnsafeHelper.fromAddress(address);
}

My JVM version running on Windows 10 Enterprise x64 is:

java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)

This explains why we get the 4 bytes pointer.

Next, we need to modify the cglib logic:

public static < T, U> Method getReferencedMethod(Supplier<T> getter, U target) {
    Class<?> klass = target.getClass();
    AtomicReference<Method> ref = new AtomicReference<>();
    Enhancer enhancer = new Enhancer();

    enhancer.setSuperclass(klass);
    enhancer.setCallback((MethodInterceptor) (obj, method, args, proxy) -> {
        ref.set(method);
        return null;
    });
    try {
        setTarget(getter, enhancer.create());
        getter.get();
    } catch (ClassCastException e) {
        throw new IllegalArgumentException(String.format("Invalid method reference on class [%s]", klass));
    }

    Method method = ref.get();
    if (method == null) {
        throw new IllegalArgumentException(String.format("Invalid method reference on class [%s]", klass));
    }

    return method;
}

So now we get the type by calling getClass and then we modify the target of the lambda. We use pointer arithmetic once again to do that:

static void setTarget(Object lambda, Object target){
    long targetAddress = UnsafeHelper.toAddress(target);
    long lambdaAddress = UnsafeHelper.toAddress(lambda);
    Unsafe unsafe = UnsafeHelper.getUnsafe();
    unsafe.putByte(lambdaAddress + 15, (byte) (targetAddress >> 24 & 0xFF));
    unsafe.putByte(lambdaAddress + 14, (byte) (targetAddress >> 16 & 0xFF));
    unsafe.putByte(lambdaAddress + 13, (byte) (targetAddress >> 8 & 0xFF));
    unsafe.putByte(lambdaAddress + 12, (byte) (targetAddress & 0xFF));
}

Thanks to that we can now write:

parameters.set(parameters::parameter, "Some value");

This is the third part of the Playing With args4j series. For your convenience you can find other parts in the table of contents in Part 1 – Mixins

Let’s see how we can implement the setters. Idea is as following: we will pass a lambda indicating the getter of the parameter. Next, we will create a dynamic proxy using cglib, call the getter (which should have no side effects) and then we will intercept the actual call to extract the getter method. Next, we will parse the annotation and call the setter.

First, we add the magical setter:

default < T, U> void set(Function<T, U> getter, U value, Class<T> klass) {
    Method method = MethodReferenceUtils.getReferencedMethod(getter, klass);
    set(method.getAnnotation(Option.class).name(), value);
}

We accept three parameters: lambda representing the getter, value to set, and a class representing the object. We extract the getter method, get its annotation and execute the normal setter.

Next, we implement the reflection magic:

import net.sf.cglib.proxy.Enhancer;
import net.sf.cglib.proxy.MethodInterceptor;
import net.sf.cglib.proxy.MethodProxy;

import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.util.concurrent.atomic.AtomicReference;
import java.util.function.Consumer;
import java.util.function.Function;
import java.util.function.Supplier;

public class MethodReferenceUtils {
    @SuppressWarnings("unchecked")
    public static <T, U> Method getReferencedMethod(Function<T, U> getter, Class<T> klass) {
        AtomicReference<Method> ref = new AtomicReference<>();
        Enhancer enhancer = new Enhancer();

        enhancer.setSuperclass(klass);
        enhancer.setCallback((MethodInterceptor) (obj, method, args, proxy) -> {
            ref.set(method);
            return null;
        });
        try {
            getter.apply((T) enhancer.create());
        } catch (ClassCastException e) {
            throw new IllegalArgumentException(String.format("Invalid method reference on class [%s]", klass));
        }

        Method method = ref.get();
        if (method == null) {
            throw new IllegalArgumentException(String.format("Invalid method reference on class [%s]", klass));
        }

        return method;
    }
}

We create an enhancer to capture the call. Next, we execute the lambda (which is a getter), capture it and store the reference. Finally, we return the method.

Now we can call it like this:

parameters.set(IJobParameters::parameter, "Value", JobParameters.class);

And this works like a charm.

This is the second part of the Playing With args4j series. For your convenience you can find other parts in the table of contents in Part 1 – Mixins

Last time we created mixins to be able to compose classes with multiple inherited parameters. We immediately noticed that we have a lot of code duplication because we need to specify parameter names in multiple places. In this part we will see how to cut the boilerplate.

We will implement custom Setter to automatically populate the fields. Here it is:

import org.kohsuke.args4j.CmdLineException;
import org.kohsuke.args4j.spi.FieldSetter;
import org.kohsuke.args4j.spi.Setter;

import java.lang.reflect.AnnotatedElement;
import java.lang.reflect.Method;

public class MixinSetter implements Setter {
    private ParametersMixin bean;
    private String name;
    private Method method;

    public MixinSetter(ParametersMixin bean, String name, Method method) {
        this.bean = bean;
        this.name = name;
        this.method = method;
    }

    @Override
    public void addValue(Object o) throws CmdLineException {
        bean.set(name, o);
    }

    @Override
    public Class getType() {
        return this.method.getReturnType();
    }

    @Override
    public boolean isMultiValued() {
        return false;
    }

    @Override
    public FieldSetter asFieldSetter() {
        return null;
    }

    @Override
    public AnnotatedElement asAnnotatedElement() {
        return this.method;
    }
}

In addValue we just call a bean method populating the dictionary. How do we get the name of the parameter? In this way:

private void parseInterfaceMethods(Class c, Object bean, CmdLineParser parser){
    for(Method m : c.getDeclaredMethods()){
        Option o = m.getAnnotation(Option.class);
        if(o != null){
            parser.addOption(new MixinSetter((ParametersMixin) bean, o.name(), m), o);
        }
    }
}

We just extract the name from the Option annotation.

Now we can simplify the parameters and remove redundant methods:

@Option(name="--parameter", usage = "Some parameter")
default String parameter(){
    return get("Some default value");
}

We still specify annotation with name and usage. We also need to specify a default value which we do by calling the method get which is:

default < T> T get(T defaultValue){
    for(StackTraceElement e : Thread.currentThread().getStackTrace()){
        try {

            Option o = Class.forName(e.getClassName()).getMethod(e.getMethodName(), new Class[0]).getAnnotation(Option.class);
            if(o != null){
                return get(o.name(), defaultValue);
            }
        } catch (NoSuchMethodException | ClassNotFoundException e1) {
        }
    }

    return defaultValue;
}

So we get the stack trace and look for the first method with the Option annotation (remember about inlining!). When we find it, we just extract the parameter value as usual.

So we still have a strongly typed parameter with getter, and setter is created automatically. So we can do that:

parameters.parameter()

And this returns the value of the parameter. However, to set the parameter, we need to do this:

parameters.set("--parameter", "Value");

So we need to have the name again. Can we do better? We will see in the next part.

This is the first part of the Playing With args4j series. For your convenience you can find other parts using the links below (or by guessing the address):
Part 1 — Mixins
Part 2 — Automatic getters
Part 3 — Nice setters
Part 4 — Nicer setters
Part 5 — Safe setters

Imagine that you are working on a command line tool and you need to parse arguments somehow. You can use pretty nice library called args4j.

So you write a class for your job parameters:

import org.kohsuke.args4j.CmdLineException;
import org.kohsuke.args4j.CmdLineParser;
import org.kohsuke.args4j.Option;

public class JobParameters {
    @Option(name="--parameter", usage = "Some parameter")
    private String parameter = "value1";
}

And then you parse it:

public class Job {
	public void run(){
            JobParameters parameters = new JobParameters();
            CmdLineParser parser = new CmdLineParser(parameters);
            parser.parseArgument(args);
	}
}

Looks great. Some time later you write another job with another parameter:

public class JobParameters2 {
    @Option(name="--parameter2", usage = "Some parameter2")
    private String parameter2 = "valu2";
}

Time goes by and you need to write another job which should have both of the parameters. What can you do? This is Java so you can’t use multiple inheritance of state. However, there are default interface implementations which we can use to simulate mixins!

Let’s start with this:

public interface ParametersMixin {
    Map<String, Object> parameters();

    default < T> T get(String name, T defaultValue){
        return (T)parameters().getOrDefault(name, defaultValue);
    }

    default < T> void set(String name, T value){
        parameters().put(name, value);
    }
}

We create an interface with methods for getting and setting parameters. We also have one abstract method providing collection. We could use field here but then it would be static so we wouldn’t be able to create multiple instances at the same time.

Now, let’s rework first parameter class:

public interface IJobParameters extends ParametersMixin {
    default String parameter(){
        return get("parameter", "value1");
    }

    @Option(name="--parameter", usage = "Some parameter")
    default void parameter(String parameter){
        set("parameter", parameter);
    }
}

The same goes for second parameter class:

public interface IJobParameters2 extends ParametersMixin {
    default String parameter2(){
        return get("parameter2", "value2");
    }

    @Option(name="--parameter2", usage = "Some parameter2")
    default void parameter2(String parameter2){
        set("parameter2", parameter2);
    }
}

Now we need to have an implementation for collection:

class ParametersMixinImpl implements ParametersMixin{
    private Map<String, Object> parameters = new HashMap<>();
    @Override
    public Map<String, Object> parameters() {
        return parameters;
    }
}

One more thing: we need to have concrete classes with parameters but this is trivial:

public class JobParameters extends ParametersMixinImpl implements IJobParameters {}
public class JobParameters2 extends ParametersMixinImpl implements IJobParameters2 {}

Awesome… but it doesn’t work. The reason is that args4j doesn’t support methods on interfaces — it just doesn’t parse them. Let’s fix that:

public void run(){
		JobParameters parameters = new JobParameters();
		CmdLineParser parser = new CmdLineParser(parameters);
		parseInterfaces(parameters, parser);
		parser.parseArgument(args);
}

private void parseInterfaces(Object bean, CmdLineParser parser){
	// args4j doesn't support methods on interfaces, we need to parse them manually
	Set<Class> interfaces = new HashSet<>();
	Deque<Class> classes = new ArrayDeque<>();
	classes.add(bean.getClass());

	while(!classes.isEmpty()){
		Class c = classes.getFirst();
		classes.removeFirst();
		if(c.isInterface()){
			interfaces.add(c);
		}

		if(c.getSuperclass() != null){
			classes.addLast(c.getSuperclass());
		}

		classes.addAll(Arrays.asList(c.getInterfaces()));
	}

	for(Class c : interfaces) {
		parseInterfaceMethods(c, bean, parser);
	}
}

private void parseInterfaceMethods(Class c, Object bean, CmdLineParser parser){
	for(Method m : c.getDeclaredMethods()){
		Option o = m.getAnnotation(Option.class);
		if(o != null){
			parser.addOption(new MethodSetter(parser, bean, m), o);
		}
	}
}

Yep, that was long but now we are ready to create class for third job:

public class JobParameters3 extends ParametersMixinImpl implements IJobParameters, IJobParameters2 {}

Done! I believe you can clearly see the advantages and that doing that in such a simple example is not very beneficial but can be great if you have multiple combinations of parameters for different jobs.

There is one terrible thing, though. Instead of having one field we need to have two methods now. We will fix that one day.