How to handle authentication failures from custom authentication handler - WSO2 API Manager

When we are implementing a custom Authentication Handler for wso2 API Manager we need to handle the case when authentication handler returns false. That mean authentication got failed and we need to send error back to client. Following is the sample code of the handleRequest method.

public boolean handleRequest(MessageContext messageContext) {
    try {
        if (authenticate(messageContext)) {
            return true;
        }
        else{
          //Your logic should go here.
        }
    } catch (APISecurityException e) {
        e.printStackTrace();
    }
    return false;
}

Ideally you need to call handleAuthFaliure method with message context.

private void handleAuthFailure(MessageContext messageContext, APISecurityException e) 

When you call that method please create APISecurityException object(as listed below) and pass it. Then error code and error messages will be picked from there and automatically send error to client(by default we return 401 for security exceptions and if defined then send defined error code and message).

public class APISecurityException extends Exception {
    private int errorCode;
    public APISecurityException(int errorCode, String message) {
        super(message);
        this.errorCode = errorCode;
    }
    public APISecurityException(int errorCode, String message, Throwable cause) {
        super(message, cause);
        this.errorCode = errorCode;
    }
    public int getErrorCode() {
        return errorCode;
    }
}

As you may already know you can generate error within your code and sent it back to client. But since API Manager have this capability you can easily use this without writing your own logic. And another advantage is if you pass error like this then it will go through auth_faliure_handler sequence and you can do any transformation there in a way it effect to all authentication failures.