Hystrix is a perfect tool for handling communication with remote systems. One of the topics it covers for you is timeouts. But it might not be as straightforward as it may seem.
Before jumping to the main issue I want to discuss a little theory about how Hystrix works.
On which thread my code runs?
Hystrix will run your main method in two isolation modes. Semaphore or thread. If you select semaphore isolation the main method will run on the same thread as the caller. If you choose thread isolation it will run in separate thread. This is pretty straightforward. But on which thread the fallback method runs depends on the reason why the fallback was invoked. If your main method threw an exception it will run on the same thread. But if the main method failed because of the timeout it will run on the timer thread. This is important to bare in mind. In consequence anything that is thread bound is not guaranteed to be available in fallback method.
Default value is Thread isolation
How timeouts work?
Hystrix supports timeouts.
They are implemented by timeout listener which are triggered after time set by clients in timeout properties. One of these listeners is registered in HystrixObservableTimeoutOperator. This operator emits
HystrixTimeoutException when this timer is called before the actual method returns. Interrupt is called on the thread which executed the main method only if isolation level is Thread. If isolation level is Semaphore interrupting is not an option because the method does not run on separate thread.
Interrupting the thread
And here comes the problem. Thread interruption in Java does not actualy interrupt the execution of the thread! It sets the interrupted flag on that thread. It is up to the implementation to handle this flag. Unfortunately most implementations in
java.net package does not respond to it. Any long running computation won’t probably respond to it.
Without interruption handling your main method will continue after the blocking call is completed even if fallback method was executed because of timeout!
Basic example can be found in this gist. Output of that example is
Main method runs in hystrix-Demo-1 Fallback method runs in HystrixTimer-1 Fallback because of HystrixTimeoutException Supplier returned on hystrix-Demo-1 Thread hystrix-Demo-1 is interrupted
This behavior might lead to unexpected bugs especialy when there is some state changing code in the main method after the blocking call.
How to fix it
Fixes can vary depending on type of the code that is being executed. But if canceling the operation immediately is not needed you can use
com.netflix.hystrix.AbstractCommand#isResponseFromFallback method to query if fallback was executed and throw an exception from the main method in that case.
Thread.currentThread.isInterrupted() will be true only in case of thread isolation thus cannot be used both in Semaphore and Thread isolated commands.
In case of apache http client you can call
org.apache.http.client.methods.AbstractExecutionAwareRequest#abort which will cause the
org.apache.http.client.HttpClient#execute(org.apache.http.client.methods.HttpUriRequest) method to throw an
SocketException and to close the current connection. In other situations you have to handle the situations yourself.
If you are using
hystrix-javanica project be aware of this bug.
Null is passed instead of timeout exception to the fallback method.