Application does not start with BlockHound and ReactorDebugAgent installed on java 8

[robotmrv]

BlockHound and ReactorDebugAgent cannot work together on java 8.
Application fails with error

java.util.ServiceConfigurationError: com.sun.tools.attach.spi.AttachProvider: Provider sun.tools.attach.WindowsAttachProvider could not be instantiated
java.lang.ExceptionInInitializerError
Caused by: java.lang.IllegalStateException: Error during attachment using: reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound@53aac487
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:400)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:374)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:342)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:328)
	at reactor.tools.agent.ReactorDebugAgent.init(ReactorDebugAgent.java:41)
	at com.example.demo.ReactorDebugApplication.<clinit>(ReactorDebugApplication.java:27)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at reactor.tools.shaded.net.bytebuddy.agent.Attacher.install(Attacher.java:99)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:395)
	... 5 more
Caused by: com.sun.tools.attach.AttachNotSupportedException: no providers installed
	at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:203)
	... 11 more
Exception in thread "main" 

If I try to start application using java 11 or just disable BlockHound or ReactorDebugAgent (in the code and spring-boot integration) everything works fine.

Expected Behavior

application starts

Actual Behavior

application start fails

Steps to Reproduce

try to start application on java 8
https://github.com/robotmrv/reactor-debug-error

Possible Solution

Your Environment

• Reactor version(s) used:
  io.projectreactor:reactor-core:3.3.0.RELEASE
  io.projectreactor:reactor-tools:3.3.0.RELEASE
  io.projectreactor.tools:blockhound:1.0.1.RELEASE
• Other relevant libraries versions (eg. netty, ...):
• JVM version (javar -version):
  java version "1.8.0_231"
  Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
  Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode)
• OS and version (eg uname -a):
  Win 10

[bsideup]

HI @robotmrv,

Thanks for reporting!
Have you tried different combinations, e.g. "only BlockHound" or "only reactor-tools?

Just trying to understand whether it fails only if both are installed.

[robotmrv]

Hi @bsideup,
If only one of BlockHound / ReactorDebugAgent is installed it works as expected

[bsideup]

@robotmrv does it happen with BlockHound 1.0.0.RELEASE?

[robotmrv]

@bsideup yes, the same with BlockHound 1.0.0.RELEASE

[bsideup]

@robotmrv thanks for verifying! 👍 on it...

[robotmrv]

Hi @bsideup
I've tested example application using linux based docker image openjdk:8

Linux c199ba7a5c0d 4.9.184-linuxkit #1 SMP Tue Jul 2 22:58:16 UTC 2019 x86_64 GNU/Linux
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)

and got similar result.

So it is not Windows specific

 09:37:42.177 [main] DEBUG reactor.util.Loggers$LoggerFactory - Using Slf4j logging framework
 Exception in thread "main" java.lang.ExceptionInInitializerError
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
      at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
      at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
      at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52)
 Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: Error during attachment using: reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound@311d617d
      at reactor.blockhound.BlockHound$Builder.install(BlockHound.java:328)
      at reactor.blockhound.BlockHound.install(BlockHound.java:94)
      at com.example.demo.ReactorDebugApplication.<clinit>(ReactorDebugApplication.java:22)
      ... 8 more
 Caused by: java.lang.IllegalStateException: Error during attachment using: reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound@311d617d
      at reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:608)
      at reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:581)
      at reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:533)
      at reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:510)
      at reactor.blockhound.BlockHound$Builder.install(BlockHound.java:309)
      ... 10 more
 Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at reactor.shaded.net.bytebuddy.agent.Attacher.install(Attacher.java:106)
      at reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:603)
      ... 14 more
 Caused by: java.lang.UnsatisfiedLinkError: Native Library /usr/local/openjdk-8/jre/lib/amd64/libattach.so already loaded in another classloader
      at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1900)
      at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1838)
      at java.lang.Runtime.loadLibrary0(Runtime.java:870)
      at java.lang.System.loadLibrary(System.java:1122)
      at sun.tools.attach.LinuxVirtualMachine.<clinit>(LinuxVirtualMachine.java:342)
      at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63)
      at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
      ... 20 more

[bsideup]

Caused by: java.lang.UnsatisfiedLinkError: Native Library /usr/local/openjdk-8/jre/lib/amd64/libattach.so already loaded in another classloader

Interesting...

/cc @raphw

[raphw]

Probably a bieffect of shading with the way attach works on Java 8 where tools.jar must be loaded into a new class loader. Byte Buddy checks if it already loaded tools.jar previously and reuses that attachment if possible but with the shading, I assume another, unshaded copy of Byte Buddy attempts the same what is impossible in Java 8 as the native code in tools.jar can only loaded once. What other tools you use load tools.jar? (Set a break point in com.sun.tools.attach.VirtualMachine.)

The solutions:

1. Use emulated attach (add JNA). It does not set any VM global state.
2. Avoid shading Byte Buddy if an unshaded copy is attempting the attach.
3. Upgrade to Java 9+ where this is fixed in the JVM.

[bsideup]

@raphw thank you for the explanation!

I just tried adding JNA but it still fails, this time with:

Caused by: java.lang.IllegalStateException: Error during attachment using: reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound@66498326
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:400)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:374)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:342)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:328)
	at reactor.tools.agent.ReactorDebugAgent.init(ReactorDebugAgent.java:41)
	at com.example.demo.ReactorDebugApplication.<clinit>(ReactorDebugApplication.java:22)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at reactor.tools.shaded.net.bytebuddy.agent.Attacher.install(Attacher.java:99)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:395)
	... 5 more
Caused by: java.lang.UnsatisfiedLinkError: Native Library /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre/lib/libattach.dylib already loaded in another classloader
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1907)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1845)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1122)
	at sun.tools.attach.BsdVirtualMachine.<clinit>(BsdVirtualMachine.java:307)
	at sun.tools.attach.BsdAttachProvider.attachVirtualMachine(BsdAttachProvider.java:63)
	at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
	... 11 more