Don't play catch-up with Spring – Explyt Spring plugin for IDEA Community
Introduction
Hi all. After my previous articles about the Maven plugin, where I proposed a new approach to implementation and created my own version for IDEA (instead of writing my own mini-Maven, I delegated all the main work to it through the Maven plugin), I was invited to work on Spring plugin at IT startup Explyt. The company is engaged in automatic generation of tests based on AI and formal methods. While working, I encountered problems similar to those that I solved in my Maven plugin. I had some déjà vu and thought: why not use a similar approach to refine and improve the Dependency Injection Explyt Spring plugin? A text for those who work with Spring plugins and want to figure out how to effectively use ready-made Spring logic for new tasks. Go to the cat, I’ll tell you in more detail about our plugin, the problems that it solved, and I’ll share some code examples.
Problem
One of the primary and main tasks of the Spring plugin is to support Dependency Injection, namely, you need to be able to understand whether a class is part of the context. In addition to general highlighting of beans and navigation through them (declaration/use), this is also often necessary when creating inspections, code “compliments,” etc. What problems could there be?
Let's look at examples. First, you need to determine the main packages of the application, where the search for beans begins. And if a string is used, then everything seems to be nothing.
@SpringBootApplication(scanBasePackages = "com.example.simple")
public class DemoApplication {
}What if we have some kind of non-standard expression or some kind of regular expression that needs to be calculated? In addition, the @ComponentScan annotation allows you to create very complex filters for packets with the ability to partially exclude and so on.
@SpringBootApplication(scanBasePackages = "com.*.spel")
public class DemoApplicationSpel {
}And this is not an easy task at all.
With @ annotationEnableJpaRepositories and similar problems similar to it. Starting from simple cases where the availability of repositories is enabled by a line with the package name, to complex expressions with custom logic.
In addition, whether a bean is in context or not is also affected by the @Conditional/@DependOn/@Profile annotations and many others. And if we have the @Conditional annotation with custom code, then this is equivalent to mission impossible.
@SpringBootApplication
public class DemoApplicationConditional {
@Autowired ConditionalBean bean;
public static void main(String[] args) {
SpringApplication.run(DemoApplicationConditional.class, args);
}
}
@Conditional(MyCondition.class)
@Component
class ConditionalBean {}
class MyCondition implements Condition {
@Override
public boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata) {
//same user logic
return false;
}
}To evaluate such an expression, we need to compile the code with all its dependencies and execute it. Having only data about the code model as input, it is almost impossible to do this based on its analysis. This doesn't even work in IDEA Ultimate (all comparisons are with Ultimate version 2022.2, when it was still possible)
Such examples can be given for a very long time. In fact, we cannot support all possible cases. And we act according to Pareto’s law, that 20% of efforts give 80% of results. This approach was already used at the time I joined the team, when everything was implemented based on the analysis of when, this is the most typical approach and it is used one way or another in all spring plugins, including Spring Ultimate.
To support all cases is, in fact, to write your own spring, only instead of java runtime with *.class files, in this case we have idea code model. And this is a very labor-intensive task, and even a lifetime is not enough to support all these possibilities. In fact, we write our spring on minimal wages.
While working, the thought constantly hovered in my head: why not try to reuse the ready-made logic from Spring for finding beans, rather than writing it myself. When playing catch-up with Spring, you always find yourself lagging behind. Since it is practically impossible to reproduce it one to one and it takes too many resources. Yes, and we are in different conditions. They have a full-fledged java runtime as input, our code model is source code. So errors and inaccuracies in reproducing spring logic are inevitable. That is, this is exactly the same thing that I tried to fix in my IDEA Maven plugin. Let's see what comes of it.
Idea
As you know, the Spring context initialization process, very roughly speaking, consists of two main steps:
building metadata – bean definition
creating bean instances – directly instances of the class
You can read more about this here And here.
Bean definition is a special spring structure that represents metadata about beans. All possible filters have already been applied to this metadata: @Conditional/@DependOn/@Profile, etc. At the next stage, instances of beans are created directly from them and additional validation occurs – whether the required bean is in the context, etc.
That is, after the first step, we already have all the necessary data about the available bins. The idea was the following: we need to somehow wedge ourselves into this process of raising the context and interrupt it after the first step.
We don’t need to initialize/create beans; database migration scripts (liquibase/flyway), @PostConstruct methods, and starting the web server can already start executing here. This can lead to many unwanted side effects (database changes or the port may be busy if the application is already running in the background). And besides, it is the process of creating beans that takes up most of the application startup time.
A small clarification: further we will talk only about Spring Boot projects, since entry points for launching a project are easily found for them, and according to the same Pareto law, this covers most of the current cases.
Implementation
Here So looks like the main part of creating a context in an abstract class AbstractApplicationContextfrom which all other application contexts are inherited (ServletWebServerApplicationContext, ReactiveWebServerApplicationContext, etc.)
public void refresh() throws BeansException, IllegalStateException {
synchronized (this.startupShutdownMonitor) {
StartupStep contextRefresh = this.applicationStartup.start("spring.context.refresh");
// Prepare this context for refreshing.
prepareRefresh();
// Tell the subclass to refresh the internal bean factory.
ConfigurableListableBeanFactory beanFactory = obtainFreshBeanFactory();
// Prepare the bean factory for use in this context.
prepareBeanFactory(beanFactory);
try {
// Allows post-processing of the bean factory in context subclasses.
postProcessBeanFactory(beanFactory);
StartupStep beanPostProcess = this.applicationStartup.start("spring.context.beans.post-process");
// Invoke factory processors registered as beans in the context.
invokeBeanFactoryPostProcessors(beanFactory);
// Register bean processors that intercept bean creation.
registerBeanPostProcessors(beanFactory);
beanPostProcess.end();
// Initialize message source for this context.
initMessageSource();
// Initialize event multicaster for this context.
initApplicationEventMulticaster();
// Initialize other special beans in specific context subclasses.
onRefresh();
// Check for listener beans and register them.
registerListeners();
// Instantiate all remaining (non-lazy-init) singletons.
finishBeanFactoryInitialization(beanFactory);
// Last step: publish corresponding event.
finishRefresh();
} catch (BeansException ex) {
....
}
}
}On line beanPostProcess.end() The process of building metadata is completed, and we need to somehow interrupt this process after this step.
The first thing that came to mind was to write your own BeanFactoryPostProcessor and throw an Exception in it. But the catch is that it must be registered as a bean in the application and put in a package that will be scanned by @PackageScan, which we do not know in advance, and the order of its execution must be configured so that it is the last one. In general, this can all be solved, but this option seemed to me the least promising, especially since I already had other thoughts in my head.
Idea number one
The idea was to use cglib which is part of the Spring-framework, in order to throw an error before calling the method initMessageSource().
The standard method for launching a Spring Boot application looks like this:
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}Where in the static method run an instance is created SpringApplication and running it: new SpringApplication(DemoApplication.class)).run(args). I quickly created an example, rewriting the main method of launching the application like this:
@SpringBootApplication
public class DemoApplicationV1 {
public static void main(String[] args) {
SpringApplication springApplication = new SpringApplication(DemoApplicationV1.class) {
@Override
protected ConfigurableApplicationContext createApplicationContext() {
ConfigurableApplicationContext applicationContext = super.createApplicationContext();
Enhancer enhancer = new Enhancer();
enhancer.setSuperclass(applicationContext.getClass());
enhancer.setCallback(new MethodInterceptorCglib());
return (ConfigurableApplicationContext) enhancer.create();
}
};
springApplication.run(args);
}
static class MethodInterceptorCglib implements MethodInterceptor {
@Override
public Object intercept(Object obj, Method method, Object[] args, MethodProxy proxy) throws Throwable {
if (method.getName().equals("initMessageSource")) {
//printBeans((ConfigurableApplicationContext) obj);
throw new RuntimeException("I am Explyt Plugin!!!");
} else {
return proxy.invokeSuper(obj, args);
}
}
}
}
And it worked… As you can see in the screenshot, the obj variable is our context, which is being initialized. It is already filled with the beanDefinitionMap data we need. It turns out that we just need to “rewrite” the method responsible for launching the project. There are no problems with registering a new bean and the package where it is located. But there is a nuance: in cglib, the context object is created by the constructor without parameters “enhancer#create()”, and this may not always be the case, but in practice it works.
Idea number two
Quite by accident, my attention during the debugging process was drawn to the interface SpringApplicationRunListenerwhich contains callback methods for the context initialization steps:
public interface SpringApplicationRunListener {
default void starting(ConfigurableBootstrapContext bootstrapContext) {
}
default void environmentPrepared(ConfigurableBootstrapContext bootstrapContext,
ConfigurableEnvironment environment) {
}
default void contextPrepared(ConfigurableApplicationContext context) {
}
default void contextLoaded(ConfigurableApplicationContext context) {
}
default void started(ConfigurableApplicationContext context, Duration timeTaken) {
}
default void ready(ConfigurableApplicationContext context, Duration timeTaken) {
}
default void failed(ConfigurableApplicationContext context, Throwable exception) {
}
}At the time of processing the error that I created at the method execution stage initMessageSource()the control process went to the failed method of the default listener. I thought: maybe this listener has a method that is called after the beanDefinitionMap is completed and before the beans are initialized?
Then you can abandon cglib and rewrite the launch method without code generation. However, there was no such method. There are methods that are called before the stage of obtaining metadata and after creating bean instances.
Then the line caught my attention beanPostProcess.end() in the context initialization method. What kind of beanPostProcess object is this? This is the interface StartupStepwhich is responsible for … read the documentation – “a step-by-step recording of indicators of a specific phase or action that occurs during application launch.” There is a default for it implementationwhich contains no logic at all and has empty stub methods. In addition, the SpringApplication class, which is responsible for launching the Spring Boot application, has a public method where you can set your own implementation, then the application launch method can be rewritten without code generation like this:
@SpringBootApplication
public class DemoApplicationV2 {
public static void main(String[] args) {
SpringApplication springApplication = new SpringApplication(DemoApplicationV2.class);
springApplication.setApplicationStartup(new ExplytApplicationStartup());
SpringApplicationHook applicationHook = application -> new ExplytSpringApplicationRunListener();
SpringApplication.withHook(applicationHook, () -> springApplication.run(args));
}
private static class ExplytSpringApplicationRunListener implements SpringApplicationRunListener {
@Override
public void failed(ConfigurableApplicationContext context, Throwable exception) {
//printBeans(context);
}
}
}We indicate our implementation StartupStep in the method SpringApplication#setApplicationStartupwhich throws an error when called beanPostProcess.end() in phase this.applicationStartup.start(“spring.context.beans.post-process”) . Next we through SpringApplication.withHook register our listener, where in the method SpringApplicationRunListener#failed we get a reference to the current context object, where there is beanDefinitionMap metadata, from which we can get everything we need. It seems that everything is fine, we were able to get what we needed without any code generation using public methods of the framework.
But there is again a nuance… SpringApplication.withHook is only available starting from Spring Boot 3.0, which is not very good since many projects still use version 2.0. We won’t talk about version 1.0, it’s like the joke: “We don’t need losers.”
Idea number three
We need to somehow correct the situation. This is what came to mind:
@SpringBootApplication
public class DemoApplicationV3 {
public static void main(String[] args) {
ExplytApplicationStartup applicationStartup = new ExplytApplicationStartup();
SpringApplication springApplication = new SpringApplication(DemoApplicationV3.class) {
@Override
protected ConfigurableApplicationContext createApplicationContext() {
applicationStartup.context = super.createApplicationContext();
return applicationStartup.context;
}
};
springApplication.setApplicationStartup(applicationStartup);
springApplication.run(args);
}
}Overriding the method SpringApplication#createApplicationContextin which we save a link to the context. When we throw an exception in applicationStartup to interrupt the process and not create beans, we will have a link to the context with the data we need. Thus, we added support starting from version 2.4 without any code generation and other illegal tricks, only using the public api. Let me remind you that version 2.4 was released on 10.2020, that is, it has a statute of limitations. For older versions, I decided to leave the option with cglib, which you can use at your own risk by enabling the “explyt.spring.native.old” option in the Registry IDEA settings.
How does it end up working?
What did we get in the end and how does it work?
The launch code given above is formatted as a separate jar file. And to get metadata about beans, you need to do the following:
compile our application,
add our jar file to the -classpath of the application launch,
run the application using the main method from the jar file,
via the “-D” parameter, pass the class name of the source application of our project, which needs to be launched inside SpringApplication,
output in any way (process output or as a file) data about the bins that our process generates in a format that we can easily process,
load this data into the IDE.
The command to start a java process will look something like this:
java -Dexplyt.spring.appClassName=com.example.DemoApplication
-classpath explyt-spring-boot-bean-reader-0.1.jar:other.jars. com.explyt.spring.boot.bean.reader.SpringBootBeanReaderStarterSince our plugin can already create a configuration run, we thought it would be convenient to tie the process launch to the configuration run and use it as a starting point for loading context data. In addition, the user can set additional env parameters there, which can affect active spring profiles, for example.
The operating logic of our plugin is as follows. When opening a spring project, we try to recognize the beans based on the IDEA model code. As a rule, this is sufficient for typical cases.
Next, it is possible to load data about beans directly from Spring using Explyt Spring RunConfiguration. It is thanks to the early configuration that we can easily launch our java process in IDEA by adding our custom jar file to the classpath and changing the launch method. (Spring Boot “icon” with a magnifying glass)
Yes, this requires compiling the project. This may not be very pleasant, but IDEA can do incremental compilation and the requirements for compiling a project in the IDE are more relaxed than the requirement for running the application. I had to work on projects that were not possible to run locally and all development was carried out through tests. After loading the data, a panel with data about the project bins is displayed, where you can double-click to go directly to the bin.
We use this data to determine whether a class is a Spring component. In fact, this is the basis for all subsequent functionality.
To create this panel I used External System Integrationwhich he spoke about in more detail in his article. There are a couple of defects with duplication and extra “icons” at the top of the panel. I opened an issue for this (once And two) and even applied git-patch, but things are still there.
In this panel, you can also search for beans by name and use some kind of dependency analyzer, which has added the ability to search for beans by type.
For example:
public interface MyInterface {}
@Service
class FooBean implements MyInterface {}
@Service
class BazBean implements MyInterface {}
class BarBean implements MyInterface {}This is what the dependency analyzer panel would look like when trying to find all the beans that implement MyInterface:
On one of the past projects, I could not understand who was creating the bean for jackson conversion in the context. It turned out that it was a custom starter developed internally, which, among other things, prevented it from being overridden. Of course, this can be understood from the logs by turning on the debug mode, but this, in my opinion, is much more convenient.
In the process of working with a project, if we do not do anything complex, but create simple beans without complex conditions such as @Conditional, we do not need to constantly run the process of synchronizing data from spring. We can recognize simple cases ourselves and pick them up on the fly if they are in the root packages of the project, otherwise we will need to do manual synchronization.
You can add multiple projects to the panel if you have several @SpringBootApplications in your repository. It also displays available profiles, similar to the Maven panel, which can be enabled/disabled by double-clicking. Data about enabled profiles is synchronized with the corresponding ran configuration.
Data about the synchronization process is located in the Build Tab. Compilation errors will be displayed there if the project could not be compiled and Spring startup logs. In the event of an emergency, all errors will be there. This simplifies the debugging process and analysis of user defects.
Ultimate has a similar panel that appears when we launch a spring from configuration wounds. It shows data about the bins from the actuator. But as far as I can tell, this data is not used to obtain data about the beans – it is simply an opportunity to look at the data from the actuators. This is a more difficult process, since the application can take a very long time to start up and be demanding in terms of the external environment (database, queue, etc.). In addition, it is not always possible to run the project locally. On the other hand, this data is more accurate because beans can be created by complex FactoryBeans, such as Spring Data Repository. In our plugin we support such cases for the most famous spring libraries, but this is potentially one of the drawbacks of our approach.
Let's sum it up
Let's look at the pros and cons of the native approach to obtaining beans.
Pros:
Spring logic is used,
easier to maintain – we don’t write unnecessary code,
less chance of making mistakes
IDE independent solution.
Since we run a separate spring process with all the project dependencies, we can use the output of our process (bean information) in any IDE and with minimal effort add spring support to any development environment in the most native way possible. Also, using this approach we were able to easily add support for @Aspect. Since we are in the spring process, we can use all of its utility methods to find all PointCut expressions and check if “java.lang.reflect.Method” matches the condition. Doing this only through analyzing the code model is very labor-intensive, since you need to create your own PointCut expression calculation engine based on the project source codes. In Ultimate this is implemented based on source codes.
In addition, with this approach we get “free” support for xml configurations. Some may say that no one uses this, but as they say, it’s a small thing, but it’s nice. Moreover, we didn’t do anything special for this.
Cons:
currently only Spring Boot is supported due to the ease of finding root points for projects and running them,
requires compilation,
if there is custom code in main then this is a problem,
for beans that are created through complex FactoryBeans, you need to write custom extractors, and we cannot know about all such cases in advance.
Conclusion
In addition to bean recognition, the plugin also contains a large number of inspections and code completions and context recognition for various Spring and related projects: Spring-Core, Spring-Data, Spring-Web, Spring-AOP, Spring-Initializr, JPQL, OpenApi.
Supported languages: Java/Kotlin/Scala.
Code examples used in this article are available at github.
You can download here.
Small examples of use – how load bins directly from Spring and how work from Spring-Web.
For feedback and bug reports: github
For communication: t.me/explytspring
We will be glad to receive any feedback and suggestions.
