Allow for usage of `DefaultProjectParser` outside of Gradle Daemon process
Opened this issue · 5 comments
BoykoAlex commented
What problem are you trying to solve?
Re-use DefaultProjectParser outside of Gradle daemon process. There are various markers added throughout the ASTs via the project parser. The recipes rely on these markers and therefore it is important to place exactly these markers for the OR recipes to work as expected. There is 3rd party code developing its own project parsers attempting to parse the source and place the same markers - essentially reinventing the wheel with a high possibility of introducing bugs.
Describe the solution you'd like
Modify DefaultProjectParser to work with a proxy to/wrapper of org.gradle.api.Project rather than org.gradle.api.Project directly. Project parser only requires a small subset of the data from org.gradle.api.Project hence the suggestion is aggregate this required data in an interface, lets name in ProjectInfo for now. The hope is that API client could fetch ProjectInfo from Gradle Daemon via the Gradle Tooling API by implementing a serializable ProjectInfoImpl class and then with the fetched ProjectInfo use DefaultProjectParser for parsing projects.
Have you considered any alternatives or workarounds?
- Parse the project on the Gradle Daemon process side and then with Gradle Tooling API fetch the parsed project, i.e. list of source files. This requires changes across many OR projects to make AST classes implement
Serializable. Firstly making all AST classes implementSerializableand work with Java serialization gets out of control (just too many classes with some requiringwriteObject/readObjectimplemented) and secondly this kind of serialize/deserialize ASTs will take very, very long time and huge amounts of memory. This solutions seemed like a complete insanity - Try fetching
org.gradle.api.Projectvia Gradle tooling api. Of course implementations oforg.gradle.api.Projectiare not serializable, however, it is possible to serialize parts of it and have a trimmed down version oforg.gradle.api.Projectoutside of Gradle Daemon process that would work for the current implementation ofDefaultProjectParser. This won't be easy to achieve given the calls such as:
//noinspection DataFlowIssue
if (subproject.getPlugins().hasPlugin("org.jetbrains.kotlin.gradle.dsl.KotlinMultiplatformExtension") ||
subproject.getExtensions().findByName("kotlin") != null && subproject.getExtensions().findByName("kotlin").getClass().getCanonicalName().startsWith("org.jetbrains.kotlin.gradle.dsl.KotlinMultiplatformExtension")) {
return parseMultiplatformKotlinProject(subproject, exclusions, alreadyParsed, projectProvenance, ctx);
}Instead with the ProjectInfo approach i would just have a boolean property isMultiplatformKotlinProject
Reflection API usage make it even more challenging:
Object kotlinExtension = subproject.getExtensions().getByName("kotlin");
NamedDomainObjectContainer<KotlinSourceSet> sourceSets;
try {
Class<?> clazz = kotlinExtension.getClass().getClassLoader().loadClass("org.jetbrains.kotlin.gradle.dsl.KotlinMultiplatformExtension");
//noinspection unchecked
sourceSets = (NamedDomainObjectContainer<KotlinSourceSet>) clazz.getMethod("getSourceSets")
.invoke(kotlinExtension);
} catch (Exception e) {
logger.warn("Failed to resolve KotlinMultiplatformExtension from {}. No sources files from KotlinMultiplatformExtension will be parsed.",
subproject.getPath());
return Stream.empty();
}Instead with the ProjectInfo approach i would have a collection of kotlin source sets on the ProjectInfo where each sourceset has a name, compile and implementation classpaths etc.
Overall this approach seems rather fragile and hard to maintain in the future. Since usage of org.gradle.api.Project is limitless on the rewrite-gradle-plugin we'd have support more and more features from org.gradle.api.Project and ensure they are serializable. At some point this just may stop working...
Are you interested in contributing this feature to OpenRewrite?
Of course! I have the POC for the ProjectInfo approach and if you are okay with the approach we can polish up the ProjectInfo API, naming etc.
BoykoAlex commented
@sambsnyd I can propose the API for ProjectInfo described above that worked for me which we could polish up perhaps if the proposed solution seems acceptable :-)
sambsnyd commented
Yeah that would bean improvement I think. Put it in rewirte-gradle-tooling-model
BoykoAlex commented
@sambsnyd I'd like to get some help with this :-)
I have completed majority of the work in rewrite-gradle-tooling-model
- Ability to fetch
GradleProjectDatamodel via Gradle Tooling API ProjectParserin the new module parser that parses the project using the fetchedGradleProjectData
Rewrite Tooling Model PR: openrewrite/rewrite-gradle-tooling-model#8
I have also attempted to remove project (sources) parsing logic from the rewrite-gradle-plugin and re-use the ProjectParser from above
Rewrite Gradle Plugin PR: #238
However, i got stuck on the rewrite-gradle-plugin :-\ The isolated classpath loader is the issue... I always get LinkageErrors:
Caused by: java.lang.LinkageError: loader constraint violation: loader org.gradle.internal.classloader.VisitableURLClassLoader$InstrumentingVisitableURLClassLoader @5f74a59a wants to load interface org.openrewrite.gradle.toolingapi.parser.GradleProjectData. A different interface with the same name was previously loaded by org.openrewrite.gradle.RewriteClassLoader @7a259b89. (org.openrewrite.gradle.toolingapi.parser.GradleProjectData is in unnamed module of loader org.openrewrite.gradle.RewriteClassLoader @7a259b89, parent loader org.gradle.internal.classloader.VisitableURLClassLoader$InstrumentingVisitableURLClassLoader @5f74a59a)
at org.gradle.internal.classloader.VisitableURLClassLoader$InstrumentingVisitableURLClassLoader.findClass(VisitableURLClassLoader.java:186)
at org.openrewrite.gradle.parser.ProjectParser.repositoryRoot(ProjectParser.java:104)
at org.openrewrite.gradle.parser.ProjectParser.<init>(ProjectParser.java:83)
at org.openrewrite.gradle.isolated.DefaultProjectParser.getProjectParser(DefaultProjectParser.java:509)
at org.openrewrite.gradle.isolated.DefaultProjectParser.parse(DefaultProjectParser.java:541)
at org.openrewrite.gradle.isolated.DefaultProjectParser.listResults(DefaultProjectParser.java:582)
at org.openrewrite.gradle.isolated.DefaultProjectParser.dryRun(DefaultProjectParser.java:241)
at org.openrewrite.gradle.DelegatingProjectParser.lambda$dryRun$3(DelegatingProjectParser.java:107)
at org.openrewrite.gradle.DelegatingProjectParser.unwrapInvocationException(DelegatingProjectParser.java:147)
at org.openrewrite.gradle.DelegatingProjectParser.dryRun(DelegatingProjectParser.java:106)
at org.openrewrite.gradle.RewriteDryRunTask.run(RewriteDryRunTask.java:49)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:125)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:58)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
at org.gradle.api.internal.tasks.execution.TaskExecution$3.run(TaskExecution.java:248)
at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
at org.gradle.api.internal.tasks.execution.TaskExecution.executeAction(TaskExecution.java:233)
at org.gradle.api.internal.tasks.execution.TaskExecution.executeActions(TaskExecution.java:216)
at org.gradle.api.internal.tasks.execution.TaskExecution.executeWithPreviousOutputFiles(TaskExecution.java:199)
at org.gradle.api.internal.tasks.execution.TaskExecution.execute(TaskExecution.java:166)
at org.gradle.internal.execution.steps.ExecuteStep.executeInternal(ExecuteStep.java:105)
at org.gradle.internal.execution.steps.ExecuteStep.access$000(ExecuteStep.java:44)
at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:59)
at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:56)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:56)
at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:44)
at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:67)
at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:37)
at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:41)
at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:74)
at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:55)
at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:50)
at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:28)
at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.executeDelegateBroadcastingChanges(CaptureStateAfterExecutionStep.java:100)
at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:72)
at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:50)
at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:40)
at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:29)
at org.gradle.internal.execution.steps.BuildCacheStep.executeWithoutCache(BuildCacheStep.java:179)
at org.gradle.internal.execution.steps.BuildCacheStep.lambda$execute$1(BuildCacheStep.java:70)
at org.gradle.internal.Either$Right.fold(Either.java:175)
at org.gradle.internal.execution.caching.CachingState.fold(CachingState.java:59)
at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:68)
at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:46)
at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:36)
at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:25)
at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:36)
at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:22)
at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:91)
at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$2(SkipUpToDateStep.java:55)
at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:55)
at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:37)
at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:65)
at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:36)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:37)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:27)
at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:77)
at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:38)
at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:94)
at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:49)
at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:71)
at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:45)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.executeWithNonEmptySources(SkipEmptyWorkStep.java:177)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:81)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:53)
at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:32)
at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:21)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:38)
at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:36)
at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:23)
at org.gradle.internal.execution.steps.CleanupStaleOutputsStep.execute(CleanupStaleOutputsStep.java:75)
at org.gradle.internal.execution.steps.CleanupStaleOutputsStep.execute(CleanupStaleOutputsStep.java:41)
at org.gradle.internal.execution.steps.ExecuteWorkBuildOperationFiringStep.lambda$execute$2(ExecuteWorkBuildOperationFiringStep.java:66)
at org.gradle.internal.execution.steps.ExecuteWorkBuildOperationFiringStep.execute(ExecuteWorkBuildOperationFiringStep.java:66)
at org.gradle.internal.execution.steps.ExecuteWorkBuildOperationFiringStep.execute(ExecuteWorkBuildOperationFiringStep.java:38)
at org.gradle.internal.execution.steps.AssignWorkspaceStep.lambda$execute$0(AssignWorkspaceStep.java:32)
at org.gradle.api.internal.tasks.execution.TaskExecution$4.withWorkspace(TaskExecution.java:293)
at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:30)
at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:21)
at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:37)
at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:27)
at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:47)
at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:34)
at org.gradle.internal.execution.impl.DefaultExecutionEngine$1.execute(DefaultExecutionEngine.java:64)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:146)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:135)
at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:74)
at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:42)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:331)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:318)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.lambda$execute$0(DefaultTaskExecutionGraph.java:314)
at org.gradle.internal.operations.CurrentBuildOperationRef.with(CurrentBuildOperationRef.java:80)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:314)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:303)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:463)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:380)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:47)
I can add GradleProjectData to RewriteClassLoader loadFromParent list but then I get a similar error about ProjectParser and then types in its method signatures and so on...
I'd really need some kind of guidance at this point :-)