Oracle - Ignore Dependencies

Question

Oracle - Ignore Dependencies

greenmonster1 opened this issue 5 years ago · 27 comments

Hi - On obevo init / deploy can we allow an option to ignore dependencies for packages. Meaning a ignore dependency option which will not account for any package references and let us recompile the packages post deployment so that they become valid then. Reason we ask this is we have complex pl/sql packages (350+) with lot of dependencies within them. Today we are facing issues during deploy failing computing dependency

at com.gs.obevo.impl.graph.GraphUtil.validateNoCycles(GraphUtil.java:137)
at com.gs.obevo.impl.graph.GraphEnricherImpl.createDependencyGraph(GraphEnricherImpl.java:161)
at com.gs.obevo.impl.changetypes.RerunnableChangeTypeCommandCalculator.getObjectChangesRequiringRecompilation(RerunnableChangeTypeCommandCalculator.java:152)
at com.gs.obevo.impl.changetypes.RerunnableChangeTypeCommandCalculator.processRerunnableChanges(RerunnableChangeTypeCommandCalculator.java:116)
at com.gs.obevo.impl.changetypes.RerunnableChangeTypeCommandCalculator.calculateCommands(RerunnableChangeTypeCommandCalculator.java:94)
at com.gs.obevo.impl.changecalc.ChangesetCreatorImpl.determineChangeset(ChangesetCreatorImpl.kt:34)
at com.gs.obevo.impl.MainDeployer.executeInternal(MainDeployer.kt:250)
at com.gs.obevo.impl.MainDeployer.execute(MainDeployer.kt:93)
at com.gs.obevo.impl.context.AbstractDeployerAppContext.deploy(AbstractDeployerAppContext.java:139)
at com.gs.obevo.db.cmdline.DbDeployerMain.start(DbDeployerMain.java:79)
at com.gs.obevo.cmdline.AbstractMain.start(AbstractMain.java:129)
at com.gs.obevo.dist.Main$11.value(Main.java:234)
at com.gs.obevo.dist.Main$11.value(Main.java:231)
at com.gs.obevo.dist.Main.execute(Main.java:119)
at com.gs.obevo.dist.Main.execute(Main.java:87)
at com.gs.obevo.dist.Main.main(Main.java:69)

Answer 1 · 2019-07-17T12:47:14.000Z

In general, this could be doable but I want to dig into your use case first

Is the dependency issue with the package or package body? Ie if it were to deploy all packages first (counting dependencies) and then deployed the package bodies, would that work?

In the stack trace that you sent, was there a string preceding it showing the full cycle? Can you send that over (only if it doesn't contain sensitive info)

Can you clarify what you mean by recompiling in your case? Wanted an idea of the workflow you are planning

Answer 2 · 2019-07-18T14:32:09.000Z

Is the dependency issue with the package or package body? Ie if it were to deploy all packages first (counting dependencies) and then deployed the package bodies, would that work? -We have not tried this but we are going to run this test now, I am sure you know that as of today obevo reverse engineer's spec and body to one file (I understand that you want us to try it so that you may change the reverse engineering to give spec and body in separate files).

In the stack trace that you sent, was there a string preceding it showing the full cycle? Can you send that over (only if it doesn't contain sensitive info) - It does not have the full cycle

Can you clarify what you mean by recompiling in your case? Wanted an idea of the workflow you are planning - The whole idea is we wanted obevo to ignore dependency for the packages and once packages are deployed in order for all the invalid package we run the following command to compile them again
--Package Spec
Alter compile;
--Package Body
Alter compile body;

Answer 3 · 2019-07-18T15:15:46.000Z

Thanks for the info

Regarding splitting the package spec and body, I had replied on #240 -> you may not need them in a
separate file as they are already deployed separately. I think if I add the additional fix mentioned in this current ticket, it may ultimately fix your issues

I'll test out your example - may need a couple weeks

Answer 4 · 2019-07-18T17:46:58.000Z

When you meant additional fix, did you mean

Make obevo run thru the package spec deployment first and the do package body (though they are on the same files)
or
Allow ignore dependency for packages?

Also couple weeks might be a little hard on us because we were all into migrate our DB schemas and hit this snag? Anything which can make this sooner will be really helpful.

Thanks!

Answer 5 · 2019-07-18T18:14:56.000Z

I meant the second - to allow ignoring dependency for packages

I'll see what I can do to get you something to try sooner

Answer 6 · 2019-07-23T01:04:58.000Z

I have a snapshot that you can try out:
https://www.dropbox.com/s/mpn3ho3erobezrb/obevo-cli-local-SNAPSHOT-dist.zip?dl=0

The code still needs some cleanup, but I want to give you a snapshot to try so that we can get feedback sooner

The change here is for package bodies to ignore the names of packages when calculating dependencies. You should not need to do anything extra for this to work

FYI - the issue mentioned in #243 is also resolved w/ this latest snapshot

Answer 7 · 2019-07-23T14:27:09.000Z

I have attempted to run a DEPLOY with the latest binaries and see that OBEVO still tries to derive dependencies for PACKAGES. The DEPLOY failed with the same exception.

    at com.gs.obevo.impl.graph.GraphUtil.validateNoCycles(GraphUtil.java:137)
    at com.gs.obevo.impl.graph.GraphEnricherImpl.createDependencyGraph(GraphEnricherImpl.java:161)
    at com.gs.obevo.impl.changetypes.RerunnableChangeTypeCommandCalculator.getObjectChangesRequiringRecompilation(RerunnableChangeTypeCommandCalculator.java:152)
    at com.gs.obevo.impl.changetypes.RerunnableChangeTypeCommandCalculator.processRerunnableChanges(RerunnableChangeTypeCommandCalculator.java:116)
    at com.gs.obevo.impl.changetypes.RerunnableChangeTypeCommandCalculator.calculateCommands(RerunnableChangeTypeCommandCalculator.java:94)
    at com.gs.obevo.impl.changecalc.ChangesetCreatorImpl.determineChangeset(ChangesetCreatorImpl.kt:34)
    at com.gs.obevo.impl.MainDeployer.executeInternal(MainDeployer.kt:266)
    at com.gs.obevo.impl.MainDeployer.execute(MainDeployer.kt:96)
    at com.gs.obevo.impl.context.AbstractDeployerAppContext.deploy(AbstractDeployerAppContext.java:139)
    at com.gs.obevo.db.cmdline.DbDeployerMain.start(DbDeployerMain.java:79)
    at com.gs.obevo.cmdline.AbstractMain.start(AbstractMain.java:129)
    at com.gs.obevo.dist.Main$5.value(Main.java:185)
    at com.gs.obevo.dist.Main$5.value(Main.java:182)
    at com.gs.obevo.dist.Main.execute(Main.java:119)
    at com.gs.obevo.dist.Main.execute(Main.java:87)
    at com.gs.obevo.dist.Main.main(Main.java:69)

Answer 8 · 2019-07-23T15:00:27.000Z

Can you send the full log file, including the messages you see before that stack trace?

Answer 9 · 2019-07-23T19:09:11.000Z

Here it is:

obevo-DEPLOY-20190723102013.txt

Answer 10 · 2019-07-24T02:23:25.000Z

You can try this new snapshot - https://www.dropbox.com/s/mpn3ho3erobezrb/obevo-cli-local-SNAPSHOT-dist.zip?dl=0

It may not work, but I added some additional logging

If it doesn't work, please send the new log file over, as well as the following package files (I see they form a cycle, so I'd like to see this as a reference example if possible)

AAA_BBC_BBEN
FPR_OUTS
AAA_AAL_LMTY

Answer 11 · 2019-07-24T22:20:56.000Z

I have executed a DEPLOY with the new snapshot and it failed with the same EXCEPTION.

The new log file contains 1,050,292 lines, most of it proprietary code.

I will work on a different solution that will be easily shared with you.

Answer 12 · 2019-07-25T23:11:13.000Z

If the log file is difficult, then at least would the content to these objects work for you?

AAA_BBC_BBEN
FPR_OUTS
AAA_AAL_LMTY

Answer 13 · 2019-07-30T03:47:18.000Z

I've uploaded a new snapshot without the excess logs (so it should get down to a more palatable number as before), but still the same behavior - https://www.dropbox.com/s/mpn3ho3erobezrb/obevo-cli-local-SNAPSHOT-dist.zip?dl=0

Would you be able to send me the file contents for the following objects? (i.e. for the smallest cycle mentioned in the logs, which I saw as a 3-node cycle w/ these objects). It will help in debugging the issue - though if you rerun with the latest snapshot, send me that whole log as well

AAA_BBC_BBEN
FPR_OUTS
AAA_AAL_LMTY

Answer 14 · 2019-07-31T00:01:17.000Z

Sorry for delayed response, my co-worker who was working on this is on vacation, he will run a test once he is back on monday.

Answer 15 · 2019-08-06T03:46:54.000Z

Acknowledged - thanks!

Answer 16 · 2019-08-06T17:19:15.000Z

There is proprietary code in
AAA_BBC_BBEN
FPR_OUTS
AAA_AAL_LMTY

Here is the new log:
obevo-DEPLOY-20190806083021.log

Answer 17 · 2019-08-06T19:29:49.000Z

Would it be possible to anonymize the code in those 3 stored procedures?

At the very least what I'm looking for are:

The structure of those files (e.g. if you have any //// METADATA, //// CHANGE, or //// BODY entries in there
What the main "create procedure" line says
Any references to the object names that you see in that code (e.g. AAA_BBC_BBEN references in FPR_OUTS)

I'll try to work without it if needed, but seeing the anonymized versions would be helpful

Answer 18 · 2019-08-09T03:38:07.000Z

Note - I found an issue with the cycle log printing on my side.

You can hold off on any action on your side until I fix this issue

Answer 19 · 2019-08-09T03:52:41.000Z

Amazing thank you for the update!!!

…

Sent from my iPhone on VZW

On Aug 8, 2019, at 11:38 PM, shantstepanian ***@***.***> wrote: Note - I found an issue with the cycle log printing on my side. You can hold off on any action on your side until I fix this issue — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

Answer 20 · 2019-08-09T13:13:52.000Z

Couple more questions - are you keeping these objects in the /routine packages folder or something else?

And I assume your package file contents are as follows?

Create package abc ...

//// BODY
create package body abc ...

Answer 21 · 2019-08-13T12:17:23.000Z

Correct, the objects are in the routinepackage folder. The package file has been reverse engineered using OBEVO, so yes, the package contents do follow your assumption above.

Answer 22 · 2019-08-20T16:43:51.000Z

I downloaded the fix and tested it, but seems that it is a MongoDB version. Would you please double-check me?

Answer 23 · 2019-08-20T17:07:10.000Z

The latest snapshot version (and the released 7.2.0 version) should have the code that I worked on where I attempted to have package-bodies not scanned for dependencies (which seems to work for me locally, but not for your use case per the earlier conversations). The mongo stuff happens to be in there, but your code should be working as it was before

That said - I haven't yet added my improvement on the error message logging that I mentioned earlier ("Note - I found an issue with the cycle log printing on my side")

What behavior are you observing now?

Answer 24 · 2019-08-20T18:10:05.000Z

Please ignore my last comment. I tested 7.2.0 and it is an Oracle version. OK on the improvement that you are working on. I will be glad to test it when it becomes available. Thanks.

Answer 25 · 2019-08-22T14:14:35.000Z

I've uploaded a snapshot with the improved cycle error messages here - https://www.dropbox.com/s/mpn3ho3erobezrb/obevo-cli-local-SNAPSHOT-dist.zip?dl=0

You likely would still see exceptions, but it should be more useful for me this time.

Please try it again and send me the cycle error message logs that you see (or just your full log file)

Answer 26 · 2019-08-27T21:42:41.000Z

Here is the new log:
obevo-DEPLOY-20190826100436.log

Answer 27 · 2019-09-01T19:08:29.000Z

This is proving to be a challenge on your large schema :) We will work through this - appreciate your patience on this

Please try this snapshot:
https://www.dropbox.com/s/mpn3ho3erobezrb/obevo-cli-local-SNAPSHOT-dist.zip?dl=0

I've allowed the underlying graph algorithm to be parameterized. Please run this using the various algorithms until one doesn't give the out of memory error like you saw

To do this, set OBEVO_JAVA_OPTS as follows before running your command

SET OBEVO_JAVA_OPTS="-DgraphAlgorithm=tiernan"
c:\yourobevo\deploy yourArgsAsNormal

And then try the following variations:

SET OBEVO_JAVA_OPTS="-DgraphAlgorithm=tarjan"

SET OBEVO_JAVA_OPTS="-DgraphAlgorithm=lauer"

SET OBEVO_JAVA_OPTS="-DgraphAlgorithm=johnson"