Constantly getting failed formation on Cloud-API and rollbacks.
michaelcuneo opened this issue · 22 comments
I've been designing an API for a while, and i've had to often adjust, remove, tweak parts of the cloud-api. At least once a week I'm left with an entirely disabled awsmobile system, where I have to delete my awsmobile project and start from scratch because no matter of enable/disable cloud-api or manually removing and updating API's or Cloud Formations will fix the state. I'm currently having three x API's all with DELETE_SKIPPED. So I delete them manually, or just remove the entire formation. I remove the front and back build of the awsmobile setup in my app... try to start it all from nothing. Sometimes I even remove the awsmobile section entirely from my app, and awsmobile init [app-id] again. But nothing works. I'm about to again, for the 100th time this month I think, delete the entire project, and start it from scratch because the cloud formation is entirely screwed.
The error usually starts off with a Rollback_complete. I go in to find out why it rolled back and can't find any information about why it failed, so I try it some more... nothing. Tried disabling it, can't it rolls back the delete, can't add new ones, it fails to make new ones.
I just tried to redeploy just now, and it said it was trying to deploy... and at the end, it says 'DELETE COMPLETE' ... I didn't even try to delete them, I was in the middle of a deploy after awsmobile push!
If I have a look at the latest error...
Embedded stack arn:aws:cloudformation:ap-southeast-2:885973830331:stack/MOBILEHUB-934180729-Development-CloudLogic-RQXP8AM9VAND/8f83da30-67b3-11e8-be2c-5081eeee243d was not successfully created: The following resource(s) failed to create: [imageSetCRUDDevelopmentfc7a3ec620c9a48eMobileHub934180729, imageSetCRUDimageSetmobilehub934180729, sampleCloudApisampleLambdamobilehub934180729, settingsCRUDsettingsmobilehub934180729].
It doesn't tell me why they couldn't create, it just says that they didn't, but... once it does this, there's no way to recover, ever. I have to delete the entire awsmobile project and start again.
How can I manipulate the awsmobile CloudFormation, using the awsmobile cli without stuffing it up every single time?
Thanks for reporting this issue.
Cloud you please provide more detailed reproduce steps? It would be great if you can provide an example of modifications you made to your apis that cloud cause this issue. This will help us a lot for investigating this issue.
The steps appear to be arbitrary. I don't know if I'm not waiting long enough between create/push/pull commands when I'm using the console vs using the awsmobile client, it usually just happens at some random point, throws me errors on the command line usually as well, timeouts and such. I'll do everything slower and document each change I make from now on so if it happens again from my next rebuild from scratch, I can write it all up here. It's not too much of a problem to restart it all, because I have my cognito user pool created, the databases are empty already anyway, there's a few things I need in S3 that I can just cut and paste... but it's still a bit annoying.
Today I made 4 databases, and 4 cloud api's. I realised one of them wasn't right, 'the database' and had to remodel the database. I also had to remove and reinstate the cloud-api because the paths wouldn't work properly in a CRUD generated with different partition keys and sort keys. So I did the usual methods.
I removed the database from my awsmobile setup from the console, and remade it from scratch.
I ran
awsmobile pull ... to get the latest database information after rebuilding them.
awsmoblie cloud-api configure ... remove API.
awsmobile push
to remove the databases API CRUD. (Everything worked fine. UPDATE_COMPLETE)
I ran awsmoblie cloud-api configure ... add API.
awsmobile push
to re add the databases. (Everything failed, Rollback due to failure again)
It appears as though my adding/removing of databases has been fine at this point, it has failed when updating the 'sampleCloudApi' that is always there...
Embedded stack arn:aws:cloudformation:ap-southeast-2:885973830331:stack/MOBILEHUB-1286103221-Development-CloudLogic-FFJGEZT1DQ4M/e05616b0-6918-11e8-be2c-5081eeee243d was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [userExclusionSetCRUDuserExclusionSetmobilehub1286103221, userExclusionSetCRUDDevelopment59e1d9125876110bMobileHub1286103221, imageSetsCRUDimageSetsmobilehub1286103221, userSettingsCRUDMobileHub1286103221].
Embedded stack arn:aws:cloudformation:ap-southeast-2:885973830331:stack/MOBILEHUB-1286103221-Development-CloudLogic-FFJGEZT1DQ4M/e05616b0-6918-11e8-be2c-5081eeee243d was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [userExclusionSetCRUDuserExclusionSetmobilehub1286103221, imageSetsCRUDimageSetsmobilehub1286103221, userSettingsCRUDuserSettingsmobilehub1286103221]. The following resource(s) failed to update: [sampleCloudApisampleLambdamobilehub1286103221].
Further removal and updating of cloudapi doesn't fix anything. From now on the entire project is disabled.
Latest AWSMobile Push resulted in
downloading cloud-api lambda function codebases/usr/local/lib/node_modules/awsmobile-cli/node_modules/aws-sdk/lib/request.js:31
throw err;
^
TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received type undefined
at assertPath (path.js:39:11)
at Object.join (path.js:1157:7)
at Response.lambdaClient.getFunction (/usr/local/lib/node_modules/awsmobile-cli/lib/backend-operations/ops-cloud-api.js:160:46)
at Request. (/usr/local/lib/node_modules/awsmobile-cli/node_modules/aws-sdk/lib/request.js:364:18)
at Request.callListeners (/usr/local/lib/node_modules/awsmobile-cli/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
at Request.emit (/usr/local/lib/node_modules/awsmobile-cli/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/usr/local/lib/node_modules/awsmobile-cli/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/usr/local/lib/node_modules/awsmobile-cli/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/awsmobile-cli/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/local/lib/node_modules/awsmobile-cli/node_modules/aws-sdk/lib/state_machine.js:26:10
My next step was to try
awsmobile cloud-api disable
awsmobile push
So far so good. Your API(S) are being deleted...
The delete worked succesfully... now if I try and recreate them, it should fail... I'll do that now.
So I run
awsmobile cloud-api enable
awsmobile cloud-api configure
Create CRUD API for existing database table. (For all 4 databases that I have)
awsmobile push
N.B. it has created sampleLambda again... I didn't ask it to, the Database does not exist.
It rolled back again due to a failure... so yes, now that it is broken, I can no longer use this project.
ARRGH
I think the client needs to ask if the sampleLambda is needed, because it's stuffing up the entire system somehow. It seems to demand that the database be there, and if you mess with it, to make your project clean, I think this is where the problem lies.
Embedded stack arn:aws:cloudformation:ap-southeast-2:885973830331:stack/MOBILEHUB-1286103221-Development-CloudLogic-VET8HWANXDX7/c7a5abf0-6925-11e8-961f-50fae957fcba was not successfully created: The following resource(s) failed to create: [userExclusionSetCRUDuserExclusionSetmobilehub1286103221, userExclusionSetCRUDDevelopment4235c42283a2d75aMobileHub1286103221, imageSetsCRUDimageSetsmobilehub1286103221, userSettingsCRUDMobileHub1286103221].
Your API(s) have been rolled back.
ROLLBACK_COMPLETE
View resource details
sampleCloudApisampleLambdamobilehub1286103221Lambda Function | | DELETE_SKIPPED
This time I even removed the CloudFormation Stack... so that it didn't even exist and had to be recreated. I removed the awsmobile init from my project. Removed the folder and it's entire contents. Reinitialised the project, then recreated the cloud-api and it still fails, and rolls back.
@michaelcuneo: really sorry to hear about your difficulty with using the tools. I'm a PM at AWS Mobile - It would be great if we can get on a call with you with our engineers to try and diagnose what the issue is. If you can share your email (DM me on twitter @TheSwaminator), I'd be happy to set some time up and walk through your issue.
Any progress with this? I have a project with the exact same issue. I do not know where to begin with troubleshooting this. :(
Any update on this issue? Im also seeing this problem. Also, I would appreciate if the AWS mobile treated this issue with some urgency. This bug is blocking further development on my backend and causes it to be overwritten when I make changes because my local version cannot be synced
2 days ago I had an hour phone call with AWS discussing this issue. These issues have also effected my project and has wasted dozens of hours of time/money.
I hope they get to fixing this quickly.
I never ended up PM'ing swaminator and running through the issues with him yet, because my issues appeared to be arbitrary, there were never any specific set steps to cause the issue. It happened after anything at a random state, but I think the problem was with my speed of working pace... I have deliberately gone away from the computer and come back, while waiting on changes to take place now, when I do an awsmobile push or awsmobile pull after modifying anything on either side (console, or client) and it appears to have fixed the issue.
I think there's some kind of time limit between running these commands that breaks everything if you hammer it, or adjust something on the console while it's doing a push or pull. It was always telling me that I had the wrong copy of a state in the console. "You do not have the latest something or other... refresh now to get the latest" ... this would often occur after no changes at all, other than a 'awsmobile pull' on the client side. Somehow a pull was triggering changes to the console while it was opened, and if I hit refresh expecting it to just get the refresh the page or something... I believe that may have caused some kind of pull on the console end while I was pulling on the client end?
Me walking away and not seeing this alert and acting on it, I think has caused the problem to go away...
Sorry for the delayed response while we were investigating this issue. We think this is related to AWS Mobile Hub not deleting your Lambda functions when you disable the cloud-api feature via the CLI/console. This is why you were unable to redeploy the stack when you made changes. Looking back at a previous comment, I can see that your Lambda function wasn't deleted due to its configuration in the CloudFormation stack.
sampleCloudApisampleLambdamobilehub1286103221Lambda Function | | DELETE_SKIPPED
A solution here could be to delete your Lambda functions via the Lambda console and then re-trying your cloud-api operation. Let me know if that works.
I am not 100% sure, but I believe that I did delete all traces of the Lambda Function from everywhere, including the local .awsmobile/ #current-backend-info and backend. But it somehow just kept popping up and failing. I have not had it happen for quite some time because my database and cloud API have been planned well enough now that I don't need to remove or replace any API's at the moment. I'll see how I got in my next project.
Been having similar issues this week. I saw the following error in CloudFormation which led me to think that I should reduce the size of my lambdas. Not wanting to stress about the size of the cloud-api Lambda I decided to just move all my logic out to a different Lambda deploy platform and leave cloud-api basically empty. Of course now awsmobile push works again. Tomorrow I'll continue getting my new Lambdas up and running, unless there is a resolution to this limit. Apologies in advance if this is a total red herring.
having the same issue, from command line
- configure the cloud-api
- Create CRUD API for an existing Amazon DynamoDB table
all seems well, then push to awsmobile, but then the CRUD does not exist, aws-exports.js not updated with CRUD info. I checked out my APIs in the console under the project and they have failed 'rolled back' / 'DELETE_SKIPPED'. I have been able to easily create these APIs before within the same stack -- but now it's failing..
I also have the same problem trying to create CRUD API from an existing Amazon DynamoDB table:
Embedded stack arn:aws:cloudformation:us-east-1:181355217625:stack/MOBILEHUB-1853879279-Development-CloudLogic-1WYN3TW4JO8BW/1da2e6c0-9aa4-11e9-bba4-0a456a010dd4 was not successfully created: The following resource(s) failed to create: [FilesCRUDFilesmobilehub1853879279, sampleCloudApisampleLambdamobilehub1853879279].
same problem!
Hello @davjohnsonGH
The awsmobilecli is now deprecated I would recommend setting up your project using the Amplify CLI.
$ npm install -g @aws-amplify/cli
$ amplify configure