Support to separate step functions into different yml files

Question

Support to separate step functions into different yml files

Pjack opened this issue 7 years ago · 13 comments

Pjack commented 7 years ago

This is a Feature Proposal

Description

For feature proposals:

There are several step machines in the serverless.yml. The declaration is too long that not easy to maintain. I would like to know if it can declare as array and include from another file, just like lambda function? Or is there any other possible solution?
https://serverless.com/framework/docs/providers/aws/guide/functions/

In my imagination, we can declare it as following:

stepFunctions:
  stateMachines:
  - ${file(resources/sf-statemachine1.yml)}
  - ${file(resources/sf-statemachine2.yml)}

Thanks for help

Answer 1 · 2019-03-13T19:26:24.000Z

It would be really nice if it was a clean copy/paste out of the Step Function interface on the AWS Dashboard.

Answer 2 · 2019-03-14T00:21:52.000Z

@Pjack The Serverless framework has built-in support for this, and I just tried and it works out-of-the-box.

You can even mix & match:

service: aws-nodejs

plugins:
  - serverless-step-functions

provider:
  name: aws
  runtime: nodejs8.10

stepFunctions:
  stateMachines:
    hello: ${file(sfn-1.yml)}
    world:
      name: world
      definition:      
        StartAt: Done
        States:
          Done:
            Type: Pass
            End: true

and sfn-1.yml looks like this:

name: hello
definition:      
  StartAt: Done
  States:
    Done:
      Type: Pass
      End: true

Answer 3 · 2019-03-14T00:25:11.000Z

@dearmond if you want something that can be easily copy and pasted out into the console, then you should use JSON instead. e.g.

stepFunctions:
  stateMachines:
    hello: ${file(sfn-1.yml)}
    world:
      name: world
      definition: ${file(sfn-2.json)}

and sfn-2.json looks like this:

{ 
  "StartAt": "Done",
  "States": {
    "Done": {
      "Type": "Pass",
      "End": true
    }
  }
}

the trick is to only put the state definitions in the JSON file

Answer 4 · 2019-03-14T14:33:50.000Z

....also worth noting that the serverless-pseudo-parameters and/or serverless parameters still work in the JSON version as well, just stick them in double quotes...for example here's a single state with params:

Might be worth throwing in the readme.

"CrawlerCheck": {
   "Type": "Task",
   "Parameters": {
     "crawlerName": "crawler"
   },
   "Resource": "arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}:function:${self:service}-${opt:stage}-checkCrawlerHasRun",
   "ResultPath": "$.crawlerStatus",
   "Next": "CrawlerCompletion"
 }

Answer 5 · 2019-03-14T16:53:11.000Z

@Pjack The Serverless framework has built-in support for this, and I just tried and it works out-of-the-box.

You can even mix & match:

service: aws-nodejs

plugins:
  - serverless-step-functions

provider:
  name: aws
  runtime: nodejs8.10

stepFunctions:
  stateMachines:
    hello: ${file(sfn-1.yml)}
    world:
      name: world
      definition:      
        StartAt: Done
        States:
          Done:
            Type: Pass
            End: true

and sfn-1.yml looks like this:

name: hello
definition:      
  StartAt: Done
  States:
    Done:
      Type: Pass
      End: true

It works!
Actually, I tried this before. Obviously, I must made mistake somewhere and cause compile error.
Thanks a lot for this easy solution~

Answer 6 · 2019-03-14T17:53:04.000Z

@bmatson fancy putting in a PR to update the README? :-P add a section on how to separate definitions out to a separate YML/JSON file

Answer 7 · 2019-07-11T12:13:37.000Z

@dearmond if you want something that can be easily copy and pasted out into the console, then you should use JSON instead. e.g.
stepFunctions:
  stateMachines:
    hello: ${file(sfn-1.yml)}
    world:
      name: world
      definition: ${file(sfn-2.json)}
and sfn-2.json looks like this:
{ 
  "StartAt": "Done",
  "States": {
    "Done": {
      "Type": "Pass",
      "End": true
    }
  }
}
the trick is to only put the state definitions in the JSON file

Thanks a lot for the example. Really illustrative!

Answer 8 · 2021-01-15T15:05:59.000Z

Do you have any progress on this?

Answer 9 · 2021-01-15T15:17:14.000Z

@usamedin I doubt there will be "progress" on this, what the OP was asking for is already completely possible, just using a different (probably better) pattern that is documented in the comments. If you'd like to make a PR to update the readme with the discussion in the comments I'm sure the maintainers would appreciate it.

Answer 10 · 2021-01-15T15:37:13.000Z

Thanks, @bmatson, using JSON as an input file for step functions will be very helpful. I will not promise but I will try during the weekend to fix the Readme.

Answer 11 · 2022-02-04T05:18:41.000Z

Hi,
Is it possible to split states into multiple files too?

I've few reusable states that can easily be reused across different state-machiens if we can perform above point.

Answer 12 · 2022-12-30T15:13:58.000Z

Hi @nitsshukla1,

Were you ever able to achieve splitting a function up into multiple files? I'm in a very similar situation where I'd love to be able to reuse certain steps.

Answer 13 · 2023-01-02T04:10:46.000Z

@cwardcode sorry for the late response, and yes i was able to achieve it. In your serverless.yml have this entry:

stepFunctions:
  stateMachines:
    UploadTest: ${file(step-function/stepfunction-upload-test.yml)}

And in file stepfunction-upload-test.yml, start like this:

name: test
definition:
  StartAt: ProcessUploadFile
  States:
     ...