`clasp push` changes the execution order of script files in Apps Script projects

Question

`clasp push` changes the execution order of script files in Apps Script projects

DimuDesigns opened this issue 7 years ago · 36 comments

Expected Behavior

clasp push should retain the order of the script files in the Apps Script Editor not just visually but also in terms of order of execution.

Actual Behavior

clasp push re-orders the files so that they are executed in alphabetical order even though visually they retain the expected order in the sidebar of the editor. This bug is hazardous to projects that rely on script files to execute as listed in the Apps Script editor.

Steps to Reproduce the Problem

Create a stand-alone script project
In that project create 6 script files in sequence and name them Z, Y, X, C, B and A; each with a Logger.log() that logs their respective names.
Create a 7th script file and add the following function:
Run that function from the menu bar Run>Run function>runTest and then view the logs View>Logs and you will get the following:
Open your terminal and use clasp clone [PROJECT-FILE-ID] and then clasp push from the command line (project file id omitted from screenshot):
Then run the test function again and then view the logs and you'll see the following:

The execution order of the scripts (though visually correct in the editor's sidebar) is changed to run in alphabetical order!

This bug wreaked havoc on one of my projects before I found it. I would love for clasp to be a part of my development workflow but that won't happen until this issue is resolved.

Specifications

Node version - v6.11.0
Clasp Version - 3.10.10
OS - Windows

Answer 1 · 2018-03-22T04:10:12.000Z

Thanks for the detailed report.
I will address this issue in the next major release.

As a workaround, encapsulate all statements in functions rather than global scope as a best practice to avoid execution order dependencies.

Answer 2 · 2018-03-22T11:13:40.000Z

@DimuDesigns the previous command line tool also had this behavior (gapps)
And it's pretty normal to not rely on something as random as creation order of the files.

As grant underlined, you should not have any global executing code. It's advised to know exactly where are your entry points.
Using global scope may cause unexpected variables overwriting and such.

@grant it would be great to also keep the alphabetical ordering capability

Answer 3 · 2018-03-22T13:45:30.000Z

Best practice aside, sometimes there is a need to store configuration options (some generated from custom classes) as global constants/enums. I could wrap said properties into getter/accessor functions or create some kind of singleton composed of such functions but that feels superfluous since I know I can rely on execution order. Unfortunately, clasp breaks that reliability.

@JeanRemiDelteil Following a clasp push the script files are re-ordered to execute alphabetically (in truth lexicographical-ly) instead of the order in which they were created. However, that ordering is not reflected visually in the Apps Script editor so I don't see the value in retaining this weird ordering glitch. But there is an option to sort files alphabetically under the view menu in the Apps Script editor.

Answer 4 · 2018-03-22T14:15:32.000Z

For visual ordering, you may want to try my AppsScriptColor extension that will display the files always alphabetically, and with folders if you name your files with '/' inside (works well with CLASP).

Answer 5 · 2018-03-22T15:43:38.000Z

Technically, I believe execution order is based on file creation date within the IDE.
In the API, the order of the files in updateContent. It causes lots of bugs for users, so we generally recommend putting all code in functions.

Answer 6 · 2018-03-22T20:27:04.000Z

@grant Using the old import/export method that existed prior to the apps script api (documented here) seems to retain the correct execution order. Maybe you could incorporate those methods to make clasp more robust.

Or (using the new api) before calling updateContent do a getContent to work out the correct order and then push based on that.

EDIT
Noticed that the old import/export method had ids for each of the files in the project. However the new apps script api does not when getContent is called. Maybe your team can bring the two methods (import/export and the new api) more inline to smooth out these issues.

Answer 7 · 2018-03-23T00:04:18.000Z

Is execution order well defined behaviour or were people using it just because it works?

What should happen if we modify the order files are sorted? View -> Sort Alphabetically (And possibly new ones in the future)?
Should it also modify execution order? What if I need a certain execution order, but enjoy viewing files in a totally different order?
What about asynchronous "order"? Is there a reason scripts can't be run in parallel? 😃

If this order issue is really a thing that can't be easily fixed by creating functions, I would rather have the option to edit the order explicitly in the manifest file than relying on naming conventions.

Answer 8 · 2018-03-23T12:46:00.000Z

@rafasc I can't speak for other developers, but in my experience, sorting the files alphabetically within a project via View -> Sort does not impact execution order. From what I can tell execution order is equivalent to the order in which the files are created within a project which appears to be independent of view order.

As for running scripts in parallel (asynchronously), there are folks in the apps script community who have managed to achieve this to a degree (linked here). But I suspect that, in terms of granularity, parallelism is only achievable at the session level, ie. multiple instances of a project are spun up (say via google.script.run from the client-side) and run/execute concurrently. Whereas individual script files/functions within a project probably cannot be made to do so within the current session (but I'm guessing here).

Answer 9 · 2018-03-23T13:27:41.000Z

The question of running script in parallel in totally different from the matter of file execution order.

In AppsScript, strictly speaking, real parallelism or asynchronous behavior like you would expect with Promise, timeOut or thread does not exist at the moment.

All files are read and executed in their creation order, which is why using global is not recommended.
This creation order was historically the order of file creation in the web editor.

And only for standalone script (not container bound script, eg: script linked to a spreadsheet and such), you could get and set the content as a Drive file.
In such a case, execution order was the order in which files were in the drive file (with GAPPS, that was alphabetical order).

The new GAS API is totally different in its disclosed manner of action, in the way that it is a API working on AppsScript projects, and not on Drive files. As such it is not bound the standalone restriction, it can work on standalone and container bound both (which is a GREAT improvement, waited for many years).

Relying in wacky hacks due to Script execution order should not be the way in AppsScript.
There are way better ways to achieve initialization with better control.

For example, you should know were all your entry points are located, and at the beginning of all those functions, just call the function initializing everything (by returning an object).
Just to mention, in AppsScript, the 'this' object is the global object.

I have an example of this in this simple project: monitor-firebase-status (and I would be happy to discuss over this method of initialization).

Answer 10 · 2018-03-27T11:14:49.000Z

@grant I just realized that this issue also highlights another oddity. If you look at the timestamps in the logs (screenshots 4 and 6) the timezone changed following the call to clasp push. It was EDT before and now its PDT after. I checked the Project Properties -> Info and the timezone is set to Eastern Time as expected but the log timezones are now PDT. Weird...

Answer 11 · 2018-07-13T09:42:31.000Z

What is the status of this bug?
I just ran into this problem with a third party library and it took me quite some time to figure out what went wrong.

Answer 12 · 2018-07-13T16:25:33.000Z

I still need to add the createTime and updateTime of a File here:
https://github.com/google/clasp/blob/master/src/files.ts#L123

Hopefully that would fix this bug.

Answer 13 · 2018-07-14T09:52:21.000Z

That would be great. Otherwise I probably have to rewrite the library.

Answer 14 · 2018-07-31T18:10:13.000Z

Btw, this is the bug for reordering script in the script editor: https://issuetracker.google.com/issues/36756144

Answer 15 · 2018-07-31T19:39:56.000Z

I've spent a few hours trying to solve this issue. This is what I've done:

Look at how to set the creation/modified date of a unix file.
- A: It seems impossible, you can only touch a file to set the modification date to now.
Look how gapps may have done this.
- A: It looks like gapps didn't modify the creation date. Not sure how this was solved in that tool.
Look how import-export may have done this:
- It looks like there's simply an array of files that you can put in or out. I guess if you manually stitched the files correctly, it would work, but that's a bit manual.

The solution seems to be this:

clasp pull: Does a unix touch in the order of the file.
- Delays each fs.writeFile with a setTimeout(() => {}, 1);
clasp push: Reads all files and does a fs.statSync to organize the array of files before push.

Answer 16 · 2018-08-01T16:30:19.000Z

@grant : I'm sure there are reason to fix this 'bug' however keeping the alphabetical ordering as an option would make sense.
(Though i'm not to stuck up on this, as for all I've said above the file order does not really matter, and my extension and the original UI allows to show the files in order !)

Answer 17 · 2018-08-02T04:44:44.000Z

If alphabetical ordering (_.gs, a.gs, b/z.gs, x.gs) would fix everyone's issues then I'd gladly implement that and update clasp this week. I hate that this large issue hasn't been solved since March and affects some claspers.

Is there anyone opposed here to that?
@DimuDesigns @rafasc @Nebucatnetzer

Answer 18 · 2018-08-02T06:41:56.000Z

I'm not sure I understand it.
Isn't alphabetical ordering the way it currently works and that is causing problems?
At least my workaround for the problem was to name the files Z01_xxx, Z02_xxx, etc.
So that they keep their order.

Answer 19 · 2018-08-02T09:10:36.000Z

Yes i'm pretty sure it's how the current implementation works.

It's just something you need to take in account when coding.
Alphabetical order is so easy to understand instead of invisible shaddy creation dates imho !
It is also reproducible on all system and fit for a streamlined work process (as the result will always be the same).

Answer 20 · 2018-08-02T09:28:01.000Z

I personally can live with that, however then it is more a problem for the Google Apps Script Browser UI.
Because there the scripts are shown in the order they get created and not in alphabetical order.
I don't know if @grant can do something about that.

Answer 21 · 2018-08-02T10:56:30.000Z

Look how import-export may have done this:

It looks like there's simply an array of files that you can put in or out. I guess if you manually stitched >the files correctly, it would work, but that's a bit manual.

I've found that the arrays retrieved using import-export actually retain the execution order of the script files in an app script project (as seen in the IDE).

I don't like the idea of a tool automatically changing underlying structures that can adversely affect my code. So if at all possible please retain the execution order.

Answer 22 · 2018-08-03T08:04:38.000Z

@Nebucatnetzer It's possible to show the scripts in alphabetical order in the Google Apps Script Browser UI.
You have to check a menu:

Answer 23 · 2018-08-03T08:50:46.000Z

Ah okay, but that doesn't change the execution order?

Answer 24 · 2018-08-03T09:08:01.000Z

No. It's display only. (Which is why alphabetical order execution in this case makes sense)

I myself don't write any code in the browser UI any more, even the files are created on my desktop, then pushed to the script when testing, so using the GUI is anecdotal..

Answer 25 · 2018-08-03T09:38:55.000Z

Ah okay, I agree that alphabetical execution order makes the most sense.
At least it's something which is easy to understand/lookup.

However that still breaks existing projects which rely on the date execution order.

Answer 26 · 2018-08-03T15:46:19.000Z

EDIT: This plan doesn't work because filesystems don't store original creation times after cloning scripts.

It seems like preserving existing execution order (which is based on the create date for each file you've created) is the only solution for preserving old script execution order.

I'll eventually add this feature, but it's going to take some time and testing since filesystems don't have a way to manually override file creation time (ctime). Here would be the implementation plan.

WRITE to FS (after `clasp pull`):

projects.getContent
With the response's files array:
- setTimeout for 1 ms
- Write the file

READ from FS (`clasp push`):

For every non-ignored file:
- fs.statSync#ctimeMs
- Sort the file in order of oldest to newest ctimeMs
- Push

This would preserve file creation order in and out of the script.google.com UI.

Answer 27 · 2018-08-03T16:31:44.000Z

Ok, Just a question on this.

We are only using READ from FS. never Write, or juste to init a project from existing really old script.
With your plan, you will order files against a creation time that came from just creating files when developing the project on the FS.
Those files are stored in a version control system, that can delete and create them at will, hence changing randomly the creation date.

As this makes no sense, but needs to be solved, I propose that you store the existing files order when pulling them, in the .clasp.json settings file.
Then use this order first when pushing, and add all non-mentioned files in alphabetical order (and do not update the .clasp.json with them unless pull is used again).

This method should solve this issue and also provide an easy / simple / consistent way to keep existing file ordering or manually / programmaticaly temper with file ordering !

Answer 28 · 2018-08-03T19:17:02.000Z

I agree that deleting/creating/moving the files will alter the order. Even copying the files from one folder to another would break the order. I'm open to any long-term solution, but I still don't see one.

My initial reply would be: Storing the push order of files doesn't work because you may rename/move the file.

However, a push order may be the best and most accurate way to push files in order.

Here's an idea: I could create a list of files that clasp should push first, but that doesn't seem scalable.

export interface ProjectSettings {
  scriptId: string;
  rootDir: string;
  projectId: string;
  filePushOrder: string[];
}

In this example, you could define filenames in your .clasp.json. It's up to you to ensure that list is up-to-date.

Would this work for everyone?

Answer 29 · 2018-08-06T06:02:42.000Z

For me this would be fine.

Answer 30 · 2018-08-06T09:30:09.000Z

@grant : and if there are files not mentioned in this list, I expect them to be pushed alphabetically ordered.
That would be ok!

About scalable:
The whole 'creation time' ordering is not scalable from the start. There are no serious project/solutions out there that use this kind of behaviour.
So it's not a real problem: the issue here is to prevent old project from breaking when pulling and pushing again.

About file being referenced by name: again, that is not a real issue ! If a user changes a filename (which is the only ID of the file), then he will probably make other changes as well and he know what he is doing.

Answer 31 · 2018-10-04T23:34:17.000Z

@DimuDesigns @JeanRemiDelteil @Nebucatnetzer @rafasc
Can you install clasp@1.6.2 and use the new filePushOrder setting?

It should work like this:

clasp status
Detected filePushOrder setting. Pushing these files first:
└─ main.js

Not ignored files:
└─ main.js
└─ 1.js
└─ 2.js
└─ appsscript.json
└─ fstest.js

Ignored files:
└─ .clasp.json

cat .clasp.json
{
  "scriptId":"1pbtnbRjwdPOAUdmTViOZnWSrqoYPa3Y3xWFzAxOZh1HvesRRKiLu7S3j",
  "filePushOrder":["main.js"]
}

Assuming the API is not buggy, it sends the files in an array as listed in the "Not ignored files" list.

Answer 32 · 2018-10-07T08:17:08.000Z

Thanks I'll check it out next week.

Answer 33 · 2018-10-08T16:24:14.000Z

I'm closing this issue as I think it's fixed with the new filePushOrder feature. If it's still broken, we can re-open the issue.

Answer 34 · 2018-10-09T15:40:10.000Z

@grant the documentation mention

All other files are pushed after this list of files.

Maybe you can detail in which order ? (alphabetical I suppose ?)

All other files are pushed in alphabetical order after this list of files.

I find this important as it's the default behavior.

Answer 35 · 2018-10-09T15:59:07.000Z

@grant I just tested, the "filePushOrder" PR reopen issue #48

Meaning my folder/file architecture is named with a "\" instead of a "/"
This breaks all the HTML imports based on file name.

The final file names should be consistent even when pushed from different OS (window / linux / mac)

Answer 36 · 2018-10-11T14:57:54.000Z

@grant I just tested, the "filePushOrder" PR reopen issue #48

Meaning my folder/file architecture is named with a "" instead of a "/"
This breaks all the HTML imports based on file name.

The final file names should be consistent even when pushed from different OS (window / linux / mac)

@grant, I'm having the same problem here regarding HTML file import.
Any idea how to solve the problem?