We found 51 tets flaky
Closed this issue · 1 comments
We run your test cases for 50 times and found that the following tests are flaky [1] (i.e., they pass or fail non-deterministically):
- testPrefaceHiddenNotDisplayed
- testPrefaceFullDisplayed
- testBackFromSettingsShouldReturnToPreviousFragment
- books_sortOrderAfterSettingsChange
- testOpenSettings
- testChangeCreatedAtPropertyResultsShouldBeReordered
- testCondition
- testSortOrder
- testNewNote
- testClearDatabaseWithFragmentsInBackStack
- testScheduledWithRepeaterToDoneFromNoteFragment
- testBookTitleChangeOnPrefaceEdit
- testScheduledWithRepeaterToDoneFromBook
- testBookReparseOnStateConfigChange
- testDeleteNote
- testChangingStateSettingsFromNoteFragment
- testMetadataShowSelectedOnNoteLoad
- testDeSelectRemovedNoteInSearch
- testInheritedTagsAfterMovingNote
- testExportSavedSearches
- testDisplayedContentInBook
- testChangeDefaultPrioritySearchResultsShouldBeReordered
- testLowestPriorityUpdateOnDefaultPriorityChange
- testDefaultPriorityUpdateOnLowestPriorityChange
- testAddingNewTodoKeywordInSettingsAndChangingStateToItForNewNote
- testNewNoteDefaultStateIsSetInitially
- testNewNoteDefaultStateIsInitiallyVisibleInSummarytestLowercaseStateConvertedToUppercase
- testAddingNewTodoKeywordInSettingsNewNoteShouldHaveDefaultState
- testStatesDuplicateDetectedIgnoringCase
- testStateSummaryAfterNoStates
- testAutoSyncIsTriggeredAfterCreatingNote
- testForceLoadingMultipleTimes
- testBookParsingAfterKeywordsSettingChange
- testPrefaceModificationMakesBookOutOfSync
- testInheritedTagsAfterDemotingSubtree
- testSelectingNoteThenOpeningAnotherBook
- testChangeDefaultPriorityAgendaResultsShouldBeReordered
- testNewBookDialogShouldSurviveScreenRotation
- testReposActivityFragments
- testChangingPrioritySettingsFromNoteFragment
- testDropboxRepoWithPercentCharacter
- testImportingGettingStartedFromGettingStartedNotebook
- testRenamingReposRemovesLinksWhatUsedThem
- testForceLoadingBookWithLink
- testSettingLinkToRenamedRepo
- testSavingProperties
- testForceLoadingAfterModification
- testMainActivityFragments
- testSaveAfterRotation
- testCabStaysOpenOnRotationInQueryFragment
- testTrimmingTitleInNoteFragment
We ran these tests on a generic Android Emulator (AVD) with Android API version 28.
To reproduce failures on these tests, please run the following script on gist:
https://gist.github.com/shaker-project/a7c7f6efccb8d04fadd1daea4d4af545
If you didn't understand any point of the script we can explain.
We require that stress-ng is installed. If not, run $> sudo apt-get install stress-ng.
If you confirm this behavior, our suggestion is to add the annotation @testflaky on these tests. See https://developer.android.com/reference/androidx/test/filters/FlakyTest
If you wish for a repair proposal, please let us know.
[1] https://hackernoon.com/flaky-tests-a-war-that-never-ends-9aa32fdef359
Sounds like animations have not been disabled.