[Performance] unnecessary file accesses in media folder (videos, screenshots, marques, etc..) during startup

I did some analysis as to why my boot up time was so long... ~13 minutes. (I have 10000s of games on a NAS)

I went the path of putting all the meta data (videos, screenshots, etc...) files in the same system folder (ex. gba, nes, n64, etc...)
This can contain multiple files per each game.

As I was just halting the program, I found most threads were all on the function call Utils::FileSystem::isDirectory within the function void SystemData::populateFolder(FileData* folder) which can be expensive, and most were in my media folder where it would not find any games.

At first, to fix this, I just a string comparison on the filename to the hardcoded folder name. I my case it was "media", because that's what Skyscraper defaults too. This then brought my load time down to about 8 minutes (a significant improvement!).

But... I don't believe doing a hardcoded string comparison is the best way to go for up streaming this. I'm thinking that there should be either a .ignorefolders (or some other name) file that would list out folders it shouldn't recurse into such as a folder that would contain the files for the metadata.

Unmodified Code

EmulationStation/es-app/src/SystemData.cpp

Line 69 in c6bbd38

void SystemData::populateFolder(FileData* folder)

isGame = false;
if(std::find(mEnvData->mSearchExtensions.cbegin(), mEnvData->mSearchExtensions.cend(), extension) != mEnvData->mSearchExtensions.cend())
{
	FileData* newGame = new FileData(GAME, filePath, mEnvData, this);

	// preventing new arcade assets to be added
	if(!newGame->isArcadeAsset())
	{
		folder->addChild(newGame);
		isGame = true;
	}
}

//add directories that also do not match an extension as folders
if(!isGame && Utils::FileSystem::isDirectory(filePath))
{
	FileData* newFolder = new FileData(FOLDER, filePath, mEnvData, this);
	populateFolder(newFolder);

	//ignore folders that do not contain games
	if(newFolder->getChildrenByFilename().size() == 0)
		delete newFolder;
	else
		folder->addChild(newFolder);
}

Modified Code (with hardcoded check)

isGame = false;
isMedia = false;
if(std::find(mEnvData->mSearchExtensions.cbegin(), mEnvData->mSearchExtensions.cend(), extension) != mEnvData->mSearchExtensions.cend())
{
	FileData* newGame = new FileData(GAME, filePath, mEnvData, this);

	// preventing new arcade assets to be added
	if(!newGame->isArcadeAsset())
	{
		folder->addChild(newGame);
		isGame = true;
	}
}
else if(Utils::FileSystem::getFileName(filePath) == "media")
{
	// preventing from entering the metadata folder
	isMedia = true;
}

//add directories that also do not match an extension as folders
if((!isGame && !isMedia) && Utils::FileSystem::isDirectory(filePath))
{
	FileData* newFolder = new FileData(FOLDER, filePath, mEnvData, this);
	populateFolder(newFolder);

	//ignore folders that do not contain games
	if(newFolder->getChildrenByFilename().size() == 0)
		delete newFolder;
	else
		folder->addChild(newFolder);
}

If it is believed that .ignorefolders is a good way to "fix" this issue. I can look in to implementing it.

@XenuIsWatching thanks for raising this. Question: are you using the latest dev build, and can you repeat the same test there if not?

I'd be keen in learning about the performance changes there, if any.

Two questions/remarks:

Are you using "Parse gamelists only", or letting it actually scan the folders? I am assuming the latter - which is alright, but confirming.
There's a profiling capability added to the latest dev code. It needs to be build with that flag, and then adding the profiling code to the specific functions you'd want to monitor. If you'd see interest in exploring it further, I'm happy to help. The specifics are in #716 .

From reading the code, you're suffering from an issue I also run into with my own NAS, which is the fact that stat calls are absurdly expensive. I'd really like to get to the bottom of it, but I'm a bit at a loss for the moment. I don't disagree with adding a .noroms or .ignorefolder file that prevents scanning that folder and children, but I still feel that's only part of the problem. Happy to keep going through this.

What system are you running this on, and what is your NAS on? Would both of them be Pi4s?

Folders starting with . should already be ignored, That was added that a long time ago and I have renamed my media folder to ".media"

@tomaz82 Thanks for the info.

I performed the following commands to move my media folder to the new location in each folder

find . -type f -name 'gamelist.xml' -exec sed -i 's#/media/#/.media/#g'  {} \;
find . -type d -name media -execdir mv '{}' .media ';'