Temporary file/directory support

Required Skills: Python or C++

Difficulty: Easy

It would be nice if we could create temporary files/directories in our current file system abstraction API. We currently only use temporary structures in the converter, but they could be useful in other places, e.g. for unzipping archives or on-disk caching.

Creation of temporary files could be implemented in our Python file API or our C++ file API or in both. The Python-side implementation is more preferrable at the moment, since the converter is written Python and temporary files already have an active use case there.

The easiest way to implement support probably is to add a method that calls the underlying system libraries for temporary file/directory creation:

Python: tempfile module
C++: std::filesystem and std::tmpfile

The retrieved path is then wrapped in our Path/Directory classes and returned by the method.

Tasks:

Create temp file/dir and wrap them in openage file system API (Python)
Create temp file/dir and wrap them in openage file system API (C++)
Delete temp file/dir explicitely when object is destroyed
Use new implementation in code (currently only in converter)
(optional) Add method for checking if a file/dir is temporary

Further Reading

Hey @heinezen , I'm interested in working with this issue.
I'm new to open source but I've beginner level experience in working with Python and C++ both.
Can you please assign me this issue.

And also can you please elaborate more about the tasks that exactly what I've to do.
Thanks for your cooperation :)

@Ashhar-24 Hey, I assigned you to to the issue nw :)

Most of what we want from the task should be in the issue description already. The general goal is to make temporary file creation/access more easier than it is right now.

For a start, I think you should try out these steps:

Compile and run the project one if you haven't already. Development works much better when you can test your own code :D

Look at the current usage of tempfile in our code, e.g.

openage/openage/convert/tool/subtool/acquire_sourcedir.py

Lines 253 to 295 in 821a6d1

    
           def download_trial() -> AnyStr: 
        
               """ 
        
               Download and extract the AoC trial version. 
        
               Does not work yet. TODO: Find an exe unpack solution that works on all platforms 
        
               """ 
        
               print(f"Downloading AoC trial version from {TRIAL_URL}") 
        
               # pylint: disable=consider-using-with 
        
               tempdir = tempfile.mkdtemp() 
        
               with urlopen(TRIAL_URL) as response: 
        
                   with tempfile.NamedTemporaryFile(delete=False) as tmp_file: 
        
                       shutil.copyfileobj(response, tmp_file) 
        
                       from ....cabextract.cab import CABFile 
        
                       cab = CABFile(tmp_file, 0x65678) 
        
                       sourcedir = Directory(tempdir).root 
        
                       print(f"Extracting game files to {sourcedir}...") 
        
                       dirs = [cab.root] 
        
                       # Loop over all files in the CAB archive and extract them 
        
                       # to the tempdir 
        
                       while len(dirs) > 0: 
        
                           cur_src_dir = dirs[0] 
        
                           cur_tgt_dir = sourcedir 
        
                           for part in cur_src_dir.parts: 
        
                               cur_tgt_dir = cur_tgt_dir[part] 
        
                           cur_tgt_dir.mkdirs() 
        
                           dirs.remove(cur_src_dir) 
        
                           for path in cur_src_dir.iterdir(): 
        
                               if path.is_dir(): 
        
                                   dirs.append(path) 
        
                               if path.is_file(): 
        
                                   with cur_tgt_dir[path.name].open("wb") as target_file: 
        
                                       with path.open("rb") as source_file: 
        
                                           target_file.write(source_file.read()) 
        
               return tempdir

openage has its own path interface that acts as a wrapper around system paths. It should be extended to get tempfile support. I think the easiest way to implement it in there is to add a staticmethod that first creates a temporary file using tempfile, then wraps it in our Path object and returns the reulting Path.

I think this should give you a good idea how to start. If you get stuck, you can ask more questions in the Matrix chat.

Hey @heinezen , I'm interested in working with this issue.
I'm new to open source but I've 10+ experience Python and C++ both.
I see some of the tasks in this issue are still open.
Can I work on this issue ?

Thank you :)

go for it!

@viswes31 Everything from the tasklist that's not ticked is still up for grabs ^^

	def download_trial() -> AnyStr:
	"""
	Download and extract the AoC trial version.

	Does not work yet. TODO: Find an exe unpack solution that works on all platforms
	"""
	print(f"Downloading AoC trial version from {TRIAL_URL}")
	# pylint: disable=consider-using-with
	tempdir = tempfile.mkdtemp()
	with urlopen(TRIAL_URL) as response:
	with tempfile.NamedTemporaryFile(delete=False) as tmp_file:
	shutil.copyfileobj(response, tmp_file)

	from ....cabextract.cab import CABFile

	cab = CABFile(tmp_file, 0x65678)

	sourcedir = Directory(tempdir).root
	print(f"Extracting game files to {sourcedir}...")
	dirs = [cab.root]

	# Loop over all files in the CAB archive and extract them
	# to the tempdir
	while len(dirs) > 0:
	cur_src_dir = dirs[0]
	cur_tgt_dir = sourcedir

	for part in cur_src_dir.parts:
	cur_tgt_dir = cur_tgt_dir[part]
	cur_tgt_dir.mkdirs()

	dirs.remove(cur_src_dir)

	for path in cur_src_dir.iterdir():
	if path.is_dir():
	dirs.append(path)

	if path.is_file():
	with cur_tgt_dir[path.name].open("wb") as target_file:
	with path.open("rb") as source_file:
	target_file.write(source_file.read())

	return tempdir