Deleted packages not removed from repodata

Steps that lead me to this issue:

I built rust-unicase_serde in COPR
I built rust-rocket_cors that depends on the previous rust-unicase_serde in COPR
I got rust-unicase_serde accepted in Fedora, so it's already in Fedora's repo
I remove rust-unicase_serde from COPR
I try to rebuild rust-rocket_cors in COPR

Behavior:
It failed with

Not finished - interrupted by error: Status code: 404 for https://download.cFailed to download packages
 Librepo error: Status code: 404 for https://download.copr.fedorainfracloud.org/results/wojnilowicz/activitywatch/fedora-40-x86_64/07436781-rust-unicase_serde/rust-unicase_serde+default-devel-0.1.0-1.fc40.noarch.rpm (IP: 108.138.85.40)

Additional information:
copr regenerate-repos doesn't help.

Suggested fix:
Could you make copr regenerate-repos to purge whole repodata beforehand, so no nonexistent packages linger there?

A workaround is to go to the Settings tab and under "3. Other options" set "Repo priority:" to "100". After that rust-unicase_serde is installed from the Fedora repository and not my nonexisting COPR package.

hmmm this seems like a regression, could this be a bug from adding the pulp features and moving corresponding code around, @FrostyX ?

I think COPR used to regenerate /devel repo(with enabled "Create repositories manually") when a build is deleted, but doesn't do it anymore.

Thank you for the hint. That sounds like it could be the problem, but I think Copr still regenerates the devel repo when deleting a build. At least to me it seems so.

I created a project with manual createrepo enabled and Copr generated this repository
https://copr-be.cloud.fedoraproject.org/results/frostyx/test-devel/fedora-40-x86_64/devel/
(I am linking a non-CDN backend URL here)

The repository was created at
2024-Nov-19 22:35:00

Then I submitted a build. The repository was regenerated at
2024-Nov-19 22:37:29

Then I deleted the build and the repository was regenerated at
2024-Nov-19 22:45:40

So it seems fine to me. Do you have a different reproducer?

@wojnilowicz can you please link the specific Copr build where this happened?

Suggested fix:
Could you make copr regenerate-repos to purge whole repodata beforehand, so no nonexistent packages linger there?

Looking at the code, we should do exactly this.

copr/backend/copr_backend/storage.py

Lines 152 to 172 in a01d746

    
           # Run createrepo first and then remove the files (to avoid old 
        
           # repodata temporarily pointing at non-existing files)! 
        
           # In srpm-builds we don't create repodata at all 
        
           if chroot != "srpm-builds": 
        
               repo = call_copr_repo( 
        
                   chroot_path, delete=subdirs, devel=self.devel, 
        
                   appstream=self.appstream, logger=self.log) 
        
               if not repo: 
        
                   result = False 
        
           for build_id in build_ids or []: 
        
               log_paths = [ 
        
                   os.path.join(chroot_path, build_chroot_log_name(build_id)), 
        
                   # we used to create those before 
        
                   os.path.join(chroot_path, 'build-{}.rsync.log'.format(build_id)), 
        
                   os.path.join(chroot_path, 'build-{}.log'.format(build_id))] 
        
               for log_path in log_paths: 
        
                   try: 
        
                       os.unlink(log_path) 
        
                   except OSError: 
        
                       self.log.debug("can't remove %s", log_path)

I dunno... I did the same steps (created a project with manual createrepo enabled, submitted a build with fedpkg copr-build, deleted the build via the web interface) and it's still there:
https://copr-be.cloud.fedoraproject.org/results/solopasha/devel-test/fedora-40-x86_64/devel/repodata/

❯ dnf --refresh --repofrompath 'test,https://copr-be.cloud.fedoraproject.org/results/solopasha/devel-test/fedora-40-x86_64/devel/' rq --repo=test
Updating and loading repositories:
 test                                                                                                       100% |   3.5 KiB/s |   1.5 KiB |  00m00s
Repositories loaded.
dummy-package-0:0-1.fc40.src
dummy-package-0:0-1.fc40.x86_64

Thank you very much for helping me debug this @solopasha. I probably wouldn't find the culprit otherwise.
What revealed the problem is that for my repository, it worked as expected but for yours, it indeed didn't work. And the difference was ... I used the "Delete" button in the build detail page, and you used the "Delete all marked" button.

I played with it, and it seems to ignore the manual createrepo flag. I will send a PR to fix it.

hmmm this seems like a regression, could this be a bug from adding the pulp features and moving corresponding code around, @FrostyX ?

@nikromen it indeed seems to be a regression caused by the features you are mentioning. We previously did this
c5166a1#diff-717d47c392da08d2b79bec8392ecfd78bb404644caeab3b717d3ae193307e20aL232
and the line got lost

	# Run createrepo first and then remove the files (to avoid old
	# repodata temporarily pointing at non-existing files)!
	# In srpm-builds we don't create repodata at all
	if chroot != "srpm-builds":
	repo = call_copr_repo(
	chroot_path, delete=subdirs, devel=self.devel,
	appstream=self.appstream, logger=self.log)
	if not repo:
	result = False

	for build_id in build_ids or []:
	log_paths = [
	os.path.join(chroot_path, build_chroot_log_name(build_id)),
	# we used to create those before
	os.path.join(chroot_path, 'build-{}.rsync.log'.format(build_id)),
	os.path.join(chroot_path, 'build-{}.log'.format(build_id))]
	for log_path in log_paths:
	try:
	os.unlink(log_path)
	except OSError:
	self.log.debug("can't remove %s", log_path)