Removing a stale Solr result in production

[pgwillia]

View details in Rollbar: https://app.rollbar.com/a/ualbertalib/fix/item/jupiter/1918

Removing a stale Solr result, be28d90c-6a87-4547-bd06-399e5ca0d708: {"id"=>"be28d90c-6a87-4547-bd06-399e5ca0d708", "visibility_ssim"=>["http://terms.library.ualberta.ca/public"], "record_created_at_dtsi"=>"2024-06-11T19:41:22.338Z", "date_ingested_dtsi"=>"2024-06-11T19:41:22.338Z", "title_tesim"=>["Decoding the Heart"], "title_ssi"=>"Decoding the Heart", "sort_year_isim"=>[2024], "sort_year_isi"=>2024, "member_of_paths_dpsim"=>["e14a6354-b557-4a15-8e99-70ad237be121/c9f59edb-6d07-41bf-a324-9b6caca431c5"], "subject_tesim"=>["Research image"], "creators_tesim"=>["DasGupta, Ishani"], "created_tesim"=>["2024-06-01"], "created_ssi"=>"2024-06-01", "languages_tesim"=>["http://id.loc.gov/vocabulary/iso639-2/eng"], "license_tesim"=>["http://creativecommons.org/licenses/by-nc/4.0/"], "item_type_ssim"=>["http://purl.org/ontology/bibo/Image"], "has_model_ssim"=>["ArItem"], "owner_ssim"=>["1"], "description_tesim"=>["The image portrays the synergistic relation between the two main aspects of my research - medical image analysis and coding strategies. I dynamically engage with both, symbolized by the reflections seen within the frames. The aim is to efficiently and accurately identify the left ventricle (LV) from 3D heart ultrasound (echocardiography) images. \nBy examining heart images, I am developing a machine learning (ML) assisted automated system that can efficiently and accurately locate the LV and analyze heart function. Working with cardiologists at the University Hospital, I strive to improve the efficiency of identifying heart diseases. I design, code, and train neural networks that continuously learn the principal features in these heart images. Once the trained ML model has generated its predicted outputs, I use medical imaging software to view, examine, and compare these ML outputs with the expert LV identification.\nIn my research, the fundamental interplay between these two domains is essential for advancing the understanding and capabilities of cardiovascular imaging.\n"], "_version_"=>1801595009808465920, "timestamp"=>"2024-06-11T19:41:29.419Z", "score"=>1.0}

jupiter/app/models/jupiter_core/solr_services/deferred_faceted_solr_query.rb

Lines 146 to 154 in 66ca00d

	rescue ActiveRecord::RecordNotFound
	# This _should_ only crop up in tests, where truncation of tables is bypassing callbacks that clean up
	# solr. BUT, I want to track this just in case.
	msg = "Removing a stale Solr result, #{res['id']}: #{res.inspect}"
	Rollbar.warning(msg)
	Rails.logger.warn(msg)
	JupiterCore::SolrServices::Client.instance.remove_document(res['id'])
	nil
	end

[pgwillia]

@murny says, "Looks like at ~1:41 PM yesterday something happen where a bunch of things couldn’t be found in the database 🤔. So these Solr errors are when we try to look up an object in ActiveRecord and it’s not found (we assume Solr is stale, so we remove the record from Solr). But this also happen in Sidekiq, where we saw a big uptick of ActiveRecord Not Found errors in DOICreateJob’s.
It hasn’t happen again since 1PM yesterday (2024-06-11) and no more new errors for this. Don’t understand how this error could happen in a blimp like this though, unless we swung to a different DB for a few seconds or something 🤷 (cause if DB went down or went offline, you would get a different error (timeout or connection error, not record not found)
These objects do exist though and I can view them no problem in the search results and show page in production as of right now"

Seems to be a one off and nothing is missing. We'll watch for this to re-occur.

[pgwillia]

@nmacgreg I wonder if this was related to IST resolving the PostgreSQL database backup/firewall the other day.