Environment variable INSIGHTS_FILTERS_ENABLED no longer works

The environment variable INSIGHTS_FILTERS_ENABLED can be used to turn off filtering when processing archives. Setting this env var to False turns off the filter checking and filtering of specs that are included in archives so that applications processing the archive using custom rules will see all of the data collected. After insights-core-3.3.21 release this env var is no longer working correctly. If it is set to False then filtered specs will not be loaded and if set to True they will be loaded but the default filters will be applied meaning that only data such as column headings will be present in the data. The True setting works as expected, but the False setting should enable loading all of the data from filtered specs in archives.

Thanks, @bfahr, for reporting this issue, the root cause is that the original validation of the filterable specs didn't care about this environment

insights-core/insights/core/spec_factory.py

Lines 195 to 198 in de92cb5

    
           if (self.ds and 
        
                   any(s.filterable for s in dr.get_registry_points(self.ds)) and 
        
                   not get_filters(self.ds)): 
        
               raise NoFilterException("Skipping %s due to no filters." % dr.get_name(self.ds))

And in #4082, the getting filters are enabled for SerializedArchiveContext (insights-archive from core collection).

So, in runtime against the archive collected by core collection, the process of filterable specs will be terminated due to the above validation.

I raised #4104 to fix it, please have a look as well.

Once it gets approved, I will release it this Thu.

Sorry for bringing this trouble to you.

Without the #4104, the result is:

~/work/insights/insights-core (master) ❯ INSIGHTS_FILTERS_ENABLED=False insights-cat insights.specs.Specs.insights_client_conf /tmp/insights-20240412012625
Traceback (most recent call last):
  File "/home/liuxc/work/insights/insights-core/insights/core/plugins.py", line 101, in invoke
    return self.component(broker)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/liuxc/work/insights/insights-core/insights/core/spec_factory.py", line 654, in __call__
    return self.kind(ctx.locate_path(self.path), root=ctx.root,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/liuxc/work/insights/insights-core/insights/core/spec_factory.py", line 180, in __init__
    self.validate()
  File "/home/liuxc/work/insights/insights-core/insights/core/spec_factory.py", line 198, in validate
    raise NoFilterException("Skipping %s due to no filters." % dr.get_name(self.ds))
insights.core.exceptions.NoFilterException: Skipping insights.specs.default.DefaultSpecs.insights_client_conf due to no filters.

Missing Dependencies:
    At Least One Of:
        insights.specs.default.DefaultSpecs.insights_client_conf

With the #4104 being patched, the result looks good

~/work/insights/insights-core (fix_4103) ❯ INSIGHTS_FILTERS_ENABLED=False insights-cat insights.specs.Specs.insights_client_conf /tmp/insights-20240412012625
SerializedOutputProvider("'/tmp/insights-20240412012625/data/etc/insights-client/insights-client.conf'")
[insights-client]
#authmethod=BASIC
# username to use when authmethod is BASIC
# password ******** use when authmethod is BASIC
#auto_update=True
# core_collect=True
core_collect=False
ros_collect=True
test=abc

	if (self.ds and
	any(s.filterable for s in dr.get_registry_points(self.ds)) and
	not get_filters(self.ds)):
	raise NoFilterException("Skipping %s due to no filters." % dr.get_name(self.ds))