No threshold in config file for ReAct

[PyJulie]

File "/mnt/sda/julie/projects/OSR/OpenOOD-main/OpenOOD-main/openood/postprocessors/react_postprocessor.py", line 41, in postprocess
output = net.forward_threshold(data, self.threshold)
AttributeError: 'ReactPostprocessor' object has no attribute 'threshold'

[zjysteven]

Could you include more info, for example what were you running/calling when seeing this error? self.threshold is set in

OpenOOD/openood/postprocessors/react_postprocessor.py

Lines 47 to 52 in 63e0c79

	def set_hyperparam(self, hyperparam: list):
	self.percentile = hyperparam[0]
	self.threshold = np.percentile(self.activation_log.flatten(),
	self.percentile)
	print('Threshold at percentile {:2d} over id data is: {}'.format(
	self.percentile, self.threshold))

, which should be automatically taken care of if you use our evaluation pipeline or api.

[PyJulie]

I just followed the provided script and run it. It seems that set_hyperapram did not initially work before postprocess function?

[zjysteven]

Were you running without APS mode? If so, I might know what's the problem.

[zjysteven]

@PyJulie Can you try again with the new commit?

[PyJulie]

Hi, I was truly running in APS mode.

I found that

OpenOOD/openood/evaluation_api/evaluator.py

Lines 118 to 157 in 63e0c79

	postprocessor.setup(net, dataloader_dict['id'], dataloader_dict['ood'])
	self.id_name = id_name
	self.net = net
	self.preprocessor = preprocessor
	self.postprocessor = postprocessor
	self.dataloader_dict = dataloader_dict
	self.metrics = {
	'id_acc': None,
	'csid_acc': None,
	'ood': None,
	'fsood': None
	}
	self.scores = {
	'id': {
	'train': None,
	'val': None,
	'test': None
	},
	'csid': {k: None
	for k in dataloader_dict['csid'].keys()},
	'ood': {
	'val': None,
	'near':
	{k: None
	for k in dataloader_dict['ood']['near'].keys()},
	'far': {k: None
	for k in dataloader_dict['ood']['far'].keys()},
	},
	'id_preds': None,
	'id_labels': None,
	'csid_preds': {k: None
	for k in dataloader_dict['csid'].keys()},
	'csid_labels': {k: None
	for k in dataloader_dict['csid'].keys()},
	}
	# perform hyperparameter search if have not done so
	if (self.postprocessor.APS_mode
	and not self.postprocessor.hyperparam_search_done):
	self.hyperparam_search()

line 118 postprocessor.setup works before line 157 self.hyperparam_search().

Unfortunately, new commit may not work either since self.activation_log should be pre-defined.

I added it into def postprocess(self, net: nn.Module, data: Any) function,

and the codes ran successfully but I am not sure if I got the expected results.

I will further try to solve this issue and pull a commit.

Thank you!

[zjysteven]

If you were running in APS mode then that's weird because during hyperparameter search the evaluator will call set_hyperparam which in turn sets self.threshold for react. Without APS mode indeed it was a bug that self.threshold was missing.

[PyJulie]

Yep, it is quiet weird since I did not get bugs when I was running other methods with the requirements of APS mode.

But I checked my .yml config and training log, which shos the APS mode is on:

[PyJulie]