Unexpected behavior when reclaiming resources after a thread completes on it's own

[ASeidelt]

Hi,
we want to implement a system that does spawn multiple threads during its lifetime. Some of them are terminated by other threads, most of them exit the entry function after completion. Several threads may be started and stopped multiple times during the life time of the system.
We are using ThreadX V6.2.0, target is STM32 ARM.

When reading the documentation and analyzing the behavior of ThreadX with test code we have made the following observations:

Common:

• There is a callback mechanism using thread_entry_exit_notify() which does notify of thread entry and exit event.
• No matter how the thread is terminated, thread_entry_exit_notify() is called with type==TX_THREAD_EXIT (0x01).
• No matter how the thread is terminated, it remains in the list of known threads (checked with ThreadX Thread list view in Eclipse).
• To remove the thread from the list of known threads tx_thread_delete() has to be called.
• The stack for the threads needs to be freed 'manually' after tx_thread_delete() has been called.
• The thread_entry_exit_notify() callback is executed with preemption disabled (_tx_thread_shell_entry) or disabled IRQs (depending on TX_NOT_INTERRUPTABLE settings).

TERMINATE:

• If a thread is terminated by calling tx_thread_terminate(), thread_entry_exit_notify() executes in the context of the thread calling tx_thread_terminate().
• It uses the stack of the calling thread to execute thread_entry_exit_notify().
• The state of the thread is TX_TERMINATED (0x02) when the thread_entry_exit_notify() function is called.
• If a thread is terminated by calling tx_thread_terminate() its state is TERMINATED.
• This behavior was expected by us.

COMPLETE:

• If a thread terminates by leaving the entry function, thread_entry_exit_notify() executes in the context of the completing thread.
• It uses the stack of the completing thread to execute thread_entry_exit_notify().
• The state of the thread is TX_COMPLETED (0x01) when the thread_entry_exit_notify() function is called.
• If a thread terminates by leaving the entry function its state is COMPLETED.
• This is the problematic behavior for our use case.

We now have the following questions:

• How do we reclaim the stack/memory of completed threads? If we free the memory in thread_entry_exit_notify() and call tx_thread_delete() this only works for threads that were terminated by calling tx_thread_terminate(). If a thread exits by leaving its entry function the thread_entry_exit_notify() function is executed in the context and using the stack of the exiting thread. A thread freeing its own stack sounds like a dangerous operation.
• If thread_entry_exit_notify() is the wrong place to free stack/thread memory, what is the recommended procedure to do so?
• Is a periodic 'garbage collection' of completed/terminated threads the way to go? What is the suggested way to iterate the known threads? Using tx_thread_info_get() and the next_thread-pointer?
• Would it be possible to add an example to ThreadX/update the documentation that covers the stack management in more detail?
• Is the following code for a thread_entry_exit_notify() callback considered safe in the context of the current ThreadX architecture?

void fi_init_thread_exit_notify_and_deinit(TX_THREAD *thread, UINT cond)
{
  if(thread && cond==TX_THREAD_EXIT) {
    tx_thread_delete(thread);  // remove thread from list
    tx_byte_release(thread->tx_thread_stack_start); // free stack
  }
}

regards
Andre