ViT for regression task such as Real Estate Price Prediction or Stock Exchange Datasets, any regression dataset.
saifhassan opened this issue · 7 comments
How can we modify ViT Pytorch code to work on regression datasets like stock prediction, real estate price prediction and etc.
How can we modify ViT Pytorch code to work on regression datasets like stock prediction, real estate price prediction and etc.
or any transformer model working on regression task
Hi @saifhassan, instead of modifying the ViT architecture for any prediction task, I would recommend considering the use of TFT (Temporal Fusion Transformers) specifically designed for prediction. Neural networks are often black boxes, making it difficult to understand how the model reached a particular decision. However, TFT comes with interpretable results, meaning that it can explain the reasoning behind its predictions or decisions in detail.
For example, if the TFTs model is forecasting future stock prices, it can provide a comprehensive explanation of how it arrived at the forecast. This could include information on which input features (such as news sentiment, financial reports, or market trends) were most important in making the forecast, how those features were combined to produce the final forecast, and any other relevant information about the model's decision-making process.
By providing such a high level of detail and transparency, the TFTs model can help users understand why it made a particular prediction or decision. This feature can be especially useful in situations where the stakes are high or the decisions are complex. I hope this response answers your question.
I can contribute by adding the feature "regression mode" to the ViT. This could be useful when using images to solve regression problems. (I have implemented it localy in my computer and works well!) Do you accept opensource contributions Mr. @lucidrains ? Thank you.
I can contribute by adding the feature "regression mode" to the ViT. This could be useful when using images to solve regression problems. (I have implemented it localy in my computer and works well!) Do you accept opensource contributions Mr. @lucidrains ? Thank you.
hello @ignaciocordova maybe im wrong, but this could be really interesting for my use case, which is the use of ViT for counting instances in images (output being a single digit). Most of the models that I've checked add a "regression head" that is used to map the results of the ViT to the regression prediction, and I want to know more about this. Could you contribute with this feature you talk about? I would be really interested on checking it for my project. Thank you.
I can contribute by adding the feature "regression mode" to the ViT. This could be useful when using images to solve regression problems. (I have implemented it localy in my computer and works well!) Do you accept opensource contributions Mr. @lucidrains ? Thank you.
Thanks for your response, I have implemented ViT for regression tasks for images.
Hi @saifhassan, instead of modifying the ViT architecture for any prediction task, I would recommend considering the use of TFT (Temporal Fusion Transformers) specifically designed for prediction. Neural networks are often black boxes, making it difficult to understand how the model reached a particular decision. However, TFT comes with interpretable results, meaning that it can explain the reasoning behind its predictions or decisions in detail.
For example, if the TFTs model is forecasting future stock prices, it can provide a comprehensive explanation of how it arrived at the forecast. This could include information on which input features (such as news sentiment, financial reports, or market trends) were most important in making the forecast, how those features were combined to produce the final forecast, and any other relevant information about the model's decision-making process.
By providing such a high level of detail and transparency, the TFTs model can help users understand why it made a particular prediction or decision. This feature can be especially useful in situations where the stakes are high or the decisions are complex. I hope this response answers your question.
yeah, you are right, but i was looking for image-based regression, however, I have solved that by adding regression-based parameters to the same ViT model.
yeah, you are right, but i was looking for image-based regression, however, I have solved that by adding regression-based parameters to the same ViT model.
Can I ask what exactly did you implement as "regression parameters"? I'm looking for something similar and I would be glad to know more about it.