How much time to train from all 3 data set ?

Question

How much time to train from all 3 data set ?

phpmind opened this issue 7 years ago · 8 comments

phpmind commented 7 years ago

Hi
How much time it takes to train from all data sets.

Ubuntu Dialogue Corpus:

Cornell Movie-Dialogs:

Reddit comments

Also, how do you combine all of there 3 data sets and how to prove that they work together.

Thanks

Answer 1 · 2017-10-12T17:48:44.000Z

That is going to be very hardware-dependent, especially on what GPU you are using. On a GTX 1080, training starts to converge/plateau after about 8-10 hours.

Also, how do you combine all of there 3 data sets and how to prove that they work together.

I'm not sure what you mean by this. If you want to combine the datasets, you could simply concatenate the files together across datasets by name. For example, assuming you're running on a Mac/Linux machine, you could get the train_from file saved to train_from_combined.txt (just an example) by running:

cat cornell/train_from.txt reddit/train_from.txt ubuntu/train_from.txt > train_from_combined.txt

Just make sure you do the same order for each of the four files (the order above is cornell, reddit, ubuntu). That should be all that's needed. I haven't tried doing this, but I doubt the results would be that great considering each dataset has quite different content. Might be fun to play around with though.

If I misunderstood/didn't answer your questions, please let me know and I'll do my best to clarify.

Answer 2 · 2017-10-12T18:15:45.000Z

Hi Brandon,
Thanks for the great and detailed answer.
I have 1080Ti, so it will take less time to train, I guess!
I am a beginner to please be patient with me :)

I have another question how to extend this chatbot to selecting some query and returning appropriate reply. I have tons of data and really a lot of Elasticsearch indexes which have different kind of data set, they are my source to query.

Now, I would like to make this the way Wit.AI, APi.AI or AWS Lex works. They take intend and entities and based on that they decide what to do. And we can define alot of them.

This chatbot is great for common question answering but I need a bit more which is the database query integration.

Thanks

Answer 3 · 2017-10-12T18:26:43.000Z

I haven't tried setting this up with Elasticsearch, so I can't help much there. However, in terms of using a model after it is trained, have you looked through the webpage directory? If you're familiar with Flask and SQLAlchemy, it shows the code I wrote for the corresponding website to host the models (taken down because money). Specifically, I think web_bot.py and main/views.py within the deepchat folder here might be a good reference point.

Happy to help with specifics, but like I said I don't have much experience with the services you listed. Sorry!

Also, although the approach I took in this project for freezing/hosting the models was the "standard" approach when I wrote it, there have since been more developments in the tf.saved_model module and the TensorFlow Serving framework that you may want to explore. I may get around to writing some code for this project that uses TensorFlow Serving, since I think their documentation/examples is somewhat poor, and folks could benefit from seeing more working/advanced use cases.

Answer 4 · 2017-10-12T18:35:28.000Z

Brandon - I have some experience with Elasticsearch and other tools you have mentioned I just wanted to know your input querying any kind of database either elastic search or something else.
But this is very interesting project you have and I am willing to extend this with some good ideas :)
Let me play with this and run what you have given to me!
I will talk to you soon.
Thanks Again for your quick reply.
I am your fan now.
Om

Answer 5 · 2017-10-12T18:37:50.000Z

Thanks! I've been meaning to update/polish some parts of the project, as tensorflow has been changing quite rapidly from month-to-month. I'd love to see any extensions you make, and remember that contributions are welcome! Best of luck.

Answer 6 · 2017-10-12T18:39:25.000Z

Thanks Brandon 👍

I am very new but willing to take this challenge!

Answer 7 · 2017-10-13T02:07:25.000Z

I would like also to work on it. Please guide me too. Is there the data that will get to run make a simple chatbot. My target to make 1 chatbot that works to answer a question using some FAQ. And the conversation more to normally talk conversation. Is it possible?

Answer 8 · 2017-10-13T02:13:42.000Z

Hi @timbezaz , have you been able to run a chatbot with the data on my MEGA account? There is a link to it in the beginning of the project README.

You can certainly train a bot to memorize a FAQ, although the models in this project are designed for more sophisticated use cases than simple QA lookups. It may be a good starter check to ensure you're setup correctly.

Also, as mentioned in the README, please let me know if you're having any trouble getting the project up and running. I'd like it to be as stress-free as possible.