Couldn't do file parsing - file downloading or video parsing issue

Question

Couldn't do file parsing - file downloading or video parsing issue

Closed this issue 9 years ago · 17 comments

Dear Louis,

Your project is great and helpful; thanks.

I always receive this log; is it file downloading issue or parsing issue? Any help please.

FYI.. It is a YouTube video

2014-10-16 14:17:09,778 - lib_general_download.py:610 - INFO - Not FLV
2014-10-16 14:17:09,780 - factory.py:377 - WARNING - Name <open file '/tmp/tmp1EP17r', mode 'r' at 0xa3ea4f0> is not string or buffer
2014-10-16 14:17:09,781 - factory.py:235 - INFO - No Type found by Extension (). Trying all parsers.
2014-10-16 14:17:09,860 - factory.py:241 - DEBUG - trying video/quicktime
2014-10-16 14:17:09,885 - factory.py:241 - DEBUG - trying video/flv
2014-10-16 14:17:09,974 - factory.py:241 - DEBUG - trying video/mpeg
2014-10-16 14:17:09,975 - mpeg.py:611 - INFO - trying mpeg-pes scan
2014-10-16 14:17:10,041 - factory.py:241 - DEBUG - trying video/avi
2014-10-16 14:17:10,042 - lib_general_download.py:78 - DEBUG - mime not found
2014-10-16 14:17:10,044 - lib_general_download.py:818 - DEBUG - Content-length: 1855797.0

Answer 1 · 2014-10-16T12:41:07.000Z

It's fine, it's just that in case of not FLV file the quality estimation is not as precise,so you have this log

You can reduce logging on command line by specifying "-q"

On Thu, Oct 16, 2014 at 1:31 PM, abahnihi notifications@github.com
wrote:

Dear Louis,
Your project is great and helpful; thanks.
I always receive this log; is it file downloading issue or parsing issue? Any help please.
FYI.. It is a YouTube video
2014-10-16 14:17:09,778 - lib_general_download.py:610 - INFO - Not FLV
2014-10-16 14:17:09,780 - factory.py:377 - WARNING - Name <open file '/tmp/tmp1EP17r', mode 'r' at 0xa3ea4f0> is not string or buffer
2014-10-16 14:17:09,781 - factory.py:235 - INFO - No Type found by Extension (). Trying all parsers.
2014-10-16 14:17:09,860 - factory.py:241 - DEBUG - trying video/quicktime
2014-10-16 14:17:09,885 - factory.py:241 - DEBUG - trying video/flv
2014-10-16 14:17:09,974 - factory.py:241 - DEBUG - trying video/mpeg
2014-10-16 14:17:09,975 - mpeg.py:611 - INFO - trying mpeg-pes scan
2014-10-16 14:17:10,041 - factory.py:241 - DEBUG - trying video/avi
2014-10-16 14:17:10,042 - lib_general_download.py:78 - DEBUG - mime not found

2014-10-16 14:17:10,044 - lib_general_download.py:818 - DEBUG - Content-length: 1855797.0

Reply to this email directly or view it on GitHub:
#4

Answer 2 · 2014-10-16T14:17:13.000Z

Thanks a lot. But this makes the result doesn't include the video meta-data details. The following results are not generated:
VideoType, VideoDuration, EncodingRate, DownloadInterruptions, BufferingDuration, PlaybackDuration and BufferDurationAtEnd.

Is it Kaa Meta-Data related issue? Do you have a plan to fix it?

Thanks, Ahmed

Answer 3 · 2014-10-18T10:26:42.000Z

I think the video file is not downloaded where the file size = 0

We are planning to use your pretty good tool for doing the same as you did with NANOG in Egypt. So do you think your code need some update from a quite busy man? Or it is not applicable to follow YouTube updates and codec?

can we do some contribution?

Answer 4 · 2014-10-18T12:06:04.000Z

Can't you use the download URI of flv file instead of the best quality (mp4)? Pafy may do help

Answer 5 · 2014-10-28T05:22:21.000Z

HI All,
I am new to using this tool ,I downloaded the windows exe available at "https://code.google.com/p/pytomo/downloads/detail?name=Pytomo_windows_x86-3.0.3.zip&can=2&q=" but database never gets populated with this so i tried using the source code available on github. I was able to start crawlng also I installed pyrrd 0.1.0 after that server also got started but while trying to open the graphs on the browser it gives attribute error saying rrdtool create attribute is not available.Is there anything that i am missing ? Please help i am unable to fix this.

Although as a workaround I copy the database .db file which gets generated after starting crawling to my exe database folder and then see all the graphs.
Although following results do not gets generated:VideoType, VideoDuration, EncodingRate, DownloadInterruptions, BufferingDuration, PlaybackDuration and BufferDurationAtEnd..
Are you able to fix this?PLease help i am juggling with this tool for so many weeks.

Thanks
Priyanka

Answer 6 · 2014-10-28T05:31:56.000Z

abahnihi,

Just a quick question are you using pytomo on linux or windows and which version of OS .Can u just let me know what all you installed along with there versions to make the server side compile successfully and see the results on the browser without any hassles?

Louis, please find out time to reply to us !
Thanks
Priyanka

Answer 7 · 2014-10-28T10:02:30.000Z

Hi abahnihi,

we don't get the meta-data for non-flv files, so these fields are empty

there is an option to specify best quality instead of normal one, use "-H" or "--hd"
NB: you can find all options with "-h"

also as stated on the google code page, all future developments are done on github, and the exe bundle is given on sourceforge, but it's not maintained any more: I don't have any Windows computer handy and I'm not aware of any working cross-compilation tool for python
anyway, if you intend to use the tool extensively, install python on windows: this is the best option

Louis

Answer 8 · 2014-10-28T10:06:25.000Z

Priyanka,

be sure that the video you are downloading is having flv format

also for rrd to plot values, you need at least 3 downloads: either the same one multiple times or by crawling, find out in the help which options is suiting you best

Louis

Answer 9 · 2014-10-28T11:00:14.000Z

Hey Louis ,

Really Thanks for finding time to reply !

I gave youtube videos in flv format for crawling and specified the URL's in config_pytomo file i.e
STATIC_URL_LIST = ['http://www.youtube.com/watch?v=GLq0EVMGCx8','http://www.youtube.com/watch?v=GLq0EVMGCx8','http://www.youtube.com/watch?v=GLq0EVMGCx8','http://www.youtube.com/watch?v=GLq0EVMGCx8','http://www.youtube.com/watch?v=GLq0EVMGCx8']

even after this i do not see VideoType, VideoDuration, EncodingRate, DownloadInterruptions, BufferingDuration, PlaybackDuration and BufferDurationAtEnd statistics coming up in the graph.I also ensured that atleast three videos are downloaded.
Also i checked at source forge the latest windows exe availabe has issues as database doesn;t get populated .I believe the change that you did in lib_youtube_download.py is not picked up hence resulting in latest exe not working .

Is there any way i can get the latest exe picking up the lib_youtube_download.py fix.

Thanks
Priyanka

Answer 10 · 2014-10-28T11:05:07.000Z

if the video is not FLV you won't have these stats as they come from
meta-data parsing that we do only for FLV

as I told you there is no maintenance any more for the windows exe bundle:
you can generate it yourself with pyinstaller: if it's working I can grant
you upload access to sourceforge

On Tue, Oct 28, 2014 at 11:00 AM, priyaash75 notifications@github.com
wrote:

Hey Louis ,

Really Thanks for finding time to reply !

I gave youtube videos in flv format for crawling and specified the URL's
in config_pytomo file i.e
STATIC_URL_LIST = ['
http://www.youtube.com/watch?v=GLq0EVMGCx8','http://www.youtube.com/watch?v=GLq0EVMGCx8','http://www.youtube.com/watch?v=GLq0EVMGCx8','http://www.youtube.com/watch?v=GLq0EVMGCx8','http://www.youtube.com/watch?v=GLq0EVMGCx8
']

even after this i do not see VideoType, VideoDuration, EncodingRate,
DownloadInterruptions, BufferingDuration, PlaybackDuration and
BufferDurationAtEnd statistics coming up in the graph.I also ensured that
atleast three videos are downloaded.
Also i checked at source forge the latest windows exe availabe has issues
as database doesn;t get populated .I believe the change that you did in
lib_youtube_download.py is not picked up hence resulting in latest exe not
working .

Is there any way i can get the latest exe picking up the
lib_youtube_download.py fix.

Thanks
Priyanka

—
Reply to this email directly or view it on GitHub
#4 (comment).

Answer 11 · 2014-10-28T12:00:44.000Z

but despite this there are no results coming for VideoType, VideoDuration, EncodingRate, DownloadInterruptions, BufferingDuration, PlaybackDuration and BufferDurationAtEnd..
What could be the issue any suggestions?

Answer 12 · 2014-10-28T12:50:46.000Z

No video parsing is supported any more even the flv (I will confirm this on today).

The tricky one is in the following function "get_data_duration" in lib_general_download. I used pafy instead (https://pypi.python.org/pypi/Pafy). This will makes the code be able to calculate the encoding_rate by the function "compute_encoding_rate"

To be honest, I used pafy to get the video_url_list as well in the file lib_youtube_download.

NB. I am not a developer, so may be my terminology is not perfect to you.

BR,
abahnihi

Answer 13 · 2014-10-29T05:00:38.000Z

Hi abahnihi,

Were you able yo get any results ?Plz confirm..

Thanks
Priyanka

Answer 14 · 2014-11-03T13:59:29.000Z

Hi abahnihi,

HI abahnihi,

Were you able yo get any results ?Plz find time to reply to this.I Tried myself few things but could not make video metadata parse and provide the info .

PLease Help!

Thanks
Priyanka

Answer 15 · 2014-11-03T16:54:47.000Z

Hi Priyanka,

Actually I am traveling till the end of this week. What I can say at the
moment, what worked with me is the estimation way, I mean without parsing.
The estimation way means the video bit rate is calculated and assuming it
equally distributed over time.

Actually I am not a developer and I just wanted to proof the concept. So
what I added needs a little extra work to make it looks nice.

If you didn't figure it out till next week, then let me share what I did.

BR, abahnihi
On Nov 3, 2014 3:59 PM, "priyaash75" notifications@github.com wrote:

Hi abahnihi,

HI abahnihi,

Were you able yo get any results ?Plz find time to reply to this.I Tried
myself few things but could not make video metadata parse and provide the
info .

PLease Help!

Thanks
Priyanka

Thanks
Priyanka

—
Reply to this email directly or view it on GitHub
#4 (comment).

Answer 16 · 2014-11-18T19:23:35.000Z

HI Abahnihi,

I am still juggling with this ,Could you please share how you did an estimation of video quality.This might help me with my own analysis.How have you predicted the encoding rates for a SD or HD youtube video ?

Answer 17 · 2014-11-19T10:51:36.000Z

Hi,

I've installed pytomo on a new computer (MacOS X) and it's working perfectly

as mentionned already, Windows exe are not supported any more

so if you don't have a Mac or a Linux host available, install python on
Windows and it should be working fine

also the encoding rates are found in Youtube API

Louis

On Tue, Nov 18, 2014 at 7:23 PM, priyaash75 notifications@github.com
wrote:

HI Abahnihi,

I am still juggling with this ,Could you please share how you did an
estimation of video quality.This might help me with my own analysis.How
have you predicted the encoding rates for a SD or HD youtube video ?

—
Reply to this email directly or view it on GitHub
#4 (comment).