WebCrawler
-
Created a basic Spider to find specific target based on the pattern
-
Expaned the Spider to do a BFS search for similiar attrs
-
Created an enumerated script based on the url pattern i have found
-
Apply multiprocessing to the script since i think it's an I/O bound task
-
Implemented a @timeit decorator and make it pickleable for multiprocessing
-
CPU analysis when querying 50 url and download images on a 4-core CPU The up-down looks reall nice and has a harmony in it!
