scrapyd compatibility issue

[ebergam]

Hi @datawizard1337 , is this repo still maintained?
I quickly tried to run it both on windows and linux and it broke down in different ways.

Should it work out of the box still or it needs maintenance?

[cordoba27]

Hi @ebergam, yes, this repository is still being maintained.
Could you please specify the problems that you encountered?

[ebergam]

Hi @cordoba27 , thanks for the quick response.
Testing on Windows10 x64, Python v3.6.
The .exe application doesn't open it right away.

I have tried debugging with ARGUS.py, and the problem is line 37:
script_dir = os.path.dirname(__file__)

It doesn't catch correctly the basepath, and returns empty value (same probelm with nogui.py)

Changing it to:
script_dir = os.path.dirname(os.path.abspath(__file__)) solves the issue, and opens GUI.

After creating the split files, tries to open the web server unsuccessfully (refuses connection):

I am probably running in some compatibility issues?

[ebergam]

So I reinstalled Python, the relative paths error is solved, but it still fails to connect to local server still fails to load.
Any idea about what it might be?

[datawizard1337]

Are you able to start the scrapyd server manually? Try the command "scrapyd" in a separate command prompt please.

[ebergam]

Hey thanks, yes indeed the problem was in the scrapyd + twisted versions.
I have reinstalled and now it's working well on the scraping part, it starts the server and shoots the requests.

Surely there is some more scrapyd error, as I read in your intro page, as the .jl file is empty and there's no output in the file where it reads the URL list. Output is only in the "chunks" subfolder, and postprocessing yields:

Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Users\Enrico\AppData\Local\Programs\Python\Python39\lib\tkinter\__init__.py", line 1884, in __call__
    return self.func(*args)
  File "C:\Users\Enrico\Desktop\ARGUS-master\ARGUS-master\ARGUS.py", line 535, in start_postprocessing
    subprocess.run(r"TSKILL scrapyd")
  File "C:\Users\Enrico\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 501, in run
    with Popen(*popenargs, **kwargs) as process:
  File "C:\Users\Enrico\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 947, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Users\Enrico\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 1416, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] Impossibile trovare il file specificato

So there's probably still some conflict there?

By the way, are you planning a linux package along the Mac version?
I think this work is terrific, thanks for making it open source :)

[cordoba27]

Actually, the output that you find in the 'chunks' folder is the relevant information. The post-processing function only merges all the output chunks into a singular file. I guess there might be another path-related problem that prevents the post-processing script to work on your computer.
Right now, there are unfortunately no plans regarding a linux package or a Mac version.

[ebergam]

Indeed, adding:
subprocess.run(r"TSKILL scrapyd", shell=True) to subprocess.run(r"TSKILL scrapyd"), solves the problem, and the postprocessing works. Thanks a lot!