jakim/ig-monitoring

Collecting data... Please come back tomorrow.

suparub opened this issue · 35 comments

Hi Jakim and thanks for the great work. I configured the monitoring with Blaze socks5 proxies (3 different) and after 3 days I still see this message on the 3 Instagram account I have added. Any idea of what's wrong ? Everything went well during the installation process.

Thanks in advance for you help !

jakim commented

Hi @RubensBeN
Sorry for the late response.
The system does not work with this kind of proxy, I think that's the problem.

I tried with no luck the http and sock5 proxy from Blazing (service recommended from the application)

happy to be enlighted as I can configure the right type of proxy to use :)

jakim commented

@RubensBeN I just checked the fresh installation, the software works as it should. I used a dedicated proxy from BlazingSeo.
Check if the worker is well set and working, you should have a log file in the directory PROJECT_FULL_PATH/runtime/logs/supervisor.log.

Let me know how it went.

Alright Ive checked the worker and everything seems fine except this log. In order for me to backtest, I would like to know : how long does it takes to gather information of a ~1k follower account by experience ? "come back tomorrow" is a bit too wide 😅

jakim commented

Check whether the supervisor is running, see the supervisor documentation for instructions.
If you were doing the installation exactly as in README, then the data should appear after an hour (see the last point in the worker configuration

alright I realized havent setup the the cron commands, I will update you on this. But even having launched the commands manually, I don't see any supervisor.log in the /runtime/logs/ folder. Here is more :

/home/suparub# service supervisor status

● supervisor.service - LSB: Start/stop supervisor
Loaded: loaded (/etc/init.d/supervisor)
Active: active (running) since Wed 2019-02-06 07:44:21 CET; 4 days ago
CGroup: /system.slice/supervisor.service
└─9905 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/supervisord.conf

Feb 06 07:44:21 systemd[1]: Started LSB: Start/stop supervisor.

nano /etc/supervisor/supervisord.conf

; supervisor config file

[unix_http_server]
file=/var/run/supervisor.sock ; (the path to the socket file)
chmod=0700 ; sockef file mode (default 0700)

[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
childlogdir=/var/log/supervisor ; ('AUTO' child log dir, default $TEMP)

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL for a unix socket

[program:yii-queue-worker]
process_name=%(program_name)s_%(process_num)02d
command=/usr/bin/php7.1 /home/admin/web/projectpath/public_html/ig-monitoring/yii queue/listen --verbose=1 --color=0
autostart=true
autorestart=true
user=nginx
numprocs=2
redirect_stderr=true
stdout_logfile=/home/admin/web/projectpath/public_html/ig-monitoring/runtime/logs/supervisor.log

[include]
files = /etc/supervisor/conf.d/*.conf

php7.1 /home/admin/web/projectpath/public_html/ig-monitoring/yii stats/update-accounts
OK!

php7.1 /home/admin/web/projectpath/public_html/ig-monitoring/yii stats/update-tags
OK!

/usr/bin/php7.1 /home/admin/web/projectpath/public_html/ig-monitoring/yii queue/listen --verbose=1 --color=0
yii\base\Exception: No proxy available.
2019-02-10 08:18:06 [262] app\jobs\AccountUpdate (attempt: 1, pid: 27682) - Started
2019-02-10 08:18:06 [262] app\jobs\AccountUpdate (attempt: 1, pid: 27682) - Error (0.226 s)
yii\db\Exception: SQLSTATE[22008]: Datetime field overflow: 1441 Datetime function: datetime field overflow
2019-02-10 08:18:06 [263] app\jobs\AccountUpdate (attempt: 1, pid: 27682) - Started
2019-02-10 08:18:06 [263] app\jobs\AccountUpdate (attempt: 1, pid: 27682) - Error (0.011 s)
yii\base\Exception: No proxy available.
2019-02-10 08:18:06 [264] app\jobs\AccountUpdate (attempt: 1, pid: 27682) - Started
2019-02-10 08:18:06 [264] app\jobs\AccountUpdate (attempt: 1, pid: 27682) - Error (0.154 s)
yii\db\Exception: SQLSTATE[22008]: Datetime field overflow: 1441 Datetime function: datetime field overflow
2019-02-10 08:18:06 [265] app\jobs\AccountUpdate (attempt: 1, pid: 27682) - Started
2019-02-10 08:18:06 [265] app\jobs\AccountUpdate (attempt: 1, pid: 27682) - Error (0.013 s)
etc.......

Alright, I guess I have found the issue :

root@sd-93047:/home/admin/conf/web# /home/admin/web/projectdir/public_html/ig-monitoring/yii queue/listen --verbose=1 --color=0
Failed loading /usr/lib/php/20160303/ioncube_loader_lin_7.1.so: /usr/lib/php/20160303/ioncube_loader_lin_7.1.so: undefined symbol: zend_interrupt_function

2019-02-12 14:12:26 [pid: 6779] - Worker is started
2019-02-12 14:12:26 [pid: 6779] - Worker is stopped (0:00:00)
Exception 'ParseError' with message 'syntax error, unexpected '?', expecting variable (T_VARIABLE)'

in /home/admin/web/projectdir/public_html/ig-monitoring/vendor/symfony/process/Process.php:140

Stack trace:
#0 /home/admin/web/projectdir/public_html/ig-monitoring/vendor/composer/ClassLoader.php(322): Composer\Autoload\includeFile('/home/admin/web...')
#1 [internal function]: Composer\Autoload\ClassLoader->loadClass('Symfony\Compone...')
#2 /home/admin/web/projectdir/public_html/ig-monitoring/vendor/yiisoft/yii2-queue/src/cli/Command.php(183): spl_autoload_call('Symfony\Compone...')
#3 /home/admin/web/projectdir/public_html/ig-monitoring/vendor/yiisoft/yii2-queue/src/cli/Command.php(125): yii\queue\cli\Command->handleMessage('15', 'O:22:"app\jobs\...', '300', 4)
#4 [internal function]: yii\queue\cli\Command->yii\queue\cli{closure}('15', 'O:22:"app\jobs\...', '300', 4)
#5 /home/admin/web/projectdir/public_html/ig-monitoring/vendor/yiisoft/yii2-queue/src/cli/Queue.php(144): call_user_func(Object(Closure), '15', 'O:22:"app\jobs\...', '300', 4)
#6 /home/admin/web/projectdir/public_html/ig-monitoring/vendor/yiisoft/yii2-queue/src/drivers/db/Queue.php(83): yii\queue\cli\Queue->handleMessage('15', 'O:22:"app\jobs\...', '300', 4)
#7 [internal function]: yii\queue\db\Queue->yii\queue\db{closure}(Object(Closure))
#8 /home/admin/web/projectdir/public_html/ig-monitoring/vendor/yiisoft/yii2-queue/src/cli/Queue.php(117): call_user_func(Object(Closure), Object(Closure))
#9 /home/admin/web/projectdir/public_html/ig-monitoring/vendor/yiisoft/yii2-queue/src/drivers/db/Queue.php(93): yii\queue\cli\Queue->runWorker(Object(Closure))
#10 /home/admin/web/projectdir/public_html/ig-monitoring/vendor/yiisoft/yii2-queue/src/drivers/db/Command.php(76): yii\queue\db\Queue->run(true, 3)
#11 [internal function]: yii\queue\db\Command->actionListen(3)
#12 /home/admin/web/projectdir/public_html/ig-monitoring/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#13 /home/admin/web/projectdir/public_html/ig-monitoring/vendor/yiisoft/yii2/base/Controller.php(157): yii\base\InlineAction->runWithParams(Array)
#14 /home/admin/web/projectdir/public_html/ig-monitoring/vendor/yiisoft/yii2/console/Controller.php(148): yii\base\Controller->runAction('listen', Array)
#15 /home/admin/web/projectdir/public_html/ig-monitoring/vendor/yiisoft/yii2/base/Module.php(528): yii\console\Controller->runAction('listen', Array)
#16 /home/admin/web/projectdir/public_html/ig-monitoring/vendor/yiisoft/yii2/console/Application.php(180): yii\base\Module->runAction('queue/listen', Array)
#17 /home/admin/web/projectdir/public_html/ig-monitoring/vendor/yiisoft/yii2/console/Application.php(147): yii\console\Application->runAction('queue/listen', Array)
#18 /home/admin/web/projectdir/public_html/ig-monitoring/vendor/yiisoft/yii2/base/Application.php(386): yii\console\Application->handleRequest(Object(yii\console\Request))
#19 /home/admin/web/projectdir/public_html/ig-monitoring/yii(20): yii\base\Application->run()
#20 {main}
root@sd-93047:/home/admin/conf/web#

found the solution, my default php command version was outdated. Keeping you updated here :)

well not really finally, still happening with the right defaut php version :

php -v
PHP 7.1.26-1+020190113101856.12+jessie1.gbp7077bb (cli) (built: Jan 13 2019 15:14:34) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2018 Zend Technologies
with the ionCube PHP Loader (enabled) + Intrusion Protection from ioncube24.com (unconfigured) v10.3.2, Copyright (c) 2002-2018, by ionCube Ltd.
with Zend OPcache v7.1.26-1+020190113101856.12+jessie1.gbp7077bb, Copyright (c) 1999-2018, by Zend Technologies

any direction ?

jakim commented

Not really.

What happens when you manually run yii queue/listen -v=1?
Jobs are done correctly?

Got this

/home/admin/web/projectpath/public_html/ig-monitoring/yii queue/listen -v=1
Failed loading /usr/lib/php/20160303/ioncube_loader_lin_7.1.so: /usr/lib/php/20160303/ioncube_loader_lin_7.1.so: undefined symbol: zend_interrupt_function
2019-02-12 17:52:24 [pid: 25042] - Worker is started
2019-02-12 17:52:24 [pid: 25042] - Worker is stopped (0:00:00)
Exception 'ParseError' with message 'syntax error, unexpected '?', expecting variable (T_VARIABLE)'

in /home/admin/web/projectpath/public_html/ig-monitoring/vendor/symfony/process/Process.php:140

Stack trace:
#0 /home/admin/web/projectpath/public_html/ig-monitoring/vendor/composer/ClassLoader.php(322): Composer\Autoload\includeFile('/home/admin/web...')
#1 [internal function]: Composer\Autoload\ClassLoader->loadClass('Symfony\Compone...')
#2 /home/admin/web/projectpath/public_html/ig-monitoring/vendor/yiisoft/yii2-queue/src/cli/Command.php(183): spl_autoload_call('Symfony\Compone...')
#3 /home/admin/web/projectpath/public_html/ig-monitoring/vendor/yiisoft/yii2-queue/src/cli/Command.php(125): yii\queue\cli\Command->handleMessage('3', 'O:22:"app\jobs\...', '300', 7)
#4 [internal function]: yii\queue\cli\Command->yii\queue\cli{closure}('3', 'O:22:"app\jobs\...', '300', 7)
#5 /home/admin/web/projectpath/public_html/ig-monitoring/vendor/yiisoft/yii2-queue/src/cli/Queue.php(144): call_user_func(Object(Closure), '3', 'O:22:"app\jobs\...', '300', 7)
#6 /home/admin/web/projectpath/public_html/ig-monitoring/vendor/yiisoft/yii2-queue/src/drivers/db/Queue.php(83): yii\queue\cli\Queue->handleMessage('3', 'O:22:"app\jobs\...', '300', 7)
#7 [internal function]: yii\queue\db\Queue->yii\queue\db{closure}(Object(Closure))
#8 /home/admin/web/projectpath/public_html/ig-monitoring/vendor/yiisoft/yii2-queue/src/cli/Queue.php(117): call_user_func(Object(Closure), Object(Closure))
#9 /home/admin/web/projectpath/public_html/ig-monitoring/vendor/yiisoft/yii2-queue/src/drivers/db/Queue.php(93): yii\queue\cli\Queue->runWorker(Object(Closure))
#10 /home/admin/web/projectpath/public_html/ig-monitoring/vendor/yiisoft/yii2-queue/src/drivers/db/Command.php(76): yii\queue\db\Queue->run(true, 3)
#11 [internal function]: yii\queue\db\Command->actionListen(3)
#12 /home/admin/web/projectpath/public_html/ig-monitoring/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#13 /home/admin/web/projectpath/public_html/ig-monitoring/vendor/yiisoft/yii2/base/Controller.php(157): yii\base\InlineAction->runWithParams(Array)
#14 /home/admin/web/projectpath/public_html/ig-monitoring/vendor/yiisoft/yii2/console/Controller.php(148): yii\base\Controller->runAction('listen', Array)
#15 /home/admin/web/projectpath/public_html/ig-monitoring/vendor/yiisoft/yii2/base/Module.php(528): yii\console\Controller->runAction('listen', Array)
#16 /home/admin/web/projectpath/public_html/ig-monitoring/vendor/yiisoft/yii2/console/Application.php(180): yii\base\Module->runAction('queue/listen', Array)
#17 /home/admin/web/projectpath/public_html/ig-monitoring/vendor/yiisoft/yii2/console/Application.php(147): yii\console\Application->runAction('queue/listen', Array)
#18 /home/admin/web/projectpath/public_html/ig-monitoring/vendor/yiisoft/yii2/base/Application.php(386): yii\console\Application->handleRequest(Object(yii\console\Request))
#19 /home/admin/web/projectpath/public_html/ig-monitoring/yii(20): yii\base\Application->run()
#20 {main}

figured it out, it was the defaut php version that was 7.0 and not 7.1, Here to change to future ones facing the problem : https://tecadmin.net/switch-between-multiple-php-version-on-debian/

Now is what I have :

yii\base\Exception: No proxy available.
2019-02-12 18:07:32 [572] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Started
2019-02-12 18:07:32 [572] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Error (0.012 s)
yii\base\Exception: No proxy available.
2019-02-12 18:07:32 [573] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Started
2019-02-12 18:07:32 [573] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Error (0.012 s)
yii\base\Exception: No proxy available.
2019-02-12 18:07:32 [574] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Started
2019-02-12 18:07:33 [574] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Error (0.011 s)
yii\base\Exception: No proxy available.
2019-02-12 18:07:33 [575] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Started
2019-02-12 18:07:33 [575] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Error (0.011 s)
yii\base\Exception: No proxy available.
2019-02-12 18:07:33 [576] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Started
2019-02-12 18:07:33 [576] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Error (0.012 s)
yii\base\Exception: No proxy available.
2019-02-12 18:07:33 [577] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Started
2019-02-12 18:07:33 [577] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Error (0.011 s)
yii\base\Exception: No proxy available.
2019-02-12 18:07:33 [578] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Started
2019-02-12 18:07:33 [578] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Error (0.012 s)
yii\base\Exception: No proxy available.
2019-02-12 18:07:33 [579] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Started
2019-02-12 18:07:33 [579] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Error (0.012 s)
yii\base\Exception: No proxy available.
2019-02-12 18:07:33 [580] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Started
2019-02-12 18:07:33 [580] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Error (0.011 s)
yii\base\Exception: No proxy available.
2019-02-12 18:07:33 [581] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Started
2019-02-12 18:07:33 [581] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Error (0.012 s)
yii\base\Exception: No proxy available.
2019-02-12 18:07:33 [582] app\jobs\AccountUpdate (attempt: 1, pid: 26987) - Started

jakim commented

Probably all proxies have been reserved and not released.
Clear queue:
yii queue/clear
and in the database console run:
update proxy set reservation_uid = null;

alright thanks, things seems to be better even if I got an error on the gui :

" Invalid account
type: Unknown reason
attempts: 1
next try: Feb 13, 2019, 9:18:55 AM"

on the terminal :

root@sd-93047:/home/suparub# /home/admin/web/projectpath/public_html/ig-monitoring/yii queue/listen -v=1
2019-02-13 08:13:31 [pid: 8771] - Worker is started
2019-02-13 08:14:34 [628] app\jobs\AccountUpdate (attempt: 1, pid: 8771) - Started
2019-02-13 08:16:43 [628] app\jobs\AccountUpdate (attempt: 1, pid: 8771) - Error (129.306 s)

yii\db\Exception: SQLSTATE[22008]: Datetime field overflow: 1441 Datetime function: datetime field overflow
2019-02-13 08:16:43 [629] app\jobs\AccountUpdate (attempt: 1, pid: 8771) - Started
2019-02-13 08:18:55 [629] app\jobs\AccountUpdate (attempt: 1, pid: 8771) - Done (131.016 s)
2019-02-13 08:18:55 [630] app\jobs\AccountUpdate (attempt: 1, pid: 8771) - Started
2019-02-13 08:18:55 [630] app\jobs\AccountUpdate (attempt: 1, pid: 8771) - Done (0.208 s)
2019-02-13 08:19:37 [628] app\jobs\AccountUpdate (attempt: 2, pid: 8771) - Started

jakim commented

it's probably related to account/tag invalidation_count.
The easiest way is to clear the database and add accounts and tags again, if you can not, start with:
update account set invalidation_count=null, update_stats_after=null where invalidation_count> 0;
update tag set invalidation_count=null, update_stats_after=null where invalidation_count> 0;

alright, well executed but still no log and an uknown yellow error in the gui :

root/home/suparub# /home/admin/web/project/public_html/ig-monitoring/yii queue update tag set invalidation_count=null, update_stats_after=null where invalidation_count> 0;

root/home/suparub# /home/admin/web/project/public_html/ig-monitoring/yii queue update account set invalidation_count=null, update_stats_after=null where invalidation_count> 0;

root/home/suparub# /home/admin/web/project/public_html/ig-monitoring/yii queue /home/admin/web/project/public_html/ig-monitoring/yii queue/update account set invalidation_count=null, update_stats_after=null where invalidation_count> 0;

root/home/suparub# /home/admin/web/project/public_html/ig-monitoring/yii queue update proxy set reservation_uid = null;
Jobs

  • waiting: 0
  • delayed: 0
  • reserved: 0
  • done: 0

root/home/suparub# /home/admin/web/project/public_html/ig-monitoring/yii queue/listen -v=1
2019-02-14 07:36:12 [pid: 32680] - Worker is started
2019-02-14 07:36:51 [1] app\jobs\AccountUpdate (attempt: 1, pid: 32680) - Started
2019-02-14 07:36:51 [1] app\jobs\AccountUpdate (attempt: 1, pid: 32680) - Done (0.209 s)

Still have this, I have double checked the socks5 proxy and it works great (no user/pass)

2019-02-14 09:00:02 [2] app\jobs\AccountUpdate (attempt: 1, pid: 32680) - Started
2019-02-14 09:00:02 [2] app\jobs\AccountUpdate (attempt: 1, pid: 32680) - Done (0.208 s)
2019-02-14 09:00:02 [3] app\jobs\AccountUpdate (attempt: 1, pid: 32680) - Started
2019-02-14 09:00:02 [3] app\jobs\AccountUpdate (attempt: 1, pid: 32680) - Error (0.011 s)

yii\base\Exception: No proxy available.
2019-02-14 09:00:02 [4] app\jobs\AccountUpdate (attempt: 1, pid: 32680) - Started
2019-02-14 09:00:02 [4] app\jobs\AccountUpdate (attempt: 1, pid: 32680) - Error (0.011 s)
yii\base\Exception: No proxy available.
2019-02-14 09:05:05 [3] app\jobs\AccountUpdate (attempt: 2, pid: 32680) - Started
2019-02-14 09:05:06 [3] app\jobs\AccountUpdate (attempt: 2, pid: 32680) - Done (0.210 s)
2019-02-14 09:05:06 [4] app\jobs\AccountUpdate (attempt: 2, pid: 32680) - Started

jakim commented

update account set... and update proxy set ... these are database commands not yii

update proxy set reservation_uid = null
7 rows affected. (Query took 0.0001 seconds.)
update account set invalidation_count=null, update_stats_after=null where invalidation_count> 0
2 rows affected. (Query took 0.0002 seconds.)
update tag set invalidation_count=null, update_stats_after=null where invalidation_count> 0
3 rows affected. (Query took 0.0002 seconds.)

Then I'm adding new IG accounts :

Invalid account
type: Unknown reason
attempts: 1
next try: Feb 14, 2019, 2:47:34 PM

Same problem, and still no logs created

2019-02-14 15:10:05 [14] app\jobs\AccountUpdate (attempt: 3, pid: 29834) - Error (0.012 s)

yii\base\Exception: No proxy available.

When the proxy added (sock5 without user/pass) works great :/

Hi @jakim , any directions ? I would love to make this work :)

jakim commented

@RubensBeN
I think I already said that.
I did not check the system for a socket proxy, use the http proxy, everything works fine.
As for the supervisor, start by checking the file creation rights or try running it with root privileges.

jakim commented

@RubensBeN any news?

Hi @jakim , no still the same problem this is not working. Honestly I don't believe it comes from the proxy as Ive tried everything.

Everytime :

  • I have emptied the database from the values
  • the cons are giving me OK! signals :
    php7.1 /home/admin/web/thebox/public_html/ig-monitoring/yii stats/update-tags
    php7.1 /home/admin/web/thebox/public_html/ig-monitoring/yii stats/update-accounts
  • still errors >> yii\base\Exception: No proxy available.
jakim commented

I checked the fresh installation according to the readme steps, with a dedicated http proxy from blazing and everything works. It must be related to your environment and unfortunately I can not help you. I leave the task open, maybe someone else will help you.
Good luck

jakim commented

no answer

I have same problem with free proxies, php7.2 :(

jakim commented

Hi @grig191

On https://demo.igmonitoring.com there is beta1 version, private proxy, php 7.3.5, debian and there is no data problem. I keep betting that it's a proxy fault.

You can send me a proxy (ip, username, pass, port, and whatever I need to connect) to admin @ igmonitoring.com, I will check.

I have chosen another proxy and it has updated in exactly one second without errors but there is no data

@jakim is there any way to avoid using proxies? I use only 1-2 accounts from my server so I don't need it and I don't have money to buy paid proxy

jakim commented

I have chosen another proxy and it has updated in exactly one second without errors but there is no data

Probably a cache, each request is kept for an hour. Try to clean it yii cahce/flush-all

is there any way to avoid using proxies? I use only 1-2 accounts from my server so I don't need it and I don't have money to buy paid proxy

No way. You have to program it.

jakim commented

After recent changes in ig, there is no possibility that it works on free proxies. They are always banned in ig.
Using the ones recommended by me guarantees success 😎

https://github.com/jakim/ig-monitoring#before-you-start