squid-ecap-pagetitle: A C++ repository from redneon

##Squid eCAP PageTitle Logger Module

adapter_pagetitle.cc

v0.1-1 (2016.06.15)

by red_neon https://github.com/redneon

Squid eCAP PageTitle Logger Module

based on adapter_modifying.cc (ecap_adapter_sample-1.0.0) http://e-cap.org/

Logging of titles of html pages into log-file via eCap module for Squid.
Support of pages, which compressed via Gzip or Deflate. And you can enable to keep only these types of compress (see squid.conf settings)
Support of page titles with JSON/JSONP callback on sites:

google.
yandex.
go.mail.ru
youtube.

Support of pages with codepages: UTF-8, Windows-1251 (all another will be saved as is)
Support of chunked pages (limit 64000 bytes) See README for more information.

Sources:

http://www.squid-cache.org/

http://e-cap.org/

https://github.com/danielaparker/jsoncons

https://github.com/Iyamoto/iconv-lite

http://windrealm.org/tutorials/decompress-gzip-stream.php

[RUS]

Модуль для журналирования заголовков <title>ЗАГОЛОВОК</title> на страницах html в Squid.

умеет читать страницы, сжатые с помощью gzip и deflate.
поддерживает изменение заголовка запросов "Accept-Encoding" на понятные адаптеру методы сжатия. Если там будет что то отличное от gzip, deflate, identity, то лишнее будет вычищено
поддерживает поиск заголовков внутри JSON/JSONP на сайтах гугла, яндекса мейл.ру, ютюба.
умеет собирать chunked ответы (лимит 64000 байт) без задержек.
поддерживает конвертирование cp1251 в utf-8 на основе выдаваемого типа содержимого в заголовке ответа.

В коде полно комментов и дебаг принтов, позже возможно будет почищено.

INSTALLATION

Required libecap v1.0.*

If you don't have installed libecap v1.0.*:

Debian wheezy/jessie:

echo "deb-src http://http.debian.net/debian testing main" >> /etc/apt/sources.list.d/testing.list \
  && apt-get build-dep -y libecap \
  && apt-get purge libecap2 \
  && apt-get update \
  && cd /usr/src \
  && apt-get source libecap3/testing \
  && cd /usr/src/libecap-*/ \
  && dpkg-buildpackage -us -uc -nc -d \
  && cd /usr/src/ \
  && dpkg -i $(ls libecap3*.deb)

Debian stretch:

apt-get install libecap3

This project include jsoncons libs v0.99.1. You can get last version of jsoncons here https://github.com/danielaparker/jsoncons and put src/jsoncons and src/jsoncons_ext into src/

cd squid_ecap_pagetitle_module/
./configure CXXFLAGS=-O3
 make
 make install

You can find the adapter in /usr/local/lib/ecap_adapter_pagetitle.so

If you want to re-create autoconf files: autoreconf -vfi

SETTINGS

Don't forget to change setting of "logfile" in ecap_service, default: logfile=/var/log/squid3/page_titles.log

Example of settings in squid.conf configuration file

# send needed info from squid into pagetitle adapter
adaptation_send_client_ip on
adaptation_send_username on
adaptation_meta X-Squid-Timestamp "%ts.%03tu"
# you can add into X-PageTitle-Custom header all what you want to see in PageTitle logs. This info will shows on end of line.
adaptation_meta X-PageTitle-Custom "referer %{Referer}>h \"%{User-Agent}>h\" %>eui"

ecap_enable on
loadable_modules /usr/local/lib/ecap_adapter_pagetitle.so
ecap_service eRespmod respmod_precache bypass=on uri=ecap://example.com/ecap_pagetitle logfile=/var/log/squid3/page_titles.log
acl http_status_ok http_status 200
adaptation_access eRespmod allow http_status_ok
## eReqmod is needed for changing of header "Accept-Encoding". This will be keep only: gzip, deflate, identity values.
## You can to remove lines below, but for example, google.com uses "sdch" compression, which the adapter does not know.
ecap_service eReqmod reqmod_precache bypass=on uri=ecap://example.com/ecap_pagetitle logfile=/var/log/squid3/page_titles.log
acl is_web_proto proto HTTP HTTPS
adaptation_access eReqmod allow is_web_proto

LOGS

How to read lines in log:

timestamp client_ip username domain "page_title" url "content_type" custom_meta

Example:

1465966583.025 192.168.111.100 - www.google.ru "Google" https://www.google.ru/ "text/html; charset=UTF-8" referer https://www.google.ru/ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36" 0b:0a:22:13:53:13

LOGROTATE

You must do "squid -k reconfigure" after rotation of log, and only "mv" (not rm). if you deleted log with "rm", then you must restart the Squid.