Hayabusa

Hayabusa: A Simple and Fast Full-Text Search Engine for Massive System Log Data

Concept

Pure python implement
Parallel SQLite processing engine
SQLite3 FTS(Full Text Search)
Core-scale architecture

Architecture

Design of the directory structure
- By specifying a search range of time in ”the directory path + yyyy + mm + dd + hh + min.db”, the search program can select the search time systematically.
```
/targetdir/yyyy/mm/dd/hh/min.db
```

StoreEngine

sample code

import os.path import sqlite3
db_file = ’test.db’ log_file = ’1m.log’

if not os.path.exists(db_file):
    conn = sqlite3.connect(db_file) conn.execute("CREATE VIRTUAL TABLE SYSLOG USING FTS3(LOGS)");
    conn.close()
conn = sqlite3.connect(db_file)

with open(log_file) as fh:
    lines = [[line] for line in fh] 
    conn.executemany(’INSERT INTO SYSLOG VALUES ( ? )’, lines) 
    conn.commit()

SearchEngine

sample command

$ python search_engine.py -h
usage: search_engine.py [-h] [--time TIME] [--match MATCH] [-c] [-s] [-v]

optional arguments:
  -h, --help     show this help message and exit
  --time TIME    time explain regexp(YYYY/MM/DD/HH/MIN). eg: 2017/04/27/10/*
  --match MATCH  matching keyword. eg: noc or 'noc Login'
  -e             exact match
  -c             count
  -s             sum
  -v             verbose
 
 $ python search_engine.py --time 2017/05/11/13/* --match 'keyword' -c

Architecture image

Search condition

case-insensitive
- no distinguish uppercase or lowercase
Exact match
```
-e --match '192.168.0.1'
```
AND
```
--match 'Hello World'
```
OR
```
--match 'Hello OR World'
```
NOT
```
--match 'Hello World -Wide'
```

PHRASE

--match '"Hello World"'
--match '\"192.168.0.1\"' <- IP address case(same as -e flag)
--match '\"192.168.0.1\" src sent' <- PHRASE + AND search

asterisk(*)
```
--match 'H* World'
```
HAT
```
--match '^Hello World'
```

Development environment

CentOS 7.3
Python 3.5.1(use anaconda packages)
SQLite3(version 3.9.2)

Dependency softwares

Python 3
SQLite3
GNU Parallel

Benchmark

Compare with Apache Spark

Hayabusa and Spark time comparison
Comarison of distributes Spark environment and the stand-alone Hayabusa

hirolovesbeer/hayabusa

Hayabusa

Concept

Architecture

Search condition

Development environment

Dependency softwares

Benchmark

Compare with Apache Spark