shiny-succotash: A Python repository from kolyaflash

Documentation

See docs for docs to get docs and see docs.

Requirements

python3.6, postgres, redis, sl_mqlib

Install for devs

pip install -r ./requirements/local.txt
APP_CONFIG_MODULE=sgateway.config.local python manage.py run_server

Install on prod

pip install -r ./requirements.txt
export APP_CONFIG_MODULE='sgateway.config.prod'
python ./manage.py credentials decrypt --passphrase 123 --override 
alembic upgrade head
gunicorn wsgi:app --bind 0.0.0.0:8000 --worker-class sanic.worker.GunicornWorker -w 2

Testing

pip install -r ./requirements/tests.txt
pytest ./tests/

DB managment

New revision (migration):

APP_CONFIG_MODULE=sgateway.config.local alembic revision --autogenerate

Apply latest revision (migration):

APP_CONFIG_MODULE=sgateway.config.local alembic upgrade head

Rollback last revision:

APP_CONFIG_MODULE=sgateway.config.local alembic downgrade -1

Configuration

Env vars

LOG_LEVEL (default info)

DB_URL (postgresql://localhost:5432/sgateway_default)

INTERNAL_GATEWAY_KEY

REDIS_HOST (localhost) REDIS_PORT (default 6379) REDIS_DB (default 0)

MESSAGE_BUS_AMQP_URL (Semilimes Message Bus RabbitMQ)

DOCS_API_URL (https://docs.semilimes.info)

Credentials vars

Just refer to vars in local.cfg.

Credentials Config

Credentials (e.g. third party services tokens) are stored in encrypted .cfg files. Filename must follow config module name (e.g. local.cfg for sgateway.config.local).

If you know passphrase for needed credentials config, you can decrypt it and unencrypted .cfg file will be automatically loaded by the app and settings available at app.config.

For example, for local:

APP_CONFIG_MODULE=sgateway.config.local ./manage.py credentials decrypt --passphrase 123

After you added/modified config vars in credentials config, you encrypt it back and commit to repo:

APP_CONFIG_MODULE=sgateway.config.local ./manage.py credentials encrypt --passphrase 123
git add config/credentials/local.cfg.aes && git commit -m 'update credentials config'

Business goals

provide highly available service. for example, we have service for currency exchange rates (Service). so when the request to get latest exchange rates (Method) came in, and fixer.io (Provider) is currently down, we, instead of return error to client, trying to get rates from (currencylayer.com).
cost efficiency. depending on a requester party, we could be in favour of provider A rather than provider B, because calling provider A is cheaper because of, let's say, requester's country of residency. Same time, it can be cheaper to call some provider A for particular task (e.g. convert currency), but expensive for another one (e.g. get all currency rates).
apply charges for requesting service gateway. but the idea is to charge user independently of used provider. so cost efficiency matters, because saved costs is our revenue.
incapsulation. Provides the optimal API for each client. Simplifies the client by moving logic for calling multiple services from the client to API gateway. Translates from a “standard” public web-friendly API protocol to whatever protocols are used internally. Etc.
be analysable. we must know as much metrics of services usage as possible. also, we must be able to easily find a particular request to find out about e.g. it's cost, source and destination.

Architecture overview

Central Config

???

Services Blueprint

sgateway.services.views.ServicesBlueprint is a special Sanic's blueprint that automatically discover services (specified in config), discover gateway middlewares and adds routes with custom handler (that handler runs gateway middlewares) and generated urls.

Services Registry

sgateway.services.registry.ServiceRegistry is a class (singleton) that holds list of services and providers. Registry is also able to provide meta schema of services (used in python client).

ServiceRequest/ServiceResponse

sgateway.services.request.ServiceRequest / sgateway.services.request.ServiceResponse are abstractions that intended to incapsulate different sources: regular HTTP requests, webhooks callbacks, websockets or even Semilimes Message Queue messages. They are used all over underlying levels of request processing starting from Router's handler.

Request Pipeline

Request Pipeline is basically a set of middlewares (sgateway.services.base.middleware.BaseGatewayMiddleware) specified in Blueprint.

Those middlewares works with request/response abstractions and can change request flow or modify response; do extra things, like apply billing costs or check API rate limits.

Service/Provider

Service (sgateway.services.base.service.BaseService) can expose it's methods to be accessible by clients. To do the job, Service do call one of the Service Providers. E.g. Emails is a Service. send is a Service Method and Sendrid is a Provider. Service inderectly returns data to client.

Provider Strategy

Service may require a Provider to do the job. Different Strategies (sgateway.services.base.strategy.BaseProviderChoiceStrategy) can be used to define which Provider is better in taking this particular job.

There is also a few special ways of choosing provider:

sgateway.services.base.service.BaseService#failover_provider_call can try to accomplish result by calling different Providers (sequentially) until one of them finally done without errors.
Always using same Provider when request means not provider-agnostic operations. E.g. if we purchased a domain name from GoDaddy - we should also manage this domain thought GoDaddy.

provider = await self.get_provider(service_request, provider_name='godaddy])

Persistence Layer

Postgres

Can be used for anything, from Core to Providers. aiopg + SQLAlchemy (without ORM obviously) is used to access database.

One can simply use Table and use global metadata, so table will be registered in migrations (just make sure that module containing Model is importing during app startup).

from sqlalchemy import Table, Column, Integer, String
from sgateway.core.db import metadata

UserModel = Table('user', metadata,
    Column('user_id', Integer, primary_key=True),
    Column('user_name', String(16), nullable=False),
    Column('email_address', String(60)),
    Column('password', String(20), nullable=False)
)

# ...

async def handler(app):
	async with app.db.connection() as conn:
		cursor = await conn.execute(UserModel.select())
		user = await cursor.fetch_one()
		return user

Redis

Redis is available via aioredis.

async def handler(app):
	with await app.redis_pool as conn:
		conn.set('key', 'cached val')
	return

Semilimes Message Queue

See sl_mqlib. AsyncioUniversalChannel from sl_mqlib is used.

There is a sgateway.core.mq.GatewayMQHandler that listening for messages on a queue and turn them into Service calls (one way only; no request's responses).

Publisher is available:

from sl_mqlib.serializer import JsonSerializer

async def handler(app):
	data = {'foo': 'bar'}
	app.mq_channel.publish('sl.topic', data, routing_key='test.test', serializer_class=JsonSerializer)

Service Example

class Provider1(BaseServiceProvider):
		
	@provide_method()
	def send_email(self, email):
		self.send_via_http(email)


class Provider2(BaseServiceProvider):
	
	@provide_method()
	def send_email(self, email):
		self.send_via_smtp(email)

class EmailCostSaving(RoundRobinStrategy):

	def select(self, request, providers):
		return self.best_for_this_email(request.get_data()['to_email'], providers)


registry = ServiceRegistry()

@registry.register()
class TestService(BaseService):
    __name__ = 'test_service'
    __version__ = 1
    providers = (Provider1, Provider2)
    provider_strategy = EmailCostSaving 

    @expose_method(http_method='POST', request_schema=EmailSchema)
    async def test_method(self, service_request):
    	provider = await self.get_provider(service_request, required_methods=['send_email'])
    	provider.send_email(service_request.get_data())
        return self.result({'success': True})

kolyaflash/shiny-succotash