beiyouhao/Backend

Django Project for MetroFlow

更新内容

戳我传送

项目介绍

环境配置

框架依赖

Django==2.2.1
djangorestframework==3.9.4
djangorestframework-simplejwt==4.3.0
mysqlclient==1.4.2.post1
numpy==1.16.4

MySQL版本&引擎

1. 检查MySQL版本
   

2. 确保MySQL使用utf-8编码

   1. 查看MySQL的字符编码

   mysql> show variables like "%char%";
   +--------------------------+-----------------------------------------------------------+
   | Variable_name            | Value                                                     |
   +--------------------------+-----------------------------------------------------------+
   | character_set_client     | utf8                                                      |
   | character_set_connection | utf8                                                      |
   | character_set_database   | latin1                                                    |
   | character_set_filesystem | binary                                                    |
   | character_set_results    | utf8                                                      |
   | character_set_server     | utf8                                                      |
   | character_set_system     | utf8                                                      |
   | character_sets_dir       | /usr/local/mysql-5.7.26-macos10.14-x86_64/share/charsets/ |
   +--------------------------+-----------------------------------------------------------+

   2. 修改对应表项的值
      

   set character_set_database='utf8'

3. 确保MySQL引擎为InnoDB

   show variables like '%storage_engine%';

   

本地运行

1. 在settings.py中配置数据库连接
2. 在/Backend/下执行

   python manage.py makemigrations
   python manage.py migrate

3. 执行如下命令以测试各项API

   python manage.py runserver

API列表

User/

• POST: register/注册
  • 数据格式

    {"username": "username", "password": "password"}
    

• POST: obtain_token/令牌获取
  • 数据格式

    {"username": "username", "password": "password"}
    

• GET: admin_user/管理用户(需要Access Token&管理员权限)
  • 在请求头中带上Access Token
  • 设置管理员账号

    python manage.py createsuperuser

Flow/

• GET: show_flow/获取某日期(范围内)某(些)站点的客流量
  • 数据格式

    {"year": 2019, "month": 5, "dates": [date_1, date_2], "stations": [s_1, s_2]}
    

Station/

• GET: get_station/获取某(些)站点的信息
  • 数据格式

    • 无json请求, 返回所有站点的信息
      
      

    • 获取选中站点的信息

      {"stations": [1, 2, 3]}
      # station键值必须是列表
      {"stations": [1]}

      
      
      

• POST: edit_station/编辑站点信息
  • 数据格式

    • 创建新站点

      {"station_info": {"station_name": "Station1"},
      # 下面这项可以省略, 省略的时候默认创建一个不属于任何线路的站点
       "relationship": {"route_id": 1, "seq": 2}
      }

      
      
      
      

    • 编辑现有站点信息

      {"station_info": {"station_name": "Station1"},
       "relationship": {"route_id": 1, "seq": 3}
      }

      需要注意的是如果relationship中的route_id不存在, 或者是route_id X seq的组合存在的话, 会返回报错信息

      • 例如:
        
        
      • 正确请求
        
        
        
      • 错误请求
        
        • 存在站点
          
          
        • 试图覆盖
          
          
          

Route/

• GET: get_route/获取某(些)线路信息
  • 数据格式

    {"route": [1, 2, 3]}
    {"route": [1]}
    # 不提供请求数据则默认返回所有路线信息

• POST: edit_route/编辑某线路信息
  • 数据格式

    {
        # 如果是不存在的路线, 将会新建一个路线;
        # 而如果存在, 则会修改对应路线的信息
        "route_info": {
            "route_name": "Route 1"
        },
        # 和Staiton里的相似, 不提供该项的时候将不会修改该路线的附属站点信息
        # relation里面的元素还是一个字典: "station": {"station_id", "seq"}
        # 跟station中对应API的数据格式有区别
        "relationship": {
            "s1": {
                "station_id": 1,
                "seq": 1
            },
            "s2": {
                "station_id": 2,
                "seq": 2
            },
        }
    }

    

功能模块

用户认证

登录&注册

注册

• API

  http://127.0.0.1:8000/User/register/
  

• 数据格式

  {"username": "Zenyatta", "password": "sr4400"}

• 返回值:

  • Token
    • Access Token
      • 对于需要身份验证的API需要在请求头部带上
        • PostMan
        • cUrl

          curl -H "Authorization: Bearer [Access Token]" [url] (-X POST)
          

        • httpie

          http [url] "Authorization: Bearer [Access Token]"
          

    • Refresh Token
      • Access Token的有效时限是五分钟，五分钟后Access Token过期，届时若要访问受限API需要重新通过/User/obtain_token/获取新的Access Token; 或者可以选择在当前Access Token未过期前带上Refresh Token访问/User/refresh_token/获取新的Access Token，并在加下来的请求中以新的Access Token代替请求头中旧的Access Token延续当前的登录状态
  • Menu
    • 适用于当前用户角色权限的可选操作菜单

客流量查询

• API

  http://127.0.0.1:8000/Flow/show_flow/

• 数据格式

  {"year": 2019, "month": 5, "dates":[1, 2, 3], "stations": [1, 2, 3]}
  # dates & stations 两项的键值必须是可迭代的展开列表，可以是单一项，但必须是列表
  # 例如
  {"year": 2019, "month": 5, "dates": [1], "stations": [1]}

站点&路线(待更新)

功能菜单(待更新)

角色权限管理(待更新)

配置单元测试&数据

Flow 模块

• 数据文件

  /Backend/data.npy
  

  

• 配置脚本(单元测试配置有问题，所以在DjangoManageShell中调配(Ctrl-C&Ctrl-V))

  1. /Backend/目录下执行python manage.py shell
  2. 复制如下脚本

     # 以下是依赖项
     from App.Flow.models import get_flow_model()
     import numpy as np
     import pickle
     # 以下是数组转换函数
     def trans(inlist):
         return dict((k_1, dict((k_2, dict((k_3, v_3) for k_3, v_3 in zip(['in', 'out'], v_2))) for k_2, v_2 in zip(range(1, len(v_1)+1), v_1))) for k_1, v_1 in zip(range(1, len(inlist)+1), inlist))
     def retrive_one_station(i, Dic):
         return {k_1: v_1[i] for k_1, v_1 in Dic.items()} 
     def regroup(flatDic):
         return {i+1: {j: flatDic[i*144+j] for j in range(1, 145)} for i in range(19)}    
     # 以下是文件读取, np数组的去归一化
     nparray = np.loads('data.npy')
     intlist = [[[int(i*3000) for i in j] for j in k] for k in nparray.tolist()]
     Dic = trans(intlist)
     # 以下是将Dic中的(2376, 81, 2)字典重排为(81, 144, 2)字典, 并存到中转变量dbCache中
     dbCache = {}
     for i in range(1, 82):
         dbCache[i] = regroup(retrive_one_station(i, Dic))
     # dbCache的格式应该是{station:{date: {in & out}}}
     for i in range(1, 82):
         flow = get_flow_model().objects.create(id=i)
         for j in range(1, 20):
             pkled = pickle.dumps(dbCache[i][j])
             setattr(flow, 'date_%s' % j, pkled)
         flow.save()

开发笔记

• 基于角色的权限管理
  • 重构用户模型, 增添用户属性

    class Account(AbstractUser):
        CATEGORY_CHOICE = (
            (),
            ()
        )
        category = models.()

  • 自定义permission_classes
    • 参考rest_framework.permissions.IsAuthenticated

      class IsAuthenticated(BasePermission):
          """
          Allows access only to authenticated users.
          """
      
          def has_permission(self, request, view):
              return bool(request.user and request.user.is_authenticated)

    1. 思路一

       • 重写has_permission()在函数内完成权限的验证

         Django-REST Framework Object Level Permissions and User Level Permissions

             class IsAuthenticated(BasePermission):
                 def has_permission(self, request, view):
                     return bool(request.user and 
                             request.user.is_authenticated and 
                             request.user.category == 1)

         • 缺陷: request.user是一个AnonymousUser,没有category属性

           Django JWT authentication - user is anonymous in middleware

         • 可能的解决方案: 自定义中间件middleware,在django-rest-frame的中间件拦截HttpRequest,并对user进行转换前保留对应属性
           

           A Django Rest Framework Jwt middleware to support request.user

           How add Authenticate Middleware JWT django?

    2. 思路二

       • 在JWT的Payload中添加所需的用户属性, 在用户发起请求验证token的时候解析token完成验证, 从request.META中取出HTTP_AUTHORIZATION并且反序列化
       1. 修改response_payload_handler
          • 对于django-restframework-jwt:
            1. 参考rest_framework_jwt.utils.jwt_reponse_payload_handler()自定义new_jwt_response_payload_handler():

               def jwt_response_payload_handler(token, user=None, request=None):
                   return {
                       'token': token,
                       'username': user.username,
                       'user_id' : user.id,
                       'email' : user.email
                   }

            2. 在settings.py中修改对应的处理函数

                   'JWT_RESPONSE_PAYLOAD_HANDLER':
                   'accounts.utils.jwt_response_payload_handler',

               Getting user id returned with JWT Ask Question Store more than default information in django-rest-framework-jwt

          • 对于rest_framework_simplejwt:
            1. 继承TokenObtainPairSerializer,并添加属性:

               class MyTokenObtainPairSerializer(TokenObtainPairSerializer):
                   def validate(self, attrs):
                       data = super().validate(attrs)
                       refresh = self.get_token(self.user)
                       data['refresh'] = str(refresh)
                       data['access'] = str(refresh.access_token)
               
                       # Add extra responses here
                       data['username'] = self.user.username
                   return data
               
               
               class MyTokenObtainPairView(TokenObtainPairView):
                   serializer_class = MyTokenObtainPairSerializer

            2. 在urls.py中修改对应选项:

               from Account.utils import MyTokenObtainPairView
               
               urlpatterns = [
                   path('admin/', admin.site.urls),
                   path('obtain/', MyTokenObtainPairView.as_view()),
               ]

       • 缺陷: 修改的只是response_payload, 而不是payload, 相关属性在生成token的时候已经被编码
       2. 修改payload_handler
          • 对于rest_framework_jwt:

            1. 参考rest_framework_jwt.utils.jwt_payload_handler实现自己的jwt_payload_handler但即便是复制原始代码, 修改JWT_PAYLOAD_HANDLER后会出现能取得token, 但在后续验证的过程中报错的情况:

               "GET /images/all/ HTTP/1.1" 401 58"
               "detail": "Authentication credentials were not provided."
               

               Django Rest Framework : Authentication credentials were not provided

               How can i make django-rest-framework-jwt return token on registration?

       3. 修改JWTMiddleWare

          import jwt
          import traceback
          
          from django.utils.functional import SimpleLazyObject
          from django.utils.deprecation import MiddlewareMixin
          from django.contrib.auth.models import AnonymousUser, User
          from django.conf import LazySettings
          from django.contrib.auth.middleware import get_user
          
          settings = LazySettings()
          
          
          class JWTAuthenticationMiddleware(MiddlewareMixin):
              def process_request(self, request):
                  request.user = SimpleLazyObject(lambda: self.__class__.get_jwt_user(request))
          
              @staticmethod
              def get_jwt_user(request):
          
                  user_jwt = get_user(request)
                  if user_jwt.is_authenticated():
                      return user_jwt
              # token = request.META.get('HTTP_AUTHORIZATION', None)
                  token = request.META.get('AUTHORIZATION', None)
          
                  user_jwt = AnonymousUser()
                  if token is not None:
                      try:
                          user_jwt = jwt.decode(
                                      token,
                                      settings.WP_JWT_TOKEN,
                                      )
                          user_jwt = User.objects.get(
                              id=user_jwt['data']['user']['id']
                          )
                      except Exception as e: # NoQA
                          traceback.print_exc()
                      return user_jwt

    3. 思路三

       Django Permission进阶使用

       1. 在每个对象上添加不同的权限 在DjangoShell中

          from django.contrib.auth.models import Group, Permission
          from django.contrib.contenttypes.models import ContentType
          
          content_type = ContentType.objects.get(app_label='school', model='Discussion')
          permission = Permission.objects.create(codename='can_publish',
                                              name='Can Publish Discussions',
                                              content_type=content_type)

       2. 对用户进行分组 在Django后台进行分组权限操作的时候会报错(Django==2.0.2) no such table: main.auth_permission__old

          no such table: main.auth_user__old

       3. 对不同分组进行授权
       4. 在permission_classes的has_permission中进行验证

          class IsCat0(BaseException):
              def has_permission(self, request, view):
                  return request.user.has_perm('auth.group.can_add_userq')