The Dutch Forensics Reporting System

🎯 About

The Dutch Forensic Reporting System aims to provide a comprehensive reporting service for identifying flaws in ICT systems across various organizations. This is a group project, that is part of the Postgraduate Program at the University of Essex.

Module: Secure Software Development.
Group Members & Contributors: Nkosilathi Tauro, Ales Tekavcic, Francis Muwalo & Abdulahi Alihu Ngamjeh.

🧰 Tools and Libraries

The following languages and tools were used in this project:

Python v3.10
Django v4.2.2
Serverless PostgreSQL Database cluster via CockroachDB
Serverless Redis Cache cluster via Railway
Tailwind CSS
Docker: Container for Deployment
Railway Microservice Deployment
Brevo email smtp server

Testing, Debugging & Security Tools:

Linter

Pylint with Framework specific pylint-django for addtional type checking.

🏁 Starting, Usage & Testing

📺 Live

You can visit either the Production or Development servers to view the deployed project.

⚙️ Locally

Note: it is recommended that you create a virtual environment when working with Django.
For more information visit: Creation of virtual environments

Requirements
Note: The project was created and tested with python 3.10 and 3.11

Python 3.10 or above installed.

# Clone this project
$ git clone https://github.com/nkosi-tauro/dfrs_domain

# Access directory
$ cd dfrs_domain

# install pip packages
$ pip install -r requirements.txt

# Run the project
$ python manage.py runserver

#For Linux/Mac use:
$ python3 manage.py runserver

# The server will initialize on <http://127.0.0.1:8000>

🗃️ Adding Packages

#After installing new packages or tools via pip, run:
$ pip freeze > requirements.txt
#To add the packages to the requirements.txt file.

🧪 Testing

We are using The Default testing module in Django TestCase to create our unittests.
Note: The tests take a while to be completed (few minutes) as a test database needs to be created.

#To run all the tests run:
$ python manage.py test

#For Linux/Mac use:
$ python3 manage.py test

Optional: The coverage module is required if you want to run the coverage commands.

# Install coverage:
$ python3 -m pip install coverage

#For Linux/Mac use:
$ python3 -m pip install coverage

Using coverage

#To view Test Coverage in terminal:
$ coverage run manage.py test && coverage report

#To view Test Coverage via HTML:
$ coverage run manage.py test && coverage report && coverage html

🔐 Security Risks and Mitigations.

1. Risk: Injection

Mitigation:
In our application, we have implemented Django's querysets which implement query parameterization to protect against SQL injection attacks (Django, N.D). By using The Django Object Relational Mapper (ORM) we avoid writing any raw SQL queries. Additionally, we utilize the built-in validators to validate and sanitize user input, ensuring adherence to specific Regex patterns which are utilized under the hood. These measures collectively enhance the security of our application by safeguarding against malicious SQL injections and promoting secure user input handling.

2. Risk(s): Insecure design + 3. Vulnerable and Outdated components

Mitigation(s):
We added a Django Dependency and Security checker. It scans the repo and helps to continuously monitor and fix common security vulnerabilities in the Django application. With this tool we can keep up to date with any changes to the tools we utilize in the project and update and or fix them if any issues arise. In addition to this we compiled 20 unit tests to validate the flows in the application

These where some security issues identified in our Project:

After fixing the issues:

4. Risk: Broken Access Control

Mitigation:
Using The Django authentication provider allowed us to implement role based access to the application. Using the @login_required(login_url='employee-login') decorators, we can secure routes behind the authentication system and any unauthenticated users will not be able to visit them. Furthermore, we added a redirect system for Authenticated users roles, admin or employee. Based on their role the authenticated user will be redirected to the relevant view where they have permissions.

Code Snippets:

#login decorator on route
@login_required(login_url='employee-login')
def someview(request):
    return render(request, 'this/this.html')

# Role based authentication
user = authenticate(username=req_user, password=password)
# This will first check if the account exists before it tries to authenticate
if user is not None:
  login(request, user)
  if user.is_authenticated:
    if User.objects.get(username=user).is_staff:         
      # Redirect to the admin view
      return redirect('adminview')
    else:
      # Redirect to the employee view
      return redirect('employeeview', user_id)

A basic Rate limiter is also implemented (Gaeddert, 2023). To implement this we made the assumption that a user who continually fails to log in to the platform is trying to brute force their way in. To Mitigate this, after 5 failed login attempts the Users IP will be flagged and blocked for the next 60 seconds from performing a login action on the application. These repeated failed attempts are also added to the Event logs that notify the admin user.

Code Snippets: extract is from user_service/views.py

def login_service(request):
  ...
  try:
    RateLimit(
      key=f"{get_client_ip(request)}",
      limit=5,
      period=60,
      cache=caches['default'],
    ).check()
  except RateLimitExceeded as e:
      systemEvent.critical(f"User with IP: {get_client_ip(request)} has been rate limitted, {e.usage} login requests failed",
                                initiator=get_client_ip(request))
      return HttpResponse(f"Rate limit exceeded. You have used {e.usage} requests, limit is {e.limit}.", status=429)

In addition to the role access and rate limiter, we have also included Session Invalidation. The current timer is 15 minutes of inactivity and the session token will be deleted, and the user logged out and redirected to the login view.

# Session Invalidation
# Logout after a period of inactivity
INACTIVE_TIME = 15 * 60  # 15 minutes
SESSION_SERIALIZER = "django.contrib.sessions.serializers.PickleSerializer"
SESSION_EXPIRE_AT_BROWSER_CLOSE = True
SESSION_COOKIE_AGE = INACTIVE_TIME  # change expired session
SESSION_IDLE_TIMEOUT = INACTIVE_TIME  # logout

5. Risk: Security Logging and Monitoring Failures:

Mitigation:
We Implemented a logging feature that logs/Tracks the following:

When a user of the General Public submits a vulnerability report
When a user attempts and fails to log in, it captures their IP.
When Employees login
When Employees submit a vulnerability report

The logs are only viewable by an admin user.

6. Risk: Cryptographic Failures:

Mitigation:
We used The Django authentication provider which is a trusted authentication library. Passwords are hashed using SHA256 while utilizing the PBKDF2 algorithm (Django, N.D). This ensures the secure handling of sensitive data. SSL/TLS are enforced on the Server which encrypts any data moving between the application and database. An additional measure was to make sure that all Keys, such as the Django Secret key, Database URLs were never checked into source code repositories by using environment variables in their place.

Server deployed on HTTPS Protocol:

Let's Encrypt Certificate:

🧵 Multithreading, Concurrency & Microservices.

Multithreading

In Django, processing emails can be time-consuming and cause the application to halt until the process is finished. This presents several problems. For instance, on our deployed server, any process that exceeds 30 seconds will be terminated. To address this issue, we have incorporated multithreading into our application for handling emails. This approach creates a separate thread or background process specifically for managing email tasks. As a result, the application is able to carry on processing other requests while the email process is running independently.

class EmailThread(threading.Thread):
    '''
    handle mutihreading for emails
    '''
    def __init__(self, email):
        self.email = email
        threading.Thread.__init__(self)

    def run(self):
        self.email.send(fail_silently=False)

Caching

We next implemented query set caching. By caching querysets, it helped reduce the number of database requests and the associated load on the database server, which should then improve its performance and allow it to handle more concurrent requests.

How it works:

On page visit/load, we first attempt to retrieve the querysets from the cache cache.get('systemlogs')
If the querysets are not found in the cache (None is returned), we fetch them from the database events = Event.objects.all()
And then we store the data in the cache cache.set('systemlogs', events)

def systemlogsview(request):
    '''
    The System Logs View
    '''
    events = cache.get('systemlogs')
    if events is None:
        events = Event.objects.all()
        cache.set('systemlogs', events)

    context = {'events': events}
    return render(request, 'adminview/eventlogs.html', context)

Issue, Problem & Solution with Caching
But we then ran into a problem. Since we were caching the data (cache default timeout is 300 seconds) if any new data came through, the user would not be able to view the updated data (Redis, N.D.). To remedy this we utilized The Django signal receiver decorator to listen for when the Model Changed, if the Model changed, the cache would be immediately deleted so that a new call to the Database can be made.

# Signal receivers to update cache on save or delete events
@receiver([post_save, post_delete], sender=Event)
def update_systemlogs_cache(sender, **kwargs):
    cache.delete('systemlogs')

🐳 Microservices

In addition to incorporating multithreading and caching, our application infrastructure is based on a microservices' architecture, allowing for seamless scalability to meet growing demands. By breaking down the application into independently deployable and manageable components (Database, Reporting System & Cache), we can scale specific microservices as needed, rather than the entire application. This approach enables efficient resource allocation, fault isolation, and optimal performance even during peak traffic periods.

🕵️ GDPR Compliance.

We are committed to ensuring compliance with the General Data Protection Regulation (GDPR) to protect the privacy and rights of individuals who interact with our website. Our GDPR compliance documentation provides detailed information on how we collect, use, and protect personal data, as well as the rights of data subjects (Woldford, N.D).

For the complete GDPR compliance policy and detailed information on data collection, processing, security measures, data subject rights, and more, please refer to our GDPR Compliance Documentation.

Testing & Linters

🧪 Testing

We created about 20 Tests, we tried to cover the most important functionality of our domain, such as:

Login Functionality - does a user get redirected to the correct views based on their role?
CRUD creating employees, updating reports, deleting reports and so forth
Views, are the correct views being rendered?
Rate limit functionality
IP address - does this capture the correct IP address value?

All 20 Tests are currently passing 🧪.

Our Current Test Coverage is 72%. See Testing section for more on coverage

Linters

Our chosen Linter for this project was Pylint. The chosen code Formatter was Black, previously it was Pep8 but we had some issues with it formatting the code.
We utilised the Linter with a few modifications.

.pylintrc file. This was so we could ignore the Default Django files and any other files that would not need to be Linted. Also, we used it to configure our preferred indent space, default is 2, but we utilize 4 spaces.
Instead of having the output in the terminal, we made use of Pylint-Report. This allowed us to set the Pylint output to JSON and then convert the JSON output to an HTML file, which allowed for easy viewing of the pylint output.

Pylint Report - See pylint_output/pylint_report.html the detailed output.

Pylint Report after fixes implemented - See pylint_output/pylint_fixedreport.html the detailed output.

🔍 References

Django. (N.D.). Password management in Django. Available from: https://docs.djangoproject.com/en/4.2/topics/auth/passwords/ [Accessed 10 July 2023]

Django. (N.D.). Security In Django. SQL Injection Protection Available from: https://docs.djangoproject.com/en/4.2/topics/security/ [Accessed 03 July 2023]

Django. (N.D.). Security In Django. Available from: https://docs.djangoproject.com/en/4.2/topics/security/ [Accessed 03 July 2023]

Gaeddert, D. (2023). Rate limiting requests in Django. Available at: https://www.forgepackages.com/guides/rate-limiting-requests/ [Accessed 7 Jul. 2023].

Redis. (N.D.). Client-side caching in Redis. Available at: https://redis.io/docs/manual/client-side-caching/.

Wolford, B. (N.D.). Writing a GDPR-compliant privacy notice (template included). GDPR.EU. Available from: https://gdpr.eu/privacy-notice/ [Accessed 06 July 2023]

📝 License

This project is under license from MIT. For more details, see the LICENSE file.