Notes about the course

Back-end development by Meta, on Coursera.

Schedule
Introduction to Back-end Development
- Week 1
- Week 2
- Week 3
Programming in Phyton
Version Control
Databases

Schedule

Monday	Tuesday	Wednesday	Thursday	Friday	Saturday	Sunday
8h-8h45m	x	x			x
9h-9h45m	x	x			x
10h-10h45m	x				x
13h-13h45m	x	x	x	x		x
14h-14h45m	x	x	x	x		x
15h-15h45m			x	x		x
17h30m-18h15m			x	x
18h30m-19h15m	x	x	x	x
19h30m-20h15m	x	x	x	x

Introduction to Back-end Development

Week 1

Capstone project overview

Create Django web server, with multilple API endpoints.

Connect it to a MySQL database.

Create a template system driven by Django views.

How the web works

Fullstack

Both front and back development. Mostly, on connecting the two.

Planning.
Architecture.
Design.
Development.
Deployment.
Maintenance.

Front-end

Design patterns of elements on a page.

Back-end

Back-end language.
Database.
APIs.
Web servers

Grace Egbo - a day in the life of a backend dev.

The day breaks out in sections.

Code review

Possible quick fixes

Meeting about ideias and needs

Coding

Soft skills/People skills

Communicate.
Listen to perspectives.
Remote world makes that even more necessary.

How the internet works

Definition: `Network`

Two or more computers connected through wired or wireless connection.

Multiples devices in a network - Network Switches

Complexity is brough down by using Network switches.

Definition: `Interconnected network`

When multiple Networks connect through Netowork Switches. E.g., the Internet.

Servers

Data centers: multitude of servers on a physical space.
Server purpose data center will have different machinery, depending on the application objetive.

Webservers

Website hosting.
Database.
Control panel.
CMS software.
Email.

Web request

Request-response cycle:

It’s the job of the web server to send you back those website content, upon requesting, by typing the URL of the website.

Designed to respond to thousands of requests of clients per second.

What are websites and webpages?

Webpage

Display content, like text, images, videos etc, on the web browser.

Website

It’s a collection of web pages linked together.

HTML, CSS and JavaScript

HTML

Hyperlink Text Markup Language Tells how to structure elements in a page.

CSS

Cascade Styling Sheets.

JavaScript

Programming language build into the browser. Tools for interactivity, data-processing and control-and-action.

Page rendering process

While interpreting each line of HTML, the browser creates a building-block that switches the visual representation you see on screen.

A response from the webserver must be a complete web page, in other to fulfill the request, to show the page, in the browser.

Web browsers

Software application used to browse the world-wide-web.

It works by sending a request to a web server, and then receives a response containing the content to be displayed on your device.

URL

Uniform Resource Locator, contains the protocol (HTTP/HTTPS), the domain name, and the file-path.

HTTP

HyperText Transfer Protocol.
Request-response cycle.

TCP - Transmission Control Protocol

Transmits messages with high precision.
Barley no data loss.
Almost always on the right addresses.
Slower than UDP.

UDP - User Datagram Protocol

Corrupt package issue solved (barely no data loss).
Easily out of order data-packages.
Possibly a lot of loss of information.

HTTP - HyperText Transfer Protocol

Standard for the web communication.
Transfers data:
Web pages,
Images,
Files.
Request-response based communication between client and server.

Makeup of a HTTP request

It must contain a

Method,
Path,
Version,
Headers.

Method

Describes the kind of action that the client wants to perform.

Most common are:

GET;
POST;
PUT;
DELETE;

Get

Retrieves information.

Post

Sends information.

Put

Update data on webserver. That is, data1 is swapped for data2.

Delete

Removes the resource.

Path

The path is a representation of where the resource is located on the webserver.

Version

Rules of what constitutes and how request and response happen.

Headers

Headers contains additional information about the request and the client that is making the request.

Makeup of a HTTP response

It’s similar to HTTP requests.

After the header, the message body contains data that is the response:

Text.
HTML Markup.
Images.
Files.
etc.

Header

HTTP response status (200, 404, 400, etc.).
Status message (OK, Not Found, Server Not Responding, etc.).

Informational

Responses ranging 100-199.

Provisional.
Interim response.
Most common: 100 CONTINUE.

Successful

Responses ranging 200-299.

If successfully processed (200 OK),

GET: Found/included.
POST: Successfully transmitted.
PUT: Successfully transmitted.
DELETE: Deleted.

Redirection

Responses ranging 300-399.

301 MOVED PERMANENTLY.
302 FOUND.

Client error

Responses ranging 400-499.

400 BAD DATA (transmitted to the server);
401 MUST LOGIN (before making the request);
403 REFUSE TO PROCESS (but valid request);
404 NOT FOUND (requested data);

Server error

Responses ranging 500-599.

500 INTERNAL SERVER ERROR (server failed to process request);

Examples about HTTP

Request Line

Every HTTP request begins with the request line.

This consists of the HTTP method, the requested resource and the HTTP protocol version.

GET /home.html HTTP/1.1

In this example, GET is the HTTP method, /home.html is the resource requested and HTTP 1.1 is the protocol used.

HTTP Method

IDEs

Integrated Development Environment offers:

Syntax highlight;
Keyword documentation;
Auto-complete suggestions;
Navigation ease;
Unified Environment for development;

Extra resources

HTTP Overview (Mozilla)

https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview

Introduction to Networking by Dr.Charles R Severance

https://www.amazon.com/Introduction-Networking-How-Internet-Works/dp/1511654945/

Chrome Developer Tools Overview (Google)

https://developer.chrome.com/docs/devtools/overview/

Firefox Developer Tools User Docs (Mozilla)

https://firefox-source-docs.mozilla.org/devtools-user/index.html

Getting Started with Visual Studio Code (Microsoft)

https://code.visualstudio.com/docs

Week 2

HTML

Table

Forms

Input tags

Text;
Password;
Checkbox;
Radio;
Textarea;
Select;

DOM

Document Object Model Server -> Web browser receives webpage -> Transforms in a DOM scheme.

CSS

How to display HTML elements.

Selector (which element to act upon);
Key-values:
Property;
Property-value;

Extra Resources

HTML and DOM

Learn more Here is a list of resources that may be helpful as you continue your learning journey.

HTML Elements Reference (Mozilla)

https://developer.mozilla.org/en-US/docs/Web/HTML/Element

The Form Element (Mozilla)

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/form

What is the Document Object Model? (W3C)

https://www.w3.org/TR/WD-DOM/introduction.html

ARIA in HTML (W3C via Github)

https://w3c.github.io/html-aria/

ARIA Authoring Practices (W3C)

https://www.w3.org/TR/wai-aria-practices-1.2/

Week 3

Working with libraries

Libraries can depend on other libraries forming a tree of dependencies.

Bundlers unify all code in one file, or few of them.
Finally, add the final bundled file into your HTML.

Responsive Design

A responsive grid happens when we combine:

Flexible grids;
Fluid images;
Media queries.

Flexible grids

Gutter: space between contents;
Margin: space between content and screen;
Sizes based on percentages.

Fluid images

Max-width: 100%;
Shrink based on container-element size;
Fit on page;
Never grow larger than original size.

Media queries

Controls:

Display size;
Orientation;
Aspect ratio.

Types of Grids

Fixed grid;
Fluid grid;
Hybrid grid;

Fixed Grid

Fixed width-columns;
Flexible-margins;

Fluid grid

Fluid width-columns;
Fixed-gutters;
Fixed-margins;

Columns either grow or shrink to adapt to the available space.

Hybrid grid

Fluid and fixed width components;
Different rules, depending on device;
Optimize experience.

Bootstrap

Responsive design with class infix

Convention for class infix made for responsive design, by Bootstrap.

Class modifiers

It’s like the conjugation of verbs, so you have different meaning, with the same functionality. E.i., buttons of alert that can mean “just pay attention”, or ”DANGER!” etc.

Static vs Dinamic websites

Static part of a website

Images;
Videos;
Text.

Dinamic part of a website

Static content that is generated, depending on the response of an application server.

To speed up the process and don’t overload the website, cache is used to store dynamically generated content.

SPA and Multi-Page Application

Multi-page Application

Loads all content at each update
Application server return the entire webpage

SPA

SPA can use both bundling and lazy-load to display dynamic content.

Bundling

When a new component or data is requested the hole section is returned and rendered.

Lazy-loading

When a new piece of a component or data is requested, only the fraction of data and component concerned for the request is loaded.

React

Extra resources

Bootstrap

Bootstrap Official Website

https://getbootstrap.com/

Bootstrap 5 Foundations by Daniel Foreman

https://www.amazon.com/Bootstrap-Foundations-Mr-Daniel-Foreman/dp/B0948GRS8W/

Responsive Web Design with HTML5 and CSS by Ben Frain

https://www.amazon.com/Responsive-Web-Design-HTML5-CSS/dp/1839211563/

Bootstrap Themes

https://themes.getbootstrap.com/

React

Learn more Here is a list of resources that may be helpful as you continue your learning journey.

React Official Website https://reactjs.org/

Choosing between Traditional Web Apps and Single Page Apps (Microsoft)

https://docs.microsoft.com/en-us/dotnet/architecture/modern-web-apps-azure/choose-between-traditional-web-and-single-page-apps

React Source Code (Github)

https://github.com/facebook/react

Introduction to React.js

The original video recorded at Facebook in 2013.

https://youtu.be/XxVg_s8xAms

Programming in Phyton

Objectives

Get started with the Python programming language and associated foundational concepts.

Learning Objectives:

Recognize common applications of the Python programming language.
Explain foundational software engineering concepts.
Use operators to program a simple output in Python.
Use control flow and loops to solve a problem

Week1-Notes - Introduction to Python

Course content

Course content During this course, you’ll cover everything you need to begin with Python development. The content of the four modules is listed below.

Module 1 - Getting started with Python In this module, you will get an introduction to the course and you’ll cover a brief summary of the history of programming. You will also learn the basics of how programming works and discover typical uses for Python in real-life. There are also some tips on how to take this course successfully.

Before embarking on any coding you’ll first establish if your current computer system is set up correctly and identify any required operating system dependencies. Then you’ll explore different ways that you can run programs through Python.

Next, you’ll study Python syntax using comments, variables, data types and user input and output. You’ll proceed to expand your Python skills by using math and logical operators so you can control the flow of your code and perform operations such as addition, subtraction, division, and multiplication.

Python has multiple ways to do code loops or looping. You will examine looping constructs to iterate your code over multiple sequences.

Module 2 - Basic programming with Python In module 2, you’ll receive an introduction to Python functions, including how to declare a function, and how to pass data to a function or return data from a function. You will also explore data structures, scope, and the concepts of lists and how they can be used in Python. You will also discover tuples, sets, dictionaries and kwargs, gaining an understanding of how their main uses.

Errors and exceptions are two very important aspects of learning Python as a new developer. In module 2, you will start with errors and exceptions, and explore what happens when something goes wrong with your code. Exception handling and Python file handling are covered, as are how to create files in Python and various methods of inserting content into a new file.

Module 3 - Programming paradigms In the third module, you will cover programming paradigms, and look at the features and concepts behind procedural programming, functional programming and object oriented programming.

A key feature of procedural programming is algorithms, you will explore how they can be used to solve problems, how algorithmic complexity is calculated and learn about Big-O notation.

Next you will learn about pure functions and recursion in functional programming, as well as the difference between maps and filters.

Finally, you will explore object-oriented programming and its four main concepts. You will explore the relationship between classes and instances in Python by creating classes, instantiating them, and accessing their variables and methods. You’ll learn about abstract classes and methods and how to implement them. The rules of method resolution and inheritance with child and parent classes are also explained. Being able to read files is essential when working with stored data in Python and you’ll discover several built-in functions to make this easier.

Module 4 - Modules, packages, libraries and tools In module 4, you learned that Python is a powerful language that allows you to build amazing things. But it can gain even more functionality with the use of modules, libraries and tools. You will learn about Modules and that they are the building blocks for adding functionality to your code, so you don’t need to continually redo everything.

You will also explore some of the commonly used Python libraries in data analysis and data science, and how they can apply to the areas of machine learning and artificial intelligence.

Finally, you’ll find out why testing is an essential component of Quality Assurance and explore the type of testing you should use. You will learn about test automation packages and the importance of automated testing and you’ll write some tests using PyTest. Finally, you will explore the evolution of Test-driven development (or TDD), and focus on how to apply a test-driven development methodology.

Module 5 - Graded assessment Here you’ll learn about the graded assessment. After you complete the individual items in this module, you’ll test your knowledge and skills by completing an end of course graded assessment.

Input

ipt = input("Please input something")
print(inp)

Format

a=1
b="abb"
print("{0} is different from {1}" .format(a, b))

a = isinstance(str, "aa")

print(a)

Week2-Notes - Python Data structures

*args, **kwargs

Args

def s(*args):
return sum(args)

print(s(3,4,5,6))

def s(*args):
return sum(*args)

print(s([3,4,5,6]))

Kwargs

def s(**kwargs):
return sum()

print(s(3,4,5,6))

Additional Resources

First

Python allows you to do quite a lot with very little code. Compared to other languages such as Java or C#, Python has a much easier learning curve. It lends itself well to the “write less, do more philosophy”. Python developers are also in high demand and learning how to program in Python makes for a good career choice.

You can access the links below to learn more about programming in Python.

Check out the Python website to find out more about built-in functions: Python

Check out W3 Schools to learn more about coding and web development: W3Schools

Check out HackerRank to practice your new acquired Python skills: HackerRank

Data Structures

Here is a list of resources that may be helpful as you continue your learning journey.

Learn more about Python data structures (Python documentation) on the Python website: Python.org - Data structures

Explore common Python data structures at the Real Python website: Real Python - Data structures

Open, close files

Open files

options

r: read (text)
rb: read (binary)
r+: read and write
w: write (overwrite file)
a: append data

`with open`

No need to use the close function.

with open('testing.txt', 'r') as file:

Close files

No arguments.

Examples

Open and read

file = open('test.txt', mode='r')

data = file.readline()

print(data)

file.close()

# alternative
with open('test.txt', mode='r') as file:
data = file.readline()
print(data)

Hello there!

Create file and populate it

try:
with open('sample/newfile.txt', 'w') as file:
file.writelines(["Hello", "\nThere", "\nThird line!"])

with open('sample/newfile.txt', 'a') as file:
file.writelines(["\nHello", "\nThere", "\nThird line!"])
except FileNotFoundError as e:
print("Error: ", e)

Only selected characters

with open('samplefile.txt', 'r') as file:
print(file.read(40))

Read a line

Reads the entire line:

with open('samplefile.txt', 'r') as file:
print(file.readline())

Reads the line until the nth character:

n=10
with open('samplefile.txt', 'r') as file:
print(file.readline(n))

Read multiple lines

readlines() read the entire contents of the file and returns it in an ordered list,

with open('samplefile.txt', 'r') as file:
lines=file.readlines()
print(len(lines))

for line in lines:
print(line)

Looping through lines

with open() as file already returns a list of lines stored in file.

with open('samplefile.txt', 'r') as file:
for line in file:
print(line)

Extra Resources

The following resources will be helpful as additional references in dealing with different concepts related to the topics you have covered in this module.

Learn more about exceptions and errors in Python on the Python website: Exceptions and Errors in Python - Python docs

Check out the PyNative website to learn more about file handling in Python: File handling in Python

Week3-Notes - Programming Paradigms

Principle DRY:

Don’t;
Repeat;
Yourself.

What is procedural programming?

Creating reusable code that is used multiple times, so to DRY.

Functional programming

# Recursive function for Tower of Hanoi
def hanoi(disks, source, helper, destination):
print('P1 - disk: {}, source: {}, helper: {}, destination: {}'.format(disks, source, helper, destination))
# Base Condition
if (disks == 1):
print('Disk {} moves from tower {} to tower {}.'.format(disks, source, destination))
return

# Recursive calls in which function calls itself
hanoi(disks - 1, source, destination, helper)
print('P2 - disk: {}, source: {}, helper: {}, destination: {}'.format(disks, source, helper, destination))
print('Disk {} moves from tower {} to tower {}.'.format(disks, source, destination))
hanoi(disks - 1, helper, source, destination)
print('P3 - disk: {}, source: {}, helper: {}, destination: {}'.format(disks, source, helper, destination))

# Driver code
# disks = int(input('Number of disks to be displaced: '))
disks = 3
'''
Tower names passed as arguments:
Source: A
Helper: B
Destination: C
'''
# Actual function call
hanoi(disks, 'A', 'B', 'C')

Slice functions

# str[start:stop:step]
trial = "rbvcrsal"
new_trial = trial[::-1]
print(new_trial)
print("---", range(len(trial)), "---")
print(trial[len(trial):-(len(trial)+1):-1])
print(new_trial == trial[len(trial):-(len(trial)+1):-1])
for i in range(len(trial)):
print(trial[i::-1])
# print(trial[i:-(len(trial)+1):-1])

Recursive reversal

trial="abcdefg"

# Idea: concatenate string in the opposite order
print(trial)
print("trial[1:]: ", trial[1:], ", trial[0]: ", trial[0], sep='')
print("trial[2:]: ", trial[2:], ", trial[1]: ", trial[1], ", trial[0]: ", trial[0], sep='')
# trial[0] -> get first character in string
def first_character(str):
return str[0]

def string_opposite(str):
if len(str) == 0:
return "" # empty string -> always will concatenate the empty string in last call
else:
# get first character and concatenate (at the end),
# with the reversed string of the orginial string, less the first character.
return string_opposite(str[1:]) + first_character(str)

print(string_opposite(trial))

a = [[96], [69]]

print(''.join(list(map(str, a))))
print(type(''.join(list(map(str, a)))))

Assignment - list comprehension

# Input data: List of dictionaries
employee_list = [
{"id": 12345, "name": "John", "department": "Kitchen"},
{"id": 12456, "name": "Paul", "department": "House Floor"},
{"id": 12478, "name": "Sarah", "department": "Management"},
{"id": 12434, "name": "Lisa", "department": "Cold Storage"},
{"id": 12483, "name": "Ryan", "department": "Inventory Mgmt"},
{"id": 12419, "name": "Gill", "department": "Cashier"}
]

# Function to be passed to the map() function. Do not change this.
def mod(employee_list):
temp = employee_list['name'] + "_" + employee_list["department"]
return temp

def to_mod_list(employee_list):
""" Modifies the employee list of dictionaries into list of employee-department strings

[IMPLEMENT ME]
1. Use the map() method to apply mod() to all elements in employee_list

Args:
employee_list: list of employee objects

Returns:
list - A list of strings consisting of name + department.
"""
### WRITE SOLUTION CODE HERE
return list(map(mod, employee_list))
#raise NotImplementedError()

def generate_usernames(mod_list):
""" Generates a list of usernames

[IMPLEMENT ME]
1. Use list comprehension and the replace() function to replace space
    characters with underscores

List comprehension looks like:
list = [ function() for <item> in <list> ]

The format for the replace() function is:

test_str.replace(“a”, “z”) # replaces every “a” in test_str with “z”

Args:
mod_list: list of employee-department strings

Returns:
list - A list of usernames consisting of name + department delimited by underscores.
"""
return [str.replace(" ", "_") for str in mod_list]
### WRITE SOLUTION CODE HERE

#raise NotImplementedError()

def map_id_to_initial(employee_list):
""" Maps employee id to first initial

[IMPLEMENT ME]
1. Use dictionary comprehension to map each employee's id (value) to the first letter in their name (key)

Dictionary comprehension looks like:
dict = { key : value for <item> in <list> }

Args:
employee_list: list of employee objects

Returns:
dict - A dictionary mapping an employee's id (value) to their first initial (key).
"""
#return {value:key for (key,value) in employ.items() for employ in employee_list}
ids=[]
names=[]
for d in employee_list:
ids.append(d["id"])
names.append(str(d["name"])[0])
return {value:key for (key, value) in zip(ids, names)}

### WRITE SOLUTION CODE HERE

#raise NotImplementedError()

def main():
mod_emp_list = to_mod_list(employee_list)
print("Modified employee list: " + str(mod_emp_list) + "\n")

print(f"List of usernames: {generate_usernames(mod_emp_list)}\n")

print(f"Initials and ids: {map_id_to_initial(employee_list)}")

if __name__ == "__main__":
main()

Classes are defined using the key-word class;
Attributes can be variables;
Behavior can be functions.

Objects

Objects are instances of Classes.

Bundle, to the *class* =employee=, the attributes =position= and =employment_status=

OOP in Python

You previously encountered the four main pillars of object oriented programming. These are: encapsulation, polymorphism, inheritance and abstraction

Istantiation process

Class definition;
New Instance;
Initializing new instance.

Attribute References

To access data defined inside the class scope.

Examples

class MyClass():
a = 5
print("Hello")

myc = MyClass()
print(MyClass.a)
print(myc.a)

class MyClass():
a = 5
def hello(self):
print("Hello, world!")

myc = MyClass()
print(MyClass.a)
print(myc.hello())
myc.hello()

class House:
'''
This is a stub for a class representing a house that can be used to create objects and evaluate different metrics that we may require in constructing it.
'''
num_rooms = 5
bathrooms = 2
def cost_evaluation(self):
print(self.num_rooms)
pass
# Functionality to calculate the costs from the area of the house

house = House()
print(house.num_rooms)
print(House.num_rooms)

house.num_rooms = 7
print(house.num_rooms)
print(House.num_rooms)

House.num_rooms = 10
house2 = House()
print(house2.num_rooms)
print(House.num_rooms)

class A:
print("Print inside A.")

def __init__(self, c):
print("---------Inside class A----------")
self.c = c

def alpha(self):
c = self.c + 1
return c


print(dir(A))
print("Instantiating A..")
# a = A(1)
# print(a.alpha())


class B:
def __init__(self, a):
print("---------Inside class B----------")
self.a = a
# print(a.alpha())
d = 5
print(d)
# print(a)

print("Instantiating B..")
# b = B(a)
# print(a)

Extra resources

The following resources will be helpful as additional references in dealing with different concepts related to the topics you have covered in this module.

Programming styles in Python https://newrelic.com/blog/nerd-life/python-programming-styles

Different types of algorithms used in Python https://www.thetechplatform.com/post/different-types-of-algorithms-in-data-structure

Introduction to Big-O notation https://dev.to/sarah_chima/the-big-o-notation-an-introduction-34f7

The following resources will be helpful as additional references in dealing with different concepts related to the topics you have covered in this lesson.

OOP Principles https://www.geeksforgeeks.org/python-oops-concepts/

In-depth understanding of MRO https://www.python.org/download/releases/2.3/mro/

OOP Principles/ Classes and objects https://realpython.com/python3-object-oriented-programming/

Week4-notes - Modules, Packages, Libraries and Testing

Imports

Only loaded once;
Can be loaded, only inside a scope;

import sys

loc = sys.path
for i in loc:
print(i)

import calendar

leapdays = calendar.leapdays(2000, 2050)
print(leapdays)

globals() # built-in function
locals() # built-in function

How To Import Modules in Python 3 (Digital Ocean)

Import modules in Python https://www.digitalocean.com/community/tutorials/how-to-import-modules-in-python-3

Python Modules (Programiz) https://www.programiz.com/python-programming/modules

Python Packages (Real Python) https://realpython.com/python-modules-packages/#python-packages

Most common libraries

OS
SYS
CSV
JSON
importlib
re
math
intertools

Data Science

numpy
scipy
nltk
pandas

Image processing and Visualization

OpenCV
matplotlib

ML and AI

Tensorflow
PyTorch
Keras

Others common:

SciPy
Scikit-learn
Theano

Web Development

Flask: Micro-Framework
Django: Full-stack

Others common:

cherry pie
pyramid
beatiful soup
selenium

Examples of libraries-use

Pandas

import pandas as pd

a = pd.DataFrame(
{
"Animals": ["Dog", "Cat", "Lion", "Cow", "Elephant"],
"Sounds": ["Barks", "Meow", "Roars", "Moo", "Trumpet"],
}
)

print(a)
print(a.describe())

b = pd.DataFrame(
{"Letters": ["a", "b", "c", "d", "e", "f"], "Numbers": [12, 7, 9, 3, 5, 1]}
)

print(b.sort_values(by="Numbers"))

b = b.assign(new_values=b["Numbers"] * 3).sort_values(by="Numbers")
print(b)

Big Data and Analysis with Python With the advent of social media and its widespread acceptance came the unprecedented need for data management. Now billions of gigabytes of data are produced every day and much of it is generated by the end-users. Organizations recognized the huge potential in harnessing this data using predictive and machine learning algorithms to generate insights. But before tackling that challenge, came the challenge of efficiently and systematically storing and handling this data in a way that made it available for quick access.

Big data is the management of large sets of data, both structured and unstructured. Today, this large amount of data is stored in the form of data warehouses and data lakes, both on servers and in the cloud. The main characteristics that are commonly identified for use of Big Data tools are Volume, Variability and Velocity.

Volume is the size of data under question and, if large enough, may require different handling to traditional data storage and management.

Variability or veracity refers to the inconsistency that may be present in this data. In huge data repositories, it is difficult to intervene manually on every wrong entry and thus enough scope for variability must be defined and established while handling such data.

Velocity is the speed of handling this data. With data sources such as social media which are continually active, there is a need for constant updates as well as robust storage. When there is a need to be processed, it should also not create a bottleneck where data retrieval takes longer. As such, velocity plays a very important role in Big Data.

This is the ability to handle a large amount of heterogeneous data with ease of access and speedy processing. The next step in this process is when this data is analyzed and is broadly called data analysis. The final step is publishing this data in form of reports, visualizations and web pages, as per the requirement.

The whole pipeline can be summarized as below:

Big Data Order of Operations Procedural sequence of actions in big data Here are several reasons why Python has found a place in the domain of Big Data:

Ease of use: Ease of use is a prerequisite for any large-scale and commonly used technology and language. Python helps setting up and running infrastructure with just a few lines of code.

Licensing structure and open-source nature: The open source paradigm has picked up immensely in recent years. Python provides many very well developed open-source libraries and frameworks, even for large scale applications. Some organizations prefer this, as it saves on cost, as well as providing easier licensing.

Active community: The Python community today is vast and very supportive. This helps with the swift resolution of issues a user may face, as well as the development of new features when required.

Libraries: Possibly the strongest reason for the acceptance of Python is the host of libraries that provide direct support for Big Data. In addition, there are many packages that also aid in bridging the gap between Python and other languages and tools enabling swift deployment of services.

High compatibility with Hadoop and Spark: Hadoop and its Hadoop distributed file system is arguably one of the best solutions for large-scale storage. The support available in Python has also helped in wider acceptance of Python. The same can be said about Spark as Python has supportive libraries such as PySpark and host of API libraries that facilitate its usage.

High processing speed: Python has support for prototyping and with its Object-oriented methodology, processing in Python is much better in comparison to other languages. With its increase in speed, Python is also able to provide adequate stability in its usage.

Portability and scalability: Broadly as mentioned before, Python’s support for cross-language platforms and operations, its ease of extensibility, various libraries, support for frameworks and API overall, makes it easy to scale and flexible.

Python tools and libraries: Most of the libraries in Python that are used for Big Data are widely common and is associate with Data Sciences and Machine Learning. Big Data includes wide-scale usage and acceptance of libraries such as: Numpy, Pandas, Scikit-learn and Scipy. To name just a few.

Additionally, here are a few more libraries that are more specific to a Big Data domain such as:

RedShift and S3: Amazon services are used with their cloud services. S3 is a storage service and RedShift is a data warehousing service.

BigQuery: Developed by Google, BigQuery is a Cloud service library that is useful with RESTful APIs.

PySpark: This is an open-source framework used for large scale data processing and works with resilient distributed datasets.

Kafka: This is a publish-subscribe messaging system that receives logs in the form of packages and is stored in partitioned spaces.

Pydoop: Pydoop provides an interface between Hadoop and Python and support for handling its Hadoop distributed file systems.

Web Frameworks

Full-stack
Microframeworks
Asynchonous

Full-stack

Offers things such as:

Form generators
Template layouts
HTTP request handling
WSGI interfaces
Database connection handling

Most common frameworks:

Django
Web2py
Pyramid

Microframworks

Do not offer as much patterns and functionalities. Used in smaller webprojects and building APIs.

Flask
Bottle
Dash
CherryPi

Asynchronous

Used to handle large amounts of connections.

Growler
AIOHTTP
Sanic

Factors to consider, when choosing framework

Available documentation
Scalability
Flexibility
Integration

Additional Resources

The following resources will be helpful as additional references in dealing with different concepts related to the topics you have covered in this module:

Features of using libraries for testing

Solve:

Bugs quickly
Gaps
Defects
Missing requirements

Helps with:

Poor design
Functionalities
Scalability
Security
AB testing
Compatibility
Assurance
Experience

What `testing` must have

Re-usability
Traceability
Efficiency

Testing Life Cycle

Planning
Preparation
Execution
Reporting

The steps to achieve this can include:

Writing scripts to test cases
Compile tests results
Correct defects based on them
Generating reports from tests

Good testing leads to

Great bug coverage
Re-usability of code
Better user experience
Reduce costs
Increase satisfaction

Types of testing

Broadly can be categorized by:

White box (accessible source code)
Black box (not accessible source code)

Types:

Compatibility testing
Ad hoc
Usability
Regression testing

How to know there was sufficient testing

Amount of test cycles
Passing percentage
Deadlines
Time intervals

Deep-in types of testing

Levels of testing:

Unit
Integration
System
Acceptance

OBS.: Other ways to categorize testing:

Functional tests
Non-functional tests
Maintenance test

Functional tests

Tests if the output is compliant with what was requested/expected.

Non-functional test

Based on metrics on overall performance and quality.

Maintenance test

When the underlining system and it’s operations and environments are corrected, changed or extended.

Levels of testing

The levels of testing are sequential and build on each other.

Unit testing

Designed for individual components, by isolating them.

Components are low-level (close to the level of the written code).

Usually involve use automation for continuous integration (CI), given their small sizes.

So, tests can be written, while writting the code.

Example

Using pytest to test components, in Python.

Integration test

Test how one component affect the other, and the flow of information between them.

Key-word: Interface.

Different approaches, depending on unit tests:

Top-down
Bottom-up
Sandwich

Example

Test if data is correctly fetched from a database. Then, send it to the webpage.

System testing

Test the performance of the application as-a-whole. Test software against the requirements, to assure completeness.

Includes measurement of the location of deployment of the application and components. So to assure that the app has:

Reliability;
Performance;
Security;
Load balancing.
Operability of the platform and OS.

OBS.: most important stage in a team of testers. And, it’s the most critical, because it’s the final stage, before deployment for end-users.

Acceptance testing

When code arrives at this stage, it’s expected to be ready for deploymeny, and bug-free, as well as compliant to the imposed standards.

Alpha.
Beta.
Regression testing.

Involves stack-holders and end-users.

Use pre-written scenarios and try to find bugs.

Final words

The key for testing is: frequent testing and early. While each of the steps are important, early testing saves time, effort and money.

As code gets larger, they become harder to fix.

Testing in Python

Generally involved in:

Prepare the test environment
Run the test scripts
Analyze the results

`Unittest` framework

Built-in testing package.

Supports:

Automation testing
Independent testing modules
Aggregation

`Pytest`

Supports:

Functional test-types:
Unit
Integration
End-to-end
Parameterized test-types:
Enables execution tests multiple times, with different parameters passed.
Parallel test-types:
Generate HTML, XML or plain text reports
Integration with different test frameworks
Pyunit
Nose
Flask
Django

Can test:

APIs
UI
Database connections
Other web-applications.

Robot

Keyword-driven development capabilities

Key-words can be pre-defined or user-defined

Used for acceptance testing, Robotic Process Automation (RPA), Test-Driven Development (TDD).

Can be used for:

Android
APIs
Mainframes

Selenium

Mostly for web-applications

Browser and OS support.

Browser-specific web drivers:

Logins
Buttons
Form-filling

Pytest testing

def add(a,b):
return a+b

def sub(a,b):
return a-b

import addition
import pytest

def test_add():
assert addition.add(4,5) == 9

def test_sub():
assert addition.sub(4,5) == -1

In the terminal,

pytest test_addition.py

Or, to test only one function,

pytest test_addition.py::test_add

TDD (Test-Driven Development)

Write test, before writting the code for a feature, app etc.

OBS.: step 5: rerun the process.

This process is also called the red-green refactor cycle.

Positives

Write tests first.
Makes sure application is compliant to tests.
Predicable behavior, from start.
Easier to integrate components, later.
Ease to refactor, in need of changes.

Other types of development cycles

Behavior-driven
Acceptance-driven
Scaling-driven
Developer-test-driven

Examples TDD

from curses.ascii import isdigit
import findstring
import pytest

def test_ispresent():
assert findstring.ispresent("Al")

def test_nodigit():
assert map(lambda p: findstring.nodigit(p), findstring.names)

names = ["Al", "Bo", "Chi", "Ma"]
def ispresent(person):
if person in names:
return True
else:
return False

def nodigit(person):
if person.isaplha():
return True
else:
return False

Additional Resources

The following resources will be helpful as additional references in dealing with different concepts related to the topics we have covered in this module.

Test-Driven Development https://testdriven.io/blog/modern-tdd/ Test-driven Development with PyTest https://stackabuse.com/test-driven-development-with-pytest/ PyTest Official website https://docs.pytest.org/en/7.1.x/ Test automation packages in Python https://www.geeksforgeeks.org/best-python-modules-for-automation/

Final Test

for x in range(1, 4):
print(int((str((float(x))))))

str = 'Pomodoro'
for l in str:
if l == 'o':
str = str.split()
print(str, end=", ")

def d():
color = "green"
def e():
nonlocal color
color = "yellow"
e()
print("Color: " + color)
color = "red"
color = "blue"
d()

num = 9
class Car:
num = 5
bathrooms = 2

def cost_evaluation(num):
num = 10
return num

class Bike():
num = 11

cost_evaluation(num)
car = Car()
bike = Bike()
car.num = 7
Car.num = 2
print(num)

Class P():
pass
Class C(P):
pass
p=P()
c=C()
# print(issubclass(p,P))
# print(issubclass(C,c))
print(issubclass(C,P))
# print(issubclass(C,P))

class A:
def a(self):
return "Function inside A"

class B:
def a(self):
return "Function inside B"

class C:
pass

class D(C, A, B):
pass

d = D()
print(d.a())

Version Control

Week 1 - Version Control and Project Managment

Version control

One of the first significant Version Control Systems was the Concurrent Versions System (CVS). It was first developed in 1986 by Walter F. Tichy at Purdue University and released publicly in 1990.

Continuous Integration

Continuous Integration, or CI, is used to automate the integration of code changes from multiple developers into a single main stream. Using a workflow whereby small changes are merged frequently, often many times per day, will reduce the number of merge conflicts.

This process is widespread in test-driven software development strategies. CI is often used to automatically compile the project and run tests on every code change to ensure that the build remains stable and prevent regressions in functionality.

Continuous Delivery

Continuous Delivery is an extension of Continuous Integration. Once the changes have been merged into the main stream, a Continuous Delivery system automatically packages the application and prepares it for deployment. This helps avoid human error when packaging the application.

Continuous Deployment

Continuous Deployment is an extension of Continuous Delivery. The goal of Continuous Deployment is to deploy and release software to customers frequently and safely. The strategy commonly involves automatically deploying to a test (also known as staging) environment first to validate the deployment package and software changes. Once validated, it can automatically deploy to the live (also known as production) environment for customers.

Git/GitHub

Version control:

Easily backroll
Easily integrate code
Keep track of collaboration (Git blame)
Keep track, in Agile setup

About Agile

Additional Resources

About Version Control

https://git-scm.com/book/en/v2/Getting-Started-About-Version-Control

List of Version Control Software

https://en.wikipedia.org/wiki/List_of_version-control_software

The benefits of a distributed version control system

https://about.gitlab.com/topics/version-control/benefits-distributed-version-control-system/

What is Cloning?

https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository

Week 2 - Unix commands

Piping and Redirection

There are three types of redirection, in Unix systems:

Input (0);
Output (1);
Error (2);

ls -l /bin/usr > text_or_texterror.txt 2>&1

We are redirecting the output, from the standard path-file (the terminal), to text_or_texterror.txt, by using =’>’= (redirection). Also, we want that, in case of error, the output also be sent to this file. So, we use the syntax 2>&1, which means, in case of error, redirect also to the conveined output-file.

Additonal Resources

Agile methodologies

https://www.planview.com/resources/guide/agile-methodologies-a-beginners-guide/

Installing git on mac and windows, detailed instructions.

https://git-scm.com/book/en/v2/Getting-Started-Installing-Git

Bash Reference Manual

https://www.gnu.org/software/bash/manual/html_node/index.html#SEC_Contents

Bash Redirections

https://www.gnu.org/software/bash/manual/html_node/Redirections.html#Redirections

Bash Cheatsheet

https://devhints.io/bash

Grep Cheatsheet

https://devhints.io/grep

Grep Manual

https://man7.org/linux/man-pages/man1/grep.1.html

History and Timeline of Unix

https://unix.org/what_is_unix/history_timeline.html

History of Vim

https://en.wikipedia.org/wiki/Vim_(text_editor)

How to work with relative and absolute paths

https://www.geeksforgeeks.org/absolute-relative-pathnames-unix/

Unix Commands Cheatsheet

https://cheatography.com/jluis/cheat-sheets/bash-and-unix-commands/

Vim Cheatsheet

https://vim.rtorr.com/

Week 3 - Git/GitHub

Git Workflow

Modified

Staged

Committed

Local vs Remote

Example, setting repo

Make fine git101,

mkdir git101

cd into git101, and initiate a new git repository,

cd git101
git init

Initialized empty Git repository in /home/buddhilw/gdrive/PP/Meta/git101/.git/

If we check for a remote, the output will be blank. We need to setup the remote repository.

cd git101
git remote

We can see the current directory this file is in has a remote,

git remote -v

We can set the new git101 to the same repository,

cd git101
git remote add origin https://github.com/BuddhiLW/Meta-backend-course.git

Then, we check and see that git101 is now attached to the remote.

cd git101
git remote -v

cd git101
git pull

There is no tracking information for the current branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.

git pull <remote> <branch>

If you wish to set tracking information for this branch you can do so with:

git branch --set-upstream-to=origin/<branch> master

cd git101
ls

cd git101
git checkout main

cd git101
ls

cd .git
cat refs/heads/main

The hash 4474b383b7d9870b8a50c155eaedd262c18e0182, in our case, points to the current commit. If we were to switch branches, this id would be different.

`git blame`

Format of output message

Selecting range of lines I want to keep track of changes

General syntax,

git blame -L <start>,<finish> <file>

For example,

git blame -L 5,15 setup.py

From, man git-blame

-L <start>,<end>, -L :<funcname>
    Annotate only the line range given by <start>,<end>, or by the
    function name regex <funcname>. May be specified multiple times.
    Overlapping ranges are allowed.

    <start> and <end> are optional.  -L <start> or -L <start>, spans
    from <start> to end of file.  -L ,<end> spans from start of file to
    <end>.

    <start> and <end> can take one of these forms:

    •   number

        If <start> or <end> is a number, it specifies an absolute line
        number (lines count from 1).

    •   /regex/

        This form will use the first line matching the given POSIX
        regex. If <start> is a regex, it will search from the end of
        the previous -L range, if any, otherwise from the start of
        file. If <start> is ^/regex/, it will search from the start of
        file. If <end> is a regex, it will search starting at the line
        given by <start>.

    •   +offset or -offset

        This is only valid for <end> and will specify a number of lines
        before or after the line given by <start>.

    If :<funcname> is given in place of <start> and <end>, it is a
    regular expression that denotes the range from the first funcname
    line that matches <funcname>, up to the next funcname line.
    :<funcname> searches from the end of the previous -L range, if any,
    otherwise from the start of file.  ^:<funcname> searches from the
    start of file. The function names are determined in the same way as
    git diff works out patch hunk headers (see Defining a custom
    hunk-header in gitattributes(5)).

Tip: mix `git blame` and `git log`

Using git blame we have the records of changes with their hash-ids. If we want to know what exactly was going one with a specific change, we can take the id and pass to git log and have a more verbose version of changes that took place in that commit.

Additional Resources

GitHub: Pricing

GitHub is free to use, but it also offers different pricing models to suit the needs of different-sized teams and organizations. Check out the link below:

https://github.com/pricing

Git: An Origin Story

https://www.linuxjournal.com/content/git-origin-story

Git Cheatsheet

https://education.github.com/git-cheat-sheet-education.pdf

Git patterns and anti-patterns for successful developers

https://youtu.be/t_4lLR6F_yk

Tech Talk: Linus Torvalds on git

https://www.youtube.com/watch?v=4XpnKHJAok8

Vim Cheatsheet https://devhints.io/vim

Databases

Week 1

Database: a form of electronic storage that holds data.

Introduction

Use-cases

Banking;
Healthcare;
Shopping.

Examples of data-stored:

Location data;
Duration on the platform;
Connection with other users.

Universal Database Tasks

Store data;
Form relationships;
Filter data;
Search data;
Perform CRUD operations.
Create;
Read;
Update;
Delete.

SQL

SQL operator techniques;
Data normalization concepts;
Data normalization techniques.

Daniel Bloomfield Ramagem

Software Engineer at Meta (Database Engineer).

Types of structured-databases

TODO: put images associeted with slides

Relational Databases

Object Oriented Databases

Graph Databases

Document databases

Relational Databases

Fields: columns, data-types associated. Records: rows, each data instance. Primary key: unique values that identify data uniquely. Foreign key: stores data related to the primary key from another relational-database table.

Business Intelligence

Business Intelligence: Analyzing data and other information to make informed decisions.

SQL

Standard language that can interact with structured data on databases.

Examples:

MySQL;
PostgreSQL;
Oracle;
Microsoft SQL Server.

Database Management System (DBMS)

Changes SQL instructions into a form understood by the database.

Common SQL Tasks

CRUD: Create, Read, Update, Delete.

SQL subsets

Terms used and their meaning:

Term	Meaning
`Object`	datastructure holding data (file, tables, JSON, etc).
`Records`	see Relational Databases.
DDL	Data Definition Language.
DML	Data Definition Language.
DQL	Data Query Language.
DCL	Data Control Language.

DDL

DDL: Data Definition Language

`Create`

Used to create storage objects in a database, like tables.

Example: create a table with certain initial fields.

Purpose: To create the database or tables inside the database

Syntax to create a table with three columns:

CREATE TABLE table_name (column_name1 datatype(size), column_name2 datatype(size), column_name3 datatype(size));

`Alter`

Modify the structure of a table object in a database.

Example: add a field.

Purpose: To change the structure of the tables in the database such as changing the name of a table, adding a primary key to a table, or adding or deleting a column in a table.

Syntax to add a field into a table:

ALTER TABLE table_name ADD (column_name datatype(size));

Syntax to add a primary key to a table:

ALTER TABLE table_name ADD primary key (column_name);

`Drop`

Remove an existing object from a database.

Example: delete a table.

Purpose: To delete a database or a table inside the database.

Syntax to drop a table:

DROP TABLE table_name;

`TRUNCATE`

Purpose: To remove all records from a table, which will empty the table but not delete the table itself.

Syntax to truncate a table:

TRUNCATE TABLE table_name;

`COMMENT`

Purpose: To add comments to explain or document SQL statements by using double dash (–) at the start of the line. Any text after the double dash will not be executed as part of the SQL statement. These comments are not there to build the database. They are only for your own use.

Syntax to COMMENT a line in SQL:

-- -> Retrieve all data from a table
SELECT * FROM table_name;

DML

Data Manipulation Language

Most CRUD operations fall under DML.

`SELECT`

Purpose: To retrieve data from tables in the database.

Syntax to select data from a table:

SELECT * FROM table_name;

`INSERT`

Insert records of data into a database-table.

Example: initialize data in the database-table, by connecting it to a file.

Purpose: To add records of data into an existing table. Syntax to insert data into three columns in a table:

INSERT INTO table_name (column1, column2, column3) VALUES (value1, value2, value3);

`UPDATE`

Edit data that already exists in a database-table.

Change some data-fields in the records.

Purpose: To modify or update data contained within a table in the database.

Syntax to update data in two columns:

UPDATE table_name SET column1 = value1, column2 = value2 WHERE condition;

`DELETE`

Delete one or more rows of data from a database-table.

Purpose: To delete data from a table in the database.

Syntax to delete data:

DELETE FROM table_name WHERE condition;

DQL

`SELECT`

Filter records, based on given criteria that can be, for example, a subset of fields, for a range of records.

DCL

Changes and controls permissions to access records and fields in the database.

`GRANT`

`REVOKE`

TCL

Transaction Control Language (TCL)

The TCL commands are used to manage transactions in the database. These are used to manage the changes made to the data in a table by utilizing the DML commands. It also allows SQL statements to be grouped together into logical transactions. This category deals with advanced functions or operations in a database. Note that this category can have a generic description of the two main commands. Use the following commands in this category:

COMMIT Command to save all the work you have already done in the database. ROLLBACK Command to restore a database to the last committed state.

Example of SQL commands

CREATE DATABASE College;
CREATE TABLE Students;

INSERT INTO Students (column_ID,   first_name, last_name, data_of_birth);
VALUES (1, Pedro, Branquinho, 1997-10-08);
INSERT INTO Students (column_ID,   first_name, last_name, data_of_birth);
VALUES (2, Cassandra, Yorker, 1990-03-22);

UPDATE Students
    SET date_of_birth = '1997-10-07'
    WHERE ID = 02;
DELETE FROM Students
        WHERE ID=02;

SELECT first_name, last_name,
FROM Students
WHERE ID = '01';

Tables types and constrains

For example, here are some of the data types that are available:

Numeric data types such as INT, TINYINT, BIGINT, FLOAT, REAL.

Date and time data types such as DATE, TIME, DATETIME.

Character and string data types such as CHAR, VARCHAR.

Binary data types such as BINARY, VARBINARY.

Miscellaneous data types such as:

Character Large Object (CLOB) for storing a large block of text in some form of text encoding.

Binary Large Object (BLOB) for storing a collection of binary data such as images.

All fields should have some kind of constrain. e.g., an email must have the symbol @. Whereas, personal names can’t.

Entity Relationship Diagram (ERD)

It is a visual representation of how the database will be implemented into tables during physical database design, using a Database Management System (DBMS) like MySQL or Oracle, for example.

Additional Resources

The following resources are some additional reading material that introduces you to the concept of a database, different types of databases, about relational databases in specific and also about the history of databases. These will add to the knowledge that you’ve got on these areas throughout this lesson.

Oracle https://www.oracle.com/uk/database/what-is-database/ Javapoint https://www.javatpoint.com/types-of-databases IBM https://www.ibm.com/cloud/learn/relational-databases Tutorialspoint https://www.tutorialspoint.com/Types-of-databases Graphdatamodeling http://graphdatamodeling.com/GraphDataModeling/History.html

The following resources are some additional reading material that introduces you to SQL. These will enable you to enhance your knowledge on SQL syntaxes and common SQL commands (DQL, DML and DDL) that you’ve learned throughout this lesson.

Javapoint https://www.javatpoint.com/dbms-sql-introduction

Beginnersbook https://beginnersbook.com/2018/11/introduction-to-sql/

The following resources are reading material that provide additional knowledge on database design and Relational Database structure.

Microsoft https://support.microsoft.com/en-us/office/database-design-basics-eb2159cf-1e30-401a-8084-bd4f9c9ca1f5 IBM https://www.ibm.com/docs/en/control-desk/7.6.0?topic=design-relational-database-structure

Week 2 - Data Types

Types

How to start sql client on terminal

Given that you installed MariaDB and configured it for buddhilw with a password. Also, you have permission to buddhilw to access the mydb database (Follow https://wiki.archlinux.org/title/MariaDB#Add_user). Then, you can:

mysql -u buddhilw -p

OBS.: if you enconter the error, after MariaDB install: error 2002 (hy000): can't connect to local server through socket '/run/mysqld/mysqld.sock' (2) archlinux. Then, follow the instructions in: https://archlinux.org/news/mariadb-replaces-mysql-in-repositories/

Example1

CREATE DATABASE mydb;
USE mydb;

CREATE TABLE w2ex1 (Device_ID int, Qnt int, Tot_Price decimal);

show tables;

+----------------+
| w2ex1          |
+----------------+
1 row in set (0,001 sec)

show columns from w2ex1;

Example2

Using CHAR and VARCHAR.

We start by lauching mysql,

mysql -u buddhilw -p

Then, we create a new table in the mydb database.

USE mydb;
CREATE TABLE w2e2 (Username CHAR(9), Full_Name VARCHAR(50), Email VARCHAR(30));

This way, we can hold the following example data:

Username	Full_Name	Email
Custom001	John Johnson	J.Johnson@email.com
Custom002	Carl Schmidt	Carl.Schmidt@email.com
Custom003	Yara Suliman	Yara.Suliman@email.com

show columns from w2e2;

`NOT NULL` and `DEFAULT`

`NOT NULL`

CREATE TABLE Customer (
customer_id int NOT NULL,
customer_name varchar(255) NOT NULL
);

`DEFAULT`

CREATE TABLE Player (
name_player varchar(50) NOT NULL,
city varchar(30) DEFAULT "Barcelona"
);

Exercises

mysql -u buddhilw -p

CREATE TABLE w2e3 (Customer_ID INT NOT NULL, Street VARCHAR(50), Postcode VARCHAR(50) DEFAULT "HA3 0AE", Town VARCHAR(30) DEFAULT "Harrow");

SHOW COLUMNS FROM w2e3;

`ALTER` - `add`, `drop` and `modify`

Let’s say we want to add a field, `Goods`

ALTER TABLE w2e3 ADD (Goods VARCHAR(30), Price DECIMAL, Sender VARCHAR(50));

We can `drop` fields

If any data-field is redundant or not needed, we can drop that field.

ALTER TABLE w2e3 DROP COLUMN Price;

Finally, `modify`

In case some field happens to hold data previously not supported, you can include it by,

ALTER TABLE w2e3 ALTER Sender VARCHAR(20);

`INSERT`-ing data into a table

INSERT INTO w2e4 (c1, c2, c3)
            VALUES (v11, v21, v31),
                   (v12, v22, v23),
                   (v13, v23, v33);

Example

mysql -u buddhilw -p

USE mydb;

CREATE TABLE w2e4 (funcID TINYINT, age TINYINT, work_time TINYINT, whentook DATE, status VARCHAR(30));

show columns from w2e4;

ALTER TABLE w2e4 MODIFY whentook DATE DEFAULT CURRENT_DATE();

Finally, we add the records,

INSERT INTO w2e4(funcID, age, work_time, whentook, status)
            VALUES (1, 27, 3, DEFAULT, "Recepcionist"),
                   (2, 30, 2, DEFAULT, "Collector");

SELECT * FROM w2e4;

In this example, we take a database called players, and select it’s country field. Then, we insert into the country database, with the new field-name as countryName.

INSERT INTO country(countryName) SELECT country FROM players

Exercise

USE mydb;
CREATE TABLE soccer (playerId int, name varchar(60), height decimal, weight decimal, avgSpeed decimal);
INSERT INTO soccer (playerId, name, height, weight, avgSpeed)
            VALUES (1, "João", 1.72, 80, 10),
                   (2, "Mario", 1.85, 86, 8);

SELECT * FROM soccer;

ALTER TABLE soccer MODIFY height decimal(1,2), weight decimal(5,2), avgSpeed decimal(5,2);

SELECT * FROM soccer;

TODO: We still have to modify the height to: 1.72, 1.85. DONE: See Update and Delete (next section).

Update and Delete

UPDATE soccer
SET height = '1.72'
WHERE playerId = 1;

UPDATE soccer
SET height = '1.85'
WHERE playerId = 2;

ALTER TABLE soccer ADD position VARCHAR(30);
-- ALTER TABLE w2e3 ADD (Goods VARCHAR(30), Price DECIMAL, Sender VARCHAR(50));

UPDATE soccer
SET position = 'runner'
WHERE weight >= 80;

SELECT * FROM soccer;

DELETE FROM soccer
WHERE name = 'João';

Exercise

CREATE TABLE customers(customerID int, customerName varchar(40), customerAddress varchar(40))

INSERT INTO customers (customerID, customerName, customerAddress) VALUES
(1, 'Jack', '115 Old street Belfast'),
(2, 'James', '24 Carlson Rd London'),
(4, 'Maria', '5 Fredrik Rd, Bedford'),
(5, 'Jade', '10 Copland Ave Portsmouth '),
(6, 'Yasmine', '15 Fredrik Rd, Bedford'),
(3, 'Jimmy', '110 Copland Ave Portsmouth');

DELETE FROM customers
WHERE customerID = 3;

Additional Resources

The following list of resources explore the meaning and role of different data types in databases. They also provide good examples for how to declare and use the data types in SQL.

W3schools https://www.w3schools.com/sql/sql_datatypes.asp W3resource https://www.w3resource.com/mysql/mysql-data-types.php LearnSQL https://learnsql.com/blog/understanding-numerical-data-types-sql/ Microsoft https://docs.microsoft.com/en-us/sql/t-sql/data-types/decimal-and-numeric-transact-sql?view=sql-server-ver16s MySQL https://dev.mysql.com/doc/refman/8.0/en/numeric-types.html

The following resources are some additional reading material that provide extra knowledge on SQL DDL and DML commands:

Tutorialspoint https://www.tutorialspoint.com/sql/index.htm Javatpoint https://www.javatpoint.com/sql-tutorial Tutorialrepublic https://www.tutorialrepublic.com/sql-tutorial/sql-create-database-statement.php

Here is a list of resources that may be helpful as you continue to explore database engineering: Tutorialspoint https://www.tutorialspoint.com/sql/index.htm Javatpoint https://www.javatpoint.com/sql-tutorial W3Schools https://www.w3schools.com/sql

Week 3 - SQL Arithmetic Operators

What are operators and why do I care? (221117114432-03)

With operators you can apply data a transformation to subsections of data.

This can be used to filter-and-select data, create a new field (column), or to select-and-apply processes.

Syntax for operators (221117115014-03)

Use the SELECT key-word to effectuate a operation. Further use FROM, in order to map operations to table-columns.

SELECT column_name1 + column_name2 FROM table_name;

Use as a filter-and-select tool

SELECT *

FROM employee

WHERE salary - tax = 50000;

Use as a select-and-apply

SELECT salary - tax  -- Apply functionality

FROM employee

WHERE hours % 2 = 1; -- Select functionality

`ORDER BY` clause - sorting (221117124242-03)

Either arrange data from a field, or group of fields, in ascending or descending order.
Types of data will have different ascending and descending sorting types (alphabetical, numerical etc.)
ASC by default.

Syntax for ordering (221117125022-03)

SELECT column_1, column_2, column_3, ... -- or "*"
FROM table_name
ORDER BY column_1 ASC, column_2 DESC;

`WHERE` Clause with `BETWEEN`, `LIKE`, `IN` (221117130430-03)

`BETWEEN`

SELECT *
FROM table_name
WHERE faculty_age BETWEEN '18' AND '33'

`LIKE`

SELECT *
FROM table_name
WHERE degree LIKE 'Sc%' -- regex matches

`IN`

SELECT *
FROM table_name
WHERE first_name IN('String1', 'String2', ...) -- regex matches

BuddhiLW/Meta-backend-course

Notes about the course

Table of Contents

Schedule

Introduction to Back-end Development

Week 1

Capstone project overview

Create Django web server, with multilple API endpoints.

Connect it to a MySQL database.

Create a template system driven by Django views.

How the web works

Fullstack

Front-end

Back-end

Grace Egbo - a day in the life of a backend dev.

Code review

Possible quick fixes

Meeting about ideias and needs

Coding

Soft skills/People skills

How the internet works

Definition: Network

Multiples devices in a network - Network Switches

Definition: Interconnected network

Servers

Webservers

Web request

What are websites and webpages?

Webpage

Website

HTML, CSS and JavaScript

HTML

CSS

JavaScript

Page rendering process

Web browsers

URL

HTTP

TCP - Transmission Control Protocol

UDP - User Datagram Protocol

HTTP - HyperText Transfer Protocol

Makeup of a HTTP request

Method

Get

Post

Put

Delete

Path

Version

Headers

Makeup of a HTTP response

Header

Informational

Successful

Redirection

Client error

Server error

Examples about HTTP

Request Line

HTTP Method

IDEs

Extra resources

Week 2

HTML

Table

Forms

Input tags

DOM

CSS

Extra Resources

HTML and DOM

Week 3

Working with libraries

Responsive Design

Flexible grids

Fluid images

Media queries

Types of Grids

Fixed Grid

Fluid grid

Definition: `Network`

Definition: `Interconnected network`

`with open`