# README This is the manual for Raw Sockets, a Python implementation of the TCP/IP protocol stack. This project is implemented using Python2.7. And the C-style structure used to compose TCP stack packets are from Python struct module. To run the program, run: ./rawhttpget URL For basic TCP/IP feature, I implemented the protocols stack with TCP/IP checksum, timeout/retransmission, congestion window and advertised window, etc. =============================================================================== REALLY IMPORTANT I utilized some feature of the network interface in Linux, and by default, 'eth0' will be used. Since the main network interfaces in different machines could be varying, so that an optional argument of the main executable 'rawhttpget' has been made. Please remember to change the correct network interface on your machine if the currently working one is not 'eth0', such as: ./rawhttpget -i eth1 URL The program will use 'eth0' by default. =============================================================================== Data Link Layer features The raw socket is in AF_PACKET family, which bypasses the operating systems layer-2 stack as well at layers 3 and 4. For Ethernet functionality, I have built my own Ethernet frames wrapping IP datagram and ARP request. Before sending any high level message to remote host, the raw socket in AF_PACKET family must query its gateway MAC address for building the subsequent Ethernet frame, so the first Ethernet frame sent is a broadcast, wrapping an ARP request for every host in the LAN to query the gateway MAC address. Once the gateway sends back an ARP reply. I can extract its MAC address from the ARP reply and embed this MAC address to our subsequent Ethernet frames. There are 2 challenges I faced when working on layer 2. 1. Before sending the broadcast ARP request for querying gateway MAC address, how to get the gateway IP address? I finally hacked it that the gateway IP address could be obtained from the file /proc/net/route, which is the static routing table file in a Linux system, and the encoded gateway IP address of the given network interface (e.g. eth0) is there. 2. I did not realize that in layer 2, the raw socket usually adds some padding bytes at the end of an Ethernet frame, which yields a packet as below: ------------------------------------------------------------------- | | | | | | | Ethernet Header | IP Header | TCP Header | HTTP data | Paddings | | | | | | | ------------------------------------------------------------------- This cannot be a problem when working with SOCK_RAW/IPPROTO_TCP, because what initially get from the raw socket is an IP datagram, and the Ethernet header and trailing paddings have been removed by the raw socket. But when using AF_PACKET, what get from the raw socket is an Ethernet frame, since the minimum size of the user data in an Ethernet frame is 46 bytes, hence the raw socket would add some trailer to some 'short' Ethernet frame. So that's why a lot of TCP checksum failure gets generated because the Ethernet frame paddings have not been removed. To illustrate such situation, we can assume an Ethernet frame containing a TCP/IP packet composed of a 20 byte IP header and 20 byte TCP header, so the raw socket would pad the payload with 6 bytes. To calculate the actual length of the packet, we need to use the total length and header length values in the IP header and the offset value in the TCP header. =============================================================================== Makefile The project root directory is raw_sockets. Run 'make' would create a symbolic link 'rawhttpget' under the root directory, which is linking to the executable Python script 'rawhttpget.py' in the ./src directory. Also the iptables would be modified during 'make' so that you can use the raw socket without losing any packets, and this needs you have sudo privilege. Run 'sudo make clean' would purge all binary, executable links and transient files inside the project directory. Run 'sudo make test' would kick off an integration testing script 'test.sh' in the ./test directory, which will run 'rawhttpget' several time to download files with the given urls. Note that this might take minutes since there is a url pointing to a 50MB file in the script. =============================================================================== The Design rawhttpget.py Main entry of the raw socket program. HttpClient.py, HttpParser.py Python modules reused from project-2, mainly process all HTTP related issues. rawurllib.py Simple wrapper of the url based application layer module, works compactly with the above 2 modules. rawsocket.py A socket module integrating the TCP/IP protocols stack, very similar to the generic socket module of Python on functionality. rawtcp.py Simple Python model for easily packing and unpacking TCP segment. rawip.py Simple Python model for easily packing and unpacking IP datagram. rawethernet.py Simple Python model for easily packing and unpacking Ethernet frame. rawarp.py Simple Python model for easily packing and unpacking ARP packet. utils.py Contains useful util functions and classes, the checksum algorithm gets implemented here for global usage through the TCP/IP stack. logger.py A simple logger that can log message in different severity level, could enter silent mode. ===============================================================================