The postcode search builder creates and populates a postcode document into azure search containing a geographical location (latitudinal and longitudinal point). This is used to search for a course in the course search index using the retrieved latitudinal and longitudinal values for a postcode and using these two values plus the distance from postcode (e.g. 25miles) to find all courses within a given distance from the postcode inputted into a request to discoverUni's Search API.
-
Course search builder is triggered when a new blob (postcodes) are stored in a storage container. It takes the path to the file and extracts the csv gzip file.
-
Following the recreation of the postcode index, the function iterates all postcode and validates they are in the region (latitude and longitude outlyers) of the UK including the Shetland islands.
-
Once validated the postcodes are stored against the postcode search index and are available for the search API to search against.
This process takes roughly 30minutes to complete
The postcode data that is uploaded to an azure storage container to which the azure function is configured to be triggered on, was taken from free map tools as a csv file.
The csv file should contain the following header row (this is validated in the azure function):
| id | postcode | latitude | longitude |
|---|---|---|---|
| 1 | AB10 1XG | 57.14416516 | -2.114847768 |
| {n} | ... | ... | ... |
There are approximately 1.75million postcodes - these will need to be updated manually by downloading the csv file from free map tools, compressing the file using gzip and uploading the file into the azure storage container that the PostcodeSearchBuilderBlobTrigger function triggers on.
Non geographic postcodes are listed below. Note these will have a latitude and longitude of an empty string (see data source)
AB99, BT58, CA99, CM92, CM98, CR44, CR90, GIR, IM99, IV99, JE5, M61, ME99, N1C, N81, NR99, NW26, PA80, PE99, RH77, SL60, SO97, SW95, SY99, WD99, WF90
Add the following to your local.settings.json:
| Variable | Default | Description |
|---|---|---|
| FUNCTIONS_WORKER_RUNTIME | python | The programming language the function worker runs on |
| AzureStorageAccountConnectionString | {retrieve from portal} | The connection string to access storage account |
| AzureWebJobsStorage | {retrieve from portal} | The default endpoint to access storage account |
| StopEtlPipelineOnWarning | false | Boolean flag to stop function worker on a warning |
| PostcodeIndexName | postcodes | The storage container that will trigger the function |
| SearchURL | {retrieve from portal} | The uri to the azure search instance |
| SearchAPIKey | {retrieve from portal} | The api key to access the azure search instance |
| AzureSearchAPIVersion | 2019-05-06 | The azure search API version for instance |
- Install .Net Core 2.2 SDK, if you haven't already.
- Install python 3.6.8 - the latest stable version that works with Azure client.
Mac user:
Install homebrew:
1) /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
2) brew install sashkab/python/python36
3) pip3.6 install -U pip setuptools
Windows user:
- Make sure Python 3.6.8 is set on your PATH, you can check this by running
python3 -vin terminal window. - Install Azure Client
Mac user:
brew tap azure/functions
brew install azure-functions-core-tools
Windows user:
- Setup Visual Studio Code, install visual studio code
- Also install the following extensions for visual studio code - documentation here
Python
Azure CLI Tools
Azure Account
Azure Functions
Azure Storage
- Sign into Azure with Visual Studio Code - follow documentation here
-
Create an azure search resource in azure portal, see Microsofts documentation here
-
Retrieve the azure search url and search api key from portal, (these will need to be added to you local.settings.json file)
-
Create/Reuse existing azure storage account, see Microsoft documentation here
-
Retrieve the azure storage account connection string and default endpoint, (these will need to be added to you local.settings.json file)
-
Create your local.settings.json file at root level of repository and include all environment variables in the configuration settings table above.
-
Create a Python virtual env to run the azure function application by running
venv .envat root level of repository. -
Run service on Python virtual env by doing the following:
source .env/bin/activate
pip install -r requirements.txt
func host start
-
Download latest csv UK postcode from here
-
Compress csv using gzip,
gzip <path to file>, the current filename isukpostcodes.csv -
Upload the gzip file to postcodes blob container, this can be done in the Azure Portal or in Visual Studio Code and will trigger a new build of postcode resource into a poscode search index.
To run tests, run the following command: pytest -v
See CONTRIBUTING for details.
See LICENSE for details.