Suggested re-write of app.md
Closed this issue · 5 comments
Please consider the following restructuring of this lesson. Some of the specific explanations may need revision, but I think overall it flows well and allows the student to build a working app as well as dig deeper into the functional components within the REPL for greater understanding. Please note it does assume some small changes to the current shell-project version of the global-growth app.
Making Your Own Web Application
The World Bank provides a data collection of world development indicators, showing the current state of global development, as well as a web API to provide access to this data. A web API is a way to provide access for one program to call another program over HTTP. In this case, the World Bank Indicators API provides access to their set of data.
We will use the World Bank Indicators API to explore some of the world development indicators for different countries. We will sort and compare certain indicators. This is a task that Clojure is well suited for.
Getting Started
- Launch Light Table
- Click File > Open folder, select the
global-growthfolder, and click the Upload button
You should now see global-growth/ at the top left of the Light Table screen. Click this to expand the directory structure, and then click core.clj to load the file in the main window.
This file has been pre-populated with 3 sections:
1. Namespace, dependencies, and constants
As seen in previous lessons, a namespace allows you to define new functions and data structures without worrying about whether the name you'd like is already taken.
Dependencies are just code libraries that others have written which you can incorporate in your own project. Click on project.clj from the directory structure in Light Table to view this file. Take a look at the :dependencies section, which includes entries like clj-http, a library that allows you to communicate over HTTP, and cheshire, another library that reads and understands the JSON format.
Click the core.clj tab at the top to go back to our main file. Notice the :require section, which is where the libraries loaded by project.clj are included and given aliases (shorter names) to be referenced later in our program when we need functionality that they provide.
2. Support functions
(defn remove-aggregate-countries
"Remove all countries that aren't actually countries, but are aggregates."
[countries]
(remove (fn [country]
(= (get-in country [:region :value]) "Aggregates")) countries))
(defn get-country-ids
"Get set of country ids so we can filter out aggregate values."
[]
(let [countries (remove-aggregate-countries (:results (get-api "/countries" {})))]
(set (map :iso2Code countries))))
(def country-ids (get-country-ids))These functions serve to build up our list of countries from data returned by the API to be used by our application. Like many well-written functions in Clojure, you can often get a good idea about what they are doing even if you don't understand exactly what each part means.
3. Web layout, controls, and routes
This is the part of the application that controls how your app will appear in the browser, and what content will be in each control (such as a drop-down menu). It also handles basic text and number formatting and what page to load based on the URL requested.
In the next section, we will add the code necessary to download, filter, and sort the data based on the selections made by the user on the web page. Before we get to that, however, let's see what happens if we run our application in its current state.
Enter this at the command line:
cd global_growth
lein ring server
BOOM! You should see some ominous looking output similar to this:
Exception in thread "main" java.lang.RuntimeException: Unable to resolve symbol: get-api in this context, compiling:(global_growth/core.clj:23:57)
at clojure.lang.Compiler.analyze(Compiler.java:6380)
at clojure.lang.Compiler.analyze(Compiler.java:6322)
...
The informative part of this error is Unable to resolve symbol: get-api in this context on line 23, character 57. If you click on the core.clj file in Light Table, you will see a line number and character count in the bottom right corner. Placing the cursor on line 23, character 57 (or whatever number is in your error message) will show you the problem, which is the call to the get-api function. Since we have not yet written that function, the program cannot run. Let's do that now.
Calling the API
Our get-api function will take two parameters to hold the path in the API we are calling (such as "/countries") and any additional query parameters that need to be added to the API URL. You can put all of the above pieces into the let part of the function. You need: base-path, query-params, response, metadata, and results. Then return a map with keys :metadata and :results, each of which has that corresponding value.
Add the following code to your core.clj file right after the parse-json line near the top of the file.
;; WORLD BANK API CALLS
(defn get-api
"Returns map representing API response."
[path qp]
(let [base-path (str base-uri path)
query-params (merge qp {:format "json" :per_page 10000})
response (parse-json (:body (client/get base-path {:query-params query-params})))
metadata (first response)
results (second response)]
{:metadata metadata
:results results}))Define a var to hold the results of calling get-api for the countries endpoint.
(def countries (get-api "/countries" {}))Notice that when we call get-api we are including the path and enabling the passing in of query-params to the API. Save your changes to core.clj.
Let's take a moment now to look at what is going on here. In order to really understand how the get-api function works, let's fire up the REPL and run through a few exercises.
REPL EXERCISE
Click the first line of the file (anywhere on the line with ns-global-growth.core) and then press cmd-shift-enter to load the project and any dependencies into the REPL session (you will see the activity in the lower left corner of the Light Table window while this is taking place).
Once everything is loaded up, press Ctrl-Space to bring up the search command box. Type "insta" and press enter when the "Instarepl: Open a Clojure instarepl" choice is highlighted.
You are now ready to execute functions.
As mentioned previously, we use the clj-http library to enable making calls over the internet with HTTP. In the Instarepl tab, type in the following line:
(require '[clj-http.client :as client])Now let's walk through the components of the get-api function to see what each one does.
Go ahead and set the address that our application will use to reach the World Bank API by entering:
(def base-uri "http://api.worldbank.org")Now whenever we need the address for the World Bank API, we can use base-uri instead of the longer URL.
Each type of data available through the World Bank API has an API endpoint. The endpoint has a specific path added to the base URL. http://api.worldbank.org/countries is the endpoint that lists all of the countries with data available. Create a var to refer to that URL by combining the base-uri and "/countries" by entering:
(def base-path (str base-uri "/countries"))Now make a call using the get function (courtesy of the clj-http library) to the full API endpoint: http://api.worldbank.org/countries, which should result in a bunch of light blue text with country names and other values appearing just to the right of what you typed within Light Table.
(client/get base-path)There, you've done it! You have successfully used Clojure to ask the World Bank for information and received back a response with the raw data. The other lines in the get-api function simply set more specific parameters for your query and assign values to vars for use elsewhere in the application.
This line,
query-params (merge qp {:format "json" :per_page 10000})just means "please send me results in the JSON format, 10,000 values at a time."
The other two lines,
(def metadata (first response))
(def results (second response))are used to assign the metadata, such as latitude and longitude for each country, and the actual indicator results, such as income level, for each country, respectively.
Filter the results
Click back to the core.clj tab, and we will continue building out our application. To add the filtering functions, place the cursor after the supporting function section, right after the following line:
(def country-ids (get-country-ids))On the next line create a function called get-value-map where the two keys that we want to look up are parameters that can be passed to the function. Put the values into a map with "into {}"
;; FILTER THE RESULTS
(defn get-value-map
"Returns relation of two keys from API response"
[path query-params key1 key2]
(let [response (get-api path query-params)]
(into {} (for [item (:results response)]
[(key1 item) (key2 item)]))))Now create another function called get-indicator-map.
(defn get-indicator-map []
"Gets map of indicators.
/topics/16/indicators: All urban development"
(get-value-map "/topics/16/indicators" {} :name :id))Define a var to hold the results of calling get-indicator-map.
(def indicator-map (get-indicator-map))Finally, create a function called get-indicator-all. The indicator name needs to be a parameter to the function. So should the data and two keys. We make these parameters so we will be able to get other indicators and pull out different pieces from the results.
(defn get-indicator-all
"Returns indicator for a specified year for all countries"
[indicator year key1 key2]
(get-value-map (str "/countries"
"/all"
"/indicators"
"/" indicator)
{:date (str year)}
key1
key2))REPL EXERCISE
To get a better grasp on what we have just accomplished, click back to the Instarepl tab and place the cursor on a new line after everything we typed in the previous exercise.
Go ahead and add in the cheshire library, which reads and understands the JSON format. Then we need a function to read (parse) the json we will receive back from the World Bank API. Passing the second argument as true tells the function to return Clojure keywords (i.e. :incomeLevel) back for the map keys.
(require '[cheshire.core :as json])
(defn parse-json [str]
(json/parse-string str true))Next, copy over the whole get-api function from core.clj, including the countries definition line, and paste in the Instarepl window on the line after the parse-json function.
We now have what we need (the ability to download and process the raw World Bank data) available in the Instarepl to allow us to walk through the components of the filtering functions.
Let's go through the raw data and pull out a couple values associated with keys in the results. You can use the for function to do this. We want to pull out the values for the "name" and "longitude" items from the results section. Those will have been turned into Clojure keywords by the cheshire json library, so you can refer to them as :name and :longitude. Add the following lines in the Instarepl:
(for [item (:results countries)]
[(:name item) (:longitude item)])Behold! A list of countries and their longitudes should appear alongside your function. Our function has taken the raw data set (countries) and returned only the specific information that we asked for, in this case country name and longitude. Look back at the get-value-map in core.clj to see how we are using similar logic to load up all of the indicator names and their values. To see this in action, copy the full get-value-map function from core.clj and paste it into the Instarepl.
"Hey, nothing happened!" That is because we simply added the function for use inside the Instarepl, but have not actually told it to run. Now let's put it to use. On the next line in the Instarepl, type:
(get-value-map "/topics/16/indicators" {} :name :id)Voila! The names and id's of all the world's development indicators (all 16 of them) should appear next to your function.
You should be able to see now how the core.clj function get-indicator-all uses this logic to build a map of all the indicators and their values for each country. Go ahead and copy the full get-indicator-all function to the Instarepl so we can use it in the next exercise.
Sort the data
After the filtering section, create a new function called sorted-indicator-map.
;; SORT THE DATA
(defn sorted-indicator-map
"Sort the map of indicator numeric values"
[inds]
(take list-size
(sort-by val >
(into {} (for [[k v] inds
:when (and v (country-ids (:id k)))]
[(:value k) (read-string v)])))))Save your changes to core.clj.
REPL EXERCISE
To help you understand what is going on here, click back to the Instarepl tab and place the cursor on a new line after everything we typed in the previous exercise.
Enter the following line:
(def inds (get-indicator-all "EP.PMP.SGAS.CD" "2012" :country :value))Since EP.PMP.SGAS.CD is the indicator name for gas pump price, you will see a list containing value pairs that describe each country with keys :id and :value, followed by a value with the actual pump price.
For the next step, you will need to first copy over the "Supporting Functions" to the Instarepl, which we talked about earlier in this lesson. These are remove-aggregate-countries and get-country-ids, as well as the line where we define country-id. Go ahead and do that.
Now enter:
(for [[k v] inds
:when (and v (country-ids (:id k)))]
[(:value k) (read-string v)])What this function does is take the list of countries (after the raw data has been processed by the supporting functions to remove aggregate countries and build a list of country names with matching id's), then creates a new list of gas pump price by country name. Next, load this list into a map by typing:
(into {} (for [[k v] inds
:when (and v (country-ids (:id k)))]
[(:value k) (read-string v)]))Notice how instead of the previous ["name" price] ["name" price] ["name" price] format, the map of countries and prices looks like ```{"name" price, "name" price, "name", price ... }.
Lastly, sort the list by price (value), and return the top 10.
(take 10
(sort-by val >
(into {} (for [[k v] inds
:when (and v (country-ids (:id k)))]
[(:value k) (read-string v)]))))That is really all sorted-indicator-map is doing. Make sense?
Running the app
Now go back to your command line in the global_growth directory and enter:
lein ring server
After several seconds, you should see a couple lines of INFO output followed by:
Started server on port 3000
To view the running application, open your web browser and go to http://localhost:3000
Congratulations, you have created your first Clojure web application!
Thanks @turbomarc. This is a nice rewrite.
I'm going to defer to @cndreisbach at this point. I'm not sure what the best way to present this is.
One thing I would like to do, though, is either pick to do the input into the core.clj source file or use the Instarepl. I would prefer not to use the Instarepl at all now.
Using this, I've done a major rewrite of first-program.md. I think the way you've got it written here, @turbomarc, is better than mine, but we're coming up against the deadline, so I'm going to leave it for now. Not closing this, as I want to circle back to it after the workshop.
Hey @cndreisbach - do you think we still need to rewrite this section? Or have later updates superseded this?
Hi, @cndreisbach and @bridgethillyer and @turbomarc!
This issue has been hanging around for seven months now--should we close it, or...? Was this added to the app curriculum after the workshop?
I believe this was incorporated. Also, I don't believe the global growth app is being maintained anymore unless someone wants to take that up. So - yes, close this.