Research eCFR API and the capabilities

[patphongs]

Summary

What we're after:
Need to research the following areas in eCFR:

• Text search
• Search by citation
• Pulling in the most up-to-date list of citations
• Highlighting
• Comparison (something our regulations app does not currently have, but would be nice to implement if we can)
• Navigation of Title 11 via the API

eCFR API documentation: https://www.ecfr.gov/developers/documentation/api/v1#/Search%20Service/get_api_search_v1_results

Related issues

• #589
• fecgov/fec-cms#6089
• fecgov/fec-proxy#393

Completion criteria

• Capabilities are documented in this ticket and plan has been determined for next steps
  - See findings in comment below

[johnnyporkchops]

Navigation:

The structure endpoint returns the nested structure of Title 11 for a given date:
/api/versioner/v1/structure:

• An execute in swaggar-docs returns a download file link where you can download a hash-named file.
• A direct call to the endpoint in a browser automatically downloads the title-11.json file.
  - https://www.ecfr.gov/api/versioner/v1/structure/2024-02-01/title-11.json
• A fetch call in an application, you can call the endpoint and capture the data to the promise without downloading the file

Text search

The results endpoint returns works for text searches
/api/search/v1/results

• Exampe text search for “firewall”:
  - https://www.ecfr.gov/api/search/v1/results?query=firewall&agency_slugs%5B%5D=federal-election-commission&per_page=20&page=1&order=relevance&paginate_by=results
  - Returns expected reg 109.21 .

Highligthting:

The queiried string in the text search above is wrapped in a strong tag in the excerpt, which gives up potential to highlight.

Content:

The full endpoint returns raw XML (without the hyplinked cross-references our Eregs content has):
/api/versioner/v1/full
Also see XML Users-Guide

• Example API call for XML for 6.103:
  - https://www.ecfr.gov/api/versioner/v1/full/2024-02-01/title-11.xml?section=6.103

• Also the undocumented eCFR “renderer” endpoint can return html with hyperlinks added (This endpoint is not in swaggar docs so I dont know if it is actually a true endpoint:
  Exampe: https://www.ecfr.gov/api/renderer/v1/content/enhanced/2024-02-08/title-11?chapter=I&part=6&section=6.103

  • To navigate to this html, click on “developer tools” button on the left sidebar while in an eCFR section.
  • Need to test this option more for this reason: While the URL returns html, calling that endpoint inside a fetch failed to return the html to my test app even though the XHR payload in the Network tab of inspector shows the html.
  • Firefox inspector > network tab gives a more revealing xhr error Response body is not available to scripts (Reason: CORS Missing Allow Origin)

Citations:

The xml includes a list of citations at the bottom of the xml inside the <cita> tag.
/api/versioner/v1/full

Example:
<cita type="N">[45 FR 15104, Mar. 7, 1980, as amended at 50 FR 50778, Dec. 12, 1985; 54 FR 34109, Aug. 17, 1989; 54 FR 48580, Nov. 24, 1989; 58 FR 42173, Aug. 6, 1993; 65 FR 38422, June 21, 2000; 68 FR 3995, Jan. 27, 2003; 68 FR 64516, Nov. 14, 2003; 68 FR 67018, Dec. 1, 2003; 73 FR 79601, Dec. 30, 2008; 82 FR 60853, Dec. 26, 2017; 84 FR 18699, May 2, 2019; 84 FR 35007, July 22, 2019]
</cita>

• Not sure about search by citation on ECFR API, but on FR API one can try searches by citation number:
  Example: 84 FR 35007 from above: https://www.federalregister.gov/api/v1/documents/84%20FR%2035007.json?fields[]=body_html_url

Comparison

• Many of the endpoints require(or allow as optional) a date to get data so this could be developed into an app using that parameter.
• Also many of the endpoints have more granular date filtering. For example, the summary endpoint has these fields which could be used for point-in-time comparison:
  /api/search/v1/summary
  date
  last_modified_after
  last_modified_on_or_after
  last_modified_before
  last_modified_on_or_before

Next Steps:

Here are some logical next steps that could be grouped into future issues.

• Meet with CFPB to find out how there application works and whether it a similar approach would work on our site
• Discussion and research on what platofrm to use (JS/Vue, Python/Django-Wagtail, something else.
• List the features we want in the product (must haves/nice to haves) and then document what endpoint(s) we would use for that feature
• Test and verify that we can manipulate the data returned from the endpoint(s) to actually build the desired functionality
• Document current and potential blockers and complications