Data Marketplace Aggregator

🔗 Live Demo: https://proto-data-market-aggregator.chrisns.workers.dev/

A Cloudflare Worker that aggregates and searches across multiple data sources including Snowflake, Databricks, ONS, Defra, Agrimetrics, AWS Marketplace, data.gov.uk, and Datarade. This service provides a unified search interface for government and commercial data marketplaces.

Features

Unified search interface across multiple data sources
Real-time data aggregation with parallel processing
Interleaved results from all sources
Responsive HTML output with GOV.UK Design System
Error handling and graceful degradation
Optimized performance with concurrent API requests
Intelligent caching strategy for improved response times
TypeScript-based implementation with strict type checking

Performance

The service is optimized for speed and efficiency:

All data source requests are executed in parallel using Promise.all
No sequential waiting between API calls
Reduced total response time
Graceful handling of slow or failed requests
Results are interleaved as they become available
Consistent date formatting across all sources

Caching Strategy

The service implements an intelligent caching strategy using Cloudflare's caching:

Aggressive caching enabled for all requests (cacheEverything: true)
Custom cache keys based on search parameters for efficient cache hits
Custom cache durations based on response status:
- Successful responses (200-299) are cached for 2 weeks
- Not Found responses (404) are cached for 1 second
- Server errors (500-599) are not cached
Cache is applied consistently across all data sources
Improves response times for repeated searches
Reduces load on upstream APIs
Provides resilience against temporary API outages
Caches all content types, not just static assets

Cache Keys

Each data source uses a unique cache key format:

Snowflake: snowflake-{search_term}
ONS: ons-{search_term}
Defra: defra-{search_term}
Agrimetrics: agrimetrics-{search_term}
Databricks: Uses default URL-based caching

Supported Data Sources

Snowflake Marketplace

Searches through Snowflake's data marketplace via their search API
Returns listings with titles, descriptions, and provider information
Direct links to data sources
POST requests with JSON payload for advanced search capabilities

Databricks Marketplace

Searches through Databricks' data marketplace
Returns dataset listings with detailed information
Client-side filtering by relevance to search terms
Supports title, description, and subtitle matching

ONS SRS Metadata Catalogue

Searches through ONS Statistical Research Service metadata
Returns dataset information with keywords and abstracts
Links to original data sources
Supports advanced search parameters

Defra Data Services

Searches through environmental data from Defra
Returns dataset information with tags and descriptions
Links to the Defra data platform
Handles HTML responses with embedded JSON data

Agrimetrics Data Marketplace

Searches through agricultural and environmental datasets
Returns comprehensive dataset information including:
- Dataset title and description
- Tags and categories
- Creator information
- Access URLs and download links
- Last modified dates
Supports advanced search parameters
Direct integration with the Agrimetrics API
Robust error handling with graceful degradation

Datarade Marketplace

Searches through Datarade's data marketplace
Returns product information including:
- Product title and description
- Provider/author information
- Direct links to product pages
- Product body text with detailed information
HTML response parsing with robust error handling
Graceful handling of malformed responses
Input validation and URL encoding
Direct integration with Datarade's search interface

AWS Marketplace

Searches through AWS Data Exchange listings
Returns comprehensive dataset information including:
- Dataset title and description
- Categories and badges
- Creator information
- Pricing details
- Direct links to AWS Marketplace listings
Supports advanced filtering for Data Exchange products
Handles zlib-compressed responses
Robust error handling with graceful degradation
Direct integration with AWS Marketplace Discovery API
Custom cache configuration for improved performance

data.gov.uk

Searches through the UK government's open data portal
Returns comprehensive dataset information including:
- Dataset title and description
- Organization information
- License details
- Resource URLs and formats
- Last modified dates
Direct integration with CKAN API
Supports advanced search parameters
Robust error handling with graceful degradation
Custom cache configuration for improved performance

OpenDataSoft

Searches through OpenDataSoft's data platform
Returns comprehensive dataset information including:
- Dataset title and description
- Publisher information
- Themes and keywords
- Last modified dates
- Direct links to dataset pages
Direct integration with OpenDataSoft Explore API v2.1
Supports advanced search parameters
Intelligent caching with 2-week duration for successful responses
Robust error handling with graceful degradation
Custom cache configuration for improved performance
UTF-8 encoding support for international queries
Parallel execution with other data sources

NHSBSA Open Data Portal

The NHS Business Services Authority (NHSBSA) Open Data Portal integration provides access to healthcare-related datasets. Features include:

Direct integration with NHSBSA's CKAN-based API
Returns comprehensive dataset information including:
- Title and description
- Publisher details
- License information
- Direct links to dataset resources
- Last modified dates
Supports full-text search across all NHSBSA datasets
Intelligent caching with 2-week duration for successful responses
Robust error handling and UTF-8 encoding support
Parallel execution with other data sources

Open Data Northern Ireland

The Open Data Northern Ireland integration provides access to datasets from the Northern Ireland public sector. Features include:

Direct integration with Open Data NI's CKAN-based API
Returns comprehensive dataset information including:
- Dataset title and description
- Organization information
- License details
- Resource URLs and formats
- Last modified dates
Supports full-text search across all Open Data NI datasets
Intelligent caching with 2-week duration for successful responses
Robust error handling and UTF-8 encoding support
Parallel execution with other data sources

SSEN Distribution

The SSEN (Scottish and Southern Electricity Networks) Distribution integration provides access to electricity network data. Features include:

Direct integration with SSEN's CKAN-based API
Returns comprehensive dataset information including:
- Dataset title and description
- Organization information
- Author details
- Tags and categories
- Resource URLs and formats
- Last modified dates
Supports full-text search across all SSEN datasets
Intelligent caching with 2-week duration for successful responses
Robust error handling and UTF-8 encoding support
Parallel execution with other data sources
Base URL: https://ckan-prod.sse.datopian.com/api/action/package_search
Query Parameters:
- q: The search query

London Datastore

The London Datastore integration provides access to datasets from the Greater London Authority. Features include:

Direct integration with London Datastore's CKAN-based API
Returns comprehensive dataset information including:
- Dataset title and description
- Organization information
- License details
- Resource URLs and formats
- Last modified dates
Direct integration with CKAN API
Supports advanced search parameters
Robust error handling with graceful degradation
Custom cache configuration for improved performance
Base URL: https://data.london.gov.uk/api/action/package_search
Query Parameters:
- q: The search query

York Open Data

The York Open Data integration provides access to datasets from the City of York Council. Features include:

Direct integration with York Open Data's CKAN-based API
Returns comprehensive dataset information including:
- Dataset title and description
- Organization information
- Author and maintainer details
- Groups and categories
- Resource URLs and formats
- Last modified dates
Supports full-text search across all York Open Data datasets
Intelligent caching with 2-week duration for successful responses
Robust error handling and UTF-8 encoding support
Parallel execution with other data sources
Base URL: https://data.yorkopendata.org/api/action/package_search
Query Parameters:
- q: The search query

Health Data Gateway

The Health Data Gateway integration provides access to health-related datasets from various UK health organizations. Features include:

Direct integration with Health Data Gateway's search API
Returns comprehensive dataset information including:
- Dataset title and description
- Publisher information
- Dataset type and subtype
- Keywords and categories
- Contact information
- Population size data
Supports full-text search across all Health Data Gateway datasets
Intelligent caching with 2-week duration for successful responses
Robust error handling and UTF-8 encoding support
Parallel execution with other data sources
Base URL: https://api.healthdatagateway.org/api/v1/search/datasets
Query Parameters:
- view_type: mini
- perPage: 25
- page: 1
- sort: score:desc
POST request with JSON payload:
```
{
  "query": "search_term"
}
```

API Integration Details

Response Format

All data sources are normalized to a common format:

interface ListingResult {
    id: string;
    title: string;
    description: string;
    subtitle: string;
    provider: {
        title: string;
        description: string;
    };
    url: string;
    source: string;
    updated: string;
}

AWS Marketplace API

The integration with AWS Marketplace uses their Discovery API with the following details:

Base URL: https://aws.amazon.com/marketplace/api/awsmpdiscovery
Method: POST
Headers:
- Accept: application/json
- Accept-Encoding: deflate, gzip
- Content-Type: application/x-amz-json-1.1
- X-Amz-Target: AWSMPDiscoveryService.SearchListings
Request Body:
- SearchText: The search query
- MaxResults: 20 (configurable)
- Filters: Configured for DATA_EXCHANGE products
- Sort: By relevance, descending order
Response Format:

interface AWSMarketplaceListingSummary {
    Id: string;
    DisplayAttributes: {
        Title: string;
        LongDescription: string;
    };
    Categories: Array<{
        DisplayName: string;
    }>;
    ProductAttributes: {
        Creator: {
            DisplayName: string;
        };
    };
    OfferSummary: {
        PricingSummary: string;
    };
}

Response Handling:
- Handles zlib-compressed responses
- Decompresses using Web Streams API
- Maps to common ListingResult format
- Includes pricing information
- Preserves category information

Datarade API

The integration with Datarade uses their search interface with the following details:

Base URL: https://datarade.ai/search/products
Query Parameters:
- keywords: The search query
Response Format:

interface DataradeProduct {
    title: string;      // Product title from h3.product-card__title
    subtitle: string;   // Author/provider from span.product-card__subtitle__author
    link: string;       // Full URL to product page
    body: string;       // Detailed description from div.product-card__body
}

Agrimetrics API

The integration with Agrimetrics uses their catalog API with the following parameters:

Base URL: https://app.agrimetrics.co.uk/backend/catalog/api/catalog/data-sets
Query Parameters:
- exchange=agrimetrics: Specifies the data exchange
- tagRelationship=narrower: Defines tag relationship type
- extendedText={search_term}: The search query
- onlyFeatured=false: Include all results
- onlyOwned=false: Include all public datasets
- showHidden=false: Exclude hidden datasets
- showEditable=false: Exclude editable datasets
- identities=PUBLIC: Show only public datasets
- offset=0: Starting point for results
- limit=13: Maximum number of results
- sort=relevance: Sort by relevance

data.gov.uk API

The integration with data.gov.uk uses their CKAN API with the following details:

Base URL: https://ckan.publishing.service.gov.uk/api/action/package_search
Method: GET
Query Parameters:
- q: The search query
Response Format:

interface DataGovUKDataset {
    id: string;
    title: string;
    notes: string;
    metadata_modified: string;
    organization: {
        title: string;
        description: string;
    };
    license_title: string;
    resources: Array<{
        url: string;
        format: string;
    }>;
}

Response Handling:
- Maps to common ListingResult format
- Extracts organization information
- Preserves license information
- Includes resource URLs
- Maintains update timestamps

Usage

Search Examples

Basic Search:

https://proto-data-market-aggregator.chrisns.workers.dev/?search=soil

Multi-word Search:

https://proto-data-market-aggregator.chrisns.workers.dev/?search=soil%20quality

Response Format

The response is an HTML page containing interleaved results from all sources. Each result includes:

Title
Description
Source information
Provider details
Last updated date (formatted consistently as DD/MM/YYYY HH:mm)
Direct link to the data

Development

Prerequisites

Node.js and npm
Wrangler CLI for Cloudflare Workers
Understanding of Cloudflare's caching mechanisms
TypeScript knowledge

Setup

Clone the repository
Install dependencies:
```
npm install
```

Testing

Run the test suite:

npm test

The test suite includes:

Unit tests for each data source integration
Integration tests for the combined search functionality
Error handling tests
URL construction tests
Response formatting tests
Cache configuration tests

Deployment

Deploy to Cloudflare Workers:

npm run deploy

Performance Considerations

When developing new features or modifying existing ones:

Maintain the parallel request architecture
Utilize appropriate cache settings for new API integrations
Consider cache implications when modifying API calls
Test performance with and without cached responses
Be mindful of aggressive caching when implementing new endpoints
Consider cache bypass strategies for time-sensitive data
Use appropriate cache keys for new data sources
Ensure consistent date formatting across all sources
Handle API timeouts gracefully

Error Handling

The service implements graceful degradation:

If any individual data source fails, other sources continue to function
Parallel request architecture ensures a single slow API doesn't block other results
Invalid search parameters are handled gracefully
Network errors are caught and logged
Empty results are handled appropriately
Timeouts and connection issues are handled gracefully
Caching provides resilience against temporary API failures
All errors are logged for monitoring

Contributing

Fork the repository
Create a feature branch
Make your changes
Add tests for new functionality
Submit a pull request

License

This project is licensed under the MIT License - see the LICENSE file for details.

chrisns/proto-data-market-aggregator

Data Marketplace Aggregator

Features

Performance

Caching Strategy

Cache Keys

Supported Data Sources

Snowflake Marketplace

Databricks Marketplace

ONS SRS Metadata Catalogue

Defra Data Services

Agrimetrics Data Marketplace

Datarade Marketplace

AWS Marketplace

data.gov.uk

OpenDataSoft

NHSBSA Open Data Portal

Open Data Northern Ireland

SSEN Distribution

London Datastore

York Open Data

Health Data Gateway

API Integration Details

Response Format

AWS Marketplace API

Datarade API

Agrimetrics API

data.gov.uk API

Usage

Search Examples

Response Format

Development

Prerequisites

Setup

Testing

Deployment

Performance Considerations

Error Handling

Contributing

License