This Python script retrieves data from an Airtable base, identifies relational columns, and builds a knowledge graph using the retrieved data. The knowledge graph is then visualized using the pyvis library and displayed in a web application built with Flask.
It works well with small Airtables but I'm having some issues when the Airtable is too big.
Before running the script, ensure that you have the following:
- Python 3.x installed
- Required Python packages:
requests
,flask
,pyairtable
,pyvis
- Airtable API key (see instructions below)
- Clone the repository:
git clone https://github.com/yoheinakajima/airtable_to_graph.git
- Navigate to the project directory:
cd airtable_to_graph
- Install the required Python packages:
pip install -r requirements.txt
To retrieve data from Airtable, you need to provide your Airtable API key. Here's how you can find your API key:
- Log in to your Airtable account.
- Go to the Developer Hub settings by clicking on your profile picture in the top-right corner and selecting "Developer Hub".
- In the Personal Key page, click on "Create New Token" button on top right.
- Give the token a name, and give it the Scopes permissions of data.records:read and schema.bases:read.
- Add all bases, or select the base you want to connect.
- Click "Create Token" button and copy the generated API key.
- Set the following environment variables with your Airtable API key and base ID:
export AIRTABLE_API_KEY=your_api_key_here
Replace your_api_key_here
with your actual Airtable API key and your_base_id_here
with the ID of the Airtable base you want to retrieve data from.
- Run the script:
python main.py
- Open a web browser and navigate to
http://localhost:5000
to view the knowledge graph.
- The script retrieves all table names from the specified Airtable base using the Airtable API.
- It retrieves data from each table using parallel processing with
ThreadPoolExecutor
. - The script identifies relational columns in the retrieved data by looking for fields that contain a list of strings starting with "rec".
- It builds a knowledge graph using the
pyvis
library, representing each record as a node and creating edges based on the relational columns. - The knowledge graph is saved as an HTML file in the
static
directory. - The script uses Flask to render the knowledge graph in a web application, which can be accessed at
http://localhost:5000
.
- You can customize the appearance of the knowledge graph by modifying the
height
,width
,bgcolor
, andfont_color
parameters in thebuild_knowledge_graph
function. - If you want to change the shape of the nodes, you can modify the
shape
parameter in thenet.add_node
function.
Contributions are welcome! If you find any issues or have suggestions for improvements, please open an issue or submit a pull request on the GitHub repository.
This project is licensed under the MIT License.