PCH API changed - data is not being fetched anymore

[nrodday]

Hi,

Action: Updating Traixroute´s DB

Expected behaviour: New files should be fetched and stored under database/
Actual behaviour: For PCH the html code is printed in all files

Detailed steps:
When I do "traixroute -u -process" I get:

traixroute -u -process
Updating the database...
Started downloading PDB dataset.
Started downloading PCH dataset.
Started downloading RouteViews dataset.
PCH dataset has been updated successfully.
Routeviews has been updated successfully.
PDB dataset has been updated successfully.
Database has been updated successfully.

Looking at db.txt I find:

cat db.txt
Imported 16 Reserved Subnets.
Extracted 0 IXP IPs from additional_info.txt.
Extracted 0 IXP Subnets from additional_info.txt.
Extracted 33654 IXP IPs from PDB.
Extracted 0 IXP IPs from PCH.
Extracted 841 IXP Subnets from PDB.
Extracted 0 IXP Subnets from PCH.
Extracted 33654 (no) dirty IXP IPs after merging PDB, PCH and additional_info.txt.
Detected 0 dirty IXP IPs after merging PDB, PCH and additional_info.txt.
Extracted 841 IXP Subnets after merging PDB, PCH and additional_info.txt.

Which suprised me, as PCH seems to have no content. Further investigating /database/PCH/ I find three files:
ixp_exchange.csv ixp_membership.csv ixp_subnets.csv

All of which contain the extact same HTML code from the PCH website. It is not the data stored in those files, but instead something must go wrong when fetching the data such that the website source code is stored. I believe PCH changed the API after the tool was released.

Fixing the ixp_exchange.csv in the /configuration/config is easy as it is still provided as CSV, just under a different URL:
https://www.pch.net/api/ixp/directory?format=csv

For the other two it is not as straight forward. PCH does not seem to provide an entire list anymore. Instead they provide an API that allows querying for each IXP-member id, according to the IXP directory which was dumped into the ixp_exchange.csv. To recreate the old structure one would need to query for each member id at:
For Subnets: https://www.pch.net/api/ixp/subnets/IXP_ID(?format=csv)
For IPs: https://www.pch.net/api/ixp/subnet_details/IXP_ID

Could you take a look and let me know if you plan to implement the changes?

Thanks!

Cheers,
NIls