Mirroring with IRRD: manually load and update source data doesn't work as expected
Closed this issue · 3 comments
Describe the bug
Hi, i am trying to use IRRD as our NRTM client and server. I load a db file for a source with an initial serial number, then update the db with new db file, but the serial number is not increased from the initial number, from 1 instead. Also, the NRTM query response doesn't work as expected as well. I am not sure whether i did something wrong. Could you have a look? Thanks
To Reproduce
Configuration file
irrd:
database_url: 'postgresql://irrd:irrd@localhost:5432/irrd'
redis_url: 'redis://localhost:6379'
piddir: /irrd/irrd-venv/var/run
user: justin.yang
group: admin
access_lists:
generic_nrtm_access:
- '0.0.0.0/0'
server:
http:
interface: 127.0.0.1
port: 8080
whois:
interface: 127.0.0.1
port: 43
email:
footer: ''
notification_header: |
This is to notify you of changes in the {sources_str} database
or object authorisation failures.
You may receive this message because you are listed in
the notify attribute on the changed object(s), because
you are listed in the mnt-nfy or upd-to attribute on a maintainer
of the object(s), or the upd-to attribute on the maintainer of a
parent of newly created object(s).
from: example@example.com
smtp: localhost
auth:
gnupg_keyring: /irrd/irrd-venv/gnupg-keyring
log:
logfile_path: /irrd/irrd-venv/var/log/irrd.log
level: DEBUG
rpki:
roa_source: null
sources:
JPNIC:
authoritative: true
keep_journal: true
nrtm_access_list: generic_nrtm_access`
Load initial data
% cat jpnic-test.db
inetnum: 27.96.54.48 - 27.96.54.55
netname: Y2JI-NET
descr: y2ji, Inc.
country: JP
admin-c: TT53002JP
tech-c: TT53002JP
remarks: This information has been partially mirrored by APNIC from
remarks: JPNIC. To obtain more specific information, please use the
remarks: JPNIC WHOIS Gateway at
remarks: http://www.nic.ad.jp/en/db/whois/en-gateway.html or
remarks: whois.nic.ad.jp for WHOIS client. (The WHOIS client
remarks: defaults to Japanese output, use the /e switch for English
remarks: output)
last-modified: 2023-07-01T19:50:03Z
source: JPNIC
./bin/irrd_load_database --source JPNIC ~/Downloads/jpnic-test.db --serial 10
Update data
% cat jpnic-test-update.db
inetnum: 27.96.54.48 - 27.96.54.55
netname: Y2JI-NET
descr: y2ji, Inc.
country: JP
admin-c: TT53002JP
tech-c: TT53002JP
remarks: This information has been partially mirrored by APNIC from
remarks: JPNIC. To obtain more specific information, please use the
remarks: JPNIC WHOIS Gateway at
remarks: http://www.nic.ad.jp/en/db/whois/en-gateway.html or
remarks: whois.nic.ad.jp for WHOIS client. (The WHOIS client
remarks: defaults to Japanese output, use the /e switch for English
remarks: output)
last-modified: 2023-07-01T19:50:03Z
source: JPNIC
aut-num: AS63795
as-name: ROUTEL
descr: RoutelNet
import: from AS59103 100 accept ANY
import: from AS59105 100 accept ANY
export: to AS59103 announce AS63795
export: to AS59105 announce AS63795
country: JP
admin-c: JP00227706
tech-c: JP00227706
last-modified: 2022-05-27T19:17:08Z
source: JPNIC
./bin/irrd_update_database --source JPNIC ~/Downloads/jpnic-test-update.db
NRTM query response
% telnet localhost 43
Trying ::1...
Connection failed: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
-g JPNIC:3:1-LAST
%START Version: 3 JPNIC 1-3
ADD 1
aut-num: AS63795
as-name: ROUTEL
descr: RoutelNet
import: from AS59103 100 accept ANY
import: from AS59105 100 accept ANY
export: to AS59103 announce AS63795
export: to AS59105 announce AS63795
country: JP
admin-c: JP00227706
tech-c: JP00227706
source: JPNIC
last-modified: 2023-07-03T23:01:58Z
ADD 2
aut-num: AS63795
as-name: ROUTEL
descr: RoutelNet
import: from AS59103 100 accept ANY
import: from AS59105 100 accept ANY
export: to AS59103 announce AS63795
export: to AS59105 announce AS63795
country: JP
admin-c: JP00227706
tech-c: JP00227706
source: JPNIC
last-modified: 2023-07-03T23:01:58Z
ADD 3
inetnum: 27.96.54.48 - 27.96.54.55
netname: Y2JI-NET
descr: y2ji, Inc.
country: JP
admin-c: TT53002JP
tech-c: TT53002JP
remarks: This information has been partially mirrored by APNIC from
remarks: JPNIC. To obtain more specific information, please use the
remarks: JPNIC WHOIS Gateway at
remarks: http://www.nic.ad.jp/en/db/whois/en-gateway.html or
remarks: whois.nic.ad.jp for WHOIS client. (The WHOIS client
remarks: defaults to Japanese output, use the /e switch for English
remarks: output)
source: JPNIC
last-modified: 2023-07-03T23:01:58Z
%END JPNIC
Connection closed by foreign host.
Expected behaviour
The NTM query response should return just one ADD for aut-num: AS63795 with serial 11
IRRd version you are running
E.g. 4.3.0-POST1
This behaviour is consistent with the documentation but I can see how this is confusing. Your setup does not have synchronised serials as nrtm_host is not set, therefore your NRTM stream generates its own independent serial set.
The serial parameter to irrd_load_database is confusing in this combination though, it sets the "seen" serial which is a rather abstract concept. I am not sure yet whether we should change this behaviour and to what.
Thanks for your reply. If a serial parameter passed to irrd_load_database, could we increase the serial number from that parameter not from 0 instead? For example, in the case mentioned earlier, if I pass 10 as the initial serial number, any object update would increment it to 11. Otherwise, when we migrate the source data to IRRD, we need to fill in gap between 0 and initial serial number with dummy object operations so that the local serial can start from the desired serial number, but this approach is not ideal.
@mxsasha Happy new year!
I am looking more deeply into this issue again.
I load an initial dump file for KRNIC with serial number 100 like irrd_load_database --source KRNIC --serial 100 {file}. I can see the database_status table having the correct record which is done by this code.
irrd=# select * from database_status where source='KRNIC';
-[ RECORD 1 ]---------+-------------------------------------
pk | d87f6d39-09b9-4069-bc70-31bd19558feb
source | KRNIC
serial_oldest_seen | 100
serial_newest_seen | 100
serial_oldest_journal |
serial_newest_journal |
Then, I added a new object via the http, but the status record has been updated as below and the journal table has one new record with serial_nrtm=1.
irrd=# select * from database_status where source='KRNIC';
-[ RECORD 1 ]---------+-------------------------------------
pk | d87f6d39-09b9-4069-bc70-31bd19558feb
source | KRNIC
serial_oldest_seen | 1
serial_newest_seen | 100
serial_oldest_journal | 1
serial_newest_journal | 1
The expected result would be the journal table will have one record with serial_nrtm=101 and the database_status table is like
pk | d87f6d39-09b9-4069-bc70-31bd19558feb
source | KRNIC
serial_oldest_seen | 100
serial_newest_seen | 101
serial_oldest_journal | 101
serial_newest_journal | 101
By looking at the source codes, I find that when running dh.commit() everytime like this in the irrd_load_database command, it will call the reset() function and this function will clean the _new_serials_per_source dict which has the source KRNIC with initial serial number 100. See https://github.com/irrdnet/irrd/blob/main/irrd/storage/database_handler.py#L182 and https://github.com/irrdnet/irrd/blob/main/irrd/storage/database_handler.py#L1190
Because the _new_serials_per_source dict is empty, IRRD will use the COALESCE(MAX(serial_nrtm), 0) + 1 when generating the serial_nrtm in the journal table per codes for the subsequent objects update operations. Since all the records in journal table have been cleaned by the irrd_load_database command, the serail_nrtm would be 1 for this case and the record in database_status table will be updated accordingly as above.
Not sure how others use the initial serial in the irrd_load_database command, but I think increasing the nrtm_serial number from that initial serial not from 0 makes more sense. Maybe create another dict for storing the initial serial per sources and not clean it? How do you think?