[FEAT]: Records API, filterable 'fqdn' field
Closed this issue · 4 comments
Is your feature request related to a problem? Please describe.
The separation between name and zone requires multiple queries if name contains periods when trying to find a record.
Describe the solution you'd like
I would like to be able to query the API like:
GET /api/plugins/netbox-dns/records/?fqdn=some.rec.example.com
Describe alternatives you've considered
Currently, I have to walk the request and do multiple queries:
GET /api/plugins/netbox-dns/records/?zone=rec.example.com&name=some
GET /api/plugins/netbox-dns/records/?zone=example.com&name=some.rec
Additional context
I'm working on a CoreDNS plugin that uses netbox-plugin-dns as a backend. Having to walk zones is leading to some ridiculous design complexities and having a queryable fqdn field would simplify this quite a lot.
An FQDN is already compiled for the display field (ex. some.rec.example.com [A]).
Hi @wranders, thanks for that feature request.
There are several ways in which this can be implemented, and all of them have one thing in common: They are more or less bad.
Using a filter() method on the DB level
This doesn't work at all for the simple reason that the fqdn() method isn't exactly straightforward. While the naive approach suggests to simply concatenate the name of the record and the name of the zone and be don, it quickly shows that this doesn't work at all - just think of @ and absolute record names for a start. Even though it can be implemented on DB level with some effort, that would still leave us with two different methods (home-grown DB function and dnspython) of deriving an FQDN from a record object, which is generally a very bad idea.
Using a custom filter() method on the ORM level
Slightly better. The general mechanism works by taking a queryset, performing the selection of the desired result by executing a custom function on the elements of the queryset, and then returning a filtered queryset. The problem here: The original queryset generally consists of all records in the database, which means all DNS records that are known to NetBox DNS. I have at least one user who has millions of records in the DB - well, no. Doesn't work either.
Storing the FQDN in a separate model field for Record
The logic is simple: On save(), get the FQDN of the record and store it in a model field, which can then be used for filtering easily. The culprit here is that it denormalises the database model (the same information is present in two places, the name of the record and zone and the FQDN of the record), and leads to a performance penalty on every save() operation for Record objects. Not only that: Every time a zone name is changed, a save() operations for all records in the zone must be performed so the fqdn field gets updated as well. So it has a performance impact for every record and zone saved, while filtering is almost instantaneous. You have to keep some things in mind, though, such as that it doesn't work for bulk_create(), which affects some use cases such as mass importing data using custom scripts.
Using a custom object manager for Record
The problem is roughly the same as for the ORM level filter, but more complex and with worse and less obvious implications on readability and in some cases usability. See this StackOverflow discussion for more detail. Not an idea I would care to explore further.
Bottom line: The least problematic approach is to implement an fqdn field on the Record model and use that for filtering. So far I did not feel the urgency to go down that road enough to accept the drawbacks, but your FR is the second time in a short while that I thought it might be a good idea after all ...
As a side note:
An FQDN is already compiled for the
displayfield (ex.some.rec.example.com [A]).
Correct. The problem is: That information is created on the fly from record and zone information, which is easy. The other way around it is not, and requires storing and updating an auxiliary field to be acceptable performance-wise. So while it's nice to have the FQDN in the display field (and some other places as well), it doesn't really help with filtering.
Awesome! Thanks for the quick turnaround despite the reservations.
Been experimenting with this for a few hours and it seems to solve my problems.
We'll see how bad the performance impact on renaming zones is. I see some potential for minor optimisations, but first I wanted to get that feature working. It's better in the new NetBox 4 compatibility branch anyway because there are some under-the-hood improvements in the plugin.
Performance impact on saving records is not that bad as the FQDN is generated anyway during validation (we need to check for length restrictions, IDN correctness etc.), so saving it is not a massive overhead and we get some of it back in other places where the FQDN is used.