Sync fails for very large AD groups
libalis opened this issue · 5 comments
libalis commented
Hello,
I get the following error message.
`/var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/lib/pg_ldap_sync/application.rb:114:in `block in retrieve_array_attribute': undefined method `first' for nil:NilClass (NoMethodError)
from /var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/lib/pg_ldap_sync/application.rb:102:in `loop'
from /var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/lib/pg_ldap_sync/application.rb:102:in `retrieve_array_attribute'
from /var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/lib/pg_ldap_sync/application.rb:154:in `block (2 levels) in search_ldap_groups'
from /var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/lib/pg_ldap_sync/application.rb:153:in `each'
from /var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/lib/pg_ldap_sync/application.rb:153:in `block in search_ldap_groups'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap.rb:786:in `block (3 levels) in search'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap/connection.rb:448:in `block (2 levels) in search'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap/connection.rb:402:in `loop'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap/connection.rb:402:in `block in search'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap/instrumentation.rb:19:in `instrument'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap/connection.rb:391:in `search'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap.rb:784:in `block (2 levels) in search'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap.rb:1311:in `use_connection'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap.rb:783:in `block in search'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap/instrumentation.rb:19:in `instrument'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap.rb:782:in `search'
from /var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/lib/pg_ldap_sync/application.rb:131:in `search_ldap_groups'
from /var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/lib/pg_ldap_sync/application.rb:367:in `start!'
from /var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/lib/pg_ldap_sync/application.rb:416:in `run'
from /var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/exe/pg_ldap_sync:6:in `<top (required)>'
from /usr/local/bin/pg_ldap_sync:25:in `load'
from /usr/local/bin/pg_ldap_sync:25:in `<main>'`
I am trying to synchronize an AD group with about 1600 members. When I use ldapsearch, I see that the members are accessed via range attribute, and our AD can export a maximum of 1500 entries at once.
member;range=0-1499
Could you please fix your program in general to handle the usecases with large AD groups iteratively with editing 1500 entries per step without errors? Or maybe there is a parameter and I can limit the range for the LDAP query myself.
Thank you in advance.
larskanis commented
pg_ldap_sync-0.4.0 added support for range retrieval and groups of more than 1500 members.
Could you please print the output of pg_ldap_sync with option -vvv, or at least the relevant part of the attribute retrieval, which is retrieved attribute range etc.
libalis commented
It starts to list the found users like this....
(I anonymize the names)
`xxxx@xxxx:~$ pg_ldap_sync -c ldap_sync_xx_xxx.yaml -vvv -t
I, [2023-08-24T11:08:49.484135 #14794] INFO -- : found user-dn: CN=xxxxxxx.xxxxxxxxx xxxxxxxx,OU=xxx-xxxxx,DC=xxxxx,DC=xxx,DC=xx
D, [2023-08-24T11:08:49.484826 #14794] DEBUG -- : dn:
D, [2023-08-24T11:08:49.485083 #14794] DEBUG -- : --->"CN=xxxxxxx.xxxxxxxxx xxxxxxxx,OU=xxx-xxxxx,DC=xxxxx,DC=xxx,DC=xx"
D, [2023-08-24T11:08:49.485352 #14794] DEBUG -- : samaccountname:
D, [2023-08-24T11:08:49.485599 #14794] DEBUG -- : --->"xxxxxxx.xxxxxxxxx"
..........`
And here the program breaks off.
`I, [2023-08-24T11:08:52.214508 #14794] INFO -- : found user-dn: CN=xxxxx.xxxxx xxxxxxxx,OU=xxx-xxxxx,DC=xxxxx,DC=xxx,DC=xx
D, [2023-08-24T11:08:52.214673 #14794] DEBUG -- : dn:
D, [2023-08-24T11:08:52.214777 #14794] DEBUG -- : --->"CN=xxxxx.xxxxx xxxxxxxx,OU=xxx-xxxxx,DC=xxxxx,DC=xxx,DC=xx"
D, [2023-08-24T11:08:52.214884 #14794] DEBUG -- : samaccountname:
D, [2023-08-24T11:08:52.214979 #14794] DEBUG -- : --->"xxxxx.xxxxx"
I, [2023-08-24T11:08:52.490383 #14794] INFO -- : found group-dn: CN=xx_xxx,OU=xxxxxxxx_xxxxxxxxxxxx_xxxx,OU=xxx-xxxxx,DC=xxxxx,DC=xxx,DC=xx
D, [2023-08-24T11:08:52.491047 #14794] DEBUG -- : retrieved attribute range :"member;range=0-1499" of dn CN=xx_xxx,OU=xxxxxxxx_xxxxxxxxxxxx_xxxx,OU=xxx-xxxxx,DC=xxxxx,DC=xxx,DC=xx
D, [2023-08-24T11:08:52.638013 #14794] DEBUG -- : retrieved attribute range :"member;range=1500-*" of dn CN=xx_xxx,OU=xxxxxxxx_xxxxxxxxxxxx_xxxx,OU=xxx-xxxxx,DC=xxxxx,DC=xxx,DC=xx
/var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/lib/pg_ldap_sync/application.rb:114:in `block in retrieve_array_attribute': undefined method `first' for nil:NilClass (NoMethodError)
from /var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/lib/pg_ldap_sync/application.rb:102:in `loop'
from /var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/lib/pg_ldap_sync/application.rb:102:in `retrieve_array_attribute'
from /var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/lib/pg_ldap_sync/application.rb:154:in `block (2 levels) in search_ldap_groups'
from /var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/lib/pg_ldap_sync/application.rb:153:in `each'
from /var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/lib/pg_ldap_sync/application.rb:153:in `block in search_ldap_groups'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap.rb:786:in `block (3 levels) in search'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap/connection.rb:448:in `block (2 levels) in search'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap/connection.rb:402:in `loop'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap/connection.rb:402:in `block in search'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap/instrumentation.rb:19:in `instrument'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap/connection.rb:391:in `search'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap.rb:784:in `block (2 levels) in search'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap.rb:1311:in `use_connection'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap.rb:783:in `block in search'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap/instrumentation.rb:19:in `instrument'
from /var/lib/gems/3.0.0/gems/net-ldap-0.18.0/lib/net/ldap.rb:782:in `search'
from /var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/lib/pg_ldap_sync/application.rb:131:in `search_ldap_groups'
from /var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/lib/pg_ldap_sync/application.rb:367:in `start!'
from /var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/lib/pg_ldap_sync/application.rb:416:in `run'
from /var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/exe/pg_ldap_sync:6:in `<top (required)>'
from /usr/local/bin/pg_ldap_sync:25:in `load'
from /usr/local/bin/pg_ldap_sync:25:in `<main>'`
As written my LDAP filter includes 1 AD group including just under 1600 users. Listing of single user accounts seems to work completely. And with the AD group the program breaks at the step "member;range=1500-*".
The program itself runs on Ubuntu 22.04 LTS. Our LDAP server is the Microsoft Active Directory (Windows Server 2019).
larskanis commented
Damn, there is a typo in the code! Could you please change the file /var/lib/gems/3.0.0/gems/pg-ldap-sync-0.4.0/lib/pg_ldap_sync/application.rb in line 106 and insert a "+" like so:
diff --git a/lib/pg_ldap_sync/application.rb b/lib/pg_ldap_sync/application.rb
index 2c6d9ae..747761b 100644
--- a/lib/pg_ldap_sync/application.rb
+++ b/lib/pg_ldap_sync/application.rb
@@ -103,7 +103,7 @@ class Application
array += entry[ranged_attr]
log.debug "retrieved attribute range #{ranged_attr.inspect} of dn #{entry_dn}"
- if ranged_attr =~ /;range=\d\-\*\z/
+ if ranged_attr =~ /;range=\d+\-\*\z/
break
end
That should fix the issue.
libalis commented
Yes, this works perfectly.
You can commit this change.
This closes the issue.
Thanks for quick solution.
larskanis commented
pg-ldap-sync-0.5.0 is released now and it fixes this issue.