Huge sudden drop in number of visitors (sessions) after upgrading to v2.9.0

Describe the Bug

We upgraded from 2.7.0 to v2.9.0 on December 19th. This caused a huge drop in the visitors (sessions) count for all all websites as can be seen below:

This continues into January.

Since the views (pageview events) has not changed much, this can only mean views are now more concentrated across less visitors (sessions). We went from ~2 views/visitor/day on average, to ~10 views/visitor/day on average.

Running this query:

SELECT s.session_id, top_sessions.count AS pageviews, s.hostname, s.browser, s.os, s.device, s.screen FROM session s JOIN (SELECT e.session_id, count(e.session_id) FROM website_event e WHERE e.event_type = 1 AND e.website_id = 'mywebsiteid' AND e.created_at BETWEEN '2023-12-07' AND '2023-12-18' GROUP BY e.session_id ORDER BY count(e.session_id) DESC LIMIT 10) AS top_sessions ON s.session_id = top_sessions.session_id;

We can get the get the Top 10 session_ids by pageview count in the 12 days from December 7th to December 18th, before the upgrade:

              session_id              | pageviews |    hostname     | browser |     os     | device  |  screen
--------------------------------------+-----------+-----------------+---------+------------+---------+-----------
 e85067d6-6569-5afb-9d38-84c8c96467cb |        41 | www.website.com | ios     | iOS        | mobile  | 393x852
 e24894c3-76d1-582a-a1ff-7becec3c8c3d |        31 | www.website.com | chrome  | Windows 10 | desktop | 1920x1080
 5509fd77-5a93-5372-b0ec-223d33ccbf76 |        31 | www.website.com | safari  | Mac OS     | desktop | 2048x1152
 e3c476f7-221f-5e24-9f18-f09e269d0ca8 |        27 | www.website.com | ios     | iOS        | mobile  | 393x852
 c90eab1e-ef35-5f90-9c00-3e019a5605c1 |        21 | www.website.com | firefox | Linux      | laptop  | 1366x768
 459a7bb0-e9ed-5cc2-a285-4d1eca9a7fb9 |        20 | www.website.com | ios     | iOS        | mobile  | 393x852
 9d8cb764-4f6c-50cc-9a0e-96face7eee83 |        18 | www.website.com | ios     | iOS        | mobile  | 414x896
 33e410ad-8f53-53df-a743-4799785d44f3 |        18 | www.website.com | ios     | iOS        | mobile  | 390x844
 bc070bf5-d5da-5147-8222-1e520817f284 |        17 | www.website.com | chrome  | Windows 10 | desktop | 1920x1080
 ffd00f48-e680-5b56-84eb-f3f3eafd4979 |        16 | www.website.com | chrome  | Windows 10 | desktop | 1920x1080

And from December 20th and December 31st, after the upgrade:

              session_id              | pageviews |    hostname     |    browser    |     os     | device  |  screen
--------------------------------------+-----------+-----------------+---------------+------------+---------+-----------
 7d66612e-3661-5fc7-aad4-f2b7e8e28bd5 |      2702 | www.website.com | ios           | iOS        | mobile  | 430x932
 9a3ae813-616d-5ea0-a782-35d76892bf8d |      1112 | www.website.com | chrome        | Android OS | mobile  | 412x1006
 300147ec-b51e-5934-9a97-5da8c054edc6 |       615 | www.website.com | chrome        | Windows 10 | laptop  | 1280x720
 30a62591-9d5d-514b-92eb-d2aee1b057d9 |       427 | www.website.com | ios           | iOS        | mobile  | 430x932
 78748dc1-88d8-59d2-8262-329895cc000c |       404 | www.website.com | ios           | iOS        | mobile  | 375x812
 4b5980de-84da-59ce-ae5a-2a561b8a9f0a |       334 | www.website.com | crios         | iOS        | mobile  | 375x812
 0262e952-38e5-574a-bf6c-1f16f6eb5786 |       252 | www.website.com | safari        | Mac OS     | laptop  | 834x1194
 199020fc-5857-5d15-afa2-984d3e4c5148 |       208 | www.website.com | ios-webview   | iOS        | mobile  | 393x852
 1cd4f507-d10f-5449-be64-8c324d5a87a1 |       205 | www.website.com | edge-chromium | Windows 10 | desktop | 1920x1080
 4b980cf4-fdeb-56ab-bba1-221f4c703cd3 |       198 | website.com     | ios           | iOS        | mobile  | 390x844

It's clear that views are accumulating across less sessions.

This means less distinct sessions are being created, which means something is making session ids be the same across different arguments.

The arguments used to generate the session UUID are websiteId, hostname, ip, userAgent and haven't changed:

umami/src/lib/session.ts

Lines 66 to 69 in 784237b

    
           const { userAgent, browser, os, ip, country, subdivision1, subdivision2, city, device } = 
        
             await getClientInfo(req, payload); 
        
           const sessionId = uuid(websiteId, hostname, ip, userAgent);

I checked how they're collected and it also hasn't changed (location data has changed, but they're not used in the UUID):

umami/src/lib/detect.ts

Lines 124 to 137 in 784237b

    
           export async function getClientInfo(req: NextApiRequestCollect, { screen }) { 
        
             const userAgent = req.headers['user-agent']; 
        
             const ip = getIpAddress(req); 
        
             const location = await getLocation(ip, req); 
        
             const country = location?.country; 
        
             const subdivision1 = location?.subdivision1; 
        
             const subdivision2 = location?.subdivision2; 
        
             const city = location?.city; 
        
             const browser = browserName(userAgent); 
        
             const os = detectOS(userAgent); 
        
             const device = getDevice(screen, os); 
        
             return { userAgent, browser, os, ip, country, subdivision1, subdivision2, city, device }; 
        
           }

Edit: the below is irrelevant, please read the next comment for the real issue

The only thing that changed related to this that I found was that a monthly salt() was added as an argument to the hash that gets encoded into the UUID:

umami/src/lib/crypto.ts

Lines 9 to 19 in 784237b

    
           export function salt() { 
        
             const ROTATING_SALT = hash(startOfMonth(new Date()).toUTCString()); 
        
             return hash(secret(), ROTATING_SALT); 
        
           } 
        
           export function uuid(...args: any) { 
        
             if (!args.length) return v4(); 
        
             return v5(hash(...args, salt()), v5.DNS); 
        
           }

This happened in this commit trying to fix this issue and was released with v2.9.0.

I don't know how this could cause the problem, but it's the only related change that I found. Any other ideas?

Database

PostgreSQL

I found the issue, it's not Umami.

My service is behind Nginx and my request headers look like:

'x-real-ip': '100.28.50.121',
'user-agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 17_2_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Mobile/15E148 Safari/604.1',
'x-forwarded-for': '::ffff:127.0.0.1'

So the real IP is in x-real-ip and x-forwarded-for always stays the same, but the request-ip library gives x-forwarded-for priority over x-real-ip:

https://github.com/pbojinov/request-ip/blob/e1d0f4b89edf26c77cf62b5ef662ba1a0bd1c9fd/lib/index.js#L45-L65

So every session gets the ::ffff:127.0.0.1 IP. This makes visitors using the same userAgent get the same session id.

request-ip hasn't changed in 2 years, so it was probably myself that changed some Nginx config or something like that at the same time I upgraded Umami.

I fixed the problem by adding proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; to my Nginx configuration.

Luckily I have the real ips, times and user agents in the nginx logs, so I can just parse it and force the correct data into the umami DB.

Glad you were able to solve it! FYI, there's also an environment variable to tell Umami which header to check, CLIENT_IP_HEADER. See https://umami.is/docs/environment-variables

Thanks @mikecao!

To fix the incorrect data I created a tool to parse the NGINX logs and force the correct session data in the DB. It worked for my case and can be found here if anyone has the same problem.

	const { userAgent, browser, os, ip, country, subdivision1, subdivision2, city, device } =
	await getClientInfo(req, payload);

	const sessionId = uuid(websiteId, hostname, ip, userAgent);

	export async function getClientInfo(req: NextApiRequestCollect, { screen }) {
	const userAgent = req.headers['user-agent'];
	const ip = getIpAddress(req);
	const location = await getLocation(ip, req);
	const country = location?.country;
	const subdivision1 = location?.subdivision1;
	const subdivision2 = location?.subdivision2;
	const city = location?.city;
	const browser = browserName(userAgent);
	const os = detectOS(userAgent);
	const device = getDevice(screen, os);

	return { userAgent, browser, os, ip, country, subdivision1, subdivision2, city, device };
	}

	export function salt() {
	const ROTATING_SALT = hash(startOfMonth(new Date()).toUTCString());

	return hash(secret(), ROTATING_SALT);
	}

	export function uuid(...args: any) {
	if (!args.length) return v4();

	return v5(hash(...args, salt()), v5.DNS);
	}