Forbidden 403
hildebrau opened this issue · 222 comments
Version
develop tag Auto-Southwest Check-In v7.2
Browser Version
Using browser version: 121.0.6167.184
Description
I'm getting consistent 403 errors when Retrieving reservation information. I purposely limited my config down to a single record locator.. removing the account login stuff and setting check_fares to false. I updated to the latest develop tag just now as well. I ran it 4-5 times in a row and got the same result each time.
To Reproduce
Running it via docker-compose w/ config.json w/ a single record locator.
Expected Behavior
I'd expect it to schedule a checkin at minimum, and ideally successfully check in at that scheduled time.
Relevant logs and program output
my config.json
{
"check_fares": false,
"notification_urls": "gchat://XXXXXXX",
"notification_level": 1,
"retrieval_interval": 6,
"reservations": [
{"confirmationNumber": "XXXXXX", "firstName": "XXXXXX", "lastName": "XXXXXXXXXX"}
]
}
docker logs output:
2024-02-26 15:23:21 DEBUG MainProcess[log:23]: Initialized the application
2024-02-26 15:23:21 DEBUG MainProcess[main:112]: Auto-Southwest Check-In v7.2
2024-02-26 15:23:21 DEBUG MainProcess[main:113]: Called with 0 arguments
2024-02-26 15:23:21 DEBUG MainProcess[config:120]: Initializing configuration file
2024-02-26 15:23:21 DEBUG MainProcess[config:149]: Reading the configuration file
2024-02-26 15:23:21 DEBUG MainProcess[config:163]: Reading configuration from environment variables
2024-02-26 15:23:21 DEBUG MainProcess[config:58]: Setting check fares to False
2024-02-26 15:23:21 DEBUG MainProcess[config:80]: Setting notification level to <NotificationLevel.INFO: 1>
2024-02-26 15:23:21 DEBUG MainProcess[config:93]: Using 1 notification services
2024-02-26 15:23:21 DEBUG MainProcess[config:97]: Setting retrieval interval to 6 hours
2024-02-26 15:23:21 DEBUG MainProcess[config:139]: Creating configurations for 1 reservations
2024-02-26 15:23:21 DEBUG MainProcess[main:142]: Monitoring 0 accounts and 1 reservations
2024-02-26 15:23:21 DEBUG Process-1[reservation_monitor:60]: Acquiring lock...
2024-02-26 15:23:21 DEBUG Process-1[reservation_monitor:62]: Lock acquired
2024-02-26 15:23:21 DEBUG Process-1[checkin_scheduler:50]: Refreshing headers for current session
2024-02-26 15:23:21 DEBUG Process-1[webdriver:106]: Starting webdriver for current session
2024-02-26 15:23:23 DEBUG Process-1[webdriver:122]: Using browser version: 121.0.6167.184
2024-02-26 15:23:23 DEBUG Process-1[webdriver:126]: Loading Southwest Check-In page
2024-02-26 15:23:27 DEBUG Process-1[webdriver:64]: Waiting for valid headers
2024-02-26 15:23:27 DEBUG Process-1[webdriver:155]: Waiting for headers_set to be set
2024-02-26 15:23:27 DEBUG Process-1[webdriver:159]: headers_set set successfully
2024-02-26 15:23:27 DEBUG Process-1[reservation_monitor:84]: Scheduling flight check-ins for 1 reservations
2024-02-26 15:23:27 DEBUG Process-1[checkin_scheduler:76]: Retrieving reservation information
2024-02-26 15:23:41 DEBUG Process-1[utils:39]: Failed to make request after 20 attempts: Forbidden 403
2024-02-26 15:23:41 DEBUG Process-1[utils:42]: Response body: {
"code": 403050700
}
2024-02-26 15:23:41 DEBUG Process-1[checkin_scheduler:82]: Failed to retrieve reservation info. Error: Forbidden 403. Exiting
2024-02-26 15:23:41 DEBUG Process-1[notification_handler:76]: Sending failed reservation retrieval notification...
Failed to retrieve reservation for XXXXXXX XXXXXXXXXXX with confirmation number XXXXXX. Reason: Forbidden 403.
Make sure the reservation information is correct and try again.
2024-02-26 15:23:41 DEBUG Process-1[checkin_scheduler:57]: 0 flights found under current reservation
2024-02-26 15:23:41 DEBUG Process-1[checkin_scheduler:43]: 0 total flights were found
2024-02-26 15:23:41 DEBUG Process-1[checkin_scheduler:100]: 0 new flights found
2024-02-26 15:23:41 DEBUG Process-1[checkin_scheduler:104]: Scheduling 0 flights for check-in
2024-02-26 15:23:41 DEBUG Process-1[checkin_scheduler:116]: 0 flights are currently scheduled. Removing old flights
2024-02-26 15:23:41 DEBUG Process-1[checkin_scheduler:132]: Successfully removed old flights. 0 flights are now scheduled
2024-02-26 15:23:41 DEBUG Process-1[reservation_monitor:71]: No more flights are scheduled for check-in. Exiting...Additional context
Synology DSM 7.x
Docker version 20.10.3, build 55f0773
Btw, from the same household and thus the same source IP, I can hit southwest.com and do a test "CHANGE" flight w/ the flight details and it shows the flight itinerary. I am not sure which method your code is using to retrieve the flight details, though.
Oh, and the record locator in question is currently within the 24 hour checkin period for the departure flight.. and was checked in manually yesterday afternoon. The return flight is on this Wednesday afternoon on the same record locator.
This is the new issue that has been come across in #226 and #201. I will consolidate those issues into this one.
I can also reproduce this issue in Docker too (both using Chromium and Chrome). I believe this is just a change from #201 that Southwest did that now returns the forbidden error instead of a 429 (which makes more sense).
You probably have already seen this; but in case not, see the Outro section on this post regarding methods of bypassing reCaptcha. The reason I bring this up is we may be dealing with an invisible reCaptcha style algorithm.
@hildebrau Hey. If you dont mind sharing, how did you get the webdriver to work in a Synology NAS? I am getting the following error
2024/02/27 00:03:33 | stdout | 2024-02-27 08:01:16 DEBUG Process-1[webdriver:103]: Starting webdriver for current session
[...]
2024/02/27 00:03:33 | stdout | File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 348, in execute
2024/02/27 00:03:33 | stdout | self.error_handler.check_response(response)
2024/02/27 00:03:33 | stdout | File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
2024/02/27 00:03:33 | stdout | selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:9222
2024/02/27 00:03:33 | stdout | raise exception_class(message, screen, stacktrace)
2024/02/27 00:03:33 | stdout | from chrome not reachable
I am guessing it can't find a web browser anywhere on the NAS. How did you get around that? Thanks
how did you get the webdriver to work in a Synology NAS?
@Agiang42, I never saw such an issue. Are you running this code as a docker container? If not, that's probably the issue. I have not tried running the python directly on the Synology because I don't believe all the prerequisites could be installed.
I'm seeing this same issue outside of docker, as well.
Successfully logged in to XXXX XXXX's account
Failed to retrieve reservation for XXXX XXXX with confirmation number XXXXXX.
Reason: Forbidden 403
Failed to retrieve reservation for XXXX XXXX with confirmation number XXXXXX.
Reason: Forbidden 403
Encountered a Too Many Requests error while logging in. Skipping reservation retrieval
I have two flights booked for the end of March, so if there are any logs that would be helpful or anything I can try with my account between now and then, I'll be happy to do so.
I am also having the same issue running outside docker with develop branch. Ubuntu 22.04 python 3.10.12 Chrome 122.0.6261.69
Account type config fails and so does the Reservation type.
Same issue for me as well. Any ideas on a fix? Is there any sense on the logic behind how southwest is determining when to issue the 403?
Any ideas on a fix? Is there any sense on the logic behind how southwest is determining when to issue the 403?
I believe this is the same error as #201, just returning a different status code due to Southwest’s improvements they did. That issue’s thread has many details on what I and others have tried, but there is no known fix to this as of yet. Most people are only getting the error in Docker, so you can try to run it with Python instead for now.
Just wanted to give a heads up that I'm running into the 403 issue in develop running the script, so it's probably not just a Docker issue.
Within Docker, I can still reproduce this issue. However, the requests seem to go through successfully in some parts and fail in others (although they fail at different requests).
I've been analyzing the requests that have gone through, used headed mode for the browser within Docker (using Xvfb in SeleniumBase), and messing around with the initialization flags that are being used to start the browser. However, nothing has been working as of yet.
Very strange how most people are only seeing it within Docker, but a select few are also seeing it outside of Docker. If this issue starts occurring for me outside of Docker, it will be easier to find out what is wrong but I don't currently have anymore ideas to see what the issue is/how to fix it.
I can repro this 403 issue using a linux server running the python command script directly, and also using docker on the same box. However, everything works on a Mac that is my daily driver. Southwest could possibly be doing some fingerprinting and blocking the connection from untrusted or unseen clients that it assumes to be bots?
This makes sense, as I running raw linux and python and seeing the error. Can we spoof the user-agent maybe?
I just tried this from a Ubuntu 22.04 desktop with Python 3.10.12 and Google Chrome 122.0.6261.111 and the master branch, and it seems to work fine along with the fare check... The system that fails to work is Ubuntu 22.04 server with Python 3.10.12 and Google Chrome 122.0.6261.111 running in an LXC virtual machine same internal network as the machine that works.. If you want logs let me know.
Can we spoof the user-agent maybe?
SeleniumBase already changes the user agent to line up with what headed Chrome has. The user agent can also be changed by adding user_agent="<agent>" on line 121 of lib/webdriver.py.
It seems like a lot of people are running into issues specifically with a server, so I will try on an Ubuntu server VM to see if I can reproduce the issue.
I seem to be encountering a combination of this and #96 on my Unraid box.
Running without privileged: true in the docker-compose just hangs until timing out (like in #96 and #147), and adding it allows the webdriver to actually start before hitting the 403
2024-03-06 20:24:52 DEBUG Process-4[reservation_monitor:86]: Scheduling flight check-ins for 1 reservations
2024-03-06 20:24:52 DEBUG Process-4[checkin_scheduler:76]: Retrieving reservation information
2024-03-06 20:25:08 DEBUG Process-4[utils:40]: Failed to make request after 20 attempts: Forbidden 403
2024-03-06 20:25:08 DEBUG Process-4[utils:43]: Response body: {
"code": 403050700
}
2024-03-06 20:25:08 DEBUG Process-4[checkin_scheduler:82]: Failed to retrieve reservation info. Error: Forbidden 403. Exiting
Can we spoof the user-agent maybe?
SeleniumBase already changes the user agent to line up with what headed Chrome has. The user agent can also be changed by adding
user_agent="<agent>"on line 121 oflib/webdriver.py.It seems like a lot of people are running into issues specifically with a server, so I will try on an Ubuntu server VM to see if I can reproduce the issue.
Hey sir! Funny thing, I just setup a droplet/vm (Digital Ocean) to do this, Ubuntu Server 22.04 LTS, getting same error 403, if you are interested I can DM you details how to get into it if you need a setup to troubleshoot that is already built?
It seems like a lot of people are running into issues specifically with a server, so I will try on an Ubuntu server VM to see if I can reproduce the issue.
I can reproduce the issue with an Ubuntu headless server too so that definitely widens the scope of the issue and will make it easier for me to debug (and hopefully fix).
Running without privileged: true in the docker-compose just hangs until timing out (like in #96 and #147), and adding it allows the webdriver to actually start before hitting the 403
@drippyer If this is still an issue after the 403 issue is fixed, feel free to make a separate issue for it.
if you are interested I can DM you details how to get into it if you need a setup to troubleshoot that is already built?
@j0nsgh I appreciate the offer. I was able to reproduce it on an Ubuntu server, so now I can troubleshoot this further.
@jdholtz Is there a fix for this? Getting 403s on every reservation I put into the config.json. Same error when just using the command line. Same result in docker and not.
@jdholtz Is there a fix for this?
Currently not. It has been working only on my laptop, which has graphics (desktop, window manager, etc.) so if you aren't already running the script on a device that has graphics you can try that--although it isn't the most convenient. You can also try changing line 118 in lib/webdriver.py from headless=True to headed=True and see if that works.
I'll have more time to work on figuring out this issue after the middle of next week.
FWIW, I was hitting these issues as well. Pulled the latest as of today and ran this through a Docker and it's working.
I've also been monitoring this issue for a fix, but noticed yesterday (without updating versions) that the 403 is gone and was able to retrieve reservations. Perhaps a change was made on Southwest's side.
I'll have more time to work on figuring out this issue after the middle of next week.
Tried those options, no dice. Brand new install of Ubuntu on a laptop.
I also tried using username and password and got this error:
Encountered a Too Many Requests error while logging in. Skipping reservation retrieval.
With this comment: python3 southwest.py CONFIRMATION FIRST LAST, here is the stack trace:
Process Process-1:
Traceback (most recent call last):
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/connection.py", line 198, in _new_conn
sock = connection.create_connection(
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 793, in urlopen
response = self._make_request(
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 491, in _make_request
raise new_e
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request
self._validate_conn(conn)
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
conn.connect()
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/connection.py", line 616, in connect
self.sock = sock = self._new_conn()
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/connection.py", line 205, in _new_conn
raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0x762eb8369240>: Failed to resolve 'mobile.southwest.com' ([Errno -3] Temporary failure in name resolution)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/schmeed/.local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen
retries = retries.increment(
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='mobile.southwest.com', port=443): Max retries exceeded with url: /api/mobile-air-booking/v1/mobile-air-booking/page/view-reservation/ZZZCCC?first-name=FIRSTNAME&last-name=LASTNAME (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x762eb8369240>: Failed to resolve 'mobile.southwest.com' ([Errno -3] Temporary failure in name resolution)"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/schmeed/auto-southwest-check-in/lib/reservation_monitor.py", line 44, in monitor
self._monitor()
File "/home/schmeed/auto-southwest-check-in/lib/reservation_monitor.py", line 70, in _monitor
self._schedule_reservations([reservation])
File "/home/schmeed/auto-southwest-check-in/lib/reservation_monitor.py", line 88, in _schedule_reservations
self.checkin_scheduler.process_reservations(confirmation_numbers)
File "/home/schmeed/auto-southwest-check-in/lib/checkin_scheduler.py", line 41, in process_reservations
flights.extend(self._get_flights(confirmation_number))
File "/home/schmeed/auto-southwest-check-in/lib/checkin_scheduler.py", line 56, in _get_flights
reservation_info = self._get_reservation_info(confirmation_number)
File "/home/schmeed/auto-southwest-check-in/lib/checkin_scheduler.py", line 77, in _get_reservation_info
response = make_request("GET", site, self.headers, info)
File "/home/schmeed/auto-southwest-check-in/lib/utils.py", line 30, in make_request
response = requests.get(url, headers=headers, params=info)
File "/home/schmeed/.local/lib/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/home/schmeed/.local/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/home/schmeed/.local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/home/schmeed/.local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/home/schmeed/.local/lib/python3.10/site-packages/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='mobile.southwest.com', port=443): Max retries exceeded with url: /api/mobile-air-booking/v1/mobile-air-booking/page/view-reservation/ZZZCCC?first-name=FIRSTNAME&last-name=LASTNAME (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x762eb8369240>: Failed to resolve 'mobile.southwest.com' ([Errno -3] Temporary failure in name resolution)"))
schmeed@schmeed-UX31A:~/auto-southwest-check-in$ I've also been monitoring this issue for a fix, but noticed yesterday (without updating versions) that the 403 is gone and was able to retrieve reservations. Perhaps a change was made on Southwest's side.
Unfortunately, I am still getting these issues with Docker and an Ubuntu server. There are random intervals people get where it works, so this may just be one of those times.
I also tried using username and password and got this error:
Encountered a Too Many Requests error while logging in. Skipping reservation retrieval.
@schmeed I was getting this error with Chromium, so try Google Chrome if you aren't already using it (uninstall Chromium too so the script doesn't try to use Chromium). As for the traceback, it seems that your DNS might not be resolving correctly due to the NameResolutionError.
I have noticed that southwest is automatically or accelerated blocking many IPs automatically that are known VPS or hosting. I have run the script from both Vultr, DO, and home connection.
Vultr is auto blocked (403) no matter how many IPs I request.
DO gets 403/429 with more than 1-2 reservations per IP.
My home connection has no problems at all with 4 separate reservations on same flight.
All using Ubuntu 22.04 with headless Google Chrome 122.0.6261.128. Same results with reservations and username/password.
Thank you @jdholtz. I was using the Chromium browser and switching to Chrome resolved the issue. Maybe "Any Chromium based browser" needs revised?
Maybe "Any Chromium based browser" needs revised?
I can change it if many people aren’t running into the issue with Chrome. However, my laptop runs Chromium just fine with this script. On the other hand, I have Chrome on an Ubuntu server and am still reaching the 403.
Is there a way to switch with the docker container?
Is there a way to switch with the docker container?
I have a Dockerfile that uses Chrome that I can send in a few hours. I haven’t been having much success with it, but it would be good to have someone else test it.
This person just posted on a couple of other similar projects. Just passing this along.
@FuzzyMistborn here's the Dockerfile. To build, just run docker build -f Dockerfile -t auto-southwest-check-in .
FROM python:3.12-slim
WORKDIR /app
ENV AUTO_SOUTHWEST_CHECK_IN_DOCKER 1
RUN apt-get update && apt-get upgrade
RUN apt-get install -y wget
RUN wget -q https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
RUN apt-get install -y ./google-chrome-stable_current_amd64.deb
COPY requirements.txt requirements.txt
RUN pip3 install --upgrade pip && pip3 install --no-cache-dir -r requirements.txt
COPY . .
ENTRYPOINT ["python3", "-u", "southwest.py"]Still got a Forbidden 403 using the Chrome dockerfile :-(
Edit: Running this via python works just fine as you seem to have also run into. So weird.
Today I restarted the script and now I'm getting 403's again even in Python. So it's not just a Docker thing it seems.
I've done a little more digging and here's what I've found (no solution yet unfortunately):
https://mobile.southwest.com/api/mobile-air-booking/v1/mobile-air-booking/feature/shopping-details is the
first website to hit a 403 as it needs special headers that the browser generates from the JS code given by Southwest. This means that SW is detecting the bot before this request is done (not right before the "Retrieving reservation information").
Using valid headers from my laptop on an Ubuntu server makes the 403 issue go away. This indicates that the issue comes directly from the browser incorrectly generating headers (meaning changing anything with the way the requests module is used will not affect it).
I've also inspected the browser window object that the script's browser formulates on my laptop (which does not hit a 403) and a server (which receives a 403) but there doesn't appear to be any differences between them. Additionally, the original request headers (user agent, content-type, etc.--not the cryptic headers generated by the JS) are the same which indicates Southwest isn't detecting the browser with a simple user agent check.
I'm inclined to think that some sort of display is "needed" which isn't installed on servers (and causes the browser to generate invalid headers). This would also explain why it is working for people on desktops and laptops (although some have reported it hitting a 403 still which contradicts this. I'm not entirely sure if they were running on a server or headed computer though).
https://mobile.southwest.com/api/mobile-air-booking/v1/mobile-air-booking/feature/shopping-details is the first website to hit a 403 as it needs special headers that the browser generates from the JS code given by Southwest. This means that SW is detecting the bot before this request is done (not right before the "Retrieving reservation information").
Do you have that JS code to generate the headers?
Do you have that JS code to generate the headers?
You can find it in a series of requests made when you load https://mobile.southwest.com/check-in (in the network tab of your browser). People have actually tried to reverse engineer the API before (see here), but their engineers appear to have done a very good job to only have it run in a browser environment (hence, why this project needs to use a browser to generate the headers).
For the record, I had the script running in a Docker container using username/password. It successfully found the two flights in the round trip and checked in for the first one. Then when it went to do the second flight days later, it got the 403. So I assume the error can't be because of the environment I was running it in.
So I assume the error can't be because of the environment I was running it in.
I've been getting similar results, where about 25% of the time I don't get a 403 and it schedules my flights fine. A possible explanation is that they are using AI through a third-party service to detect bots (the link to the article explaining this can be found here: #201 (comment)) which would explain the inconsistency.
For reference, I am running an Ubuntu server as a virtual machine on my laptop hitting the 403, while my host works perfectly fine. That's my main reasoning as to why I think it is the 'server' environment as Docker also has a very similar environment to servers (headless, no graphics server, etc.)
man that sucks, I wish they would just let us be! lol
Got 403 errors in Docker on both headless linux and headed Mac servers.
Managed to get it working by running python script directly on headed Mac Mini server by setting headed=True as suggested in #230 (comment)
Thanks!
Have you tried installing a XFCE desktop environment on the remote Ubuntu server and configure it to run a VNC server? - Something like this https://vitux.com/ubuntu-vnc-server/ . I'm thinking that way you may be able to create impression that the ubuntu server is a machine that does have a display, and prevent the Forbidden 403 error. Could be a reasonable workaround if this works.
Have you tried installing a XFCE desktop environment on the remote Ubuntu server and configure it to run a VNC server?
I have set up a VNC server through Docker, but I unfortunately still saw the issue. I'm also seeing the issue now with a desktop so it's definitely doing more to detect the bot than just seeing if the client is using a desktop or not (I'm not sure how that detection would actually be done).
I would like to add that they're definitely doing more to detect the bot. Still seems hit or miss though whether its 429 or 403. I've done several tweaks to try to slow down and randomize when things are done, but may be more to it...
Maybe this has been covered before, but is there any particular reason why the mobile site has to be used for this script? Many moons ago, I had written a SWA check-in script in PHP, using the standard website URL as the target. I'm sure it's changed significantly since then, but maybe the main site is less restrictive when it comes to bots/scripts?
I'm running an Ubuntu VM (via Unraid). I was getting a 403 on every attempt. The suggestion in comment #230 resolved the issue for me.
@rnegrette - I'm running it directly on my Ubuntu server which will work normally. However it is hit or miss at this point in it's detection and throwing errors of 429 and 403.
I'm running it directly in python on a linux server and getting 403 errors each time.
I can confirm that installing a lightweight window manager (even if the logged-in session isn't using it) works.
I can confirm that installing a lightweight window manager (even if the logged-in session isn't using it) works.
So, can someone try doing that in a docker container? Seems super silly, but hell, if it works, it works? I like the randomized wait times, too..
On Mac, running the application locally:
if self.is_fork_ctx:
^^^^^^^^^^^^^^^^
AttributeError: 'Lock' object has no attribute 'is_fork_ctx'
Through docker:
Reason: Forbidden 403.
Make sure the reservation information is correct and try again.
@devtrf for the issue you are running into locally, can you make an issue or discussion for it?
Ladies and Gents you won't believe it. I was able to successfully check in with v5.0 no errors and price checks successfully. Running on Docker with Synology. Try it out, let me know
My observation is that SeleniumBase chrome could be the cause for headless in Docker
Edit:
v5.0 works! - STABLE
v6.1 works! - STABLE (using it for my check ins) - successfully logged in and checked in 4 flights without any errors. Will stay on this version.
v7.0 works! - unstable
v7.3 works! - unstable
v7.2 does not work!
v7.4 does not work!
vDevelop does not work!
@dmytrokoren I'm still getting:
Failed to retrieve reservation for <MY_NAME> with confirmation number <CONF_CODE>. Reason: Forbidden 403.
I'm launching this via:
docker run --name <docker_name> --volume <config_json>:/app/config.json -d jdholtz/auto-southwest-check-in:v6.1
update:
Running on v5.0 is working for me so far.
v5.0 is working for me as well.
@dmytrokoren I'm still getting:
Failed to retrieve reservation for <MY_NAME> with confirmation number <CONF_CODE>. Reason: Forbidden 403.I'm launching this via:
docker run --name <docker_name> --volume <config_json>:/app/config.json -d jdholtz/auto-southwest-check-in:v6.1update:
Running on v5.0 is working for me so far.
v5.0 is the most stable as of now for most but I'm still having luck with v6.1 via login or reservation.
other issue with those two (for me) is that I need the version for raspberry pi but I think those are only 7 and above haha
Perfecto!!!
V7.4 (latest) is working for me now. Successful login, check-in and fare check... (4 flights checked in)
I hope this work for you.
What I noticed is that all were checked for price under 1 attempt. 😀
Workaround:
All you need to change is the GPG_KEY, take the values from v5.0
"GPG, also known as GNU Privacy Guard, is very commonly used to digitally sign files in order to guarantee their authenticity."
** I guess Southwest checks for authenticity. The gpg_key in v7.4 is not verified, I would assume.
Can someone confirm if it's working for them?
Perfecto!!!
V7.4 (latest) is working for me now. Successful login, check-in and fare check... (4 flights checked in)
I hope this work for you. What I noticed is that all were checked for price under 1 attempt. 😀
Workaround: All you need to change is the GPG_KEY, take the values from v5.0
"GPG, also known as GNU Privacy Guard, is very commonly used to digitally sign files in order to guarantee their authenticity."
** I guess Southwest checks for authenticity. The gpg_key in v7.4 is not verified, I would assume.
Can someone confirm if it's working for them?
Didn't last too long, after 5/27 the fare checker getting 429
I was looking into the southwest api, and came to know that these values are important.
"code": 403050700 is thrown if the values below are not there or not matching, changing one letter or digit will give the error code, even if you have the correct cookies.
EE30zvQLWf-a:
EE30zvQLWf-b:
EE30zvQLWf-c:
EE30zvQLWf-d:
EE30zvQLWf-f:
EE30zvQLWf-z:
** I have done some validations, that cookie session changes but we can still keep the same values for EE30zvQLWf (a, b, c, d, f, z).
I have logged out in UI and ran the api and the request was unauthorized due to cookies marked as UNAUTHORIZED. I logged back in UI and updated the cookies in api (left the EE30zvQLWf unchanged) and the request came as 200 with upcoming-trips data.
user-agent, not that important - I removed it and the request still was 200.
can someone suggest how to pass headers values to chrome via docker?
Maybe this has been covered before, but is there any particular reason why the mobile site has to be used for this script?
@agentdr8 the mobile site has always been less restrictive (at least, since the start of this project) so the script has always used it over the regular site. I did try the regular site at one point a few months ago but still got the same 403 (and the regular API is much more complicated). It seems now that they are also getting their mobile API to a similar level of security.
I can confirm that installing a lightweight window manager (even if the logged-in session isn't using it) works.
Has someone been able to verify this?
@dmytrokoren thanks for trying the different versions. I would really like to not use v5.0 as undetected-chromedriver is used instead of SeleniumBase and undetected-chromedriver is a much lower quality project than SeleniumBase (and rarely maintained). Has v5.0 ran successfully for you every time? That's definitely something to look into to see why that is working well and v7.4 is not at all (mostly between undetected-chromedriver and SeleniumBase).
I was looking into the southwest api, and came to know that these values are important.
Those seem to be generated by JavaScript using techniques described here. That's probably how they mostly detect bots and the reason a browser is even needed in the first place.
can someone suggest how to pass headers values to chrome via docker?
I'm not too sure how you can do it via Chrome, but you can use valid headers from your local machine, if that is working, and have the script use those headers in Docker. That allows the script to work correctly in the Docker container but obviously that's not an actual solution.
@agentdr8 the mobile site has always been less restrictive (at least, since the start of this project) so the script has always used it over the regular site. I did try the regular site at one point a few months ago but still got the same 403 (and the regular API is much more complicated). It seems now that they are also getting their mobile API to a similar level of security.
Makes sense, and it's rather unfortunate they're tightening things up.
To add some more context to this particular issue, I did not encounter an issue using v7.4 on my Ubuntu 22.04.4 host (which does have XFCE4 installed) via screen session when I flew back yesterday. It checked me in just fine.
I have another pair of flights next week, so I'll be keeping tabs on them as well. If you need any debugging output, please let me know.
@dmytrokoren thanks for trying the different versions. I would really like to not use v5.0 as undetected-chromedriver is used instead of SeleniumBase and undetected-chromedriver is a much lower quality project than SeleniumBase (and rarely maintained). Has v5.0 ran successfully for you every time? That's definitely something to look into to see why that is working well and v7.4 is not at all (mostly between undetected-chromedriver and SeleniumBase).
v5.0 is not working any more, unable to login for unknown reason.
I was looking into the southwest api, and came to know that these values are important.
Those seem to be generated by JavaScript using techniques described here. That's probably how they mostly detect bots and the reason a browser is even needed in the first place.
We don't need to generate every time, just need to inject header with those (EE30zvQLWf a,b,c,d,f,x) values and it will be working every time. I was able to verify via postman. All I update is cookies. @jdholtz this is valid solution. Can we give it a try? I'm not strong in docker
can someone suggest how to pass headers values to chrome via docker?
I'm not too sure how you can do it via Chrome, but you can use valid headers from your local machine, if that is working, and have the script use those headers in Docker. That allows the script to work correctly in the Docker container but obviously that's not an actual solution.
How to get a script with headers to run in docker?
Here are screenshots of fair check.
(cookies are not required for fair check)
Checking with highlighted values:
Checking with disability single value:
Injecting the headers, resolved the 403 and 429 issue, but if you do multiple logins on and off even on Mac you will get too much requests error. But it goes away after ~20 min or so.
I created my own docker image and tested with Synology on v7.4 all is working now. I have updated SeleniumBase version now using chrome v125. and injected headers with suggested key:values. Let me know if you want to test, I can share my Image.
I'll be happy to test your image on my Synology.
I'll be happy to test your image on my Synology.
Here you go, try it out: https://hub.docker.com/r/dmytrokoren/auto-southwest-check-in
If any one wants to give it a try for arm64 and arm/v7
That's working for me, @dmytrokoren !
@jdholtz ill raise a PR for the fix. Will you be able to approve it?
@jdholtz ill raise a PR for the fix. Will you be able to approve it?
Yeah, definitely if it fixes the issue! I'm confused on what the fix is though.
@jdholtz ill raise a PR for the fix. Will you be able to approve it?
Yeah, definitely if it fixes the issue! I'm confused on what the fix is though.
It's the headers, only 7-8 lines of code (clean code :) ) - it might be a bandaid but at least it doesn't hurt lol. I'm currently running it on Synology docker and it's working every time and no errors.
It's the headers
Where does it get the headers from? Currently, the script runs the browser and snags the headers that it used and attaches them to requests made after that (not using the browser).
It's the headers
Where does it get the headers from? Currently, the script runs the browser and snags the headers that it used and attaches them to requests made after that (not using the browser).
Correct. I'm substituting some of the key:values and user-agent to be the same as if you ran on Mac safari (doesn't matter if we using Chrome). The rest of the values it's getting from the request_headers.
See the issue we facing in docker is that those values could be wrongly generated at times or all the time now. That is why we facing 403.
We are not facing running it locally because they are being generated properly on Mac or Windows.
@dmytrokoren is your method to manually pull the headers and then input them? Or am I misunderstanding something?
Doesn't those after sometime change? I remember coming across another similar project and in his project he ran up against issues with the header as they'd change over time?
@dmytrokoren is your method to manually pull the headers and then input them? Or am I misunderstanding something?
Doesn't those after sometime change? I remember coming across another similar project and in his project he ran up against issues with the header as they'd change over time?
They might change, after some time - no time frame. I'm not 100% sure, but with same values I've already done postman api request for 3 days, and they are holding up
@dmytrokoren So I wonder if it is better to compare and see what headers the script is pulling if any in some instances to help determine what the problem is.
I wonder if there is some dependencies for those headers? Because if the script is running and searching for the headers and pulling it in shouldn't matter if manually inputed or not I would assume?
Maybe something is getting truncated? I dunno...just something to think about...hard coding it would mean it would just be a little more work to update.
I wonder if it's worth adding into a config file to either use your own header or allow the script/selenium base do pull the headers?
@dmytrokoren So I wonder if it is better to compare and see what headers the script is pulling if any in some instances to help determine what the problem is.
I wonder if there is some dependencies for those headers? Because if the script is running and searching for the headers and pulling it in shouldn't matter if manually inputed or not I would assume?
Maybe something is getting truncated? I dunno...just something to think about...hard coding it would mean it would just be a little more work to update.
I wonder if it's worth adding into a config file to either use your own header or allow the script/selenium base do pull the headers?
@babehboi Im looking into it to see what is actually happening in docker container. I'm not sure if it's possible that values get truncated because same script works fine locally.
I'm not manually pulling header. I'm just intercepting and putting in corrected key:values (8 key values). If these values expire (no one knows the duration) then the best bet would be using from config file. Good suggestion, but let's see.
@dmytrokoren Another thing to think about is why does it happen pretty randomly too? So here is an instance, I have I think 7 different reservations already setup for the year...of the 7 only 1 hit the 403 error today, probably by tomorrow it should successfully retrieve it, as it has been working all week for the same 7.
So then what's driving it, I may try your attempt and pull those headers myself and see if it makes a difference, but I feel some bot detection is still at play...
@dmytrokoren Another thing to think about is why does it happen pretty randomly too? So here is an instance, I have I think 7 different reservations already setup for the year...of the 7 only 1 hit the 403 error today, probably by tomorrow it should successfully retrieve it, as it has been working all week for the same 7.
I'm not sure what is going on there, but yesterday around 4pm CST, the api was unable to login because on SW side they were doing something.. I also tried to manually login and was getting 500 error.
So then what's driving it, I may try your attempt and pull those headers myself and see if it makes a difference, but I feel some bot detection is still at play...
Bot detection is there, this is how probably "EE30zvQLWf" key values are generated. Since we getting headers from automated chrome, it is definatly identifies it.
:) There bot detection did not work well last time, that why on v5.0 it worked (no longer works) - that's why we had no issue with 403.
@dmytrokoren, I just tried your forked image and I haven't seen it throw an error yet.. but it's only be 20 mins or so. Promising thus far, though!
I did see this debug message though. That's kind of a bummer; but a separate issue all together.. would love to see this be supported @jdholtz ..
2024-05-30 15:53:56 DEBUG Process-1[reservation_monitor:112]: Fare check is not supported with companion passes. Skipping fare check
Can it simply not see the value of the original fare if a companion pass is resting on top of it? Perhaps we can hardcode the value into the config file or something? I just want to be alerted if the fare drops.. then I can decide if I want to cancel the companion record and redo the booking manually or not.
@dmytrokoren, I just tried your forked image and I haven't seen it throw an error yet.. but it's only be 20 mins or so. Promising thus far, though!
I did see this debug message though. That's kind of a bummer; but a separate issue all together.. would love to see this be supported @jdholtz .. 2024-05-30 15:53:56 DEBUG Process-1[reservation_monitor:112]: Fare check is not supported with companion passes. Skipping fare check Can it simply not see the value of the original fare if a companion pass is resting on top of it? Perhaps we can hardcode the value into the config file or something? I just want to be alerted if the fare drops.. then I can decide if I want to cancel the companion record and redo the booking manually or not.
this looks like companion details: https://www.southwest.com/api/loyalty-management/v2/loyalty-management/accounts/self/companion-details-secure
but I have no companions there so I don't see any data:
{
"meta": {
"api_version": "2.14.001-1"
},
"data": {}
}
@hildebrau - What happens when you manually check your reservation? Does it show up any prices for you? Because the script is just scraping data if it can see the data? Just to be sure, you're checking the Primary - non-companion pass holder's reservation right?
I don't have a companion pass so I can't confirm. I should note if you look for a closed issue for companion pass. (#78)
I did see this debug message though. That's kind of a bummer; but a separate issue all together.. would love to see this be supported @jdholtz ..
2024-05-30 15:53:56 DEBUG Process-1[reservation_monitor:112]: Fare check is not supported with companion passes. Skipping fare check
Can it simply not see the value of the original fare if a companion pass is resting on top of it? Perhaps we can hardcode the value into the config file or something? I just want to be alerted if the fare drops.. then I can decide if I want to cancel the companion record and redo the booking manually or not.
It checks the fare of the primary flyer just fine. Companion flights are free, so it can't check the fare for your companion.
For example, I have both my wife and I in the config; I have a companion pass and she is my companion. It checks my fare for decrease/increase fine; on my wife's account, it skips it, since she's my companion. But for flights where she is flying alone, it checks those fine too.
I didn't mean to derail this towards companion pass discussion.. sorry! But, the log entry I showed above was from it logging in as the primary (non-companion) account and seeing the booking. It skipped it for fare check seemingly because it has a companion tied to it. It may also matter that it was a purchase by points.
@babehboi asked what happens when I manually check the reservation. I'm not sure which logic path the script uses to find the original fare of the booking. Once upon a time the "change flight" flow could be used.. but due to "Companion itinerary linked" to this reservation, that flow is blocked. But now I'm really curious how this works for @borski for fare checking his itineraries that have companion itineraries linked to them.
If I could manually enter in the USD or point value of the reservation in the config file, then perhaps the script could be smart enough to monitor that flight segment or segments cost and compare it in USD or points.. whichever unit was given in the config and alert if the fare went down. I guess that's a feature request.
@hildebrau if I remember correctly the script basically will hit the "change" flight button. If that's no longer available is probably the reason why because it will do a check based on your flight time and get the fares accordingly
I'll be happy to test your image on my Synology.
Here you go, try it out: https://hub.docker.com/r/dmytrokoren/auto-southwest-check-in
If any one wants to give it a try for arm64 and arm/v7
I am having this issue in docker, but this image fixed it!
I'll be happy to test your image on my Synology.
Here you go, try it out: https://hub.docker.com/r/dmytrokoren/auto-southwest-check-in
If any one wants to give it a try for arm64 and arm/v7I am having this issue in docker, but this image fixed it!
Well, maybe not.. Getting random 403's within docker again.
Well, maybe not.. Getting random 403's within docker again.
Strange, I had mine already running for 6+ hours now. At 10am cst it did the fair check. Again at 7pm it ran, received proper headers and done the fair check all in 1 attempt. 4 flights scheduled.
Try again. Pull latest docker image and run the script. Keep us posted!
Well, maybe not.. Getting random 403's within docker again.
Strange, I had mine already running for 6+ hours now. At 10am cst it did the fair check. And at 7pm it had ran for the proper headers and done the fair check all in 1 attempt. 4 flights scheduled.
Try again. Pull latest docker image and run the script. Keep us posted!
I am wondering if it's because I have a config with multiple accounts in the config. Maybe spin up three containers, one for each login?
I am wondering if it's because I have a config with multiple accounts in the config. Maybe spin up three containers, one for each login?
I would recommend running one account per container. SW is cracking down on bots. Spin up 3 containers and let me know the outcome.
I am wondering if it's because I have a config with multiple accounts in the config. Maybe spin up three containers, one for each login?
I would recommend running one account per container. SW is cracking down on bots. Spin up 3 containers and let me know the outcome.
Damn, something again on SW. I'll into this more but right now it's failing...
I have it running in three containers, one of them got a 403 immediately on fare check. We will see what it does today during my checkin. 10:20am CST checkin.
Checked in fine. (I got B55 though 😢 )
That's even more sinister! Don't fail the check-in, just make it really slow so everyone gets bad seats.
…
On Fri, May 31, 2024 at 11:40 AM edshaw42 @.> wrote: Checked in fine. (I got B55 though 😢 ) — Reply to this email directly, view it on GitHub <#230 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGG6AKNZMFBZNNDNMHYCJ3ZFCRW3AVCNFSM6AAAAABD2NNZQSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBSGYZDGMRZGU . You are receiving this because you are subscribed to this thread.Message ID: @.>
The check-in happens at same speed as it is. No extra time was added. Possibly SW or there lots of folks that opt-in for early bird )
@edshaw42 @babehboi try pull the docker image again (latest tag). I'm not seeing any failure with 2 reservations or login option with 4 flights scheduled and fair check.
@jdholtz can you also try and maybe you can review my PR.
I have it running again, will update over next few hours what it does.
2 containers running for each login.
@edshaw42 @babehboi try pull the docker image again (latest tag). I'm not seeing any failure with 2 reservations or login option with 4 flights scheduled and fair check.
@jdholtz can you also try and maybe you can review my PR.
I have it running again, will update over next few hours what it does.
2 containers running for each login.
You could now do multiple logins under one container