New HSBC statements page breaks OBIS
Closed this issue · 21 comments
I'm aware that HSBC UK have now updated their statements pages, meaning that OBIS no longer works. I'll be updating the parser soon.
Commit 02f7b65 introduces a parser for the new dojo-based HSBC UK web app, which means you can run OBIS from a [ View statement ] page again.
I'm afraid multi-statement download is a more difficult job than before, and I've not tackled it yet, meaning it's a one-at-a-time task to download your statements in OFX/QIF/JSON/etc. formats. :(
Also note: HSBC does not correctly calculate running-balances any more, so you'll see a few "Calculation discrepancy" warnings in the OBIS statement-viewer.
The exported data should be fine despite this, although it will not be consistent with the old formats, so be careful if you plan to re-import the same statements again.
This is absolutely fab shuckster. Really appreciate the work on this. Any news on an update to address the new statement pages? Any way this parser can download multiple statements from two separate bank accounts with HSBC at the same time?
Thanks masterell, glad you’ve found it helpful! I must admit I’ve not made any further changes since this report. I do need to use OBIS very soon though, so expect a few tweaks when I come across the same issues as you!
I’m not sure about multiple statement download for the time being. This was much easier to achieve with their old website and worked well. I’ll see what I can do though.
Thanks! Really appreciate that. I guess even if OBIS opens all the different accounts/statements in different tabs with the download options available in each individual tab, it would be handy. HSBC don't make things easy!!
Just had a look through myself. I'm not particularly well versed with Dojo but from what I can see there doesn't appear to be an easy way to parse multiple statements at once (let alone multiple statements from multiple accounts). One could attempt to parse the statements via the PDFs as their links are readily available from the "View/Download Statements" page, but this would be quite messy and not perfect. I've started looking into the various scripts HSBC runs to generate the statements.
It appears this is the form template they use:
Which uses the following script:
Which takes:
https://www.content.online-banking.hsbc.co.uk/app/group/gpib/cmn/bijit/RefererTool.js?uk317
Wow, thanks so much for the digging! As you can probably tell I'm being pulled away from this at the moment, so apologies for my late replies. But I'll be back on it before too long!
I used OBIS about a year ago and it worked fine (thanks!) but had to reinstall the bookmarklet today after a Chrome issue and can't get it to run at all. Nothing happens when I hit the bookmark. Any suggestions?
Hi @Sprocket2020, I'll take a look at it this weekend. Been a while since I've used it myself!
Many thanks. HSBC is so awful with their statements that I can hardly believe that I'm still using it!
I hear that!
I've just tried OBIS and it seems to be working okay. Here's a screenshot:
You do have to download statements from My Accounts > Manage > Statements, then click View statement for the statement you wish to download, then the OBIS bookmark:
It may take a moment or two for the controls in the first screenshot to appear. It depends on how responsive Github Pages is. If it's been a while since you used the site you could try clearing your cookies/cache for the hsbc.co.uk domain.
Let me know how it goes!
I just spent a miserable afternoon cutting and pasting transactions from pdf files to Excel. I'll try OBIS again right now.
Wow! My fault, it seems. I didn't actually pull a single statement down before using the bookmarklet - how dumb can you get? :-(
The good news is that I have another account to do which I was just about to start so that's going to make life lots easier. Many thanks!
Excuse my (total) lack of modern coding experience, but am I right in thinking that the bank creates the pdf files on demand for the older statement downloads and OBIS compiles its various data formats from the same source? If so, why don't they allow useful downloads for at least the period required for UK tax reporting?
I can't be sure that HSBC's own PDF statement are generated on-the-fly, but certainly they would use their own API to build them. OBIS uses the data in the HTML page to offer different formats.
Actually, last time I checked the statement-data can now be accessed in a global JS object, so the parser could be simplified.
But quite why HSBC can't offer formats other than PDF is probably more to do with political reasons, since it's not exactly a difficult technical problem for them!
Have recent changes broken this? I can't get statements on the web page any more (like in the 10 Jan screenshot above). There is just a page for downloading old pdf statements (and MiData which only goes back 12 months). Any ideas how to reach the previous statement pages, or gone now?
Hi @prisonblues - I've been working on-and-off on a new version of OBIS that's compatible with more recent HSBC changes, but wasn't aware that the old one had broken.
I'll take a closer look in the next couple days, but I'm trying to prioritise the new version currently!
Turns out I had to update the user-content URLs, but doing so has not completely solved the problem.
Unfortunately, the HSBC statements-page has been completely updated and is only offering PDF downloads. There is no HTML page that would be easy to scrape anymore.
The other problem is that the updated page has a new CSP -- or Content Security Policy -- forbidding the Bookmarklet to load. It was highly unlikely that OBIS would work anyway on this new statements page, but the CSP is another obstacle to getting it working again.
Anyway, I'm still tinkering around. If I have any luck I'll let you know!
I used OBIS in 2o19 but had to resort to a commercial product last year. Bank2CSV by ProperSoft worked OK for taking the details from the PDFs and I only needed to subscribe for a month at a time so not wasteful, given I only need to use it once a year. The best part for me was that You can set it to work on all twelve months in one batch which is a big time saver. I hope you manage to get OBIS running again and I'll revert.
Hey @Sprocket2020 , and everyone else too.
I've completely rewritten OBIS. It works a treat locally, but it looks like Chrome's CORB security-measures prevent OBIS from loading via Github.
Still, you can try it out locally if you want by cloning the project. I use pnpm, but npm should work fine:
git clone https://github.com/shuckster/OBIS.git
cd OBIS
pnpm i
pnpm run debug:liveAt this point an express-server should be running, and a local bookmarklet will have been built in dist/ that points to it. Click it after logging into your HSBC UK account.
Let me know how you get on!
Better instructions now in README.