Public certstream-server down

Question

Public certstream-server down

Closed this issue 7 months ago · 9 comments

Hey Ryan and the CaliDog team,

first off I highly appreciate the work you do and hope are doing well.
I have been using your server as a client, since I didn't want to run my own server for a side-project.
According to my logs and datadog, the server went black Sunday night and hasn't come back up.

https://p.datadoghq.com/sb/e72980047-c37b10c02bf48fa80b013416ac899d15?refresh_mode=sliding&theme=dark&from_ts=1694116226902&to_ts=1701892226902&live=true

Also, could you please elaborate on this part of the README:

single Hetzner dedicated server without issue (~250TB of data every month!).

What specs does the server have and how much of that 250 TB is distributed between http watcher and the websocket clients?
If I run it myself would I get away with significantly less traffic? Have you ever contacted Hetzner or been contacted concerning traffic?

Kind regards

Answer 1 · 2023-12-06T21:13:00.000Z

went down around the same time for me too.

https://twitter.com/OmgImAlexis/status/1732362070723792963

Answer 2 · 2023-12-06T21:14:08.000Z

if this server doesnt come back up soon i may stand my own version of it up. 🙏

would also be interested in what the average stats would be to run one of these for my own projects and/or the public. 🤔

Answer 3 · 2023-12-07T08:49:16.000Z

If its any help, the server went down at
2023-12-05 04:14:45.2244-ERROR: The server returned status code '521' when status code '101' was expected.

I have my own instance running elsewhere so I've moved some of my services to that.

Answer 4 · 2023-12-08T02:15:09.000Z

Are there alternative solutions?

Answer 5 · 2023-12-08T04:08:08.000Z

I've deployed this and it's been working well for several months now. https://github.com/d-Rickyy-b/certstream-server-go

…

On Fri, 8 Dec 2023, 02:15 iuime, ***@***.***> wrote: Are there alternative solutions? — Reply to this email directly, view it on GitHub <#110 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AENQH25BVENI4FLCAOC4V73YIJZ3RAVCNFSM6AAAAABAJ5Z7UKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBWGQ2DSOBXGE> . You are receiving this because you commented.Message ID: ***@***.***>

Answer 6 · 2023-12-14T01:06:33.000Z

Hi, apologies for the downtime.

The public server is not really meant to be consumed for more than simple proofs of concepts - if you're ingesting this data yourself you should consider running your own server - https://github.com/CaliDog/certstream-server. There are congestion issues that might cause you to lose certificates when the server's bandwidth is maxed out (which is usually the case despite it being a gigabit connection).

Things are back up but please consider running your own server if you're using it for something important!

Answer 7 · 2023-12-15T06:49:55.000Z

Hi, this is huge volume of data … focus on certificate_update message type, it should cut all of staging, pre certificate, etc. non actionable data

…

On 14 Dec 2023, at 21:40, MaJunior420 ***@***.***> wrote: ~250TB

Answer 8 · 2023-12-15T07:26:10.000Z

I ve had it running for a few months now so I will see if I can get some metrics on bandwidth. I only have one local client reading the messages and firing hits to another server. Bandwidth from that will be minimal.

…

On Thu, 14 Dec 2023, 19:40 MaJunior420, ***@***.***> wrote: I've deployed this and it's been working well for several months now. https://github.com/d-Rickyy-b/certstream-server-go I am trying to do the same. Do you have an idea of how much data this would be to feed only a couple of connections? I read ~250TB of data every month but is that serving the public? Thanks. — Reply to this email directly, view it on GitHub <#110 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AENQH22SATD2DRXAVGJ7Q6DYJNI3ZAVCNFSM6AAAAABAJ5Z7UKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJWGQ3TANZXHA> . You are receiving this because you commented.Message ID: ***@***.***>

Answer 9 · 2023-12-17T00:54:08.000Z

I am running my own version of the certstream-server (that has been mentioned above) and it needs about 13.5 Mbit/s bandwidth downstream (from the CT Log) and 1-20 Mbit/s upstream per client (depending on the stream).

root@certstream:~$ vnstat -d

 enp0s3  /  daily

          day        rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     2023-12-09    37.17 GiB |  776.66 MiB |   37.93 GiB |    3.77 Mbit/s
     2023-12-10   129.65 GiB |    7.23 GiB |  136.87 GiB |   13.61 Mbit/s
     2023-12-11   139.13 GiB |   80.71 GiB |  219.84 GiB |   21.86 Mbit/s
     2023-12-12   133.95 GiB |  164.79 GiB |  298.74 GiB |   29.70 Mbit/s
     2023-12-13   135.06 GiB |  162.04 GiB |  297.10 GiB |   29.54 Mbit/s
     2023-12-14   136.94 GiB |  165.05 GiB |  301.99 GiB |   30.02 Mbit/s
     2023-12-15   131.88 GiB |  163.10 GiB |  294.98 GiB |   29.33 Mbit/s
     2023-12-16   127.01 GiB |  159.98 GiB |  287.00 GiB |   28.53 Mbit/s
     2023-12-17     9.66 GiB |   10.93 GiB |   20.58 GiB |   31.02 Mbit/s
     ------------------------+-------------+-------------+---------------
     estimated    146.36 GiB |  165.60 GiB |  311.96 GiB |

Some explanation on the data: I started traffic recording on 2023-12-09 some time in the evening, so I only got a fraction of the data. rx is the data received from the ct log (and of course all other requests to the server).

tx is the transmitted data from the certstream server to the client. Starting on 2023-12-11 there is a single client that connects to the full-cert websocket and consumes about 17 Mbit/s of bandwidth.

While this might not be directly comparable to the original certstream-server it does give an approximate indication on how much data is being processed.