Make a staticman app instead of staticmanapp to avoid reaching quotas

Question

Make a staticman app instead of staticmanapp to avoid reaching quotas

Closed this issue 6 years ago · 68 comments

The public instance of Staticman is in trouble: API calls are now regularly hitting the Github quotas of 5000 hits per hour per user.

The issue is that every comment on any site using this instance is going through the staticmanapp user, therefore staticmanapp easily reaches its API quotas.

This causes the problems that have been repeatedly mentioned in #227, #222, or #242 for example.

I got in touch with the Github staff about this and here's their recommendation:

instead of using a single account to make all those requests, you should build an app so that your users can authorize/install the app. That way, the rate limits will scale much better -- for OAuth Apps, each user has their own quota and for GitHub Apps each installation has its own quota. So, your total rate limit would scale with the number of users instead of being static, which is what you want -- you want the limits to grow with your userbase.

@eduardoboucas you know your software - would this be possible?

Answer 1 · 2018-12-12T05:32:24.000Z

@robinmetral
I am in process of creating an GitHub app that tries to work as staticman. It is still in development and is in a private repository. I can extend invitation if you want to look or contribute.

I expect it to be completed by end of this month.

Answer 2 · 2018-12-12T06:37:05.000Z

Good to hear @Erised!
Let me know when it's ready, I can help with testing 🙂

Answer 3 · 2018-12-12T11:02:42.000Z

@robinmetral Sure. I will let you know.

Answer 4 · 2018-12-14T10:18:55.000Z

Just a friendly wake-up call to @eduardoboucas. Staticman is awesome, but right now, it's hitting its limits and people are developing alternatives. Please find the time to develop a solution so we can continue using this fantastic app without this very annoying limitation.

Answer 5 · 2018-12-14T11:20:59.000Z

Hi everyone. I'm sorry that some people are frustrated with the project, I can relate to that. But please remember that the code is fully open-source, which means anyone can simply run their own instance, with their own GitHub account, and bypass all these limitations. The issue we're seeing here is really a problem with the free, public instance I decided to host for everyone. In hindsight, this probably wasn't the best of ideas, because it puts pressure on me to be the sole gatekeeper of this service.

I've not abandoned the project and I'm thinking of solutions to solve both problems: the problem that a single GitHub account acting on behalf of everyone isn't maintainable, and the problem that in the current scheme of things, where the public instance that I run is the centrepiece of the project, I represent a bottleneck.

For example, I'm keen on the idea of rebuilding Staticman as a Netlify function, so that effectively everyone is running their own instance (for free) rather than relying on a centralised service (which I'm covering the costs for). In this scenario, people would provide access to their own GitHub account, which commits would be made from, thus removing the issue of quota limits.

All I can say is that your patience is much appreciated and I'd love to hear everyone's thoughts on how we can make this more manageable for everyone.

Answer 6 · 2018-12-14T11:34:21.000Z

@eduardoboucas The best solution in my opinion, would simply be a GitHub app.
GitHub Apps also have their call limits, but they have this per installation.

If repo A & repo B, install an app on their repos, each gets 5k hits per hour.
I have already started working on this, I will present it here when it is prsentable.

Till then, I am also open to any discussion that how we can make it work for everybody.

Answer 7 · 2018-12-15T04:01:24.000Z

Thanks for the updates @eduardoboucas ! 🙌

A Netlify function would be great as a simpler solution to people wanting to self-host their instance.

However one of the things I love with Staticman is how simple it is to set it up only using a Github repo!

I think that many people would benefit from such a "centralized" service, be it for testing or to allow comments on small websites and blogs (GH pages for example), where setting up Netlify+AWS Lamba to run a self-hosted instance seems like a lot of trouble.

In this case a Github app would probably be the best solution! @Erised can you consider making your working repo public so that we can take a look and maybe contribute? 🙂

Answer 8 · 2018-12-15T09:34:23.000Z

@robinmetral Please give me a day or two.

I want to finish a few tasks by myself.
I don't want myself to be an embarrasment as this is my first project.
I just need it to be presentable with a few features. Then, i will work on it with everyone.

Answer 9 · 2018-12-15T10:33:30.000Z

I don't want myself to be an embarrasment as this is my first project.

No reason to feel embarrassed at all! We all appreciate the effort you're putting in. Whenever you feel comfortable showing your code, I'm happy to review and help you change anything that needs tweaking.

Answer 10 · 2018-12-15T21:25:53.000Z

@eduardoboucas Thank you. It means a great deal. I am looking over staticman's code all the time to see how everything works.
I am sure will let you know.

Answer 11 · 2018-12-16T01:18:34.000Z

@Erised if you need any help I'd be happy to assist as well.

Answer 12 · 2018-12-17T13:55:22.000Z

Thanks for your reply @eduardoboucas. Staticman running in a netlify function sounds like a very good idea. I'm looking forward to you testing this out and, if found feasible, giving us feedback on how to set it up for ourselves. Keep up the good work :D

Answer 13 · 2018-12-29T07:26:23.000Z

If i understand things correctly, Netlify provides a way to translate form submissions into the Functions (aws lambda events). However, the free plan limits this to 100 form submissions per month (even though the Functions limit for the free plan is much higher). Just FYI.

Meanwhile, if i understand things correctly (and I may not) the /connect controller is invoking the GET /user/repository_invitations API method (https://developer.github.com/v3/repos/invitations/#list-a-users-repository-invitations) which will only return the first 30 invitations. I do not see any handling of pagination (although possibly it's built into the github client library you are using by default, or i am missing some setup someplace). Thus, many calls to connect are likely to fail with Invitation Not Found if there are more than 30 queued up. And, their continuing retry attempts help exhaust the API limits.

If the above analysis is correct a few things would greatly help:

set the page size to 100 (the max github allows)
handle pagination. if the api returns them in FIFO order, then perhaps starting from the tail would help - which really means get the first page, then jump to the last page (the link should be provided in the api response) and work toward the front until found. This would favor recent invitations.
Possibly, when traversing the list of invitations, reject all those older than N hours.
Cache the list of invitations for N minutes to reduce API hits to github. Suspect that if you reject the old invitations then the current list at any given time should be maintainable at <100 (so a single page)

Sorry I don't have time to do it myself - but maybe someone else can contribute improvements here. My suggestion would be to start with just setting the per_page parameter and rejecting stale invitations (to unclog things and keep them unclogged). Then pagination support and caching probably become unnecessary,. Although users with pending invitations who have not yet given up will have to re-invite, at least they will likely be successful at that time.

Update: Was late when i originally posted this. While lying in bed i wondered why instead you don't just accept all invitations. Each time you getRepoInvites, just accept them all. This will keep the queue low. And, if you just schedule this to happen say each minute, then you won't need folks to hit the endpoint to connect at all. I'm assuming the point of this was only to trigger the invite acceptance.

Answer 14 · 2018-12-30T07:25:07.000Z

@rliebling great analysis, I didn't even think about that. I'm more than happy to provide a PR for this.

Answer 15 · 2019-01-12T15:10:34.000Z

Hi all.

I had a go at implementing Staticman as a GitHub App, which should fix many of the issues people are seeing at the moment. Can I ask for some volunteers to help me test it? Here's how:

Remove staticmanapp as a collaborator
Go to https://github.com/apps/staticman-net and install the application on your repository
Submit a comment to the new v3 endpoint, using dev.staticman.net as the base URL – i.e. https://dev.staticman.net/v3/entry/github/[USERNAME]/[REPOSITORY]/[BRANCH]

Any help is much appreciated.

Answer 16 · 2019-01-12T20:06:53.000Z

Hi @eduardoboucas

Great to hear the news from you!

Not sure if i've just done something wrong but i'm getting 500 errors. I removed staticmanapp as a collaborator. I installed the staticman-net github app with access to my repo. And, i tried submitting a comment to https://dev.staticman.net/v2/entry/rliebling/my_blog/master/comments. After retrying a couple times i tried curl'ing the /v2/connect/rliebling/my_blog endpoint and also got a 500 response.

Note that i'm using Hugo with the engimo them which has staticman support built in, but i've never successfully used staticman before (as my invitation was "not found"). Also I'm testing using Hugo on my localhost - assume the post-id stuff wouldn't cause the problem, but i mention in case i'm wrong. The request that's failing is (as copied from chrome debugger, removing user agent and cookies:

curl -i  'https://dev.staticman.net/v2/entry/rliebling/my_blog/master/comments' -H 'authority: dev.staticman.net' -H 'cache-control: max-age=0' -H 'origin: http://localhost:1313' -H 'upgrade-insecure-requests: 1' -H 'content-type: application/x-www-form-urlencoded' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-US,en;q=0.9' --data 'options%5BpostId%5D=dc33308b4aef09b40e86a6783d501abd&options%5Bredirect%5D=http%3A%2F%2Flocalhost%3A1313%2Fposts%2Fmore_on_tech_debt%2F%23submission-success&options%5BredirectError%5D=http%3A%2F%2Flocalhost%3A1313%2Fposts%2Fmore_on_tech_debt%2F%23submission-failure&fields%5Bhoneypot%5D=&fields%5Bpermalink%5D=%2Fposts%2Fmore_on_tech_debt%2F&fields%5Bparent_id%5D=&fields%5Bcontent%5D=test+comment&fields%5Bauthor%5D=rich&fields%5Bemail%5D=rliebling%40gmail.com&fields%5Bsite%5D=https%3A%2F%2Fexample.com' --compressed

Answer 17 · 2019-01-12T20:31:18.000Z

Thanks for the feedback. I’ll try to debug it tonight using your sample request and will come back with my findings.

Thank you all for your patience.

Answer 18 · 2019-01-12T23:37:29.000Z

@rliebling Your site doesn't seem to be configured properly. I don't see a configuration file for Staticman on https://github.com/rliebling/my_blog.

Answer 19 · 2019-01-13T01:44:18.000Z

Ah! My bad! Had only configured locally and never pushed to GH as my invite had not been accepted. And, did this testing having forgotten all about that!

Sorry - I should have checked and figured this out myself.

I've fixed that now, however, and still getting 500 response. I don't want you having to go about debugging my config if you think that's likely the issue. I'll try looking at the code to understand better what it's doing. But, one quick thing to confirm: if i configure path: "data/comments/{options.postId}" inside staticman.yml should i need that directory/path to already exist?

Note: i've also enabled commenting on my live site now -- just not working yet (eg https://rich.liebling.us/posts/more_on_tech_debt/)

Answer 20 · 2019-01-13T16:10:45.000Z

@rliebling Your config is fine, it was an issue with an environment variable on the development instance. I've fixed it, tested again and it seems to be working.

You can see a submission here: rliebling/my_blog@e36206d

Answer 21 · 2019-01-13T16:15:09.000Z

@eduardoboucas Thanks so much for this project, moving it to a github app, and for your help here!

Answer 22 · 2019-01-13T21:38:56.000Z

Hi all.

I had a go at implementing Staticman as a GitHub App, which should fix many of the issues people are seeing at the moment. Can I ask for some volunteers to help me test it? Here's how:

Remove staticmanapp as a collaborator

Go to https://github.com/apps/staticman-net and install the application on your repository

Submit a comment as usual, but use https://dev.staticman.net instead of https://api.staticman.net as the base URL.

Any help is much appreciated.

Thanks @eduardoboucas. This seems to be working great for me on my project. https://github.com/simonarnell/GDPRDPIAT

Answer 23 · 2019-01-13T23:35:02.000Z

I've updated the comment above to point to the new v3 endpoint. The idea is that people will carry on using v1 or v2 endpoints if they're using the legacy staticmanapp authentication method, whilst people that have installed the new GitHub App will use the v3 endpoints.

Answer 24 · 2019-01-14T02:25:59.000Z

@eduardoboucas Tried this out and this works great. Thanks so much for making this change.

Answer 25 · 2019-01-14T03:14:05.000Z

Staticman app v3 works great at https://spinningnumbers.org/a/staticman.html <https://spinningnumbers.org/a/staticman.html> [Edit: Unfortunately I have to withdraw the 'working great'. Staticman app is delivering new comments via Staticman v3. The comments show up in _data/comments folder. But, github-pages is not triggering a rebuild of the affected article pages. In fact, I'm having trouble imagining how github-pages can tell which article needs regeneration, short of regenerating my whole site.] thanks so much [edit: still! ] for this valuable service. - Willy McAllister

Answer 26 · 2019-01-14T14:09:34.000Z

@eduardoboucas this is amazing!! 🎉

Works perfectly on my end, THANK YOU for everything! 😃

Answer 27 · 2019-01-14T19:55:12.000Z

Looks like this is working great for me. I am having a few issues with Mailgun, but I think that is on my side. Everything else looks good (reCaptcha, posts, etc.).

Edit:

Mailgun is working great now, was an issue on my end.

Answer 28 · 2019-01-16T19:21:17.000Z

When I submit a comment it shows up in _data/comments. However, github-pages/Jekyll does not trigger a rebuild of the parent article.

I'm using the App with the v3 endpoint, per Eduardo's edited instructions. I waited the suggested 10 minutes for github-pages to respond. If I make a trivial change to an unrelated article and push, the rebuild is triggered promptly and the comment appears.

My site is on github-pages.
https://spinningnumbers.org/a/staticman.html, https://github.com/willymcallister/willymcallister.github.io

Answer 29 · 2019-01-19T16:09:59.000Z

Hello. I'm attempting to use this GitHub application for a project I'm associated with and I'm running into some issues.

When using v3 of the API listed above, I'm receiving an INVALID_VERSION error. I've removed staticmanapp as a collaborator. I don't have admin access to the repo, so I'm checking with someone to ensure the application is pointing to that repo correctly. If this information is valid however, is there any other reason why one would receive a versioning error?

Answer 30 · 2019-01-19T16:16:55.000Z

I've authenticated the staticman app but I'm getting a too may requests error. Below is the cURL of hte request being sent by my form, any idea if i've got this set up incorrectly?

curl "https://api.staticman.net/v3/entry/github/adamebell/adamebell.github.io/master" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" -H "Accept-Language: en-US,en;q=0.5" --compressed -H "Referer: https://adamebell.com/Books-I-Read-In-2018-Pt-1/" -H "Content-Type: application/x-www-form-urlencoded" -H "DNT: 1" -H "Connection: keep-alive" -H "Cookie: __cfduid=df5ab0bcb2c608a9ab02df3ab3355c80f1547911561" -H "Upgrade-Insecure-Requests: 1" -H "Pragma: no-cache" -H "Cache-Control: no-cache" --data "options"%"5Bredirect"%"5D="%"2FBooks-I-Read-In-2018-Pt-1"%"2F&options"%"5Bslug"%"5D=Books-I-Read-In-2018-Pt-1&fields"%"5Bname"%"5D=adam&fields"%"5Bemail"%"5D=notme"%"40gmail.com&fields"%"5Bmessage"%"5D=test+test+test&action="

Answer 31 · 2019-01-19T16:51:09.000Z

@adamebell

Use dev.staticman.net as the API endpoint if you are using the app. It looks like you are still using api.staticman.net.

Answer 32 · 2019-01-19T16:56:54.000Z

The new functionality is still on the development instance (dev.staticman.net). I’ll release it to the main API sometime this week, now that it seems to work well for everyone.

Answer 33 · 2019-01-19T17:53:57.000Z

I've noticed others are using curl express the error they are receiving, so here is a request from my repo: curl -d "" -X POST https://dev.staticman.net/v3/entry/github/duellinksmeta/duellinksmeta/master {"success":false,"errorCode":"INVALID_VERSION"}

Answer 34 · 2019-01-19T18:39:42.000Z

Hi, I appreciate you're working hard on this and this may just be a user config error, but I just get a 500 error back with no error message other than {success:false}.
My repo is at:

https://github.com/hortigraph/hortigraph.github.io/

The request / response is:

Request URL: https://dev.staticman.net/v3/entry/github/hortigraph/hortigraph.github.io/master/comments
Request Method: POST
Status Code: 500
Remote Address: 104.27.141.78:443
Referrer Policy: no-referrer-when-downgrade

Response headers:

access-control-allow-headers: Origin, X-Requested-With, Content-Type, Accept
access-control-allow-origin: *
cf-ray: 49bb3c136f74bbd8-LHR
content-length: 17
content-type: application/json; charset=utf-8
date: Sat, 19 Jan 2019 18:06:21 GMT
etag: W/"11-UIVUdQWNarX1D9mk06okyEMbpS8"
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
status: 500
via: 1.1 vegur
x-powered-by: Express

Request Headers:

Provisional headers are shown
Accept: /
Content-Type: application/x-www-form-urlencoded
Origin: https://hortigraph.github.io
Referer: https://hortigraph.github.io/rhs/r-2101-2/
User-Agent: Mozilla/5.0 (X11; CrOS x86_64 11316.82.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.59 Safari/537.36

Form data:

fields[message]: comment
fields[name]: name
fields[email]: nigel@nigelwheeler.com
fields[url]:
options[slug]: r-2101-2
fields[hidden]:

/staticman.yml


comments:
  # (*) REQUIRED
  #
  # Names of the fields the form is allowed to submit. If a field that is
  # not here is part of the request, an error will be thrown.
  allowedFields: ["name", "email", "url", "message"]

  # (*) REQUIRED WHEN USING NOTIFICATIONS
  #
  # When allowedOrigins is defined, only requests sent from one of the domains
  # listed will be accepted. The origin is sent as part as the `options` object
  # (e.g. <input name="options[origin]" value="http://yourdomain.com/post1")
  # allowedOrigins: ["yourdomain.com"]

  # (*) REQUIRED
  #
  # Name of the branch being used. Must match the one sent in the URL of the
  # request.
  branch: "master"

  commitMessage: "New comment by {fields.name}"

  # (*) REQUIRED
  #
  # Destination path (filename) for the data files. Accepts placeholders.
  filename: "comment-{@timestamp}"

  # The format of the generated data files. Accepted values are "json", "yaml"
  # or "frontmatter"
  format: "yaml"

  # List of fields to be populated automatically by Staticman and included in
  # the data file. Keys are the name of the field. The value can be an object
  # with a `type` property, which configures the generated field, or any value
  # to be used directly (e.g. a string, number or array)
  generatedFields:
    date:
      type: "date"
      options:
        format: "iso8601" # "iso8601" (default), "timestamp-seconds", "timestamp-milliseconds"

  # Whether entries need to be approved before they are published to the main
  # branch. If set to `true`, a pull request will be created for your approval.
  # Otherwise, entries will be published to the main branch automatically.
  moderation: true

  # Akismet spam detection.
  # akismet:
  #   enabled: true
  #   author: "name"
  #   authorEmail: "email"
  #   authorUrl: "url"
  #   content: "message"
  #   type: "comment"

  # Name of the site. Used in notification emails.
  # name: "Your Site"

  # Notification settings. When enabled, users can choose to receive notifications
  # via email when someone adds a reply or a new comment. This requires an account
  # with Mailgun, which you can get for free at http://mailgun.com.
  # notifications:
    # Enable notifications
    # enabled: true

    # (!) ENCRYPTED
    #
    # Mailgun API key
    # apiKey: ""

    # (!) ENCRYPTED
    #
    # Mailgun domain (encrypted)
    # domain: ""

  # (*) REQUIRED
  #
  # Destination path (directory) for the data files. Accepts placeholders.
  path: "_data/comments/{options.slug}" # "_data/comments/{options.slug}" (default)

  # Names of required files. If any of these isn't in the request or is empty,
  # an error will be thrown.
  requiredFields: ["name", "email", "message"]

  # List of transformations to apply to any of the fields supplied. Keys are
  # the name of the field and values are possible transformation types.
  transforms:
    email: md5

  # reCaptcha
  # Register your domain at https://www.google.com/recaptcha/ and choose reCAPTCHA V2
  reCaptcha:
    enabled: false
    siteKey: #"6LdRBykTAAAAAFB46MnIu6ixuxwu9W1ihFF8G60Q"
    # Encrypt reCaptcha secret key using Staticman /encrypt endpoint
    # For more information, https://staticman.net/docs/encryption
    secret: #"PznnZGu3P6eTHRPLORniSq+J61YEf+A9zmColXDM5icqF49gbunH51B8+h+i2IvewpuxtA9TFoK68TuhUp/X3YKmmqhXasegHYabY50fqF9nJh9npWNhvITdkQHeaOqnFXUIwxfiEeUt49Yoa2waRR7a5LdRAP3SVM8hz0KIBT4="

Answer 35 · 2019-01-19T19:53:57.000Z

@eduardoboucas @robinmetral thanks for the quick reply, that was definitely an error on my part but now I'm getting the same "Invalid Version" error that @unpreceden7ed is getting

Answer 36 · 2019-01-19T19:55:07.000Z

My POST url above is suffixed with /comments
My understanding was that this is then used by staticman.yml to allow different configs, e.g. to /posts
If I post to the 'raw' URL https://dev.staticman.net/v3/entry/github/hortigraph/hortigraph.github.io/master
I also get an error response

{
    "success": false,
    "errorCode": "INVALID_VERSION"
}

UPDATE:

I removed the comments: key from staticman.yml: I still get the INVALID_VERSION Response.

If I then try posting with /comments appended to the URL, I get:

{
    "success": false,
    "rawError": {
        "_smErrorCode": "MISSING_CONFIG_BLOCK"
    },
    "errorCode": "MISSING_CONFIG_BLOCK"
}

So, it seems like the different configs are available (eg. to /posts /comments) : is the comments: block accepted as default by the API, even when just posting to the 'raw' URL?

Answer 37 · 2019-01-19T20:14:17.000Z

It is my understanding, and how I I have always used the API, that you have your username, then repo, then branch, and finally the directory where you want your comments placed. I have never used it with the raw branch.

In my brief testing it looks like the directory is required after the branch in the API URL.

Answer 38 · 2019-01-19T20:23:07.000Z

I took a look again and it looks like it is a change going from v2 to v3...

On the v2 without the optional PROPERTY Field after the branch it works fine. But, it throws an error on v3. Interesting.

Answer 39 · 2019-01-19T20:26:17.000Z

@dancwilliams thanks.
Does the path: in staticman.yml not specify the directory, and the final part of the url relate to a 'block' with that name, e.g. comments: in my file above?
It looks like the API is reading the config file somehow, rather than having auth problems or something.
The problem is when I post to https://dev.staticman.net/v3/entry/github/hortigraph/hortigraph.github.io/master/comments
I get no meaningful error (just success:false) nor have any way to debug this.

If I post to /_data I get 'Missing Config Block' again.

If I post to /_data/comments I just get a 404.

The INVALID_VERSION error only seems to crop up today. Could I please ask @robinmetral if your instances is still working as perfectly as 5 days ago? - It would be good to determine if this is a new issue or various people having a bad config moment...

Answer 40 · 2019-01-19T20:40:32.000Z

@cloudwheels

Yes, I misspoke in my comment above. My apologies. The property field refers to the block in your config file.

When I perform a POST to your API endpoint I get the following:

iPad:~# curl -d "" -X POST https://dev.staticman.net/v3/entry/github/hortigraph/hortigraph.github.io/master/comments
{"success":false,"errorCode":"MISSING_PARAMS","data":["fields"]}

This is “good” as it means it is looking for the required fields as defined in you config file.

It is the same thing I get when posting to mine, which I know is working:

iPad:~# curl -d "" -X POST https://dev.staticman.net/v3/entry/github/dancwilliams/networkhobo/master/comments             
{"success":false,"errorCode":"MISSING_PARAMS","data":["fields"]}

Answer 41 · 2019-01-19T20:45:03.000Z

I'm also getting MISSING_CONFIG_BLOCK now, and it seems to only happen if I have a validly formatted .yml file as staticman.yml in my directory (I had a junk file in there where I improperly re-downloaded the sample file and it threw a different error).

I'm trying to narrow down if the issues lie with the yml file itself or something with my call.

cURL:
curl "https://dev.staticman.net/v3/entry/github/adamebell/adamebell.github.io/master/data/" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" -H "Accept-Language: en-US,en;q=0.5" --compressed -H "Referer: https://adamebell.com/What-is-this/" -H "Content-Type: application/x-www-form-urlencoded" -H "DNT: 1" -H "Connection: keep-alive" -H "Cookie: __cfduid=df5ab0bcb2c608a9ab02df3ab3355c80f1547911561" -H "Upgrade-Insecure-Requests: 1" -H "Pragma: no-cache" -H "Cache-Control: no-cache" --data "options"%"5Bredirect"%"5D="%"2FWhat-is-this"%"2F&options"%"5Bslug"%"5D=What-is-this&fields"%"5Burl"%"5D="%"2FWhat-is-this"%"2F&fields"%"5Bpostname"%"5D+type"%"3D=What+is+this+going+to+be"%"3F&fields"%"5Bname"%"5D=Adam&fields"%"5Bemail"%"5D=notme"%"40gmail.com&fields"%"5Bmessage"%"5D=wooooooork&action="

Answer 42 · 2019-01-19T20:46:32.000Z

@dancwilliams thanks again: I noticed other folks with recent errors are mainly posting to an endpoint without a field.

Yes, I think the app is accessing the repo somehow.
It's when I post with fields I get the error. If I post with required fields missing, or fields that aren't specified in the accepted values, I don't get any specific error messages though, just success:false

Answer 43 · 2019-01-19T20:48:32.000Z

With the following comment between @dancwilliams and @cloudwheels, I was able to get the instance working.

As Dan mentioned, going from v2 to v3, the property parameter within the API is no longer optional (at least seems that way). This was discovered through cloud's comment. After adding that to my API and adjusting the staticman.yml file with this property, I was able to successfully submit a POST and received the expected file within the correct place (i.e. the path variable with in the YAML file).

Thanks again 👍

Answer 44 · 2019-01-19T20:51:35.000Z

@adamebell

The path field is missing from your config file. See my functional config file here

I am looking at your config here. You have the entire path under your filename field. It just needs to be broken into the two separate fields it seems.

Answer 45 · 2019-01-19T20:57:45.000Z

@unpreceden7ed I'm very happy for you 👍

Now to work out what my problem is, cos I think the config is correct, I'm posting to a url + FIELD with a matching config block, and it looks like it's not a problem with the API as others are working.

This is Jekyll on GH pages, which requires a _config.yml, which is where staticman V1 picked up it's config: maybe some confusion there...dunno.

Oh for a meaningful error message @eduardoboucas please ???

Looking forward to that moment of clarity....

Answer 46 · 2019-01-19T21:06:09.000Z

@cloudwheels

I agree, your config file does look good. I am looking around to see if anything else jumps out at me.

Answer 47 · 2019-01-19T21:10:13.000Z

@dancwilliams Awesome, it's working now. For anyone else still struggling, I'll post what I have.

I took Dan's config file and just modified the filenames and other relevant fields, and then used the following HTML for the comment form (relies on materialize css, is currently ugly but functional). I'm using jekyll, so I have this in an _includes file called statcman.html and then in my post layout I just include it at the bottom. pasting in code mode isn't working so I've attached the file)
statcman.txt

Answer 48 · 2019-01-19T21:13:26.000Z

@dancwilliams Much appreciated!

FYI this started with a 'client-side' issue with the jekyll template which is here: mmistakes/minimal-mistakes#2034

But I'm testing with postman: I don't think it's that.

I'll have a go at replacing the config with yours...

Answer 49 · 2019-01-19T21:28:52.000Z

@cloudwheels

I'm running curl against your repository and I'm noticing something interesting. You have three required fields in your YAML file. However, if I fulfill the requirements of one and not the others, I'm not receiving an error message that states something to fact of 'x' amount of fields are required. I know this message can be generated, as I received this when I was testing with my repository and missed a few fields. This makes me think there may be an error with parsing your YAML file, though it may not be within the file itself.

Do you have the bot configured correctly against your repository and do you have the staticmanapp removed as a contributor? The missing error seems to stem between the parsing of your YAML file, though it's not certain if the cause is the file itself or the application.

Here is my curl output: curl -d "fields[email]=foobar@gmail.com" -X POST https://dev.staticman.net/v3/entry/github/hortigraph/hortigraph.github.io/master/comments

Answer 50 · 2019-01-19T21:52:07.000Z

@unpreceden7ed Thanks for the moral support.
Agree, there's some problem parsing the yaml.
I've checked, removed and re-added theapp, and added to my personal repos in case it was anything to do with the repo being under an organisation. In any event, I appear to be talking to the app, so it's there I think, just having problems, which pretty much must be to do with the yaml, I just can't figure it out!

Answer 51 · 2019-01-19T22:11:52.000Z

@dancwilliams @unpreceden7ed and all.
I am at peace.

In the chain of things this MUST boil down to the content of my staticman.yml (unless the test after reinstalling the app didn't reflect that change)
I replaced with the standard sample: https://github.com/eduardoboucas/staticman/edit/master/staticman.sample.yml
changing only the sitename.

So, thanks for this great advancement @eduardoboucas : 3 more confirmed happy customers for v3 within the last hour. Perhaps some clarification on docs & improved error messages needed?

My issues are all user support, so no follow up required thanks, but I will report this back to mmistakes/minimal-mistakes#2034 as referncved earlier which may need changes to the bundled staticman.yml

Thanks again 👍

Answer 52 · 2019-01-19T22:40:19.000Z

@cloudwheels

I wanted to test a theory similar to yours and I saw different results. I decided to create a repository from scratch, take the sample YAML file from Staticman, not edit it (it was renamed for configuration purposes, but the file itself was not edited) and use curl against it. My results proved successful.

curl -d "fields[email]=foobar@gmail.com" -X POST https://dev.staticman.net/v3/entry/github/unpreceden7ed/staticman-test/master/comments {"success":false,"data":["name","message"],"rawError":{"_smErrorCode":"MISSING_REQUIRED_FIELDS","data":["name","message"]},"errorCode":"MISSING_REQUIRED_FIELDS"}
In above response, I only gave the parameters an email and received the message I spoke about earlier with the missing parameters.

curl -d "fields[email]=foobar@gmail.com&fields[name]=John Doe&fields[message]=A message to you" -X POST https://dev.staticman.net/v3/entry/github/unpreceden7ed/staticman-test/master/comments {"success":true,"fields":{"email":"726f7c3769f32d3da4656ea906d02adb","name":"John Doe","message":"A message to you","date":1547936943}}
In this response, I gave it the required parameters and received successful response.

The only step left would be actually implement a form within my repository (https://github.com/unpreceden7ed/staticman-test) and use it against the staticman bot itself. The fact that I'm not able to reproduce at least this much with your repository seems odd, but I wouldn't know where else to tell you begin searching at this point. 🤔

Answer 53 · 2019-01-25T08:03:51.000Z

I can confirm that V3 works well for me.

Answer 54 · 2019-01-30T15:58:41.000Z

I just got it working on my website. Thanks all of you for being so awesome and wholesome!

Answer 55 · 2019-02-04T22:40:28.000Z

Staticman App requires moderated comments. My site (https://spinningnumbers.org/a/staticman.html) was not automatically rebuilding with unmoderated comments. Here is the full diagnosis from GitHub Support...

It looks like your site builds are failing with the following error:
Validation failed: User must be a human
To avoid abuse, GitHub Pages sites can't be automatically rebuilt when a GitHub App pushes to a repository. As StaticMan is a GitHub App it isn't able to automatically trigger a rebuild of your site.

I notice that this application has a moderation-mode, that will create pull requests for you to merge manually. Enabling this mode will let this app create pull requests, and will then trigger a build when you manually merge them.

I'd recommend reaching out to the project maintainer for more help on this, as this will need to be something that is changed on their side.

Answer 56 · 2019-02-12T06:21:17.000Z

I use a botpot field that is intended for a user to leave blank. However, these blank fields are appearing in the comment files added by Staticman. Is it possible to strip empty fields from the comment files?

Maybe a feature request to strip blank fields that are not in allowedFields would be more appropriate.

Answer 57 · 2019-02-12T07:46:22.000Z

@zinefer @willymcallister maybe you should post new issues for this!

Your problem will have much more visibility than as a message on a thread with 50+ comments, and it'll let us keep issues organized 🙂

Closing this for now, feel free to reopen if relevant 👋

Answer 58 · 2019-02-25T21:25:18.000Z

Testing the v3 api I only get this error:
curl -d "fields[name]=dom&fields[email]=truc@truc.net&fields[message]=test" -X POST https://dev.staticman.net/v3/entry/github/domguard/onlefay-comm/master/comments
{"success":false}
Does my github repository need to be public for the app to be able to push comments ?

Answer 59 · 2019-03-10T07:58:02.000Z

I started to add Staticman to my project today and ran into troubles with V2, presumably because of the rate limiting. Decided to give V3 a go and it just worked great first time 👍

Might be worth considering updating the docs on the website because V2 was pretty much unusable for me.

Answer 60 · 2019-03-10T08:17:01.000Z

Agreed @MyGuySi, I'm sure @eduardoboucas would be open to a PR 🙂

Answer 61 · 2019-03-16T23:27:35.000Z

Hi all.

I had a go at implementing Staticman as a GitHub App, which should fix many of the issues people are seeing at the moment. Can I ask for some volunteers to help me test it? Here's how:

Remove staticmanapp as a collaborator

Go to https://github.com/apps/staticman-net and install the application on your repository

Submit a comment to the new v3 endpoint, using dev.staticman.net as the base URL – i.e. https://dev.staticman.net/v3/entry/github/[USERNAME]/[REPOSITORY]/[BRANCH]

Any help is much appreciated.

I followed the instructions except that I do not understand step 3. I typed in the URL and got "Cannot GET /v3/entry/github/valzi/light-transmuter/master". Where am I supposed to go to submit the comment?

Answer 62 · 2019-03-17T08:37:44.000Z

@valzi this URL is the one that your comment form has to POST to. The Get started doc is a good place to start building with Staticman 🙂

Answer 63 · 2019-03-17T18:49:31.000Z

Thanks, I get it now.

Answer 64 · 2019-05-11T23:23:05.000Z

I don't have reCAPTCHA set but keep getting this error:

{"success":false,"message":"Missing reCAPTCHA API credentials","rawError":{"_smErrorCode":"RECAPTCHA_MISSING_CREDENTIALS"},"errorCode":"RECAPTCHA_MISSING_CREDENTIALS"}

Answer 65 · 2019-06-16T17:19:35.000Z

I think the v3 Staticman app may be broken, but I'm not certain. See #294 for more details.

Answer 66 · 2019-06-16T17:49:23.000Z

Thanks for this update. I'm sure the app just scaled past what his service could handle. I'll probably look for another solution (like Netlify).

…

On Sun, Jun 16, 2019, 1:19 PM Daniel Schroeder ***@***.***> wrote: I think the v3 Staticman app may be broken, but I'm not certain. See #294 <#294> for more details. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#243?email_source=notifications&email_token=ACWVJSI7XOW6S6IXM4LQZBDP2ZY2VA5CNFSM4GJUOO7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXZRLOY#issuecomment-502470075>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACWVJSLZAYHYQ5NCPNP7LXTP2ZY2VANCNFSM4GJUOO7A> .

Answer 67 · 2019-07-05T09:55:09.000Z

I don't have reCAPTCHA set but keep getting this error:

{"success":false,"message":"Missing reCAPTCHA API credentials","rawError":{"_smErrorCode":"RECAPTCHA_MISSING_CREDENTIALS"},"errorCode":"RECAPTCHA_MISSING_CREDENTIALS"}

@pacollins That's a duplicate of #223.

Answer 68 · 2019-07-06T11:41:53.000Z

@cloudwheels

Hi, I appreciate you're working hard on this and this may just be a user config error, but I just get a 500 error back with no error message other than {success:false}.
My repo is at:

https://github.com/hortigraph/hortigraph.github.io/

The request / response is:

Request URL: https://dev.staticman.net/v3/entry/github/hortigraph/hortigraph.github.io/master/comments
Request Method: POST
Status Code: 500
Remote Address: 104.27.141.78:443
Referrer Policy: no-referrer-when-downgrade

Response headers:

access-control-allow-headers: Origin, X-Requested-With, Content-Type, Accept
access-control-allow-origin: *
cf-ray: 49bb3c136f74bbd8-LHR
content-length: 17
content-type: application/json; charset=utf-8
date: Sat, 19 Jan 2019 18:06:21 GMT
etag: W/"11-UIVUdQWNarX1D9mk06okyEMbpS8"
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
status: 500
via: 1.1 vegur
x-powered-by: Express

Request Headers:

Provisional headers are shown
Accept: /
Content-Type: application/x-www-form-urlencoded
Origin: https://hortigraph.github.io
Referer: https://hortigraph.github.io/rhs/r-2101-2/
User-Agent: Mozilla/5.0 (X11; CrOS x86_64 11316.82.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.59 Safari/537.36

Form data:

fields[message]: comment
fields[name]: name
fields[email]: nigel@nigelwheeler.com
fields[url]:
options[slug]: r-2101-2
fields[hidden]:

/staticman.yml


comments:
  # (*) REQUIRED
  #
  # Names of the fields the form is allowed to submit. If a field that is
  # not here is part of the request, an error will be thrown.
  allowedFields: ["name", "email", "url", "message"]

  # (*) REQUIRED WHEN USING NOTIFICATIONS
  #
  # When allowedOrigins is defined, only requests sent from one of the domains
  # listed will be accepted. The origin is sent as part as the `options` object
  # (e.g. <input name="options[origin]" value="http://yourdomain.com/post1")
  # allowedOrigins: ["yourdomain.com"]

  # (*) REQUIRED
  #
  # Name of the branch being used. Must match the one sent in the URL of the
  # request.
  branch: "master"

  commitMessage: "New comment by {fields.name}"

  # (*) REQUIRED
  #
  # Destination path (filename) for the data files. Accepts placeholders.
  filename: "comment-{@timestamp}"

  # The format of the generated data files. Accepted values are "json", "yaml"
  # or "frontmatter"
  format: "yaml"

  # List of fields to be populated automatically by Staticman and included in
  # the data file. Keys are the name of the field. The value can be an object
  # with a `type` property, which configures the generated field, or any value
  # to be used directly (e.g. a string, number or array)
  generatedFields:
    date:
      type: "date"
      options:
        format: "iso8601" # "iso8601" (default), "timestamp-seconds", "timestamp-milliseconds"

  # Whether entries need to be approved before they are published to the main
  # branch. If set to `true`, a pull request will be created for your approval.
  # Otherwise, entries will be published to the main branch automatically.
  moderation: true

  # Akismet spam detection.
  # akismet:
  #   enabled: true
  #   author: "name"
  #   authorEmail: "email"
  #   authorUrl: "url"
  #   content: "message"
  #   type: "comment"

  # Name of the site. Used in notification emails.
  # name: "Your Site"

  # Notification settings. When enabled, users can choose to receive notifications
  # via email when someone adds a reply or a new comment. This requires an account
  # with Mailgun, which you can get for free at http://mailgun.com.
  # notifications:
    # Enable notifications
    # enabled: true

    # (!) ENCRYPTED
    #
    # Mailgun API key
    # apiKey: ""

    # (!) ENCRYPTED
    #
    # Mailgun domain (encrypted)
    # domain: ""

  # (*) REQUIRED
  #
  # Destination path (directory) for the data files. Accepts placeholders.
  path: "_data/comments/{options.slug}" # "_data/comments/{options.slug}" (default)

  # Names of required files. If any of these isn't in the request or is empty,
  # an error will be thrown.
  requiredFields: ["name", "email", "message"]

  # List of transformations to apply to any of the fields supplied. Keys are
  # the name of the field and values are possible transformation types.
  transforms:
    email: md5

  # reCaptcha
  # Register your domain at https://www.google.com/recaptcha/ and choose reCAPTCHA V2
  reCaptcha:
    enabled: false
    siteKey: #"6LdRBykTAAAAAFB46MnIu6ixuxwu9W1ihFF8G60Q"
    # Encrypt reCaptcha secret key using Staticman /encrypt endpoint
    # For more information, https://staticman.net/docs/encryption
    secret: #"PznnZGu3P6eTHRPLORniSq+J61YEf+A9zmColXDM5icqF49gbunH51B8+h+i2IvewpuxtA9TFoK68TuhUp/X3YKmmqhXasegHYabY50fqF9nJh9npWNhvITdkQHeaOqnFXUIwxfiEeUt49Yoa2waRR7a5LdRAP3SVM8hz0KIBT4="

In the reCATPCHA section of your Staticman config, even if enabled: false, if you didn't comment out the unused siteKey and secret, that can contribute to an error. Here, you've passed the value null to these two attributes.

To avoid that you may either comment out the whole line (e.g. https://github.com/VincentTam/BJPubTest1)

  # reCaptcha
  # Register your domain at https://www.google.com/recaptcha/ and choose reCAPTCHA V2
  reCaptcha:
    enabled: false
    #siteKey: "6LdRBykTAAAAAFB46MnIu6ixuxwu9W1ihFF8G60Q"
    # Encrypt reCaptcha secret key using Staticman /encrypt endpoint
    # For more information, https://staticman.net/docs/encryption
    #secret: "PznnZGu3P6eTHRPLORniSq+J61YEf+A9zmColXDM5icqF49gbunH51B8+h+i2IvewpuxtA9TFoK68TuhUp/X3YKmmqhXasegHYabY50fqF9nJh9npWNhvITdkQHeaOqnFXUIwxfiEeUt49Yoa2waRR7a5LdRAP3SVM8hz0KIBT4="

or use an empty string "" instead of null. (e.g. Hugo Future Imperfect Slim)

  # reCaptcha
  # Register your domain at https://www.google.com/recaptcha/ and choose reCAPTCHA V2
  reCaptcha:
    enabled: false
    siteKey: ""
    # Encrypt reCaptcha secret key using Staticman /encrypt endpoint
    # For more information, https://staticman.net/docs/encryption
    secret: ""

Reference:

My experiement with Staticman's reCAPTCHA support in daattali/beautiful-jekyll#514 (comment)
My blog post about different types of Staticman errors