zipmark/rspec_api_documentation

Documentation no longer include JSON response body when using with Rack >= 2.1.0

sikachu opened this issue · 16 comments

Hello,

This issue is pretty much for reporting the incompatibility to the gem author, and hopefully it will help anyone who runs into this problem to understand what's going on.

Basically, after we upgrade our dependencies to use Rack 2.1.1, we noticed that our generated documentation no longer show JSON response but instead showing [binary data] instead.

Digging in further, we found out that in rack/rack@8c62821, especially this change, MockResponse#body now creates a buffer and use << to join the content together. However, on line 195, the author uses String.new without specifying the encoding, resulting in Ruby creating a new String with ASCII-8BIT encoding by default.

As it turns out, rspec_api_documentation relies on string encoding to determine if it should include the response body in the documentation or not:

if response_body.encoding == Encoding::ASCII_8BIT
"[binary data]"
else
formatter = RspecApiDocumentation.configuration.response_body_formatter
return formatter.call(response_content_type, response_body)
end

Hence, the change in Rack broke this conditional.

I've reported this issue to Rack in rack/rack#1486, and hopefully we can solve this soon.

The solution right now for us is to lock Rack to ~> 2.0.8 for now.

Thank you very much.

Any other workaround?

I ran into the same issue recently. I don't really like this solution, but I fixed it using mokey patching:
config/initializers/rspec_api_documentation.rb

module RspecApiDocumentation
  class RackTestClient < ClientBase
    def response_body
      last_response.body.encode("utf-8")
    end
  end
end

Indeed, it seems to be caused by an encoding issue where utf-8 become ascii-8bit

This seems to also be the result after Rails 6 upgrade.

Just created a new rails 6 app and had this problem, Tao's response above fixed it (#456 (comment))

Tao-Galasse's solution worked but I found that I needed to stick it right before the RspecApiDocumentation.configure block instead of the initializers directory.

There's an additional issue with endpoints that use send_data:

     Failure/Error: last_response.body.encode("utf-8")
     
     Encoding::UndefinedConversionError:
       "\xD3" from ASCII-8BIT to UTF-8
     # ./config/initializers/rspec_api_documentation.rb:6:in `encode'
     # ./config/initializers/rspec_api_documentation.rb:6:in `response_body'

I've tweaked Tao's fix, it ain't perfect but it works for my projects:

module RspecApiDocumentation
  class RackTestClient < ClientBase
    def response_body
      if last_response.headers["Content-Type"].include?("json")
        last_response.body.encode("utf-8")
      else
        "[binary data]"
      end
    end
  end
end

On my side, I fixed it by two mechanisms :

  1. @skibox's @Tao-Galasse solution remade mine
  2. Use response_body_formatter like it is intended to filter the different kind of output and make it OK for apitome

Resulting in this spec/support/rspec_api_documentation.rb file :

# frozen_string_literal: true

# Fix a bug that generate erroneous "[binary data]" not yet fixed on rspec_api_documentation:6.1.0
module RspecApiDocumentation
  class RackTestClient < ClientBase
    def response_body
      body = last_response.body
      if body.empty? || last_response.headers['Content-Type'].include?('json')
        body.encode('utf-8')
      else
        '"[binary data]"'
      end
    rescue Encoding::UndefinedConversionError
      '"[binary data]"'
    end
  end
end

# configure rspec_api_documentation
RspecApiDocumentation.configure do |config|
# ..... some unrelated config here  

  # Change how the response body is formatted by default
  # Is proc that will be called with the response_content_type & response_body
  # by default response_content_type of `application/json` are pretty formated.
  config.response_body_formatter = lambda do |response_content_type, response_body|
    if response_content_type.include?('application/json')
      JSON.parse(response_body)
      return response_body
    elsif response_content_type.include?('text') || response_content_type.include?('txt')
      # quote it for JSON Parser in documentation reader like APITOME
      return "\"#{response_body}\""
    else
      return '"[binary data]"'
    end
  rescue JSON::ParseError
    '"[binary data]"'
  end

# ... Other unrelated config
end

Just to make sure people notice, this issue would be fixed by #458 🙇‍♂️

For anyone following this, Ive merged #458 please let me know if this resolves the issue for you. cc @artofhuman

Uysim commented

Hope this can be fixed. We need to work with Rails 6

@Uysim please test and let everyone know

Doesn't work with application/vnd.api+json content type. Probably it's better to use include?('json')?

@incubus Note the change in #458 does not, in itself alter the gem's behavior.
If you use the default response_body_formatter, then you will experience the same problem as before.

To solve your problem, please:

  1. Make sure you use the master branch (at least until there's a new release).
  2. Define a custom response_body_formatter in your configuration, like so:
RspecApiDocumentation.configure do |config|
  config.response_body_formatter = 
    Proc.new do |content_type, response_body|
      if content_type =~ /application\/.*json/
        JSON.pretty_generate(JSON.parse(response_body))
      else
        response_body
      end
    end
end

Note that the above, compared with the new default (see below) introduced in #458 would stop filtering out potentially binary data.

elsif content_type =~ /application\/.*json/

As the test response_body.encoding == Encoding::ASCII_8BIT has become unreliable to filter binary data, it will be under your responsibility to figure out, in your context, what you can check to determine whether you want to display a given response or not.

Here's a more complex example:

RspecApiDocumentation.configure do |config|
  config.response_body_formatter =
    Proc.new do |content_type, response_body|
      # http://www.libpng.org/pub/png/spec/1.2/PNG-Rationale.html#R.PNG-file-signature
      if response_body[0,8] == "\x89PNG\r\n\u001A\n"
        "<img src=\"data:image/png;base64,#{Base64.strict_encode64(response_body)}\" />"
      elsif content_type =~ /application\/.*json/
        JSON.pretty_generate(JSON.parse(response_body))
      elsif response_body.encoding == Encoding::ASCII_8BIT # note 1
        "[binary?]" # note 2
      else
        response_body
      end
    end
end

Notes:

  1. This is the old check, still unreliable, but because it happens after checking for JSON or PNG, then you would be able to display those two properly. You might lose the display of very plain text (the else part) if Rack returns everything as ASCII_8BIT (same problem as before).

  2. To alleviate the above, you might want to return a better string than just [binary?] if you aren't sure your test is accurate. You could for example return something like this:

    <details><summary>[binary?]</summary>
      #{response_body}
    </details>

This would look like this:

[binary?] ˇÿˇ‡��JFIF�����`�`��ˇ·�ÄExif��MM�*�����������������������������J�����������R�(����������ái���������Z�������`�������`������†�����������†���������������ˇ· !http://ns.adobe.com/xap/1.0/� �ˇÌ�8Photoshop 3.0�8BIM��������8BIM�%������‘�åŸè�≤�ÈÄ òϯB~ˇ‚�ËICC_PROFILE������ÿappl� ��mntrRGB XYZ �Ÿ����������acspAPPL����appl������������������ˆ÷������”-appl������������������������������������������������desc�������odscm���x���úcprt�������8wtpt���L����rXYZ���`����gXYZ���t����bXYZ���à����rTRC���ú����chad���¨���,bTRC���ú����gTRC���ú����desc��������Generic RGB Profile������������Generic RGB Profile��������������������������������������������������mluc����������� skSK���(���ÑdaDK���.���¨caES���$���⁄viVN���$���˛ptBR���&���"ukUA���*���HfrFU���(���rhuHU���(���özhTW�������¬nbNO���&���ÿcsCZ���"���˛heIL������� itIT���(���>roRO���$���fdeDE���,���äkoKR�������∂svSE���&���ÿzhCN�������ÃjaJP�������‚elGR���"���¸ptPO���&����nlNL���(���DesES���&����thTH���$���ltrTR���"���êfiFI���(���≤hrHR���(���⁄plPL���,����ruRU���"���.arEG���&���PenUS���&���v�V�a�e�o�b�e�c�n�˝� �R�G�B� �p�r�o�f�i�l�G�e�n�e�r�e�l� �R�G�B�-�b�e�s�k�r�i�v�e�l�s�e�P�e�r�f�i�l� �R�G�B� �g�e�n�Ë�r�i�c�C�•�u� �h�Ï�n�h� �R�G�B� �C�h�u�n�g�P�e�r�f�i�l� �R�G�B� �G�e�n�È�r�i�c�o���0�3�0�;�L�=�8�9� �?�@�>�D�0�9�;� �R�G�B�P�r�o�f�i�l� �g�È�n�È�r�i�q�u�e� �R�V�B�¡�l�t�a�l�·�n�o�s� �R�G�B� �p�r�o�f�i�lê�u(� �R�G�B� Çr_icœè�G�e�n�e�r�i�s�k� �R�G�B�-�p�r�o�f�i�l�O�b�e�c�n�˝� �R�G�B� �p�r�o�f�i�l�‰�Ë�’�‰�Ÿ�‹� �R�G�B� �€�‹�‹�Ÿ�P�r�o�f�i�l�o� �R�G�B� �g�e�n�e�r�i�c�o�P�r�o�f�i�l� �R�G�B� �g�e�n�e�r�i�c�A�l�l�g�e�m�e�i�n�e�s� �R�G�B�-�P�r�o�f�i�l«|º�� �R�G�B� ’�∏\” «|fnê�� �R�G�B� cœèeáNˆN�Ç,� �R�G�B� 0◊0Ì0’0°0§0Î�ì�µ�Ω�π�∫�Ã� �¿�¡�ø�∆�Ø�ª� �R�G�B�P�e�r�f�i�l� �R�G�B� �g�e�n�È�r�i�c�o�A�l�g�e�m�e�e�n� �R�G�B�-�p�r�o�f�i�e�l�B���#�D���%�L� �R�G�B� ���1�H�'�D���G�e�n�e�l� �R�G�B� �P�r�o�f�i�l�i�Y�l�e�i�n�e�n� �R�G�B�-�p�r�o�f�i�i�l�i�G�e�n�e�r�i� �k�i� �R�G�B� �p�r�o�f�i�l�U�n�i�w�e�r�s�a�l�n�y� �p�r�o�f�i�l� �R�G�B���1�I�8�9� �?�@�>�D�8�;�L� �R�G�B�E�D�A� �*�9�1�J�A� �R�G�B� �'�D�9�'�E�G�e�n�e�r�i�c� �R�G�B� �P�r�o�f�i�l�etext����Copyright 2007 Apple Inc., all rights reserved.�XYZ ������ÛR�������œXYZ ������tM��=Ó���–XYZ ������Zu��¨s���4XYZ ������(����ü��∏6curv���������Õ��sf32������ B���fiˇˇÛ&���í��˝ëˇˇ˚¢ˇˇ˝£���‹��¿lˇ¿���������"�������ˇƒ���������������������������� �ˇƒ�µ����������������}��������!1A��Qa�"q�2Åë°�#B±¡�R—$3brÇ �����%&'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyzÉÑÖÜáàâäíìîïñóòôö¢£§•¶ß®©™≤≥¥µ∂∑∏π∫¬√ƒ≈∆«»… “”‘’÷◊ÿŸ⁄·‚„‰ÂÊÁËÈÍÒÚÛÙıˆ˜¯˘˙ˇƒ���������������������������� �ˇƒ�µ����������������w�������!1��AQ�aq�"2Å��Bë°±¡ #3R�br— �$4·%Ò����&'()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyzÇÉÑÖÜáàâäíìîïñóòôö¢£§•¶ß®©™≤≥¥µ∂∑∏π∫¬√ƒ≈∆«»… “”‘’÷◊ÿŸ⁄‚„‰ÂÊÁËÈÍÚÛÙıˆ˜¯˘˙ˇ€�C�������0��0D000D\DDDD\t\\\\\tåttttttåååååååå®®®®®®ƒƒƒƒƒ‹‹‹‹‹‹‹‹‹‹ˇ€�C�"$$848`44`ÊúÄúÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊˇ›����ˇ⁄� ��������?�È(¢ä�ˇŸ

@davidstosik it works, thanks!

I ran into the same issue recently. I don't really like this solution, but I fixed it using mokey patching:
config/initializers/rspec_api_documentation.rb

module RspecApiDocumentation
  class RackTestClient < ClientBase
    def response_body
      last_response.body.encode("utf-8")
    end
  end
end

Indeed, it seems to be caused by an encoding issue where utf-8 become ascii-8bit

It worked for me. Thank you so much

frozen_string_literal: true

Fix a bug that generate erroneous "[binary data]" not yet fixed on rspec_api_documentation:6.1.0

module RspecApiDocumentation
class RackTestClient < ClientBase
def response_body
body = last_response.body
if body.empty? || last_response.headers['Content-Type'].include?('json')
body.encode('utf-8')
else
'"[binary data]"'
end
rescue Encoding::UndefinedConversionError
'"[binary data]"'
end
end
end

I did a slight variation on this one so I could get the responses to show up as pretty printed:

RspecApiDocumentation.configure do |config|
  config.response_body_formatter = lambda do |response_content_type, response_body|
    if response_content_type.include?('application/json')
      return JSON.pretty_generate(JSON.parse(response_body))
    elsif response_content_type.include?('text') || response_content_type.include?('txt')
      # quote it for JSON Parser in documentation reader like APITOME
      return "\"#{response_body}\""
    else
      return '"[binary data]"'
    end
  rescue JSON::ParseError
    '"[binary data]"'
  end
end