magento/magento2

X-Magento-Tags header too large

katef opened this issue · 72 comments

katef commented

Preconditions

I've been reading about people using Magento2 and the size of the X-Magento-Tags header.

Steps to reproduce

For example http://www.maxbucknell.com/blog/2016/2/10/troubleshooting-varnish-with-magento-2 writes:

The rub here is that Magento adds a tag for every product in a category. And the tag is of the form "catalog_product_{{PRODUCT_ID}}".

Actual result

That's an integer and 20 bytes of string per product, which gives some users headers several kB in size, when many products are in one category. In that case, the author describes hitting Varnish's limit at 400 products.

Expected result

I work for a CDN where some of our users use Magento, and we cache their content with Varnish. Increasing Varnish's http_resp_hdr_len size may suit that one person's situation, but it is infeasible for us, because it would affect all of our users.

Proposed solution

I have a suggestion to offer, for how to use header space more efficiently:

Each product ID is an integer. The X-Magento-Tags header describes the presence or absence of products only. That information can be done with a single bit per product. The index of the bit would give the product ID.

That's usually known as a bit vector, or a bitfield, or a bitmap. My suggestion is to switch the header to use a bitmap, encoded in hex. That'd take one bit per product, instead of over 20 bytes.

More on bitmaps: https://en.wikipedia.org/wiki/Bit_array

stamster commented

Excellent point!
But I'm afraid such change is not going to happen anytime soon, maybe for Magento 3.0... and this is only one of the topics out there to discuss when it comes to optimization.

  • 👎4
katef commented

Thank you - Although I'd request that you consider this a blocking item, rather than an optimisation, because this header is so large that it prevents certain users from using Varnish.

  • 👍11
melnikovi commented

@katef thank you for reporting this issue, internal ticket created MAGETWO-57851

  • ❤️3
stamster commented

@katef : I'm not representative of Magento Commerce Inc. ... :) I completely agree with you, of course it is blocking for certain use cases out there.

pynej commented

Some more detail:
The catalog_category_product_ and catalog_category_ could easily be changed to 2 characters that would still be unique and cut down the length of this header by ~80%.

Of critical issue though is that in Magento 2.1 all versions of a configured product are incorrectly added to this header field. In 2.0 only the parent item was added. As a result the size of the field grows exponentially and makes varnish unusable. I am trying to find the code in question that is incorrectly adding all versions so I can comment it out and restore 2.0 functionality and and have an actual working site.

  • 👍3
pynej commented

This workaround fixes but wastes lots or memory.

  • 👍1
pynej commented

I did this workaround to get my site working again. Basically disabling the addition of the child products of configurable items to the tag. There are cons't defined that specify the catalog_product_ tag, so that should be pretty straightforward to shorten.

--- vendor/magento/module-configurable-product/Plugin/Model/Product.php.orig    2016-09-07 14:10:22.201010000 -0400
+++ vendor/magento/module-configurable-product/Plugin/Model/Product.php 2016-09-07 14:08:17.107250000 -0400
@@ -26,9 +26,9 @@
         /** @var Configurable $productType */
         $productType = $product->getTypeInstance();
         if ($productType instanceof Configurable) {
-            foreach ($productType->getChildrenIds($product->getId())[0] as $productId) {
-                $result[] = \Magento\Catalog\Model\Product::CACHE_TAG . '_' . $productId;
-            }
+            #foreach ($productType->getChildrenIds($product->getId())[0] as $productId) {
+            #    $result[] = \Magento\Catalog\Model\Product::CACHE_TAG . '_' . $productId;
+            #}
         }
         return $result;
     }
  • 👍4
pboisvert commented

@choukalos can you ensure this item gets the magetwo ticket? pulled the improvement label

choukalos commented

Tracking internally - MAGETWO-58265 ; Thanks @pynej for diagnosing and the code diff.

veloraven commented

Internal ticket for 2.1 - MAGETWO-58362

choukalos commented

Closing; this was resolved in 2.1.3

katef commented

@choukalos What was the resolution?

  • 👍1
djfordz commented

I'm still having issue with this in version 2.1.3

Error 503 Backend fetch failed

With Category Pages that have tons of categories.

dcabrejas commented

@choukalos I think this ticket should be reopen, I have encountered this issue on a Magento site running on 2.1.5. Category pages containing a large number of products return a X-Magento-Tag header which is bigger than the size allowed by the HTTP specification.

We have implemented this workaround for now http://devdocs.magento.com/guides/v2.0/config-guide/varnish/tshoot-varnish-503.html

But this is not a solution, just a workaround because the http header size is still larger than the limit set by the HTTp specification.

  • 👍4
redelschaap commented

Any update on this? This is still present in 2.1.7.

  • 👍5
tedsalmon commented

As previously stated, this is still an issue in 2.1.8. I have implemented a Plugin around Magento\Catalog\Model\Product like so:

<?php

namespace YourCompany\YourModule\Plugin;

class Product{

    protected $_request;

    public function __construct(\Magento\Framework\App\Request\Http $request){
       $this->_request = $request;
    }

    /*
     * Return no simple product tags when returning category pages
     */
    public function afterGetIdentities(\Magento\Catalog\Model\Product $subject, $tags){
        if(!$this->_request){
            return $tags;
        }
        $r = $this->_request;
        $fullRoute = sprintf(
            '%s_%s_%s', $r->getRouteName(), $r->getControllerName(), $r->getActionName()
        );
        if($fullRoute === 'catalog_category_view'){
            return [];
        }else{
            return $tags;
        }
    }
}

?>

I very much realize that this is a hack, but we simply have too many products (over 64kB of tags) for the tag shortening to work. This will still show any parent product tags, FWIW.

  • 👍5
  • 👎1
satishgadhave commented

Can you pls provide link to pull request?

  • 👍1
mpchadwick commented

This also impacts Apache + PHP-FPM (https://maxchadwick.xyz/blog/http-response-header-size-limit-with-mod-proxy-fcgi). I see this was marked as done in 2.3-develop. Is there a link to a PR that fixes this?

sambolek commented

@mpchadwick

Believe this is the commit 2529ec4 , but it's just another workaround.

It only delays the error, which will happen anyways, if you have a category with large amount of products.

Edit: found another commit that affects tags, but it's related only to purge requests e1f8de3

quickshiftin commented

May I ask what is the purpose of the X-Magento-Tags header? I'm troubleshooting 'Varnish' issues on an M2 site with a large number of categories. The X-Magento-Tags header is absolutely massive!

Varnish obviously is not the culprit here, it's Magento. Why on earth does this header need to be so large? Could the data not be embedded on the page? I'm trying to download pages that are resulting in 503's through Varnish with tried-and-true CLI HTTP clients - guess what the results are when using

wget: HTTP request sent, awaiting response... Read error (Cannot allocate memory) in headers.
curl: curl: (27) Avoided giant realloc for header (max is 102400)!

I looked for options to expand memory allotment for headers in these two CLI clients, could not find them. It seems there are several pieces of software out there indicating this is a gross abuse of HTTP headers!

sambolek commented

Here's a potential solution which splits X-Magento-Tags header into multiple short ones that might help your issue #12831

tedsalmon commented

@quickshiftin,

The purpose of the header is so the cache can be cleared on the fly for any page that contains the tag of the entity being updated in the Admin panel.

The data could always be gzipped and then base64 encoded into the header but at some point you're going to run into this issue again.

sambolek commented

That header is used to determine which pages need to be flushed from Varnish when e.g. a product is saved or cms page is updated. Essentially, Magento tags each page with set of tags pulled in from content that's being displayed on page.

More info https://inviqa.com/blog/how-full-page-cache-works-magento-2

quickshiftin commented

@sambolek, @tedsalmon - Thank you guys for the lighting fast responses!

quickshiftin commented

Hi guys, after looking over the blog page about how the M2 FPC works, other comments on this thread and getting things working on my site I've got a couple things I'd like to share.

First off, merely increasing http_resp_hdr_len was not cutting it. There were several other settings I increased in Varnish to stop the 503's

  • workspace_backend - 256k
  • workspace_client - 256k
  • workspace_session - 256k
  • workspace_thread - 8k (max)

Not sure if all of those are necessary, but the pages are loading. Stumbled into this thread when working on possibly related 504 responses on the same site about a week ago. I also brought http_resp_hdr_len to 2MB, based on what I saw requesting pages directly from the webserver (that would load successfully through curl lol).

The other thing is I think the patch in 2.2.3 is reasonable. Pretty much I agree with @pynej, it's not a perfect solution, but it's definitely the low hanging fruit here, and should dramatically reduce the X-Magento-Tags header size. And for an exhaustive solution like what @sambolek has built, I would still suggest the patch in 2.2.3 go along with it, because why store a bunch of extra data if you don't need to!

  • 👍2
alexblueacorn commented

I'm on 2.3.2 and we still have this issue. I can fix it on my local mac, just by increasing proxy memory buffer, but we have production on ubuntu , apache php-fpm and we having issues to configure it right,
For now we just disabled FPC , but I think do the same @pynej suggested. Just exclude childs from the list.

` /**
* Add child identities to product identities
*
* @param Product $subject
* @param array $identities
* @return array
*/
public function afterGetIdentities(Product $subject, array $identities): array
{
// foreach ($this->configurableType->getChildrenIds($subject->getId()) as $childIds) {
// foreach ($childIds as $childId) {
// $identities[] = Product::CACHE_TAG . '_' . $childId;
// }
// }

    return array_unique($identities);
}`
Tjitse-E commented

I had the same problem on Magento 2.3.2. In my opinion, the cleanest approach is just to disable the \Magento\ConfigurableProduct\Model\Plugin\Frontend\ProductIdentitiesExtender::afterGetIdentities plugin. I did this by creating a new module and then disable the plugin via DI.xml.

Create etc/frontend/di.xml with content:

<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework:ObjectManager/etc/config.xsd">
    <!-- disable \Magento\ConfigurableProduct\Model\Plugin\Frontend\ProductIdentitiesExtender::afterGetIdentities -->
    <type name="Magento\Catalog\Model\Product">
        <plugin name="product_identities_extender" disabled="true"/>
    </type>
</config>
  • 👍4
  • 👎1
diazwatson commented

I can confirm that this issue is still happening in 2.3.2.
Maybe we could rescue the work done in #12831 and apply the changes suggested by @ishakhsuvarov to have this finally fixed?

m2-assistant commented

Hi @engcom-Echo. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

  • 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

    DetailsIf the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

  • 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

  • 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

  • 4. Verify that the issue is reproducible on 2.4-develop branch

    Details- Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
    - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
    - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

  • 5. Add label Issue: Confirmed once verification is complete.

  • 6. Make sure that automatic system confirms that report has been added to the backlog.

magento-engcom-team commented

@engcom-Echo Thank you for verifying the issue.

Unfortunately, not enough information was provided to acknowledge ticket. Please consider adding the following:

  • Add "Reproduced on " label(s) to this ticket based on verification result
  • Update issue content to follow format required by Issue Reporting Guidelines

Once all required information is added, please add label "Issue: Confirmed" again.
Thanks!

magento-engcom-team commented

@engcom-Echo Thank you for verifying the issue.

Unfortunately, not enough information was provided to acknowledge ticket. Please consider adding the following:

Once all required information is added, please add label "Issue: Confirmed" again.
Thanks!

magento-engcom-team commented

✅ Confirmed by @engcom-Echo
Thank you for verifying the issue. Based on the provided information internal tickets MC-29755 were created

Issue Available: @engcom-Echo, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

hostep commented

We are also running against this problem on our local machines (no Varnish, no Redis) which seems to be caused by a configurable product on which 510 simple products are linked.

Does anybody know what settings to tweak in php-fpm and/or apache2 to get around this?

The solution linked by @mpchadwick on his blog doesn't seem to resolve the issue over here unfortunately.

Disabling the plugin as mentioned by @Tjitse-E does seem to work, but doesn't sound like a good solution.

@engcom-Hotel: can you try to bump the priority of this issue internally, because this is an issue on which you can spend many hours staring at a blank screen, finding no decent errors in log files and then accidentally finding this thread here.

  • 👍1
mpchadwick commented

@hostep I haven't found any way to get around this with Apache and PHP-FPM. We've had to move to nginx to get around this.

  • 👍1
engcom-Hotel commented

Hello @hostep
I will try to advance this task.

Thanks for contributing!

KevinMace commented

We're also seeing this issue on a category containing a large configurable product (+1000 variations) along with a dozen simple products.

Magento Version 2.3.1

engcom-Hotel commented

Hello @sdzhepa

Could you raise the priority of this issue by talking to internal Magento teams?
The problem really needs a solution as @hostep wrote earlier.

Thanks for collaboration!

  • 👀1
dhorytskyi commented

It looks like \Magento\ConfigurableProduct\Model\Plugin\Frontend\ProductIdentitiesExtender plugin, which adds child product identities to a configurable product, can be safely removed. Another plugin adds parent identities to child products. So if child product is changed, cache for configurable product will be flushed even if page doesn't have cache tag of child product.

  • 👍5
m2-assistant commented

Hi @o-iegorov. Thank you for working on this issue.
Looks like this issue is already verified and confirmed. But if you want to validate it one more time, please, go though the following instruction:

  • 1. Add/Edit Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

  • 2. Verify that the issue is reproducible on 2.4-develop branch

    Details- Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
    - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
    - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

  • 3. If the issue is not relevant or is not reproducible any more, feel free to close it.

o-iegorov commented

Internal Magento team is working on this issue right now.

  • 👀2
gianstraf commented

It looks like \Magento\ConfigurableProduct\Model\Plugin\Frontend\ProductIdentitiesExtender plugin, which adds child product identities to a configurable product, can be safely removed. Another plugin adds parent identities to child products. So if child product is changed, cache for configurable product will be flushed even if page doesn't have cache tag of child product.

I can confirm that it's true.
The plugin is not useful and causes a varnish 503, when Magento is behind varnish.

  • 👍1
drew7721 commented

I believe this plugin is should not have been added to configurable products.

On stores with over 350k products this is what happens :

  1. It increases the size of the tag header from 815b to 18k (showing 48 products per page)
  2. Varnish RAM usage is increased because of the size of the tags.
  3. Header buffer sizes need to be increased for nginx buffers

I really don't see the point of having all these tags. Note that this was introduced in this commit that was attempting to fix this issue #19313

This was a fix for bundle products but was unfortunately also applied to the configurable products.

This might be a good solution for a store that has only a few products, not many children for configurable products and showing a small number of products per page in the category listings. Not so much for a big store that has configurable products with hundreds of children and shows many products per page.

This also seems to increase the PURGE requests size since it will have all the child tags. I believe that children products should simply return the parent tags and when a change is made to a child or a product the cache should be flushed by the parent tag. In other words, the child/parent relation should be checked when flushing the cache instead of adding all these tags to all the pages.

Maybe @sivaschenko can give us more insight on the reasoning for merging this and the testing that was done to ensure that this change does not break the infrastructure.

  • 👍3
danemacmillan commented

I always see the same faces in Magento's tickets related to performance and huge catalogs. ✋

  • 😄3
sivaschenko commented

@drew7721 the problem is that child products affect parent product representation while parent products do not affect child products representation on the frontend, so in order to be sure that the cache is valid - the cache of pages with parent products should be invalidated when a child product is changed.

drew7721 commented

@sivaschenko I agree that the child products affect parent product representation. But I strongly believe that we are addressing this issue the wrong way, as far as configurable products are concerned.

Goal : clear parent product caches when the child product is modified.

First approach : (the one implemented now)

  • Add all child product cache tags on parent product pages
  • Add all child products as well as parent products on category pages.
  • Clear caches by the child product cache tag and we get the wanted results.
    To me, this seems like a back-end/admin issue being fixed by adding non essential data to the front-end.

Pros: I tried to find some

  • We can clear caches by child product tag.
  • Makes the save operation faster (?) - not certain of this and I don't want to benchmark this.
  • No need to figure out the parent product of a child when changes are done on the back-end (not sure if this is a pro or con actually)

Cons:

  • Header size is increased due to thousands of product tags
    • Think of category with 48 products per page & 5 size super-attributes & 5 colors super-attributes per product => translates into over 1248 tags; That's huge!
  • Increased overhead processing for all category pages.
  • Every time a category page loaded needs to figure out all the tags of all the child products
  • Server config changes needed to support the increased header size.
  • Higher use of RAM by Varnish
  • Increased bandwidth and page sizes
  • Decreased performance overall - IMO

Second approach: (what I suggest as a more viable solution)

  • Always us the parent product tags on categories, configurable product pages and even the child product pages.
  • When a change is made to a child product, figure out the parent(s) and flush caches based on that tag.
  • When changes are made to a parent product, clear caches by the parent product cache tag.
    In other words, approach the issue when the caches are getting flushed (back-end), not when the category is getting generated for the front-end.

Cons:

  • Changes to a child product will flush caches for all siblings (if they're visible individually).
  • Might make back-end operation less performant (save and maybe indexing) - again, not certain of this and I don't want to benchmark this but feel free to and share the result.

Pros:

  • Fixes all the cons of the first approach.
  • It seems a more viable solution overall.

Now, this might not be a good approach for bundled products. I'm not very familiar with this type of products but I get the gist of it. I'm quite positive that a similar approach could be taken.

Please, let me know if this makes sense.

  • 👍1
ilnytskyi commented

For those who faced this issue caused by custom code. e.g. import of product/categories
Make sure you have all indexers set up tu run "Update by Schedule"

Because after each save from entity repository in the loop it will add tags to
\Magento\Framework\Indexer\CacheContext or other storage and will cause reindexing and purging more entities, not only saved/updated ones

MartinAarts commented

It looks like \Magento\ConfigurableProduct\Model\Plugin\Frontend\ProductIdentitiesExtender plugin, which adds child product identities to a configurable product, can be safely removed. Another plugin adds parent identities to child products. So if child product is changed, cache for configurable product will be flushed even if page doesn't have cache tag of child product.

We fixed the problem of an entire category giving a 503 error with this solution on Magento 2.3.4. We just commented out a couple of lines in this file. This is a dirty fix, but this fix can easily be made without downtime, and for as far as I know without any consequences. Used this fixed, assuming this file will soon be changed in the update to 2.3.5 to really solve this problem.

gamort commented

A product[parent or not] knows what categories it was in and what categories it is now in when it is saved.
A child product can lookup what parent products it is in before and after it is saved.
for an edge case, a child product can lookup what categories it's parent product is in when it is saved.

Therefore when a product is saved, a product is able to trigger a flush for all related category pages.
When a child product is saved, it can trigger a flush for all it's parent products.

A category page only has to have the tags associated with it for the category.
A product page only has to have the tags associated directly with it.

There are some tricky problems here to address with the staging feature of Commerce in regards to indexing and caching - but they are already broken to begin with so they can be addressed with an EE specific fix.

magento-engcom-team commented

Hi @katef, @o-iegorov.

Thank you for your report and collaboration!

The issue was fixed by Magento team.

The fix will be available with the upcoming 2.3.5 release.

  • 🎉3
magento-engcom-team commented

Hi @katef, @o-iegorov.

Thank you for your report and collaboration!

The issue was fixed by Magento team. The fix was delivered into magento/magento2:2.3-develop branch(es).
Related commit(s):

The fix will be available with the upcoming 2.3.5 release.

katef commented

@magento-engcom-team Wonderful, thank you!

The commits you linked to 404 though. Was that branch rebased and now they have different hashes now, perhaps?

magento-engcom-team commented

Hi @katef, @o-iegorov.

Thank you for your report and collaboration!

The issue was fixed by Magento team. The fix was delivered into magento/magento2:2.4-develop branch(es).
Related commit(s):

The fix will be available with the upcoming 2.4.0 release.

katef commented

Hi @magento-engcom-team! That's great, thank you!
All your links 404 for me, though. Are those private perhaps?

bmitchell-ldg commented

Hi @magento-engcom-team we would also like to see the related commits for 2.4.0 if you would be able to resolve those 404 links please?

ecoprince commented

Hi @katef, @o-iegorov.

Thank you for your report and collaboration!

The issue was fixed by Magento team. The fix was delivered into magento/magento2:2.4-develop branch(es).
Related commit(s):

The fix will be available with the upcoming 2.4.0 release.

@magento-engcom-team All links goes to 404. Please update it.

ihor-sviziev commented

@ecoprince,
Now these commits are available. Here is merge commit that contains all these changes: 3b611a7

  • 👍2
katef commented

Thank you!

websecureNL commented

Any way to get this fixed on 2.3.5? I understand the fix is made for 2.4 develop but for our production store we are using 2.3.5-p1 and having this problem.

websecureNL commented

@ihor-sviziev, I tried this and it seems to work properly. We will do some further testing. Since making patches was new to me a quick hint for other users. We used this file (based on merge commit 3b611a7) to patch using instructions https://devdocs.magento.com/guides/v2.4/comp-mgr/patching.html

  • 👍1
websecureNL commented

The problem also occurs when using catalogsearch. No idea if this also impacts 2.4 but since the patch did not fix this it could be.

ihor-sviziev commented
websecureNL commented

@ihor-sviziev. Of course I can do this but isn't this originated from Magento 2.3? We are having problems with 2.3.5 and would like to get it fixed there. I will check 2.4-develop asap.

elvinristi commented

if you compare commits then actually nothing was done. was reverted probably something what didn't exist yet in M2.1 and therefor no matter of the version, issue continues.

temporary fix can be currently #6401 (comment)

but longer run should be similar what was done here but closed and ignored due inactivity #12831 which splits headers.

ihor-sviziev commented

if you compare commits then actually nothing was done. was reverted probably something what didn't exist yet in M2.1 and therefor no matter of the version, issue continues.

temporary fix can be currently #6401 (comment)

but longer run should be similar what was done here but closed and ignored due inactivity #12831 which splits headers.

@elevinskii #6401 (comment) is really hack and not a correct solution, it breaks automatic cache flushing for category pages when any product was changed in the admin. definitely don't recommend it.

#12831 this solution is quite good, but it's not finalized in order to be accepted. You could create new PR based on this PR and apply requested changes. Only one concern - such change might break some services like Fastly (for instance used in Magento Cloud), so probably it should be double checked additionally

Also would like to add that there was a fix for configurable products that mostly caused this issue, this fix you can see in #6401 (comment)

BTW if you have steps to reproduce of this issue on 2.4-develop - please report it separately

toonvd commented

I just landed on this issue, a more complete solution would be to implement #6401 (comment). And create an observer on product save that gets parents on simple products and invalidates those too.

amenk commented

Still happens on some circumstances on 2.4.2. Is this header only needed if varnish is used? Can it be safely removed, if not?

hostep commented

@amenk: as far as I'm aware: yes! The X-Magento-Tags header is only used by Varnish to indicate what tags are associated to a certain url. So when Magento tells Varnish to purge some tags later on, it will purge all url's from its cache where that particular tag got associated to it.
So I believe you should be able to safely remove the header if you aren't using Varnish (or a similar reverse proxy that works with this header).

  • 👍3
amenk commented

@hostep Thanks - we have a patch on StackOverflow: https://magento.stackexchange.com/questions/287621/ah01070-error-parsing-script-headers-in-php-fpm-when-accessing-certain-product .

What should be noted is that the clearing should happen after the tags are extracted from the Header and used for the built-in cache as well.

  • 👍1
atty31 commented

This happens on 2.4.3-p1 . Any updates or patches ? Thanks

  • 👍2
mlaurense commented

After having raised the http_resp_hdr_len to 1MB (derived by calculating the largest number of products in a category multiplied by 21, as advised by Magento: https://support.magento.com/hc/en-us/articles/360034631211-Troubleshooting-503-error-caused-by-necessity-to-change-default-Varnish-settings), we were still experiencing intermittent 503 errors. These would happen on static files, e.g. javascripts and images.

To further debug this, I opened a varnishlog using the following command:
varnishlog -g request -q "RespStatus >= 500"
This returns all requests with a response code of 500 or higher. Now we just had to wait for an error to appear.

Soon after, the errors appeared:

--  FetchError     Workspace overflow
--  FetchError     overflow
[...]
--  BerespStatus   503
--  BerespReason   Service Unavailable
--  BerespReason   Backend fetch failed
--  Error          out of workspace (bo)

This is indicating a clear error, and pointed me in the right direction to further raise the workspace parameters as well. I am now using these parameters and the errors haven't reappeared:

    -p workspace_backend=2097152
    -p http_resp_size=1082768
    -p http_resp_hdr_len=1050000
    -p workspace_client=2097152
    -p http_req_size=65536
    -p http_req_hdr_len=32768

Of course, this shouldn't be necessary, but given the fact that Magento uses these very large tag headers, I'm afraid we have no choice other than to tune these varnish parameters.

  • 👍2
  • 😕1
hostep commented

Potential fix: #33468

  • 👍2