streams over paginate loop over the same record over and over again

Question

streams over paginate loop over the same record over and over again

Opened this issue a year ago · 10 comments

Describe the bug

create an invoice item
list pending invoice items
use the stream pagination
watch as it loops endlessly over the same record
some sample code:

#[tracing::instrument(skip(self))]
    async fn list_pending_invoice_items(&self, customer_id: &str) -> Result<Vec<crate::payments::types::InvoiceItem>> {
        // List all the invoice items that are PENDING for the customer.
        let mut invoice_items: Vec<crate::payments::types::InvoiceItem> = Default::default();

        let params = stripe::ListInvoiceItems {
            customer: Some(stripe::CustomerId::from_str(customer_id)?),
            limit: Some(100),
            pending: Some(true),
            ..Default::default()
        };
        let list = stripe::InvoiceItem::list(&self.client, &params).await?.paginate(params);
        let mut stream = list.stream(&self.client);
        // For each invoice in the stream add it to our array.
        while let Some(Ok(next)) = stream.next().await {
            invoice_items.push(next.into());
        }

        Ok(invoice_items)
    }

To Reproduce

see above sorry!

Expected behavior

Previous we were on version 0.21 now on 0.26 we see this bug. Without the code changing it appeared. So i would expect the stream to stop after its gotten all the records

Code snippets

No response

OS

macos

Rust version

1.75.0

Library version

0.26.0

API version

whatever the default is for the library?

Additional context

No response

Answer 1 · 2024-01-03T23:02:37.000Z

ah okay so 0.25 works, 0.26 doesnt and I think it has to do with what is noted in the changelog about changes to List

Answer 2 · 2024-01-07T20:08:40.000Z

Thanks @jessfraz - hoping this will be fixed by pagination changes in #452. If possible, it would be extremely helpful if you could capture the sequence of stripe responses to your sample code (or just debug printing the first 2 returned List I guess)? Then I can add as a simple regression test that doesn't need to hit the stripe API

Answer 3 · 2024-01-08T18:57:37.000Z

yes will get you some json from our "test mode" today! thank you!

Answer 4 · 2024-01-08T22:26:02.000Z

you know what is weird... it just started working fine on 0.26, my test that was failing is now passing, when i tried to get you the debug output

Answer 5 · 2024-01-08T22:39:14.000Z

maybe it was a server side stripe thing because I swear its the same code

Answer 6 · 2024-01-08T22:47:29.000Z

Thanks for looking into it - that's strange! Think this is good to leave open regardless because I wouldn't be surprised if it's a weird pagination edge case.

Answer 7 · 2024-01-08T22:56:36.000Z

Okay it is def a weird edge case, in that previously it was happening on listing invoice items, but now it is happening on listing invoices.

here is the debug ooutput of listing invoices when it happens

This is in our stripe test mode so nothing is bad or a secret.

list: ListPaginator { page: List { data: [Invoice { id: InvoiceId("in_1OWRfIGC2bHVgrBXnRrx3rka"), account_country: Some("US"), account_name: Some("Zoo Corporation"), account_tax_ids: None, amount_due: Some(0), amount_paid: Some(0), amount_remaining: Some(0), amount_shipping: Some(0), application: None, application_fee_amount: None, attempt_count: Some(0), attempted: Some(false), auto_advance: Some(true), automatic_tax: Some(AutomaticTax { enabled: false, status: None }), billing_reason: Some(Manual), charge: None, collection_method: Some(ChargeAutomatically), created: Some(1704754288), currency: Some(USD), custom_fields: None, customer: Some(Id(CustomerId("cus_PJGZy9fkkikP3k"))), customer_address: None, customer_email: Some("test@create_invoice_item_beta.com"), customer_name: Some("Test User"), customer_phone: None, customer_shipping: None, customer_tax_exempt: Some(None), customer_tax_ids: Some([]), default_payment_method: None, default_source: None, default_tax_rates: Some([]), deleted: false, description: Some("I CAD to bill you."), discount: None, discounts: Some([]), due_date: None, effective_at: None, ending_balance: None, footer: None, from_invoice: None, hosted_invoice_url: None, invoice_pdf: None, last_finalization_error: None, latest_revision: None, lines: List { data: [InvoiceLineItem { id: InvoiceLineItemIdWebhook(InvoiceLineItemIdWebhook("il_1OWRfHGC2bHVgrBXdZPrmDTs")), amount: 150, amount_excluding_tax: Some(150), currency: USD, description: Some("12e4a6f1-8292-47bd-8e09-4438aa6abd84 GET /ws/modeling/commands 200 OK: 3 minutes"), discount_amounts: Some([DiscountsResourceDiscountAmount { amount: 150, discount: Id(DiscountId("di_1OWRfHGC2bHVgrBXJk0OkEe8")) }]), discountable: true, discounts: Some([Id(DiscountId("di_1OWRfHGC2bHVgrBXJk0OkEe8"))]), invoice_item: Some(Id(InvoiceItemId("ii_1OWRfHGC2bHVgrBXLFmhySEG"))), livemode: false, metadata: {"api_call_id": "12e4a6f1-8292-47bd-8e09-4438aa6abd84", "api_call_minutes": "3", "beta": "true", "user_id": "3e147226-5f6d-490d-888c-507bd7df4daa", "api_call_method": "GET", "api_call_endpoint": "/ws/modeling/commands", "api_call_status_code": "200 OK"}, period: Some(Period { end: Some(1704754287), start: Some(1704754287) }), plan: None, price: Some(Price { id: PriceId("price_1OWRfHGC2bHVgrBXv5rcq2qo"), active: Some(false), billing_scheme: Some(PerUnit), created: Some(1704754287), currency: Some(USD), currency_options: None, custom_unit_amount: None, deleted: false, livemode: Some(false), lookup_key: None, metadata: Some({}), nickname: None, product: Some(Id(ProductId("prod_PL7vbJk7ufq4zP"))), recurring: None, tax_behavior: Some(Inclusive), tiers: None, tiers_mode: None, transform_quantity: None, type_: Some(OneTime), unit_amount: Some(150), unit_amount_decimal: Some("150") }), proration: false, proration_details: Some(InvoicesResourceLineItemsProrationDetails { credited_items: None }), quantity: Some(1), subscription: None, subscription_item: None, tax_amounts: Some([]), tax_rates: Some([]), type_: InvoiceItem, unit_amount_excluding_tax: Some("150") }], has_more: false, total_count: Some(1), url: "/v1/invoices/in_1OWRfIGC2bHVgrBXnRrx3rka/lines" }, livemode: Some(false), metadata: Some({}), next_payment_attempt: Some(1704757888), number: None, on_behalf_of: None, paid: Some(false), paid_out_of_band: Some(false), payment_intent: None, payment_settings: Some(InvoicesPaymentSettings { default_mandate: None, payment_method_options: None, payment_method_types: None }), period_end: Some(1704754288), period_start: Some(1704754288), post_payment_credit_notes_amount: Some(0), pre_payment_credit_notes_amount: Some(0), quote: None, receipt_number: None, rendering_options: None, shipping_cost: None, shipping_details: None, starting_balance: Some(0), statement_descriptor: None, status: Some(Draft), status_transitions: Some(InvoicesStatusTransitions { finalized_at: None, marked_uncollectible_at: None, paid_at: None, voided_at: None }), subscription: None, subscription_details: Some(SubscriptionDetailsData { metadata: None }), subscription_proration_date: None, subtotal: Some(0), subtotal_excluding_tax: Some(0), tax: None, test_clock: None, threshold_reason: None, total: Some(0), total_discount_amounts: Some([DiscountsResourceDiscountAmount { amount: 150, discount: Id(DiscountId("di_1OWRfHGC2bHVgrBXJk0OkEe8")) }]), total_excluding_tax: Some(0), total_tax_amounts: Some([]), transfer_data: None, webhooks_delivered_at: Some(1704754289) }, Invoice { id: InvoiceId("in_1L8zBKGC2bHVgrBXetniCtUq"), account_country: Some("US"), account_name: Some("Zoo Corporation"), account_tax_ids: None, amount_due: Some(0), amount_paid: Some(0), amount_remaining: Some(0), amount_shipping: Some(0), application: None, application_fee_amount: None, attempt_count: Some(0), attempted: Some(false), auto_advance: Some(false), automatic_tax: Some(AutomaticTax { enabled: false, status: None }), billing_reason: Some(Manual), charge: None, collection_method: Some(ChargeAutomatically), created: Some(1654834246), currency: Some(USD), custom_fields: None, customer: Some(Id(CustomerId("cus_Lqg9WtmoSWfZhj"))), customer_address: None, customer_email: Some("kc+test@jessfraz.com"), customer_name: Some("test f name test l name"), customer_phone: Some("test-phone"), customer_shipping: None, customer_tax_exempt: Some(None), customer_tax_ids: Some([]), default_payment_method: None, default_source: None, default_tax_rates: Some([]), deleted: false, description: None, discount: None, discounts: Some([]), due_date: None, effective_at: None, ending_balance: None, footer: None, from_invoice: None, hosted_invoice_url: None, invoice_pdf: None, last_finalization_error: None, latest_revision: None, lines: List { data: [], has_more: false, total_count: Some(0), url: "/v1/invoices/in_1L8zBKGC2bHVgrBXetniCtUq/lines" }, livemode: Some(false), metadata: Some({}), next_payment_attempt: None, number: None, on_behalf_of: None, paid: Some(false), paid_out_of_band: Some(false), payment_intent: None, payment_settings: Some(InvoicesPaymentSettings { default_mandate: None, payment_method_options: None, payment_method_types: None }), period_end: Some(1654834246), period_start: Some(1654834246), post_payment_credit_notes_amount: Some(0), pre_payment_credit_notes_amount: Some(0), quote: None, receipt_number: None, rendering_options: None, shipping_cost: None, shipping_details: None, starting_balance: Some(0), statement_descriptor: None, status: Some(Draft), status_transitions: Some(InvoicesStatusTransitions { finalized_at: None, marked_uncollectible_at: None, paid_at: None, voided_at: None }), subscription: None, subscription_details: Some(SubscriptionDetailsData { metadata: None }), subscription_proration_date: None, subtotal: Some(0), subtotal_excluding_tax: Some(0), tax: None, test_clock: None, threshold_reason: None, total: Some(0), total_discount_amounts: Some([]), total_excluding_tax: Some(0), total_tax_amounts: Some([]), transfer_data: None, webhooks_delivered_at: Some(1654834246) }], has_more: false, total_count: None, url: "/v1/invoices" }, params: ListInvoices { collection_method: None, created: None, customer: None, due_date: None, ending_before: None, expand: [], limit: Some(100), starting_after: None, status: Some(Draft), subscription: None } }

Answer 8 · 2024-01-08T22:57:35.000Z

I think it happens when there is only one item in the list?

Answer 9 · 2024-01-09T16:01:15.000Z

I see the same and based on the above tip, this workaround is helping

    let mut result: Vec<Customer> = Default::default();

    let list = Customer::list(&client, &params).await?;
    if list.has_more {
        let mut stream = list.paginate(params).stream(&client);
        while let Some(Ok(next)) = stream.next().await {
            result.push(next);
        }
    } else {
        result = list.data;
    }

EDIT: actually not working 100% of the time

Answer 10 · 2024-01-09T16:52:18.000Z

I am hoping this issue will be nuked once the rewrite lands. If you are able to produce a unit test or even a set of json requests / responses I will try to understand what is wrong.