More context is included than selected

[tsoernes]

In this example, I have set the request to add a buffer to the context, with the user prompt. I have also set it to exclude any previous messages. When inspecting the JSON query, it is not correct. It includes both the buffer, and then some code that should not be there.

{
  "model": "m-gpt-4o-mini-18072024",
  "messages": [
    {
      "role": "system",
      "content": "You are a large language model and a careful programmer. First explain your code, then provide the code.\n\nImprove this code"
    },
    {
      "role": "assistant",
      "content": "Request context:\n\nIn buffer `polars.py`:\n\n```python\n\"\"\"A collection of tools and utilities for Polars\"\"\"\n\nimport polars as pl\nimport polars.selectors as cs\n\n\ndef drop_empty_list_columns(df: pl.DataFrame, verbose: bool = True) -> pl.DataFrame:\n    \"\"\"Drop columns with just empty lists\"\"\"\n    for col in df.columns:\n        if isinstance(df[col].dtype, pl.List):\n            if df[col].list.len().sum() == 0:\n                df = df.drop(col)\n                if verbose:\n                    print(f\"Dropped empty list column {col}\")\n    return df\n\n\ndef drop_empty_list_columns_in_place(df: pl.DataFrame, verbose: bool = True) -> None:\n    \"\"\"Drop columns with just empty lists\"\"\"\n    for col in df.columns:\n        if isinstance(df[col].dtype, pl.List):\n            if df[col].list.len().sum() == 0:\n                df.drop_in_place(col)\n                if verbose:\n                    print(f\"Dropped empty list column {col}\")\n\n\ndef empty_string_to_null(\n    df: pl.Series | pl.DataFrame, columns: str | list[str] | None = None, verbose: bool = True\n) -> pl.Series | pl.DataFrame:\n    \"\"\"\n    Replace empty strings with nulls. If no column(s) are given then all suitable columns are processed.\n    \"\"\"\n    if isinstance(df, pl.Series):\n        is_series = True\n        df = df.to_frame()\n    else:\n        is_series = False\n\n    if isinstance(columns, str):\n        columns = [columns]\n    # Select string columns\n    string_cols = columns if columns else df.select(cs.by_dtype(pl.String)).columns\n\n    # Create a mask for empty strings\n    masks = {col: df[col].str.len_bytes().eq(0) for col in string_cols}\n\n    # Apply the mask to set empty strings to null\n    df = df.with_columns(\n            pl.when(masks[col]).then(None).otherwise(pl.col(col)).alias(col)\n            for col in string_cols\n    )\n\n    if verbose:\n        for col in string_cols:\n            if n_empty := masks[col].sum():\n                print(f\"Replaced {n_empty} empty strings with null in column {col}\")\n\n    if is_series:\n        df = df.to_series(0)\n\n    return df\n\n\ndef empty_list_to_null(\n    df: pl.DataFrame | pl.Series,\n    columns: str | list[str] | None = None,\n    verbose: bool = True\n) -> pl.DataFrame | pl.Series:\n    \"\"\"Given a polars DataFrame, convert all empty lists in the given column(s) to nulls.\n    If no columns are specified, the all list columns will be processed.\"\"\"\n    if isinstance(df, pl.Series):\n        is_series = True\n        df = df.to_frame()\n    else:\n        is_series = False\n\n    if columns is None:\n        columns = [col for col in df.columns if isinstance(df[col].dtype, pl.List)]\n    elif isinstance(columns, str):\n        columns = [columns]\n\n    for column in columns:\n        if column in df.columns:\n            df = df.with_columns(\n                pl.when(pl.col(column).list.len().eq(0))\n                .then(None)\n                .otherwise(pl.col(column))\n                .alias(column)\n            )\n        elif verbose:\n            print(f\"{column=} not in df\")\n\n    if is_series:\n        df = df.to_series(0)\n    return df\n\n\ndef unnest_struct_columns(\n    df: pl.DataFrame,\n    columns: str | list[str] | None = None,\n    recurse: bool = True,\n    sep: str = \".\",\n    verbose: bool = True,\n) -> pl.DataFrame:\n    \"\"\"\n    Flatten/unnest struct (from nested JSON or dicts) columns in the DataFrame.\n    If no column(s) are given then all suitable columns are processed.\n    \"\"\"\n    # Identify JSON columns by checking for struct type\n    if columns:\n        if isinstance(columns, str):\n            struct_columns = [columns]\n        else:\n            struct_columns = columns\n    else:\n        struct_columns = [\n            col for col in df.columns if isinstance(df[col].dtype, pl.Struct)\n        ]\n\n    # No JSON columns found\n    if not struct_columns:\n        return df\n\n    # Process each JSON column\n    flattened_dfs = []\n    remaining_cols = [col for col in df.columns if col not in struct_columns]\n\n    # Keep non-JSON columns\n    if remaining_cols:\n        flattened_dfs.append(df.select(remaining_cols))\n\n    # Flatten each JSON column\n    for col in struct_columns:\n        flattened = df.select(pl.col(col)).unnest(col)\n        # Prefix column names with original column name\n        new_names = {c: f\"{col}{sep}{c}\" for c in flattened.columns}\n        flattened = flattened.rename(new_names)\n        if recurse:\n            flattened = unnest_struct_columns(flattened, recurse=True)\n        if verbose:\n            new_cols = \", \".join([f\"'{c}'\" for c in flattened.columns])\n            print(f\"Unnested '{col}' to {new_cols}\")\n        flattened_dfs.append(flattened)\n\n    # Combine all DataFrames horizontally\n    return pl.concat(flattened_dfs, how=\"horizontal\")\n\n\ndef convert_date_columns(\n    df: pl.DataFrame, columns: str | list[str], format: str = \"%Y-%m-%d\"\n) -> pl.DataFrame:\n    \"\"\"Convert (parse) specified string column(s) to date type.\"\"\"\n    if isinstance(columns, str):\n        columns = [columns]\n    for col in columns:\n        if col in df.columns:\n            df = df.with_columns(\n                [pl.col(col).str.strptime(pl.Date, format=format).alias(col)]\n            )\n        else:\n            print(f\"{col=} not in df\")\n    return df\n\n\ndef cast_columns(\n        df: pl.DataFrame, columns: list[str], dtype, verbose: bool = True,\n) -> pl.DataFrame:\n    \"\"\"Convert specified columns to given type.\"\"\"\n    columns_ = []\n    for col in columns:\n        if col in df.columns:\n            columns_.append(col)\n        elif verbose:\n            print(f\"{col=} not in df\")\n    df = df.with_columns([pl.col(col).cast(dtype).alias(col) for col in columns_])\n    return df\n\n...\n```\n\n\n    elif isinstance(columns, str):\n        columns = [columns]\n\n    for column in columns:\n        if column in df.columns:\n            df = df.with_columns"
    }
  ],
  "stream": true,
  "temperature": 1.0
}

Look at the assitant content. It contains the buffer, with the file name, quoted in markdown. That is as expected. But after the markdown quoted code, it also includes more code, that is not selected.

[karthink]

It's hard to read the code as included in the content field above. Could you do the following?

1. Run (setq gptel-expert-commands t)
2. Set up the same context, but instead of calling gptel-send/RET,
3. Run the dry run command to view the request data as a Lisp structure:

Then paste that result here.

[tsoernes]

@karthink

Here is the same content. The only thing I have added is newlines. It should make it easier to distinguish content that should be there (from the buffer) vs content that is not supposed to be there. There is only one triple-quoted markdown block in the content, which is from the file. The rest I have no idea where is from.

{
  "model": "m-gpt-4o-mini-18072024",
  "messages": [
    {
      "role": "system",
      "content": "You are a large language model and a careful programmer. First explain your code, then provide the code.\n\nImprove this code"
    },
    {
      "role": "assistant",
      "content": "Request context:\n\n
In buffer `polars.py`:\n\n
```python\n\"\"\"A collection of tools and utilities for Polars\"\"\"\n\nimport polars as pl\nimport polars.selectors as cs\n\n\ndef drop_empty_list_columns(df: pl.DataFrame, verbose: bool = True) -> pl.DataFrame:\n    \"\"\"Drop columns with just empty lists\"\"\"\n    for col in df.columns:\n        if isinstance(df[col].dtype, pl.List):\n            if df[col].list.len().sum() == 0:\n                df = df.drop(col)\n                if verbose:\n                    print(f\"Dropped empty list column {col}\")\n    return df\n\n\ndef drop_empty_list_columns_in_place(df: pl.DataFrame, verbose: bool = True) -> None:\n    \"\"\"Drop columns with just empty lists\"\"\"\n    for col in df.columns:\n        if isinstance(df[col].dtype, pl.List):\n            if df[col].list.len().sum() == 0:\n                df.drop_in_place(col)\n                if verbose:\n                    print(f\"Dropped empty list column {col}\")\n\n\ndef empty_string_to_null(\n    df: pl.Series | pl.DataFrame, columns: str | list[str] | None = None, verbose: bool = True\n) -> pl.Series | pl.DataFrame:\n    \"\"\"\n    Replace empty strings with nulls. If no column(s) are given then all suitable columns are processed.\n    \"\"\"\n    if isinstance(df, pl.Series):\n        is_series = True\n        df = df.to_frame()\n    else:\n        is_series = False\n\n    if isinstance(columns, str):\n        columns = [columns]\n    # Select string columns\n    string_cols = columns if columns else df.select(cs.by_dtype(pl.String)).columns\n\n    # Create a mask for empty strings\n    masks = {col: df[col].str.len_bytes().eq(0) for col in string_cols}\n\n    # Apply the mask to set empty strings to null\n    df = df.with_columns(\n            pl.when(masks[col]).then(None).otherwise(pl.col(col)).alias(col)\n            for col in string_cols\n    )\n\n    if verbose:\n        for col in string_cols:\n            if n_empty := masks[col].sum():\n                print(f\"Replaced {n_empty} empty strings with null in column {col}\")\n\n    if is_series:\n        df = df.to_series(0)\n\n    return df\n\n\ndef empty_list_to_null(\n    df: pl.DataFrame | pl.Series,\n    columns: str | list[str] | None = None,\n    verbose: bool = True\n) -> pl.DataFrame | pl.Series:\n    \"\"\"Given a polars DataFrame, convert all empty lists in the given column(s) to nulls.\n    If no columns are specified, the all list columns will be processed.\"\"\"\n    if isinstance(df, pl.Series):\n        is_series = True\n        df = df.to_frame()\n    else:\n        is_series = False\n\n    if columns is None:\n        columns = [col for col in df.columns if isinstance(df[col].dtype, pl.List)]\n    elif isinstance(columns, str):\n        columns = [columns]\n\n    for column in columns:\n        if column in df.columns:\n            df = df.with_columns(\n                pl.when(pl.col(column).list.len().eq(0))\n                .then(None)\n                .otherwise(pl.col(column))\n                .alias(column)\n            )\n        elif verbose:\n            print(f\"{column=} not in df\")\n\n    if is_series:\n        df = df.to_series(0)\n    return df\n\n\ndef unnest_struct_columns(\n    df: pl.DataFrame,\n    columns: str | list[str] | None = None,\n    recurse: bool = True,\n    sep: str = \".\",\n    verbose: bool = True,\n) -> pl.DataFrame:\n    \"\"\"\n    Flatten/unnest struct (from nested JSON or dicts) columns in the DataFrame.\n    If no column(s) are given then all suitable columns are processed.\n    \"\"\"\n    # Identify JSON columns by checking for struct type\n    if columns:\n        if isinstance(columns, str):\n            struct_columns = [columns]\n        else:\n            struct_columns = columns\n    else:\n        struct_columns = [\n            col for col in df.columns if isinstance(df[col].dtype, pl.Struct)\n        ]\n\n    # No JSON columns found\n    if not struct_columns:\n        return df\n\n    # Process each JSON column\n    flattened_dfs = []\n    remaining_cols = [col for col in df.columns if col not in struct_columns]\n\n    # Keep non-JSON columns\n    if remaining_cols:\n        flattened_dfs.append(df.select(remaining_cols))\n\n    # Flatten each JSON column\n    for col in struct_columns:\n        flattened = df.select(pl.col(col)).unnest(col)\n        # Prefix column names with original column name\n        new_names = {c: f\"{col}{sep}{c}\" for c in flattened.columns}\n        flattened = flattened.rename(new_names)\n        if recurse:\n            flattened = unnest_struct_columns(flattened, recurse=True)\n        if verbose:\n            new_cols = \", \".join([f\"'{c}'\" for c in flattened.columns])\n            print(f\"Unnested '{col}' to {new_cols}\")\n        flattened_dfs.append(flattened)\n\n    # Combine all DataFrames horizontally\n    return pl.concat(flattened_dfs, how=\"horizontal\")\n\n\ndef convert_date_columns(\n    df: pl.DataFrame, columns: str | list[str], format: str = \"%Y-%m-%d\"\n) -> pl.DataFrame:\n    \"\"\"Convert (parse) specified string column(s) to date type.\"\"\"\n    if isinstance(columns, str):\n        columns = [columns]\n    for col in columns:\n        if col in df.columns:\n            df = df.with_columns(\n                [pl.col(col).str.strptime(pl.Date, format=format).alias(col)]\n            )\n        else:\n            print(f\"{col=} not in df\")\n    return df\n\n\ndef cast_columns(\n        df: pl.DataFrame, columns: list[str], dtype, verbose: bool = True,\n) -> pl.DataFrame:\n    \"\"\"Convert specified columns to given type.\"\"\"\n    columns_ = []\n    for col in columns:\n        if col in df.columns:\n            columns_.append(col)\n        elif verbose:\n            print(f\"{col=} not in df\")\n    df = df.with_columns([pl.col(col).cast(dtype).alias(col) for col in columns_])\n    return df
\n\n...\n```

\n\n\n    elif isinstance(columns, str):\n        columns = [columns]\n\n    for column in columns:\n        if column in df.columns:\n            df = df.with_columns"
    }
  ],
  "stream": true,
  "temperature": 1.0
}

[karthink]

Where was the cursor when you sent this request?

[tsoernes]

bottom of buffer, no text selected.

[karthink]

bottom of buffer, no text selected.

Then it's working as intended -- without a region selected, gptel sends everything up to the cursor, as well as any context included with gptel-add. If you want to send the current buffer up to point, you don't need to run gptel-add.

[karthink]

Closing since this is working as intended. Feel free to continue the discussion here if you have more questions.