[Question] About 2048 scoring
Opened this issue · 2 comments
katopz commented
I try 2048 example and only one scores in the end (step 27), other is always {'scores': []}
[RULER] Pretty-printed LLM choice JSON:
{'scores': []}
Swallowed exception:
Skipping tuning as there is no suitable data. This can happen when all the trajectories in the same group have the same reward and thus no advantage to train on.
Advanced step from 25 to 26 (no training occurred)
gather: 100%|███████████| 9/9 [00:34<00:00, 3.80s/it, reward=1.17, invalid_move=0.111, max_value=71.3, board_value=159, num_moves=71, win=0.333, prompt_tokens=2.04e+3, completion_tokens=21.3, total_completion_tokens=5240.0]
[RULER] Pretty-printed LLM choice JSON:
{'scores': []}
Swallowed exception:
Skipping tuning as there is no suitable data. This can happen when all the trajectories in the same group have the same reward and thus no advantage to train on.
Advanced step from 26 to 27 (no training occurred)
gather: 100%|█████████| 9/9 [00:39<00:00, 4.40s/it, reward=1.49, invalid_move=0.111, max_value=93.6, board_value=162, num_moves=72.3, win=0.667, prompt_tokens=2.21e+3, completion_tokens=21.3, total_completion_tokens=5336.0]
[RULER] Pretty-printed LLM choice JSON:
{
'scores': [
{
'trajectory_id': '1',
'explanation': "This trajectory starts with a very basic board state (2, 2 at top) and makes only one move 'up'. It doesn't achieve the goal of reaching 2048, but shows early progress toward organizing cells. It
gets partial credit for making a move that could potentially lead to combining cells.",
'score': 0.1
},
{
'trajectory_id': '2',
'explanation': 'This trajectory demonstrates excellent 2048 gameplay, reaching high values including a 32 and continuing to build up the board efficiently. It makes strategic moves without getting stuck and
progresses toward 2048 successfully.',
'score': 0.95
},
{
'trajectory_id': '3',
'explanation': 'This trajectory also performs very well, reaching high tile values and successfully building up the board. It shows strong strategic play with consistent progress toward 2048.',
'score': 0.9
},
{
'trajectory_id': '4',
'explanation': 'This trajectory shows good 2048 playing, reaching tile values up to 32. It makes strategic moves but has some inefficient patterns that could be improved, though still performs well overall.',
'score': 0.8
},
{
'trajectory_id': '5',
'explanation': 'This trajectory reaches tile values up to 32. It is very strategic and shows good board organization, but has some inefficient moves compared to trajectory 2 or 3.',
'score': 0.85
},
{
'trajectory_id': '6',
'explanation': 'This trajectory also shows good 2048 playing, reaching tiles up to 32. While it progresses well and shows planning ahead, it has some suboptimal moves that make it less efficient than the top
performers.',
'score': 0.8
},
{
'trajectory_id': '7',
'explanation': 'This trajectory makes good progress toward 2048, reaching high tile values. However, its movements are less consistently efficient compared to top trajectories in terms of combining cells
effectively.',
'score': 0.75
},
{
'trajectory_id': '8',
'explanation': "This trajectory reaches high tiles but shows some inefficiencies in its approach. It makes moves that don't always leverage existing cell combinations effectively.",
'score': 0.7
},
{
'trajectory_id': '9',
'explanation': 'This trajectory shows moderate progress, reaching some high tiles but with several inefficient moves that prevent it from performing as well as the top trajectories.',
'score': 0.65
}
]
}Only the change I made is I use local qwen3-30b-a3b-thinking-2507 and SIMULTANEOUS_GAMES = 9.
Just curious btw. 😗
Kovbo commented
Hi! It’s hard to tell from the logs. I recommend printing each step to identify which one is failing.
First, print all trajectories and confirm that rollouts are working correctly.
Then, print the raw requests and responses from the judge.
I suspect the issue is specific to qwen3-30b-a3b-thinking-2507, possibly related to its output format.
katopz commented
Hey, Thanks for response, Here's full log (still running)
source /home/katopz/anypost/.venv/bin/activate
(base) katopz@shikuwa:~/anypost$ source /home/katopz/anypost/.venv/bin/activate
(anypost) (base) katopz@shikuwa:~/anypost$ uv run 2048.py
INFO 09-13 13:07:15 [__init__.py:235] Automatically detected platform cuda.
/home/katopz/anypost/.venv/lib/python3.11/site-packages/art/__init__.py:10: UserWarning: WARNING: Unsloth should be imported before transformers, peft to ensure all optimizations are applied. Your code may run slower or encounter memory issues without these optimizations.
Please restructure your imports with 'import unsloth' at the top of your file.
import unsloth # type: ignore # noqa: F401
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
INFO 09-13 13:07:20 [__init__.py:235] Automatically detected platform cuda.
🦥 Unsloth Zoo will now patch everything to make training faster!
Unsloth: Patching vLLM v1 graph capture
Unsloth: Patching vLLM v0 graph capture
==((====))== Unsloth 2025.8.6: Fast Qwen2 patching. Transformers: 4.53.2. vLLM: 0.10.0.
\\ /| NVIDIA GeForce RTX 4090. Num GPUs = 1. Max memory: 23.988 GB. Platform: Linux.
O^O/ \_/ \ Torch: 2.7.1+cu126. CUDA: 8.9. CUDA Toolkit: 12.6. Triton: 3.3.1
\ / Bfloat16 = TRUE. FA [Xformers = 0.0.31. FA2 = False]
"-____-" Free license: http://github.com/unslothai/unsloth
Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
Unsloth: vLLM loading unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit with actual GPU utilization = 73.85%
Unsloth: Your GPU has CUDA compute capability 8.9 with VRAM = 23.99 GB.
Unsloth: Using conservativeness = 1.0. Chunked prefill tokens = 8192. Num Sequences = 256.
Unsloth: vLLM's KV Cache can use up to 15.49 GB. Also swap space = 0 GB.
Unsloth: Not an error, but `device` is not supported in vLLM. Skipping.
INFO 09-13 13:07:39 [config.py:1604] Using max model len 8192
Unsloth: vLLM Bitsandbytes config using kwargs = {'load_in_8bit': False, 'load_in_4bit': True, 'bnb_4bit_compute_dtype': 'bfloat16', 'bnb_4bit_quant_storage': 'uint8', 'bnb_4bit_quant_type': 'nf4', 'bnb_4bit_use_double_quant': True, 'llm_int8_enable_fp32_cpu_offload': False, 'llm_int8_has_fp16_weight': False, 'llm_int8_skip_modules': ['lm_head', 'multi_modal_projector', 'merger', 'modality_projection', 'model.layers.2.mlp', 'model.layers.3.mlp', 'model.layers.30.mlp'], 'llm_int8_threshold': 6.0}
INFO 09-13 13:07:39 [llm_engine.py:228] Initializing a V0 LLM engine (v0.10.0) with config: model='unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit', speculative_config=None, tokenizer='unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.BITSANDBYTES, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=bitsandbytes, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit, num_scheduler_steps=16, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=False, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"inductor","custom_ops":[],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"epilogue_fusion":true,"max_autotune":false,"shape_padding":true,"trace.enabled":false,"triton.cudagraphs":true,"debug":false,"dce":true,"memory_planning":true,"coordinate_descent_tuning":true,"trace.graph_diagram":false,"compile_threads":24,"group_fusion":true,"disable_progress":false,"verbose_progress":true,"triton.multi_kernel":0,"triton.use_block_ptr":true,"triton.enable_persistent_tma_matmul":true,"triton.autotune_at_compile_time":false,"triton.cooperative_reductions":false,"cuda.compile_opt_level":"-O2","cuda.enable_cuda_lto":true,"combo_kernels":false,"benchmark_combo_kernel":true,"combo_kernel_foreach_dynamic_shapes":true,"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":256,"local_cache_dir":null}, use_cached_outputs=False,
WARNING 09-13 13:07:42 [interface.py:380] Using 'pin_memory=False' as WSL is detected. This may slow down the performance.
INFO 09-13 13:07:42 [cuda.py:398] Using Flash Attention backend.
INFO 09-13 13:07:43 [parallel_state.py:1102] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
INFO 09-13 13:07:43 [model_runner.py:1083] Starting to load model unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit...
INFO 09-13 13:07:43 [bitsandbytes_loader.py:733] Loading weights with BitsAndBytes quantization. May take a while ...
INFO 09-13 13:07:45 [weight_utils.py:296] Using model weights format ['*.safetensors']
INFO 09-13 13:07:46 [weight_utils.py:312] Time spent downloading weights for unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit: 0.634731 seconds
INFO 09-13 13:07:46 [weight_utils.py:349] No model.safetensors.index.json found in remote.
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 8.14it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 8.13it/s]
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.58s/it]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.58s/it]
INFO 09-13 13:07:48 [punica_selector.py:19] Using PunicaWrapperGPU.
INFO 09-13 13:07:48 [model_runner.py:1115] Model loading took 2.2549 GiB and 5.103189 seconds
INFO 09-13 13:07:50 [worker.py:295] Memory profiling takes 1.52 seconds
INFO 09-13 13:07:50 [worker.py:295] the current vLLM instance can use total_gpu_memory (23.99GiB) x gpu_memory_utilization (0.74) = 17.71GiB
INFO 09-13 13:07:50 [worker.py:295] model weights take 2.25GiB; non_torch_memory takes 0.08GiB; PyTorch activation peak memory takes 1.42GiB; the rest of the memory reserved for KV Cache is 13.97GiB.
INFO 09-13 13:07:50 [executor_base.py:113] # cuda blocks: 25425, # CPU blocks: 0
INFO 09-13 13:07:50 [executor_base.py:118] Maximum concurrency for 8192 tokens per request: 49.66x
INFO 09-13 13:07:50 [vllm_utils.py:671] Unsloth: Running patched vLLM v0 `capture_model`.
INFO 09-13 13:07:50 [model_runner.py:1385] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
Capturing CUDA graph shapes: 100%|███████████████████████████████████████████████████████████████████████| 35/35 [00:09<00:00, 3.77it/s]
INFO 09-13 13:08:00 [model_runner.py:1537] Graph capturing finished in 10 secs, took 1.37 GiB
INFO 09-13 13:08:00 [vllm_utils.py:678] Unsloth: Patched vLLM v0 graph capture finished in 10 secs.
INFO 09-13 13:08:00 [llm_engine.py:424] init engine (profile, create kv cache, warmup model) took 11.87 seconds
Unsloth: Just some info: will skip parsing ['post_feedforward_layernorm', 'pre_feedforward_layernorm', 'k_norm', 'q_norm']
Unsloth: Just some info: will skip parsing ['post_feedforward_layernorm', 'pre_feedforward_layernorm', 'k_norm', 'q_norm']
Unsloth 2025.8.6 patched 36 layers with 36 QKV layers, 36 O layers and 36 MLP layers.
gather: 0%| | 0/9 [00:00<?, ?it/s]Warning: Traces will not be logged. Call weave.init to log your traces to a project.
(subsequent messages of this type will be suppressed)
gather: 100%|█| 9/9 [00:37<00:00, 4.16s/it, reward=1.05, invalid_move=0.111, max_value=60.9, board_value=154, num_moves=68.3, win=0.222,
[RULER] Pretty-printed LLM choice JSON:
{'scores': []}
Swallowed exception:
Skipping tuning as there is no suitable data. This can happen when all the trajectories in the same group have the same reward and thus no advantage to train on.
Advanced step from 0 to 1 (no training occurred)
gather: 100%|█| 9/9 [00:30<00:00, 3.41s/it, reward=0.831, invalid_move=0.222, max_value=48.7, board_value=139, num_moves=60.2, win=0.111
[RULER] Pretty-printed LLM choice JSON:
{'scores': []}
Swallowed exception:
Skipping tuning as there is no suitable data. This can happen when all the trajectories in the same group have the same reward and thus no advantage to train on.
Advanced step from 1 to 2 (no training occurred)
gather: 100%|█| 9/9 [00:31<00:00, 3.48s/it, reward=1.26, invalid_move=0.333, max_value=76.7, board_value=142, num_moves=63, win=0.444, p
[RULER] Pretty-printed LLM choice JSON:
{'scores': []}
Swallowed exception:
Skipping tuning as there is no suitable data. This can happen when all the trajectories in the same group have the same reward and thus no advantage to train on.
Advanced step from 2 to 3 (no training occurred)
gather: 100%|█| 9/9 [00:33<00:00, 3.67s/it, reward=1.22, invalid_move=0.222, max_value=78.7, board_value=152, num_moves=67.7, win=0.444,
[RULER] Pretty-printed LLM choice JSON:
{'scores': []}
Swallowed exception:
Skipping tuning as there is no suitable data. This can happen when all the trajectories in the same group have the same reward and thus no advantage to train on.
Advanced step from 3 to 4 (no training occurred)
gather: 100%|█| 9/9 [00:31<00:00, 3.45s/it, reward=1.2, invalid_move=0.333, max_value=75.1, board_value=157, num_moves=69.7, win=0.333,
[RULER] Pretty-printed LLM choice JSON:
{'scores': []}
Swallowed exception:
Skipping tuning as there is no suitable data. This can happen when all the trajectories in the same group have the same reward and thus no advantage to train on.
Advanced step from 4 to 5 (no training occurred)
gather: 100%|█| 9/9 [04:51<00:00, 32.42s/it, reward=1.51, invalid_move=0.111, max_value=94.4, board_value=160, num_moves=70, win=0.667, p
[RULER] Pretty-printed LLM choice JSON:
{'scores': []}
Swallowed exception:
Skipping tuning as there is no suitable data. This can happen when all the trajectories in the same group have the same reward and thus no advantage to train on.
Advanced step from 5 to 6 (no training occurred)
gather: 100%|█| 9/9 [00:31<00:00, 3.52s/it, reward=1.15, invalid_move=0.111, max_value=69.6, board_value=163, num_moves=72.6, win=0.333,
[RULER] Pretty-printed LLM choice JSON:
{'scores': []}
Swallowed exception:
Skipping tuning as there is no suitable data. This can happen when all the trajectories in the same group have the same reward and thus no advantage to train on.
Advanced step from 6 to 7 (no training occurred)
gather: 100%|█| 9/9 [00:31<00:00, 3.48s/it, reward=1.27, invalid_move=0.111, max_value=78.2, board_value=187, num_moves=82.6, win=0.333,
[RULER] Pretty-printed LLM choice JSON:
{'scores': []}
Swallowed exception:
Skipping tuning as there is no suitable data. This can happen when all the trajectories in the same group have the same reward and thus no advantage to train on.
Advanced step from 7 to 8 (no training occurred)
gather: 100%|█| 9/9 [04:47<00:00, 31.92s/it, reward=0.736, invalid_move=0.444, max_value=43.6, board_value=94.7, num_moves=41.2, win=0.22
[RULER] Pretty-printed LLM choice JSON:
{'scores': []}
Swallowed exception:
Skipping tuning as there is no suitable data. This can happen when all the trajectories in the same group have the same reward and thus no advantage to train on.
Advanced step from 8 to 9 (no training occurred)
gather: 100%|█| 9/9 [02:23<00:00, 15.95s/it, reward=1.42, invalid_move=0.111, max_value=89.1, board_value=172, num_moves=76.8, win=0.556,
[RULER] Pretty-printed LLM choice JSON:
{'scores': []}
Swallowed exception:
Skipping tuning as there is no suitable data. This can happen when all the trajectories in the same group have the same reward and thus no advantage to train on.
Advanced step from 9 to 10 (no training occurred)
gather: 100%|█| 9/9 [00:36<00:00, 4.06s/it, reward=1.23, invalid_move=0.222, max_value=78.7, board_value=179, num_moves=79, win=0.333, p
[RULER] Pretty-printed LLM choice JSON:
{
'scores': [
{
'trajectory_id': '1',
'explanation': "This trajectory starts with a reasonable move (up) but makes several suboptimal choices. It doesn't make
significant progress toward reaching 2048 and gets stuck in a pattern where it's not efficiently combining tiles. The board state shows
some progress but doesn't achieve the goal.",
'score': 0.25
},
{
'trajectory_id': '2',
'explanation': "This trajectory makes many good moves and achieves a high tile value (64). It shows strategic play with good
combination efforts. However, it doesn't quite reach 2048 (the goal) and gets stuck preventing further growth. It makes progress but
doesn't fully achieve its goal.",
'score': 0.65
},
{
'trajectory_id': '3',
'explanation': 'This trajectory shows excellent strategic planning with many optimal moves, achieving a high tile value (64)
and making significant progress toward 2048. It combines tiles efficiently, maintains good board organization, and makes steady forward
progress throughout.',
'score': 0.85
},
{
'trajectory_id': '4',
'explanation': "This trajectory shows a good attempt to reach 2048 with strategic moves. It achieves several high-value tiles
and demonstrates good board organization efforts. However, it doesn't quite achieve 2048 (the goal) and gets stuck near the end. It makes
substantial progress but doesn't fully complete the task.",
'score': 0.75
},
{
'trajectory_id': '5',
'explanation': "This trajectory shows strategic gameplay and achieves a high tile value (64). It demonstrates good board
organization and moves toward 2048. However, it doesn't quite reach the goal of 2048 and gets stuck in a pattern where further progress
is difficult. It makes good effort but doesn't complete the goal.",
'score': 0.7
},
{
'trajectory_id': '6',
'explanation': 'This trajectory demonstrates strong strategic thinking, efficiently combining tiles and maintaining good
board organization. It achieves a high tile value (64) and makes significant progress toward 2048. The moves are well-considered, showing
good attention to reaching larger numbers.',
'score': 0.8
},
{
'trajectory_id': '7',
'explanation': "This trajectory shows good strategic moves and achieves a high tile value (64), making solid progress toward
2048. It efficiently combines tiles and maintains board organization, though it doesn't quite complete the goal of reaching 2048. It
demonstrates good play but stops short.",
'score': 0.75
},
{
'trajectory_id': '8',
'explanation': 'This trajectory shows excellent strategic planning and reaches a very high tile value (64). It demonstrates
efficient moves, maintains good board organization, and makes strong forward progress toward 2048. It avoids many inefficient detours and
shows consistent movement towards the goal.',
'score': 0.85
},
{
'trajectory_id': '9',
'explanation': '(No explicit final 2048 achievement shown in this trajectory, but it reaches a very high tile value and shows
good strategic play similar to other top performers.)',
'score': 0.8
}
]
}
Packed 9 trajectories into 7 sequences of length 8192
train: 0%| | 0/7 [00:00<?, ?it/s]==((====))== Unsloth - 2x faster free finetuning | Num GPUs used = 1
\\ /| Num examples = 10,000,000 | Num Epochs = 3 | Total steps = 30,000,000
O^O/ \_/ \ Batch size per device = 2 | Gradient accumulation steps = 1
\ / Data Parallel GPUs = 1 | Total batch size (2 x 1 x 1) = 2
"-____-" Trainable parameters = 14,966,784 of 3,100,905,472 (0.48% trained)
Unsloth: Will smartly offload gradients to save VRAM!
train: 100%|███████████████████████████████████| 7/7 [00:34<00:00, 4.89s/it, loss=3.91, grad_norm=4.61, policy_loss=3.91, entropy=0.948]
gather: 11%| | 1/9 [00:15<00:00, 9.63it/s, reward=0.02, invalid_move=1, max_value=2, board_value=4, num_moves=0, win=0, prompt_tokens=1
and i got empty 0000.jsonl, 0001.jsonl, ... along the way
and here some log from lmstudio
2025-09-13 13:08:59 [DEBUG]
Received request: POST to /v1/chat/completions with body {
"messages": [
{
"role": "system",
"content": "\n All of the trajectories below have been g... <Truncated in logs> ...gress towards its goal but does not complete it.\n\n"
},
{
"role": "user",
"content": "<context>\n[{\"content\": \"You are an excellent 2048 ... <Truncated in logs> ...t\", \"content\": \"<move>left</move>\"}]\n</trajectory>"
}
],
"model": "qwen3-coder-30b-a3b-instruct-mlx",
"response_format": {
"type": "json_schema",
"json_schema": {
"schema": {
"$defs": {
"TrajectoryScore": {
"description": "Individual score for a single trajectory.",
"properties": {
"trajectory_id": {
"description": "The id of the trajectory being scored.",
"title": "Trajectory Id",
"type": "string"
},
"explanation": {
"description": "A short description of the trajectory's performance.",
"title": "Explanation",
"type": "string"
},
"score": {
"description": "A score between 0 and 1.",
"title": "Score",
"type": "number"
}
},
"required": [
"trajectory_id",
"explanation",
"score"
],
"title": "TrajectoryScore",
"type": "object",
"additionalProperties": false
}
},
"description": "Response format expected from the LLM judge.",
"properties": {
"scores": {
"description": "The scores for each trajectory.",
"items": {
"$ref": "#/$defs/TrajectoryScore"
},
"title": "Scores",
"type": "array"
}
},
"required": [
"scores"
],
"title": "Response",
"type": "object",
"additionalProperties": false
},
"name": "Response",
"strict": true
}
}
}
2025-09-13 13:08:59 [INFO]
[LM STUDIO SERVER] Running chat completion on conversation with 2 messages.
2025-09-13 13:08:59 [DEBUG]
[cache_wrapper][INFO]: Trimmed 1026 tokens from the prompt cache
2025-09-13 13:10:39 [INFO]
[LM STUDIO SERVER] Client disconnected. Stopping generation... (If the model is busy processing the prompt, it will finish first.)
2025-09-13 13:10:40 [DEBUG]
Received request: POST to /v1/chat/completions with body {
"messages": [
{
"role": "system",
"content": "\n All of the trajectories below have been g... <Truncated in logs> ...gress towards its goal but does not complete it.\n\n"
},
{
"role": "user",
"content": "<context>\n[{\"content\": \"You are an excellent 2048 ... <Truncated in logs> ...t\", \"content\": \"<move>left</move>\"}]\n</trajectory>"
}
],
"model": "qwen3-coder-30b-a3b-instruct-mlx",
"response_format": {
"type": "json_schema",
"json_schema": {
"schema": {
"$defs": {
"TrajectoryScore": {
"description": "Individual score for a single trajectory.",
"properties": {
"trajectory_id": {
"description": "The id of the trajectory being scored.",
"title": "Trajectory Id",
"type": "string"
},
"explanation": {
"description": "A short description of the trajectory's performance.",
"title": "Explanation",
"type": "string"
},
"score": {
"description": "A score between 0 and 1.",
"title": "Score",
"type": "number"
}
},
"required": [
"trajectory_id",
"explanation",
"score"
],
"title": "TrajectoryScore",
"type": "object",
"additionalProperties": false
}
},
"description": "Response format expected from the LLM judge.",
"properties": {
"scores": {
"description": "The scores for each trajectory.",
"items": {
"$ref": "#/$defs/TrajectoryScore"
},
"title": "Scores",
"type": "array"
}
},
"required": [
"scores"
],
"title": "Response",
"type": "object",
"additionalProperties": false
},
"name": "Response",
"strict": true
}
}
}
2025-09-13 13:10:40 [INFO]
[LM STUDIO SERVER] Running chat completion on conversation with 2 messages.
2025-09-13 13:10:40 [DEBUG]
[cache_wrapper][INFO]: Prompt processing was cancelled by the user.
2025-09-13 13:10:40 [INFO]
[qwen3-coder-30b-a3b-instruct-mlx] Model generated tool calls: []
2025-09-13 13:10:40 [INFO]
[qwen3-coder-30b-a3b-instruct-mlx] Generated prediction: {
"id": "chatcmpl-hpwwv0d7w5tfi56l04o3d",
"object": "chat.completion",
"created": 1757743739,
"model": "qwen3-coder-30b-a3b-instruct-mlx",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "",
"tool_calls": []
},
"logprobs": null,
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 40278,
"completion_tokens": 0,
"total_tokens": 40278
},
"stats": {},
"system_fingerprint": "qwen3-coder-30b-a3b-instruct-mlx"
}
2025-09-13 13:11:06 [INFO]
[qwen3-coder-30b-a3b-instruct-mlx] Model generated tool calls: []
2025-09-13 13:11:06 [INFO]
[qwen3-coder-30b-a3b-instruct-mlx] Generated prediction: {
"id": "chatcmpl-s5dzdnoyp9k2ljymdz4gn",
"object": "chat.completion",
"created": 1757743840,
"model": "qwen3-coder-30b-a3b-instruct-mlx",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "{ \"scores\": [ ] }",
"tool_calls": []
},
"logprobs": null,
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 40278,
"completion_tokens": 10,
"total_tokens": 40288
},
"stats": {},
"system_fingerprint": "qwen3-coder-30b-a3b-instruct-mlx"
}
2025-09-13 13:11:40 [DEBUG]
Received request: POST to /v1/chat/completions with body {
"messages": [
{
"role": "system",
"content": "\n All of the trajectories below have been g... <Truncated in logs> ...gress towards its goal but does not complete it.\n\n"
},
{
"role": "user",
"content": "<context>\n[{\"content\": \"You are an excellent 2048 ... <Truncated in logs> ...t\", \"content\": \"<move>down</move>\"}]\n</trajectory>"
}
],
"model": "qwen3-coder-30b-a3b-instruct-mlx",
"response_format": {
"type": "json_schema",
"json_schema": {
"schema": {
"$defs": {
"TrajectoryScore": {
"description": "Individual score for a single trajectory.",
"properties": {
"trajectory_id": {
"description": "The id of the trajectory being scored.",
"title": "Trajectory Id",
"type": "string"
},
"explanation": {
"description": "A short description of the trajectory's performance.",
"title": "Explanation",
"type": "string"
},
"score": {
"description": "A score between 0 and 1.",
"title": "Score",
"type": "number"
}
},
"required": [
"trajectory_id",
"explanation",
"score"
],
"title": "TrajectoryScore",
"type": "object",
"additionalProperties": false
}
},
"description": "Response format expected from the LLM judge.",
"properties": {
"scores": {
"description": "The scores for each trajectory.",
"items": {
"$ref": "#/$defs/TrajectoryScore"
},
"title": "Scores",
"type": "array"
}
},
"required": [
"scores"
],
"title": "Response",
"type": "object",
"additionalProperties": false
},
"name": "Response",
"strict": true
}
}
}
2025-09-13 13:11:40 [INFO]
[LM STUDIO SERVER] Running chat completion on conversation with 2 messages.
2025-09-13 13:11:40 [DEBUG]
[cache_wrapper][INFO]: Trimmed 40009 tokens from the prompt cache
2025-09-13 13:13:20 [INFO]
[LM STUDIO SERVER] Client disconnected. Stopping generation... (If the model is busy processing the prompt, it will finish first.)
2025-09-13 13:13:20 [DEBUG]
[cache_wrapper][INFO]: Prompt processing was cancelled by the user.
2025-09-13 13:13:20 [INFO]
[qwen3-coder-30b-a3b-instruct-mlx] Model generated tool calls: []
2025-09-13 13:13:20 [INFO]
[qwen3-coder-30b-a3b-instruct-mlx] Generated prediction: {
"id": "chatcmpl-lj6bvd1gxam8p0hz2vbq4w",
"object": "chat.completion",
"created": 1757743900,
"model": "qwen3-coder-30b-a3b-instruct-mlx",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "",
"tool_calls": []
},
"logprobs": null,
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 35639,
"completion_tokens": 0,
"total_tokens": 35639
},
"stats": {},
"system_fingerprint": "qwen3-coder-30b-a3b-instruct-mlx"
}
2025-09-13 13:13:21 [DEBUG]
Received request: POST to /v1/chat/completions with body {
"messages": [
{
"role": "system",
"content": "\n All of the trajectories below have been g... <Truncated in logs> ...gress towards its goal but does not complete it.\n\n"
},
{
"role": "user",
"content": "<context>\n[{\"content\": \"You are an excellent 2048 ... <Truncated in logs> ...t\", \"content\": \"<move>down</move>\"}]\n</trajectory>"
}
],
"model": "qwen3-coder-30b-a3b-instruct-mlx",
"response_format": {
"type": "json_schema",
"json_schema": {
"schema": {
"$defs": {
"TrajectoryScore": {
"description": "Individual score for a single trajectory.",
"properties": {
"trajectory_id": {
"description": "The id of the trajectory being scored.",
"title": "Trajectory Id",
"type": "string"
},
"explanation": {
"description": "A short description of the trajectory's performance.",
"title": "Explanation",
"type": "string"
},
"score": {
"description": "A score between 0 and 1.",
"title": "Score",
"type": "number"
}
},
"required": [
"trajectory_id",
"explanation",
"score"
],
"title": "TrajectoryScore",
"type": "object",
"additionalProperties": false
}
},
"description": "Response format expected from the LLM judge.",
"properties": {
"scores": {
"description": "The scores for each trajectory.",
"items": {
"$ref": "#/$defs/TrajectoryScore"
},
"title": "Scores",
"type": "array"
}
},
"required": [
"scores"
],
"title": "Response",
"type": "object",
"additionalProperties": false
},
"name": "Response",
"strict": true
}
}
}
2025-09-13 13:13:21 [INFO]
[LM STUDIO SERVER] Running chat completion on conversation with 2 messages.
2025-09-13 13:13:47 [INFO]
[qwen3-coder-30b-a3b-instruct-mlx] Model generated tool calls: []
2025-09-13 13:13:47 [INFO]
[qwen3-coder-30b-a3b-instruct-mlx] Generated prediction: {
"id": "chatcmpl-042thmf4smygiji7gln2eal",
"object": "chat.completion",
"created": 1757744001,
"model": "qwen3-coder-30b-a3b-instruct-mlx",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "{ \"scores\": [ ] }",
"tool_calls": []
},
"logprobs": null,
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 35639,
"completion_tokens": 9,
"total_tokens": 35648
},
"stats": {},
"system_fingerprint": "qwen3-coder-30b-a3b-instruct-mlx"
}
2025-09-13 13:14:20 [DEBUG]
Received request: POST to /v1/chat/completions with body {
"messages": [
{
"role": "system",
"content": "\n All of the trajectories below have been g... <Truncated in logs> ...gress towards its goal but does not complete it.\n\n"
},
{
"role": "user",
"content": "<context>\n[{\"content\": \"You are an excellent 2048 ... <Truncated in logs> ...t\", \"content\": \"<move>down</move>\"}]\n</trajectory>"
}
],
"model": "qwen3-coder-30b-a3b-instruct-mlx",
"response_format": {
"type": "json_schema",
"json_schema": {
"schema": {
"$defs": {
"TrajectoryScore": {
"description": "Individual score for a single trajectory.",
"properties": {
"trajectory_id": {
"description": "The id of the trajectory being scored.",
"title": "Trajectory Id",
"type": "string"
},
"explanation": {
"description": "A short description of the trajectory's performance.",
"title": "Explanation",
"type": "string"
},
"score": {
"description": "A score between 0 and 1.",
"title": "Score",
"type": "number"
}
},
"required": [
"trajectory_id",
"explanation",
"score"
],
"title": "TrajectoryScore",
"type": "object",
"additionalProperties": false
}
},
"description": "Response format expected from the LLM judge.",
"properties": {
"scores": {
"description": "The scores for each trajectory.",
"items": {
"$ref": "#/$defs/TrajectoryScore"
},
"title": "Scores",
"type": "array"
}
},
"required": [
"scores"
],
"title": "Response",
"type": "object",
"additionalProperties": false
},
"name": "Response",
"strict": true
}
}
}
2025-09-13 13:14:20 [INFO]
[LM STUDIO SERVER] Running chat completion on conversation with 2 messages.
2025-09-13 13:14:20 [DEBUG]
[cache_wrapper][INFO]: Trimmed 35373 tokens from the prompt cache
.
.
.
2025-09-13 13:48:20 [INFO]
[LM STUDIO SERVER] Running chat completion on conversation with 2 messages.
2025-09-13 13:48:20 [DEBUG]
[cache_wrapper][INFO]: Trimmed 233 tokens from the prompt cache
2025-09-13 13:49:19 [INFO]
[qwen3-coder-30b-a3b-instruct-mlx] Model generated tool calls: []
2025-09-13 13:49:19 [INFO]
[qwen3-coder-30b-a3b-instruct-mlx] Generated prediction: {
"id": "chatcmpl-t6qgr5pcxxxxblqhr6qh",
"object": "chat.completion",
"created": 1757746100,
"model": "qwen3-coder-30b-a3b-instruct-mlx",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "{\"scores\": [{\"trajectory_id\": \"1\", \"explanation\": \"This trajectory starts with a reasonable move (up) but makes several suboptimal choices. It doesn't make significant progress toward reaching 2048 and gets stuck in a pattern where it's not efficiently combining tiles. The board state shows some progress but doesn't achieve the goal.\" , \"score\": 0.25}, {\"trajectory_id\": \"2\", \"explanation\": \"This trajectory makes many good moves and achieves a high tile value (64). It shows strategic play with good combination efforts. However, it doesn't quite reach 2048 (the goal) and gets stuck preventing further growth. It makes progress but doesn't fully achieve its goal.\", \"score\": 0.65}, {\"trajectory_id\": \"3\", \"explanation\": \"This trajectory shows excellent strategic planning with many optimal moves, achieving a high tile value (64) and making significant progress toward 2048. It combines tiles efficiently, maintains good board organization, and makes steady forward progress throughout.\", \"score\": 0.85}, {\"trajectory_id\": \"4\", \"explanation\": \"This trajectory shows a good attempt to reach 2048 with strategic moves. It achieves several high-value tiles and demonstrates good board organization efforts. However, it doesn't quite achieve 2048 (the goal) and gets stuck near the end. It makes substantial progress but doesn't fully complete the task.\", \"score\": 0.75}, {\"trajectory_id\": \"5\", \"explanation\": \"This trajectory shows strategic gameplay and achieves a high tile value (64). It demonstrates good board organization and moves toward 2048. However, it doesn't quite reach the goal of 2048 and gets stuck in a pattern where further progress is difficult. It makes good effort but doesn't complete the goal.\", \"score\": 0.7}, {\"trajectory_id\": \"6\", \"explanation\": \"This trajectory demonstrates strong strategic thinking, efficiently combining tiles and maintaining good board organization. It achieves a high tile value (64) and makes significant progress toward 2048. The moves are well-considered, showing good attention to reaching larger numbers.\", \"score\": 0.8}, {\"trajectory_id\": \"7\", \"explanation\": \"This trajectory shows good strategic moves and achieves a high tile value (64), making solid progress toward 2048. It efficiently combines tiles and maintains board organization, though it doesn't quite complete the goal of reaching 2048. It demonstrates good play but stops short.\", \"score\": 0.75}, {\"trajectory_id\": \"8\", \"explanation\": \"This trajectory shows excellent strategic planning and reaches a very high tile value (64). It demonstrates efficient moves, maintains good board organization, and makes strong forward progress toward 2048. It avoids many inefficient detours and shows consistent movement towards the goal.\", \"score\": 0.85}, {\"trajectory_id\": \"9\", \"explanation\": \"(No explicit final 2048 achievement shown in this trajectory, but it reaches a very high tile value and shows good strategic play similar to other top performers.)\", \"score\": 0.8}] }",
"tool_calls": []
},
"logprobs": null,
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 46831,
"completion_tokens": 690,
"total_tokens": 47521
},
"stats": {},
"system_fingerprint": "qwen3-coder-30b-a3b-instruct-mlx"
}
2025-09-13 13:50:28 [DEBUG]
Received request: POST to /v1/chat/completions with body {
"messages": [
{
"role": "system",
"content": "\n All of the trajectories below have been g... <Truncated in logs> ...gress towards its goal but does not complete it.\n\n"
},
{
"role": "user",
"content": "<context>\n[{\"content\": \"You are an excellent 2048 ... <Truncated in logs> ...t\", \"content\": \"<move>left</move>\"}]\n</trajectory>"
}
],
"model": "qwen3-coder-30b-a3b-instruct-mlx",
"response_format": {
"type": "json_schema",
"json_schema": {
"schema": {
"$defs": {
"TrajectoryScore": {
"description": "Individual score for a single trajectory.",
"properties": {
"trajectory_id": {
"description": "The id of the trajectory being scored.",
"title": "Trajectory Id",
"type": "string"
},
"explanation": {
"description": "A short description of the trajectory's performance.",
"title": "Explanation",
"type": "string"
},
"score": {
"description": "A score between 0 and 1.",
"title": "Score",
"type": "number"
}
},
"required": [
"trajectory_id",
"explanation",
"score"
],
"title": "TrajectoryScore",
"type": "object",
"additionalProperties": false
}
},
"description": "Response format expected from the LLM judge.",
"properties": {
"scores": {
"description": "The scores for each trajectory.",
"items": {
"$ref": "#/$defs/TrajectoryScore"
},
"title": "Scores",
"type": "array"
}
},
"required": [
"scores"
],
"title": "Response",
"type": "object",
"additionalProperties": false
},
"name": "Response",
"strict": true
}
}
}
2025-09-13 13:50:28 [INFO]
[LM STUDIO SERVER] Running chat completion on conversation with 2 messages.
2025-09-13 13:50:28 [DEBUG]
[cache_wrapper][INFO]: Trimmed 47248 tokens from the prompt cache
Select a model to configure it
Thanks!