Unable to transliterate from Roman to other Indic langauges other than Hindi

[Chaitanya-Jadhav]

Hi Team,
I am trying to transliterate from Roman to Tamil.
However, the model is only transliterating to Hindi even though I am passing target language code as ta
fairseq-interactive corpus-bin --path transformer/indicxlit.pt --task translation_multi_simple_epoch --beam 10 --nbest 10 --source-lang en --target-lang ta --encoder-langtok src --lang-dict lang_list.txt --input source.txt --lang-pairs as-en,bn-en,brx-en,gom-en,gu-en,hi-en,kn-en,ks-en,mai-en,ml-en,mni-en,mr-en,ne-en,or-en,pa-en,sa-en,sd-en,si-en,ta-en,te-en,ur-en

/usr/local/lib/python3.8/dist-packages/torch/cuda/init.py:497: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
2023-01-20 07:12:21 | INFO | fairseq.tasks.text_to_speech | Please install tensorboardX: pip install tensorboardX
2023-01-20 07:12:23 | INFO | fairseq_cli.interactive | {'_name': None, 'common': {'_name': None, 'no_progress_bar': False, 'log_interval': 100, 'log_format': None, 'log_file': None, 'aim_repo': None, 'aim_run_hash': None, 'tensorboard_logdir': None, 'wandb_project': None, 'azureml_logging': False, 'seed': 1, 'cpu': False, 'tpu': False, 'bf16': False, 'memory_efficient_bf16': False, 'fp16': False, 'memory_efficient_fp16': False, 'fp16_no_flatten_grads': False, 'fp16_init_scale': 128, 'fp16_scale_window': None, 'fp16_scale_tolerance': 0.0, 'on_cpu_convert_precision': False, 'min_loss_scale': 0.0001, 'threshold_loss_scale': None, 'amp': False, 'amp_batch_retries': 2, 'amp_init_scale': 128, 'amp_scale_window': None, 'user_dir': None, 'empty_cache_freq': 0, 'all_gather_list_size': 16384, 'model_parallel_size': 1, 'quantization_config_path': None, 'profile': False, 'reset_logging': False, 'suppress_crashes': False, 'use_plasma_view': False, 'plasma_path': '/tmp/plasma'}, 'common_eval': {'_name': None, 'path': 'transformer/indicxlit.pt', 'post_process': None, 'quiet': False, 'model_overrides': '{}', 'results_path': None}, 'distributed_training': {'_name': None, 'distributed_world_size': 1, 'distributed_num_procs': 1, 'distributed_rank': 0, 'distributed_backend': 'nccl', 'distributed_init_method': None, 'distributed_port': -1, 'device_id': 0, 'distributed_no_spawn': False, 'ddp_backend': 'pytorch_ddp', 'ddp_comm_hook': 'none', 'bucket_cap_mb': 25, 'fix_batches_to_gpus': False, 'find_unused_parameters': False, 'gradient_as_bucket_view': False, 'fast_stat_sync': False, 'heartbeat_timeout': -1, 'broadcast_buffers': False, 'slowmo_momentum': None, 'slowmo_base_algorithm': 'localsgd', 'localsgd_frequency': 3, 'nprocs_per_node': 1, 'pipeline_model_parallel': False, 'pipeline_balance': None, 'pipeline_devices': None, 'pipeline_chunks': 0, 'pipeline_encoder_balance': None, 'pipeline_encoder_devices': None, 'pipeline_decoder_balance': None, 'pipeline_decoder_devices': None, 'pipeline_checkpoint': 'never', 'zero_sharding': 'none', 'fp16': False, 'memory_efficient_fp16': False, 'tpu': False, 'no_reshard_after_forward': False, 'fp32_reduce_scatter': False, 'cpu_offload': False, 'use_sharded_state': False, 'not_fsdp_flatten_parameters': False}, 'dataset': {'_name': None, 'num_workers': 1, 'skip_invalid_size_inputs_valid_test': False, 'max_tokens': None, 'batch_size': 1, 'required_batch_size_multiple': 8, 'required_seq_len_multiple': 1, 'dataset_impl': None, 'data_buffer_size': 10, 'train_subset': 'train', 'valid_subset': 'valid', 'combine_valid_subsets': None, 'ignore_unused_valid_subsets': False, 'validate_interval': 1, 'validate_interval_updates': 0, 'validate_after_updates': 0, 'fixed_validation_seed': None, 'disable_validation': False, 'max_tokens_valid': None, 'batch_size_valid': None, 'max_valid_steps': None, 'curriculum': 0, 'gen_subset': 'test', 'num_shards': 1, 'shard_id': 0, 'grouped_shuffling': False, 'update_epoch_batch_itr': False, 'update_ordered_indices_seed': False}, 'optimization': {'_name': None, 'max_epoch': 0, 'max_update': 0, 'stop_time_hours': 0.0, 'clip_norm': 0.0, 'sentence_avg': False, 'update_freq': [1], 'lr': [0.25], 'stop_min_lr': -1.0, 'use_bmuf': False, 'skip_remainder_batch': False, 'debug_param_names': False}, 'checkpoint': {'_name': None, 'save_dir': 'checkpoints', 'restore_file': 'checkpoint_last.pt', 'continue_once': None, 'finetune_from_model': None, 'reset_dataloader': False, 'reset_lr_scheduler': False, 'reset_meters': False, 'reset_optimizer': False, 'optimizer_overrides': '{}', 'save_interval': 1, 'save_interval_updates': 0, 'keep_interval_updates': -1, 'keep_interval_updates_pattern': -1, 'keep_last_epochs': -1, 'keep_best_checkpoints': -1, 'no_save': False, 'no_epoch_checkpoints': False, 'no_last_checkpoints': False, 'no_save_optimizer_state': False, 'best_checkpoint_metric': 'loss', 'maximize_best_checkpoint_metric': False, 'patience': -1, 'checkpoint_suffix': '', 'checkpoint_shard_count': 1, 'load_checkpoint_on_all_dp_ranks': False, 'write_checkpoints_asynchronously': False, 'model_parallel_size': 1}, 'bmuf': {'_name': None, 'block_lr': 1.0, 'block_momentum': 0.875, 'global_sync_iter': 50, 'warmup_iterations': 500, 'use_nbm': False, 'average_sync': False, 'distributed_world_size': 1}, 'generation': {'_name': None, 'beam': 10, 'beam_mt': 0, 'nbest': 10, 'max_len_a': 0.0, 'max_len_b': 200, 'max_len_a_mt': 0.0, 'max_len_b_mt': 200, 'min_len': 1, 'match_source_len': False, 'unnormalized': False, 'no_early_stop': False, 'no_beamable_mm': False, 'lenpen': 1.0, 'lenpen_mt': 1.0, 'unkpen': 0.0, 'replace_unk': None, 'sacrebleu': False, 'score_reference': False, 'prefix_size': 0, 'no_repeat_ngram_size': 0, 'sampling': False, 'sampling_topk': -1, 'sampling_topp': -1.0, 'constraints': None, 'temperature': 1.0, 'diverse_beam_groups': -1, 'diverse_beam_strength': 0.5, 'diversity_rate': -1.0, 'print_alignment': None, 'print_step': False, 'lm_path': None, 'lm_weight': 0.0, 'iter_decode_eos_penalty': 0.0, 'iter_decode_max_iter': 10, 'iter_decode_force_max_iter': False, 'iter_decode_with_beam': 1, 'iter_decode_with_external_reranker': False, 'retain_iter_history': False, 'retain_dropout': False, 'retain_dropout_modules': None, 'decoding_format': None, 'no_seed_provided': False, 'eos_token': None}, 'eval_lm': {'_name': None, 'output_word_probs': False, 'output_word_stats': False, 'context_window': 0, 'softmax_batch': 9223372036854775807}, 'interactive': {'_name': None, 'buffer_size': 1, 'input': 'source/source.txt'}, 'model': None, 'task': Namespace(_name='translation_multi_simple_epoch', aim_repo=None, aim_run_hash=None, all_gather_list_size=16384, amp=False, amp_batch_retries=2, amp_init_scale=128, amp_scale_window=None, azureml_logging=False, batch_size=None, batch_size_valid=None, beam=10, beam_mt=0, best_checkpoint_metric='loss', bf16=False, bpe=None, broadcast_buffers=False, bucket_cap_mb=25, buffer_size=0, checkpoint_shard_count=1, checkpoint_suffix='', combine_valid_subsets=None, constraints=None, continue_once=None, cpu=False, cpu_offload=False, criterion='cross_entropy', curriculum=0, data='corpus-bin', data_buffer_size=10, dataset_impl=None, ddp_backend='pytorch_ddp', ddp_comm_hook='none', decoder_langtok=False, decoding_format=None, device_id=0, disable_validation=False, distributed_backend='nccl', distributed_init_method=None, distributed_no_spawn=False, distributed_num_procs=1, distributed_port=-1, distributed_rank=0, distributed_world_size=1, diverse_beam_groups=-1, diverse_beam_strength=0.5, diversity_rate=-1.0, empty_cache_freq=0, enable_lang_ids=False, enable_reservsed_directions_shared_datasets=False, encoder_langtok='src', eos=2, eos_token=None, extra_data=None, extra_lang_pairs=None, fast_stat_sync=False, find_unused_parameters=False, finetune_from_model=None, fix_batches_to_gpus=False, fixed_dictionary=None, fixed_validation_seed=None, force_anneal=None, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, fp32_reduce_scatter=False, gen_subset='test', gradient_as_bucket_view=False, grouped_shuffling=False, heartbeat_timeout=-1, ignore_unused_valid_subsets=False, input='source/source.txt', iter_decode_eos_penalty=0.0, iter_decode_force_max_iter=False, iter_decode_max_iter=10, iter_decode_with_beam=1, iter_decode_with_external_reranker=False, keep_best_checkpoints=-1, keep_inference_langtok=False, keep_interval_updates=-1, keep_interval_updates_pattern=-1, keep_last_epochs=-1, lang_dict='lang_list.txt', lang_pairs='as-en,bn-en,brx-en,gom-en,gu-en,hi-en,kn-en,ks-en,mai-en,ml-en,mni-en,mr-en,ne-en,or-en,pa-en,sa-en,sd-en,si-en,ta-en,te-en,ur-en', lang_tok_replacing_bos_eos=False, lang_tok_style='multilingual', langs=None, langtoks=None, langtoks_specs=['main'], left_pad_source='True', left_pad_target='False', lenpen=1, lenpen_mt=1, lm_path=None, lm_weight=0.0, load_alignments=False, load_checkpoint_on_all_dp_ranks=False, localsgd_frequency=3, log_file=None, log_format=None, log_interval=100, lr_scheduler='fixed', lr_shrink=0.1, match_source_len=False, max_len_a=0, max_len_a_mt=0, max_len_b=200, max_len_b_mt=200, max_source_positions=1024, max_target_positions=1024, max_tokens=None, max_tokens_valid=None, max_valid_steps=None, maximize_best_checkpoint_metric=False, memory_efficient_bf16=False, memory_efficient_fp16=False, min_len=1, min_loss_scale=0.0001, model_overrides='{}', model_parallel_size=1, nbest=10, no_beamable_mm=False, no_early_stop=False, no_epoch_checkpoints=False, no_last_checkpoints=False, no_progress_bar=False, no_repeat_ngram_size=0, no_reshard_after_forward=False, no_save=False, no_save_optimizer_state=False, no_seed_provided=False, not_fsdp_flatten_parameters=False, nprocs_per_node=1, num_shards=1, num_workers=1, on_cpu_convert_precision=False, optimizer=None, optimizer_overrides='{}', pad=1, path='transformer/indicxlit.pt', patience=-1, pipeline_balance=None, pipeline_checkpoint='never', pipeline_chunks=0, pipeline_decoder_balance=None, pipeline_decoder_devices=None, pipeline_devices=None, pipeline_encoder_balance=None, pipeline_encoder_devices=None, pipeline_model_parallel=False, plasma_path='/tmp/plasma', post_process=None, prefix_size=0, print_alignment=None, print_step=False, profile=False, quantization_config_path=None, quiet=False, replace_unk=None, required_batch_size_multiple=8, required_seq_len_multiple=1, reset_dataloader=False, reset_logging=False, reset_lr_scheduler=False, reset_meters=False, reset_optimizer=False, restore_file='checkpoint_last.pt', results_path=None, retain_dropout=False, retain_dropout_modules=None, retain_iter_history=False, sacrebleu=False, sampling=False, sampling_method='concat', sampling_temperature=1.5, sampling_topk=-1, sampling_topp=-1.0, sampling_weights=None, sampling_weights_from_file=None, save_dir='checkpoints', save_interval=1, save_interval_updates=0, score_reference=False, scoring='bleu', seed=1, shard_id=0, simul_type=None, skip_invalid_size_inputs_valid_test=False, slowmo_base_algorithm='localsgd', slowmo_momentum=None, source_dict=None, source_lang='en', suppress_crashes=False, target_dict=None, target_lang='ta', task='translation_multi_simple_epoch', temperature=1.0, tensorboard_logdir=None, threshold_loss_scale=None, tokenizer=None, tpu=False, train_subset='train', truncate_source=False, unk=3, unkpen=0, unnormalized=False, update_epoch_batch_itr=False, update_ordered_indices_seed=False, upsample_primary=1, use_plasma_view=False, use_sharded_state=False, user_dir=None, valid_subset='valid', validate_after_updates=0, validate_interval=1, validate_interval_updates=0, virtual_data_size=None, virtual_epoch_size=None, wandb_project=None, warmup_updates=0, write_checkpoints_asynchronously=False, zero_sharding='none'), 'criterion': {'_name': 'cross_entropy', 'sentence_avg': True}, 'optimizer': None, 'lr_scheduler': {'_name': 'fixed', 'force_anneal': None, 'lr_shrink': 0.1, 'warmup_updates': 0, 'lr': [0.25]}, 'scoring': {'_name': 'bleu', 'pad': 1, 'eos': 2, 'unk': 3}, 'bpe': None, 'tokenizer': None, 'ema': {'_name': None, 'store_ema': False, 'ema_decay': 0.9999, 'ema_start_update': 0, 'ema_seed_model': None, 'ema_update_freq': 1, 'ema_fp32': False}, 'simul_type': None}
2023-01-20 07:12:23 | INFO | fairseq.data.multilingual.multilingual_data_manager | loaded language list from lang_list.txt as they are ordered in file
2023-01-20 07:12:23 | INFO | fairseq.data.multilingual.multilingual_data_manager | [en] dictionary: 54 types
2023-01-20 07:12:23 | INFO | fairseq.data.multilingual.multilingual_data_manager | [ta] dictionary: 806 types
2023-01-20 07:12:23 | INFO | fairseq_cli.interactive | loading model(s) from transformer/indicxlit.pt
2023-01-20 07:12:24 | INFO | fairseq_cli.interactive | NOTE: hypothesis and token scores are output in base 2
2023-01-20 07:12:24 | INFO | fairseq_cli.interactive | Type the input sentence and press return:
2023-01-20 07:12:24 | INFO | fairseq.tasks.translation_multi_simple_epoch | start batch sampler: mem usage: used=997.40625Mb; avail=11672.80078125Mb
2023-01-20 07:12:24 | INFO | fairseq.tasks.translation_multi_simple_epoch | [None] @batch_sampler order indices time: 0:00:00.000790
2023-01-20 07:12:24 | INFO | fairseq.tasks.translation_multi_simple_epoch | mem usage: used=997.40625Mb; avail=11672.80078125Mb
2023-01-20 07:12:24 | INFO | fairseq.tasks.translation_multi_simple_epoch | [None] @batch_sampler filter_by_size time: 0:00:00.000076
2023-01-20 07:12:24 | INFO | fairseq.tasks.translation_multi_simple_epoch | mem usage: used=997.40625Mb; avail=11672.80078125Mb
2023-01-20 07:12:24 | INFO | fairseq.tasks.translation_multi_simple_epoch | [None] @batch_sampler batch_by_size time: 0:00:00.000985
2023-01-20 07:12:24 | INFO | fairseq.tasks.translation_multi_simple_epoch | [None] per epoch batch_sampler set-up time: 0:00:00.002610
2023-01-20 07:12:24 | INFO | fairseq.tasks.translation_multi_simple_epoch | mem usage: used=997.40625Mb; avail=11672.80078125Mb
S-0 en s k y b a g
W-0 0.191 seconds
H-0 -0.23068146407604218 स ् क ा य ब ॅ ग
D-0 -0.23068146407604218 स ् क ा य ब ॅ ग
P-0 -0.0003 -0.0002 -0.0004 -0.1929 -0.5797 -0.0003 -1.3016 -0.0003 -0.0003
H-0 -0.2412191480398178 स ् क ा य ब ै ग
D-0 -0.2412191480398178 स ् क ा य ब ै ग
P-0 -0.0003 -0.0002 -0.0004 -0.1929 -0.5797 -0.0003 -1.3965 -0.0002 -0.0005
H-0 -0.3573142886161804 स ् क ा ई ब ै ग
D-0 -0.3573142886161804 स ् क ा ई ब ै ग
P-0 -0.0003 -0.0002 -0.0004 -0.1929 -1.8929 -0.0001 -1.1284 -0.0002 -0.0004
H-0 -0.4048730432987213 स ् क ा ई ब ॅ ग
D-0 -0.4048730432987213 स ् क ा ई ब ॅ ग
P-0 -0.0003 -0.0002 -0.0004 -0.1929 -1.8929 -0.0001 -1.5565 -0.0003 -0.0003
H-0 -0.5313842296600342 स ् क े ब ै ग
D-0 -0.5313842296600342 स ् क े ब ै ग
P-0 -0.0003 -0.0002 -0.0004 -3.0147 -0.0006 -1.2342 -0.0002 -0.0005
H-0 -0.5332987308502197 स ् क े ब ॅ ग
D-0 -0.5332987308502197 स ् क े ब ॅ ग
P-0 -0.0003 -0.0002 -0.0004 -3.0147 -0.0006 -1.2494 -0.0004 -0.0004
H-0 -0.5979421734809875 स ् क ा य ब े ग
D-0 -0.5979421734809875 स ् क ा य ब े ग
P-0 -0.0003 -0.0002 -0.0004 -0.1929 -0.5797 -0.0003 -4.6069 -0.0002 -0.0006
H-0 -0.6184923052787781 स ् क ा इ ब ै ग
D-0 -0.6184923052787781 स ् क ा इ ब ै ग
P-0 -0.0003 -0.0002 -0.0004 -0.1929 -4.0428 -0.0001 -1.3291 -0.0002 -0.0006
H-0 -1.4913733005523682 स ् क ा य ब ै ग ्
D-0 -1.4913733005523682 स ् क ा य ब ै ग ्
P-0 -0.0003 -0.0002 -0.0004 -0.1929 -0.5797 -0.0003 -1.3965 -0.0002 -12.7172 -0.0260
H-0 -1.6366398334503174 स ् क ा य ब ॅ क
D-0 -1.6366398334503174 स ् क ा य ब ॅ क
P-0 -0.0003 -0.0002 -0.0004 -0.1929 -0.5797 -0.0003 -1.3016 -12.6541 -0.0001
2023-01-20 07:12:24 | INFO | fairseq_cli.interactive | Total time: 0.770 seconds; translation time: 0.191

Could you please help me understanding why is it happening so?

[yashmadhani97]

Hi @Chaitanya-Jadhav , Please pass encoder-langtok "tgt" instead of encoder-langtok "src" in fairseq-interactive command.

[Chaitanya-Jadhav]

Thank you @yashmadhani97. It worked for me!