Line Segmentation is an important phase in the digitization of the historical documents for downstream text recognition. It is a challenging task for palm leaves which varies with degradation of the medium, noise at the background and overlapping of lines and wide variety of handwriting style. Traditional rule-based methods of pre-processing work well for historical documents and deep learning based methods for historical documents for line segmentation approach do not carry over to deep learning based Vision Transformers approach for palm leaves. In our proposed work, palm leaf chunks are line are pre-annotated with unique color code, which are treated as objects to leverage the ability of Segformer, a multi head attention mechanism Vision Transformer methodology based segmentation framework. A major benefit of Segformer is that lines are pre-defined through a line annotation tool called CVAT which allows unwanted noise to be ignored and grasping in-detail textual information.