<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Markdown Files — Spark Programming for Data Engineering</title>
<link rel="stylesheet" href="_static/css/index.73d71520a4ca3b99cfee5594769eaaae.css">
<link rel="stylesheet"
href="_static/vendor/fontawesome/5.13.0/css/all.min.css">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
<link rel="stylesheet"
href="_static/vendor/open-sans_all/1.44.1/index.css">
<link rel="stylesheet"
href="_static/vendor/lato_latin-ext/1.44.1/index.css">
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/sphinx-book-theme.40e2e510f6b7d1648584402491bb10fe.css" type="text/css" />
<link rel="stylesheet" type="text/css" href="_static/togglebutton.css" />
<link rel="stylesheet" type="text/css" href="_static/copybutton.css" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.css" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-thebe.css" />
<link rel="stylesheet" type="text/css" href="_static/panels-main.c949a650a448cc0ae9fd3441c0e17fb0.css" />
<link rel="stylesheet" type="text/css" href="_static/panels-variables.06eb56fa6e07937060861dad626602ad.css" />
<link rel="preload" as="script" href="_static/js/index.3da636dd464baa7582d2.js">
<script id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
<script src="_static/jquery.js"></script>
<script src="_static/underscore.js"></script>
<script src="_static/doctools.js"></script>
<script src="_static/togglebutton.js"></script>
<script src="_static/clipboard.min.js"></script>
<script src="_static/copybutton.js"></script>
<script >var togglebuttonSelector = '.toggle, .admonition.dropdown, .tag_hide_input div.cell_input, .tag_hide-input div.cell_input, .tag_hide_output div.cell_output, .tag_hide-output div.cell_output, .tag_hide_cell.cell, .tag_hide-cell.cell';</script>
<script src="_static/sphinx-book-theme.d31b09fe5c1d09cb49b26a786de4a05d.js"></script>
<script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
<script type="text/x-mathjax-config">MathJax.Hub.Config({"tex2jax": {"inlineMath": [["\\(", "\\)"]], "displayMath": [["\\[", "\\]"]], "processRefs": false, "processEnvironments": false}})</script>
<script async="async" src="https://unpkg.com/thebelab@latest/lib/index.js"></script>
<script >
const thebe_selector = ".thebe,.cell"
const thebe_selector_input = "pre,.cell_input div.highlight"
const thebe_selector_output = ".output,.cell_output"
</script>
<script async="async" src="_static/sphinx-thebe.js"></script>
<link rel="canonical" href="https://github.com/ngdeepak/testbook/README.html" />
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Chapter 1 Spark Basics" href="chapters/chapter1-spark-basics.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en" />
<!-- Opengraph tags -->
<meta property="og:url" content="https://github.com/ngdeepak/testbook/README.html" />
<meta property="og:type" content="article" />
<meta property="og:title" content="Markdown Files" />
<meta property="og:description" content="Markdown Files Whether you write your book’s content in Jupyter Notebooks (.ipynb) or in regular markdown files (.md), you’ll write in the same flavor of markd" />
<meta property="og:image" content="https://github.com/ngdeepak/testbook/_static/sparklogo.png" />
<meta name="twitter:card" content="summary" />
</head>
<body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
<div class="container-xl">
<div class="row">
<div class="col-12 col-md-3 bd-sidebar site-navigation show" id="site-navigation">
<div class="navbar-brand-box">
<a class="navbar-brand text-wrap" href="index.html">
<img src="_static/sparklogo.png" class="logo" alt="logo">
<h1 class="site-logo" id="site-title">Spark Programming for Data Engineering</h1>
</a>
</div><form class="bd-search d-flex align-items-center" action="search.html" method="get">
<i class="icon fas fa-search"></i>
<input type="search" class="form-control" name="q" id="search-input" placeholder="Search this book..." aria-label="Search this book..." autocomplete="off" >
</form>
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<ul class="nav sidenav_l1">
<li class="toctree-l1 current active">
<a class="reference internal" href="#">
Markdown Files
</a>
</li>
</ul>
<p class="caption collapsible-parent">
<span class="caption-text">
Chapters
</span>
</p>
<ul class="nav sidenav_l1">
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter1-spark-basics.html">
Chapter 1 Spark Basics
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter1-spark-basics.html#chapter-outline">
Chapter Outline
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter1-spark-basics.html#test">
123test
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter1-spark-basics.html#chpter-learning-objectives">
Chpter Learning Objectives
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter1-spark-basics.html#id1">
test
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter1-spark-basics.html#ashahahakhakhdak">
ashahahakhakhdak
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter2-dataframe.html">
Chapter 1 Basics
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter3-data-types.html">
Chapter 1 Basics
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter4-schema.html">
Chapter 1 Basics
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter5-data-ingestion.html">
Chapter 1 Basics
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter6-user-defined-function.html">
Chapter 1 Basics
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter7-string-column.html">
Chapter 1 Basics
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter8-number-column.html">
Chapter 1 Basics
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter9-date-column.html">
Chapter 1 Basics
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter10-array-column.html">
Chapter 1 Basics
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter11-map-column.html">
Chapter 1 Basics
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter12-struct-column.html">
Chapter 1 Basics
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter13-null-column.html">
Chapter 1 Basics
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter14-columns%20operations.html">
Chapter 1 Basics
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter15-rows-operations.html">
Chapter 1 Basics
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter16-spark-with-pandas.html">
Chapter 1 Basics
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter17-columns%20operations.html">
Chapter 1 Basics
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapters/chapter18-rows-operations.html">
Chapter 1 Basics
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="intro.html">
Jupiter Book
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="test.html">
Creating a Data Frame
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="test.html#twisting-spinning-data-in-apache-spark-interactive-data-engineer-s-guide">
Twisting & Spinning Data in Apache Spark - Interactive Data Engineer’s Guide
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="SparkDataFrame.html">
Spark Data Frame
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="SparkDataFrame.html#creating-a-spark-dataframe">
Creating a Spark DataFrame
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="SparkDataFrame.html#creating-a-spark-dataframe123">
Creating a Spark DataFrame123
</a>
</li>
</ul>
</nav> <!-- To handle the deprecated key -->
<div class="navbar_extra_footer">
Powered by <a href="https://jupyterbook.org">Jupyter Book</a>
</div>
</div>
<main class="col py-md-3 pl-md-4 bd-content overflow-auto" role="main">
<div class="row topbar fixed-top container-xl">
<div class="col-12 col-md-3 bd-topbar-whitespace site-navigation show">
</div>
<div class="col pl-2 topbar-main">
<button id="navbar-toggler" class="navbar-toggler ml-0" type="button" data-toggle="collapse"
data-toggle="tooltip" data-placement="bottom" data-target=".site-navigation" aria-controls="navbar-menu"
aria-expanded="true" aria-label="Toggle navigation" aria-controls="site-navigation"
title="Toggle navigation" data-toggle="tooltip" data-placement="left">
<i class="fas fa-bars"></i>
<i class="fas fa-arrow-left"></i>
<i class="fas fa-arrow-up"></i>
</button>
<div class="dropdown-buttons-trigger">
<button id="dropdown-buttons-trigger" class="btn btn-secondary topbarbtn" aria-label="Download this page"><i
class="fas fa-download"></i></button>
<div class="dropdown-buttons">
<!-- ipynb file if we had a myst markdown file -->
<!-- Download raw file -->
<a class="dropdown-buttons" href="_sources/README.md"><button type="button"
class="btn btn-secondary topbarbtn" title="Download source file" data-toggle="tooltip"
data-placement="left">.md</button></a>
<!-- Download PDF via print -->
<button type="button" id="download-print" class="btn btn-secondary topbarbtn" title="Print to PDF"
onClick="window.print()" data-toggle="tooltip" data-placement="left">.pdf</button>
</div>
</div>
<!-- Source interaction buttons -->
<div class="dropdown-buttons-trigger">
<button id="dropdown-buttons-trigger" class="btn btn-secondary topbarbtn"
aria-label="Connect with source repository"><i class="fab fa-github"></i></button>
<div class="dropdown-buttons sourcebuttons">
<a class="repository-button"
href="https://github.com/ngdeepak/testbook"><button type="button" class="btn btn-secondary topbarbtn"
data-toggle="tooltip" data-placement="left" title="Source repository"><i
class="fab fa-github"></i>repository</button></a>
<a class="issues-button"
href="https://github.com/ngdeepak/testbook/issues/new?title=Issue%20on%20page%20%2FREADME.html&body=Your%20issue%20content%20here."><button
type="button" class="btn btn-secondary topbarbtn" data-toggle="tooltip" data-placement="left"
title="Open an issue"><i class="fas fa-lightbulb"></i>open issue</button></a>
<a class="edit-button" href="https://github.com/ngdeepak/testbook/edit/master/README.md"><button
type="button" class="btn btn-secondary topbarbtn" data-toggle="tooltip" data-placement="left"
title="Edit this page"><i class="fas fa-pencil-alt"></i>suggest edit</button></a>
</div>
</div>
<!-- Full screen (wrap in <a> to have style consistency -->
<a class="full-screen-button"><button type="button" class="btn btn-secondary topbarbtn" data-toggle="tooltip"
data-placement="bottom" onclick="toggleFullScreen()" aria-label="Fullscreen mode"
title="Fullscreen mode"><i
class="fas fa-expand"></i></button></a>
<!-- Launch buttons -->
</div>
<!-- Table of contents -->
<div class="d-none d-md-block col-md-2 bd-toc show">
<div class="tocsection onthispage pt-5 pb-3">
<i class="fas fa-list"></i> Contents
</div>
<nav id="bd-toc-nav">
<ul class="nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#what-is-myst">
What is MyST?
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#what-are-roles-and-directives">
What are roles and directives?
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#using-a-directive">
Using a directive
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#using-a-role">
Using a role
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#adding-a-citation">
Adding a citation
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#executing-code-in-your-markdown-files">
Executing code in your markdown files
</a>
</li>
</ul>
</li>
</ul>
</nav>
</div>
</div>
<div id="main-content" class="row">
<div class="col-12 col-md-9 pl-md-3 pr-md-0">
<div>
<div class="section" id="markdown-files">
<h1>Markdown Files<a class="headerlink" href="#markdown-files" title="Permalink to this headline">¶</a></h1>
<p>Whether you write your book’s content in Jupyter Notebooks (<code class="docutils literal notranslate"><span class="pre">.ipynb</span></code>) or
in regular markdown files (<code class="docutils literal notranslate"><span class="pre">.md</span></code>), you’ll write in the same flavor of markdown
called <strong>MyST Markdown</strong>.</p>
<div class="section" id="what-is-myst">
<h2>What is MyST?<a class="headerlink" href="#what-is-myst" title="Permalink to this headline">¶</a></h2>
<p>MyST stands for “Markedly Structured Text”. It
is a slight variation on a flavor of markdown called “CommonMark” markdown,
with small syntax extensions to allow you to write <strong>roles</strong> and <strong>directives</strong>
in the Sphinx ecosystem.</p>
</div>
<div class="section" id="what-are-roles-and-directives">
<h2>What are roles and directives?<a class="headerlink" href="#what-are-roles-and-directives" title="Permalink to this headline">¶</a></h2>
<p>Roles and directives are two of the most powerful tools in Jupyter Book. They
are kind of like functions, but written in a markup language. They both
serve a similar purpose, but <strong>roles are written in one line</strong>, whereas
<strong>directives span many lines</strong>. They both accept different kinds of inputs,
and what they do with those inputs depends on the specific role or directive
that is being called.</p>
<div class="section" id="using-a-directive">
<h3>Using a directive<a class="headerlink" href="#using-a-directive" title="Permalink to this headline">¶</a></h3>
<p>At its simplest, you can insert a directive into your book’s content like so:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>```{mydirectivename}
My directive content
```
</pre></div>
</div>
<p>This will only work if a directive with name <code class="docutils literal notranslate"><span class="pre">mydirectivename</span></code> already exists
(which it doesn’t). There are many pre-defined directives associated with
Jupyter Book. For example, to insert a note box into your content, you can
use the following directive:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>```{note}
Here is a note
```
</pre></div>
</div>
<p>This results in:</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Here is a note</p>
</div>
<p>In your built book.</p>
<p>For more information on writing directives, see the
<a class="reference external" href="https://myst-parser.readthedocs.io/">MyST documentation</a>.</p>
</div>
<div class="section" id="using-a-role">
<h3>Using a role<a class="headerlink" href="#using-a-role" title="Permalink to this headline">¶</a></h3>
<p>Roles are very similar to directives, but they are less-complex and written
entirely on one line. You can insert a role into your book’s content with
this pattern:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>Some content {rolename}`and here is my role's content!`
</pre></div>
</div>
<p>Again, roles will only work if <code class="docutils literal notranslate"><span class="pre">rolename</span></code> is a valid role’s name. For example,
the <code class="docutils literal notranslate"><span class="pre">doc</span></code> role can be used to refer to another page in your book. You can
refer directly to another page by its relative path. For example, the
role syntax <code class="docutils literal notranslate"><span class="pre">{doc}`intro`</span></code> will result in: <a class="reference internal" href="intro.html"><span class="doc">Jupiter Book</span></a>.</p>
<p>For more information on writing roles, see the
<a class="reference external" href="https://myst-parser.readthedocs.io/">MyST documentation</a>.</p>
</div>
<div class="section" id="adding-a-citation">
<h3>Adding a citation<a class="headerlink" href="#adding-a-citation" title="Permalink to this headline">¶</a></h3>
<p>You can also cite references that are stored in a <code class="docutils literal notranslate"><span class="pre">bibtex</span></code> file. For example,
the following syntax: <code class="docutils literal notranslate"><span class="pre">{cite}`holdgraf_evidence_2014`</span></code> will render like
this: <span class="bibtex" id="id1">[holdgraf_evidence_2014]</span>.</p>
<p>Moreoever, you can insert a bibliography into your page with this syntax:
The <code class="docutils literal notranslate"><span class="pre">{bibliography}</span></code> directive must be used for all the <code class="docutils literal notranslate"><span class="pre">{cite}</span></code> roles to
render properly.
For example, if the references for your book are stored in <code class="docutils literal notranslate"><span class="pre">references.bib</span></code>,
then the bibliography is inserted with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>```{bibliography} references.bib
```
</pre></div>
</div>
<p>Resulting in a rendered bibliography that looks like:</p>
<p id="bibtex-bibliography-README-0"></p>
</div>
<div class="section" id="executing-code-in-your-markdown-files">
<h3>Executing code in your markdown files<a class="headerlink" href="#executing-code-in-your-markdown-files" title="Permalink to this headline">¶</a></h3>
<p>If you’d like to include computational content inside these markdown files,
you can use MyST Markdown to define cells that will be executed when your
book is built. Jupyter Book uses <em>jupytext</em> to do this.</p>
<p>First, add Jupytext metadata to the file. For example, to add Jupytext metadata
to this markdown page, run this command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">jupyter</span><span class="o">-</span><span class="n">book</span> <span class="n">myst</span> <span class="n">init</span> <span class="n">markdown</span><span class="o">.</span><span class="n">md</span>
</pre></div>
</div>
<p>Once a markdown file has Jupytext metadata in it, you can add the following
directive to run the code at build time:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>```{code-cell}
print("Here is some code to execute")
```
</pre></div>
</div>
<p>When your book is built, the contents of any <code class="docutils literal notranslate"><span class="pre">{code-cell}</span></code> blocks will be
executed with your default Jupyter kernel, and their outputs will be displayed
in-line with the rest of your content.</p>
<p>For more information about executing computational content with Jupyter Book,
see <a class="reference external" href="https://myst-nb.readthedocs.io/">The MyST-NB documentation</a>.</p>
<div class="toctree-wrapper compound">
</div>
</div>
</div>
</div>
<script type="text/x-thebe-config">
{
requestKernel: true,
binderOptions: {
repo: "ngdeepak/testbook",
ref: "master",
},
codeMirrorConfig: {
theme: "abcdef",
mode: "python"
},
kernelOptions: {
kernelName: "python3",
path: "./."
},
predefinedOutput: true
}
</script>
<script>kernelName = 'python3'</script>
</div>
</div>
</div>
<div class='prev-next-bottom'>
<a class='right-next' id="next-link" href="chapters/chapter1-spark-basics.html" title="next page">Chapter 1 Spark Basics</a>
</div>
<footer class="footer mt-5 mt-md-0">
<div class="container">
<p>
By Deepak Gowda<br/>
© Copyright 2020.<br/>
</p>
</div>
</footer>
</main>
</div>
</div>
<script src="_static/js/index.3da636dd464baa7582d2.js"></script>
</body>
</html>