Python plugin/extra to load data files from an external source (such as AWS S3) to a local directory

<!DOCTYPE html> <html class=”writer-html5” lang=”en” > <head>

<meta charset=”utf-8” /><meta name=”generator” content=”Docutils 0.17.1: http://docutils.sourceforge.net/” />

<meta name=”viewport” content=”width=device-width, initial-scale=1.0” /> <title>Data Loader Plugin - Python — data-loader-plugin 0.0.1 documentation</title>

<link rel=”stylesheet” href=”_static/pygments.css” type=”text/css” /> <link rel=”stylesheet” href=”_static/css/theme.css” type=”text/css” />

<!–[if lt IE 9]>
<script src=”_static/js/html5shiv.min.js”></script>

<![endif]–>

<script data-url_root=”./” id=”documentation_options” src=”_static/documentation_options.js”></script> <script src=”_static/jquery.js”></script> <script src=”_static/underscore.js”></script> <script src=”_static/doctools.js”></script>

<script src=”_static/js/theme.js”></script> <link rel=”index” title=”Index” href=”genindex.html” /> <link rel=”search” title=”Search” href=”search.html” />

</head>

<body class=”wy-body-for-nav”>

<div class=”wy-grid-for-nav”>

<nav data-toggle=”wy-nav-shift” class=”wy-nav-side”>

<div class=”wy-side-scroll”>

<div class=”wy-side-nav-search” >

<a href=”#” class=”icon icon-home”> data-loader-plugin

</a>

<div class=”version”>: 0.0.1

</div>

<div role=”search”>

<form id=”rtd-search-form” class=”wy-form” action=”search.html” method=”get”>: <input type=”text” name=”q” placeholder=”Search docs” /> <input type=”hidden” name=”check_keywords” value=”yes” /> <input type=”hidden” name=”area” value=”default” />

</form>

</div>

</div><div class=”wy-menu wy-menu-vertical” data-spy=”affix” role=”navigation” aria-label=”Navigation menu”>: <!– Local TOC –> <div class=”local-toc”><ul>

<li><a class=”reference internal” href=”#”>Data Loader Plugin - Python</a></li> <li><a class=”reference internal” href=”#table-of-content-toc”>Table of Content (ToC)</a></li> <li><a class=”reference internal” href=”#overview”>Overview</a></li> <li><a class=”reference internal” href=”#references”>References</a><ul> <li><a class=”reference internal” href=”#python-module”>Python module</a></li> <li><a class=”reference internal” href=”#python-virtual-environments”>Python virtual environments</a></li> </ul> </li> <li><a class=”reference internal” href=”#installation”>Installation</a><ul> <li><a class=”reference internal” href=”#clone-this-git-repository”>Clone this Git repository</a></li> <li><a class=”reference internal” href=”#python-environment”>Python environment</a></li> </ul> </li> <li><a class=”reference internal” href=”#usage”>Usage</a><ul> <li><a class=”reference internal” href=”#install-the-data-loader-plugin-module”>Install the <code class=”docutils literal notranslate”><span class=”pre”>data-loader-plugin</span></code> module</a><ul> <li><a class=”reference internal” href=”#install-in-the-python-user-space”>Install in the Python user space</a></li> <li><a class=”reference internal” href=”#installation-in-a-dedicated-python-virtual-environment”>Installation in a dedicated Python virtual environment</a></li> </ul> </li> <li><a class=”reference internal” href=”#use-data-loader-plugin-as-a-module-from-another-python-program”>Use <code class=”docutils literal notranslate”><span class=”pre”>data-loader-plugin</span></code> as a module from another Python program</a></li> </ul> </li> <li><a class=”reference internal” href=”#development-contribution”>Development / Contribution</a><ul> <li><a class=”reference internal” href=”#test-the-data-loader-plugin-python-module”>Test the data loader plugin Python module</a></li> </ul> </li> </ul> </div>

</div>

</div>

</nav>

<section data-toggle=”wy-nav-shift” class=”wy-nav-content-wrap”><nav class=”wy-nav-top” aria-label=”Mobile navigation menu” >

<i data-toggle=”wy-nav-top” class=”fa fa-bars”></i> <a href=”#”>data-loader-plugin</a>

</nav>

<div class=”wy-nav-content”>

<div class=”rst-content”>
<div role=”navigation” aria-label=”Page navigation”>

<ul class=”wy-breadcrumbs”>
<li><a href=”#” class=”icon icon-home”></a> »</li> <li>Data Loader Plugin - Python</li> <li class=”wy-breadcrumbs-aside”>

<a href=”_sources/README.md.txt” rel=”nofollow”> View page source</a>

</li>

</ul> <hr/>

</div>

<div role=”main” class=”document” itemscope=”itemscope” itemtype=”http://schema.org/Article”>
<div itemprop=”articleBody”>

<h1>Data Loader Plugin - Python<a class=”headerlink” href=”#data-loader-plugin-python” title=”Permalink to this headline”></a></h1> </section> <section class=”tex2jax_ignore mathjax_ignore” id=”table-of-content-toc”> <h1>Table of Content (ToC)<a class=”headerlink” href=”#table-of-content-toc” title=”Permalink to this headline”></a></h1> <ul class=”simple”> <li><p><a class=”reference external” href=”#data-loader-plugin-python”>Data Loader Plugin - Python</a></p></li> <li><p><a class=”reference external” href=”#table-of-content-toc”>Table of Content (ToC)</a></p></li> <li><p><a class=”reference external” href=”#overview”>Overview</a></p></li> <li><p><a class=”reference external” href=”#references”>References</a></p> <ul> <li><p><a class=”reference external” href=”#python-module”>Python module</a></p></li> <li><p><a class=”reference external” href=”#python-virtual-environments”>Python virtual environments</a></p></li> </ul> </li> <li><p><a class=”reference external” href=”#installation”>Installation</a></p> <ul> <li><p><a class=”reference external” href=”#clone-this-git-repository”>Clone this Git repository</a></p></li> <li><p><a class=”reference external” href=”#python-environment”>Python environment</a></p></li> </ul> </li> <li><p><a class=”reference external” href=”#usage”>Usage</a></p> <ul> <li><p><a class=”reference external” href=”#install-the-data-loader-plugin-module”>Install the <code class=”docutils literal notranslate”><span class=”pre”>data-loader-plugin</span></code> module</a></p> <ul> <li><p><a class=”reference external” href=”#install-in-the-python-user-space”>Install in the Python user space</a></p></li> <li><p><a class=”reference external” href=”#installation-in-a-dedicated-python-virtual-environment”>Installation in a dedicated Python virtual environment</a></p></li> </ul> </li> <li><p><a class=”reference external” href=”#use-data-loader-plugin-as-a-module-from-another-python-program”>Use <code class=”docutils literal notranslate”><span class=”pre”>data-loader-plugin</span></code> as a module from another Python program</a></p></li> </ul> </li> <li><p><a class=”reference external” href=”#development-contribution”>Development / Contribution</a></p> <ul> <li><p><a class=”reference external” href=”#test-the-data-loader-plugin-python-module”>Test the data loader plugin Python module</a></p></li> </ul> </li> </ul> <p><small><i><a href=’http://ecotrust-canada.github.io/markdown-toc/’>Table of contents generated with markdown-toc</a></i></small></p> </section> <section class=”tex2jax_ignore mathjax_ignore” id=”overview”> <h1>Overview<a class=”headerlink” href=”#overview” title=”Permalink to this headline”></a></h1> <p>The <a class=”reference external” href=”https://github.com/cloud-helpers/python-plugin-data-loader”>data loader plugin</a>, aims at supporting running programs (<em>e.g.</em>, API service backends) when downloading data from cloud services such as <a class=”reference external” href=”https://aws.amazon.com/s3”>AWS S3</a>. It provides a base Python library, namely <code class=”docutils literal notranslate”><span class=”pre”>data-loader-plugin</span></code>, offering a few methods to download data files from AWS S3.</p> </section> <section class=”tex2jax_ignore mathjax_ignore” id=”references”> <h1>References<a class=”headerlink” href=”#references” title=”Permalink to this headline”></a></h1> <section id=”python-module”> <h2>Python module<a class=”headerlink” href=”#python-module” title=”Permalink to this headline”></a></h2> <ul class=”simple”> <li><p>GitHub: https://github.com/cloud-helpers/python-plugin-data-loader/tree/master/data_loader_plugin</p></li> <li><p>PyPi: https://pypi.org/project/data-loader-plugin/</p></li> <li><p>Read the Docs (RTD): https://readthedocs.org/projects/data-loader-plugin/</p></li> </ul> </section> <section id=”python-virtual-environments”> <h2>Python virtual environments<a class=”headerlink” href=”#python-virtual-environments” title=”Permalink to this headline”></a></h2> <ul class=”simple”> <li><p>Pyenv and pipenv: http://github.com/machine-learning-helpers/induction-python/tree/master/installation/virtual-env</p></li> </ul> </section> </section> <section class=”tex2jax_ignore mathjax_ignore” id=”installation”> <h1>Installation<a class=”headerlink” href=”#installation” title=”Permalink to this headline”></a></h1> <section id=”clone-this-git-repository”> <h2>Clone this Git repository<a class=”headerlink” href=”#clone-this-git-repository” title=”Permalink to this headline”></a></h2> <div class=”highlight-bash notranslate”><div class=”highlight”><pre><span></span>$ mkdir -p ~/dev/infra <span class=”o”>&&</span> <span class=”se”></span>

git clone git@github.com:cloud-helpers/python-plugin-data-loader.git ~/dev/infra/python-plugin-data-loader

$ <span class=”nb”>cd</span> ~/dev/infra/python-plugin-data-loader </pre></div> </div> </section> <section id=”python-environment”> <h2>Python environment<a class=”headerlink” href=”#python-environment” title=”Permalink to this headline”></a></h2> <ul class=”simple”> <li><p>If not already done so, install <code class=”docutils literal notranslate”><span class=”pre”>pyenv</span></code>, Python 3.9 and, <code class=”docutils literal notranslate”><span class=”pre”>pip</span></code> and <code class=”docutils literal notranslate”><span class=”pre”>pipenv</span></code></p> <ul> <li><p>PyEnv:</p></li> </ul> </li> </ul> <div class=”highlight-bash notranslate”><div class=”highlight”><pre><span></span>$ git clone https://github.com/pyenv/pyenv.git <span class=”si”>${</span><span class=”nv”>HOME</span><span class=”si”>}</span>/.pyenv $ cat >> ~/.profile2 <span class=”s”><< _EOF</span>

<span class=”s”># Python</span> <span class=”s”>eval "$(pyenv init –path)"</span>

<span class=”s”>_EOF</span> $ cat >> ~/.bashrc <span class=”s”><< _EOF</span>

<span class=”s”># Python</span> <span class=”s”>export PYENV_ROOT="${HOME}/.pyenv"</span> <span class=”s”>export PATH="${PYENV_ROOT}/bin:${PATH}"</span> <span class=”s”>. ~/.profile2</span> <span class=”s”>if command -v pyenv 1>/dev/null 2>&1</span> <span class=”s”>then</span> <span class=”s”> eval "$(pyenv init -)"</span> <span class=”s”>fi</span> <span class=”s”>if command -v pipenv 1>/dev/null 2>&1</span> <span class=”s”>then</span> <span class=”s”> eval "$(pipenv –completion)"</span> <span class=”s”>fi</span>

<span class=”s”>_EOF</span> $ . ~/.bashrc </pre></div> </div> <ul class=”simple”> <li><p>Python 3.9:</p></li> </ul> <div class=”highlight-bash notranslate”><div class=”highlight”><pre><span></span>$ pyenv install <span class=”m”>3</span>.9.8 <span class=”o”>&&</span> pyenv <span class=”nb”>local</span> <span class=”m”>3</span>.9.8 </pre></div> </div> <ul class=”simple”> <li><p><code class=”docutils literal notranslate”><span class=”pre”>pip</span></code>:</p></li> </ul> <div class=”highlight-bash notranslate”><div class=”highlight”><pre><span></span>$ python -mpip install -U pip </pre></div> </div> <ul class=”simple”> <li><p><code class=”docutils literal notranslate”><span class=”pre”>pipenv</span></code>:</p></li> </ul> <div class=”highlight-bash notranslate”><div class=”highlight”><pre><span></span>$ python -mpip install -U pipenv </pre></div> </div> </section> </section> <section class=”tex2jax_ignore mathjax_ignore” id=”usage”> <h1>Usage<a class=”headerlink” href=”#usage” title=”Permalink to this headline”></a></h1> <section id=”install-the-data-loader-plugin-module”> <h2>Install the <code class=”docutils literal notranslate”><span class=”pre”>data-loader-plugin</span></code> module<a class=”headerlink” href=”#install-the-data-loader-plugin-module” title=”Permalink to this headline”></a></h2> <ul class=”simple”> <li><p>There are at least two ways to install the <code class=”docutils literal notranslate”><span class=”pre”>data-loader-plugin</span></code> module, in the Python user space with <code class=”docutils literal notranslate”><span class=”pre”>pip</span></code> and in a dedicated virtual environment with <code class=”docutils literal notranslate”><span class=”pre”>pipenv</span></code>.</p> <ul> <li><p>Both options may be installed in parallel</p></li> <li><p>The Python user space (typically, <code class=”docutils literal notranslate”><span class=”pre”>/usr/local/opt/python@3.9</span></code> on MacOS or <code class=”docutils literal notranslate”><span class=”pre”>~/.pyenv/versions/3.9.8</span></code> on Linux) may already have many other modules installed, parasiting a fine-grained control over the versions of every Python dependency. If all the versions are compatible, then that option is convenient as it is available from the whole user space, not just from this sub-directory</p></li> </ul> </li> <li><p>In the remainder of that <a class=”reference external” href=”#usage”>Usage section</a>, it will be assumed that the <code class=”docutils literal notranslate”><span class=”pre”>data-loader-plugin</span></code> module has been installed and readily available from the environment, whether that environment is virtual or not. In other words, to adapt the documentation for the case where <code class=”docutils literal notranslate”><span class=”pre”>pipenv</span></code> is used, just add <code class=”docutils literal notranslate”><span class=”pre”>pipenv</span> <span class=”pre”>run</span></code> in front of every Python-related command.</p></li> </ul> <section id=”install-in-the-python-user-space”> <h3>Install in the Python user space<a class=”headerlink” href=”#install-in-the-python-user-space” title=”Permalink to this headline”></a></h3> <ul class=”simple”> <li><p>Install and use the <code class=”docutils literal notranslate”><span class=”pre”>data-loader-plugin</span></code> module in the user space (with <code class=”docutils literal notranslate”><span class=”pre”>pip</span></code>):</p></li> </ul> <div class=”highlight-bash notranslate”><div class=”highlight”><pre><span></span>$ python -mpip uninstall data-loader-plugin $ python -mpip install -U data-loader-plugin </pre></div> </div> </section> <section id=”installation-in-a-dedicated-python-virtual-environment”> <h3>Installation in a dedicated Python virtual environment<a class=”headerlink” href=”#installation-in-a-dedicated-python-virtual-environment” title=”Permalink to this headline”></a></h3> <ul class=”simple”> <li><p>Install and use the <code class=”docutils literal notranslate”><span class=”pre”>data-loader-plugin</span></code> module in a virtual environment:</p></li> </ul> <div class=”highlight-bash notranslate”><div class=”highlight”><pre><span></span>$ pipenv shell <span class=”o”>(</span>python-…-JwpAHotb<span class=”o”>)</span> ✔ python -mpip install -U data-loader-plugin <span class=”o”>(</span>python-…-JwpAHotb<span class=”o”>)</span> ✔ python -mpip install -U data-loader-plugin <span class=”o”>(</span>python-…-JwpAHotb<span class=”o”>)</span> ✔ <span class=”nb”>exit</span> </pre></div> </div> </section> </section> <section id=”use-data-loader-plugin-as-a-module-from-another-python-program”> <h2>Use <code class=”docutils literal notranslate”><span class=”pre”>data-loader-plugin</span></code> as a module from another Python program<a class=”headerlink” href=”#use-data-loader-plugin-as-a-module-from-another-python-program” title=”Permalink to this headline”></a></h2> <ul class=”simple”> <li><p>Check the data file with the AWS command-line (CLI):</p></li> </ul> <div class=”highlight-bash notranslate”><div class=”highlight”><pre><span></span>$ aws s3 ls –human s3://nyc-tlc/trip<span class=”se”></span>data/yellow_tripdata_2021-07.csv –no-sign-request <span class=”m”>2021</span>-10-29 <span class=”m”>20</span>:44:34 <span class=”m”>249</span>.3 MiB yellow_tripdata_2021-07.csv </pre></div> </div> <ul class=”simple”> <li><p>Module import statements:</p></li> </ul> <div class=”highlight-python notranslate”><div class=”highlight”><pre><span></span><span class=”gp”>>>> </span><span class=”kn”>import</span> <span class=”nn”>importlib</span> <span class=”gp”>>>> </span><span class=”kn”>from</span> <span class=”nn”>types</span> <span class=”kn”>import</span> <span class=”n”>ModuleType</span> <span class=”gp”>>>> </span><span class=”kn”>from</span> <span class=”nn”>data_loader_plugin.base</span> <span class=”kn”>import</span> <span class=”n”>DataLoaderBase</span> </pre></div> </div> <ul class=”simple”> <li><p>Create an instance of the DataLoaderBase Python class:</p></li> </ul> <div class=”highlight-python notranslate”><div class=”highlight”><pre><span></span><span class=”gp”>>>> </span><span class=”n”>plugin</span><span class=”p”>:</span> <span class=”n”>ModuleType</span> <span class=”o”>=</span> <span class=”n”>importlib</span><span class=”o”>.</span><span class=”n”>import_module</span><span class=”p”>(</span><span class=”s2”>"data_loader_plugin.copyfile"</span><span class=”p”>)</span> <span class=”gp”>>>> </span><span class=”n”>data_loader</span><span class=”p”>:</span> <span class=”n”>DataLoaderBase</span> <span class=”o”>=</span> <span class=”n”>plugin</span><span class=”o”>.</span><span class=”n”>DataLoader</span><span class=”p”>(</span> <span class=”go”> local_path='/tmp/yellow_tripdata_2021-07.csv',</span> <span class=”go”> external_url='s3://nyc-tlc/tripdata/yellow_tripdata_2021-07.csv',</span> <span class=”go”> )</span> <span class=”gp”>>>> </span><span class=”n”>data_load_success</span><span class=”p”>,</span> <span class=”n”>message</span> <span class=”o”>=</span> <span class=”n”>data_loader</span><span class=”o”>.</span><span class=”n”>load</span><span class=”p”>()</span> </pre></div> </div> </section> </section> <section class=”tex2jax_ignore mathjax_ignore” id=”development-contribution”> <h1>Development / Contribution<a class=”headerlink” href=”#development-contribution” title=”Permalink to this headline”></a></h1> <ul class=”simple”> <li><p>Build the source distribution and Python artifacts (wheels):</p></li> </ul> <div class=”highlight-bash notranslate”><div class=”highlight”><pre><span></span>$ rm -rf _skbuild/ build/ dist/ .tox/ __pycache__/ .pytest_cache/ MANIFEST .egg-info/ $ pipenv run python setup.py sdist bdist_wheel </pre></div> </div> <ul class=”simple”> <li><p>Upload to Test PyPi (no Linux binary wheel can be uploaded on PyPi):</p></li> </ul> <div class=”highlight-bash notranslate”><div class=”highlight”><pre><span></span>$ <span class=”nv”>PYPIURL</span><span class=”o”>=</span><span class=”s2”>"https://test.pypi.org"</span> $ pipenv run twine upload -u __token__ –repository-url <span class=”si”>${</span><span class=”nv”>PYPIURL</span><span class=”si”>}</span>/legacy/ dist/ Uploading distributions to https://test.pypi.org/legacy/ Uploading data_loader_plugin-0.0.1-py3-none-any.whl <span class=”m”>100</span>%<span class=”p”>|</span>███████████████████████████████████████<span class=”p”>|</span> <span class=”m”>23</span>.1k/23.1k <span class=”o”>[</span><span class=”m”>00</span>:02<<span class=”m”>00</span>:00, <span class=”m”>5</span>.84kB/s<span class=”o”>]</span> Uploading data-loader-plugin-0.0.1.tar.gz <span class=”m”>100</span>%<span class=”p”>|</span>███████████████████████████████████████<span class=”p”>|</span> <span class=”m”>23</span>.0k/23.0k <span class=”o”>[</span><span class=”m”>00</span>:01<<span class=”m”>00</span>:00, <span class=”m”>15</span>.8kB/s<span class=”o”>]</span>

View at: https://test.pypi.org/project/data-loader-plugin/0.0.1/ </pre></div> </div> <ul class=”simple”> <li><p>Upload/release the Python packages onto the <a class=”reference external” href=”https://pypi.org”>PyPi repository</a>:</p> <ul> <li><p>Register the authentication token for access to PyPi:</p></li> </ul> </li> </ul> <div class=”highlight-bash notranslate”><div class=”highlight”><pre><span></span>$ <span class=”nv”>PYPIURL</span><span class=”o”>=</span><span class=”s2”>"https://upload.pypi.org"</span> $ pipenv run keyring <span class=”nb”>set</span> <span class=”si”>${</span><span class=”nv”>PYPIURL</span><span class=”si”>}</span>/ __token__ Password <span class=”k”>for</span> <span class=”s1”>'__token__'</span> <span class=”k”>in</span> <span class=”s1”>'${PYPIURL}/'</span>: </pre></div> </div> <ul class=”simple”> <li><p>Register the authentication token for access to PyPi:</p></li> </ul> <div class=”highlight-bash notranslate”><div class=”highlight”><pre><span></span>$ pipenv run twine upload -u __token__ –repository-url <span class=”si”>${</span><span class=”nv”>PYPIURL</span><span class=”si”>}</span>/legacy/ dist/* Uploading distributions to https://upload.pypi.org/legacy/ Uploading data_loader_plugin-0.0.1-py3-none-any.whl <span class=”m”>100</span>%<span class=”p”>|</span>███████████████████████████████████████<span class=”p”>|</span> <span class=”m”>23</span>.1k/23.1k <span class=”o”>[</span><span class=”m”>00</span>:02<<span class=”m”>00</span>:00, <span class=”m”>5</span>.84kB/s<span class=”o”>]</span> Uploading data-loader-plugin-0.0.1.tar.gz <span class=”m”>100</span>%<span class=”p”>|</span>███████████████████████████████████████<span class=”p”>|</span> <span class=”m”>23</span>.0k/23.0k <span class=”o”>[</span><span class=”m”>00</span>:01<<span class=”m”>00</span>:00, <span class=”m”>15</span>.8kB/s<span class=”o”>]</span>

View at: https://pypi.org/project/data-loader-plugin/0.0.1/ </pre></div> </div> <ul class=”simple”> <li><p>Note that the documentation is built automatically by ReadTheDocs (RTD)</p> <ul> <li><p>The documentation is available from https://data-loader-plugin.readthedocs.io/en/latest/</p></li> <li><p>The RTD project is setup on https://readthedocs.org/projects/data-loader-plugin/</p></li> </ul> </li> <li><p>Build the documentation manually (with <a class=”reference external” href=”http://sphinx-doc.org”>Sphinx</a>):</p></li> </ul> <div class=”highlight-bash notranslate”><div class=”highlight”><pre><span></span>$ pipenv run python setup.py build_sphinx running build_sphinx Running Sphinx v4.3.0 <span class=”o”>[</span>autosummary<span class=”o”>]</span> generating autosummary <span class=”k”>for</span>: README.md myst v0.15.2: …, <span class=”nv”>words_per_minute</span><span class=”o”>=</span><span class=”m”>200</span><span class=”o”>)</span> building <span class=”o”>[</span>mo<span class=”o”>]</span>: targets <span class=”k”>for</span> <span class=”m”>0</span> po files that are out of date building <span class=”o”>[</span>html<span class=”o”>]</span>: targets <span class=”k”>for</span> <span class=”m”>1</span> <span class=”nb”>source</span> files that are out of date updating environment: <span class=”o”>[</span>new config<span class=”o”>]</span> <span class=”m”>1</span> added, <span class=”m”>0</span> changed, <span class=”m”>0</span> removed reading sources… <span class=”o”>[</span><span class=”m”>100</span>%<span class=”o”>]</span> README … looking <span class=”k”>for</span> now-outdated files… none found pickling environment… <span class=”k”>done</span> checking consistency… <span class=”k”>done</span> preparing documents… <span class=”k”>done</span> writing output… <span class=”o”>[</span><span class=”m”>100</span>%<span class=”o”>]</span> README … build succeeded.

The HTML pages are <span class=”k”>in</span> build/sphinx/html. </pre></div> </div> <ul class=”simple”> <li><p>Re-generate the Python dependency files (<code class=”docutils literal notranslate”><span class=”pre”>requirements.txt</span></code>) for the CI/CD pipeline (currently Travis CI):</p></li> </ul> <div class=”highlight-bash notranslate”><div class=”highlight”><pre><span></span>$ pipenv –rm<span class=”p”>;</span> rm -f Pipfile.lock<span class=”p”>;</span> pipenv install<span class=”p”>;</span> pipenv install –dev $ git add Pipfile.lock $ pipenv lock -r > ci/requirements.txt $ pipenv lock –dev -r > ci/requirements-dev.txt $ git add ci/requirements.txt ci/requirements-dev.txt $ git commit -m <span class=”s2”>"[CI] Upgraded the Python dependencies for the Travis CI pipeline"</span> </pre></div> </div> <section id=”test-the-data-loader-plugin-python-module”> <h2>Test the data loader plugin Python module<a class=”headerlink” href=”#test-the-data-loader-plugin-python-module” title=”Permalink to this headline”></a></h2> <ul class=”simple”> <li><p>Enter into the <code class=”docutils literal notranslate”><span class=”pre”>pipenv</span></code> Shell:</p></li> </ul> <div class=”highlight-bash notranslate”><div class=”highlight”><pre><span></span>$ pipenv shell <span class=”o”>(</span>python-…-iVzKEypY<span class=”o”>)</span> ✔ python -V Python <span class=”m”>3</span>.9.8 </pre></div> </div> <ul class=”simple”> <li><p>Uninstall any previously installed <code class=”docutils literal notranslate”><span class=”pre”>data-loader-plugin</span></code> module/library:</p></li> </ul> <div class=”highlight-bash notranslate”><div class=”highlight”><pre><span></span><span class=”o”>(</span>python-…-iVzKEypY<span class=”o”>)</span> ✔ python -mpip uninstall data-loader-plugin </pre></div> </div> <ul class=”simple”> <li><p>Launch a simple test with <code class=”docutils literal notranslate”><span class=”pre”>pytest</span></code></p></li> </ul> <div class=”highlight-bash notranslate”><div class=”highlight”><pre><span></span><span class=”o”>(</span>python-iVzKEypY<span class=”o”>)</span> ✔ python -mpytest <span class=”nv”>tests</span> <span class=”o”>===================</span> <span class=”nb”>test</span> session <span class=”nv”>starts</span> <span class=”o”>==================</span> platform darwin – Python <span class=”m”>3</span>.9.8, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 rootdir: ~/dev/infra/python-plugin-data-loader plugins: cov-3.0.0 collected <span class=”m”>3</span> items

tests/test_copyfile.py . <span class=”o”>[</span> <span class=”m”>33</span>%<span class=”o”>]</span> tests/test_s3.py .. <span class=”o”>[</span><span class=”m”>100</span>%<span class=”o”>]</span> <span class=”o”>======================</span> <span class=”m”>3</span> passed <span class=”k”>in</span> <span class=”m”>1</span>.22s <span class=”o”>==================</span> </pre></div> </div> <ul class=”simple”> <li><p>Exit the <code class=”docutils literal notranslate”><span class=”pre”>pipenv</span></code> Shell:</p></li> </ul> <div class=”highlight-bash notranslate”><div class=”highlight”><pre><span></span><span class=”o”>(</span>python-…-iVzKEypY<span class=”o”>)</span> ✔ <span class=”nb”>exit</span> </pre></div> </div> </section> </section>

</div>

</div> <footer>

<hr/>

<div role=”contentinfo”>

<p>© Copyright 2021, Stanislav Khrapov.
<span class=”lastupdated”>Last updated on Nov 16, 2021. </span></p>

</div>

Built with <a href=”https://www.sphinx-doc.org/”>Sphinx</a> using a
<a href=”https://github.com/readthedocs/sphinx_rtd_theme”>theme</a> provided by <a href=”https://readthedocs.org”>Read the Docs</a>.

</footer>

</div>

</div>

</section>

</div> <script>

jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);

});

</script>

</body> </html>