Python pour la data science

Lino Galiana

doi:10.5281/zenodo.8229676

This part of the site presents something that is not specific to Python but is nevertheless essential: the practice of Git.

A large part of the content of this part comes from a dedicated course I gave with Romain Avouac (in French).

Scroll down to read the slides below or click here to display the slides full screen (the slides are in French).

1 Introduction

1.1 The problem

The natural approach when you start working for a long time on a project for which you want to avoid erasing and losing your code because of a human error is to duplicate files and create multiple versions of the same code that are snapshots at a given point in time:

This is a way of saving the evolution of your code and therefore the life of a project. Nevertheless, it is a very artisanal way of proceeding that does not eliminate human error, since it is always possible to make a mistake when duplicating the file or to forget to save a version that satisfied us.

This practice has many drawbacks. The first is that searching for relevant information, for example about the introduction of a bug, is complicated by this duplicated structure. It is difficult, without going into the detail of each file, to know its actual evolution between two dates. For information extraction, it would be much more relevant to build files describing the evolution between one version and another, but this requires many manual steps and a considerable amount of time. The second problem is that we do not necessarily know, when we come back to the project a few days later, why we saved a given version: on 28 November, do we remember what distinguishes the versions of 4 May at 12:37 and at 16:02?

If we add the collaborative dimension of working on code, other drawbacks of this artisanal approach quickly appear. First, someone taking over the project will have trouble getting into it. Moreover, it will be even harder for this person to look for relevant information about the version choices that were made: this history is ultimately useless to them. Especially since the question of sharing this code arises: through what channel is this set of files shared? By email? On a shared drive? But what happens if several participants in the project work on it at the same time? How can collaboration be organised and changes reconciled if they happen jointly?

1.2 The technical solution: `Git`

Git provides a technical answer to these many questions. This software, specialised in version control, that is, in tracking the evolution of a project, solves many problems related to carrying out data science projects in organisations. The purpose of this chapter is to present a few concepts needed to understand Git and to show how it helps manage the evolution of a software project. The next chapter will introduce how Git smooths collaboration within teams involved in data science projects. This has become essential because the era when data scientists all worked alone on notebooks is over. Data science projects have become, in most organisations, more ambitious and often involve several people with varied profiles¹, so discipline is needed for collaboration to be smooth.

2 How to use `Git` when doing `Python` ?

Git is version control software, that is, software in charge of recording the evolutions of a file over time (what is called versioning). It is not a Python package, we will not use it that way.

Python users can use Git through two intermediaries: through the command line or through graphical extensions in their development environments (VSCode, Jupyter, etc.). On the SSPCloud, the recommended infrastructure for this course, we can do Git with both approaches but we will mainly use the VSCode graphical interface, which lowers the cost of entry into Git.

Where to find additional information for this course?

Git is part of the collaborative practices that have become standard in the open-source world but are also increasingly common in data science administrations and companies.

There are many resources on using Git on the internet. Unfortunately, many are technical and assume an already advanced knowledge of some computing notions useful for understanding Git. This course will not make such assumptions, except for a minimal knowledge of the logic of a filesystem, that is, of how files are organised on a computer.

Regarding content close to this one, a series of training resources has been gathered by Insee on this site (in French).

3 Content of this part

Learning Git is split into two chapters:

The first is devoted to presenting the general logic of Git and its important concepts, and to illustrating them through practice in an individual work setting.
The second chapter is devoted to the challenges of collaborative work

The general objective of this part is to demystify Git. While self-taught discovery can be particularly painful, this is not the case when accompanied by a resource that illustrates the important concepts through practice and gradually increases the complexity of the setting in which Git is used.

👉️ A number of terms that are new when you discover Git, but which are the concepts useful for understanding it, are defined in the margins of the next two chapters, as shown here.

Informations additionnelles

Python environment

This site was built automatically through a Github action using the Quarto reproducible publishing software (version 1.10.18).

The environment used to obtain the results is reproducible via uv. The pyproject.toml file used to build this environment is available on the linogaliana/python-datascientist repository

pyproject.toml

[project]
name = "python-datascientist"
version = "0.1.0"
description = "Source code for Lino Galiana's Python for data science course"
readme = "README.md"
requires-python = ">=3.13,<3.14"
dependencies = [
    "altair>=6.0.0",
    "cartiflette",
    "contextily==1.6.2",
    "duckdb>=0.10.1",
    "folium>=0.19.6",
    "gdal==3.11.4",
    "graphviz==0.20.3",
    "great-tables>=0.12.0",
    "gt-extras>=0.0.8",
    "ipykernel>=6.29.5",
    "jupyter>=1.1.1",
    "jupyter-cache>=1.0.0",
    "kaleido>=0.2.1",
    "langchain-community>=0.3.27",
    "loguru==0.7.3",
    "markdown>=3.8",
    "nbclient>=0.10.0",
    "nbformat>=5.10.4",
    "nltk>=3.9.1",
    "pandas>=3.0",
    "pip>=25.1.1",
    "plotly>=6.1.2",
    "plotnine>=0.15",
    "polars>=1.8.2",
    "pyarrow>=17.0.0",
    "pynsee>=0.1.8",
    "python-dotenv>=1.0.1",
    "python-frontmatter>=1.1.0",
    "pywaffle>=1.1.1",
    "requests>=2.32.3",
    "scikit-image>=0.24.0",
    "scikit-learn>=1.8.0",
    "scipy>=1.13.0",
    "seaborn>=0.13.2",
    "selenium<4.39.0",
    "spacy>=3.8.4",
    "webdriver-manager>=4.0.2",
    "wordcloud==1.9.3",
]

[tool.uv.sources]
cartiflette = { git = "https://github.com/inseefrlab/cartiflette" }
gdal = [
  { index = "gdal-wheels", marker = "sys_platform == 'linux'" },
  { index = "geospatial_wheels", marker = "sys_platform == 'win32'" },
]

[[tool.uv.index]]
name = "geospatial_wheels"
url = "https://nathanjmcdougall.github.io/geospatial-wheels-index/"
explicit = true

[[tool.uv.index]]
name = "gdal-wheels"
url = "https://gitlab.com/api/v4/projects/61637378/packages/pypi/simple"
explicit = true

[dependency-groups]
dev = [
    "nb-clean>=4.0.1",
]

To use exactly the same environment (version of Python and packages), please refer to the documentation for uv.

File history

md`This file has been modified __${table_commit.length}__ times since its creation on ${creation_string} (last modified on ${last_modification_string})`

html`<div>${git_history_table}</div>`

html`<div>${git_history_graph}</div>`

SHA	Date	Author	Description
345804e7	2026-09-26 09:17:19	Lino Galiana	Création d’un PDF avec typst (#703)
40bd4299	2026-09-20 12:45:39	linogaliana	Correction en masse des problèmes de traduction
fe573ec0	2025-12-23 12:54:11	Lino Galiana	Un syllabus sous la forme d’un joli tableau (#667)
eeb949c8	2025-08-21 17:38:40	Lino Galiana	Fix a few problems detected by AI agent (#641)
c3d51646	2025-08-12 17:28:51	Lino Galiana	Ajoute un résumé au début de chaque chapitre (première partie) (#634)
94648290	2025-07-22 18:57:48	Lino Galiana	Fix boxes now that it is better supported by jupyter (#628)
21db4dbf	2024-09-30 11:42:26	lgaliana	Marges
3e04253c	2024-09-30 10:11:32	Lino Galiana	Grosse mise à jour de la partie Git (#557)
c9f9f8a7	2024-04-24 15:09:35	Lino Galiana	Dark mode and CSS improvements (#494)
005d89b8	2023-12-20 17:23:04	Lino Galiana	Finalise l’affichage des statistiques Git (#478)
09654c71	2023-11-14 15:16:44	Antoine Palazzolo	Suggestions Git & Visualisation (#449)
57f108fa	2023-11-10 10:59:36	linogaliana	Intro git
154f09e4	2023-09-26 14:59:11	Antoine Palazzolo	Des typos corrigées par Antoine (#411)
9a4e2267	2023-08-28 17:11:52	Lino Galiana	Action to check URL still exist (#399)
3bdf3b06	2023-08-25 11:23:02	Lino Galiana	Simplification de la structure 🤓 (#393)
30823c40	2023-08-24 14:30:55	Lino Galiana	Liens morts navbar (#392)
2dbf8533	2023-07-05 11:21:40	Lino Galiana	Add nice featured images (#368)
f21a24d3	2023-07-02 10:58:15	Lino Galiana	Pipeline Quarto & Pages 🚀 (#365)
34cc32c3	2022-10-14 22:05:47	Lino Galiana	Relecture Git (#300)
f394b233	2022-10-13 14:32:05	Lino Galiana	Dernieres modifs geopandas (#298)
0e01c33f	2021-11-10 12:09:22	Lino Galiana	Relecture @antuki API+Webscraping + Git (#178)
9a3f7ad8	2021-10-31 18:36:25	Lino Galiana	Nettoyage partie API + Git (#170)
aeb39950	2021-07-06 11:11:03	avouacr	Relecture et ajouts sur anaconda + jupyter (#116)
4cdb759c	2021-05-12 10:37:23	Lino Galiana	:sparkles: :star2: Nouveau thème hugo :snake: :fire: (#105)

creation = d3.min(
  table_commit.map(d => new Date(d.Date))
)

last_modification = d3.max(
  table_commit.map(d => new Date(d.Date))
)

creation_string = creation.toLocaleString("fr", {
  "day": "numeric",
  "month": "long",
  "year": "numeric"
})

last_modification_string = last_modification.toLocaleString("fr", {
  "day": "numeric",
  "month": "long",
  "year": "numeric"
})

git_history_table = Inputs.table(
  table_commit,
  {
    format: {
      SHA: x => md`[${x}](${github_repo}/commit/${x})`,
      Description: x => md`${replacePullRequestPattern(x, github_repo)}`,
      /*Date: x => x.toLocaleString("fr", {
        "month": "numeric",
        "day": "numeric",
        "year": "numeric"
        })
      */
    }
  }
)

// Historique du fichier sous forme d'arbre git (style: styles/styles.css, `.fh-*`).
// L'historique d'un fichier est linéaire (`git log --follow`) : une seule ligne,
// du commit le plus récent (en haut) à la création du fichier (en bas).
git_history_graph = {
  const lang = document.documentElement.lang === "en" ? "en" : "fr"
  const labels = {
    fr: {latest: "dernière version", created: "création", plain: "commit", pr: "commit issu d'une pull request", by: "par"},
    en: {latest: "latest", created: "created", plain: "commit", pr: "commit from a pull request", by: "by"}
  }[lang]

  const commits = (table_commit || []).map(d => {
    // "(#667)" en fin de message: numéro de la pull request, affiché sous forme de pastille
    const pr = d.Description.match(/\s*\(#(\d+)\)\s*$/)
    return {
      sha: d.SHA,
      author: d.Author,
      // "YYYY-MM-DD HH:MM:SS" -> ISO 8601, que tous les navigateurs savent lire
      date: new Date(d.Date.replace(" ", "T")),
      message: pr ? d.Description.slice(0, pr.index) : d.Description,
      pr: pr ? pr[1] : null
    }
  })

  if (commits.length === 0) return html`<em>-</em>`

  const fmt = new Intl.DateTimeFormat(lang, {day: "numeric", month: "short", year: "numeric"})
  const fmtFull = new Intl.DateTimeFormat(lang, {dateStyle: "full", timeStyle: "short"})

  // une ligne par commit, avec un séparateur à chaque changement d'année
  const rows = []
  commits.forEach((c, i) => {
    const year = c.date.getFullYear()
    if (i === 0 || year !== commits[i - 1].date.getFullYear()) {
      rows.push(html`<li class="fh-row fh-year ${i === 0 ? "fh-top" : ""}">${year}</li>`)
    }
    const isFirst = i === 0
    const isLast = i === commits.length - 1
    const head = isFirst ? html`<span class="fh-pill fh-pill-head">${labels.latest}</span>` : ""
    const root = isLast ? html`<span class="fh-pill fh-pill-root">${labels.created}</span>` : ""
    const pr = c.pr ? html`<a class="fh-pill fh-pill-pr" href="${github_repo}/pull/${c.pr}">#${c.pr}</a>` : ""
    const title = html`<div class="fh-title">${head}${root}<a class="fh-sha" href="${github_repo}/commit/${c.sha}">${c.sha}</a><span class="fh-msg">${c.message}</span>${pr}</div>`
    const meta = html`<div class="fh-meta" title="${fmtFull.format(c.date)}">${fmt.format(c.date)} · ${labels.by} ${c.author}</div>`
    rows.push(html`<li class="fh-row fh-commit ${c.pr ? "fh-pr" : ""} ${isFirst ? "fh-first" : ""} ${isLast ? "fh-last fh-root" : ""}"><span class="fh-dot"></span>${title}${meta}</li>`)
  })

  // Les gabarits html tiennent sur une seule ligne: en sortie Typst, ce code n'est pas
  // interprété par Quarto et pandoc le lit comme du markdown; une balise `<div` en début
  // de ligne y ouvrirait une vraie div.
  const legend = html`<div class="fh-legend"><span><span class="fh-dot"></span>${labels.plain}</span><span class="fh-pr"><span class="fh-dot"></span>${labels.pr}</span></div>`
  return html`<div class="fh"><div class="fh-scroll"><ol class="fh-list">${rows}</ol></div>${legend}</div>`
}

function replacePullRequestPattern(inputString, githubRepo) {
    // Use a regular expression to match the pattern #digit
    var pattern = /#(\d+)/g;

    // Replace the pattern with ${github_repo}/pull/#digit
    var replacedString = inputString.replace(pattern, '[#$1](' + githubRepo + '/pull/$1)');

    return replacedString;
}

github_repo = "https://github.com/linogaliana/python-datascientist"

table_commit = {

// Get the HTML table by its class name
var table = document.querySelector('.commit-table');

// Check if the table exists
if (table) {
    // Initialize an array to store the table data
    var dataArray = [];

    // Extract headers from the first row
    var headers = [];
    for (var i = 0; i < table.rows[0].cells.length; i++) {
        headers.push(table.rows[0].cells[i].textContent.trim());
    }

    // Iterate through the rows, starting from the second row
    for (var i = 1; i < table.rows.length; i++) {
        var row = table.rows[i];
        var rowData = {};

        // Iterate through the cells in the row
        for (var j = 0; j < row.cells.length; j++) {
            // Use headers as keys and cell content as values
            rowData[headers[j]] = row.cells[j].textContent.trim();
        }

        // Push the rowData object to the dataArray
        dataArray.push(rowData);
    }
  }

  return dataArray

}

// Get the element with class 'git-details'
{
  var gitDetails = document.querySelector('.commit-table');

  // Check if the element exists
  if (gitDetails) {
      // Hide the element
      gitDetails.style.display = 'none';
  }
}

Back to top

Footnotes

This observation is the starting point of the course “Putting data science projects into production”, available at ensae-reproductibilite.github.io/website/ (in French), which Romain Avouac and I teach at the end of the ENSAE curriculum.↩︎

Citation

BibTeX citation:

@book{galiana2025,
  author = {Galiana, Lino},
  title = {Python Pour La Data Science},
  date = {2025},
  url = {https://pythonds.linogaliana.fr/},
  doi = {10.5281/zenodo.8229676},
  langid = {en}
}

For attribution, please cite this work as:

Galiana, Lino. 2025. Python Pour La Data Science. https://doi.org/10.5281/zenodo.8229676.