Method

🔹 Python: The Backbone of Data Processing

Definition: Python is a high-level, open-source programming language widely used in data science, machine learning, and network analysis due to its readability, extensibility, and rich ecosystem of libraries.

Why Python?

• Ecosystem: Libraries like Pandas, NetworkX, and Matplotlib are industry standards for data manipulation and analysis.
• Compatibility: Seamless integration with OpenAlex API and other data sources.
• Reproducibility: Scripts can be easily shared and reused (via Jupyter Notebooks or GitHub).

Key Features Used in MLKN.lab:

Feature	Use Case in MLKN.lab	Example Libraries/Tools
Data Cleaning	Normalizing OpenAlex metadata (e.g., author names, discipline labels).	Pandas, NumPy
Automation	Batch processing of large datasets.	Python scripts, Cron jobs
API Interactions	Fetching data from OpenAlex, ORKG, Wikidata.	requests, aiohttp
Modularity	Organizing code into reusable functions.	Custom Python modules

Example Code Snippet:

import pandas as pd
    import requests

    # Fetch data from OpenAlex API
    response = requests.get("https://api.openalex.org/works?filter=publication_year:2020")
    data = response.json()
    df = pd.DataFrame(data["results"])  # Convert to DataFrame for analysis

Learn More: Python Official Website | Pandas for Data Science

🔹 Pandas: Data Cleaning & Transformation

Definition: Pandas is a Python library for data manipulation and analysis, offering high-performance, easy-to-use data structures (e.g., DataFrames) and tools.

Why Pandas?

• Efficiency: Optimized for large datasets (e.g., OpenAlex’s 200M+ works).
• Flexibility: Handles missing data, duplicates, and complex transformations seamlessly.
• Integration: Works with NetworkX, Matplotlib, and PyVis for end-to-end workflows.

Key Features Used in MLKN.lab:

Feature	Use Case in MLKN.lab	Example Code
Data Cleaning	Removing duplicates, standardizing labels.	df.drop_duplicates()
Filtering	Extracting relevant disciplines/subfields.	df[df["discipline"] == "AI"]
Aggregation	Counting papers per discipline.	df.groupby("discipline").size()
Merging	Combining datasets (e.g., OpenAlex + Scopus).	pd.merge(df1, df2, on="id")

Example Workflow:

1. Load raw OpenAlex data into a DataFrame.
2. Clean (remove duplicates, normalize text).
3. Transform (extract disciplines, count co-occurrences).
4. Export to NetworkX for graph analysis.

Learn More: Pandas Documentation

🔹 NetworkX: Network Construction & Analysis

Definition: NetworkX is a Python library for creating, manipulating, and studying the structure, dynamics, and functions of complex networks.

Why NetworkX?

• Standard for Network Science: Used in academic research and industry.
• Algorithms: Built-in functions for centrality, clustering, and path analysis.
• Visualization: Basic plotting capabilities (complemented by D3.js for interactivity).

Key Features Used in MLKN.lab:

Feature	Use Case in MLKN.lab	Example Code
Graph Construction	Creating nodes (disciplines) and edges (relationships).	G = nx.DiGraph()
Centrality Metrics	Identifying key disciplines (e.g., hubs).	nx.degree_centrality(G)
Community Detection	Finding clusters of related disciplines.	nx.algorithms.community.greedy_modularity_communities(G)
Path Analysis	Tracing connections between fields.	nx.shortest_path(G, "AI", "Ethics")

Example: Building a Knowledge Graph

import networkx as nx

    # Create a directed graph
    G = nx.DiGraph()

    # Add nodes (disciplines)
    G.add_node("Computer Science", type="domain")
    G.add_node("AI", type="discipline", parent="Computer Science")

    # Add edges (relationships)
    G.add_edge("Computer Science", "AI", relationship="hierarchical")

    # Calculate centrality
    centrality = nx.degree_centrality(G)
    print(f"Most central discipline: {max(centrality, key=centrality.get)}")

Learn More: NetworkX Documentation

🔹 Matplotlib/Seaborn: Static Visualization

Definition: Matplotlib is a Python library for creating static, publication-quality visualizations, while Seaborn provides a high-level interface for statistical graphics.

Why Matplotlib/Seaborn?

• Exploratory Analysis: Quickly visualize distributions, correlations, and trends in the data.
• Publication-Ready: Export graphs in high resolution (PNG, SVG, PDF).
• Complement to D3.js: Used for static analyses (e.g., histograms of discipline sizes), while D3.js handles interactivity.

Key Features Used in MLKN.lab:

Feature	Use Case in MLKN.lab	Example Code
Histograms	Distribution of papers per discipline.	plt.hist(df["discipline"])
Heatmaps	Co-occurrence matrices between disciplines.	sns.heatmap(co_occurrence_matrix)
Network Plots	Basic static graphs (before D3.js).	nx.draw(G, with_labels=True)

Example: Discipline Distribution

import matplotlib.pyplot as plt

    df["discipline"].value_counts().plot(kind="bar", title="Papers per Discipline")
    plt.xlabel("Discipline")
    plt.ylabel("Number of Papers")
    plt.show()

Learn More: Matplotlib Gallery | Seaborn Tutorial

🔹 PyVis: Interactive Network Visualization (Python Side)

Definition: PyVis is a Python library that generates interactive network visualizations in HTML/JS using Vis.js (complementary to D3.js).

Why PyVis?

• Rapid Prototyping: Generates interactive visualizations in a few lines of code.
• Exportable: Produces standalone HTML files (no server required).
• Complement to D3.js: Used for quick visualizations during development.

Key Features Used in MLKN.lab:

Feature	Use Case in MLKN.lab	Example Code
Interactive Graphs	Visualizing small/medium networks.	net.show("network.html")
Node Customization	Color nodes by domain/discipline.	net.show_buttons(filter_=["physics"])
Edge Customization	Style edges by relationship type.	net.show_edges(labels=True)

Example: Visualizing a Sub-Network

from pyvis.network import Network

    net = Network(notebook=True, height="750px", width="100%")
    net.from_nx(G)  # Convert NetworkX graph to PyVis
    net.show("knowledge_network.html")

Learn More: PyVis Documentation

🔹 D3.js: The Heart of MLKN.hypergraph’s Interactive Visualization

Definition: D3.js (Data-Driven Documents) is a JavaScript library for producing dynamic, interactive data visualizations in web browsers. It binds data to DOM elements and applies transformations to the document based on that data.

Why D3.js?

• Flexibility: Full control over every pixel (unlike PyVis, which is more limited).
• Performance: Optimized for large graphs (320K+ connections in MLKN.hypergraph).
• Interactivity: Supports zoom, drag-and-drop, tooltips, and clicks to explore the data.
• Force-Directed Layout: Ideal algorithm for visualizing complex networks (e.g., scientific knowledge).

Key Concepts in MLKN.hypergraph:

Concept	Description	Implementation in D3.js
Force-Directed Layout	Simulates a physical system where nodes repel each other and links attract.	d3.forceSimulation() + d3.forceLink()
Nodes	Represent domains/disciplines.	circle SVG elements with r (radius) based on centrality.
Edges	Represent hierarchical/semantic relationships.	line SVG elements with stroke-width based on weight.
Tooltips	Show metadata on hover (e.g., name, number of subdisciplines).	d3.tip() or custom HTML tooltips.
Zoom/Pan	Explore the graph at different scales.	d3.zoom() + d3.drag()
Color Coding	Nodes colored by domain (e.g., red for Cognitive Sciences).	d3.scaleOrdinal() for domain colors.

How the Force-Directed Layout Works:

1. Initialization: Nodes are placed randomly in the SVG space. Links are represented as springs (shorter links = stronger forces).
2. Simulation:
   • forceManyBody: Repulsion between all nodes (avoids overlaps). Key Parameter: strength (e.g., -100 for strong repulsion).
   • forceLink: Attraction along links (simulates springs). Key Parameter: distance (e.g., 50 for ideal node spacing).
   • forceCenter: Attracts all nodes to the SVG center. Key Parameter: x and y (center coordinates).
3. Stabilization: The simulation balances out after a few iterations (nodes stabilize). Key Parameter: alphaTarget (lower = faster stabilization).

Example Code (Simplified):

 // 1. Set up the SVG
 const svg = d3.select("body").append("svg")
     .attr("width", 800)
     .attr("height", 600);

// 2. Load the data (nodes and links)
d3.json("data/knowledge-network.json").then(data => {
const nodes = data.nodes;
const links = data.links;

// 3. Create the simulation
const simulation = d3.forceSimulation(nodes)
     .force("link", d3.forceLink(links).id(d => d.id).distance(50))
     .force("charge", d3.forceManyBody().strength(-100))
     .force("center", d3.forceCenter(400, 300));

// 4. Draw the links
const link = svg.append("g")
     .selectAll("line")
     .data(links)
     .enter().append("line")
     .attr("stroke", "#999")
     .attr("stroke-width", 1);

// 5. Draw the nodes
const node = svg.append("g")
     .selectAll("circle")
     .data(nodes)
     .enter().append("circle")
     .attr("r", d => Math.sqrt(d.size) * 5) // Size based on centrality
     .attr("fill", d => color(d.domain)); // Color by domain

// 6. Update positions on each tick
simulation.on("tick", () => {
link
     .attr("x1", d => d.source.x)
     .attr("y1", d => d.source.y)
     .attr("x2", d => d.target.x)
     .attr("y2", d => d.target.y);

node
     .attr("cx", d => d.x)
     .attr("cy", d => d.y);
     });
});

Advantages for MLKN.hypergraph:

• Scalability: Handles hundreds of thousands of nodes/links without lag.
• Customization: Allows adding labels, legends, and custom interactions.
• Integration: Works with React, Vue.js, or vanilla JS (compatible with your current site).

Learn More: D3.js Official Website | Force-Directed Layout Tutorial

The data processing pipeline transforms raw OpenAlex data into a structured, polyhierarchical knowledge network ready for analysis and visualization.

Key Steps:

1. Data Ingestion:
   • Fetching data from OpenAlex API (e.g., works, authors, concepts).
   • Supplementary sources: Scopus, MeSH, IEEE Thesaurus (to enrich classifications).
2. Cleaning & Normalization:
   • Deduplication: Removing duplicate entries (e.g., same paper in multiple sources).
   • Standardization: Normalizing discipline names (e.g., "AI" → "Artificial Intelligence").
   • Disambiguation: Resolving ambiguous author names (e.g., "J. Smith" → "John Smith, Harvard").
3. Transformation:
   • Extracting Hierarchies: Identifying parent-child relationships (e.g., "Neuroscience" → "Cognitive Neuroscience").
   • Co-Occurrence Analysis: Counting how often disciplines/concepts appear together.
   • Network Construction: Converting cleaned data into a graph structure (nodes = disciplines, edges = relationships).

Tools Used:

• Python + Pandas: For cleaning and transformation.
• NetworkX: For building the graph.
• Jupyter Notebooks: For documenting each step (reproducibility).

Example Workflow:

# 1. Load raw data
df = pd.read_csv("data/raw/openalex_works.csv")

# 2. Clean duplicates
df = df.drop_duplicates(subset=["doi"])

# 3. Normalize discipline names
discipline_mapping = {"AI": "Artificial Intelligence", "CS": "Computer Science"}
df["discipline"] = df["discipline"].replace(discipline_mapping)

# 4. Extract co-occurrences
co_occurrence = pd.crosstab(df["discipline1"], df["discipline2"])

# 5. Save processed data
df.to_csv("data/processed/disciplines_clean.csv", index=False)

At MLKN.lab, we are committed to open science principles: transparency, reproducibility, and accessibility. All our data, code, and methodologies are publicly available.

Key Components:

1. Open Data:
   • Zenodo: Datasets with DOI for citation (e.g., 10.5281/zenodo.21363227).
   • GitHub: Raw and processed data (via Git LFS for large files).
2. Open Code:
   • GitHub Repository: FrancoisPapin/MLKN-lab.
   • Jupyter Notebooks: Step-by-step tutorials for data cleaning, analysis, and visualization.
   • License: MIT (allows free reuse).
3. Open Methodology:
   • Documentation: Detailed explanations of every step (from data collection to visualization).
   • Tutorials: Guides for reproducing our results (e.g., README.md, tutorials/ folder).

How to Reproduce Our Results:

1. Clone the Repository:

   git clone https://github.com/FrancoisPapin/MLKN-lab.git
       cd MLKN-lab

2. Install Dependencies:

   pip install -r requirements.txt

3. Download Data:
   • Download datasets from Zenodo or use git lfs pull for Git LFS files.
4. Run Notebooks:
   • Open notebooks/data_cleaning.ipynb in Jupyter to reproduce the data processing.

Learn More: Open Science Principles | FAIR Principles

🔹 Repository Structure

MLKN-lab/
    ├── data/            # Raw and processed data (Git LFS)
    │   ├── raw/         # Raw OpenAlex data
    │   └── processed/   # Cleaned data (CSV/JSON)
    ├── notebooks/       # Jupyter Notebooks (step-by-step tutorials)
    │   ├── data_cleaning.ipynb
    │   └── network_analysis.ipynb
    ├── src/             # Source code
    │   ├── python/      # Data processing (Pandas, NetworkX)
    │   └── js/          # D3.js code for MLKN.hypergraph
    ├── docs/            # Documentation
    └── README.md        # Setup instructions

🔹 How to Contribute

1. Fork the Repository: Create your own copy on GitHub.
2. Clone Locally:

   git clone https://github.com/FrancoisPapin/MLKN-lab.git

3. Create a Branch:

   git checkout -b new-feature

4. Commit Changes:

   git commit -m "Add new classification layer"

5. Submit a Pull Request: Propose your changes for review.

🔹 Key Files

File/Folder	Description	Link
data/raw/	Raw OpenAlex data (via Git LFS).	Zenodo Dataset
notebooks/	Jupyter Notebooks for data cleaning and analysis.	View on GitHub
src/python/	Python scripts (Pandas, NetworkX).	View on GitHub
src/js/	D3.js code for MLKN.hypergraph.	View on GitHub
README.md	Instructions for setup and usage.	View on GitHub