Using Git Repository Sources

As an alternative to local files, pgEdge Document Loader can clone and process documentation directly from Git repositories. This is useful for:

• Loading documentation from remote repositories without manual cloning
• Processing specific branches or tags (e.g., versioned documentation)
• Automated pipelines that fetch and load docs from source control

Git Source Options

*Either --source or --git-url is required, but not both.

Basic Usage

Clone a repository and process all supported files from the root:

pgedge-docloader \
    --git-url https://github.com/org/docs-repo.git \
    --db-host localhost \
    --db-name mydb \
    --db-user myuser \
    --db-table documents \
    --col-doc-content content \
    --col-file-name filename

Processing a Specific Directory

Use --git-doc-path to process files from a specific directory within the repository:

pgedge-docloader \
    --git-url https://github.com/org/project.git \
    --git-doc-path docs/api \
    --db-host localhost \
    --db-name mydb \
    --db-user myuser \
    --db-table documents \
    --col-doc-content content

The --git-doc-path option supports glob patterns:

# Process only markdown files in the docs directory
pgedge-docloader \
    --git-url https://github.com/org/project.git \
    --git-doc-path "docs/**/*.md" \
    --config config.yml

Multiple Source Patterns

You can specify multiple --git-doc-path options to process files from different locations:

# Process both docs directory and root-level markdown files
pgedge-docloader \
    --git-url https://github.com/org/project.git \
    --git-doc-path "docs/**/*.md" \
    --git-doc-path "*.md" \
    --config config.yml

In a configuration file, use a YAML list:

git-url: https://github.com/org/project.git
git-doc-path:
    - "docs/**/*.md"
    - "*.md"

Working with Branches and Tags

Checkout a Specific Branch

pgedge-docloader \
    --git-url https://github.com/org/docs.git \
    --git-branch main \
    --git-doc-path docs \
    --config config.yml

Checkout a Specific Tag

Use tags for versioned documentation:

pgedge-docloader \
    --git-url https://github.com/org/project.git \
    --git-tag v2.0.0 \
    --git-doc-path docs \
    --set-column version="2.0.0" \
    --config config.yml

Accepted Branch and Tag Names

Branch and tag names are passed to git as command line arguments, so they are validated before use to make sure they cannot be read as options. A name is accepted only when it:

• Starts with a letter or a digit.
• Contains only letters, digits and the characters ., _, / and -.
• Does not contain .., and does not end with / or .lock.

Anything else is rejected with an error before git is run. This covers ordinary names such as main, release/18-stable and v1.2.3, while refusing input such as --upload-pack=... that git would otherwise interpret as an option.

Symbolic Links in Cloned Repositories

Documents are read through a directory scoped to the source tree, so a symbolic link cannot be used to pull a file in from elsewhere on the machine. This matters when loading a repository you do not control: without it, a link named guide.md pointing at ~/.ssh/id_rsa would load that key into your database.

• Relative links that stay inside the source tree are followed as usual.
• Links that resolve outside the tree are skipped, and the run reports an error for each one.
• Absolute links are always skipped, because their target cannot be checked against the source tree.

Persistent Clone Directory

By default, repositories are cloned to a temporary directory and removed after processing. For repeated runs, you can specify a persistent clone directory:

pgedge-docloader \
    --git-url https://github.com/org/docs.git \
    --git-clone-dir /var/cache/docloader/repos \
    --git-keep-clone \
    --config config.yml

On subsequent runs with --git-skip-fetch, the tool will reuse the existing clone without fetching updates:

pgedge-docloader \
    --git-url https://github.com/org/docs.git \
    --git-clone-dir /var/cache/docloader/repos \
    --git-keep-clone \
    --git-skip-fetch \
    --config config.yml

Configuration File Example

Git source options can also be specified in a configuration file:

# Git source configuration
git-url: https://github.com/org/docs-repo.git
git-branch: main
git-doc-path:
    - "docs/**/*.md"
    - "*.md"
git-clone-dir: /var/cache/docloader/repos
git-keep-clone: true

# Database configuration
db-host: localhost
db-name: mydb
db-user: myuser
db-table: documents

# Column mappings
col-doc-content: content
col-file-name: filename
col-doc-title: title

# Custom metadata
custom-columns:
    source: "git-repo"
    project: "my-project"

Then run with:

pgedge-docloader --config config.yml

Authentication

HTTPS URLs

For public repositories, use the HTTPS URL directly:

--git-url https://github.com/org/public-repo.git

For private repositories, you can use a personal access token in the URL:

--git-url https://TOKEN@github.com/org/private-repo.git

Or configure Git credential helpers before running the tool.

SSH URLs

For SSH authentication, ensure your SSH keys are configured:

--git-url git@github.com:org/repo.git

Error Handling

The tool will fail with a clear error message if:

• Git is not installed on the system
• The repository URL is invalid or inaccessible
• The repository URL or the branch or tag name is not in an accepted form
• The specified branch or tag does not exist
• The --git-doc-path does not exist in the repository

Best Practices

1. Use tags for versioned docs: When loading documentation for specific software versions, use --git-tag to ensure consistency.

2. Cache clones for repeated runs: Use --git-clone-dir and --git-keep-clone to avoid re-cloning on every run.

3. Use --git-skip-fetch carefully: Only skip fetching when you're sure the local clone is up-to-date.

4. Set version metadata: Use --set-column to add version information when processing tagged releases:

   --git-tag v1.2.3 --set-column version="1.2.3"