depgraph/DEVELOPING.md

DepGraph

DepGraph is a small website and utility for querying information about project dependencies count and their licenses.

This file defines the design of the project and basic guidelines for what will be develop.

Developing

Tech Stack

• Back-end: Go;
  • Routing: Smalltrip.
• Front-end: Server-side generated with html/template
  • HTMX for bits of interactivity;
  • TailwindCSS for some or all styling;
  • Hot Reloading will be provided by x/xtemplate.
• Tools:
  • Linting: golangci-lint, version 2;
  • Formatting: gofumpt and goimports, both already provided by golangci-lint;
  • Deployment & Building: Docker and Nix;

Design

The project design should be language/ecosystem-agnostic, so any sort of module system of any language can be developed for the project. However, the main focus will be primarily Go and JavaScript.

DepGraph
 |- cache        # Key-Value Cache provider or adapter for DBs
 |- repository   # Retrievers to get data from Gitea, GitLab, etc
 |- service      # Transform the data from repositories into a fs.FS
 |                 filesystem that adapters can use
 |- adapter      # Adapters are who does the magic, they get the
 |                 filesystem and detect files such as go.mod,
 |                 package.json, etc to know what adapter to use
 |- adapter/go     # Adapter for Go's module system
 |- adapter/npm    # Adapter for NodeJS/NPM/JavaScript module system
 |- router       # Router for the web interface and API
 |- templates    # HTML templates for the front-end
 |- cmd          # CLI interface of the aplication
 |- main.go      # Entry point of the program

After the project is more mature and has a solid working, it may be refactored to add some more structure to the project and a better CLI, Web API and interface and Go interface:

DepGraph
 |- cache        # Key-Value Cache provider or adapter for DBs
 |- repository   # Retrievers to get data from Gitea, GitLab, etc
 |- service      # Transform the data from repositories into a fs.FS
 |                 filesystem that adapters can use
 |- adapter      # Adapters are who does the magic, they get the
 |                 filesystem and detect files such as go.mod,
 |                 package.json, etc to know what adapter to use
 |- adapter/go     # Adapter for Go's module system
 |- adapter/npm    # Adapter for NodeJS/NPM/JavaScript module system
-|- router       # Router for the web interface and API
-|- template     # HTML templates for the front-end
+|- api          # Code related to the web api and interface
+|- api/router     # Router for the web interface and API
+|- api/template   # HTML templates for the front-end
 |- cmd          # CLI interface of the aplication
+|- cmd/cmd.go     # Root command of the application
+|- cmd/api.go     # Subcommand for running the web interface
 |- main.go      # Entry point of the program
+|- depgraph.go    # Entry point for the Go API interface

Adapters

The Dependency Data Format

Since for displaying the dependencies we just need a minimal amount of information, we will probably end up probably end up with a data format/struct something similar to this pseudo-code:

Dependency{
	url: "https://example.com" # The URL of the dependency
	                             (NPM, pkg.go.dev, git repository, etc)
	version: "v1.6.9"          # The version of the dependency
	hash: "ABCDEFGHIJKLMNOPQR" # The HASH of the dependency
	dependencies: []Dependency # The dependency's depedencies
+   license: "MIT"             # SPDX License, info: https://spdx.dev/
}

Note: the license field is not necessary for the start of development and is more related for future (possible) features.

In other words, it will be a Tree.

The Go Adapter

The Go adapter should, in principle, be simple and mostly rely on the conventions of the Go module system. To reduce the amount of requests we may need to do for third-party websites, we will try to find multiple files, from least- to most-request-heavy:

1. Check for a go.sum file
   • If the file exists we quit and use it as the "de-facto" list of dependencies;
   • To parse it we just need to iterate over each line and get the module name, version and hash that are separated by spaces. Check the go.sum file documentation;
   • The go.sum file already has a hash that we can use to pin the dependency and use on the cache.
2. Check for a go.mod file
   • If it exists, we use golang.org/x/mod/modfile to parse it and have a list of dependencies;
   • To reduce the amount of requests, we just hash the module-name+version combination to use in the cache. We could in the future request each module files and hash their go.mod file like how go.sum does it.
3. If no go.sum or go.mod file exists, and we want to be feature-complete, we would then need to download the module source code and use one of: - Use golang.org/x/tools/go/packages - Use go list -f '{{join .Imports "\n"}}' ., but it doesn't return the versions of the packages and needs the source code to be on the OS's filesystem (There may be some Go interface for this instead of CLI).

• We also would check for go.work.sum and go.work in the same way to get workspaces' dependencies.
• This process could be recursive to get all dependencies in the project and don't be limited by just the root. It depends if also Gitea/GitHub's and other providers' APIs have some endpoint to search specific files.