diff --git a/.gitea/DEVELOPING.md b/.gitea/DEVELOPING.md new file mode 100644 index 0000000..3e68f20 --- /dev/null +++ b/.gitea/DEVELOPING.md @@ -0,0 +1,76 @@ +# DepGraph + +DepGraph is a utility for querying information about project dependencies +count and their licenses. + +## Developing + +### Tech Stack + +- Language: [Go](https://go.dev); +- Tools: + - Linting: [`golangci-lint`, version 2](https://golangci-lint.run/); + - Formatting: [`gofumpt`](https://github.com/mvdan/gofumpt) and + [`goimports`](https://pkg.go.dev/golang.org/x/tools/cmd/goimports), + both already provided by `golangci-lint`; + +### Design + +The project design should be language/ecosystem-agnostic, so any sort of +module system of any language can be developed for the project. However, +the main focus will be primarily Go and JavaScript. + +``` +DepGraph + |- repository # Retrievers to get data from Gitea, GitLab, etc + |- service # Transform the data from repositories into a fs.FS + | filesystem that adapters can use + |- adapter # Adapters are who does the magic, they get the + | filesystem and detect files such as go.mod, + | package.json, etc to know what adapter to use + |- adapter/go # Adapter for Go's module system + |- adapter/npm # Adapter for NodeJS/NPM/JavaScript module system + |- cmd # CLI interface of the project + |- depgraph.go # API interface of the project +``` + +For now, the project will focus on providing a Go API and CLI interface to fetch +and process dependencies. When the interfaces are mature, we will build on +top of it a website that consumes the returned list of dependencies and shows +it on a graph and other more visual features. + +#### Adapters + +The core workflow of the adapters/dependencies parsing is defined below: + +![Depgraph Spec Core](./depgraph-design-core.svg) + +This from were the final list of dependencies will be created and from were +specific language adapters are called from. + +##### Go Adapter + +![DepGraph Spec Golang](./depgraph-spec-golang.svg) + +Since Go doesn't have a unified package repository, the adapter has a lot +of different options and paths it can work with. But thankfully, Go does +provide a proxy that caches the source code of public packages, which +we can use to retrieve the source code if we need it very easily for most +packages. + +Useful resources to read and use: +- [Go Modules Reference](https://go.dev/ref/mod) + - [`go.mod` files](https://go.dev/ref/mod#go-mod-file) + - [`go.sum` files](https://go.dev/ref/mod#go-sum-files) + - [GOPROXY protocol](https://go.dev/ref/mod#goproxy-protocol) + - [proxy.golang.org](https://proxy.golang.org/) + - [Module Cache](https://go.dev/ref/mod#module-cache) + - [Module zip files](https://go.dev/ref/mod#zip-files) +- Packages to handle modules + - [golang.org/x/mod/sumdb/dirhash](https://pkg.go.dev/golang.org/x/mod/sumdb/dirhash) + - [golang.org/x/mod/mofile](https://pkg.go.dev/golang.org/x/mod/modfile) + - [golang.org/x/mod/zip](https://pkg.go.dev/golang.org/x/mod@v0.32.0/zip) + - [golang.org/x/mod/module](https://pkg.go.dev/golang.org/x/mod/module) +- The [golang.org/x/tools/go/packages](https://pkg.go.dev/golang.org/x/tools@v0.41.0/go/packages) + package may also be useful to parse packages, specially from source code, + but the current design/spec illustrated above does not take it into account. diff --git a/.gitea/depgraph-design-core.svg b/.gitea/depgraph-design-core.svg new file mode 100644 index 0000000..b2a5ef1 --- /dev/null +++ b/.gitea/depgraph-design-core.svg @@ -0,0 +1,2 @@ +InputOpen withos.OpenRoot()Is Localpathfs.FS InterfacehashfileSystem: fs.FSdepedencies: []*DependencyDependencyAdd to Queue*Dependency*Dependency*Dependency*Dependency*Dependency...Process Queue(Go channel)Worker...Queue Workers Group(Go goroutines)WorkerWorkerWorkerWorkerProcessdependencyon availableworkerWorker*Dependencymutex: will have methods to be ableto change properties concurrentlyprovideread-onlydependencyIterate over available managersrun managerattributes: map[string]anymerge returned attributesempty map[hash]*Dependencyemtpy map[string]anyempty hash*Dependency*Dependency*Dependency*Dependency...found dependenciesadd as child dependenciesadd each dependency to queue to be processedjoined/final hash*Dependencyhashdependencies*Dependencyhashdependencies*DependencyhashdependenciesProcessed DependenciesCheck every processeddependencyCheck every dependency'sdependencies (not recursively)Check if every dependency has a hashAll dependencies' dependencies have ahash?No: continuechecking untilyesEvery process iscompletedYesClose channel(?)OutputGoNPMAvailable managersRust...use all attributes to generate hashadd fs.FS to dependency, if it doesn't has one already \ No newline at end of file diff --git a/.gitea/depgraph-spec-golang.svg b/.gitea/depgraph-spec-golang.svg new file mode 100644 index 0000000..b155c31 --- /dev/null +++ b/.gitea/depgraph-spec-golang.svg @@ -0,0 +1,2 @@ +Input:(Read-Only)DependencyParse go.mod filehashfileSystem: fs.FSattributes: map[string]anyHas go.mod fileFor each required moduleParse go.sum file(if exists)hashfileSystem: fs.FSdepedencies: []*DependencyDependencymutex: will have methods to be ableto change properties concurrentlyattributes: map[string]any(Read Only) Dependencyempty hashAttributesGO-Version: module.VersionGO-ZipHash: stringRequireget versionIs on go.sum?Is there a .ziphashof the module onGOMODCACHE?GO-ModHash: stringIs there a .zip onGOMODCACHE?NoNoget zip hashget go.mod hashHash go.mod file(if exists)Hash contents withgolang.org/x/mod/sumdb/dirhashSave as fs.FSIs the source codeon GOMODCACHE?NoDownload zip fromGOPROXYNoDownload from VCSNot foundNot found / unableto downloadCrawl pkg.go.dev pageto get location ofgo.mod fileGive upNot foundUnable todownloadErrors: []errorAdd error to notify thatthe dependency wasn't fullyable to parse/get infoAttributesGO-ZipHash?: stringGO-ModHash?: stringErrors: []error*Dependency*Dependency*Dependency...add todependencies listGO-Version?: module.VersionPath?: stringVesion?: stringFinal AttributesGO-ZipHash?: stringGO-ModHash?: stringErrors: []errorGO-Version?: module.VersionPath?: stringVesion?: stringUse fs.FSHas filesystem?YesHas GO-Version.Path?NoHas GO-Version.Version?YesNoWe don't haveany way to fetchthe dependency.Return ErrorIs there a .zip onGOCACHE?Is the source codeon GOMODCACHE?NoDownload zip fromGOPROXYNoDownload from VCSNot foundYesDowload source codeTBD: Download thelatest version of thesource code and add awarning to Errorsattribute, or justreturn a error for notbeing able to downloadthe correct versionNoHas GO-Version?Get from go.modNoHas GO-ModHash?Hash go.modNoHas GO-ZipHash?Hash the source codewith dirhashNofileSystem: fs.FSattributes: map[string]anyOutputdepedencies: []*DependencyTBD: Get from VCSNo go.mod file \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index df70dc2..b0cc9a4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,5 +1,8 @@ # Contribuiting +[See the development guide](./.gitea/DEVELOPING.md) to know what is the structure of +these initial states of development. + ## Style Guide ### Code