History

Peter Gardfjäll e28cc79c92 Improve sync performance for pull-mirrors (#19125 ) This addresses https://github.com/go-gitea/gitea/issues/18352 It aims to improve performance (and resource use) of the `SyncReleasesWithTags` operation for pull-mirrors. For large repositories with many tags, `SyncReleasesWithTags` can be a costly operation (taking several minutes to complete). The reason is two-fold: 1. on sync, every upstream repo tag is compared (for changes) against existing local entries in the release table to ensure that they are up-to-date. 2. the procedure for getting _each tag_ involves a series of git operations ```bash git show-ref --tags -- v8.2.4477 git cat-file -t 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git cat-file -p 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git rev-list --count 29ab6ce9f36660cffaad3c8789e71162e5db5d2f ``` of which the `git rev-list --count` can be particularly heavy. This PR optimizes performance for pull-mirrors. We utilize the fact that a pull-mirror is always identical to its upstream and rebuild the entire release table on every sync and use a batch `git for-each-ref .. refs/tags` call to retrieve all tags in one go. For large mirror repos, with hundreds of annotated tags, this brings down the duration of the sync operation from several minutes to a few seconds. A few unscientific examples run on my local machine: - https://github.com/spring-projects/spring-boot (223 tags) - before: `0m28,673s` - after: `0m2,244s` - https://github.com/kubernetes/kubernetes (890 tags) - before: `8m00s` - after: `0m8,520s` - https://github.com/vim/vim (13954 tags) - before: `14m20,383s` - after: `0m35,467s` I added a `foreachref` package which contains a flexible way of specifying which reference fields are of interest (`git-for-each-ref(1)`) and to produce a parser for the expected output. These could be reused in other places where `for-each-ref` is used. I'll add unit tests for those if the overall PR looks promising.		3 years ago
..
foreachref	Improve sync performance for pull-mirrors (#19125 )	3 years ago
pipeline	Change git.cmd to RunWithContext (#18693 )	3 years ago
tests/repos	rm .sample hooks which aren't used (#19101 )	3 years ago
README.md	move code.gitea.io/git to code.gitea.io/gitea/modules/git (#6364 )	6 years ago
batch_reader.go	Change git.cmd to RunWithContext (#18693 )	3 years ago
blame.go	Propagate context and ensure git commands run in request context (#17868 )	3 years ago
blame_test.go	refactor: move from io/ioutil to io and os package (#17109 )	3 years ago
blob.go	Read expected buffer size (#17409 )	3 years ago
blob_gogit.go	Add bundle download for repository (#14538 )	3 years ago
blob_nogogit.go	Make Requests Processes and create process hierarchy. Associate OpenRepository with context. (#17125 )	3 years ago
blob_test.go	Make git.OpenRepository accept Context (#19260 )	3 years ago
command.go	Refactor `git.Command.Run*`, introduce `RunWithContextString` and `RunWithContextBytes` (#19266 )	3 years ago
command_race_test.go	Refactor `git.Command.Run*`, introduce `RunWithContextString` and `RunWithContextBytes` (#19266 )	3 years ago
command_test.go	Refactor `git.Command.Run*`, introduce `RunWithContextString` and `RunWithContextBytes` (#19266 )	3 years ago
commit.go	Change git.cmd to RunWithContext (#18693 )	3 years ago
commit_convert_gogit.go	Add bundle download for repository (#14538 )	3 years ago
commit_info.go	Use native git variants by default with go-git variants as build tag (#13673 )	4 years ago
commit_info_gogit.go	format with gofumpt (#18184 )	3 years ago
commit_info_nogogit.go	format with gofumpt (#18184 )	3 years ago
commit_info_test.go	Make git.OpenRepository accept Context (#19260 )	3 years ago
commit_reader.go	Tagger can be empty, as can Commit and Author - tolerate this (#15835 )	4 years ago
commit_test.go	Make git.OpenRepository accept Context (#19260 )	3 years ago
diff.go	Make git.OpenRepository accept Context (#19260 )	3 years ago
diff_test.go	Prevent off-by-one error on comments on newly appended lines (#18029 )	3 years ago
error.go	Fix 500 Error with branch and tag sharing the same name #15592 (#16040 )	4 years ago
git.go	Refactor `git.Command.Run*`, introduce `RunWithContextString` and `RunWithContextBytes` (#19266 )	3 years ago
git_test.go	Use gitea logging module for git module (#16243 )	4 years ago
hook.go	format with gofumpt (#18184 )	3 years ago
last_commit_cache.go	Defer Last Commit Info (#16467 )	3 years ago
last_commit_cache_gogit.go	format with gofumpt (#18184 )	3 years ago
last_commit_cache_nogogit.go	Defer Last Commit Info (#16467 )	3 years ago
lfs.go	Unexport git.GlobalCommandArgs (#18376 )	3 years ago
log_name_status.go	Change git.cmd to RunWithContext (#18693 )	3 years ago
notes.go	Use native git variants by default with go-git variants as build tag (#13673 )	4 years ago
notes_gogit.go	Stop logging an error when notes are not found (#18626 )	3 years ago
notes_nogogit.go	Stop logging an error when notes are not found (#18626 )	3 years ago
notes_test.go	Make git.OpenRepository accept Context (#19260 )	3 years ago
parse_gogit.go	Add bundle download for repository (#14538 )	3 years ago
parse_gogit_test.go	Add bundle download for repository (#14538 )	3 years ago
parse_nogogit.go	Add bundle download for repository (#14538 )	3 years ago
parse_nogogit_test.go	format with gofumpt (#18184 )	3 years ago
ref.go	Refactor various strings (#17784 )	3 years ago
remote.go	Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552 )	3 years ago
repo.go	Use a more general (and faster) method to sanitize URLs with credentials (#19239 )	3 years ago
repo_archive.go	Change git.cmd to RunWithContext (#18693 )	3 years ago
repo_attribute.go	Don't show context cancelled errors in attribute reader (#19006 )	3 years ago
repo_attribute_test.go	format with gofumpt (#18184 )	3 years ago
repo_base.go	Make git.OpenRepository accept Context (#19260 )	3 years ago
repo_base_gogit.go	Make git.OpenRepository accept Context (#19260 )	3 years ago
repo_base_nogogit.go	Make git.OpenRepository accept Context (#19260 )	3 years ago
repo_blame.go	Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552 )	3 years ago
repo_blob.go	Use native git variants by default with go-git variants as build tag (#13673 )	4 years ago
repo_blob_gogit.go	Add bundle download for repository (#14538 )	3 years ago
repo_blob_nogogit.go	Add bundle download for repository (#14538 )	3 years ago
repo_blob_test.go	Make git.OpenRepository accept Context (#19260 )	3 years ago
repo_branch.go	Make git.OpenRepository accept Context (#19260 )	3 years ago
repo_branch_gogit.go	Make git.OpenRepository accept Context (#19260 )	3 years ago
repo_branch_nogogit.go	Use full output of git show-ref --tags to get tags for PushUpdateAddTag (#19235 )	3 years ago
repo_branch_test.go	Make git.OpenRepository accept Context (#19260 )	3 years ago
repo_commit.go	Change git.cmd to RunWithContext (#18693 )	3 years ago
repo_commit_gogit.go	Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552 )	3 years ago
repo_commit_nogogit.go	Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552 )	3 years ago
repo_commit_test.go	Make git.OpenRepository accept Context (#19260 )	3 years ago
repo_commitgraph.go	Use full output of git show-ref --tags to get tags for PushUpdateAddTag (#19235 )	3 years ago
repo_commitgraph_gogit.go	Add bundle download for repository (#14538 )	3 years ago
repo_compare.go	Change git.cmd to RunWithContext (#18693 )	3 years ago
repo_compare_test.go	Make git.OpenRepository accept Context (#19260 )	3 years ago
repo_gpg.go	Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552 )	3 years ago
repo_hook.go	move code.gitea.io/git to code.gitea.io/gitea/modules/git (#6364 )	6 years ago
repo_index.go	Change git.cmd to RunWithContext (#18693 )	3 years ago
repo_language_stats.go	format with gofumpt (#18184 )	3 years ago
repo_language_stats_gogit.go	Add .gitattribute assisted language detection to blame, diff and render (#17590 )	3 years ago
repo_language_stats_nogogit.go	Prevent dangling GetAttribute calls (#18754 )	3 years ago
repo_language_stats_test.go	Make git.OpenRepository accept Context (#19260 )	3 years ago
repo_object.go	Change git.cmd to RunWithContext (#18693 )	3 years ago
repo_ref.go	Use native git variants by default with go-git variants as build tag (#13673 )	4 years ago
repo_ref_gogit.go	Add bundle download for repository (#14538 )	3 years ago
repo_ref_nogogit.go	Change git.cmd to RunWithContext (#18693 )	3 years ago
repo_ref_test.go	Make git.OpenRepository accept Context (#19260 )	3 years ago
repo_stats.go	Change git.cmd to RunWithContext (#18693 )	3 years ago
repo_stats_test.go	Make git.OpenRepository accept Context (#19260 )	3 years ago
repo_tag.go	Improve sync performance for pull-mirrors (#19125 )	3 years ago
repo_tag_gogit.go	Use full output of git show-ref --tags to get tags for PushUpdateAddTag (#19235 )	3 years ago
repo_tag_nogogit.go	Use full output of git show-ref --tags to get tags for PushUpdateAddTag (#19235 )	3 years ago
repo_tag_test.go	Improve sync performance for pull-mirrors (#19125 )	3 years ago
repo_test.go	Make git.OpenRepository accept Context (#19260 )	3 years ago
repo_tree.go	Change git.cmd to RunWithContext (#18693 )	3 years ago
repo_tree_gogit.go	Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552 )	3 years ago
repo_tree_nogogit.go	Fix NPE on try to get tag reference via API (#18245 )	3 years ago
sha1.go	Use native git variants by default with go-git variants as build tag (#13673 )	4 years ago
sha1_gogit.go	Add bundle download for repository (#14538 )	3 years ago
sha1_nogogit.go	Add bundle download for repository (#14538 )	3 years ago
signature.go	Use native git variants by default with go-git variants as build tag (#13673 )	4 years ago
signature_gogit.go	Add bundle download for repository (#14538 )	3 years ago
signature_nogogit.go	Add bundle download for repository (#14538 )	3 years ago
submodule.go	Fix some lints (#17337 )	3 years ago
submodule_test.go	format with gofumpt (#18184 )	3 years ago
tag.go	format with gofumpt (#18184 )	3 years ago
tag_test.go	Fix NPE on try to get tag reference via API (#18245 )	3 years ago
tree.go	Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552 )	3 years ago
tree_blob.go	Use native git variants by default with go-git variants as build tag (#13673 )	4 years ago
tree_blob_gogit.go	format with gofumpt (#18184 )	3 years ago
tree_blob_nogogit.go	Add bundle download for repository (#14538 )	3 years ago
tree_entry.go	On open repository open common cat file batch and batch-check (#15667 )	4 years ago
tree_entry_gogit.go	Add bundle download for repository (#14538 )	3 years ago
tree_entry_mode.go	format with gofumpt (#18184 )	3 years ago
tree_entry_nogogit.go	Propagate context and ensure git commands run in request context (#17868 )	3 years ago
tree_entry_test.go	Make git.OpenRepository accept Context (#19260 )	3 years ago
tree_gogit.go	Add bundle download for repository (#14538 )	3 years ago
tree_nogogit.go	Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552 )	3 years ago
utils.go	Simplify parameter types (#18006 )	3 years ago
utils_test.go	Fix ref links in issue overviews for tags (#8742 )	5 years ago

README.md

Git Module

This module is merged from https://github.com/go-gitea/git which is a Go module to access Git through shell commands. Now it's a part of gitea's main repository for easier pull request.