Add push to remote mirror repository (#15157)
* Added push mirror model. * Integrated push mirror into queue. * Moved methods into own file. * Added basic implementation. * Mirror wiki too. * Removed duplicated method. * Get url for different remotes. * Added migration. * Unified remote url access. * Add/Remove push mirror remotes. * Prevent hangs with missing credentials. * Moved code between files. * Changed sanitizer interface. * Added push mirror backend methods. * Only update the mirror remote. * Limit refs on push. * Added UI part. * Added missing table. * Delete mirror if repository gets removed. * Changed signature. Handle object errors. * Added upload method. * Added "upload" unit tests. * Added transfer adapter unit tests. * Send correct headers. * Added pushing of LFS objects. * Added more logging. * Simpler body handling. * Process files in batches to reduce HTTP calls. * Added created timestamp. * Fixed invalid column name. * Changed name to prevent xorm auto setting. * Remove table header im empty. * Strip exit code from error message. * Added docs page about mirroring. * Fixed date. * Fixed merge errors. * Moved test to integrations. * Added push mirror test. * Added test.tokarchuk/v1.17
KN4CK3R
4 years ago
committed by
GitHub
parent
5d113bdd19
commit
440039c0cc
@ -0,0 +1,88 @@ |
||||
--- |
||||
date: "2021-05-13T00:00:00-00:00" |
||||
title: "Repository Mirror" |
||||
slug: "repo-mirror" |
||||
weight: 45 |
||||
toc: false |
||||
draft: false |
||||
menu: |
||||
sidebar: |
||||
parent: "advanced" |
||||
name: "Repository Mirror" |
||||
weight: 45 |
||||
identifier: "repo-mirror" |
||||
--- |
||||
|
||||
# Repository Mirror |
||||
|
||||
Repository mirroring allows for the mirroring of repositories to and from external sources. You can use it to mirror branches, tags, and commits between repositories. |
||||
|
||||
**Table of Contents** |
||||
|
||||
{{< toc >}} |
||||
|
||||
## Use cases |
||||
|
||||
The following are some possible use cases for repository mirroring: |
||||
|
||||
- You migrated to Gitea but still need to keep your project in another source. In that case, you can simply set it up to mirror to Gitea (pull) and all the essential history of commits, tags, and branches are available in your Gitea instance. |
||||
- You have old projects in another source that you don’t use actively anymore, but don’t want to remove for archiving purposes. In that case, you can create a push mirror so that your active Gitea repository can push its changes to the old location. |
||||
|
||||
## Pulling from a remote repository |
||||
|
||||
For an existing remote repository, you can set up pull mirroring as follows: |
||||
|
||||
1. Select **New Migration** in the **Create...** menu on the top right. |
||||
2. Select the remote repository service. |
||||
3. Enter a repository URL. |
||||
4. If the repository needs authentication fill in your authentication information. |
||||
5. Check the box **This repository will be a mirror**. |
||||
5. Select **Migrate repository** to save the configuration. |
||||
|
||||
The repository now gets mirrored periodically from the remote repository. You can force a sync by selecting **Synchronize Now** in the repository settings. |
||||
|
||||
## Pushing to a remote repository |
||||
|
||||
For an existing repository, you can set up push mirroring as follows: |
||||
|
||||
1. In your repository, go to **Settings** > **Repository**, and then the **Mirror Settings** section. |
||||
2. Enter a repository URL. |
||||
3. If the repository needs authentication expand the **Authorization** section and fill in your authentication information. |
||||
4. Select **Add Push Mirror** to save the configuration. |
||||
|
||||
The repository now gets mirrored periodically to the remote repository. You can force a sync by selecting **Synchronize Now**. In case of an error a message displayed to help you resolve it. |
||||
|
||||
:exclamation::exclamation: **NOTE:** This will force push to the remote repository. This will overwrite any changes in the remote repository! :exclamation::exclamation: |
||||
|
||||
### Setting up a push mirror from Gitea to GitHub |
||||
|
||||
To set up a mirror from Gitea to GitHub, you need to follow these steps: |
||||
|
||||
1. Create a [GitHub personal access token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) with the *public_repo* box checked. |
||||
2. Fill in the **Git Remote Repository URL**: `https://github.com/<your_github_group>/<your_github_project>.git`. |
||||
3. Fill in the **Authorization** fields with your GitHub username and the personal access token. |
||||
4. Select **Add Push Mirror** to save the configuration. |
||||
|
||||
The repository pushes shortly thereafter. To force a push, select the **Synchronize Now** button. |
||||
|
||||
### Setting up a push mirror from Gitea to GitLab |
||||
|
||||
To set up a mirror from Gitea to GitLab, you need to follow these steps: |
||||
|
||||
1. Create a [GitLab personal access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) with *write_repository* scope. |
||||
2. Fill in the **Git Remote Repository URL**: `https://<destination host>/<your_gitlab_group_or_name>/<your_gitlab_project>.git`. |
||||
3. Fill in the **Authorization** fields with `oauth2` as **Username** and your GitLab personal access token as **Password**. |
||||
4. Select **Add Push Mirror** to save the configuration. |
||||
|
||||
The repository pushes shortly thereafter. To force a push, select the **Synchronize Now** button. |
||||
|
||||
### Setting up a push mirror from Gitea to Bitbucket |
||||
|
||||
To set up a mirror from Gitea to Bitbucket, you need to follow these steps: |
||||
|
||||
1. Create a [Bitbucket app password](https://support.atlassian.com/bitbucket-cloud/docs/app-passwords/) with the *Repository Write* box checked. |
||||
2. Fill in the **Git Remote Repository URL**: `https://bitbucket.org/<your_bitbucket_group_or_name>/<your_bitbucket_project>.git`. |
||||
3. Fill in the **Authorization** fields with your Bitbucket username and the app password as **Password**. |
||||
4. Select **Add Push Mirror** to save the configuration. |
||||
|
||||
The repository pushes shortly thereafter. To force a push, select the **Synchronize Now** button. |
||||
@ -0,0 +1,86 @@ |
||||
// Copyright 2021 The Gitea Authors. All rights reserved.
|
||||
// Use of this source code is governed by a MIT-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package integrations |
||||
|
||||
import ( |
||||
"context" |
||||
"fmt" |
||||
"net/http" |
||||
"net/url" |
||||
"testing" |
||||
|
||||
"code.gitea.io/gitea/models" |
||||
"code.gitea.io/gitea/modules/git" |
||||
"code.gitea.io/gitea/modules/repository" |
||||
"code.gitea.io/gitea/modules/setting" |
||||
mirror_service "code.gitea.io/gitea/services/mirror" |
||||
|
||||
"github.com/stretchr/testify/assert" |
||||
) |
||||
|
||||
func TestMirrorPush(t *testing.T) { |
||||
onGiteaRun(t, testMirrorPush) |
||||
} |
||||
|
||||
func testMirrorPush(t *testing.T, u *url.URL) { |
||||
defer prepareTestEnv(t)() |
||||
|
||||
setting.Migrations.AllowLocalNetworks = true |
||||
|
||||
user := models.AssertExistsAndLoadBean(t, &models.User{ID: 2}).(*models.User) |
||||
srcRepo := models.AssertExistsAndLoadBean(t, &models.Repository{ID: 1}).(*models.Repository) |
||||
|
||||
mirrorRepo, err := repository.CreateRepository(user, user, models.CreateRepoOptions{ |
||||
Name: "test-push-mirror", |
||||
}) |
||||
assert.NoError(t, err) |
||||
|
||||
ctx := NewAPITestContext(t, user.LowerName, srcRepo.Name) |
||||
|
||||
doCreatePushMirror(ctx, fmt.Sprintf("%s%s/%s", u.String(), url.PathEscape(ctx.Username), url.PathEscape(mirrorRepo.Name)), user.LowerName, userPassword)(t) |
||||
|
||||
mirrors, err := models.GetPushMirrorsByRepoID(srcRepo.ID) |
||||
assert.NoError(t, err) |
||||
assert.Len(t, mirrors, 1) |
||||
|
||||
ok := mirror_service.SyncPushMirror(context.Background(), mirrors[0].ID) |
||||
assert.True(t, ok) |
||||
|
||||
srcGitRepo, err := git.OpenRepository(srcRepo.RepoPath()) |
||||
assert.NoError(t, err) |
||||
defer srcGitRepo.Close() |
||||
|
||||
srcCommit, err := srcGitRepo.GetBranchCommit("master") |
||||
assert.NoError(t, err) |
||||
|
||||
mirrorGitRepo, err := git.OpenRepository(mirrorRepo.RepoPath()) |
||||
assert.NoError(t, err) |
||||
defer mirrorGitRepo.Close() |
||||
|
||||
mirrorCommit, err := mirrorGitRepo.GetBranchCommit("master") |
||||
assert.NoError(t, err) |
||||
|
||||
assert.Equal(t, srcCommit.ID, mirrorCommit.ID) |
||||
} |
||||
|
||||
func doCreatePushMirror(ctx APITestContext, address, username, password string) func(t *testing.T) { |
||||
return func(t *testing.T) { |
||||
csrf := GetCSRF(t, ctx.Session, fmt.Sprintf("/%s/%s/settings", url.PathEscape(ctx.Username), url.PathEscape(ctx.Reponame))) |
||||
|
||||
req := NewRequestWithValues(t, "POST", fmt.Sprintf("/%s/%s/settings", url.PathEscape(ctx.Username), url.PathEscape(ctx.Reponame)), map[string]string{ |
||||
"_csrf": csrf, |
||||
"action": "push-mirror-add", |
||||
"push_mirror_address": address, |
||||
"push_mirror_username": username, |
||||
"push_mirror_password": password, |
||||
"push_mirror_interval": "0", |
||||
}) |
||||
ctx.Session.MakeRequest(t, req, http.StatusFound) |
||||
|
||||
flashCookie := ctx.Session.GetCookie("macaron_flash") |
||||
assert.NotNil(t, flashCookie) |
||||
assert.Contains(t, flashCookie.Value, "success") |
||||
} |
||||
} |
||||
@ -0,0 +1,39 @@ |
||||
// Copyright 2021 The Gitea Authors. All rights reserved.
|
||||
// Use of this source code is governed by a MIT-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package migrations |
||||
|
||||
import ( |
||||
"fmt" |
||||
"time" |
||||
|
||||
"code.gitea.io/gitea/modules/timeutil" |
||||
|
||||
"xorm.io/xorm" |
||||
) |
||||
|
||||
func createPushMirrorTable(x *xorm.Engine) error { |
||||
type PushMirror struct { |
||||
ID int64 `xorm:"pk autoincr"` |
||||
RepoID int64 `xorm:"INDEX"` |
||||
RemoteName string |
||||
|
||||
Interval time.Duration |
||||
CreatedUnix timeutil.TimeStamp `xorm:"created"` |
||||
LastUpdateUnix timeutil.TimeStamp `xorm:"INDEX last_update"` |
||||
LastError string `xorm:"text"` |
||||
} |
||||
|
||||
sess := x.NewSession() |
||||
defer sess.Close() |
||||
if err := sess.Begin(); err != nil { |
||||
return err |
||||
} |
||||
|
||||
if err := sess.Sync2(new(PushMirror)); err != nil { |
||||
return fmt.Errorf("Sync2: %v", err) |
||||
} |
||||
|
||||
return sess.Commit() |
||||
} |
||||
@ -0,0 +1,106 @@ |
||||
// Copyright 2021 The Gitea Authors. All rights reserved.
|
||||
// Use of this source code is governed by a MIT-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package models |
||||
|
||||
import ( |
||||
"errors" |
||||
"time" |
||||
|
||||
"code.gitea.io/gitea/modules/log" |
||||
"code.gitea.io/gitea/modules/timeutil" |
||||
|
||||
"xorm.io/xorm" |
||||
) |
||||
|
||||
var ( |
||||
// ErrPushMirrorNotExist mirror does not exist error
|
||||
ErrPushMirrorNotExist = errors.New("PushMirror does not exist") |
||||
) |
||||
|
||||
// PushMirror represents mirror information of a repository.
|
||||
type PushMirror struct { |
||||
ID int64 `xorm:"pk autoincr"` |
||||
RepoID int64 `xorm:"INDEX"` |
||||
Repo *Repository `xorm:"-"` |
||||
RemoteName string |
||||
|
||||
Interval time.Duration |
||||
CreatedUnix timeutil.TimeStamp `xorm:"created"` |
||||
LastUpdateUnix timeutil.TimeStamp `xorm:"INDEX last_update"` |
||||
LastError string `xorm:"text"` |
||||
} |
||||
|
||||
// AfterLoad is invoked from XORM after setting the values of all fields of this object.
|
||||
func (m *PushMirror) AfterLoad(session *xorm.Session) { |
||||
if m == nil { |
||||
return |
||||
} |
||||
|
||||
var err error |
||||
m.Repo, err = getRepositoryByID(session, m.RepoID) |
||||
if err != nil { |
||||
log.Error("getRepositoryByID[%d]: %v", m.ID, err) |
||||
} |
||||
} |
||||
|
||||
// GetRepository returns the path of the repository.
|
||||
func (m *PushMirror) GetRepository() *Repository { |
||||
return m.Repo |
||||
} |
||||
|
||||
// GetRemoteName returns the name of the remote.
|
||||
func (m *PushMirror) GetRemoteName() string { |
||||
return m.RemoteName |
||||
} |
||||
|
||||
// InsertPushMirror inserts a push-mirror to database
|
||||
func InsertPushMirror(m *PushMirror) error { |
||||
_, err := x.Insert(m) |
||||
return err |
||||
} |
||||
|
||||
// UpdatePushMirror updates the push-mirror
|
||||
func UpdatePushMirror(m *PushMirror) error { |
||||
_, err := x.ID(m.ID).AllCols().Update(m) |
||||
return err |
||||
} |
||||
|
||||
// DeletePushMirrorByID deletes a push-mirrors by ID
|
||||
func DeletePushMirrorByID(ID int64) error { |
||||
_, err := x.ID(ID).Delete(&PushMirror{}) |
||||
return err |
||||
} |
||||
|
||||
// DeletePushMirrorsByRepoID deletes all push-mirrors by repoID
|
||||
func DeletePushMirrorsByRepoID(repoID int64) error { |
||||
_, err := x.Delete(&PushMirror{RepoID: repoID}) |
||||
return err |
||||
} |
||||
|
||||
// GetPushMirrorByID returns push-mirror information.
|
||||
func GetPushMirrorByID(ID int64) (*PushMirror, error) { |
||||
m := &PushMirror{} |
||||
has, err := x.ID(ID).Get(m) |
||||
if err != nil { |
||||
return nil, err |
||||
} else if !has { |
||||
return nil, ErrPushMirrorNotExist |
||||
} |
||||
return m, nil |
||||
} |
||||
|
||||
// GetPushMirrorsByRepoID returns push-mirror informations of a repository.
|
||||
func GetPushMirrorsByRepoID(repoID int64) ([]*PushMirror, error) { |
||||
mirrors := make([]*PushMirror, 0, 10) |
||||
return mirrors, x.Where("repo_id=?", repoID).Find(&mirrors) |
||||
} |
||||
|
||||
// PushMirrorsIterate iterates all push-mirror repositories.
|
||||
func PushMirrorsIterate(f func(idx int, bean interface{}) error) error { |
||||
return x. |
||||
Where("last_update + (`interval` / ?) <= ?", time.Second, time.Now().Unix()). |
||||
And("`interval` != 0"). |
||||
Iterate(new(PushMirror), f) |
||||
} |
||||
@ -0,0 +1,49 @@ |
||||
// Copyright 2021 The Gitea Authors. All rights reserved.
|
||||
// Use of this source code is governed by a MIT-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package models |
||||
|
||||
import ( |
||||
"testing" |
||||
"time" |
||||
|
||||
"code.gitea.io/gitea/modules/timeutil" |
||||
|
||||
"github.com/stretchr/testify/assert" |
||||
) |
||||
|
||||
func TestPushMirrorsIterate(t *testing.T) { |
||||
assert.NoError(t, PrepareTestDatabase()) |
||||
|
||||
now := timeutil.TimeStampNow() |
||||
|
||||
InsertPushMirror(&PushMirror{ |
||||
RemoteName: "test-1", |
||||
LastUpdateUnix: now, |
||||
Interval: 1, |
||||
}) |
||||
|
||||
long, _ := time.ParseDuration("24h") |
||||
InsertPushMirror(&PushMirror{ |
||||
RemoteName: "test-2", |
||||
LastUpdateUnix: now, |
||||
Interval: long, |
||||
}) |
||||
|
||||
InsertPushMirror(&PushMirror{ |
||||
RemoteName: "test-3", |
||||
LastUpdateUnix: now, |
||||
Interval: 0, |
||||
}) |
||||
|
||||
time.Sleep(1 * time.Millisecond) |
||||
|
||||
PushMirrorsIterate(func(idx int, bean interface{}) error { |
||||
m, ok := bean.(*PushMirror) |
||||
assert.True(t, ok) |
||||
assert.Equal(t, "test-1", m.RemoteName) |
||||
assert.Equal(t, m.RemoteName, m.GetRemoteName()) |
||||
return nil |
||||
}) |
||||
} |
||||
@ -0,0 +1,31 @@ |
||||
// Copyright 2021 The Gitea Authors. All rights reserved.
|
||||
// Use of this source code is governed by a MIT-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package git |
||||
|
||||
import "net/url" |
||||
|
||||
// GetRemoteAddress returns the url of a specific remote of the repository.
|
||||
func GetRemoteAddress(repoPath, remoteName string) (*url.URL, error) { |
||||
err := LoadGitVersion() |
||||
if err != nil { |
||||
return nil, err |
||||
} |
||||
var cmd *Command |
||||
if CheckGitVersionAtLeast("2.7") == nil { |
||||
cmd = NewCommand("remote", "get-url", remoteName) |
||||
} else { |
||||
cmd = NewCommand("config", "--get", "remote."+remoteName+".url") |
||||
} |
||||
|
||||
result, err := cmd.RunInDir(repoPath) |
||||
if err != nil { |
||||
return nil, err |
||||
} |
||||
|
||||
if len(result) > 0 { |
||||
result = result[:len(result)-1] |
||||
} |
||||
return url.Parse(result) |
||||
} |
||||
@ -1,25 +1,164 @@ |
||||
// Copyright 2020 The Gitea Authors. All rights reserved.
|
||||
// Copyright 2021 The Gitea Authors. All rights reserved.
|
||||
// Use of this source code is governed by a MIT-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package util |
||||
|
||||
import ( |
||||
"errors" |
||||
"testing" |
||||
|
||||
"github.com/stretchr/testify/assert" |
||||
) |
||||
|
||||
func TestSanitizeURLCredentials(t *testing.T) { |
||||
var kases = map[string]string{ |
||||
"https://github.com/go-gitea/test_repo.git": "https://github.com/go-gitea/test_repo.git", |
||||
"https://mytoken@github.com/go-gitea/test_repo.git": "https://github.com/go-gitea/test_repo.git", |
||||
"http://github.com/go-gitea/test_repo.git": "http://github.com/go-gitea/test_repo.git", |
||||
"/test/repos/repo1": "/test/repos/repo1", |
||||
"git@github.com:go-gitea/test_repo.git": "(unparsable url)", |
||||
func TestNewSanitizedError(t *testing.T) { |
||||
err := errors.New("error while secret on test") |
||||
err2 := NewSanitizedError(err) |
||||
assert.Equal(t, err.Error(), err2.Error()) |
||||
|
||||
var cases = []struct { |
||||
input error |
||||
oldnew []string |
||||
expected string |
||||
}{ |
||||
// case 0
|
||||
{ |
||||
errors.New("error while secret on test"), |
||||
[]string{"secret", "replaced"}, |
||||
"error while replaced on test", |
||||
}, |
||||
// case 1
|
||||
{ |
||||
errors.New("error while sec-ret on test"), |
||||
[]string{"secret", "replaced"}, |
||||
"error while sec-ret on test", |
||||
}, |
||||
} |
||||
|
||||
for source, value := range kases { |
||||
assert.EqualValues(t, value, SanitizeURLCredentials(source, false)) |
||||
for n, c := range cases { |
||||
err := NewSanitizedError(c.input, c.oldnew...) |
||||
|
||||
assert.Equal(t, c.expected, err.Error(), "case %d: error should match", n) |
||||
} |
||||
} |
||||
|
||||
func TestNewStringURLSanitizer(t *testing.T) { |
||||
var cases = []struct { |
||||
input string |
||||
placeholder bool |
||||
expected string |
||||
}{ |
||||
// case 0
|
||||
{ |
||||
"https://github.com/go-gitea/test_repo.git", |
||||
true, |
||||
"https://github.com/go-gitea/test_repo.git", |
||||
}, |
||||
// case 1
|
||||
{ |
||||
"https://github.com/go-gitea/test_repo.git", |
||||
false, |
||||
"https://github.com/go-gitea/test_repo.git", |
||||
}, |
||||
// case 2
|
||||
{ |
||||
"https://mytoken@github.com/go-gitea/test_repo.git", |
||||
true, |
||||
"https://" + userPlaceholder + "@github.com/go-gitea/test_repo.git", |
||||
}, |
||||
// case 3
|
||||
{ |
||||
"https://mytoken@github.com/go-gitea/test_repo.git", |
||||
false, |
||||
"https://github.com/go-gitea/test_repo.git", |
||||
}, |
||||
// case 4
|
||||
{ |
||||
"https://user:password@github.com/go-gitea/test_repo.git", |
||||
true, |
||||
"https://" + userPlaceholder + "@github.com/go-gitea/test_repo.git", |
||||
}, |
||||
// case 5
|
||||
{ |
||||
"https://user:password@github.com/go-gitea/test_repo.git", |
||||
false, |
||||
"https://github.com/go-gitea/test_repo.git", |
||||
}, |
||||
// case 6
|
||||
{ |
||||
"https://gi\nthub.com/go-gitea/test_repo.git", |
||||
false, |
||||
unparsableURL, |
||||
}, |
||||
} |
||||
|
||||
for n, c := range cases { |
||||
// uses NewURLSanitizer internally
|
||||
result := NewStringURLSanitizer(c.input, c.placeholder).Replace(c.input) |
||||
|
||||
assert.Equal(t, c.expected, result, "case %d: error should match", n) |
||||
} |
||||
} |
||||
|
||||
func TestNewStringURLSanitizedError(t *testing.T) { |
||||
var cases = []struct { |
||||
input string |
||||
placeholder bool |
||||
expected string |
||||
}{ |
||||
// case 0
|
||||
{ |
||||
"https://github.com/go-gitea/test_repo.git", |
||||
true, |
||||
"https://github.com/go-gitea/test_repo.git", |
||||
}, |
||||
// case 1
|
||||
{ |
||||
"https://github.com/go-gitea/test_repo.git", |
||||
false, |
||||
"https://github.com/go-gitea/test_repo.git", |
||||
}, |
||||
// case 2
|
||||
{ |
||||
"https://mytoken@github.com/go-gitea/test_repo.git", |
||||
true, |
||||
"https://" + userPlaceholder + "@github.com/go-gitea/test_repo.git", |
||||
}, |
||||
// case 3
|
||||
{ |
||||
"https://mytoken@github.com/go-gitea/test_repo.git", |
||||
false, |
||||
"https://github.com/go-gitea/test_repo.git", |
||||
}, |
||||
// case 4
|
||||
{ |
||||
"https://user:password@github.com/go-gitea/test_repo.git", |
||||
true, |
||||
"https://" + userPlaceholder + "@github.com/go-gitea/test_repo.git", |
||||
}, |
||||
// case 5
|
||||
{ |
||||
"https://user:password@github.com/go-gitea/test_repo.git", |
||||
false, |
||||
"https://github.com/go-gitea/test_repo.git", |
||||
}, |
||||
// case 6
|
||||
{ |
||||
"https://gi\nthub.com/go-gitea/test_repo.git", |
||||
false, |
||||
unparsableURL, |
||||
}, |
||||
} |
||||
|
||||
encloseText := func(input string) string { |
||||
return "test " + input + " test" |
||||
} |
||||
|
||||
for n, c := range cases { |
||||
err := errors.New(encloseText(c.input)) |
||||
|
||||
result := NewStringURLSanitizedError(err, c.input, c.placeholder) |
||||
|
||||
assert.Equal(t, encloseText(c.expected), result.Error(), "case %d: error should match", n) |
||||
} |
||||
} |
||||
|
||||
@ -0,0 +1,452 @@ |
||||
// Copyright 2021 The Gitea Authors. All rights reserved.
|
||||
// Use of this source code is governed by a MIT-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package mirror |
||||
|
||||
import ( |
||||
"context" |
||||
"fmt" |
||||
"strings" |
||||
"time" |
||||
|
||||
"code.gitea.io/gitea/models" |
||||
"code.gitea.io/gitea/modules/cache" |
||||
"code.gitea.io/gitea/modules/git" |
||||
"code.gitea.io/gitea/modules/lfs" |
||||
"code.gitea.io/gitea/modules/log" |
||||
"code.gitea.io/gitea/modules/notification" |
||||
repo_module "code.gitea.io/gitea/modules/repository" |
||||
"code.gitea.io/gitea/modules/setting" |
||||
"code.gitea.io/gitea/modules/timeutil" |
||||
"code.gitea.io/gitea/modules/util" |
||||
) |
||||
|
||||
// gitShortEmptySha Git short empty SHA
|
||||
const gitShortEmptySha = "0000000" |
||||
|
||||
// UpdateAddress writes new address to Git repository and database
|
||||
func UpdateAddress(m *models.Mirror, addr string) error { |
||||
remoteName := m.GetRemoteName() |
||||
repoPath := m.Repo.RepoPath() |
||||
// Remove old remote
|
||||
_, err := git.NewCommand("remote", "rm", remoteName).RunInDir(repoPath) |
||||
if err != nil && !strings.HasPrefix(err.Error(), "exit status 128 - fatal: No such remote ") { |
||||
return err |
||||
} |
||||
|
||||
_, err = git.NewCommand("remote", "add", remoteName, "--mirror=fetch", addr).RunInDir(repoPath) |
||||
if err != nil && !strings.HasPrefix(err.Error(), "exit status 128 - fatal: No such remote ") { |
||||
return err |
||||
} |
||||
|
||||
if m.Repo.HasWiki() { |
||||
wikiPath := m.Repo.WikiPath() |
||||
wikiRemotePath := repo_module.WikiRemoteURL(addr) |
||||
// Remove old remote of wiki
|
||||
_, err := git.NewCommand("remote", "rm", remoteName).RunInDir(wikiPath) |
||||
if err != nil && !strings.HasPrefix(err.Error(), "exit status 128 - fatal: No such remote ") { |
||||
return err |
||||
} |
||||
|
||||
_, err = git.NewCommand("remote", "add", remoteName, "--mirror=fetch", wikiRemotePath).RunInDir(wikiPath) |
||||
if err != nil && !strings.HasPrefix(err.Error(), "exit status 128 - fatal: No such remote ") { |
||||
return err |
||||
} |
||||
} |
||||
|
||||
m.Repo.OriginalURL = addr |
||||
return models.UpdateRepositoryCols(m.Repo, "original_url") |
||||
} |
||||
|
||||
// mirrorSyncResult contains information of a updated reference.
|
||||
// If the oldCommitID is "0000000", it means a new reference, the value of newCommitID is empty.
|
||||
// If the newCommitID is "0000000", it means the reference is deleted, the value of oldCommitID is empty.
|
||||
type mirrorSyncResult struct { |
||||
refName string |
||||
oldCommitID string |
||||
newCommitID string |
||||
} |
||||
|
||||
// parseRemoteUpdateOutput detects create, update and delete operations of references from upstream.
|
||||
func parseRemoteUpdateOutput(output string) []*mirrorSyncResult { |
||||
results := make([]*mirrorSyncResult, 0, 3) |
||||
lines := strings.Split(output, "\n") |
||||
for i := range lines { |
||||
// Make sure reference name is presented before continue
|
||||
idx := strings.Index(lines[i], "-> ") |
||||
if idx == -1 { |
||||
continue |
||||
} |
||||
|
||||
refName := lines[i][idx+3:] |
||||
|
||||
switch { |
||||
case strings.HasPrefix(lines[i], " * "): // New reference
|
||||
if strings.HasPrefix(lines[i], " * [new tag]") { |
||||
refName = git.TagPrefix + refName |
||||
} else if strings.HasPrefix(lines[i], " * [new branch]") { |
||||
refName = git.BranchPrefix + refName |
||||
} |
||||
results = append(results, &mirrorSyncResult{ |
||||
refName: refName, |
||||
oldCommitID: gitShortEmptySha, |
||||
}) |
||||
case strings.HasPrefix(lines[i], " - "): // Delete reference
|
||||
results = append(results, &mirrorSyncResult{ |
||||
refName: refName, |
||||
newCommitID: gitShortEmptySha, |
||||
}) |
||||
case strings.HasPrefix(lines[i], " + "): // Force update
|
||||
if idx := strings.Index(refName, " "); idx > -1 { |
||||
refName = refName[:idx] |
||||
} |
||||
delimIdx := strings.Index(lines[i][3:], " ") |
||||
if delimIdx == -1 { |
||||
log.Error("SHA delimiter not found: %q", lines[i]) |
||||
continue |
||||
} |
||||
shas := strings.Split(lines[i][3:delimIdx+3], "...") |
||||
if len(shas) != 2 { |
||||
log.Error("Expect two SHAs but not what found: %q", lines[i]) |
||||
continue |
||||
} |
||||
results = append(results, &mirrorSyncResult{ |
||||
refName: refName, |
||||
oldCommitID: shas[0], |
||||
newCommitID: shas[1], |
||||
}) |
||||
case strings.HasPrefix(lines[i], " "): // New commits of a reference
|
||||
delimIdx := strings.Index(lines[i][3:], " ") |
||||
if delimIdx == -1 { |
||||
log.Error("SHA delimiter not found: %q", lines[i]) |
||||
continue |
||||
} |
||||
shas := strings.Split(lines[i][3:delimIdx+3], "..") |
||||
if len(shas) != 2 { |
||||
log.Error("Expect two SHAs but not what found: %q", lines[i]) |
||||
continue |
||||
} |
||||
results = append(results, &mirrorSyncResult{ |
||||
refName: refName, |
||||
oldCommitID: shas[0], |
||||
newCommitID: shas[1], |
||||
}) |
||||
|
||||
default: |
||||
log.Warn("parseRemoteUpdateOutput: unexpected update line %q", lines[i]) |
||||
} |
||||
} |
||||
return results |
||||
} |
||||
|
||||
// runSync returns true if sync finished without error.
|
||||
func runSync(ctx context.Context, m *models.Mirror) ([]*mirrorSyncResult, bool) { |
||||
repoPath := m.Repo.RepoPath() |
||||
wikiPath := m.Repo.WikiPath() |
||||
timeout := time.Duration(setting.Git.Timeout.Mirror) * time.Second |
||||
|
||||
log.Trace("SyncMirrors [repo: %-v]: running git remote update...", m.Repo) |
||||
gitArgs := []string{"remote", "update"} |
||||
if m.EnablePrune { |
||||
gitArgs = append(gitArgs, "--prune") |
||||
} |
||||
gitArgs = append(gitArgs, m.GetRemoteName()) |
||||
|
||||
remoteAddr, remoteErr := git.GetRemoteAddress(repoPath, m.GetRemoteName()) |
||||
if remoteErr != nil { |
||||
log.Error("GetRemoteAddress Error %v", remoteErr) |
||||
} |
||||
|
||||
stdoutBuilder := strings.Builder{} |
||||
stderrBuilder := strings.Builder{} |
||||
if err := git.NewCommand(gitArgs...). |
||||
SetDescription(fmt.Sprintf("Mirror.runSync: %s", m.Repo.FullName())). |
||||
RunInDirTimeoutPipeline(timeout, repoPath, &stdoutBuilder, &stderrBuilder); err != nil { |
||||
stdout := stdoutBuilder.String() |
||||
stderr := stderrBuilder.String() |
||||
|
||||
// sanitize the output, since it may contain the remote address, which may
|
||||
// contain a password
|
||||
|
||||
sanitizer := util.NewURLSanitizer(remoteAddr, true) |
||||
stderrMessage := sanitizer.Replace(stderr) |
||||
stdoutMessage := sanitizer.Replace(stdout) |
||||
|
||||
log.Error("Failed to update mirror repository %v:\nStdout: %s\nStderr: %s\nErr: %v", m.Repo, stdoutMessage, stderrMessage, err) |
||||
desc := fmt.Sprintf("Failed to update mirror repository '%s': %s", repoPath, stderrMessage) |
||||
if err = models.CreateRepositoryNotice(desc); err != nil { |
||||
log.Error("CreateRepositoryNotice: %v", err) |
||||
} |
||||
return nil, false |
||||
} |
||||
output := stderrBuilder.String() |
||||
|
||||
gitRepo, err := git.OpenRepository(repoPath) |
||||
if err != nil { |
||||
log.Error("OpenRepository: %v", err) |
||||
return nil, false |
||||
} |
||||
|
||||
log.Trace("SyncMirrors [repo: %-v]: syncing releases with tags...", m.Repo) |
||||
if err = repo_module.SyncReleasesWithTags(m.Repo, gitRepo); err != nil { |
||||
log.Error("Failed to synchronize tags to releases for repository: %v", err) |
||||
} |
||||
|
||||
if m.LFS && setting.LFS.StartServer { |
||||
log.Trace("SyncMirrors [repo: %-v]: syncing LFS objects...", m.Repo) |
||||
ep := lfs.DetermineEndpoint(remoteAddr.String(), m.LFSEndpoint) |
||||
if err = repo_module.StoreMissingLfsObjectsInRepository(ctx, m.Repo, gitRepo, ep); err != nil { |
||||
log.Error("Failed to synchronize LFS objects for repository: %v", err) |
||||
} |
||||
} |
||||
gitRepo.Close() |
||||
|
||||
log.Trace("SyncMirrors [repo: %-v]: updating size of repository", m.Repo) |
||||
if err := m.Repo.UpdateSize(models.DefaultDBContext()); err != nil { |
||||
log.Error("Failed to update size for mirror repository: %v", err) |
||||
} |
||||
|
||||
if m.Repo.HasWiki() { |
||||
log.Trace("SyncMirrors [repo: %-v Wiki]: running git remote update...", m.Repo) |
||||
stderrBuilder.Reset() |
||||
stdoutBuilder.Reset() |
||||
if err := git.NewCommand("remote", "update", "--prune", m.GetRemoteName()). |
||||
SetDescription(fmt.Sprintf("Mirror.runSync Wiki: %s ", m.Repo.FullName())). |
||||
RunInDirTimeoutPipeline(timeout, wikiPath, &stdoutBuilder, &stderrBuilder); err != nil { |
||||
stdout := stdoutBuilder.String() |
||||
stderr := stderrBuilder.String() |
||||
|
||||
// sanitize the output, since it may contain the remote address, which may
|
||||
// contain a password
|
||||
|
||||
remoteAddr, remoteErr := git.GetRemoteAddress(wikiPath, m.GetRemoteName()) |
||||
if remoteErr != nil { |
||||
log.Error("GetRemoteAddress Error %v", remoteErr) |
||||
} |
||||
|
||||
sanitizer := util.NewURLSanitizer(remoteAddr, true) |
||||
stderrMessage := sanitizer.Replace(stderr) |
||||
stdoutMessage := sanitizer.Replace(stdout) |
||||
|
||||
log.Error("Failed to update mirror repository wiki %v:\nStdout: %s\nStderr: %s\nErr: %v", m.Repo, stdoutMessage, stderrMessage, err) |
||||
desc := fmt.Sprintf("Failed to update mirror repository wiki '%s': %s", wikiPath, stderrMessage) |
||||
if err = models.CreateRepositoryNotice(desc); err != nil { |
||||
log.Error("CreateRepositoryNotice: %v", err) |
||||
} |
||||
return nil, false |
||||
} |
||||
log.Trace("SyncMirrors [repo: %-v Wiki]: git remote update complete", m.Repo) |
||||
} |
||||
|
||||
log.Trace("SyncMirrors [repo: %-v]: invalidating mirror branch caches...", m.Repo) |
||||
branches, _, err := repo_module.GetBranches(m.Repo, 0, 0) |
||||
if err != nil { |
||||
log.Error("GetBranches: %v", err) |
||||
return nil, false |
||||
} |
||||
|
||||
for _, branch := range branches { |
||||
cache.Remove(m.Repo.GetCommitsCountCacheKey(branch.Name, true)) |
||||
} |
||||
|
||||
m.UpdatedUnix = timeutil.TimeStampNow() |
||||
return parseRemoteUpdateOutput(output), true |
||||
} |
||||
|
||||
// SyncPullMirror starts the sync of the pull mirror and schedules the next run.
|
||||
func SyncPullMirror(ctx context.Context, repoID int64) bool { |
||||
log.Trace("SyncMirrors [repo_id: %v]", repoID) |
||||
defer func() { |
||||
err := recover() |
||||
if err == nil { |
||||
return |
||||
} |
||||
// There was a panic whilst syncMirrors...
|
||||
log.Error("PANIC whilst syncMirrors[%d] Panic: %v\nStacktrace: %s", repoID, err, log.Stack(2)) |
||||
}() |
||||
|
||||
m, err := models.GetMirrorByRepoID(repoID) |
||||
if err != nil { |
||||
log.Error("GetMirrorByRepoID [%d]: %v", repoID, err) |
||||
return false |
||||
} |
||||
|
||||
log.Trace("SyncMirrors [repo: %-v]: Running Sync", m.Repo) |
||||
results, ok := runSync(ctx, m) |
||||
if !ok { |
||||
return false |
||||
} |
||||
|
||||
log.Trace("SyncMirrors [repo: %-v]: Scheduling next update", m.Repo) |
||||
m.ScheduleNextUpdate() |
||||
if err = models.UpdateMirror(m); err != nil { |
||||
log.Error("UpdateMirror [%d]: %v", m.RepoID, err) |
||||
return false |
||||
} |
||||
|
||||
var gitRepo *git.Repository |
||||
if len(results) == 0 { |
||||
log.Trace("SyncMirrors [repo: %-v]: no branches updated", m.Repo) |
||||
} else { |
||||
log.Trace("SyncMirrors [repo: %-v]: %d branches updated", m.Repo, len(results)) |
||||
gitRepo, err = git.OpenRepository(m.Repo.RepoPath()) |
||||
if err != nil { |
||||
log.Error("OpenRepository [%d]: %v", m.RepoID, err) |
||||
return false |
||||
} |
||||
defer gitRepo.Close() |
||||
|
||||
if ok := checkAndUpdateEmptyRepository(m, gitRepo, results); !ok { |
||||
return false |
||||
} |
||||
} |
||||
|
||||
for _, result := range results { |
||||
// Discard GitHub pull requests, i.e. refs/pull/*
|
||||
if strings.HasPrefix(result.refName, "refs/pull/") { |
||||
continue |
||||
} |
||||
|
||||
tp, _ := git.SplitRefName(result.refName) |
||||
|
||||
// Create reference
|
||||
if result.oldCommitID == gitShortEmptySha { |
||||
if tp == git.TagPrefix { |
||||
tp = "tag" |
||||
} else if tp == git.BranchPrefix { |
||||
tp = "branch" |
||||
} |
||||
commitID, err := gitRepo.GetRefCommitID(result.refName) |
||||
if err != nil { |
||||
log.Error("gitRepo.GetRefCommitID [repo_id: %d, ref_name: %s]: %v", m.RepoID, result.refName, err) |
||||
continue |
||||
} |
||||
notification.NotifySyncPushCommits(m.Repo.MustOwner(), m.Repo, &repo_module.PushUpdateOptions{ |
||||
RefFullName: result.refName, |
||||
OldCommitID: git.EmptySHA, |
||||
NewCommitID: commitID, |
||||
}, repo_module.NewPushCommits()) |
||||
notification.NotifySyncCreateRef(m.Repo.MustOwner(), m.Repo, tp, result.refName) |
||||
continue |
||||
} |
||||
|
||||
// Delete reference
|
||||
if result.newCommitID == gitShortEmptySha { |
||||
notification.NotifySyncDeleteRef(m.Repo.MustOwner(), m.Repo, tp, result.refName) |
||||
continue |
||||
} |
||||
|
||||
// Push commits
|
||||
oldCommitID, err := git.GetFullCommitID(gitRepo.Path, result.oldCommitID) |
||||
if err != nil { |
||||
log.Error("GetFullCommitID [%d]: %v", m.RepoID, err) |
||||
continue |
||||
} |
||||
newCommitID, err := git.GetFullCommitID(gitRepo.Path, result.newCommitID) |
||||
if err != nil { |
||||
log.Error("GetFullCommitID [%d]: %v", m.RepoID, err) |
||||
continue |
||||
} |
||||
commits, err := gitRepo.CommitsBetweenIDs(newCommitID, oldCommitID) |
||||
if err != nil { |
||||
log.Error("CommitsBetweenIDs [repo_id: %d, new_commit_id: %s, old_commit_id: %s]: %v", m.RepoID, newCommitID, oldCommitID, err) |
||||
continue |
||||
} |
||||
|
||||
theCommits := repo_module.ListToPushCommits(commits) |
||||
if len(theCommits.Commits) > setting.UI.FeedMaxCommitNum { |
||||
theCommits.Commits = theCommits.Commits[:setting.UI.FeedMaxCommitNum] |
||||
} |
||||
|
||||
theCommits.CompareURL = m.Repo.ComposeCompareURL(oldCommitID, newCommitID) |
||||
|
||||
notification.NotifySyncPushCommits(m.Repo.MustOwner(), m.Repo, &repo_module.PushUpdateOptions{ |
||||
RefFullName: result.refName, |
||||
OldCommitID: oldCommitID, |
||||
NewCommitID: newCommitID, |
||||
}, theCommits) |
||||
} |
||||
log.Trace("SyncMirrors [repo: %-v]: done notifying updated branches/tags - now updating last commit time", m.Repo) |
||||
|
||||
// Get latest commit date and update to current repository updated time
|
||||
commitDate, err := git.GetLatestCommitTime(m.Repo.RepoPath()) |
||||
if err != nil { |
||||
log.Error("GetLatestCommitDate [%d]: %v", m.RepoID, err) |
||||
return false |
||||
} |
||||
|
||||
if err = models.UpdateRepositoryUpdatedTime(m.RepoID, commitDate); err != nil { |
||||
log.Error("Update repository 'updated_unix' [%d]: %v", m.RepoID, err) |
||||
return false |
||||
} |
||||
|
||||
log.Trace("SyncMirrors [repo: %-v]: Successfully updated", m.Repo) |
||||
|
||||
return true |
||||
} |
||||
|
||||
func checkAndUpdateEmptyRepository(m *models.Mirror, gitRepo *git.Repository, results []*mirrorSyncResult) bool { |
||||
if !m.Repo.IsEmpty { |
||||
return true |
||||
} |
||||
|
||||
hasDefault := false |
||||
hasMaster := false |
||||
hasMain := false |
||||
defaultBranchName := m.Repo.DefaultBranch |
||||
if len(defaultBranchName) == 0 { |
||||
defaultBranchName = setting.Repository.DefaultBranch |
||||
} |
||||
firstName := "" |
||||
for _, result := range results { |
||||
if strings.HasPrefix(result.refName, "refs/pull/") { |
||||
continue |
||||
} |
||||
tp, name := git.SplitRefName(result.refName) |
||||
if len(tp) > 0 && tp != git.BranchPrefix { |
||||
continue |
||||
} |
||||
if len(firstName) == 0 { |
||||
firstName = name |
||||
} |
||||
|
||||
hasDefault = hasDefault || name == defaultBranchName |
||||
hasMaster = hasMaster || name == "master" |
||||
hasMain = hasMain || name == "main" |
||||
} |
||||
|
||||
if len(firstName) > 0 { |
||||
if hasDefault { |
||||
m.Repo.DefaultBranch = defaultBranchName |
||||
} else if hasMaster { |
||||
m.Repo.DefaultBranch = "master" |
||||
} else if hasMain { |
||||
m.Repo.DefaultBranch = "main" |
||||
} else { |
||||
m.Repo.DefaultBranch = firstName |
||||
} |
||||
// Update the git repository default branch
|
||||
if err := gitRepo.SetDefaultBranch(m.Repo.DefaultBranch); err != nil { |
||||
if !git.IsErrUnsupportedVersion(err) { |
||||
log.Error("Failed to update default branch of underlying git repository %-v. Error: %v", m.Repo, err) |
||||
desc := fmt.Sprintf("Failed to uupdate default branch of underlying git repository '%s': %v", m.Repo.RepoPath(), err) |
||||
if err = models.CreateRepositoryNotice(desc); err != nil { |
||||
log.Error("CreateRepositoryNotice: %v", err) |
||||
} |
||||
return false |
||||
} |
||||
} |
||||
m.Repo.IsEmpty = false |
||||
// Update the is empty and default_branch columns
|
||||
if err := models.UpdateRepositoryCols(m.Repo, "default_branch", "is_empty"); err != nil { |
||||
log.Error("Failed to update default branch of repository %-v. Error: %v", m.Repo, err) |
||||
desc := fmt.Sprintf("Failed to uupdate default branch of repository '%s': %v", m.Repo.RepoPath(), err) |
||||
if err = models.CreateRepositoryNotice(desc); err != nil { |
||||
log.Error("CreateRepositoryNotice: %v", err) |
||||
} |
||||
return false |
||||
} |
||||
} |
||||
return true |
||||
} |
||||
@ -0,0 +1,242 @@ |
||||
// Copyright 2021 The Gitea Authors. All rights reserved.
|
||||
// Use of this source code is governed by a MIT-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package mirror |
||||
|
||||
import ( |
||||
"context" |
||||
"errors" |
||||
"io" |
||||
"net/url" |
||||
"regexp" |
||||
"time" |
||||
|
||||
"code.gitea.io/gitea/models" |
||||
"code.gitea.io/gitea/modules/git" |
||||
"code.gitea.io/gitea/modules/lfs" |
||||
"code.gitea.io/gitea/modules/log" |
||||
"code.gitea.io/gitea/modules/repository" |
||||
"code.gitea.io/gitea/modules/setting" |
||||
"code.gitea.io/gitea/modules/timeutil" |
||||
"code.gitea.io/gitea/modules/util" |
||||
) |
||||
|
||||
var stripExitStatus = regexp.MustCompile(`exit status \d+ - `) |
||||
|
||||
// AddPushMirrorRemote registers the push mirror remote.
|
||||
func AddPushMirrorRemote(m *models.PushMirror, addr string) error { |
||||
addRemoteAndConfig := func(addr, path string) error { |
||||
if _, err := git.NewCommand("remote", "add", "--mirror=push", m.RemoteName, addr).RunInDir(path); err != nil { |
||||
return err |
||||
} |
||||
if _, err := git.NewCommand("config", "--add", "remote."+m.RemoteName+".push", "+refs/heads/*:refs/heads/*").RunInDir(path); err != nil { |
||||
return err |
||||
} |
||||
if _, err := git.NewCommand("config", "--add", "remote."+m.RemoteName+".push", "+refs/tags/*:refs/tags/*").RunInDir(path); err != nil { |
||||
return err |
||||
} |
||||
return nil |
||||
} |
||||
|
||||
if err := addRemoteAndConfig(addr, m.Repo.RepoPath()); err != nil { |
||||
return err |
||||
} |
||||
|
||||
if m.Repo.HasWiki() { |
||||
wikiRemoteURL := repository.WikiRemoteURL(addr) |
||||
if len(wikiRemoteURL) > 0 { |
||||
if err := addRemoteAndConfig(wikiRemoteURL, m.Repo.WikiPath()); err != nil { |
||||
return err |
||||
} |
||||
} |
||||
} |
||||
|
||||
return nil |
||||
} |
||||
|
||||
// RemovePushMirrorRemote removes the push mirror remote.
|
||||
func RemovePushMirrorRemote(m *models.PushMirror) error { |
||||
cmd := git.NewCommand("remote", "rm", m.RemoteName) |
||||
|
||||
if _, err := cmd.RunInDir(m.Repo.RepoPath()); err != nil { |
||||
return err |
||||
} |
||||
|
||||
if m.Repo.HasWiki() { |
||||
if _, err := cmd.RunInDir(m.Repo.WikiPath()); err != nil { |
||||
// The wiki remote may not exist
|
||||
log.Warn("Wiki Remote[%d] could not be removed: %v", m.ID, err) |
||||
} |
||||
} |
||||
|
||||
return nil |
||||
} |
||||
|
||||
// SyncPushMirror starts the sync of the push mirror and schedules the next run.
|
||||
func SyncPushMirror(ctx context.Context, mirrorID int64) bool { |
||||
log.Trace("SyncPushMirror [mirror: %d]", mirrorID) |
||||
defer func() { |
||||
err := recover() |
||||
if err == nil { |
||||
return |
||||
} |
||||
// There was a panic whilst syncPushMirror...
|
||||
log.Error("PANIC whilst syncPushMirror[%d] Panic: %v\nStacktrace: %s", mirrorID, err, log.Stack(2)) |
||||
}() |
||||
|
||||
m, err := models.GetPushMirrorByID(mirrorID) |
||||
if err != nil { |
||||
log.Error("GetPushMirrorByID [%d]: %v", mirrorID, err) |
||||
return false |
||||
} |
||||
|
||||
m.LastError = "" |
||||
|
||||
log.Trace("SyncPushMirror [mirror: %d][repo: %-v]: Running Sync", m.ID, m.Repo) |
||||
err = runPushSync(ctx, m) |
||||
if err != nil { |
||||
log.Error("SyncPushMirror [mirror: %d][repo: %-v]: %v", m.ID, m.Repo, err) |
||||
m.LastError = stripExitStatus.ReplaceAllLiteralString(err.Error(), "") |
||||
} |
||||
|
||||
m.LastUpdateUnix = timeutil.TimeStampNow() |
||||
|
||||
if err := models.UpdatePushMirror(m); err != nil { |
||||
log.Error("UpdatePushMirror [%d]: %v", m.ID, err) |
||||
|
||||
return false |
||||
} |
||||
|
||||
log.Trace("SyncPushMirror [mirror: %d][repo: %-v]: Finished", m.ID, m.Repo) |
||||
|
||||
return err == nil |
||||
} |
||||
|
||||
func runPushSync(ctx context.Context, m *models.PushMirror) error { |
||||
timeout := time.Duration(setting.Git.Timeout.Mirror) * time.Second |
||||
|
||||
performPush := func(path string) error { |
||||
remoteAddr, err := git.GetRemoteAddress(path, m.RemoteName) |
||||
if err != nil { |
||||
log.Error("GetRemoteAddress(%s) Error %v", path, err) |
||||
return errors.New("Unexpected error") |
||||
} |
||||
|
||||
if setting.LFS.StartServer { |
||||
log.Trace("SyncMirrors [repo: %-v]: syncing LFS objects...", m.Repo) |
||||
|
||||
gitRepo, err := git.OpenRepository(path) |
||||
if err != nil { |
||||
log.Error("OpenRepository: %v", err) |
||||
return errors.New("Unexpected error") |
||||
} |
||||
defer gitRepo.Close() |
||||
|
||||
ep := lfs.DetermineEndpoint(remoteAddr.String(), "") |
||||
if err := pushAllLFSObjects(ctx, gitRepo, ep); err != nil { |
||||
return util.NewURLSanitizedError(err, remoteAddr, true) |
||||
} |
||||
} |
||||
|
||||
log.Trace("Pushing %s mirror[%d] remote %s", path, m.ID, m.RemoteName) |
||||
|
||||
if err := git.Push(path, git.PushOptions{ |
||||
Remote: m.RemoteName, |
||||
Force: true, |
||||
Mirror: true, |
||||
Timeout: timeout, |
||||
}); err != nil { |
||||
log.Error("Error pushing %s mirror[%d] remote %s: %v", path, m.ID, m.RemoteName, err) |
||||
|
||||
return util.NewURLSanitizedError(err, remoteAddr, true) |
||||
} |
||||
|
||||
return nil |
||||
} |
||||
|
||||
err := performPush(m.Repo.RepoPath()) |
||||
if err != nil { |
||||
return err |
||||
} |
||||
|
||||
if m.Repo.HasWiki() { |
||||
wikiPath := m.Repo.WikiPath() |
||||
_, err := git.GetRemoteAddress(wikiPath, m.RemoteName) |
||||
if err == nil { |
||||
err := performPush(wikiPath) |
||||
if err != nil { |
||||
return err |
||||
} |
||||
} else { |
||||
log.Trace("Skipping wiki: No remote configured") |
||||
} |
||||
} |
||||
|
||||
return nil |
||||
} |
||||
|
||||
func pushAllLFSObjects(ctx context.Context, gitRepo *git.Repository, endpoint *url.URL) error { |
||||
client := lfs.NewClient(endpoint) |
||||
contentStore := lfs.NewContentStore() |
||||
|
||||
pointerChan := make(chan lfs.PointerBlob) |
||||
errChan := make(chan error, 1) |
||||
go lfs.SearchPointerBlobs(ctx, gitRepo, pointerChan, errChan) |
||||
|
||||
uploadObjects := func(pointers []lfs.Pointer) error { |
||||
err := client.Upload(ctx, pointers, func(p lfs.Pointer, objectError error) (io.ReadCloser, error) { |
||||
if objectError != nil { |
||||
return nil, objectError |
||||
} |
||||
|
||||
content, err := contentStore.Get(p) |
||||
if err != nil { |
||||
log.Error("Error reading LFS object %v: %v", p, err) |
||||
} |
||||
return content, err |
||||
}) |
||||
if err != nil { |
||||
select { |
||||
case <-ctx.Done(): |
||||
return nil |
||||
default: |
||||
} |
||||
} |
||||
return err |
||||
} |
||||
|
||||
var batch []lfs.Pointer |
||||
for pointerBlob := range pointerChan { |
||||
exists, err := contentStore.Exists(pointerBlob.Pointer) |
||||
if err != nil { |
||||
log.Error("Error checking if LFS object %v exists: %v", pointerBlob.Pointer, err) |
||||
return err |
||||
} |
||||
if !exists { |
||||
log.Trace("Skipping missing LFS object %v", pointerBlob.Pointer) |
||||
continue |
||||
} |
||||
|
||||
batch = append(batch, pointerBlob.Pointer) |
||||
if len(batch) >= client.BatchSize() { |
||||
if err := uploadObjects(batch); err != nil { |
||||
return err |
||||
} |
||||
batch = nil |
||||
} |
||||
} |
||||
if len(batch) > 0 { |
||||
if err := uploadObjects(batch); err != nil { |
||||
return err |
||||
} |
||||
} |
||||
|
||||
err, has := <-errChan |
||||
if has { |
||||
log.Error("Error enumerating LFS objects for repository: %v", err) |
||||
return err |
||||
} |
||||
|
||||
return nil |
||||
} |
||||
Loading…
Reference in new issue