domi/forgejo-domi
1
0
Fork 0
forked from forgejo/forgejo
Code Pull requests Activity

[F3] Forgejo driver and CLI

Browse source 
user, topic, project, label, milestone, repository, pull_request,
release, asset, comment, reaction, review providers

Signed-off-by: Earl Warren <contact@earl-warren.org>

Preserve file size when creating attachments

Introduced in c6f5029708

repoList.LoadAttributes has a ctx argument now

Rename `repo.GetOwner` to `repo.LoadOwner`

bd66fa586a

upgrade to the latest gof3

(cherry picked from commit c770713656)

[F3] ID remapping logic is in place, remove workaround

(cherry picked from commit d0fee30167)

[F3] it is experimental, do not enable by default

(cherry picked from commit de325b21d0)
(cherry picked from commit 547e7b3c40)
(cherry picked from commit 820df3a56b)
(cherry picked from commit eaba87689b)
(cherry picked from commit 1b86896b3b)
(cherry picked from commit 0046aac1c6)
(cherry picked from commit f14220df8f)
(cherry picked from commit 559b731001)
(cherry picked from commit 801f7d600d)
(cherry picked from commit 6aa76e9bcf)
(cherry picked from commit a8757dcb07)

[F3] promote F3 users to matching OAuth2 users on first sign-in

(cherry picked from commit bd7fef7496)
(cherry picked from commit 07412698e8)
(cherry picked from commit d143e5b2a3)

[F3] upgrade to gof3 50a6e740ac04

Add new methods GetIDString() & SetIDString() & ToFormatInterface()
Change the prototype of the fixture function

(cherry picked from commit d7b263ff8b)
(cherry picked from commit b3eaf2249d)
(cherry picked from commit d492ddd9bb)

[F3] add GetLocalMatchingRemote with a default implementation

(cherry picked from commit 0a22015039)
(cherry picked from commit f1310c38fb)
(cherry picked from commit deb68552f2)

[F3] GetLocalMatchingRemote for user

(cherry picked from commit e73cb837f5)
(cherry picked from commit a24bc0b85e)
(cherry picked from commit 846a522ecc)

[F3] GetAdminUser now has a ctx argument

(cherry picked from commit 37357a92af)
(cherry picked from commit 660bc1673c)
(cherry picked from commit 72d692a767)

[F3] introduce UserTypeF3

To avoid conflicts should UserTypeRemoteUser be used differently by Gitea

(cherry picked from commit 6de2701bb3)

[F3] user.Put: idempotency

(cherry picked from commit 821e38573c)
(cherry picked from commit f7638f5414)

[F3] upgrade to urfave v2

(cherry picked from commit cc3dbdfd1d)

[F3] update gof3

(cherry picked from commit 2eee960751)

[F3] move f3 under forgejo-cli

* simplify the tests by re-using the forgejo-cli helpers to capture
  the output
* unify CmdF3 to be structured in the same way CmdActions is

(cherry picked from commit 4c9fe58b74)

[F3] replace f3 with forgejo-cli f3

(cherry picked from commit 7ba7ceef1b)

[F3] s/ListOptions/Paginator/

[F3] user: add unit tests

[F3] user comparison of F3 managed users is on content

[F3] issue: add unit tests

[F3] gof3 now has one more argument to Put()

[F3] re-use gof3 unit tests for the driver

(cherry picked from commit af7ee6200c)

Conflicts:
	tests/integration/integration_test.go
	because of some code removed in forgejo-development, trivial
	context conflict resolution

[F3] more idempotent tests (#1275)

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/1275
Co-authored-by: Loïc Dachary <loic@dachary.org>
Co-committed-by: Loïc Dachary <loic@dachary.org>

[F3] tests: do SQL update if nothing changes

[F3] tests comment idempotence

[F3] tests milestone idempotence

[F3] tests pull_request idempotence

[F3] tests release idempotence

[F3] tests asset idempotence

[F3] tests project idempotence

[F3] tests review idempotence

(cherry picked from commit 91038bb4e8)
(cherry picked from commit a7d2a65214)
(cherry picked from commit 59a17e5a34)

[F3] sub command of forgejo-cli

(cherry picked from commit 4d098e9b83)

[F3] implement --quiet, --debug, --verbose

(cherry picked from commit 82e2e17b45)

[F3] fix off by one error when importing repositories

(cherry picked from commit 31689b1397)

[F3] upgrade gof3

(cherry picked from commit 87b8cfe5a1)

[F3] set the logger for all drivers

The logger is set for the local Forgejo driver only. Even when --debug
is specified, the other drivers do not display debug
information. Use the gof3 context to set the logger for all of them at
once.

(cherry picked from commit 8aa7de8ba0)

[F3] the closed date of an issue may be nil

(cherry picked from commit 93d3eaf0b5)

[F3] update gof3 to support system users

there now is a workaround to hardcode system users when they are not
supported by the API

(cherry picked from commit 915484daa7)
This commit is contained in:
Earl Warren 2022-09-06 14:35:43 +10:00
parent d63c5b18a7
commit b47ac73b8a
No known key found for this signature in database
GPG key ID: 0579CB2928A78A00
41 changed files with 3960 additions and 52 deletions
Download patch file Download diff file Expand all files Collapse all files

325
tests/integration/f3_test.go Normal file
View file

@ -0,0 +1,325 @@
// SPDX-License-Identifier: MIT
package integration
import (
"context"
"fmt"
"net/http"
"net/url"
"testing"
auth_model "code.gitea.io/gitea/models/auth"
"code.gitea.io/gitea/models/unittest"
user_model "code.gitea.io/gitea/models/user"
"code.gitea.io/gitea/modules/log"
"code.gitea.io/gitea/modules/setting"
"code.gitea.io/gitea/modules/test"
"code.gitea.io/gitea/services/f3/util"
"code.gitea.io/gitea/services/migrations"
"code.gitea.io/gitea/tests"
"github.com/markbates/goth"
"github.com/stretchr/testify/assert"
f3_types "lab.forgefriends.org/friendlyforgeformat/gof3/config/types"
f3_forges "lab.forgefriends.org/friendlyforgeformat/gof3/forges"
f3_common "lab.forgefriends.org/friendlyforgeformat/gof3/forges/common"
f3_f3 "lab.forgefriends.org/friendlyforgeformat/gof3/forges/f3"
f3_forgejo "lab.forgefriends.org/friendlyforgeformat/gof3/forges/forgejo"
f3_tests "lab.forgefriends.org/friendlyforgeformat/gof3/forges/tests"
"lab.forgefriends.org/friendlyforgeformat/gof3/format"
f3_util "lab.forgefriends.org/friendlyforgeformat/gof3/util"
)
func TestF3_MirrorAPITOLocal(t *testing.T) {
onGiteaRun(t, func(t *testing.T, u *url.URL) {
AllowLocalNetworks := setting.Migrations.AllowLocalNetworks
setting.F3.Enabled = true
setting.Migrations.AllowLocalNetworks = true
AppVer := setting.AppVer
// Gitea SDK (go-sdk) need to parse the AppVer from server response, so we must set it to a valid version string.
setting.AppVer = "1.16.0"
defer func() {
setting.Migrations.AllowLocalNetworks = AllowLocalNetworks
setting.AppVer = AppVer
migrations.Init()
}()
assert.NoError(t, migrations.Init())
//
// Step 1: create a fixture
//
fixtureNewF3Forge := func(t f3_tests.TestingT, logger *f3_types.Logger, user *format.User, tmpDir string) *f3_forges.ForgeRoot {
root := f3_forges.NewForgeRoot(&f3_f3.F3{}, &f3_f3.Options{
Options: f3_types.Options{
Configuration: f3_types.Configuration{
Directory: tmpDir,
},
Features: f3_types.AllFeatures,
Logger: util.ToF3Logger(nil),
},
Remap: true,
})
return root
}
fixture := f3_forges.NewFixture(t, f3_forges.FixtureForgeFactory{Fun: fixtureNewF3Forge, AdminRequired: false})
fixture.NewUser(5432)
fixture.NewMilestone()
fixture.NewLabel()
fixture.NewIssue()
fixture.NewTopic()
fixture.NewRepository()
fixture.NewPullRequest()
fixture.NewRelease()
fixture.NewAsset()
fixture.NewIssueComment(nil)
fixture.NewPullRequestComment()
// fixture.NewReview()
fixture.NewIssueReaction()
fixture.NewCommentReaction()
//
// Step 2: mirror F3 into Forgejo
//
doer, err := user_model.GetAdminUser(context.Background())
assert.NoError(t, err)
forgejoLocalUpload := util.ForgejoForgeRoot(f3_types.AllFeatures, doer, 0)
upload := forgejoLocalUpload.Forge
options := f3_common.NewMirrorOptionsRecurse()
upload.Mirror(context.Background(), fixture.Forge, options)
//
// Step 3: mirror Forgejo into F3
//
logger := util.ToF3Logger(nil)
f3 := f3_forges.FixtureNewF3Forge(t, logger, nil, t.TempDir())
forgejoLocalDownload := util.ForgejoForgeRoot(f3_types.AllFeatures, doer, 0)
download := forgejoLocalDownload.Forge
downloadUser := download.Users.GetFromFormat(context.Background(), &format.User{UserName: fixture.UserFormat.UserName})
downloadProject := downloadUser.Projects.GetFromFormat(context.Background(), &format.Project{Name: fixture.ProjectFormat.Name})
options = f3_common.NewMirrorOptionsRecurse(downloadUser, downloadProject)
f3.Forge.Mirror(context.Background(), download, options)
//
// Step 4: verify the fixture and F3 are equivalent
//
files := f3_util.Command(context.Background(), "find", f3.GetDirectory())
assert.Contains(t, files, "/repository/git/hooks")
assert.Contains(t, files, "/label/")
assert.Contains(t, files, "/issue/")
assert.Contains(t, files, "/milestone/")
assert.Contains(t, files, "/topic/")
assert.Contains(t, files, "/pull_request/")
assert.Contains(t, files, "/release/")
assert.Contains(t, files, "/asset/")
assert.Contains(t, files, "/comment/")
// assert.Contains(t, files, "/review/")
assert.Contains(t, files, "/reaction/")
// f3_util.Command(context.Background(), "cp", "-a", f3.GetDirectory(), "abc")
})
}
func TestF3_MaybePromoteUser(t *testing.T) {
defer tests.PrepareTestEnv(t)()
//
// OAuth2 authentication source GitLab
//
gitlabName := "gitlab"
_ = addAuthSource(t, authSourcePayloadGitLabCustom(gitlabName))
//
// F3 authentication source matching the GitLab authentication source
//
f3Name := "f3"
f3 := createF3AuthSource(t, f3Name, "http://mygitlab.eu", gitlabName)
//
// Create a user as if it had been previously been created by the F3
// authentication source.
//
gitlabUserID := "5678"
gitlabEmail := "gitlabuser@example.com"
userBeforeSignIn := &user_model.User{
Name: "gitlabuser",
Type: user_model.UserTypeF3,
LoginType: auth_model.F3,
LoginSource: f3.ID,
LoginName: gitlabUserID,
}
defer createUser(context.Background(), t, userBeforeSignIn)()
//
// A request for user information sent to Goth will return a
// goth.User exactly matching the user created above.
//
defer mockCompleteUserAuth(func(res http.ResponseWriter, req *http.Request) (goth.User, error) {
return goth.User{
Provider: gitlabName,
UserID: gitlabUserID,
Email: gitlabEmail,
}, nil
})()
req := NewRequest(t, "GET", fmt.Sprintf("/user/oauth2/%s/callback?code=XYZ&state=XYZ", gitlabName))
resp := MakeRequest(t, req, http.StatusSeeOther)
assert.Equal(t, "/", test.RedirectURL(resp))
userAfterSignIn := unittest.AssertExistsAndLoadBean(t, &user_model.User{ID: userBeforeSignIn.ID})
// both are about the same user
assert.Equal(t, userAfterSignIn.ID, userBeforeSignIn.ID)
// the login time was updated, proof the login succeeded
assert.Greater(t, userAfterSignIn.LastLoginUnix, userBeforeSignIn.LastLoginUnix)
// the login type was promoted from F3 to OAuth2
assert.Equal(t, userBeforeSignIn.LoginType, auth_model.F3)
assert.Equal(t, userAfterSignIn.LoginType, auth_model.OAuth2)
// the OAuth2 email was used to set the missing user email
assert.Equal(t, userBeforeSignIn.Email, "")
assert.Equal(t, userAfterSignIn.Email, gitlabEmail)
}
func TestF3_UserMappingExisting(t *testing.T) {
onGiteaRun(t, func(t *testing.T, u *url.URL) {
AllowLocalNetworks := setting.Migrations.AllowLocalNetworks
setting.F3.Enabled = true
setting.Migrations.AllowLocalNetworks = true
AppVer := setting.AppVer
// Gitea SDK (go-sdk) need to parse the AppVer from server response, so we must set it to a valid version string.
setting.AppVer = "1.16.0"
defer func() {
setting.Migrations.AllowLocalNetworks = AllowLocalNetworks
setting.AppVer = AppVer
}()
log.Debug("Step 1: create a fixture in F3")
fixtureNewF3Forge := func(t f3_tests.TestingT, logger *f3_types.Logger, user *format.User, tmpDir string) *f3_forges.ForgeRoot {
root := f3_forges.NewForgeRoot(&f3_f3.F3{}, &f3_f3.Options{
Options: f3_types.Options{
Configuration: f3_types.Configuration{
Directory: tmpDir,
},
Features: f3_types.AllFeatures,
Logger: util.ToF3Logger(nil),
},
Remap: true,
})
return root
}
fixture := f3_forges.NewFixture(t, f3_forges.FixtureForgeFactory{Fun: fixtureNewF3Forge, AdminRequired: false})
userID := int64(5432)
fixture.NewUser(userID)
// fixture.NewProject()
log.Debug("Step 2: mirror F3 into Forgejo")
//
// OAuth2 authentication source GitLab
//
gitlabName := "gitlab"
gitlab := addAuthSource(t, authSourcePayloadGitLabCustom(gitlabName))
//
// Create a user as if it had been previously been created by the F3
// authentication source.
//
gitlabUserID := fmt.Sprintf("%d", userID)
gitlabUser := &user_model.User{
Name: "gitlabuser",
Email: "gitlabuser@example.com",
LoginType: auth_model.OAuth2,
LoginSource: gitlab.ID,
LoginName: gitlabUserID,
}
defer createUser(context.Background(), t, gitlabUser)()
doer, err := user_model.GetAdminUser(context.Background())
assert.NoError(t, err)
forgejoLocal := util.ForgejoForgeRoot(f3_types.AllFeatures, doer, gitlab.ID)
options := f3_common.NewMirrorOptionsRecurse()
forgejoLocal.Forge.Mirror(context.Background(), fixture.Forge, options)
log.Debug("Step 3: mirror Forgejo into F3")
adminUsername := "user1"
logger := util.ToF3Logger(nil)
forgejoAPI := f3_forges.NewForgeRoot(&f3_forgejo.Forgejo{}, &f3_forgejo.Options{
Options: f3_types.Options{
Configuration: f3_types.Configuration{
URL: setting.AppURL,
Directory: t.TempDir(),
},
Features: f3_types.AllFeatures,
Logger: logger,
},
AuthToken: getUserToken(t, adminUsername, auth_model.AccessTokenScopeWriteAdmin, auth_model.AccessTokenScopeAll),
})
f3 := f3_forges.FixtureNewF3Forge(t, logger, nil, t.TempDir())
apiForge := forgejoAPI.Forge
apiUser := apiForge.Users.GetFromFormat(context.Background(), &format.User{UserName: gitlabUser.Name})
// apiProject := apiUser.Projects.GetFromFormat(context.Background(), &format.Project{Name: fixture.ProjectFormat.Name})
// options = f3_common.NewMirrorOptionsRecurse(apiUser, apiProject)
options = f3_common.NewMirrorOptionsRecurse(apiUser)
f3.Forge.Mirror(context.Background(), apiForge, options)
//
// Step 4: verify the fixture and F3 are equivalent
//
files := f3_util.Command(context.Background(), "find", f3.GetDirectory())
assert.Contains(t, files, fmt.Sprintf("/user/%d", gitlabUser.ID))
})
}
func TestF3_UserMappingNew(t *testing.T) {
onGiteaRun(t, func(t *testing.T, u *url.URL) {
AllowLocalNetworks := setting.Migrations.AllowLocalNetworks
setting.F3.Enabled = true
setting.Migrations.AllowLocalNetworks = true
AppVer := setting.AppVer
// Gitea SDK (go-sdk) need to parse the AppVer from server response, so we must set it to a valid version string.
setting.AppVer = "1.16.0"
defer func() {
setting.Migrations.AllowLocalNetworks = AllowLocalNetworks
setting.AppVer = AppVer
}()
log.Debug("Step 1: create a fixture in F3")
fixtureNewF3Forge := func(t f3_tests.TestingT, logger *f3_types.Logger, user *format.User, tmpDir string) *f3_forges.ForgeRoot {
root := f3_forges.NewForgeRoot(&f3_f3.F3{}, &f3_f3.Options{
Options: f3_types.Options{
Configuration: f3_types.Configuration{
Directory: tmpDir,
},
Features: f3_types.AllFeatures,
Logger: util.ToF3Logger(nil),
},
Remap: true,
})
return root
}
fixture := f3_forges.NewFixture(t, f3_forges.FixtureForgeFactory{Fun: fixtureNewF3Forge, AdminRequired: false})
userID := int64(5432)
fixture.NewUser(userID)
log.Debug("Step 2: mirror F3 into Forgejo")
doer, err := user_model.GetAdminUser(context.Background())
assert.NoError(t, err)
forgejoLocalDestination := util.ForgejoForgeRoot(f3_types.AllFeatures, doer, 0)
options := f3_common.NewMirrorOptionsRecurse()
forgejoLocalDestination.Forge.Mirror(context.Background(), fixture.Forge, options)
log.Debug("Step 3: change the Name of the user in F3 and mirror to Forgejo")
otherusername := "otheruser"
fixture.UserFormat.UserName = otherusername
fixture.Forge.Users.Upsert(context.Background(), fixture.UserFormat)
forgejoLocalDestination.Forge.Mirror(context.Background(), fixture.Forge, options)
log.Debug("Step 4: mirror Forgejo into F3 using the changed name")
f3 := util.F3ForgeRoot(f3_types.AllFeatures, t.TempDir())
forgejoLocalOrigin := util.ForgejoForgeRoot(f3_types.AllFeatures, doer, 0)
forgejoLocalOriginUser := forgejoLocalOrigin.Forge.Users.GetFromFormat(context.Background(), &format.User{UserName: otherusername})
options = f3_common.NewMirrorOptionsRecurse(forgejoLocalOriginUser)
f3.Forge.Mirror(context.Background(), forgejoLocalOrigin.Forge, options)
//
// verify the fixture and F3 are equivalent
//
files := f3_util.Command(context.Background(), "find", f3.GetDirectory())
assert.Contains(t, files, fmt.Sprintf("/user/%d", forgejoLocalOriginUser.GetID()))
})
}