golang gitlab subgroup 构建问题

golang gitlab subgroup

字数统计: 1.6k阅读时长: 8 min

 2025/06/17 

最近给同事解决的 subgroup 问题…..

由来

开发构建 Docker 镜像报错没权限拉取依赖：

xxxapi/middlewares imports
	xxx.xxxgitlab.net/x/xx/xxx/econtext: xxx.xxxgitlab.net/x/xx/xxx@v1.6.5-rc.7.0.20250609085545-4855369c0f1e: invalid version: git ls-remote -q origin in /go/pkg/mod/cache/vcs/672638ca2205d3f2cdc0288db28840b769e58250b95fbbd31e553c6c8076fc3c: exit status 128:
	remote: 
	remote: ========================================================================
	remote: 
	remote: The project you were looking for could not be found or you don't have permission to view it.
	remote: 
	remote: ========================================================================
	remote: 
	fatal: Could not read from remote repository.
	
	Please make sure you have the correct access rights
	and the repository exists.

解决过程

镜像内有 gitlab 的 deploy keys，初步怀疑是没依赖仓库权限，让开发联系该依赖仓库负责人，去开启 Enabled deploy keys 后还是一样。

手动构建测试

登录到构建机器上，docker build 实际就是按照 Dockerfile 来 docker run 和 docker commit 的结合，找到失败的 run 的镜像 id：

1 2	$ docker ps -a b7c4be99d86d ba54b82c5b1c "/bin/sh -c 'IN_DOCK…" 4 hours ago Exited (1) 1 hours ago nervous_chandrasekhar

用上面的镜像 ID 和 command 手动测下：

1
2
3

$ docker run --rm -ti --entrypoint bash ba54b82c5b1c
root@a9114c46d496:/go/src/xxx.xxxgitlab.net/xxxxx/# IN_DOCKER=1 bash ./build.sh
...

报错依旧，GOPRIVATE GONOPROXY 啥的都配置了的，以及 ssh 相关 git config --global url."git@xxx.xxxgitlab.net:".insteadof "https://xxx.xxxgitlab.net/"都是以及配置了没问题的：

1
2
3

$ cat ~/.gitconfig
[url "git@xxx.xxxgitlab.net:"]
	insteadof = https://xxx.xxxgitlab.net/

用 git 测试也没问题：

$ ssh -T git@xxx.xxxgitlab.net
Welcome to GitLab, @xxxx_reporter!
$ git clone xxx.xxxgitlab.net/x/xx/xxx
...

看了下 go 的 help ，发现 go mod 没有 debug level 相关 cmdline，但是 go get 有：

$ go get -x  xxx.xxxgitlab.net/x/xx/xxx
# get https://xxx.xxxgitlab.net/x/xx/xxx?go-get=1
# get https://xxx.xxxgitlab.net/x/xx/xxx?go-get=1: 200 OK (0.063s)
# get https://xxx.xxxgitlab.net/docmxini/xx?go-get=1
# get https://xxx.xxxgitlab.net/x/xx?go-get=1: 200 OK (0.016s)
mkdir -p /go/pkg/mod/cache/vcs # git3 https://xxx.xxxgitlab.net/x/xx.git
# lock /go/pkg/mod/cache/vcs/672638ca2205d3f2cdc0288db28840b769e58250b95fbbd31e553c6c8076fc3c.lock
# /go/pkg/mod/cache/vcs/672638ca2205d3f2cdc0288db28840b769e58250b95fbbd31e553c6c8076fc3c for git3 https://xxx.xxxgitlab.net/x/fx.git
cd /go/pkg/mod/cache/vcs/672638ca2205d3f2cdc0288db28840b769e58250b95fbbd31e553c6c8076fc3c; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' 4855369c0f1e --
0.002s # cd /go/pkg/mod/cache/vcs/672638ca2205d3f2cdc0288db28840b769e58250b95fbbd31e553c6c8076fc3c; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' 4855369c0f1e --
cd /go/pkg/mod/cache/vcs/672638ca2205d3f2cdc0288db28840b769e58250b95fbbd31e553c6c8076fc3c; git ls-remote -q origin
0.232s # cd /go/pkg/mod/cache/vcs/672638ca2205d3f2cdc0288db28840b769e58250b95fbbd31e553c6c8076fc3c; git ls-remote -q origin
# get https://xxx.xxxgitlab.net/x/xx.git
# get https://xxx.xxxgitlab.net/x/xx.git: 200 OK (0.076s)
cd /go/pkg/mod/cache/vcs/672638ca2205d3f2cdc0288db28840b769e58250b95fbbd31e553c6c8076fc3c; git tag -l
0.002s # cd /go/pkg/mod/cache/vcs/672638ca2205d3f2cdc0288db28840b769e58250b95fbbd31e553c6c8076fc3c; git tag -l
go: xxx.xxxgitlab.net/x/xx/xxx@v1.6.5-rc.7.0.20250609085545-4855369c0f1e: invalid version: git ls-remote -q origin in /go/pkg/mod/cache/vcs/672638ca2205d3f2cdc0288db28840b769e58250b95fbbd31e553c6c8076fc3c: exit status 128:
	remote: 
	remote: ========================================================================
	remote: 
	remote: The project you were looking for could not be found or you don't have permission to view it.
	remote: 
	remote: ========================================================================
	remote: 
	fatal: Could not read from remote repository.
	
	Please make sure you have the correct access rights
	and the repository exists.

上面最后的命令报错，cd 进去执行下看看：

$ cd /go/pkg/mod/cache/vcs/672638ca2205d3f2cdc0288db28840b769e58250b95fbbd31e553c6c8076fc3c
$ git ls-remote
remote: 
remote: ========================================================================
remote: 
remote: The project you were looking for could not be found or you don't have permission to view it.
remote: 
remote: ========================================================================
remote: 
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

其实前面的 go get -x 里就有问题详细信息了，这里我是看目录下文件发现的：

$ ls -al
total 28
drwxr-xr-x  7 root root  119 Jun 17 03:53 .
drwxr-xr-x 20 root root 8192 Jun 17 03:53 ..
-rw-r--r--  1 root root   23 Jun 17 03:53 HEAD
drwxr-xr-x  2 root root    6 Jun 17 03:53 branches
-rw-r--r--  1 root root  179 Jun 17 03:53 config
-rw-r--r--  1 root root   73 Jun 17 03:53 description
drwxr-xr-x  2 root root 4096 Jun 17 03:53 hooks
drwxr-xr-x  2 root root   21 Jun 17 03:53 info
drwxr-xr-x  4 root root   30 Jun 17 03:53 objects
drwxr-xr-x  4 root root   31 Jun 17 03:53 refs
$ cat config
[core]
	repositoryformatversion = 0
	filemode = true
	bare = true
[remote "origin"]
	url = https://xxx.xxxgitlab.net/x/xx.git
	fetch = +refs/heads/*:refs/remotes/origin/*

仓库地址不对，改成 url = https://xxx.xxxgitlab.net/x/xx/xxx.git 后就可以了：

$ git ls-remote | head
From git@xxx.xxxgitlab.net:x/xx/xxx.git
74f35a28ecc0f9721a49c41fb5d7bffa071d1502	HEAD
295c3a79dfc0d27022459d0cec21411031edd5e8	refs/heads/0xxx
...

gitlab subgroup

搜了下，发现是 gitlab 的鉴权和 golang 的 get 逻辑冲突，双方都认为对方不规范，谁都不让谁，具体见文章:

解决方案只有两种，一种是 ~/.netrc 但是是明文密码，不适用于 CI/CD 构建。
使用 replace 的话需要指定一样的版本，而 go get 升级依赖的时候 replace 不会动。
写 shell 在 go build 前执行的话，怕正则边界和模糊匹配到了前缀一样的，并且 go mod edit -json 可以输出 json 信息，好在 golang 编译镜像是 ubuntu，内部有 python。

脚本解决

花了点时间写了如下 python 脚本，执行方式如下：

go mod edit -json | python3 scripts/golang-subgroup.py \
  --repo xxx.xxxgitlab.net/x/xx/xxx \
  --repo xxx.xxxgitlab.net/x/xx/xxx2

# 或者
go mod edit -json > go.mod.json
python3 scripts/golang-subgroup.py \
  --repo xxx.xxxgitlab.net/x/xx/xxx \
  --repo xxx.xxxgitlab.net/x/xx/xxx2

# -*- coding: utf-8 -*-

import argparse
import sys
import os
import json
import logging

def parse_args():
    parser = argparse.ArgumentParser(description='解决 go.mod 里的 gitlab subgroups 问题')
    parser.add_argument('--repo', action='append', help='要处理的仓库名字，例如: xx.gitlab.cn/a/b/c')
    parser.add_argument('--mod', default="./go.mod", help='go.mod path')
    return parser.parse_args()

def replace_repo(repo_list, mod_json, mod_path):
    if mod_json['Replace']:
        mod_replace_list = [ i['Old']['Path'] for i in mod_json['Replace']]
        repo_list = [repo for repo in repo_list if repo not in mod_replace_list]
        if len(repo_list) == 0:
            logging.info("已经全部替换")
            return

    replace_list = [ i for i in mod_json['Require'] if i['Path'] in repo_list and (not i.get('Indirect', False))]
    if len(replace_list) == 0:
        logging.error("未找到需要替换的仓库: %s", ','.join(repo_list))
        return
    with open(mod_path, 'a', encoding='utf-8') as f:
        for item in replace_list:
            replace_str = "replace {0} => {0}.git {1}".format(item['Path'], item['Version'])
            logging.info("添加 %s", replace_str)
            f.writelines(replace_str+'\n')

def main():
    logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
    args = parse_args()
    if args.repo is None or len(args.repo) == 0:
        return

    mod_json = {}
    mod_json_path = "go.mod.json"
    if os.isatty(0):
        if not os.path.isfile(mod_json_path):
            logging.error("请以 go mod edit -json | python3 %s 运行", __file__)
            os._exit(2)
        with open(mod_json_path, 'r') as file:
            mod_json = json.load(file)
    else:
        mod_json = json.loads(sys.stdin.read())

    
    if (not isinstance(mod_json, dict)) or mod_json.get("Module", "") == "":
        logging.error("请以 go mod edit -json | python3 %s 运行", __file__)
        os._exit(2)
    replace_repo(args.repo, mod_json, args.mod)

if __name__ == "__main__":
    main()