File obs-scm-bridge-0.2.obscpio of Package obs-scm-bridge.24636
07070100000000000081A400000000000000000000000162A070E500000179000000000000000000000000000000000000001C00000000obs-scm-bridge-0.2/Makefileprefix = /usr
PYTHON ?= python
servicedir = ${prefix}/lib/obs/service
all:
install:
install -d $(DESTDIR)$(servicedir)
install -m 0755 obs_scm_bridge $(DESTDIR)$(servicedir)
test:
flake8 set_version tests/
${PYTHON} -m unittest discover tests/
clean:
find -name "*.pyc" -exec rm {} \;
find -name '*.pyo' -exec rm {} \;
rm -rf set_versionc
.PHONY: all install test
07070100000001000081A400000000000000000000000162A070E500000820000000000000000000000000000000000000001D00000000obs-scm-bridge-0.2/README.md
Native OBS SCM bridge helper
============================
Native OBS scm support for the build recipies and additional files. This is bridging an external authorative
scm repository into OBS. Any source change or merge workflow must be provided via the scm repository
hoster in this scenario.
Only git is supported atm, but this can be extended later to further systems.
It is not recommended to put large binary files into git directly as this won't scale. Use the
asset support instead, which is described in pbuild documentation:
http://opensuse.github.io/obs-build/pbuild.html#_remote_assets
These assets will be downloaded by osc and OBS. The verification via sha256 sum is optional.
HOWTO manage a single package
=============================
The current way to define a git repository for an OBS package is using the `scmsync`
element inside the package meta.
```
<scmsync>https://github.com/foo/bar</scmsync>
```
For doing a local checkout use the currently experimental osc from
https://download.opensuse.org/repositories/home:/adrianSuSE:/OBSGIT/
This version allows you to do
# osc co $project <$package>
which will create a git repository inside of the classic osc checkout.
The only further tested functionality is to do local builds atm.
HOWTO manage an entire project
==============================
A git repository can also get defined for entire project. This can be done
via the scmsync element in project meta.
Any top level subdirectory will be handled as package container.
It is recomended to use git submodules for each package if it is a larger
project. This allows partial cloning of the specific package.
TODO
====
* Monitoring changes in referenced repository. (can currently be workarounded
via "osc service rr")
* osc upstream integration
* signature validation
* find a better way to store files in .osc and .assets of the checkout, as
they do not belong to the git repository
auto extending .gitignore? (esp. when downloading asset files?)
* make cpio generation bit identical (avoid mtime from clone)
07070100000002000081A400000000000000000000000162A070E500000506000000000000000000000000000000000000002700000000obs-scm-bridge-0.2/obs-scm-bridge.spec#
# spec file
#
# Copyright (c) 2021 SUSE LLC
#
# All modifications and additions to the file contributed by third parties
# remain the property of their copyright owners, unless otherwise agreed
# upon. The license for this file, and modifications and additions to the
# file, is the same license as for the pristine package itself (unless the
# license for the pristine package is not an Open Source License, in which
# case the license is the MIT License). An "Open Source License" is a
# license that conforms to the Open Source Definition (Version 1.9)
# published by the Open Source Initiative.
# Please submit bugfixes or comments via https://bugs.opensuse.org/
#
%if 0%{?fedora} || 0%{?rhel}
%define build_pkg_name obs-build
%else
%define build_pkg_name build
%endif
Name: obs-scm-bridge
Version: 0.0.1
Release: 0
Summary: A help service to work with git repositories in OBS
License: GPL-2.0-or-later
URL: https://github.com/openSUSE/obs-scm-bridge
Source0: %{name}-%{version}.tar.xz
Requires: %{build_pkg_name} >= 20211125
BuildArch: noarch
Recommends: python3-packaging
%description
%prep
%autosetup
%build
%install
make DESTDIR=%{buildroot} install
%files
%{_prefix}/lib/obs/service
%changelog
07070100000003000081ED00000000000000000000000162A070E500003868000000000000000000000000000000000000002200000000obs-scm-bridge-0.2/obs_scm_bridge#!/usr/bin/python3
# -*- coding: utf-8 -*-
# scm (only git atm) cloning and packaging for Open Build Service
#
# (C) 2021 by Adrian Schröter <adrian@suse.de>
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
# See http://www.gnu.org/licenses/gpl-2.0.html for full license text.
import argparse
import os
import re
import shutil
import sys
import logging
import subprocess
import tempfile
from html import escape
import urllib.parse
import configparser
outdir = None
download_assets = '/usr/lib/build/download_assets'
export_debian_orig_from_git = '/usr/lib/build/export_debian_orig_from_git'
pack_directories = False
get_assets = False
shallow_clone = True
if os.environ.get('DEBUG_SCM_BRIDGE') == "1":
logging.getLogger().setLevel(logging.DEBUG)
if os.environ.get('OBS_SERVICE_DAEMON'):
pack_directories = True
get_assets = True
if os.environ.get('OSC_VERSION'):
get_assets = True
shallow_clone = False
os.environ['LANG'] = "C"
class ObsGit(object):
def __init__(self, outdir, url):
self.outdir = outdir
self.revision = None
self.subdir = None
self.lfs = False
self.arch = []
self.url = list(urllib.parse.urlparse(url))
query = urllib.parse.parse_qs(self.url[4]);
if "subdir" in query:
self.subdir = query['subdir'][0]
del query['subdir']
self.url[4] = urllib.parse.urlencode(query)
if "arch" in query:
self.arch = query['arch']
del query['arch']
self.url[4] = urllib.parse.urlencode(query)
if "lfs" in query:
self.lfs = True
del query['lfs']
self.url[4] = urllib.parse.urlencode(query)
if self.url[5]:
self.revision = self.url[5]
self.url[5] = ''
def run_cmd(self, cmd, cwd=None, stdout=subprocess.PIPE, fatal=None):
logging.debug("COMMAND: %s" % cmd)
stderr = subprocess.PIPE
if stdout == subprocess.PIPE:
stderr = subprocess.STDOUT
proc = subprocess.Popen(cmd,
shell=False,
stdout=stdout,
stderr=stderr,
cwd=cwd)
output = proc.communicate()[0]
if isinstance(output, bytes):
output = output.decode('UTF-8')
logging.debug("RESULT(%d): %s", proc.returncode, repr(output))
if fatal and proc.returncode != 0:
print("ERROR: " + fatal + " failed: ", output)
sys.exit(proc.returncode)
return (proc.returncode, output)
def do_lfs(self, outdir):
cmd = [ 'git', '-C', outdir, 'lfs', 'fetch' ]
self.run_cmd(cmd, fatal="git lfs fetch")
def do_clone_commit(self, outdir):
cmd = [ 'git', 'init', outdir ]
self.run_cmd(cmd, fatal="git init")
cmd = [ 'git', '-C', outdir, 'remote', 'add', 'origin', urllib.parse.urlunparse(self.url) ]
self.run_cmd(cmd, fatal="git remote add origin")
cmd = [ 'git', '-C', outdir, 'fetch', 'origin', self.revision ]
if shallow_clone:
cmd += [ '--depth', '1' ]
print(cmd)
self.run_cmd(cmd, fatal="git fetch")
cmd = [ 'git', '-C', outdir, 'checkout', '-q', self.revision ]
self.run_cmd(cmd, fatal="git checkout")
def do_clone(self, outdir):
if self.revision and re.match(r"^[0-9a-fA-F]{40,}$", self.revision):
self.do_clone_commit(outdir)
if self.lfs:
self.do_lfs(outdir)
return
cmd = [ 'git', 'clone', urllib.parse.urlunparse(self.url), outdir ]
if shallow_clone:
cmd += [ '--depth', '1' ]
if self.revision:
cmd.insert(2, '-b')
cmd.insert(3, self.revision)
self.run_cmd(cmd, fatal="git clone")
if self.lfs:
self.do_lfs(outdir)
def clone(self):
if not self.subdir:
self.do_clone(self.outdir)
return
clonedir = tempfile.mkdtemp(prefix="obs-scm-bridge")
self.do_clone(clonedir)
fromdir = os.path.join(clonedir, self.subdir)
if not os.path.realpath(fromdir+'/').startswith(os.path.realpath(clonedir+'/')):
print("ERROR: subdir is not below clone directory")
sys.exit(1)
if not os.path.isdir(fromdir):
print("ERROR: subdir " + self.subdir + " does not exist")
sys.exit(1)
if not os.path.isdir(self.outdir):
os.makedirs(self.outdir)
for name in os.listdir(fromdir):
shutil.move(os.path.join(fromdir, name), self.outdir)
shutil.rmtree(clonedir)
def fetch_tags(self):
cmd = [ 'git', '-C', self.outdir, 'fetch', '--tags', 'origin', '+refs/heads/*:refs/remotes/origin/*' ]
logging.info("fetching all tags")
self.run_cmd(cmd, fatal="fetch --tags")
def cpio_directory(self, directory):
logging.info("create archivefile for %s", directory)
cmd = [ download_assets, '--create-cpio', '--', directory ]
archivefile = open(directory + '.obscpio', 'w')
self.run_cmd(cmd, stdout=archivefile, fatal="cpio creation")
archivefile.close()
def cpio_specials(self, specials):
if not specials:
return
logging.info("create archivefile for specials")
cmd = [ download_assets, '--create-cpio', '--', '.' ] + specials
archivefile = open('build.specials.obscpio', 'w')
self.run_cmd(cmd, stdout=archivefile, fatal="cpio creation")
archivefile.close()
def cpio_directories(self):
logging.debug("walk via %s", self.outdir)
os.chdir(self.outdir)
listing = sorted(os.listdir("."))
specials = []
for name in listing:
if name == '.git':
# we do not store git meta data service side atm to avoid bloat storage
# however, this will break some builds, so we will need an opt-out in future
shutil.rmtree(name)
continue
if name[0:1] == '.':
specials.append(name)
continue
if os.path.islink(name):
specials.append(name)
continue
if os.path.isdir(name):
logging.info("CPIO %s ", name)
self.cpio_directory(name)
shutil.rmtree(name)
if specials:
self.cpio_specials(specials)
for name in specials:
if os.path.isdir(name):
shutil.rmtree(name)
else:
os.unlink(name)
def get_assets(self):
logging.info("downloading assets")
cmd = [ download_assets ]
for arch in self.arch:
cmd += [ '--arch', arch ]
if pack_directories:
cmd += [ '--noassetdir', '--', self.outdir ]
else:
cmd += [ '--unpack', '--noassetdir', '--', self.outdir ]
self.run_cmd(cmd, fatal="asset download")
def copyfile(self, src, dst):
cmd = [ 'cp', '-af', self.outdir + "/" + src, self.outdir + "/" + dst ]
self.run_cmd(cmd, fatal="file copy")
def export_debian_files(self):
if os.path.isfile(self.outdir + "/debian/control") and \
not os.path.isfile(self.outdir + "/debian.control"):
self.copyfile("debian/control", "debian.control")
if os.path.isfile(self.outdir + "/debian/changelog") and \
not os.path.isfile(self.outdir + "/debian.changelog"):
self.copyfile("debian/changelog", "debian.changelog")
def get_debian_origtar(self):
if os.path.isfile(self.outdir + "/debian/control"):
# need up get all tags
if not self.subdir:
self.fetch_tags()
cmd = [ export_debian_orig_from_git, self.outdir ]
logging.info("exporting debian origtar")
self.run_cmd(cmd, fatal="debian origtar export")
def get_subdir_info(self, dir):
cmd = [ download_assets, '--show-dir-srcmd5', '--', dir ]
rcode, info = self.run_cmd(cmd, fatal="download_assets --show-dir-srcmd5")
return info.strip()
def write_info_file(self, filename, info):
infofile = open(filename, 'w')
infofile.write(info + '\n')
infofile.close()
def add_service_info(self):
info = None
if self.subdir:
info = self.get_subdir_info(self.outdir)
else:
cmd = [ 'git', '-C', outdir, 'show', '-s', '--pretty=format:%H', 'HEAD' ]
rcode, info = self.run_cmd(cmd, fatal="git show -s HEAD")
info = info.strip()
if info:
self.write_info_file(os.path.join(self.outdir, "_service_info"), info)
def write_package_xml_file(self, name, url):
xmlfile = open(name + '.xml', 'w')
xmlfile.write('<package name="' + escape(name) + '">\n')
xmlfile.write(' <scmsync>' + escape(url) + '</scmsync>\n')
xmlfile.write('</package>\n')
xmlfile.close()
def list_submodule_revisions(self):
revisions = {}
cmd = [ 'git', 'ls-tree', 'HEAD', '.' ]
rcode, output = self.run_cmd(cmd, fatal="git ls-tree")
for line in output.splitlines():
lstree = line.split(maxsplit=4)
if lstree[1] == 'commit' and len(lstree[2]) >= 40:
revisions[lstree[3]] = lstree[2]
return revisions
def generate_package_xml_files(self):
logging.debug("walk via %s", self.outdir)
os.chdir(self.outdir)
export_files = set(["_config"])
# find all top level git submodules
gitsubmodules = set()
if os.path.isfile('.gitmodules'):
gsmconfig = configparser.ConfigParser()
gsmconfig.read('.gitmodules')
revisions = None
for section in gsmconfig.sections():
if not 'path' in gsmconfig[section]:
logging.warn("path not defined for git submodule " + section)
continue
if not 'url' in gsmconfig[section]:
logging.warn("url not defined for git submodule " + section)
continue
path = gsmconfig[section]['path']
url = gsmconfig[section]['url']
if '/' in path:
# we handle only top level submodules in project mode
continue
# find revision of submodule
if revisions is None:
revisions = self.list_submodule_revisions()
revision = revisions.get(path, None)
if not revision:
logging.error("Could not determine revision of submodule " + section)
sys.exit(1)
# all good, write xml file and register the module
gitsubmodules.add(path)
url = list(urllib.parse.urlparse(url))
url[5] = revision
if self.arch:
query = urllib.parse.parse_qs(url[4]);
query['arch'] = self.arch
url[4] = urllib.parse.urlencode(query)
self.write_package_xml_file(path, urllib.parse.urlunparse(url))
self.write_info_file(path + ".info", revision)
export_files.add(path + ".xml")
export_files.add(path + ".info")
shutil.rmtree(path)
# handle plain files and directories
listing = sorted(os.listdir("."))
regexp = re.compile(r"^[a-zA-Z0-9\.\-\_\+]*$");
for name in listing:
if name == '.git':
shutil.rmtree(name)
continue
if os.path.isdir(name):
if name in gitsubmodules:
# already handled as git submodule
continue
info = self.get_subdir_info(name)
shutil.rmtree(name)
if not regexp.match(name):
logging.warn("directory name contains invalid char: " + name)
continue
# add subdir info file
self.write_info_file(name + ".info", info)
# add subdir parameter to url
url = self.url
query = urllib.parse.parse_qs(url[4])
query['subdir'] = name
url[4] = urllib.parse.urlencode(query)
self.write_package_xml_file(name, urllib.parse.urlunparse(url))
else:
if not name in export_files:
os.unlink(name)
if __name__ == '__main__':
parser = argparse.ArgumentParser(
description='Open Build Service source service for managing packaging files in git.'
'This is a special service for OBS git integration.')
parser.add_argument('--outdir', required=True,
help='output directory for modified sources')
parser.add_argument('--url',
help='REQUIRED: url to git repository')
parser.add_argument('--projectmode',
help='just return the package list based on the subdirectories')
parser.add_argument('--debug',
help='verbose debug mode')
args = vars(parser.parse_args())
url = args['url']
outdir = args['outdir']
project_mode = args['projectmode']
if not outdir:
print("no outdir specified")
sys.exit(-1)
if not url:
print("no url specified")
sys.exit(-1)
if args['debug']:
logging.getLogger().setLevel(logging.DEBUG)
logging.debug("Running in debug mode")
# workflow
obsgit = ObsGit(outdir, url)
obsgit.clone()
if project_mode == 'true' or project_mode == '1':
obsgit.generate_package_xml_files()
sys.exit(0)
if pack_directories:
obsgit.add_service_info()
if get_assets:
obsgit.get_assets()
obsgit.get_debian_origtar()
if pack_directories:
obsgit.export_debian_files()
obsgit.cpio_directories()
07070100000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000B00000000TRAILER!!!37 blocks
