[Jenkins][Issue] Artifact 的 Local Copy 毀損時仍被判定為有效而不會重新下載 (Python)

透過 Python API (JenkinsAPI 取得 artifacts 時,通常都是透過:

理想上,Artifact.save() 會檢查 local copy 的 fingerprint 是否跟 server 上的 artifact 一致,如果 local copy 有問題才會重新下載。但過程中,如果你觀察到有下面的 logs 出現,可能就得小心了…

DEBUG:requests.packages.urllib3.connectionpool:"GET /fingerprint/9fb189f24862ce8b96468df4ebc50629/api/python HTTP/1.1" 404 None
ERROR:root:Failed request at https://<JENKINS_SERVER>/fingerprint/9fb189f24862ce8b96468df4ebc50629/api/python with params: None
INFO:jenkinsapi.artifact:Local copy of <ARTIFACT_NAME> is already up to date.

因為最後的 “Local copy of … is already up to date" 可能會讓人不把上一個 ERROR 當做一回事。目前 JenkinsAPI 最新版 (0.2.25) 的設計還是 – 如果找不到 local copy 的 fingerprint,也將它視為 valid 或 up to date

fingerprint-404-not-found-python/fingerprint_not_found_404.png
Figure 1. 直接訪問 https://<JENKINS_SERVER>/fingerprint/<FINGERPRINT>/api/python 就會看到 404 Not Found 的錯誤

對大部份的人來說,這都是個大問題!一旦有 local copy,就算已經毀損也會被視為沒問題。

Note 這個問題已於 2014-08-18 回報 Local copy passes the fingerprint validation event it is broken · Issue #303,並提案增加 strict_validation 來決定是否將 negative validity 視為錯誤。

但為什麼會這樣呢?原因很簡單,因為 file fingerprinting 的功能不一定有啟用,找不到 fingerprint 並無法斷言 local copy 有問題,這在 Fingerprint.valid() 的文件有提到:

Return True / False if valid. If returns True, self.unknown is set to either True or False, and can be checked if we have positive validity (fingerprint known at server) or negative validity (fingerprint not known at server, but not really an error).

We can’t really say anything about the validity of fingerprints not found — but the artifact can still exist, so it is not possible to definitely say they are valid or not.

class Fingerprint(JenkinsBase):

    def valid(self):
        try:
            self.poll()
            self.unknown = False
        except requests.exceptions.HTTPError as err:
            response_obj = err.response
            if response_obj.status_code == 404:
                self.unknown = True
                return True
            else:
                return False

        return True

    def validate_for_build(self, filename, job, build):
        if not self.valid():
            log.info("Unknown to jenkins.")
            return False
        if self.unknown:
            # not request error, but unknown to jenkins
            return True
        # ...

也就是說 server 上找不到 client 送上來驗證的 fingerprint 時 (404),又細分成下面兩種狀況:

  • Positive Validity – 表示 local copy 確定沒問題 (self.unknown == False)

  • Negative Validity – 表示 local copy “可能" 沒問題 (self.unknown == True),不能視為一個錯誤。

看似合理的設計,但從 Artifact.save() caller 的角度來看,卻無從得知 positive/negative validity 的差異。

class Artifact(object):

    def save(self, fspath):
        log.info(msg="Saving artifact @ %s to %s" % (self.url, fspath))
        if not fspath.endswith(self.filename):
            log.warn(msg="Attempt to change the filename of artifact %s on save." % self.filename)
        if os.path.exists(fspath):
            if self.build:
                try:
                    if self._verify_download(fspath): # 1
                        log.info(msg="Local copy of %s is already up to date." % self.filename)
                        return fspath
                except ArtifactBroken:      # 2
                    log.info("Jenkins artifact could not be identified.")
            else:
                log.info("This file did not originate from Jenkins, so cannot check.")
        else:
            log.info("Local file is missing, downloading new.")
        filepath = self._do_download(fspath)
        try:
            self._verify_download(filepath) # 3
        except ArtifactBroken:
            log.warning("fingerprint of the downloaded artifact could not be verified")
        return fspath

    def _verify_download(self, fspath):
        """
        Verify that a downloaded object has a valid fingerprint.
        """
        local_md5 = self._md5sum(fspath)
        fp = Fingerprint(self.build.job.jenkins.baseurl, local_md5, self.build.job.jenkins)
        return fp.validate_for_build(os.path.basename(fspath), self.build.job.name, self.build.buildno)
1 就目前的設計而言,在 server 端找不到 local copy 的 fingerprint,也會被視為 valid。

這也就是為什麼明明 local copy 有問題時,還是會在 log 裡看到 “Local copy of … is already up to date." 的原因。

2 事實上,try clause 裡沒人會拋出 ArtifactBroken 的例外。
3 這樣的檢查在下載完 artifact 之後也會做,不過也有相同的問題。

尤其 file fingerprinting 的功能有啟用時,這樣的結果更是無法接受。在這個問題獲得改善之前,可以自己多做一層檢查,以達到 “找不到 fingerprint 肯定就表示 local copy 有問題" 的效果:

import os, logging
from jenkinsapi.fingerprint import Fingerprint
from jenkinsapi.custom_exceptions import ArtifactBroken

__all__ = ['save_artifact', 'save_artifact_to_dir']

_log = logging.getLogger(__name__)

def save_artifact(artifact, fspath, strict_validation=False):         # 1
    if os.path.exists(fspath):
        try:
            _verify_local_copy(artifact, fspath, strict_validation)
        except ArtifactBroken:
            _log.warning('Local copy [%s] is invalid, remove it.', fspath)
            os.remove(fspath)

    return _verify_local_copy(artifact, artifact.save(fspath), strict_validation)

def save_artifact_to_dir(artifact, dirpath, strict_validation=False): # 1
    fspath = os.path.join(dirpath, artifact.filename)
    return save_artifact(artifact, fspath, strict_validation)

def _validate_local_copy(artifact, fspath, strict):
    local_md5 = artifact._md5sum(fspath)
    baseurl = artifact.build.job.jenkins.baseurl
    job_name = artifact.build.job.name
    buildno = artifact.build.buildno

    fp = Fingerprint(baseurl, local_md5, artifact.build.job.jenkins)
    valid = fp.validate_for_build(os.path.basename(fspath), job_name, buildno)

    if not valid or (fp.unknown and strict): # strict = 404 as invalid 1
        raise ArtifactBroken("Artifact %s seems to be broken, check %s" % (local_md5, baseurl))
    return fspath
1 另外拉出一個 strict_validation 的參數,決定是否將 server 端找不到 fingerprint 的情況 (404) 視為一種錯誤。
廣告

發表迴響

Please log in using one of these methods to post your comment:

WordPress.com Logo

您的留言將使用 WordPress.com 帳號。 登出 / 變更 )

Twitter picture

您的留言將使用 Twitter 帳號。 登出 / 變更 )

Facebook照片

您的留言將使用 Facebook 帳號。 登出 / 變更 )

Google+ photo

您的留言將使用 Google+ 帳號。 登出 / 變更 )

連結到 %s