fix(repo): stop reporting code scanning languages added by GitHub

docs/reference/organization/repository/index.md

@@ -20,7 +20,7 @@ Definition of a Repository for a GitHub organization, the following properties a
 | _auto_init_                               | boolean                                                   | If the repository shall be auto-initialized during creation                             | only considered during creation                                                                                                                                                            |
 | _code_scanning_default_setup_enabled_     | boolean                                                   | If the repo has default code scanning enabled                                           |                                                                                                                                                                                            |
 | _code_scanning_default_query_suite_       | string                                                    | The query suite to use for default code scanning                                        | `default` or `extended`, only taken into account when `code_scanning_default_setup_enabled` is set to true                                                                                 |
-| _code_scanning_default_languages_         | list[string]                                              | The languages to analyze with code scanning default setup                               | `actions`, `c-cpp`, `csharp`, `go`, `java-kotlin`, `javascript`, `python`, `ruby`, `swift`, `rust` or `typescript`, only taken into account when `code_scanning_default_setup_enabled` is set to true. The deprecated value `javascript-typescript` is translated to `javascript` and `typescript` |
+| _code_scanning_default_languages_         | list[string]                                              | The languages to analyze at least with code scanning default setup                      | `actions`, `c-cpp`, `csharp`, `go`, `java-kotlin`, `javascript`, `python`, `ruby`, `swift`, `rust` or `typescript`, only taken into account when `code_scanning_default_setup_enabled` is set to true. An empty list leaves the languages to GitHub, languages that GitHub analyzes in addition are kept. The deprecated value `javascript-typescript` is translated to `javascript` and `typescript` |
 | _custom_properties_                       | dict\[string, string \| list\[string\]\]                  | The custom properties to set for this repository                                        |                                                                                                                                                                                            |
 | _default_branch_                          | string                                                    | Name of the default branch                                                              |                                                                                                                                                                                            |
 | _delete_branch_on_merge_                  | boolean                                                   | If branches shall automatically be deleted after a merge                                |                                                                                                                                                                                            |

docs/userguide/guides/repository-security.md

@@ -75,11 +75,29 @@ of a public repository, see
 Default setup lets GitHub configure and run CodeQL. It is configured with:
 
 - `code_scanning_default_setup_enabled`: enables default setup
-- `code_scanning_default_languages`: the languages to analyze
+- `code_scanning_default_languages`: the languages to analyze at least
 - `code_scanning_default_query_suite`: `default` or `extended`
 
 The languages and the query suite are only taken into account when default setup is enabled.
 
+### Languages analyzed by GitHub
+
+Default setup analyzes the CodeQL languages that GitHub detects in the repository, and GitHub adds the languages
+that it detects later on, e.g. once Python code is pushed to the repository, see
+[About setup types for code scanning](https://docs.github.com/en/code-security/concepts/code-scanning/setup-types).
+The configured languages are thus the minimum set of languages to analyze:
+
+- without configured languages (an empty list), the languages to analyze are left to GitHub, `plan` and
+  `check-sync` do not report the languages that GitHub analyzes
+- a configured language that is not analyzed is added to the analyzed languages
+- languages that GitHub analyzes in addition to the configured ones are kept, they are not removed and not
+  reported as a difference
+
+!!! tip "Recommendation"
+
+    Leave the languages to GitHub, and only configure a language that GitHub does not analyze on its own, e.g. a
+    compiled language.
+
 ### Supported languages
 
 `code_scanning_default_languages` only accepts the language identifiers of CodeQL, not the names of the languages
@@ -136,11 +154,11 @@ orgs.newOrg('my-project', 'my-org') {
 }
 ```
 
-1. Without configured languages, the languages to analyze are left to GitHub. Once the repository contains code,
-   configure the languages explicitly, so that the configuration shows what is analyzed.
+1. Without configured languages, the languages to analyze are left to GitHub, see
+   [Languages analyzed by GitHub](#languages-analyzed-by-github).
 
-Once the repository contains code, e.g. Java, Kotlin and TypeScript code as well as workflows, the languages are
-configured with their CodeQL identifiers:
+Once the repository contains code, e.g. Java, Kotlin and TypeScript code as well as workflows, languages to analyze
+at least are configured with their CodeQL identifiers:
 
 ``` jsonnet
 orgs.newOrg('my-project', 'my-org') {

otterdog/models/repository.py

@@ -963,6 +963,53 @@ def is_key_valid_for_diff_computation(self, key: str, expected_object: Self) ->
 
         return True
 
+    def get_difference_from(self, other: Self) -> dict[str, Change[Any]]:
+        diff: dict[str, Change[Any]] = super().get_difference_from(other)
+
+        languages_change = diff.get("code_scanning_default_languages")
+        if languages_change is not None:
+            languages_change = self._get_code_scanning_languages_change(languages_change)
+            if languages_change is None:
+                diff.pop("code_scanning_default_languages")
+            else:
+                diff["code_scanning_default_languages"] = languages_change
+
+        return diff
+
+    @classmethod
+    def _get_code_scanning_languages_change(cls, change: Change[Any]) -> Change[Any] | None:
+        """
+        Returns the change of the code scanning default languages that needs to be applied, if any.
+
+        Default setup analyzes the languages that GitHub detects in the repository and adds the languages
+        detected later on, the configured languages are thus the minimum set of languages to analyze:
+        - without any configured language, the languages to analyze are left to GitHub
+        - languages analyzed in addition to the configured ones are kept
+        """
+        expected_languages = change.to_value
+        if not isinstance(expected_languages, list) or len(expected_languages) == 0:
+            return None
+
+        current_languages = list(change.from_value) if isinstance(change.from_value, list) else []
+        analyzed_languages = {cls._get_code_scanning_analyzed_language(x) for x in current_languages}
+
+        missing_languages: list[str] = []
+        for language in expected_languages:
+            if language not in missing_languages and (
+                cls._get_code_scanning_analyzed_language(language) not in analyzed_languages
+            ):
+                missing_languages.append(language)
+
+        if len(missing_languages) == 0:
+            return None
+
+        return Change(change.from_value, current_languages + missing_languages)
+
+    @classmethod
+    def _get_code_scanning_analyzed_language(cls, language: str) -> str:
+        # CodeQL analyzes JavaScript and TypeScript together, analyzing one of them analyzes both
+        return "javascript-typescript" if language in cls._combined_code_scanning_languages else language
+
     def include_for_live_patch(self, context: LivePatchContext) -> bool:
         import fnmatch
 
@@ -1212,7 +1259,10 @@ async def get_mapping_to_provider(
 
         if "code_scanning_default_languages" in data:
             mapping.pop("code_scanning_default_languages")
-            code_scanning_mapping["languages"] = S("code_scanning_default_languages")
+            # without any configured language, GitHub analyzes the languages it detects in the repository
+            languages = data["code_scanning_default_languages"]
+            if is_set_and_valid(languages) and len(languages) > 0:
+                code_scanning_mapping["languages"] = S("code_scanning_default_languages")
 
         if len(code_scanning_mapping) > 0:
             mapping["code_scanning_default_config"] = code_scanning_mapping

otterdog/providers/github/rest/repo_client.py

@@ -698,6 +698,16 @@ async def _fill_code_scanning_config(self, org_id: str, repo_name: str, repo_dat
                 config["languages"] = [lang for lang in config["languages"] if lang != "javascript-typescript"]
             repo_data["code_scanning_default_config"] = config
 
+    async def _get_analyzed_code_scanning_languages(self, org_id: str, repo_name: str) -> list[str]:
+        current_data: dict[str, Any] = {}
+        await self._fill_code_scanning_config(org_id, repo_name, current_data)
+
+        current_config = current_data.get("code_scanning_default_config", {})
+        if current_config.get("state") != "configured":
+            return []
+
+        return list(current_config.get("languages", []))
+
     @staticmethod
     def _to_default_setup_update_languages(languages: list[str]) -> list[str]:
         # GitHub reports 'javascript' and 'typescript' separately for the default setup, but the
@@ -714,6 +724,10 @@ async def _update_code_scanning_config(self, org_id: str, repo_name: str, code_s
 
         languages = code_scanning.get("languages")
         if isinstance(languages, list):
+            if len(languages) > 0:
+                # the configured languages are the minimum set of languages to analyze, keep the ones analyzed
+                # by the current default setup, e.g. added by GitHub when it detected a new language
+                languages = [*await self._get_analyzed_code_scanning_languages(org_id, repo_name), *languages]
             code_scanning = {**code_scanning, "languages": self._to_default_setup_update_languages(languages)}
 
         try:

otterdog/webapp/home/playground-reference.json

@@ -597,8 +597,8 @@
         },
         "code_scanning_default_languages": {
           "type": "list[string]",
-          "description": "The languages to analyze with code scanning default setup",
-          "notes": "`actions`, `c-cpp`, `csharp`, `go`, `java-kotlin`, `javascript`, `python`, `ruby`, `swift`, `rust` or `typescript`, only taken into account when `code_scanning_default_setup_enabled` is set to true. The deprecated value `javascript-typescript` is translated to `javascript` and `typescript`"
+          "description": "The languages to analyze at least with code scanning default setup",
+          "notes": "`actions`, `c-cpp`, `csharp`, `go`, `java-kotlin`, `javascript`, `python`, `ruby`, `swift`, `rust` or `typescript`, only taken into account when `code_scanning_default_setup_enabled` is set to true. An empty list leaves the languages to GitHub, languages that GitHub analyzes in addition are kept. The deprecated value `javascript-typescript` is translated to `javascript` and `typescript`"
         },
         "custom_properties": {
           "type": "dict[string, string | list[string]]",

tests/models/test_repository.py

@@ -854,3 +854,98 @@ async def test_explicit_single_code_scanning_language_must_be_detected(configure
 
     assert len(failures) == 1
     assert f"'{missing}'" in failures[0]
+
+
+def _code_scanning_repo(repository_test, enabled, languages):
+    repo = Repository.from_model_data(repository_test.model_data)
+    repo.code_scanning_default_setup_enabled = enabled
+    repo.code_scanning_default_query_suite = "default"
+    repo.code_scanning_default_languages = languages
+    return repo
+
+
+@pytest.mark.parametrize(
+    "configured,analyzed",
+    [
+        # without configured languages, the languages to analyze are left to GitHub
+        ([], ["actions", "python"]),
+        ([], None),
+        # the configured languages are a minimum, GitHub might analyze additional ones
+        (["actions"], ["actions", "go", "python"]),
+        (["actions", "python"], ["python", "actions"]),
+        # CodeQL analyzes JavaScript and TypeScript together
+        (["typescript"], ["javascript"]),
+        (["javascript", "typescript"], ["actions", "javascript", "typescript"]),
+    ],
+)
+def test_code_scanning_languages_analyzed_by_github_are_no_difference(repository_test, configured, analyzed):
+    expected = _code_scanning_repo(repository_test, True, configured)
+    current = _code_scanning_repo(repository_test, True, analyzed)
+
+    assert expected.get_difference_from(current) == {}
+
+
+@pytest.mark.parametrize(
+    "configured,analyzed,applied",
+    [
+        (["actions", "java-kotlin"], ["actions", "python"], ["actions", "python", "java-kotlin"]),
+        (["c-cpp", "c-cpp"], ["actions"], ["actions", "c-cpp"]),
+        (["javascript", "typescript"], ["python"], ["python", "javascript", "typescript"]),
+        (["actions"], [], ["actions"]),
+        (["actions"], None, ["actions"]),
+    ],
+)
+def test_missing_code_scanning_languages_are_added_to_the_analyzed_ones(repository_test, configured, analyzed, applied):
+    expected = _code_scanning_repo(repository_test, True, configured)
+    current = _code_scanning_repo(repository_test, True, analyzed)
+
+    assert expected.get_difference_from(current) == {"code_scanning_default_languages": Change(analyzed, applied)}
+
+
+def test_code_scanning_languages_are_added_when_enabling_default_setup(repository_test):
+    expected = _code_scanning_repo(repository_test, True, ["actions"])
+    current = _code_scanning_repo(repository_test, False, [])
+
+    assert expected.get_difference_from(current) == {
+        "code_scanning_default_setup_enabled": Change(False, True),
+        "code_scanning_default_languages": Change([], ["actions"]),
+    }
+
+
+def test_live_patch_ignores_code_scanning_languages_added_by_github(repository_test):
+    from otterdog.models import LivePatchContext
+    from otterdog.models.organization_settings import OrganizationSettings
+
+    org_settings = OrganizationSettings.from_model_data(
+        repository_test.load_json_resource("otterdog-org-settings.json")
+    )
+    context = LivePatchContext("test-org", "", False, False, "*", org_settings, org_settings)
+
+    expected = _code_scanning_repo(repository_test, True, [])
+    current = _code_scanning_repo(repository_test, True, ["actions", "go", "python"])
+
+    patches = []
+    Repository.generate_live_patch(expected, current, None, context, patches.append)
+
+    assert patches == []
+
+
+@pytest.mark.parametrize("languages", [[], None])
+async def test_code_scanning_without_languages_to_provider(repository_test, languages):
+    repo = _code_scanning_repo(repository_test, True, languages)
+
+    provider_data = await repo.to_provider_data(repository_test.org_id, repository_test.provider)
+
+    assert provider_data["code_scanning_default_config"] == {"state": "configured", "query_suite": "default"}
+
+
+async def test_code_scanning_languages_to_provider(repository_test):
+    repo = _code_scanning_repo(repository_test, True, ["actions", "python"])
+
+    provider_data = await repo.to_provider_data(repository_test.org_id, repository_test.provider)
+
+    assert provider_data["code_scanning_default_config"] == {
+        "state": "configured",
+        "query_suite": "default",
+        "languages": ["actions", "python"],
+    }

tests/providers/github/integration/test_repositories.py

@@ -126,3 +126,24 @@ async def test_update_repo_propagates_unavailable_code_scanning_when_enabled(git
             REPO_NAME,
             {"code_scanning_default_config": {"state": "configured"}},
         )
+
+
+async def test_update_repo_keeps_code_scanning_languages_analyzed_by_github(github: GitHubProviderTestKit):
+    """Configured languages are added to the ones analyzed by the current default setup, which are kept."""
+    github.http.expect(
+        "GET",
+        CODE_SCANNING_URL,
+        response_json={"state": "configured", "languages": ["actions", "javascript-typescript", "python"]},
+    )
+    github.http.expect(
+        "PATCH",
+        CODE_SCANNING_URL,
+        request_json={"languages": ["actions", "python", "java-kotlin"]},
+        response_json={"state": "configured"},
+    )
+
+    await github.provider.update_repo(
+        ORG_ID,
+        REPO_NAME,
+        {"code_scanning_default_config": {"languages": ["java-kotlin"]}},
+    )

tests/providers/github/rest/test_repo_client.py

@@ -96,7 +96,15 @@ async def test_update_code_scanning_config_combines_javascript_and_typescript(se
         async def mock_request_json(method, url, data=None):
             return {}
 
-        mocked_restapi = pretend.stub(requester=pretend.stub(request_json=pretend.call_recorder(mock_request_json)))
+        async def mock_request_raw(method, url):
+            return (200, json.dumps({"state": "not-configured", "languages": []}))
+
+        mocked_restapi = pretend.stub(
+            requester=pretend.stub(
+                request_json=pretend.call_recorder(mock_request_json),
+                request_raw=mock_request_raw,
+            )
+        )
         repo_client = RepoClient(mocked_restapi)
 
         code_scanning = {"state": "configured", "languages": languages}
@@ -113,6 +121,58 @@ async def mock_request_json(method, url, data=None):
         # the config passed by the caller is left untouched
         assert code_scanning == {"state": "configured", "languages": languages}
 
+    @pytest.mark.parametrize(
+        "current_config,languages,expected_languages",
+        [
+            (
+                {"state": "configured", "languages": ["actions", "python"]},
+                ["actions", "java-kotlin"],
+                ["actions", "python", "java-kotlin"],
+            ),
+            (
+                {"state": "configured", "languages": ["javascript", "javascript-typescript", "typescript"]},
+                ["actions", "javascript"],
+                ["javascript-typescript", "actions"],
+            ),
+            (
+                {"state": "not-configured", "languages": ["python"]},
+                ["actions"],
+                ["actions"],
+            ),
+        ],
+    )
+    async def test_update_code_scanning_config_keeps_analyzed_languages(
+        self, current_config, languages, expected_languages
+    ):
+        async def mock_request_json(method, url, data=None):
+            return {}
+
+        async def mock_request_raw(method, url):
+            return (200, json.dumps(current_config))
+
+        mocked_restapi = pretend.stub(
+            requester=pretend.stub(
+                request_json=pretend.call_recorder(mock_request_json),
+                request_raw=pretend.call_recorder(mock_request_raw),
+            )
+        )
+        repo_client = RepoClient(mocked_restapi)
+
+        await repo_client._update_code_scanning_config(
+            "test-org", "test-repo", {"state": "configured", "languages": languages}
+        )
+
+        assert repo_client.requester.request_raw.calls == [
+            pretend.call("GET", "/repos/test-org/test-repo/code-scanning/default-setup"),
+        ]
+        assert repo_client.requester.request_json.calls == [
+            pretend.call(
+                "PATCH",
+                "/repos/test-org/test-repo/code-scanning/default-setup",
+                data={"state": "configured", "languages": expected_languages},
+            ),
+        ]
+
     async def test_update_code_scanning_config_without_languages(self):
         async def mock_request_json(method, url, data=None):
             return {}