Problem:
I’m working on an Ansible playbook to manage GitLab Runner registration and configuration. The playbook should:

Register a GitLab Runner and extract its token and ID.
Template a config file with the extracted variables, including a marker (####) at the end.
On subsequent runs, it should find the #### marker in the existing config, remove everything after it, and append new configuration content.
However, on rerunning the playbook, the #### marker is getting removed inadvertently, causing the config to be incorrect. Specifically, the issue arises when default configurations are added back, resulting in the #### marker being deleted, and the new configurations are not appended correctly.

To visualize it:

<start of template 1>
Template 1 
bla blabla
bla 
bla
####
<end of template 1>
(then i rerun and the default config gets put in so now the config file looks like this:

<start of template 1>
Template 1 
bla blabla
bla 
bla
####
<end of template 1>
<Start of default config 2>
bla bla
bla
bla
<End of default config 2>

so basically the default config deletes the marker #### upon regstering

Wanted output:

So it should see the marker (“####”) on the first template, remove the default config after extracting the token and id, and template to a config.tmp with token and id then append that config.tmp to the “main” config.toml

<start of template 1>
Template 1 
bla blabla
bla 
bla
####
<end of template 1>
(then i rerun and the default config gets put in so now the config file looks like this:

<start of template 1>
Template 1 
bla blabla
bla 
bla
<end of template 1>
<Start of template 2>
bla bla
bla
bla
####
<End of template 2>

Here is part of the playbook that is in play:

(This verison is below gitlab 15.6 so it works)
- name: register runner
  command: |
      docker compose run --rm runner register 
        --non-interactive 
        --url "{{ gitlab_runner_url }}" 
        --registration-token "{{ gitlab_runner_reg_token }}" 
        --name "{{ gitlab_runner_name }}"
        --executor "{{ gitlab_runner_executor }}" 
        --docker-image "{{ gitlab_runner_docker_image }}" 
  args:
    chdir: "{{ gitlab_path_dir }}"
  when: register_runner_confirm.user_input | lower == 'y'
  register: gitlab_runner_registred

- name: extract runner token
  shell: "grep -Po '(?<=token = ")[^"]*' {{ gitlab_path_config }} | tail -1"
  register: gitlab_runner_config_token

- name: extract runner id
  shell: "grep -Po '(?<=id = )[0-9]+' {{ gitlab_path_config }} | tail -1"
  register: gitlab_runner_config_id

- name: clear config file if marker file does not exist
  copy:
    content: ""
    dest: "{{ gitlab_path_config }}"
  when: not marker_file_result.stat.exists

- name: template runner config with variables
  template:
    src: gitlab-runner-config.j2
    dest: "{{ gitlab_path_config }}"
  vars:
    gitlab_runner_token: "{{ gitlab_runner_config_token.stdout }}"
    gitlab_runner_id: "{{ gitlab_runner_config_id.stdout | int }}"
  when: not marker_file_result.stat.exists

- name: Read existing config content
  slurp:
    src: "{{ gitlab_path_config }}"
  register: existing_config_content
  when: marker_file_result.stat.exists

- name: Find last occurrence of #### and get line number
  shell: "grep -n '####' {{ gitlab_path_config }} | tail -1 | cut -d: -f1"
  register: last_section_line
  changed_when: false
  when: marker_file_result.stat.exists

### Not sure if it works since im not getting to this part
#- name: Remove content after last #### and store in temporary file
#  shell: |
#    last_line="{{ last_section_line.stdout | default('$') }}"
#    sed "${last_line},$d" {{ gitlab_path_config }} > {{ gitlab_path_dir }}/existing_config.tmp
#  when: marker_file_result.stat.exists

#- name: Template new runner config
#  template:
#    src: gitlab-runner-config.j2
#    dest: "{{ gitlab_path_dir }}/new_config.tmp"
#  vars:
#    gitlab_runner_token: "{{ gitlab_runner_config_token.stdout }}"
#    gitlab_runner_id: "{{ gitlab_runner_config_id.stdout | int }}"
#  when: marker_file_result.stat.exists

### Not sure if it works since im not getting to this part
#- name: Concatenate cleaned config with temporary config
#  shell: | 
#   "cat <(sed '{{ last_section_line.stdout }}q' {{ gitlab_path_config }}) {{ gitlab_path_dir }} #/
#   /gitlab-runner-config.tmp > {{ gitlab_path_config }}.new /
#   && mv {{ gitlab_path_config }}.new {{ gitlab_path_config }}" /
#  when: marker_file_result.stat.exists
  
- name: Remove temporary files
  file:
    path: "{{ item }}"
    state: absent
  loop:
    - "{{ gitlab_path_dir }}/gitlab-runner-config.tmp"

- name: template registry config file
  template:
    src: registry-config.j2
    dest: "{{ gitlab_path_registry_mirror }}"


- name: create marker file to indicate first run is completed
  file:
    path: "{{ gitlab_path_dir }}/.first_run_completed"
    state: touch
  when: not marker_file_result.stat.exists

Attempted Solutions:
Initial Playbook Logic:

Registered the runner and extracted the token and ID.
Cleared the config file if it was the first run.
Added the #### marker at the end of the config.
On subsequent runs, attempted to remove content after #### and append new configuration.
However, the #### marker was being deleted, causing issues in maintaining the correct config structure.

Shell Command for Removing Content After ####:

Used sed to find the last occurrence of #### and delete everything after it.
The command sed -i ‘/####/,$d’ {{ gitlab_path_config }} worked initially but sometimes failed to correctly find and maintain the #### marker.
Updated Approach:

Read the existing config content.
Decoded the base64-encoded content.
Found the last occurrence of #### and removed content after it.
Temporarily stored the remaining content.
Created new runner configuration and appended it correctly.
Despite these attempts, I was still facing issues with the #### marker being removed incorrectly and the configurations not appending as expected.

Expectation:
I expected the playbook to correctly identify the #### marker without removing it on the subsequent runs, remove everything after it on subsequent runs, and append new configurations while retaining the #### marker. This would ensure that the configuration file remains correctly structured across multiple playbook runs as that is my end goal.