Probleme beim Update des Bitnami Ghost Helm Charts auf die Version 21.0.1

Das neueste Update (27.05.2024) des Bitnami Ghost Helm Charts bringt eine unangenehme Überraschung mit sich: einen nicht dokumentierten Migrationspfad. Ohne diesen startet der MySQL Pod nicht mehr, was wiederum den Ghost Blog lahmlegt.

Doch keine Panik, ich bin gerade die letzten 2h durch diese Tatsache gegangen und habe nun einen guten dokumentierten Weg (auf Englisch), den ihr gerne befolgen könnt:

At first I found a workaround, add this to your helm chart custom values:

#...
mysql:
  enabled: true
#...
  auth:
#...
    authenticationPolicy: "mysql_native_password"
  primary:
    configuration: |-
        [mysqld]
        authentication_policy='{{- .Values.auth.authenticationPolicy | default "* ,," }}'
        mysql_native_password=ON
        skip-name-resolve
        explicit_defaults_for_timestamp
        basedir=/opt/bitnami/mysql
        plugin_dir=/opt/bitnami/mysql/lib/plugin
        port={{ .Values.primary.containerPorts.mysql }}
        mysqlx={{ ternary 1 0 .Values.primary.enableMySQLX }}
        mysqlx_port={{ .Values.primary.containerPorts.mysqlx }}
        socket=/opt/bitnami/mysql/tmp/mysql.sock
        datadir=/bitnami/mysql/data
        tmpdir=/opt/bitnami/mysql/tmp
        max_allowed_packet=16M
        bind-address=*
        pid-file=/opt/bitnami/mysql/tmp/mysqld.pid
        log-error=/opt/bitnami/mysql/logs/mysqld.log
        character-set-server=UTF8
        slow_query_log=0
        long_query_time=10.0

        [client]
        port={{ .Values.primary.containerPorts.mysql }}
        socket=/opt/bitnami/mysql/tmp/mysql.sock
        default-character-set=UTF8
        plugin_dir=/opt/bitnami/mysql/lib/plugin

        [manager]
        port={{ .Values.primary.containerPorts.mysql }}
        socket=/opt/bitnami/mysql/tmp/mysql.sock
        pid-file=/opt/bitnami/mysql/tmp/mysqld.pid
    persistence:
#...

What does that do?
It resets the authentication_policy back to mysql_native_password. To do this, we must then set an additional option in mysql.conf, mysql_native_password=ON. Therefore, the default config was overwritten with all its default options and this value was added.

Why did this happen?
The latest Ghost Helm version also uses a new MySQL chart version from Bitnami, where a breaking change regarding authentication has been introduced. Or rather, a long announced deprecation was removed.

Source

This major bump uses mysql 8.4 image, that includes several configuration settings, for example the parameter auth.defaultAuthenticationPlugin has been removed in favor of auth.authenticationPolicy. This could potentially break your deployment and you would need to adjust the config settings accordingly.

Related notes in the deprecation removal in mysql: https://dev.mysql.com/doc/relnotes/mysql/8.4/en/news-8-4-0.html#mysqld-8-4-0-deprecation-removal

And what happens now?

With the above options, the mysql pod will start again and so will Ghost. At this point you should make a backup of your posts at the latest. From this point onwards, you can upgrade to the new authentication method. Follow this guide: https://dnsmichi.at/2024/05/11/docker-compose-ghost-msql-8-4-msql-native-password-startup-errors-mitigation-solution/#how-to-migrate-to-cachingsha2password

I am currently trying this out and will then update this comment with the specific commands.

Edit:
mysql-0 is the name of pod running mysql

kubectl exec -it mysql-0 – bash # connect into the pod with an interactive shell
mysql -u root -p # enter root password afterwards, you will find this information the related secret in your cluster

mysql> SELECT User, Host, plugin FROM mysql.user;
+------------------+-----------+-----------------------+
| User             | Host      | plugin                |
+------------------+-----------+-----------------------+
| bn_ghost         | %         | mysql_native_password |
| root             | %         | mysql_native_password |
| mysql.infoschema | localhost | caching_sha2_password |
| mysql.session    | localhost | caching_sha2_password |
| mysql.sys        | localhost | caching_sha2_password |
+------------------+-----------+-----------------------+
5 rows in set (0.00 sec)

Now we change to the new authentication method: caching_sha2_password. But you should dump your database before! Open another terminal for the next command.

kubectl exec -it mysql-0 – /bin/sh -c  "mysqldump -u root -p<EnterPasswordHereWithoutALeadingSpace> --databases <EnterDatabaseNameHere>" >dump.sql

Now go back to the previous terminal. The authentication method will now be changed.

mysql> ALTER USER 'root'@'%' IDENTIFIED WITH caching_sha2_password BY '<EnterYourPasswordForRootHere>';
Query OK, 0 rows affected, 1 warning (0.05 sec)
mysql>  ALTER USER 'bn_ghost'@'%' IDENTIFIED WITH caching_sha2_password BY 'EnterYourPasswordForbn_ghostHere';
Query OK, 0 rows affected, 1 warning (0.02 sec)
mysql> SELECT User, Host, plugin FROM mysql.user;
+------------------+-----------+-----------------------+
| User             | Host      | plugin                |
+------------------+-----------+-----------------------+
| bn_ghost         | %         | caching_sha2_password |
| root             | %         | caching_sha2_password |
| mysql.infoschema | localhost | caching_sha2_password |
| mysql.session    | localhost | caching_sha2_password |
| mysql.sys        | localhost | caching_sha2_password |
+------------------+-----------+-----------------------+
5 rows in set (0.00 sec)

Now you can exit mysql and exit the pod and remove the options added above regarding the mysql_native_password and restart your helm deployment and cross your fingers.
(remove mysql.auth.authenticationPolicy and mysql.primary.configuration)

First the mysql pod should restart and become healthy with the livenessProbe. Then the deployment of ghost should be restarted, as it is probably currently in a crash loop. After the initialization is done and this pod is also healthy, your Ingress should work again and the blog should be accessible without data loss. If something went wrong, you should now have the dump of your MySQL and the exported blog posts. You can now start from a fresh state.

Diesen Kommentar habe ich auch in das bereits geöffnete Issue gepostet. Ich bin gespannt, ob Bitnami, die Hersteller des Charts, noch etwas in die neueren Versionen des Charts übernehmen, damit dieser Migrationspfad etwas smoother wird.