Categories
Docker Kubernetes Kubernetes

Kubernetes Upgrade fails with timeout

What the heck? The latest upgrade procedure of my Kubernetes cluster gave me headaches. Not only because it failed with a timeout – mainly because the root cause was not obvious. In fact, the maintainers of Kubernetes made an infrastructure change long time ago but forgot to properly communicate to their users.

But before we start the rant, let’s check what happened – I tried to upgrade from v1.18.2 to v1.18.14. This happened:

timed out waiting for the condition
couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced
k8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade.rollbackOldManifests

So I started to re-run the upgrade with verbosity on. Nothing more information. What I saw was that the kube-apiserver won’t come up – no log file gave a reason why this could happen. I asked Google – very little information but one hint – the image pull could have been failing.

Another search revealed that Kubernetes maintainers changed their repository from gcr.io/google_containers to k8s.gcr.io – presumingly long time ago. And checking my cluster more thoroughly I found out that the old repository was being used. But why was my cluster not knowing the new one? I was upgrading each major version – since the beginning.

Next search for the information on how to change it – nothing on Kubernetes docs (WTF!) but in some change request. You need to see the kubeadm-config ConfigMap in your kube-system namespace. There you’ll find the repository address. Changing this to the correct name finally did the trick and the upgrade succeeded.

But the more I think about this challenge, the more angry I get.

  1. How can such an essential change not be communicated more prominently – especially since the old repository was abandoned with v1.18.6 – the last image version in the old repo. Every upgrade document sind 1.18 must have a warning that the old repo is out of order now and a link to the change procedure
  2. Why is the error message not telling anything useful? The stacktrace is useless for the information about what happened.
  3. And why – for God’s sake – does the upgrade procedure itself not check for this essential change? Especially since v1.18.7.

This way of maintaining software is a very unprofessional one. Kubernetes is the foundation of so many productive systems now that this essential change must be taken more seriously by the maintainers. Breaking the procedures is a danger to all these systems and a proper communication or risk mitigation is not in place.

I need to stress out that upgrading Kubernetes is always risky. I experienced so many issues in the past that blocked an upgrade. Most of them were better documented and so I could resolve them. But this infrastructure change is a sign of unprofessional risk management. And I hope they will do much better next time.

Categories
Docker Kubernetes

Composer Updates in Docker Images

I recently started to use composer-packaged applications within Docker images. But I noticed that the Docker image did not change when I rebuilt after I updated one of the module dependencies. The reason is the caching mechanism of docker builds. A very obvious solution would be the --no-cache option when building. But this is a bad solution as it would cause the complete build to make use of the cache.

Another solution I found was using ARGs that will disable the cache for a command. I personally felt this not very helpful as it would involve some awkward tweak of the build process (as I understand). So I discovered a very simple solution: the ADD command can skip the caching when using a web URL as the content to download. The command would download the content from the address and compare it to the previous cache content. We would just need a URL that returns a different content each time it is called. And luckily there is one.

So the elegant solution is to use this combination of lines in your Dockerfile and the composer modules are updated accordingly each time:

# The RUN must use cached layers, this is the hack
ADD "https://www.random.org/cgi-bin/randbyte?nbytes=10&format=h" skipcache
RUN composer install \
    && chown -R www-data:www-data /var/www/html
Categories
Applications Bugzilla for Java CSV Compiling Miscellaneous Projects RS Library RsBudget Templating

The End of Atlassian JIRA

Atlassian announced the end of their various licensed stand-alone products. This heavily affects several of my projects, especially my Open Source projects. That’s why I am preparing now the migration away from Atlassian products. Of course, I could stay with Atlassian using their Cloud offerings. I like their products as they address my needs like no other products on the market. However, there are some downsides if I’d stay with Atlassian:

  • A migration of all existing data is unavoidable and will cost time and effort. It is not clear whether the Cloud product configuration would match my needs.
  • My Open Source projects are already code-hosted by GitHub. That’s why GitHub is the natural migration target for them.
  • I want to have full control over my CI/CD pipelines. A cloud Bamboo solution will take away a lot of freedom and I am not sure whether the various secrets I require during build and develop will stay on my servers and only there.
  • A long-term availability of issues and documentation is not guaranteed if I shall be forced to abandon projects.

So, all issue trackers are migrated to GitHub by today. The JIRA server has been shutdown. Please refer to the respective GitHub repositories in case you need to report an issue or require support for any of my projects.

Bamboo will be migrated to Jenkins. However, I experience some performance issues when starting Jenkins. That’s why this migration will still take a while. However, this would not affect your activities when using any of the projects.

I deeply regret to take this decision and would have loved to stay with Atlassion JIRA and Bamboo.

Categories
Linux Miscellaneous

Raspberry Pi 3: Turn Monitor On/Off

This is the only way the touchscreen LCD at my Raspberry Pi 3 turns on and off:

vcgencmd display_power 0 # turn off
vcgencml display_power 1 # turn on
Categories
Miscellaneous PHP

Accessing a Vault in PHP

I admit it. I was “hacked”. It was not really a hack, it was my mistake not to notice that one of my deployment logs was accessible for everyone to read and – exceptionally for a specific test performed back then – revealing configuration data including a password to an AWS service. The leakage has been fixed and the password changed. So, you don’t need to search for it anymore 🙂

However, the incident was reason enough for me to further secure my applications and introduce a vault for secret information. As most of my applications are based on PHP, I tried to find some ready-to-use code (and found a few). But all these libraries and SDKs are very heavyweight as they address much more use cases than just accessing a vault in order to fetch a secret read-only.

So I wrote a lightweight version of a PHP vault that not only accesses a Hashicorp Vault but also provides an abstract API so that my applications do not need to know what vault is being used or even where the secrets are stored.

So here is a code snippet that demonstrates how to use it with a Hashicorp Vault:

// Create configuration
$config = array(
	'type'   => 'hashicorp',
	'config' => array(
		'uri'      => 'https://127.0.0.1:8200/v1',
		'roleId'   => '<app-role-id>',
		'secretId' => '<secret-id>'
	)
);

// Create the vault instance
try {
	$vault = \Vault\VaultFactory::create($config);
} catch (\Vault\VaultException $e) {
	// Vault could not be created
}

With that vault, I can now access my secrets transparently within application code:

try {
	$mySecret = $vault->get('my/secret");
	$username = $mySecret->get('username');
	$password = $mySecret->get('password');
} catch (\Vault\VaultException $e) {
	// secret was not found
}

I even can further abstract this by not even knowing that there is a vault involved:

$callback = new \Vault\CredentialsProvider($vault, 'my/secret');

try {
	$username = $callback->getUsername();
	$password = $callback->getPassword();
} catch (\Vault\VaultException $e) {
	// Secret cannot be retrieved or does not exist
}

I can now integrate this pattern in all my PHP projects without immediately putting the secrets in a Hashicorp vault. The framework already comes with vault implementations that are based on configuration files or objects.

All code is publicly available at GitHub for reuse. The documentation there gives more code examples on how to use other vaults, e.g. when you want to start slowly and only manage your secrets in a configuration file.

Categories
Kubernetes

Kubernetes Service names in HELM templates

Based on my previous post, here comes a snippet that will correctly produce a full DNS name of a service in the cluster from the same namespace.

{{/*
Makes a full hostname from the given string if it's not one already or an IP address.
Attaches ".<namespace>.svc.cluster.local" to the end and includes the release name if required.
Please note that you need to call this template with (dict "Context" . "Value" "your-value")
*/}}
{{- define "prefix.serviceName" -}}
{{- if include "prefix.isIpAddress" .Value }}
    {{- print .Value }}
{{- else -}}
    {{- $parts := splitList "." .Value -}}
    {{- if gt (len $parts) 1 -}}
        {{- print .Value }}
    {{- else -}}
        {{- if eq .Context.Chart.Name .Context.Release.Name -}}
            {{- printf "%s.%s.svc.cluster.local" .Value .Context.Release.Namespace }}
        {{- else -}}
            {{- printf "%s-%s.%s.svc.cluster.local" .Context.Release.Name .Value .Context.Release.Namespace }}
        {{- end -}}

    {{- end -}}
{{- end -}}
{{- end -}}

Please note that using the template is a bit more cumbersome due to some Go language issues:

serviceName-anIpAddress:  {{ include "prefix.serviceName" (dict "Context" . "Value" "1.2.3.4") }}
serviceName-anIpAddress2: {{ include "prefix.serviceName" (dict "Context" . "Value" "1.0.3.4") }}
serviceName-NoIpAddress:  {{ include "prefix.serviceName" (dict "Context" . "Value" "1.2.3.4.5") }}
serviceName-NoIpAddress2: {{ include "prefix.serviceName" (dict "Context" . "Value" "hello") }}
serviceName-NoIpAddress3: {{ include "prefix.serviceName" (dict "Context" . "Value" "hello.svc") }}
serviceName-NoIpAddress4: {{ include "prefix.serviceName" (dict "Context" . "Value" "hello.svc.cluster.local") }}
serviceName-NoIpAddress5: {{ include "prefix.serviceName" (dict "Context" . "Value" "1") }}

The template needs access to the root context. So the dict function is used to pass the context and the actual, simple service name.

Feel free to adjust the function when you need another namespace as an argument.

Categories
Kubernetes

HELM template to detect IP address

I was in a need to detect whether the content of a variable is an IP address or not. I guess the function is not perfect, but it fulfills the basic need:

{{/*
Test if the given value is an IP address
*/}}
{{- define "prefix.isIpAddress" -}}
{{- $rc := . -}}
{{- $parts := splitList "." . -}}
{{- if eq (len $parts) 4 -}}
    {{- range $parts -}}
        {{- if and (not (atoi .)) (ne . "0") -}}
            {{- $rc = "" -}}
        {{- end -}}
    {{- end -}}
{{- else -}}
    {{- $rc = "" -}}
{{- end -}}
{{- print $rc }}
{{- end -}}

The function at least detects these values correctly:

{{ include "prefix.isIpAddress" "1.2.3.4" }}
{{ include "prefix.isIpAddress" "1.0.3.4" }}
{{ include "prefix.isIpAddress" "1.2.3.4.5" }}
{{ include "prefix.isIpAddress" "hello" }}
{{ include "prefix.isIpAddress" "hello.svc" }}
{{ include "prefix.isIpAddress" "hello.svc.tld.com" }}
Categories
Linux Miscellaneous Shell

Using sendmail with a Relay Host

It is useful when Docker containers can send e-mail to you in case there is an error condition that needs attention. Here is how to.

Install the sendmail package from your distribution and edit the file /etc/mail/sendmail.mc. Add this line to the end of it:

define('SMART_HOST', '<dns-name-of-your-relay')

Done! Just restart sendmail:

/etc/init.d/sendmail stop
/etc/init.d/sendmail start
Categories
Bugzilla for Java

What is the current status of Bugzilla?

I am honest. I am confused about the current status of Bugzilla. My personal environment does not list anyone or any project anymore that uses the most famous bug tracker software – for years. And this situation now seems to reflect in the current development activities in Bugzilla’s official GitHub repository. The last commit listed as of today was on February 1st, almost 3 months ago. Even worse: only 10 commits are logged within the last 12 months – on their active 5.2 branch. The last official release of Bugzilla was published even before that: on February 9th, 2019.

So it appears to me that the project dies a slow death – caused by the community that has migrated away to other bug trackers. And as sad as this seems, it is not a very surprising fact. Bugzilla was ignoring the needs of their users for too long. I remember a privacy issue to hide e-mail addresses from public that is still open (after many years). Bugzilla has become a dinosaur in a modern world. It is based on Perl, hard to setup on modern systems and misses many of the flexibility that other trackers integrated much faster.

Having said this, I am convinced now – more than ever – that my personal project B4J – Bugzilla for Java will not be maintained anymore. I will most likely fix bugs. But there will be no enhancement anymore.

Good-bye old friend, Bugzilla!

Categories
Bugzilla for Java CSV Java RS Library

Several new Software Releases available

As mentioned in previous post, most of my projects have been touched lately. So here is a list of the latest releases that you might find useful to integrate:

Also, there is a new project: EventBroker – a MQTT-alike event broker to enable losely coupled microservices being synchronized. It is written in Java and runs as docker container. Check it out when you look for an easy way to signal events to REST services in your distributed environment. Fun Feature: a special subscription topic for timer events that will enable you to implement cron jobs inside microservices – triggered by a REST call.