Category: Programming languages

All programming languages

Categories
Miscellaneous PHP

Accessing a Vault in PHP

I admit it. I was “hacked”. It was not really a hack, it was my mistake not to notice that one of my deployment logs was accessible for everyone to read and – exceptionally for a specific test performed back then – revealing configuration data including a password to an AWS service. The leakage has been fixed and the password changed. So, you don’t need to search for it anymore 🙂

However, the incident was reason enough for me to further secure my applications and introduce a vault for secret information. As most of my applications are based on PHP, I tried to find some ready-to-use code (and found a few). But all these libraries and SDKs are very heavyweight as they address much more use cases than just accessing a vault in order to fetch a secret read-only.

So I wrote a lightweight version of a PHP vault that not only accesses a Hashicorp Vault but also provides an abstract API so that my applications do not need to know what vault is being used or even where the secrets are stored.

So here is a code snippet that demonstrates how to use it with a Hashicorp Vault:

// Create configuration
$config = array(
	'type'   => 'hashicorp',
	'config' => array(
		'uri'      => 'https://127.0.0.1:8200/v1',
		'roleId'   => '<app-role-id>',
		'secretId' => '<secret-id>'
	)
);

// Create the vault instance
try {
	$vault = \Vault\VaultFactory::create($config);
} catch (\Vault\VaultException $e) {
	// Vault could not be created
}

With that vault, I can now access my secrets transparently within application code:

try {
	$mySecret = $vault->get('my/secret");
	$username = $mySecret->get('username');
	$password = $mySecret->get('password');
} catch (\Vault\VaultException $e) {
	// secret was not found
}

I even can further abstract this by not even knowing that there is a vault involved:

$callback = new \Vault\CredentialsProvider($vault, 'my/secret');

try {
	$username = $callback->getUsername();
	$password = $callback->getPassword();
} catch (\Vault\VaultException $e) {
	// Secret cannot be retrieved or does not exist
}

I can now integrate this pattern in all my PHP projects without immediately putting the secrets in a Hashicorp vault. The framework already comes with vault implementations that are based on configuration files or objects.

All code is publicly available at GitHub for reuse. The documentation there gives more code examples on how to use other vaults, e.g. when you want to start slowly and only manage your secrets in a configuration file.

Categories
Linux Miscellaneous Shell

Using sendmail with a Relay Host

It is useful when Docker containers can send e-mail to you in case there is an error condition that needs attention. Here is how to.

Install the sendmail package from your distribution and edit the file /etc/mail/sendmail.mc. Add this line to the end of it:

define('SMART_HOST', '<dns-name-of-your-relay')

Done! Just restart sendmail:

/etc/init.d/sendmail stop
/etc/init.d/sendmail start
Categories
Bugzilla for Java CSV Java RS Library

Several new Software Releases available

As mentioned in previous post, most of my projects have been touched lately. So here is a list of the latest releases that you might find useful to integrate:

Also, there is a new project: EventBroker – a MQTT-alike event broker to enable losely coupled microservices being synchronized. It is written in Java and runs as docker container. Check it out when you look for an easy way to signal events to REST services in your distributed environment. Fun Feature: a special subscription topic for timer events that will enable you to implement cron jobs inside microservices – triggered by a REST call.

Categories
Apache Linux Perl

How to automate LetsEncrypt

A new service is born: Let’s Encrypt. It offers free SSL certificates that you can use for web servers, email servers or whatever service you want to secure with TLS. This blog post presents my strategy to automate certificate creation and renewal. Please, install Let’s Encrypt on your web server box before you start to follow the presented strategy.

The key to success is to have Let’s Encrypt running without any further interaction. I use webroot authentication – which allows me to leave the productive web service up and running while the certificates are being issued or renewed. Therefore, I created a file named “myserver.ini” in folder /etc/letsencrypt. This configuration file contains all details that are required for the certification process;

1
2
3
4
5
6
7
8

rsa-key-size = 4096
authenticator = webroot
webroot-path = /path/to/webroot/
server = https://acme-v01.api.letsencrypt.org/directory
renew-by-default = True
agree-tos
email = <my-email-address>
domains = domain1.com, domain2.com

The second component of my strategy is the central piece: a script called “renewCertificates.pl”:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

#!/usr/bin/perl
 
my $DOMAINS = {
    'myserver' => {
        'configFile' => '/etc/letsencrypt/myserver.ini',
        'leSubDir'   => 'domain1.com',
        'certDir'    => '/var/www/domain1.com/certs',
    },
};
 
my $domain;
my $renewed = 0;
chdir ('/usr/local/letsencrypt');
foreach $domain (keys(%{$DOMAINS})) {
    print "INFO  - $domain - START\n";
    my $cmd = '/usr/local/scripts/checkCertExpiry.sh 30 '.$DOMAINS->{$domain}->{'certDir'}.'/cert.pem >/dev/null';
    my $rc = system($cmd);
    if ($rc) {
        $cmd = './letsencrypt-auto certonly --config '.$DOMAINS->{$domain}->{'configFile'}.' --renew-by-default';
        $rc = system($cmd);
        if (!$rc) {
            $cmd = 'cp /etc/letsencrypt/live/'.$DOMAINS->{$domain}->{'leSubDir'}.'/* '.$DOMAINS->{$domain}->{'certDir'}.'/';
            $rc = system($cmd);
            if ($rc) {
                print "ERROR - $domain - Cannot deploy\n";
            } else {
                print "INFO  - $domain - Deployed\n";
                $renewed = 1;
            }
        } else {
            print "ERROR - $domain - Cannot generate certificates\n";
        }
    } else {
        print "INFO  - $domain - Certificate does not expire within 30 days\n";
    }
    print "INFO  - $domain - END");
}
 
if ($renewed) {
   system("/etc/init.d/apache2 reload");
}
 
exit 0;

This scripts allows renewal of multiple certificates by supporting multiple configurations. Lines 3-9 describe these configurations. leSubDir (line 6) is the sub directory that Let’s Encrypt creates in the certification process. It is the name of the first domain specified in the configuration file, here: domain1.com. certDir (line 7) is the target path where the certificates will be deployed to.

A second script supports this procedure by telling whether a certificate will expire within a certain number of days (see line 16 above):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

#!/bin/bash
 
# First parameter specifies if certificate expire in the next X days
DAYS=$1
 
target=$2
if [ ! -f "$target" ]; then
    echo "Certificate does not exist (RC=2)"
    exit 2;
fi
 
openssl x509 -checkend $(( 86400 * $DAYS )) -enddate -in "$target" >/dev/null 2>&1
expiry=$?
if [ $expiry -eq 0 ]; then
    echo "Certificate will not expire (RC=0)"
    exit 0
else
    echo "Certificate will expire (RC=1)"
    exit 1
fi

This script returns 0 when the given certificate will not expire, otherwise it returns a non-0 value. The Perl script above will renew certificates 30 days before expiration only.

The last piece is the Apache configuration to be used on these domains:

1
2
3
4

    SSLEngine on
    SSLCertificateFile /var/www/domain1.com/certs/cert.pem
    SSLCertificateKeyFile /var/www/domain1.com/certs/privkey.pem
    SSLCertificateChainFile /var/www/domain1.com/certs/fullchain.pem

I run the central Perl script above daily and do not need to worry about certificates anymore 🙂

Categories
Linux Perl

Nonblocking sockets and Perl’s Net::Daemon

I was writing a Perl-based proxy for line-based SMTP protocol. The main reason doing this is because I receive unwanted e-mail bounces that are not filtered out by my SpamAssassin. The idea was to hook into the mail delivery chain and to collect e-mail addresses that I use. The proxy can later then filter out any bounce message that was not originated by myself.

I decided to use the Net::Daemon module which has a quite fancy interface. One just writes a single function which handles the client connection. As I didn’t want to learn every detail of SMTP protocol, I simply use non-blocking sockets. So whoever of the two parties wants to talk, it can do so and my proxy will just listen to the chat. The IO::Select documentation tells you to do this when you have multiple sockets to react on:

1
2
3
4
5
6
7
8
9
10
11
12

# Prepare selecting
$select = new IO::Select();
$select->add($socket1);
$select->add($socket2);
 
# Run until timed out / select socket
while (@CANREAD = $select->can_read(30)) {
    foreach $s (@CANREAD) {
        #... read the socket and do your stuff
    }
}
# Whenever there is a problem (like talk ended) the loop exits here

However, this code doesn’t work as expected. The can_read() function will not return an empty list when the sockets closed. It still returns any socket and the loop goes on forever. In fact, as we are in non-blocking mode, the script now eats up CPU time. 🙁

There are two solutions to it. The first is to check whether the given socket is still connected and then exit the loop:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# Prepare selecting
$select = new IO::Select();
$select->add($socket1);
$select->add($socket2);
 
# Run until timed out / select socket
while (@CANREAD = $select->can_read(30)) {
    foreach $s (@CANREAD) {
        if (!$s->connected()) {
            return;
        }
        #... It's safe now to read the socket and do your stuff
    }
}

The second and more clean method is just to remove the closed socket from the IO::Select object:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# Prepare selecting
$select = new IO::Select();
$select->add($socket1);
$select->add($socket2);
 
# Run until timed out / select socket
while (@CANREAD = $select->can_read(30)) {
    foreach $s (@CANREAD) {
        if (!$s->connected()) {
            $select->remove($s);
            next;
        }
        #... It's safe now to read the socket and do your stuff
    }
}

Then the selector runs empty and will exit the loop as well.

Categories
Java

Multi-threading: Pitfalls when initializing static variables

As a developer you often have to initialize static variables in a multi-threaded environment. The basic solution that most programmers apply is:

1
2
3
4
5
6
7
8
9

private static Object staticVar = null;
 
public static synchronized Object getStaticVar() {
    if (staticVar == null) {
        // initialize
        staticVar = ...
    }
    return staticVar;
}

This is a simple but expensive method. Each thread that needs the variable must synchronize with each other although the variable has long been initialized. So, usually the next step is to synchronize less:

1
2
3
4
5
6
7
8
9
10
11
12

private static Object SYNCHRONIZER = new Object();
private static Object staticVar = null;
 
public static Object getStaticVar() {
    if (staticVar == null) {
        synchronized (SYNCHRONIZER) {
            // initialize
            staticVar = ...
        }
    }
    return staticVar;
}

Yep. That does it, doesn’t it? The answer is: half! Imagine two simultaneous threads entering the method at the same time. If both will see staticVar being null then both will try to enter the synchronized block. And of course, both will initialize the variable nevertheless another thread did it before. So, we add another evaluation to ensure that only one thread will initialize:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

private static Object SYNCHRONIZER = new Object();
private static Object staticVar = null;
 
public static Object getStaticVar() {
    if (staticVar == null) {
        synchronized (SYNCHRONIZER) {
            if (staticVar == null) {
               // initialize
               staticVar = ...
            }
        }
    }
    return staticVar;
}

Most books end here but omit a very crucial part. The code works perfect as long as the initialization is a simple operation only. Let’s make the initialization a bit more sophisticated:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

private static Object SYNCHRONIZER = new Object();
private static List<String> staticVar = null;
 
public static List<String> getStaticVar() {
    if (staticVar == null) {
        synchronized (SYNCHRONIZER) {
            if (staticVar == null) {
               // initialize
               staticVar = new ArrayList<String>();
               staticVar.add("value 1");
               staticVar.add("value 2");
               staticVar.add("value 3");
            }
        }
    }
    return staticVar;
}

The first glance doesn’t reveal anything. But it happened several times in one of my own applications that two threads were not correctly synchronized. Occasionally, one thread found the list to be empty. What happened?

The magic is that staticVar is being set at the very first beginning of the synchronized block. Meanwhile another thread was entering the method and saw the variable not being null. It immediately started using the list although it was not yet initialized completely. The correct solution is therefore:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

private static Object SYNCHRONIZER = new Object();
private static List<String> staticVar = null;
 
public static List<String> getStaticVar() {
    if (staticVar == null) {
        synchronized (SYNCHRONIZER) {
            if (staticVar == null) {
               // initialize
               List<String> tmp = new ArrayList<String>();
               tmp.add("value 1");
               tmp.add("value 2");
               tmp.add("value 3");
               staticVar = tmp;
            }
        }
    }
    return staticVar;
}

For readability, we could refactor the method:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

private static Object SYNCHRONIZER = new Object();
private static List<String> staticVar = null;
 
public static List<String> getStaticVar() {
    if (staticVar == null) {
        synchronized (SYNCHRONIZER) {
            if (staticVar == null) {
               staticVar = createList();
            }
        }
    }
    return staticVar;
}
 
private static List<String> createList() {
   // initialize
   List<String> tmp = new ArrayList<String>();
   tmp.add("value 1");
   tmp.add("value 2");
   tmp.add("value 3");
   return tmp;
}
Categories
Eclipse Java RsBudget

RsBudget 2.0 released

logoIt’s done. My first official Eclipse/RCP application is out. RsBudget is an Expense Tracker for everyone. I’ve been developing it now for three years while constantly using it for private purposes. That’s how it grew to its functionality as it is today. I simply used these previous versions in order to feel and learn what’s been missing. Now it’s up to you to tell me what there is to be done next (a few tasks are already waiting ;)).

The application still misses some features, e.g. nice graphical statistics. But I don’t regard them as a must-have so far. They will be added with next versions, some will be available as commercial add-ons later.

The main features are:

  • General Expense Planning
  • Monthly Expense Planning, Tracking and Control
  • Categorization of expenses
  • Comparison of planned and actual values
  • Free text field for personal notes for each month
  • Forecasting of balances and profit/loss
  • Statistics and History
  • Export of transaction records to Excel and CSV
  • Multi-language support (English and German)
  • Online Help
  • Online Update

RsBudget runs on all major desktop platforms (Windows, MacOS, Linux) with Java 7 installed. Just download your version here!

Categories
CSV Eclipse Java RS Library

Eclipse RCP Common Feature launched

Good news for all Eclipse developers that want to use some of my projects in their own Eclipse/RCP projects. I bundled some modules and projects into a Luna Eclipse Feature Plug-In – called RCP Common Feature.

You will need the Update Site http://download.ralph-schuster.eu/rcp-updates/luna/releases/ to be added in your IDE and install the feature as you would do with every other Eclipse feature plug-in.

These are the modules and projects currently bundled:

Furthermore, there are three more plug-ins available specific to Eclipse/E4 UI and logging. The feature plug-in is released under LGPL V3 license (as all projects bundled in it).

 

Categories
Java RsBudget

Luna Update: TranslationService not in context

After migrating my Eclipse/E4 application to the latest release Luna, I noticed that the TranslationService is not present anymore within the E4Workbench.getServiceContext(). At least not at the PostContextCreate stage of the application. This is not a big deal, as you can easily create this yourself:

1
2
3
4
5
6
7
8
9
10
11
12

   private static TranslationService TRANSLATIONS = getTopContext().get(TranslationService.class);
 
   public static TranslationService getTranslationService() {
      if (TRANSLATIONS == null) {
         TRANSLATIONS = getTopContext().get(TranslationService.class);
         if (TRANSLATIONS == null) {
            TRANSLATIONS = ContextInjectionFactory.make(BundleTranslationProvider.class, getTopContext());
            getTopContext().set(TranslationService.class, TRANSLATIONS);
         }
      }
      return TRANSLATIONS;
   }
Categories
Java Tomcat

Solution to “Tomcat can’t stop [Abandoned connection cleanup thread]”

I am preparing a web service for one of my larger projects, using Tomcat, Hibernate and MySQL. While developing I stumbled across several Tomcat error messages that suggest some threads could not be stopped. Most of the threads could simply be vanished by calling Hibernate’s SessionFactory.close() method. However, a few threads seem to remain, e.g.

Jul 09, 2014 10:55:40 AM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/examples] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to 
unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Jul 09, 2014 10:55:40 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/examples] appears to have started a thread named [Abandoned connection cleanup 
thread] but has failed to stop it. This is very likely to create a memory leak.

and in fact, Tomcat stopped working after a few redeployments due to memory problems. Two main reasons:

  1. The MySQL AbandonedConnectionCleanupThread does not shutdown correctly.
  2. The MySQL J/Connector driver doesn’t unregister anymore from the DriverManager.

At least with J/Connector V5.1.31 the problem disappears by using this shutdown procedure (hooked in by a ServletContextListener):

1
2
3
4
5
6
7
8
9
10
11
12
13

   try {
      com.mysql.jdbc.AbandonedConnectionCleanupThread.shutdown();
   } catch (Throwable t) {}
   // This manually deregisters JDBC driver, which prevents Tomcat 7 from complaining about memory leaks
   Enumeration<java .sql.Driver> drivers = java.sql.DriverManager.getDrivers();
   while (drivers.hasMoreElements()) {
      java.sql.Driver driver = drivers.nextElement();
      try {
         java.sql.DriverManager.deregisterDriver(driver);
      } catch (Throwable t) {}
   }
   try { Thread.sleep(2000L); } catch (Exception e) {}
</java>

I still have to observe long-term effects and will keep you informed.