The Comfy Seat

An Unsupportable Path

Sat, 14 Jun 2025 00:00:00 +0000

Back in December I wrote about how we, the community behind the open source project called Puppet, were being forced into forking the project. In the time since then, OpenVox was born and has been diligently chugging along creating, among other things, builds based off of the last truly open versions of Puppet 7 & 8. We have also been trying to work with Perforce to ensure OpenVox remains compatible with Puppet Core and Puppet Enterprise. We’ve given them extensive feedback both in writing and via Zoom meetings on the EULA that is attached to Puppet Core to try to make it workable for the community, but they will not make the necessary changes so that it is tenable for Vox Pupuli to test our modules against Puppet Core. Additionally, they are steadfast in their commitment to keep Facter as a private repository going forward. Facter is a critical, load-bearing part of the Puppet technology stack. If they make private changes that we don’t anticipate or know to test for, it risks breaking the entire ecosystem. Similar to their promises about OSP, they said they’ll push changes back into the public repo and take PRs, but given that they have done this zero times in the last 7 months on the puppet repo, this does not seem likely.

Proxying Bitcoin Core and LND with Tailscale and Nginx

Sat, 08 Feb 2025 10:30:00 +0100

Recently I decided I wanted to run my own Bitcoin and Lightning node and I wanted it to be reachable on the public internet. I didn’t, however, want it to actually reside on the server that has the static public IPv4 and IPv6 addresses available. Thus, a reverse proxy was needed. This turned out to be a pretty simple thing to solve for thanks to the Nginx Stream Proxy module and Tailscale. Here’s the basic architecture:

The Community Is Forking Puppet

Mon, 16 Dec 2024 00:00:00 +0000

So, here’s an updated tl;dr on Puppet as an OpenSource project: a fork is absolutely coming now. There was a “town hall” today in which Perforce Software made it quite clear they are going to claim they want to work with the community while not actually doing so. As a result, those of us who have been following this closely reassembled, determined there was no longer hope of really working together, and that it was time to move forward accordingly. Perforce also made it clear no community thing could use the brand mark / trademark “Puppet” as well, thus the naming discussion has started (the linked GitHub org will be renamed as soon as a name is decided upon). Additionally, governance discussions are underway as well. More will be shared about this as soon as we have come to a decision. In the mean time, some thoughts on the subject are at https://github.com/OpenPuppetProject/planning/issues/7.

Routing Across AWS Subnets

Fri, 31 May 2024 17:00:00 -0400

This morning at work I was presented with an interesting question: why can’t two instances in AWS seem to talk to each other on their internal / private network interfaces? To answer this, let’s back up a second and let me show you what the architecture of the environment is. First, take a moment and look at the diagram below and observe not only how many layers there are, but also that this is a pretty simple setup with one VPC containing two instances that are spread across two Availability Zones:

Automated Plant Watering

Wed, 01 May 2024 15:00:00 -0400

Every spring, my wife and I get really excited about all the pretty plants and flowers that we can decorate our yard with. We also generally grow some vegetables and/or herbs. The problem with this is that we live in Georgia in the US and it gets freaking hot and humid here during the summer. The oppressive heat makes us not want to go outside to water the plants. Combine this with a little bit of traveling and you have a recipe for mostly dead plants during the latter part of the growing season. Well, this year we decided to not only acknowledge this reality, but to do something about it. You see, I’m a bit of a home automation nut and my wife knows it. She was shopping on Amazon and came across an inexpensive drip irrigation kit for gardens and decided to buy it for a raised flower bed we were already planning to setup this year. When it came in, she showed it to me and said “now I just need you to make it come on automatically.” As you might be able to guess, I was more than happy to take up that challenge. I spent a couple of days doing research to find a solution that fit within these self-imposed parameters:

Custom Weather Entity

Tue, 30 Apr 2024 15:00:00 -0400

In Home Assistant 2024.4, this note was in the “Backward-incompatible changes” section of the release announcement

The previously deprecated forecast attribute of weather entities, has now been removed. Use the weather.get_forecasts service to get the forecast data instead.

(@gjohansson-ST - #110761) (documentation)

I had a heck of a time finding docs on this, so I have compiled what I did here.

Base weather integration

I am using the Tomorrow.io integration to get forecasts, but what I have done should work with any weather provider.

PowerPress Authorization Flow

Wed, 12 Jul 2023 21:45:00 -0400

There’s a proposal in the podcast namespace for an authorization tag and I think PowerPress, an existing WordPress plugin that facilitates podcast hosting, can implement the same authorization flow as a podcast hosting provider. The proposed authorization flow is described to work something like this when a service wants to confirm a user owns a podcast:

Service reads an authorization url from the podcast’s rss feed
Service generates a onetime token
Service calls authorization url with token & rss feed as parameters
Website hosting authorization url verifies it’s the home / host of the feed
Website has user log in
Website presents user a confirmation page
If user confirms, website inserts the token into the <podcast:txt> tag
Website publishes updated rss feed
If website supports it, it sends a podping to notify watchers of the updated feed
Website sends a success response if the feed is updated and a failure response otherwise

User interaction is done at this point. The service still needs to see the updated feed, but that is beyond the bits I want to talk about here.

Crash Boom Bang: PoE & Lightning Strikes

Sat, 09 Apr 2022 09:00:00 -0400

A couple of days ago there was a severe storm that rolled through my area. Lots of thunder that literally rattled the walls of my house, lightning strikes near by, and high winds. At one point the power blinked out too. No big deal… or at least it wasn’t after I realized what was going on. The mystery I am referring to is that when the storm finished I realized my PoE Zigbee Coordinator wasn’t back up and running.

Starting Over with Home Assistant - Prep Time

Sun, 06 Mar 2022 18:00:00 -0500

The other day I posted this question to Redit:

I’m seriously considering redoing my Home Assistant setup from scratch now that I know what we actually use and what’s just cruft… anyone else done this?

As expected, there were a variety of opinions. Surprisingly though, there was an overwhelming consensus that redos after having used Home Assistant for a whe were a good thing.

After reading all the comments and thinking about things more, I’ve decided I want to download all my backups, copy out several bits of yaml, export some other settings, and then take the plunge. In my “Introducing My Home Assistant Setup” post I said I’d be following up with one that breaks down all my automations. This new decision is going to delay that a bit. Instead, I’m going to start by chronicling my journey through rebuilding my setup.

That time I forgot to create alerts for a leak sensor

Sun, 20 Feb 2022 16:00:00 -0500

I bought and setup a leak sensor… but forgot to have it alert me if it detected water 🤦‍♂️ Here’s what happened and my new alerting system.

Temperature sensing for Jupiter Garage

Sun, 30 Jan 2022 22:34:00 -0500

The other day I was listening to Linux Unplugged 441 and heard Chris mention how he wished he had a way to track the temperature in the garage where the server is.

I decided that this was something I could help with, so I hit him up on Twitter:

Hey @ChrisLAS - I was listening to LUP today and heard you might need to monitor temperatures in your garage… DM me if you want a cloudless WiFi monitor based on ESPHome.

— Technical Issues (@technicalissues) January 19, 2022

We chatted a tad via direct messages and then I built this:

It’s modeled after one I have in my own garage with a couple of small modifications to suite his use case better. The setup is made up of:

a 1/2 size prototyping board
a D1 Mini (aka a small ESP8266) microcontroller
a BME280 temperature, pressure, and humidity sensor
a 3 port spring terminal block
a Dallas 1-wire temperature sensor in a waterproof housing with a cable attached

The idea is that Chris will be able to mount this on, or near, the new server cabinet with the microcontroller at the top so that the heat it generates rises above the onboard sensor. The onboard sensor (the purple part) will allow him to monitor the temperature, barometric pressure, and humidity in the garage while the corded sensor will allow for monitoring the temperature inside the server rack.

Introducing My Home Assistant Setup

Sun, 09 Jan 2022 23:00:00 -0500

A year ago today (January 9th, 2021) I deployed what I consider my first production-grade instance of Home Assistant and couldn’t be happier. It is an amazingly powerful tool that is 100% free and open source. One of Home Assistant’s key features is the fact that it takes a local first approach to everything. By that I mean that every aspect of the project makes a concerted effort to not rely on the internet or cloud services unless they are absolutely required, such as when integrating with a vendor who does not have a local api (or won’t provide access to it to the community). This means that if the internet is out I can still control the vast majority of the devices connected to Home Assistant using either the web interface or the app on my phone… and push notifications from Home Assistant to my phone will continue to work too.

Epomaker GK68XS: A great keyboard, horrific order fulfillment

Fri, 13 Nov 2020 20:10:00 -0500

3 months, 35 emails, and many Facebook Messenger messages later, I finally have all the things I paid for during the GK68XS Kickstarter. I originally ordered 3 keyboards: one of each color combination of the plastic framed wireless models. Sadly, delivery was nothing like what was advertised. Here’s the journey I had to take simply to get what I paid for.

On Aug. 14th I got the first one.
On Aug. 17th I was told by support that the shipping company missed my other ones and that both would be sent out right away.
On Sept. 2nd I got a delivery but it only contained one keyboard. On Sept. 7th support claimed ignorance and asked for more information.
On Sept. 9th they apologized and offered to refund me or fast ship the missing one. I opted for the latter as I still wanted the keyboard.
On Sept. 14th they said I’d have it in had in a week.
Two weeks later I followed up because I had not gotten anything.
On Oct. 3rd they again claimed ignorance. I again provided all the requested info and on both Oct. 9 and Oct. 17 I asked for an update.
On Oct. 17th I reached out via Facebook Messenger and asked if they could help. They said they would escalate my issue.
On Oct. 21st I posted in Messenger that I still had not heard anything.
On Oct. 22nd I got a response in Messenger that they’d pushed the team and I’d get a response the same day.
On Oct. 23rd I finally got a response back that the missing keyboard would be back in stock “in a few weeks.” I replied to support and reiterated a previous message that I needed to have the keyboard before December so that I could get it sent out in time for Christmas. They offered to send a white one immediately, which I accepted.
On Oct. 28th I followed up asking if there was a tracking number yet but did not get a response so I tried again on Nov. 1st.
On Nov. 1st I posted in Messenger again, this time referring to the conversation from Oct. 23rd. I let them know I had not heard anything at all yet (similar to what I sent support via email). The person working Messenger was quite nice and seemed to try and help.
On Nov. 2nd I got a message in Messenger saying a response had been sent to my email. I told them I did indeed get a message and that it said they’d send me tracking info in 1-2 working days. The person in Messenger asked I keep them updated on the status of things, which I was happy to do.
On Nov. 6th I followed up again via Messenger and email as I had not heard anything back.
On Nov. 8th I got a message in Messenger saying my order should be on the way in 1-2 days. I let them know I appreciated the help but that I’d already heard that same timeline six days prior. They said they’d keep an eye out for it and later that day I finally got the tracking information.
Today, Nov. 13th I got the keyboard, just as I had ordered it, which was both good and bad. The good is I got exactly what I was after. The bad is that support apparently flat out lied to me when they said they’d ship one immediately if I accepted a different color.

I am sharing all this in the open in hopes that it will encourage Epomaker to make process changes so that this doesn’t happen again. My wife and I both love our GK68XS keyboards and have even backed the GK96S, though we did so separately in hopes of avoiding a repeat of the fiasco documented here.

OpenTelemetry Part 3: v0.6.0 gems and VMPooler

Mon, 05 Oct 2020 18:20:00 -0400

For part three of my journey in using OpenTelemetry (Otel) with Sinatra I am upgrading to the 0.6.0 release of the OTel gems to get many new features, adding instrumentation to VMPooler, and learning what not to do. Part 3 also includes opening several issues and making my first code contribution to opentelemetry-ruby. Lastly, I will be sharing some more complete code examples showing how all the bits are configured.

OpenTelemetry Part 2: Redoing Instrumentation

Sun, 06 Sep 2020 21:30:00 -0400

For part two of my journey in using OpenTelemetry (Otel) with Sinatra I am replacing my Lightstep instrumentation with the OTel version. Besides updating the instrumentation, I am also deploying production instances of an OTel Collector and Jaeger. The goal of part 2 is to have my first three applications shipping traces to both a local Jaeger instance and to Lightstep in both test and production and to have Jaeger included in the Docker compose workflows used during development.

Burnout Sucks

Tue, 18 Aug 2020 23:05:00 -0400

My favorite personal project is an application called PiWeatherRock… or it was before I dove in head-first working to update it and create a community for it’s users. Tons of enthusiasm morphed into something else and, before I realized what was happening, I didn’t even want to touch my computer after work. Months have passed since I even opened anything related to the project and, as best as I can tell, this is that thing I’ve seen others talk about called “burnout.”

OpenTelemetry Part 1: Sinatra

Fri, 07 Aug 2020 18:27:00 -0400

OpenTelemetry (aka OTel) is becoming the standard for distributed tracing. This is the first in a multi-part series where I will document my trials, tribulations, and successes along the road of using OTel to instrument multiple applications. The first few are all ruby applications and some that I hope to be able to do later are written in Java. My goal is to instrument the applications using one or more standards-compliant libraries and then send the spans to an OTel collector. The OTel collector will then send them on to one or more backends such as Jaeger, Lightstep, and/or Datadog.

Fixing powerline in homebrew's vim

Tue, 12 May 2020 00:00:00 +0000

For the last few days I have been trying to figure out why, all of a sudden, powerline has stopped working in vim. Whenever I start vim I get this error:

╔ ☕️ gene:~
╚ᐅ vim
Error detected while processing /Users/gene.liverman/.vimrc_os_specific:
line 2:
Traceback (most recent call last):
 File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'powerline'
line 3:
Traceback (most recent call last):
 File "<string>", line 1, in <module>
NameError: name 'powerline_setup' is not defined
line 4:
Traceback (most recent call last):
 File "<string>", line 1, in <module>
NameError: name 'powerline_setup' is not defined
Press ENTER or type command to continue

The part that serious confused me about this was that this worked fine:

Puppet Camping in place: East meets West

Tue, 28 Apr 2020 00:00:00 +0000

I pitched a tent at Puppet Camp a couple of times before joining the company and have to say that last week’s event was superb, and it more than lived up to the standards set in ye olden times. It was great to hang out (virtually) with so many community members! There were some faces, or should I say Slack handles, that I knew, but many more I got to meet and chat with for the first time. The work these gurus are doing in their day jobs is just amazing! The best part is that a lot of what was demoed and talked about is directly applicable to the work that I and the other attendees do. Below are some of my takeaways from the event along with a boatload of reference material from the presenters and people in Slack.

Church and COVID-19

Mon, 13 Apr 2020 00:00:00 +0000

My church has been streaming our service for several years and has been live on a local FM station since before I became a member. I setup streaming initially and have been involved in all kinds of services. During that time I’ve run into just about every type of issue or challenge a sound or audio video tech in a midsize church can… or so I thought.

De-forking a Puppet Module

Wed, 26 Feb 2020 13:39:00 -0500

A couple of years ago, the team I’m on forked a Puppet module called “mrepo” that is used for creating and managing RPM-based repository mirrors. We recently had an issue arise with using the module, and I happened to notice that the upstream of our fork is now Vox Pupuli and that they had made several improvements that we could benefit from. Those changes, combined with knowing the quality work Vox Pupuli does on all of their modules, made me wonder what it would take to get off our fork and back on to the upstream version.

Fixing Vagrant's box index

Thu, 14 Nov 2019 10:36:00 -0500

I use Vagrant a lot and sometimes things on my laptop get moved around or deleted by means other than vagrant destroy. The problem with this is that when I later run vagrant global-status it will show me things that don’t actually exist anymore. Today I finally got tired of this and figured out how to fix it with minimal pain.

Reclaiming Your Accounts

Fri, 02 Aug 2019 00:00:00 +0000

This post is a quick reference guide for what I recommend people do when someone has gotten into their accounts on various platforms.

Using the Multi-Resource Declaration and Defined Types to Simplify Manifests

Thu, 09 May 2019 00:00:00 +0000

Sometimes it seems you just keep repeating the same block of code with only one or two lines changed. Sometimes a single thing you need to do more than once is made up of the same two or three resources. These two scenarios are ones that I experience fairly often.

They are also ones I regularly observe when doing code reviews for others. I am often met with interest and a response that is along the lines of “I didn’t know you could do that” when I mention the idea of simplifying the code I am reviewing by using a multi-resource declaration or a defined type. This post will introduce you to multi-resource declarations and defined types and then walk you through a real-world example of putting them to use to configure load balancing of Puppet Enterprise’s services.

Breaking Up a Large Pull Request

Wed, 17 Oct 2018 00:00:00 +0000

Ever finished up all the changes for a pull request on GitHub and realized it was just too big to review easily or to reason about what’s going on? I had just this issue recently. The solution: create multiple patches that each contain a subset of the changes and use them to generate more manageable pull requests.

For this guide let’s make a few assumptions:

the default branch in your git repo is master
the branch containing the big PR is called my_massive_change
the changes encompass many different files, many of which are in subfolders

To expedite things I am also using hub to interact with GitHub.

My journey to securing sensitive data in Puppet code

Wed, 15 Aug 2018 00:00:00 +0000

Dealing with secrets and sensitive data in Puppet is daunting, right? Nope, not at all. Let me show you how to do it. I’ve wrapped my head around the options available and want to share my journey in hopes of saving you from a few trials and tribulations. Just interested in the end result? Feel free to scroll down to the last section fittingly entitled The final product.

Dual Booting macOS High Sierra and Linux Mint

Tue, 06 Feb 2018 00:00:00 +0000

This is a step-by-step walkthrough for dual booting a MacBook Pro (Mid-2015 aka MacBookPro11,5) that already has macOS High Sierra on it with Linux Mint. The hard drive is formatted APFS and has File Vault turned on.

Before beginning I suggest reading this entire post to see how involved it is or, at a minimum, read the known issues at the bottom.

GitLab CI and Chocolatey Server

Wed, 09 Aug 2017 00:00:00 +0000

If you are not familiar with chocolatey, its an awesome package manager, like apt or yum, for Windows. You can also host your own internal chocolatey feed and there is even a Puppet module to build it for you. This can be especially useful for machines that cannot reach out to the internet to perform the installations. Chocolatey even provides a step-by-step guide on how to internalize packages, this can be a lot of manual steps from building packages, to getting them up to the Chocolatey server, keeping history, and maintaining when there are package updates.

This is why I created a quick solution for maintaining your package history in Git and using GitLab CI to automate building and deploying packages to your internal Chocolatey server. This guide assumes you have an internal GitLab instance, an internal Chocolatey server, and a Windows based GitLab Runner with powershell execution. Documentation here on GitLab Runners

The Road to Puppet 5

Wed, 09 Aug 2017 00:00:00 +0000

Not long ago Puppet released version 5 to the open source world so, naturally, it was time to start updating all my projects to be compatible with it. The first stop along the way was at the house of Vagrant… only, there’s a catch: it’s been relocated. That’s right, my Vagrant boxes got a shiny new home at app.vagrantup.com/genebean as part of some restructuring done by HashiCorp. After getting my new door key (aka account) I went next door to visit my friend Packer. I hung out in his workshop massaging my templates with the goal of updating and simplifying the boxes I build. The end result included combining all the versions of RVM into a single build and creating a new box for Puppet 5. Now, if you’ve ever hung around Mr. Packer for any length of time then you know he loves to create multiple versions of anything he helps assemble. Seeing as I want him to be happy I figured I should oblige and let him create some Docker images too.

Add Puppetfile Validation to Testing

Mon, 31 Jul 2017 00:00:00 +0000

This is a quick post about how to add validation of your Puppetfile, primarily if you are using the control-repo and r10k for deploying Puppet environments. This came about because I found myself entering incorrect syntax into this file on more than a one occasion. Additionally, there are no indications of any problem, even when importing environments in Foreman, so the only way to find out is by manually running r10k from the command line on the Puppet Server.

Python PIP Issues after Homebrew upgrade

Mon, 24 Jul 2017 00:00:00 +0000

This is just a quick note for anyone else out there who recently ran brew update && brew upgrade and then found that Python no longer worked as expected. Here are the important points:

The issue is that Homebrew introduced a breaking change and did a crappy job of documenting it.
The fix is to prefix your path with /usr/local/opt/python/libexec/bin
More details can be found at https://github.com/Homebrew/homebrew-core/issues/15746

For me, the fix was to add this to my .zshrc file:

Upgrade to Puppet 5

Sat, 15 Jul 2017 00:00:00 +0000

Today I successfully upgraded our Puppet Master from Puppet 4.x (puppetserver 2.7.2) to Puppet 5 (puppetserver 5.0.0). It was wildly helpful to go through the entire upgrade process and perform LOTS of testing and troubleshooting with the Vagrant Puppet Environmet, which is basically an exact replica of my production environment. This is an all-in-one Open Source Puppet setup and, once the next release is out, I would highly recommend for testing!

Automatically Generate GoAccess stats

Fri, 16 Jun 2017 00:00:00 +0000

I’ve been using GoAccess to look at my logs for a while now. The other day I decided I wanted be able to look at these stats for the different sites on my web server in a variety of ways including:

all data from all sites combined
all data on a per-site basis
daily stats from each site kept for a week

The thing with wanting daily stats is it helps if they are created in a way that only covers that day. That sounds simple, but the logrotate generally runs around around 3am. So what’s the solution? Cron. To be more exact, run logrotate from cron and generate stats while you’re at it.

# Puppet Name: rotate nginx logs
0 0 * * * /root/updatestats.sh

Now, if you are going to run logrotate from cron you’d better turn of the original one. Here’s how I did that:

$ cat /etc/logrotate.d/nginx
# this is managed by a cron job.
# the script that would normally be here is at /root/nginx-logroate.conf

You have to do something like this instead of just deleting the file because otherwise the next time there is an update to Nginx (or whatever web server you are running) it will just recreate the file. The reason this works is that installers generally don’t clobber existing files. Below is the referenced replacement:

$ cat /root/nginx-logroate.conf
# use date as a suffix of the rotated file
dateext

/var/log/nginx/*log {
 create 0644 nginx nginx
 daily
 rotate 10
 missingok
 notifempty
 compress
 sharedscripts
 postrotate
 /bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null` 2>/dev/null || true
 endscript
}

Node.js, CentOS7, and libhttp_parser.so.2

Wed, 07 Jun 2017 00:00:00 +0000

Don’t you just love it when package maintainers break you blog? Yeah, me too. Tonight I went to post an article (no, not this one) and found my site to be down. When I went to start it back up I got this:

[ghost ~]$ /usr/bin/npm start --production
node: error while loading shared libraries: libhttp_parser.so.2: cannot open shared object file: No such file or directory

As it turns out, the maintainer of the nodejs-6.10.3-1.el7.x86_64 package added this to their changelog:

Saying Goodbye to dd-wrt

Tue, 23 May 2017 00:00:00 +0000

Tonight I had to wave a sad goodbye to dd-wrt and revert back to a stock firmware. This travesty is because the dd-wrt firmware doesn’t support the hardware NAT function on the TP-Link Archer C7 v2 which resulted in losing over two third of my bandwidth. Being that my ISP provides me with a full gigabit upstream and down that equated to a getting only 200-300 megs each way instead of over 900 on a wired connection. On wireless things were even worse: I was getting 100-200 megs vs over 500.

Zabbix 3.2 is WAY more efficient!

Wed, 10 May 2017 00:00:00 +0000

Recently our Oracle DBA hit me up and said that all of a sudden some of his servers were showing a load average of 0.00, 0.00, 0.00. To diagnose this I started looking at our Zabbix dashboard to see when the load dropped off. I noticed it was on March the 3rd so I checked a second host and found that it also dropped off on the same day… interesting.

Solving a WordPress 'http error'

Sat, 25 Mar 2017 00:00:00 +0000

Tonight we were trying to make the first post on my wife’s blog and ran smack into a “Http error” message. When I looked in the console of my web browser I found an error 413 (Request Entity Too Large) message. After a bit of Googling it turns out that Nginx was the culprit. Apparently the default value of client_max_body_size is 1 meg. As I am sure you can imagine, most images grabbed with a camera phone are larger than that now.

Exploring Grafana

Sun, 05 Mar 2017 00:00:00 +0000

This weekend I decided to check out Grafana. My first test for it was setting up the Zabbix backend. This went much better than I had expected so I started looking at what other data I could pull in. It turns out that Grafana may well be a great tool for centralizing data and metrics from disparate sources. The consensus on the interwebs, as best as I can tell, is that InfluxDB is the backend I should store my metrics in so I’m going try that next. Once InfluxDB is setup my plan is to try out some one-off inputs to it such as:

SSL, Name-based Virtual Hosts, and Let's Encrypt

Mon, 27 Feb 2017 00:00:00 +0000

When I started switching everything I could over to https-only I was under the impression that the only option was to tie each host to a single certificate unless I wanted to shell out the big bucks for a wildcard cert. This also meant one host per IP address if I wanted to use the standard port 443. That was two or three years ago. Just a few months ago I learned that SAN certificates were recognized by all the major browsers and started taking advantage of them to reduce the burden of needing two certs to cover things like example.com and www.example.com. In my mind this still required two IP addresses though (one per domain). All this changed tonight when I decided on a whim to see if you could setup Nginx to recognize name-based virtual hosts that were all tied to a single SAN certificate on a single IP. As it turns out, this works just fine (who knew?!?). And, as the icing on the cake, Let’s Encrypt supports up to 100 SAN entries per certificate!

Updating to Puppet 4 Part 3

Tue, 07 Feb 2017 00:00:00 +0000

Hooked and Proxied

When I left off last time a webhook receiver was needed… well, its finished and published to Puppet Forge as genebean/puppetmaster_webhook. The module creates a custom Sinatra application and installs it along with RVM. The end result is that you can post messages from GitHub or GitLab and have it deploy the corresponding repository’s branch or environment.

While I was setting all this up I also decided to front everything with HAProxy so that I could simulate being behind a load balancer immediately and to prepare for the eventual high availability setup that is my end goal. As of today I have it so that all nodes talk to the Puppet master by way of the proxy. Foreman and my webhook receiver are also being fronted by the proxy.

Updating to Puppet 4 Part 2

Sat, 28 Jan 2017 00:00:00 +0000

Four repos become one…

When I last created a full Puppet environment “Roles & Profiles” were the new way to do things. Gary Larizza was posting articles that talked all about how each of these should be in their own repository and how we should use r10k and hiera and how each of them should also have a repo. What that meant was that concerns were well separated but it also made for a rather complex environment.

Updating to Puppet 4 Part 1

Wed, 25 Jan 2017 00:00:00 +0000

Better than two years ago I created a multi-node Vagrant setup based around a three node Puppet environment with boxes for:

Foreman acting as a CA, report viewer, and ENC
PuppetDB
A Puppet master with r10k

The environment also has a client node to test against.

At the time I built all this Puppet 3.x was as the latest version. Fast forward to January 2017 and Puppet 3 has been end-of-life’d, Puppet is on version 4.8, Puppet Server is on version 2.7, and control repos are a thing so I figured it was time to update my stuff.

The Ghost Lives

Mon, 16 Jan 2017 00:00:00 +0000

I decided it was time to stop procrastinating and actually update the live version of my blog to the one I have been working on for a while. So, here it is: The Comfy Seat: Ghost edition.

The new site is built on v0.11.4 of Ghost from https://github.com/TryGhost/Ghost and fronted by Nginx. Some additional improvements will be coming soon as will more posts (I hope).

ELK Stack v2 (and a correction)

Mon, 20 Jul 2015 00:00:00 +0000

I’ve learned a lot since my last post. One of those things is that I was wrong… setting up Logstash on your Redis nodes isn’t such a bad idea. Another thing that I have learned is that fluentd / td-agent is not as great as I thought it was. My revised plan as depicted in the updated design below is to use Logstash Forwarder on my non-Windows nodes and send that to a Logstash instance that does nothing but stick things into a local Redis instance. Doing this also eliminates the need for my custom receiver named Sawyer. The last change noted below is that I have upped my number of Elasticsearch data nodes and Logstash indexers to 3 each. This was a direct result of load. I also like the improved distribution of shards by having more than 2 nodes in a 5×2 shard setup.

ELK Stack Design

Fri, 19 Jun 2015 00:00:00 +0000

I’ve been working on a new logging system based around Elasticsearch, Logstash, and Kibana. One of my biggest challenges was that all the recommended designs I found said that logs should go from a shipper to Redis. The problems with this are twofold:

Logstash doesn’t seem like a good fit for Windows. The biggest issues are that it relies on Java which isn’t something that is very sellable to any Windows admin that I know. The other is that it simply didn’t work reliably in my testing. The 1.4.x series had performance issues and the copy of 1.5.1 I just tried on Windows 7 is throwing

Windows Event Log error: Invoke of: NextEvent<br></br>
Source: SWbemEventSource<br></br>
Description: Timed out

errors under even the simplest of tests. Unlike other tools it also requires specifying each Event Log that you want to monitor individually as opposed to being able to just grab them all. 2. Not everything can have an agent on it which means that I needed a way to pipe syslog into Redis

Windows 7 x64 and Underscore-CLI

Sun, 23 Nov 2014 00:00:00 +0000

Underscore-CLI is a great utility for working with JSON data. Below are the steps it took to get it running on my Windows 7 laptop:

Install node.js1. Node adds a trailing \ to it’s path… to actually use it you must remove this as Windows does not want it to be there
Install Python
Add Python to your path (something like C:\Python27)
Underscore-CLI uses node-gyp… to get that to work on Windows 7 x64 you have to follow their guide at https://github.com/TooTallNate/node-gyp/wiki/Visual-Studio-2010-Setup. Be sure to pay attention to the part about utilizing the Windows 7 SDK command prompt.

Hyper-V, CentOS 6.5 kernel panic, and 7 long hours

Wed, 28 May 2014 00:00:00 +0000

In hopes of it helping someone else not spend hours of work like I just did here is my lesson-learned from my first day of using Windows Server 2012 r2 Hyper-V.

Vagrant, Fusion, & DHCP Oddities

Sun, 11 May 2014 00:00:00 +0000

I’ve had some random weirdness that I thought was related to Vagrant’s VMware Fusion provider until I turned on debugging tonight. As it turns out, Fusion had decided at some point in the past to start storing it’s DHCP leases in vmnet-dhcpd-vmnet8.leases~ instead of vmnet-dhcpd-vmnet8.leases. The same was true for vmnet1 too. After quitting Fusion and running ‘sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-cli –stop’ I removed vmnet-dhcpd-vmnet* so that all the leases would be reset. After that I reran ‘vagrant up’ and (finally) things worked as expected.

What's Missing from GitLab?

Fri, 02 May 2014 00:00:00 +0000

The other day I was asked what GitLab was missing and I realized that, really, it’s not much. The single biggest thing to me is the inability to create new projects and interact with existing ones from a remote shell session a la gh / GitHub CLI. Other than that it really comes down to polish and aesthetics. Below is my $0.02 based on interacting with GitLab as a person who runs a server and as an end user.

Configuration Management Part 3: Vagrant & Packer

Sun, 27 Apr 2014 00:00:00 +0000

To facilitate developing my Puppet code, the Pro Puppet book suggests using Vagrant. Seeing as I’ve been meaning to get around to learning it for a while I decided now was the time to finally do so. The only problem is that, being a responsibly paranoid SysAdmin, I was never a fan of using a base for my work that I didn’t know the contents of. I also never liked the idea of basing my work off of something I didn’t understand (a Vagrant box) or that could go away at anytime.

Configuration Management Part 2: puppetlabs-apache & puppet-lint

Wed, 23 Apr 2014 00:00:00 +0000

Today was a good day. I installed puppet-lint and ran it against a custom module I’m writing for my first node and found lots of issues that it was kind enough to tell me exactly how to resolve. I then got down to using my first module from Puppet Forge: puppetlabs-apache Installing it was a piece of cake but understanding how to use it took a bit of trial and error.

Configuration Management Part 1: The Restart

Tue, 22 Apr 2014 00:00:00 +0000

As mentioned in my last post, I’ve decided to start over on my journey to doing configuration management in an environment where we treat our infrastructure as code. Today I kicked things off by setting up a new Puppet Master on CentOS 6.5. Once my usual setup was applied to the system via a PXE boot & Kickstart installed Git and the puppetmaster package and was off.

Version Control
One of my main goals is to track everything in Git so my first task was to change the group ownership of /etc/puppet to my puppetadmins group and give them write access. Then I needed to initialize a repo in that directly, tell Git that it’s a shared repository so other admins can work in it too, and tell Git to ignore the modules folder. I then applied the group permissions to everything inside the folder, did setgid on modules & manifests, and lastly I did a setfacl on modules & manifests so that us admins would retain rwx on all files and folders. Lastly I cloned my first module from our GitLab instance into a folder under modules.

Foreman: Too Much Voodoo

Mon, 21 Apr 2014 00:00:00 +0000

I finally got around to setting up Foreman at work and managing my first node with it. After digging around I found that I felt really boxed in using this setup because so much of the work is done behind the scenes in some magical way. One of my main goals is to facilitate the concept of infrastructure as code and, like my code, track changes via git and store them in our GitLab instance. The Foreman, as best I can tell, takes and hides everything it does inside a database which prevents me from being able to apply any version control to it’s settings. This is an unforeseen and unfortunate reality because the developers have made a really good looking product that can do a lot of really cool things. For me though, this is too much voodoo at this early of a stage of us doing configuration management and I think I’m going to back out my install and start over with a different approach that defines nodes in pain text .pp files. I’m sure I’ll take advantage of pulling in data from some external source like Hiera and / or other systems we have to help make decisions dynamically but I don’t think I want the configs themselves in a db… who knows; guess I’ll try it out and see.

Puppet, Factor, Dashboard, MCollective, Hiera, Foreman... Where does it end?

Sun, 27 Oct 2013 00:00:00 +0000

I want to setup a system based around the open source Puppet software stack… I am just not sure what the stack contains. My goal is a system that is holistic where Puppet, VMware vSphere, monitoring, revision control, and continuous integration all play well together. I also want to bring Vagrant and Packer into the mix for development environments. I’ve gathered so far that all the stuff listed in the title will be useful but what else is needed? Please comment below and I’ll also post again as I find answers.

Automatically Starting and Stopping Oracle Fusion Middleware on Red Hat 5

Sat, 28 Sep 2013 00:00:00 +0000

At work we utilize Oracle Fusion Middleware on Red Hat 5.8. As the primary systems administrator for the servers running FMW, I always found it to be a real pain that something as simple as a reboot required me to involve the app admin. Instead of just being annoyed I got with that app admin to learn how the services were started and stopped and then wrote a set of SysV init scripts to automate that process. These scripts seem to be reliable now so I have released the code on BitBucket at https://bitbucket.org/genebean/oracle-fmw-sysv-init. These scripts cover all the components used when running Ellucian’s Internet Native Banner and Self Service Banner.

Vagrant, Veewee, & Me (part 1)

Tue, 21 May 2013 00:00:00 +0000

I have toyed with the idea of diving into Vagrant for a while now and, tonight, decided it was time. I decided to be different and RTFM… this left me with two big questions: where can I get “boxes” from and how can I easily make my own? After a little Googling I discovered that Puppet Labs provides a small library of the boxes they use internally. On their page I also found the answer to my second question of how to make my own: Veewee. It seems I have a bit of setup to do before I can start using Veewee but I think it will be worth it. My plan is to bring up a base CentOS 6.4 x86_64 box and then make a Vagrantfile that uses Puppet to configure it for building RPM’s in. Ideally, I will start including this Vagrantfile with the source of any RPM I publish so that building a new one is easy-peasy.

Gentoo & MySQL Binary Logging

Sun, 07 Apr 2013 00:00:00 +0000

So, I learned today that the root cause of my site issues was that Gentoo apparently decided to enable binary logging by default yet did not have a max size or may days set in my.cnf like other distros do so, as a result, I had MANY gigabytes of logs which filled up /. Lesson learned. Thanks to Zabbix I knew about the issue straight away and was able to minimize my downtime.

More Monitoring... NOW!

Mon, 01 Apr 2013 00:00:00 +0000

As some may have noticed, I have had a few site issues lately. Those should be in the past now. As a result of these issues, I am now actively monitoring this site & it’s server via Zabbix. If I can compile all the pieces of the different guides I used to set it up in between other thing I’ll post it here. Until then, here’s the short version:

ClearOS @ home + v2.0 packages from EPEL = Zabbix server

Using Oracle Wallet with Wildcard Certificates

Tue, 05 Feb 2013 00:00:00 +0000

Do you have to use Oracle Wallet as part of Fusion Middleware? Do you also have a wildcard SSL certificate? If so, then this tutorial is for you. This tutorial is the result of trying to make a new install of Internet Native Banner (INB) play nicely with our wildcard certs so that if we change the hostname of the system or clone the virtual machine SSL does not break or require adjustment.

Building Android

Wed, 18 Jul 2012 20:45:45 +0000

I wonder how hard it is going to be to pick pieces I like out of one or more Android ROMs and add them to my own AOSP-based ROM. To start finding out, I am currently downloading both the AOSP repo & the CyanogenMod repo. Then I guess it will be time to dig into both via Eclipse. This should be interesting.

Putting this site to use...

Tue, 17 Jul 2012 00:00:00 +0000

So… I was perusing RootzWiki looking for the next great Jellybean ROM for my phone and noticed that frequently dev’s need a place to mirror things. With that in mind, I think I will look into a good way to mirror some files and also to provide torrents for any mirrored file. This should be interesting…

Uptimed Site Now Live

Tue, 17 Jul 2012 00:00:00 +0000

Just a quick update to let you know that uptimed.technicalissues.us is its own site now.

Sitting down for the first time…

Sat, 12 May 2012 00:00:00 +0000

Welcome to The Comfy Seat! This site will serve as the base for all my domain names and hosting needs. Other domains that will share this site from Day 1 are:

Gene Liverman

Mon, 01 Jan 0001 00:00:00 +0000

I’m a bicycle riding Eagle Scout tech geek. I’m also a genealogy nut who drinks too much coffee.

Jake Spain

Mon, 01 Jan 0001 00:00:00 +0000