Stephen Gream's Site

Rebuilding the homelab

Tue, 14 Apr 2026 00:00:00 +0000

Again we move forward the ratchet of mediocrity

About Me

Mon, 19 Jan 2026 00:00:00 +0000

G’day. My name’s Stephen and I’m here to convince you I know things. I work as a software engineer in Melbourne, Australia and mostly specialise in AWS native applications and deployments.

Here’s a summary of my previous professional experience:

ANZ Bank, 2021-2025: Tech lead for the Gearbox team. We worked on orchestrator loan applications, importing account performance data from the ANZ data lake, and providing a integration gateway for bureau calls. Core technologies were Kubernetes, GCP, Temporal and Golang.
TSA Group, 2019-2021: Working within the TSA contact centre business, I was part of the team that first brought their new AWS Connect based solution to market. My role at TSA spanned across the whole stack. As such I had exposure to everything from the React frontend, to the Go backend, to the deployment and configuration pipelines.
Versent, 2017-2019:> Working as a backend engineer, I helped build out toll processing software with Transurban. This was my first exposure to golang and AWS
Hitwise, 2016-2017: I was a Java developer with the data development team, working with data pipeline applications and ETL processes.
LexisNexis, 2012-2016: My first “real” programming job after graduating uni, I worked with a number of senior engineers to build modelling software using C# and bespoke, proprietary data formats
4D Canberra, 2009-2011: I worked as a sysadmin and IT technician to get myself through university. I assisted in administering a mixed Windows domain and Linux infrastructure within the office, as well as providing general office IT support to external clients.

Recruiters please note, if I receive an email from you without having given you my address before, you’re either not going to hear from me or receive a hostile or sarcastic response.

No, Satya, I will not stop calling it slop

Mon, 19 Jan 2026 00:00:00 +0000

I don’t care what a suit says, I’m not consuming low effort nonsense

Winter downtime

Tue, 29 Oct 2024 00:00:00 +0000

I’ve been putting off writing anything here for since April. Not because I don’t have anything to say, but I’ve been seriously lacking motivation to do much more than go to work, go to the gym and sleep.

I’ve always had a bit of a seasonal depression, though for the most part I’ve been able to manage it with proper diet and exercise. This year, however, it’s just hit me hard and I’ve not really wanted to do anything productive.

Trying to go back to Linux Daily Driving

Thu, 25 Apr 2024 00:00:00 +0000

As I’ve mentioned a few times in the past, I’m quite an avid Linux user and have been for a long time. All my rack machines run Ubuntu Jammy, and I’ve been running Arch on my laptop ever since I got it.

However, I haven’t used Linux on my desktop machine since about 2 years ago when my budget SSDs shat themselves. I quickly bought a 4TB Samsung NVME replacement and decided to go the Windows route because of games and music software. I’ve been using Windows 10 since then, and I think it affected me in odd ways.

Migrating from Codepipeline to Gitea Actions

Sat, 27 Jan 2024 00:00:00 +0000

I recently slapped a new box in my rack and installed some general purpose services on it, one of which was Gitea, which is a self hosted git service.

One of my ongoing efforts is to move as much as possible off the cloud and into my rack. While this is not necessarily the most economical decision, I have the peace of mind that I’m keeping my data in my own space rather than shipping it off to a third party. It also has the added benefit of not needing to wait until Amazon gets their shit together and updates the node build images.

Reflecting on 2023

Thu, 04 Jan 2024 00:00:00 +0000

Thought I’d post a stream of consciousness type post on my year, might help me piece together some disparate thoughts.

Others posting

I’ve seen a lot of negativity from others this year. I get that things are tough, but there seems to be an active push against acknowledging the great things in your life. I’ve personally taken to being more positive about things, even if I do crack dark jokes at times.

TIL in GCP

Tue, 28 Nov 2023 00:00:00 +0000

So today I learned that by default GCP service accounts can’t impersonate themselves. I was trying to run a Github action at work, and one of the things in the script was a call to the credentials service to generate the correct token. I passed the bound github runner service account as the account to impersonate, and got back a 403 error. I ended up forking the action and just pulling it into our repo to short circuit the impersonation function.

Linux Things I've Learned This Week

Fri, 17 Nov 2023 00:00:00 +0000

I’ve been a Linux enthusiast since I first started uni. I still have my first Kubuntu CD somewhere, and while I mostly work off the Macbook work has supplied me now, my own personal Thinkpad is running Arch.

However, one thing I never bothered diving into too deeply was systemd. This came back to bite me a bit with my openstack install, as Zun containers were hanging on boot.

This led me down the rabbit hole, and I’ve spent about 3 weeks trying to figure it out. Originally, I thought it was a Zun problem, since it just looked like it was never starting the container. After trawling through logs, I eventually tracked it down to the time when Kuryr was trying to attach the interfaces to the bridge.

Annoying OpenStack Quirks

Fri, 20 Oct 2023 00:00:00 +0000

So I’ve spent a bit of time with OpenStack now, debugging issues and getting my setup tuned and ready to go.

For the most part, Neutron and Nova work great. No complaints. Haven’t used much of Keystone directly but I’m sure it works fine. Bit of configuration and debugging to do on Magnum (I think), but again I’m pretty sure it’s a “I configured this incorrectly” problem and not a “this is a piece of shit” problem.

Running OpenStack at home

Tue, 08 Aug 2023 00:00:00 +0000

I recently found a 50% off deal on rack servers at Lenovo, and picked one up for a pretty good price. It’s 100% overkill, but I’ve always wanted my own server just because.

Since it’s complete overkill, I decided I needed to keep that train rolling and set it up as an Openstack all in one.

The Strategy

In the lead up to the server’s delivery, I looked into the different ways to set up the cloud. The first stop was microstack, which is a Canonical supported effort. Configuration was trivial, and I was able to quickly get it up and running in an Ubuntu server VM. After exploring a little bit, I decided that I wanted to experiment with Trove, Swift and Magnum, which aren’t offered in the package.

A little Sunday afternoon project

Sun, 09 Jul 2023 00:00:00 +0000

It’s raining here today, so my plans to go outside and weed everything after our holiday have been scuttled. Looking for something to do, I decided to see if I still remembered how to do React, turns out I do and I made a little amortization calculator. I’ve also bumped up Gatsby to v5 because codebuild supports node 18 now.

I did cheat and pull in a few more libraries that’ll likely blow out my build time, but eh, maybe I’ll do some more stuff with them later because I like making charts.

On Burnout

Tue, 27 Jun 2023 00:00:00 +0000

I got back from a month long holiday to Europe on Monday, and was struck by how different I felt after landing. For about 18 months now, I’ve just been completely rundown. I have had no original ideas of my own, I haven’t written any music and I haven’t had any remotely inspiring moments despite my frequent jaunts into the reserves around my home.

It starts small

A little while ago, I started feeling sick most of the time. It wasn’t a major thing, maybe a sniffling nose or a small headache that I could take a Panadol and power through. I was constantly tired from this and its impact on my sleep. It would bubble underneath the surface for a few months and then spill over all at once, leading me to take more than a couple of sick days here and there.

Writing a VM Part Four - The Assembler

Wed, 11 Jan 2023 00:00:00 +0000

I’m not 100% proud of this assembler, but here we go. I’ve hacked one together that works well enough, but I don’t think I’ll show much code here. Code is here for the curious.

Source Material

Originally I was completely lost and thought I might need to write a full tokeniser and lexer, but I found an easier way after skimming through Assemblers and Loaders by David Saloman.

My Assembly Language

My assembly language is as simple as I could make it. I’d like to add macros at some point, but it’s not really up the top of my priority list right now, so can wait.

Writing a VM - Learnings So Far

Thu, 29 Dec 2022 00:00:00 +0000

I’ve been working on this VM on and off for a month or so now, and I’ve definitely had some ideas of what I could have done better. I’ve learned from this, and I’ll certainly be putting this into practice should I ever try to do something like this again.

Planning out my entire instruction set before coding

I’m in a bit of an awkward spot at the moment with my instruction set, and it’s likely to get a little messy as I go ahead. I’ve been adding instructions piecemeal as I’ve written each blog, and as a result I’m going to end up with bitwise actions and interrupt handling in strange spots in the table.

Writing a VM Part Three

Tue, 20 Dec 2022 00:00:00 +0000

Welcome to part 3 of this blog series. We’ll be adding instructions to allow us to better control the flow of our code, as well as adding in comparison instructions. Code for this blog entry will be available here.

I’d recommend reading part one and part two before reading this part.

Overview

We’re going add new instructions to control the flow of our executing code:

Hex Value	Mnemonic	Description
0x0A	PUSH	Push the value of I1 onto the stack
0x0B	POP	Pop a value off the stack and into I2
0x0C	JMP	Set PC to the address in I1
0x0D	LESS	Run the next instruction only if I1 < I2
0x0E	LTE	Run the next instruction only if I1 <= I2
0x0F	GT	Run the next instruction only if I1 > I2
0x10	GTE	Run the next instruction only if I1 >= I2
0x11	EQ	Run the next instruction only if I1 = I2
0x12	CALL	Set the PC to the address in I1
0x13	RETURN	Return to the last `CALL` location

To facilitate these changes, we’ll be adding a new register to the machine.

Writing a VM Part Two

Tue, 06 Dec 2022 00:00:00 +0000

When we last left off, we had a basic machine that could add two numbers. This time, we’ll add a new register to act as a set of flags to convey the machine status, instructions to allow us to set and reset these flags and a few other useful arithmetic instructions.

The New Architecture

We’re going to add a new register, which isn’t a large change

This will act not as register that holds a number, like the other registers in the CPU, but as a set of 32 binary flags we can use as indicators for things like errors and interrupts. Each bit in the register will be one flag, meaning we can mask the register with the flag value in order to get the value.

The Documentation Trap

Wed, 23 Nov 2022 00:00:00 +0000

In the beginning, there was READMEs and code comments, and the great architect in the sky saw that it was good.

As software grew, so too did the tomes accompanying it, and tools were developed to help manage it. Sharepoint, wikis, and automated documentation build processes. But they were all deceived, in the darkness of Sydney, a tool was forged by the dark master Atlassian, and thus unleashed on the world was Confluence.

Writing a VM Part One

Tue, 15 Nov 2022 00:00:00 +0000

When I was a student, I did an intro to computer architecture course that was instructed by Peter Strazdins. I was struck by his enthusiasm and his apparent thirst to pass along everything he knew, and while I would never have called us friends I certainly was on very good terms with him and had more than a few very geeky chats with him after lectures and tutorials. My time learning under him certainly shaped a lot of current software engineering interests, particularly around VMs and low level coding.

Update to my last post

Thu, 01 Sep 2022 00:00:00 +0000

An update to my last post. My wife sent me this article where multiple former employees have come out and talked about conditions at the big 4 consultancies. It’s heartening to know that there’s light being shone on their awful treatment of workers, and hopefully they do better in future.

What is normal in the software industry

Wed, 24 Aug 2022 00:00:00 +0000

What is normal in the software industry?

Maybe a couple of years ago, I had an interesting encounter with a soon-to-be graduate software engineer who asked me a few questions about what the industry was actually like.

“Is being held back until 3am normal?” “How much code are you expected to produce every day?” “What things should I learn in order to keep myself in a job?”

I had an extended time with this young man, just chatting. It struck me how much he seemed worried that he’d chosen the wrong career path. I did a little further probing, and it turned out he’d done an internship with a certain big 4 consultancy (which shall remain nameless) who had clearly mistreated their interns.

Peeper

Thu, 30 Jun 2022 00:00:00 +0000

I’ve recently started having more to do with Kubernetes lately, and have as a result have learned about the Sidecar Pattern.

I thought that maybe I should see if I can do something that might be useful for that, and one of the stinging points I’ve had is authentication.

Enter Peeper

Peeper was written to abstract away authentication from REST clients and presenting an interface that makes it look like they’re being presented with an unauthenticated interface. This may save some work if you’re writing services in multiple languages and don’t want to rewrite the authentication logic every time.

What Go Taught Me

Mon, 16 May 2022 00:00:00 +0000

I’ve been using Golang professionally for a number of years now, having had my first exposure in 2017 or maybe 2018. It was sold to our team as a faster version of Java, with a lot of cleaner libraries that would help us write, build and deploy much faster than our existing Java application.

For the most part, I’ve not only found that to be true but to have completely altered some of my existing ideas on software engineering.

NES Emulators and Such

Sat, 30 Apr 2022 00:00:00 +0000

I got it in my head the other day to start writing a NES emulator, as I didn’t really have a current home project on my plate. I’ve dabbled in virtual machines before, having tried to build an implementation of the DCPU computer, but I only got so far with that one since there wasn’t really anything to run. The NES, though, has no end of documentation and people still releasing code for it.

The Important Things

Thu, 31 Mar 2022 00:00:00 +0000

I’ve been feeling a little down and snowed under emotionally lately, and as a result haven’t really had the drive to blog about anything. Today, though, things happened which cut through the fog I tend to get when I’m struggling a little.

My morning started at 6:30, when I jumped out of bed and started getting ready. After showering, I noticed that my jeans had blown themselves apart at the seams.

More Fun With Rust

Fri, 28 Jan 2022 00:00:00 +0000

I’ve been playing with Rust some more lately, and I’m starting to really appreciate a few things about the language.

I’ve been going through Hands on Rust, and building the little Roguelike game the book lays out. I’ve learned how to use Cargo, how modules work and how to stop worrying about macros. It’s been a very enlightening experience, and I’ve enjoyed it immensely. I have also been working on an implementation of the DCPU-16 computer using Rust, though I’m not quite done and still need to implement interrupts.

A Warning on Hardware

Sun, 16 Jan 2022 00:00:00 +0000

Around October last year, I built myself a high end PC. Mostly because I could, and my existing rig was starting to age a little. And when I say high end, I mean it. AMD 5950x, RTX 3090, 128GB of RAM and 8TB of NVMe storage.

I also bought a case and AIO CPU cooler combo, since I’m not really patient enough to build a custom water setup.

Cracks Form

Around November, I started getting a ton of disk write errors on my storage drive. This first manifested as unloaded textures, music and SFX in some games I was playing with some of my friends. After a reboot it came good, but I kept having similar issues sporadically. Get to early December, and I’m trying to do some pretty heavy I/O processing of some files I got off the internet somewhere, and the hard drive would just completely disappear. I downloaded CrystalDiskInfo, and it’s reporting 0s for every S.M.A.R.T statistic.

Some Productivity Experiments

Mon, 06 Dec 2021 00:00:00 +0000

This past fortnight, I’ve been trying something to maybe see if I can actually finish another home project. Like I mentioned in a previous blog, I adopted the Zettelkasten method for notetaking, which requires that you break everything down into small, bitesized cards and file them away somewhere sensible. It’s made me more focussed, and quickly able to find information that I know I’ve seen around before.

One problem I’ve continued to have, though, is being able to see my home projects through to the end. One would think that this would be as simple as setting up a kanban board with some cards, but I’ve found I still lose steam after an initial burst of motivation.

CodeCommit SSH Issues

Tue, 30 Nov 2021 00:00:00 +0000

Just a short one, went to pull some changes from CodeCommit and got the follow error

Unable to negotiate with 103.8.174.37 port 22: no matching host key type found. Their offer: ssh-rsa
fatal: Could not read from remote repository.

Looks like OpenSSH has changed around some defaults and won’t automatically accept SSH offers, no big deal. Add the following to your CodeCommit section in ~/.ssh/config

Host git-codecommit.*.amazonaws.com
   User AAAAAAAAAAAAAAAAAAAA
   IdentityFile ~/.ssh/id_codecommit
   # Add these to let the SSH client accept RSA keys
   PubkeyAcceptedAlgorithms +ssh-rsa
   HostkeyAlgorithms +ssh-rsa

Maybe eventually AWS will accept ed25519 keys and give us a longer term fix, but for now this will have to do.

Playing with Go Generics

Sat, 13 Nov 2021 00:00:00 +0000

Go 1.17 dropped a while ago, and a workmate at the time was quick to notice that with it had dropped the long awaited generics. Generics were something I used a lot of in the Java and .NET worlds, but being in the Go world for the last 5 or so years has taught me that they’re not entirely necessary.

However, if this is to become a staple feature in my language of choice in 1.18, then I should probably learn to use it. I may run across that 0.01% of problems that require it one day.

Some minor changes

Thu, 21 Oct 2021 00:00:00 +0000

As I’ve been at this for a little while now, my blog post folder was starting to get a little difficult to navigate

I’ve added some stuff in Gatsby node to make this a little easier to sort through

I think this is how it was laid out when I used heroblog, but I’ve only just got around to adding it back in now.

Here’s what I added

How Dungeons and Dragons Helped me Organise My Notes and Become a Better Engineer

Wed, 22 Sep 2021 00:00:00 +0000

I’ve been an avid tabletop gamer since I met my now wife in late 2013. I’d played D&D once or twice in uni, but didn’t really get into it. That was, until Jocelyn saw me get excited over a copy of Edge of the Empire in my FLGS and encouraged me to give it a go.

I fell in love with it all pretty quickly, eventually falling into the perma GM role in my group. Of course, with the GM chair came campaign notes that needed to be organised. Characters, stats, planning, and plot notes eventually got out of hand. I started with a small folder, eventually growing into a thick folder.

CodePipeline Notifications

Wed, 04 Aug 2021 00:00:00 +0000

So with the last blog I published, my build failed and I didn’t realise for a day or two. Not being happy with that situation, I turned to one of the things I learned in my preparation for the devops pro certification.

How this Blog Works

For some extra context, this whole site uses Gatsby to build static HTML. On the AWS side, it’s served off a private S3 bucket with a CloudFront distribution in front of it, and I use CodePipeline to build and deploy it out when I commit and push a new markdown blog post.

The Four Personalities of the Creative Process

Sun, 18 Jul 2021 00:00:00 +0000

A few years ago, I decided to do something completely unnecessary and travel to Chicago for a series of guitar workshops. It was 7 days of just hanging out with other guitarists jamming and learning from other very talented musicians. Probably not the most amazing use of my savings, but I enjoyed myself nonetheless and learned a lot about writing music and playing with other people.

Of the things I learned in that week, the most useful has been something the instructor called the “Four Personalities”, a way of breaking down the creative process and trying to get through blocks in your songwriting. I’ve found that it matches some of the things we do in software engineering, and have since adapted the way I work to somewhat mirror this process.

Adding Pagination to the Blog Listing

Mon, 28 Jun 2021 00:00:00 +0000

I added some pagination to the blog listing page, took maybe an hour or so. Kinda surprised I manged that.

Basically, I edited my gatsby-node to generate a new set of nodes depending on the posts it finds. I also removed my index.js page so I could use the same code everywhere. My node generation now looks like this

const blogPagesTemplate = path.resolve('./src/templates/blog-page.js')
const paginateLimit =  siteMetadata.paginateLimit
createPage({
  slug: `/`,
aliases:
 - /`/`,
  component: blogPagesTemplate,
  context: {
    skip: 0,
    limit: paginateLimit,
    pageNumber: 1,
  }
})
for (let i = 1; i*paginateLimit < posts.length; i ++) {
  createPage({
    slug: `/blogs/${i + 1}`,
aliases:
 - /`/blogs/${i + 1}`,
    component: blogPagesTemplate,
    context: {
      skip: i * paginateLimit,
      limit: paginateLimit,
      pageNumber: i+1,
    }
  })
}

While my updated GraphQL query looks like this

Rust as a Go Programmer

Mon, 14 Jun 2021 00:00:00 +0000

So I decided to finally sit down and check out Rust, as it’s a language that’s gaining a large amount of traction and generating buzz for its various features.

The things that got me curious were:

Memory ownership, go garbage collection can be brutal
Functional features
Generics
Potentially never having to deal with C again

It took me a little bit to start getting my head around how it was all put together, but things eventually started to fall into place as I started to map over my existing knowledge.

Finding Errors

Thu, 20 May 2021 00:00:00 +0000

My day to day grind at the moment has been configuring new environments as we have new work coming in. I’ve done it enough at this point that I know exactly where the pain points are and what’s next in line to automate, so I think by the end of next month I’ll be able to spin up a whole new environment in under 20 minutes before customisations.

The fun begins

I spun up a new environment and got all the Amazon configuration sorted. Connect was happy, I could make phone calls and I could see everything streaming out to the database properly.

Things I learned coding a chess game

Sun, 02 May 2021 00:00:00 +0000

As I alluded to in my bitboard rant a little while ago, I’ve been working on a chess game in my own time. I’ve had a few realisations while I’ve been building it, which I thought I’d share.

Realisation 1: You don’t always need objects

Coming up through uni in Java and spending a large chunk of my professional life using OOP languages, I often found myself reaching for a struct in golang. On noticing how great the bitboard solution was, I started rethinking how I define my types.

Opting out of FLoC in CloudFront

Sat, 17 Apr 2021 00:00:00 +0000

The current quiet tech battle being fought is against Google’s Federated Learning of Cohorts, or FLoC, which Google is pushing as its alternative to third party cookies. The Verge explains it better than I could, so I’ll link it and call it day on that part.

I don’t use tracking or invasive analytics on my site and never will, so let’s make sure it’s difficult for Google as well.

How do we stop it, if it’s browser based?

There’s a new HTTP header we can use, Permissions-Policy. We can set this on our Cloudfront origin easily enough to disable (we hope) FLoC in the browser on our site. W3C has a full spec available with examples if you’re interested in reading more

Making and deploying AMIs with Gitlab CI

Thu, 04 Mar 2021 00:00:00 +0000

I’ve had three weeks off from work for a little break after a particularly tough job, so I decided to start sharpening some of my AWS knowledge and actually get that DevOps Pro certification sometime soon.

One of the practice questions I was looking at dealt with quickly spinning up EC2 instances in an autoscaling group, the answer to which was to bake an AMI using a script. I realised I’d never actually done that before, so thought I’d give it a go.

Salesforce Endgame Tool

Sat, 20 Feb 2021 00:00:00 +0000

So something interesting I spotted this week was a tool called Endgame.

Originally announced by Salesforce, the tool was pulled by the time I found it, though by that point it was trivial to dig up a fork. The purpose of the tool is to be able to see whether an IAM principal can smash open an account’s resources, and to be honest it’ll be very useful to me in the future while testing attack surfaces.

FullCalendar with Gatsby

Wed, 27 Jan 2021 00:00:00 +0000

Ran into a situation a little while ago where I needed a calendar page for something I was working on, eventually settled on FullCalendar with its React docs.

I wrote up my page and got it working with the development server, only to have the following issue when I went to build my production bundle

"window" is not available during server side rendering.

See our docs page for more info on this error: https://gatsby.dev/debug-html


  1 | import * as react from 'react';
  2 | import * as reactDom from 'react-dom';
> 3 | (typeof globalThis !== 'undefined' ? globalThis : window).FullCalendarVDom = {
    | ^
  4 |     Component: react.Component,
  5 |     createElement: react.createElement,
  6 |     render: reactDom.render,


  WebpackError: ReferenceError: window is not defined

Not being a JS developer normally, this led me down a bit of a rabbit hole and it took me a couple of hours to find a solution, which is

Bitboards, or how I learned something that should have been completely obvious

Tue, 24 Nov 2020 00:00:00 +0000

I’ve always had a fascination with chess, and indeed I played and won trophies for it in school. Long gone are my glory days, however, and I haven’t played regularly for going on 12 years now.

I grew up playing Chessmaster, and always wondered how it worked. When I did my AI course at uni, the curtain was pulled back a little as I learned about search algorithms, but I put this particular problem to the back of my head until I was bored after work last week.

Fargate Task Recycling

Tue, 27 Oct 2020 00:00:00 +0000

UPDATE 11/8/2023

I got a message on LinkedIn asking for an update as one of the links had gone stale. I’ve replaced it with an archive link, no idea if this is still how Fargate operates under the hood. There is also the new task maintenance page that seems to offer a similar explanation.

Original post

Came across something curious in my investigation into why I had a couple of Cloudwatch alarms going off.

Federating Amazon Connect with Azure AD

Tue, 13 Oct 2020 21:22:00 +0000

Naturally with any large organisation, they’re only going to be interested in your software if you make it easy for them to integrate it into their workflows. To that end, let’s have a look at how we get an Amazon Connect CCP working with Azure as an IdP.

Set Up your Amazon Connect Instance

The first step is to set up your Connect instance to use SAML. This is set when creating a new instance and cannot be changed later, so make sure you do this correctly first go

Tweaking the blog again

Mon, 05 Oct 2020 00:00:00 +0000

As should be obvious if you’ve come here before, I’ve redone the place. This time, I went straight off the Gatsby blog starter and tweaked a few things here and there.

New Features

Full tagging. Before I only had broad categories and it was becoming difficult to find certain posts. Give it a go by clicking the hashtags under the title.
Simpler styling. I really hate cluttered, noisy UIs. Being able to reduce the amount of things going on on the screen makes me happy.
Shorter build times. It was taking me a long time to build and bring up the development server. Paring it all back means I can be stupid faster.
More frontmatter metadata. I have the option of either letting Gatsby generate an excerpt or specifying a short description for each post.

Lessons Learned

GraphQL is something that I’ve been curious about for a while now, it was good to get my hands dirty with it to figure out whether it’s actually something worth using. While I like it here, I’m still not convinced whether it’ll always be the right tool. Seems like a lot of overkill for most applications, maybe if I had a huge search space that needed some level of arbitary-ness to it, but I think I’d just prefer query parameters most of the time
React. I know I’m using it in the context of Gatsby right now, but I think I could actually grow to enjoy using this. The component based development style means I’ve been able to identify a couple of common components that I’ve been able to pull out, and in the process I’ve been able to better figure out how my own site works under the hood.
Gatsby in general. Previously, I was able to make small changes, now I’m feeling way more comfortable modifying the little parts that annoy me

Hopefully, I’ll be able to continue tweaking this incarnation of the blog to something I can really be proud of. It’s already lasted longer than my last few attempts, so I’ll take that as a good sign

IAM Authentication with Postgres

Tue, 15 Sep 2020 00:00:00 +0000

Getting IAM auth in Postgres is a pretty easy process. The first thing we’ll need to do is to enable it in our RDS instance

We also need to create ourselves a user within Postgres and grant it the rds_iam role.

CREATE USER iam_authed_user LOGIN;
GRANT rds_iam TO iam_authed_user;

We’re ready to get ourselves a temporary auth token. In golang, use rdsutils.BuilAuthToken

endpoint := fmt.Sprintf("%s:%d", DatabaseHost, DatabasePort)
tok, err := rdsutils.BuildAuthToken(endpoint,
  region, DatabaseUser, credentials)

Where region is the AWS region and credentials is an AWS credentials provider. Important to note that the endpoint parameter contains both the host and the port. We then use this token to connect to our database

Cloudformation API Gateway with Cognito Authorizer

Mon, 24 Aug 2020 00:00:00 +0000

I’ve been back at the Cloudformation in the last little while as we’ve been provisioning some new clients at work and I wanted to speed things up substantially. This led me down a bit of a rabbit hole experimenting with various parts that we’ve previously done using ad-hoc clickops, including Cognito user pools. I found there wasn’t really any complete examples out there for me to rip off, so I’ll dump what I came up with here.

Tunneller

Wed, 13 May 2020 00:00:00 +0000

Been a bit slack on updating this blog, so to make up for it I slapped together a quick and dirty application to tunnel through EC2 instances and into RDS services sitting off the public web.

You can find the source code on my github page

Frustrations with Fargate and ECS

Thu, 30 Apr 2020 00:00:00 +0000

I just spent a good month wrangling ECS Fargate deployments with a ton of cloudformation to boot. I’m going to outline the stupid things I’ve had to deal with and some errors I’ve found

Fargate

Fargate is an AWS offering that allows you to bring up docker containers without needing to provision servers to run them on. Aside from that, it’s much the same as a normal ECS cluster from a developer perspective as far as I can tell.

Breadth vs Depth of knowledge

Sat, 04 Apr 2020 00:00:00 +0000

Forgive the rambly nature of this blog entry, just trying to set my own thoughts straight a little bit. This isn’t intended for anyone else, really, but I think I need to put it down so I can track how my opinion on it has changed later on.

A long time ago, I thought art was a bit of a waste of time. Time we could be putting into solving real problems in the world was being sunk into creating entertainment and “expressing yourself”. At some point, my thoughts changed on this somewhat, and I started reading more about art and music I started to appreciate it a lot more and eventually picked up a guitar with a tax refund one year. Nearly ten years later, I’m a competent guitarist and have released a single EP with a band on top of regularly writing my own music for the DnD campaigns I run (most of which I’m not really proud enough of to release publicly, sorry).

Go's net/http package vs gin

Tue, 24 Mar 2020 00:00:00 +0000

UPDATE Feb 20, 2021

I checked my Google search console and realised that most of my traffic comes in through here. Here’s the summary for you:

Golang’s net/http and Gin follow much the same philosophy
Gin has wrappers around route registration, which means less boilerplate
Gin offers a few more convenience wrappers around deserialisation that’ll mean you don’t have to worry about the boilerplate around the json.Unmarshal function
Gin uses httprouter, which is apparently faster
Middleware works in much the same way as vanilla go

Personally, I still prefer Goa or using OpenAPI to generate my plumbing for me, but if you’re deadset on using a code first approach, Gin is a decent enough choice. Just be aware of the dependencies you’re pulling in.

On working from home

Mon, 23 Mar 2020 00:00:00 +0000

I’ve long been in favour of flexible work policies that allow remote work where possible. While there are many benefits in face to face contact, I personally believe that having a home office helps productivity and quality of life.

With the current pandemic, I’ve spent the last sprint working from home, and we managed to blaze through enough work to convince the executives somewhat that a remote first policy is certainly viable. I think this is being bolstered by the numbers from many other offices around the world, showing nervous middle management that their reluctance was more about the inability to move beyond their comfort zone than it was about productivity.

Writing testable Go code

Wed, 11 Mar 2020 00:00:00 +0000

Testing in golang is much like any other language, and Go even comes with some of its own tools to write your tests with.

Today I’m going to share a couple of the tricks I’ve learned to make sure the code comes out smelling like a patch of roses with a sensible amount of code coverage

Build Tools

I always have some makefile targets to run my tests and build tags in my code to run all my tests.

Black Hat Go first impressions

Tue, 03 Mar 2020 00:00:00 +0000

I picked up my order of Black Hat Go last week, and after flicking through and reading (not necessarily doing), I have a few things to say based on my skim through.

Presentation

The layout is logical, and it steps through each technique quite thoroughly in a similar way to black hat python. It’s quite an easy read if you know what you’re looking at

Modern Solutions to Modern Problems

There’s a heavy emphasis on not reinventing the wheel, which I can appreciate. One section is dedicated to interfacing with Shodan, and there’s quite a few examples that use Metasploit quite prominently.

Concurrency in Go

Wed, 26 Feb 2020 00:00:00 +0000

Something I’ve really come to appreciate over the last 2 years or so I’ve been using Go is its concurrency constructs. It’s trivial to push something out into another goroutine, and it takes virtually no effort to synchronise your workers.

The most obvious place to start is with the go keyword

package main

import (
    "log"
)

func worker(id int) {
    log.Printf("Hello from worker %d", id)
}

func main() {
    for i := 0; i < 10; i++ {
        go worker(i)
    }
    log.Printf("Goodbye from main")
}

Simply prepend it to a function call and it will push it off to the background for you. Much easier than trying to spin off a bunch of pthreads or writing a large amount of boilerplate to set up a worker pool.

Getting Started with Asterisk

Wed, 19 Feb 2020 00:00:00 +0000

I currently work for a company that deals in call centre software, and until this point I hadn’t really started looking into the VOIP side of everything. I decided I should probably change that

First Order of Business: Set up SIP

The first thing I wanted to do was set up Asterisk, and I thought the best way of doing this would be to set up the basic hello world server from the docs. You can find it on my Gitlab or you can just pull down and run the container using docker run -p5060:5060 -p5060:5060/udp registry.gitlab.com/threetoes/asterisk-container

Making Cloudformation Generic

Thu, 13 Feb 2020 00:00:00 +0000

I’ve spent the better part of the week pulling apart one of my own Cloudformation stacks trying to make it suitable to take with us between clients, and as such I’ve picked up some ideas on how to make it more portable between environments. I’ll probably read some more blogs on this stuff in the coming weeks, but here’s the small collection of tips I’ve come away with trying to get this stack into our new account.

API first web services

Wed, 05 Feb 2020 00:00:00 +0000

Recently, it’s become apparent at work that we should probably have skeletons in place to make it easier to get a new AWS service up and running. At the moment, the HTTP framework of choice is Gin, which is a nice enough framework that provides a familiar workflow:

Write your tests
Write your logic
Wire up your endpoints

Design First

However, while working elsewhere I was won around to the idea of Swagger first implementation of web services. At first I was skeptical, having come from the Spring boot world where you just slap a @Controller annotation on a class to make a new endpoint, and the spec first tools I’d used in the past in my own tinkering were a little meh to say the least.

Assumed Roles with the AWS SDK

Wed, 22 Jan 2020 00:00:00 +0000

Something I haven’t noticed before, mostly because I haven’t needed it until recently, is using assumed roles to authorise applications with the Amazon SDK. I was met with the following error when I tried to use a profile defined with a role_arn and source_profile instead of an access ID and secret

encountered error: NoCredentialProviders: no valid providers in chain. Deprecated.
 For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Took me a little bit to find the solution via Duck duck go, but it turns out you can’t use the AWS_PROFILE environment variable here.

Getting started with Cloudformation

Mon, 13 Jan 2020 00:00:00 +0000

About 2 and a half years ago, I set upon what would be the first step of my career in cloud computing. While skeptical at the time, I very quickly adapted and realised the immediate benefits I was getting from AWS, and especially how I, as a developer, was going to benefit.

I first came into contact with cloudformation when our platform team told us about our base infrastructure repo, where I was to spin up the s3 buckets I needed for something I was working on. No problem, I thought, it’s just YAML.

Minds.com Profile

Sat, 11 Jan 2020 00:00:00 +0000

I created a minds.com profile, because I hate Twitter and refuse to use it. You can find me on there as @threetoes

Git Driven Command and Control Part 1

Sat, 04 Jan 2020 00:00:00 +0000

Happy new year! I’ve been busy visiting family for the last little bit, so haven’t had a lot of time to write blogs. Today I’m going to show the basics of a command and control server, which I’ve loosely based of Black Hat Python but decided to write from scratch just for the fun of it. Code is available on my github

The point

This is a thing I’ve used in the past to control some raspberry pis I kept under my living room coffee table for a while. I had a github repo set up similarly to the one I’ll detail in this post, and they would bot around and scrape websites based on what I’d put in my config repo.

Fyne GUIs

Sat, 21 Dec 2019 00:00:00 +0000

Something that took my interest at GopherConAU this year was Steve O’Connor’s talk The Fyne GUI Toolkit, where he showcased what Fyne was capable of.

Anyone that’s worked with me knows I have WinForms induced PTSD when it comes to GUI toolkits, long ago coming to the conclusion that anything I wrote will just be a console app or web API. While in the past, I’ve found WPF quite comfortable and had minimal amounts of success using the Angular framework with Material, it’s just too fiddly and annoying for a colourblind person like me to care that much about GUI programming.

Spamming the Spammers

Thu, 12 Dec 2019 00:00:00 +0000

So I’ve been getting some intriguing emails lately, featuring auto replies from contact us forms saying thank you for your message.

Finally got one that included the body that was sent and sure enough it had a link to a “German” SEO company. Digging further, I found their contact form submitted to a 3rd party site, an “American” ad agency which doesn’t seem to exist at all.

I dug through the site a bit more, just to confirm that they were definitely fake. Just a hint, if you want to sell yourself as tech savvy make sure your website isn’t a hot mess like mine. What intrigued me, though, is how their webform didn’t seem to require any sort of CSRF token, nor did it seem to demand any additional state beyond the form data.

Black Hat Python - ARP Cache Poisoner

Wed, 04 Dec 2019 00:00:00 +0000

Code is on my gihub

December is upon us, so my wife and I have started our annual Christmas movie binge. I finished this script up while we had Die Hard on, and kept wandering back into the study to check my desktop’s ARP cache as the book suggested.

I’ve done my usual thing of adding an argument parser to it, and I’ve made a couple of hacks to try and make everything shutdown a little more cleanly.

Black Hat Python - Network Scanner

Wed, 27 Nov 2019 00:00:00 +0000

Code is on my gihub

Another one where the script was more or less perfect as it was, minus the usual socket read/writes and print statements. One really odd thing is I guess there was a print error in the book.

The IP header structure was defined as

_fields_ = [
    ("ihl",  c_ubyte, 4),
    ("version",  c_ubyte, 4),
    ("tos",  c_ubyte),
    ("len",  c_ushort),
    ("id",  c_ushort),
    ("offset", c_ushort),
    ("ttl",  c_ubyte),
    ("protocol_num",  c_ubyte),
    ("sum",  c_ushort),
    ("src",  c_ulong),
    ("dst",  c_ulong)
]

Which I needed to change to

Black Hat Python - SSH Proxy

Mon, 18 Nov 2019 00:00:00 +0000

This one actually ended up being pretty much a straight port over. I decided to keep with the quick and dirty approach here because I’ve been procrastinating about doing it for the last week, so just getting it over and done with for now. The code is available here

Core lesson

This script shows off the transport mode of Paramiko, and since we don’t really do anything with the data we read off the sockets there really wasn’t a lot to do around the socket reads and writes. In theory we could intercept these messages and munge them as we wanted.

Black Hat Python - SSH with Paramiko

Sun, 03 Nov 2019 00:00:00 +0000

I finished this one up in Sydney airport this afternoon, though I think I went a little overboard while working on it. I gave a go of keying a dictionary with SSH channel objects and it seems to work, so I improved on my message reading loop from last time a little bit.

The Premise

This chapter presented us with a single server/single client to execute commands on a connecting client. I’ve changed that a little to be a more persistent connection to multiple clients, though you could probably just pump a shell script down the pipe and close the connection.

Black Hat Python TCP Proxy Rewrite

Thu, 24 Oct 2019 00:00:00 +0000

This next section was pretty easy to port. My implementation isn’t as clean as it could be, but for an implementation that took me an hour or so to get working then another hour or so to clean up a little. The repo is here

The Main Problem

Exactly the same as the netcat script. The socket methods don’t return strings any more, so let’s see what we can do about that

Updating Black Hat Python: Netcat Replacement

Wed, 16 Oct 2019 00:00:00 +0000

First order of business is updating the netcat replacement Seitz has written so it works again. My code is here

The issues

Simply put our biggest problem here is types.

The socket.recv and socket.send no longer deal with strings and expect a bytes type, so our updated script will need to be aware of and work with those, meaning we can’t really rely on newline characters to denote the end of a message. Our new implementation will need to change this up to utilise those new types

Updating Black Hat Python Scripts for Python 3

Mon, 14 Oct 2019 00:00:00 +0000

I’ve been working through the book Black Hat Python, and it’s got some good code recipes, but the code doesn’t really work as is.

I’m thinking I could do a blog series where I update each script to work with Python 3, then I might port it all to golang for laughs. I’ll start with the Replacing Netcat chapter over the course of this week, and we’ll see how that goes, then take it from there. I hope to not only rewrite the scripts, but go into depth about why they broke, if only to get my own thoughts straight on it.

Interviews from the other side of the table

Sun, 13 Oct 2019 00:00:00 +0000

Have you been stuck for tech interview ideas?

I’ve had a lot of chats with people about this as of late (don’t really know why, I’ve never run an interview before), but it seems to be something that comes up a lot because there’s no manual on how to do it.

Having spent the last couple of weeks looking for work, I’ve had a few interviews now, and I’m here to give you some general feedback on my experiences and what I think worked really well and why, and the inverse where I just felt like I wasted my time.

Time to try again

Fri, 04 Oct 2019 00:00:00 +0000

Having issues with Tumblr, time to move again. Luckily I learned about Gatsby and markdown driven static sites, so I’m going to give this a go using S3 and a codepipeline job to push it out to the live site.

Going to have to change a few things in this app, it seems to insist on a cover image for each blog post when it’s only really a nice to have for me. I just want something I can edit in vim and push out with git.