Switching to Hugo

    written by Zach Morrissey on 2020-02-24

    Hugo bills itself as 'The world’s fastest framework for building websites'

    One of the biggest changes for this blog happened just now - this blog switched from Lektor to Hugo. For me, it’s been a revolution in how personally productive I can be. I’m expecting that this will improve how quickly I can write and publish blog articles here and, maybe, even lead to me publishing more than 1x per ~3-6month period. I don’t intend this to be a dead blog, and I’m putting in the work to make that the case.

    I choose technologies in a very safe way, following a pattern:

    1. Choose technologies that already have strong adoption as opposed to niche or cutting-edge.
    2. Use technologies the way that others use them. Look for examples, re-use resources where possible.

    My original system didn’t meet either of these. This is a long-overdue follow up to my first post about the technologies I chose the first time around.

    Choosing a New Static-Site Generator

    Primary resource used: staticgen.com

    Background, I don’t particularly love being in the React ecosystem for scenarios when the value of React likely won’t be realized (which is to say, highly interactive sites that benefit from one-way data flow). This site is primarily data and not interactivity, so doesn’t necessarily lend itself to using it.

    Why Hugo?

    • It’s a single binary I can install anywhere.
    • I’m learning Go, and it reuses much of the Go ecosystem (notably, templating).
    • It’s fast.
    • It takes a simple approach to the web; that it’s a static set of HTML, CSS, and JS files. While you can wrap a build system like Webpack or Gulp in, it provides enough out-of-the-box features using Hugo Pipes that it may not be necessary.
    • Batteries included: Useful shortcodes, deployment targets, integrations.
    • It’s one of the supported static site generators for Netlify CMS, which is what I intend to use to write content from here on out.
    • The other obvious options, considering their popularity, are Gatsby and Next.js, but this site is largely static content and doesn’t necessitate the React ecosystem.
    • Support for some automatic content systems like Taxonomies and Archetypes.

    Why Not Hugo?

    In building the site on Hugo, there were a number of things that I encountered that I didn’t love; they were few and far between though.

    • Examples. The biggest single deficiency of Hugo as a static-site generator is that the examples provided by the documentation aren’t great. I’ve got strong feelings about this and I’m going to try to see if I can note times where this is missing and submit those as MRs.
    • Template debugging. Aside from printfing a bunch of variables, I didn’t find the mechanisms for figuring out what data is available where to be particularly useful (not that other SSGs have this down, though).
    • Template Lookup Order. Error messages are confusing and it’s hard to figure out where things slot into. The warnings that the hugo builder provides could be so much more useful.
    • As of writing, Netlify CMS doesn’t yet support page bundles (although this should be available soon).

    Why Not Lektor (any more)?

    My needs for the website have changed pretty drastically from when I first built it in 2016 and wrote things for it in 2018.

    • It’s not popular (ranked ~30th on StaticGen for static-site generators), and it’s not growing in popularity.
    • I don’t really use Python for anything web-facing, which is one of the reasons I originally stuck with Lektor.
    • Not enough batteries included. Many integrations are abandoned or haven’t been updated in forever.
    • Bad writing ergonomics. The editing page isn’t very helpful, and it doesn’t actually help you create content faster. There once was a Electron app that theoretically a headless CMS for the software, but it was quickly abandoned.

    Build & Deploy

    This site is now built and deployed using Github Actions, which are similar to Gitlab CI in form and function but are much easier to get up and going with if your repository is hosted there.

    Example-first Documentation

    written by Zach Morrissey on 2019-07-07

    When I read through your (library|module|framework)'s documentation I will skip everything and scan until I find the first example.

    The one unifying thing about many programmers is that they’re driven towards problem-solving, and they’re going to have a goal set when the finally reach your documentation. Documentation that aims to identify those goals and solve them as directly as possible is the best documentation. Users want the fastest way to prove and/or disprove their hypothesis.

    The most direct way, you might imagine, is examples. Lots of examples. Everywhere. Nothing comes close to how useful actual usage is.

    Two Common Ways I Use Documentation

    1. Checking how closely your examples match my intended use case. Everyone has a mental model of how they think a particular library will work prior to using it, and they will try to navigate to the right pages by searching for terms related to that. If this mental model is wrong, it’s important to know why that is and provide examples for what the alternatives look like.
    2. Attempt to install your software in a variety of ways I already know, and fall back to reading the “Getting Started” page when that doesn’t work.

    More specifically, if users can’t see the actual functions or classes they intend to use demonstrated, they will avoid your framework entirely.

    Although you’ll see more seasoned engineers throw around “RTFM” or other phrases extolling the virtues of painstakingly reading the manpages of a particular tool, but I will always have a shorter-term goal in mind than holistic understanding. Most documentation is intended as reference, but there still exists a large spectrum of user experience in navigating that reference information.

    Reasons Whey They’re Looking For Examples

    Ternary JS operator describing how seniors feel about juniors' questions when they're not 100% well-researched.
    1. Starting off and want a frame of reference.
    2. Users know what an implementation might look like, but don’t know what it’s called.
    3. StackOverflow-inspired copy/paste-heavy workflow.
    4. Asking for help on the internet sucks.
    5. Asking for help from your coworkers can suck too.

    If you believe it’s easiest to get started using a framework if the examples are good, then it’s truly the most user-focused way of helping your users out. It’s the quickest route from intention to implementation.

    Model Your Docs After Successful Software

    There are an endless number of under-documented pieces of code out there, and their success or failure is largely driven around how quickly users can get at the answer that fits their mental model of how your library works.

    Leaving some examples of great examples-driven documentation here:

    An Architecture For Personal Data Engineering

    written by Zach Morrissey on 2019-03-26

    Performing in-depth data analyses on things in my own life have always been more difficult than in my working life, primarily because there I’ve normally had teams of engineers who have helped in setting up and maintaining most of the analytics-focused infrastructure that I’ve used. Supporting myself with the same sort of infrastructure I was used to was an adventure, one that started much earlier and has since grown from there.

    Miscellaneous fake graphs for fun.

    Scope

    To most ends, downloading CSVs and plugging them into your spreadsheet software of choice will do most tasks well enough. For my purposes though, I wanted to scale this past what manual analysis was going to get me. There’s a few key areas that I wanted this for:

    • Personal finances. Most of the data that I create finds it’s way into financial systems that are hard to get your information out of. I’m not against the bank having my spending data, but I am frustrated that it’s so hard for me to get it too.
    • Fantasy sports. I love sports and I love stats.
    • House shopping. This data is heaaaavily guarded and it’s hard to find anything that isn’t breaking some sort of ToS to get.

    This doesn’t necessarily create a system that requires dedicated data engineering work, but some of the goals that I had for it did. These were:

    • Automated. I’d like to see how repeatable and reproducible these analyses can be.
    • Modeled. I’d like to build and train some models related to how I live my life to see if there’s any predictive benefit to these things.
    • Interactive. - For everything that you see here, I’d like to have some sort of interface that I can open, preferably via web browser.

    The Bits n’ Pieces

    For doing this sort of work, I set up a few infrastructural components. Hardware is a low-spec server with Proxmox (for VMs) and Docker/Portainer.

    Diagram of my personal data engineering architecture.
    • Postgres database - My primary datastore. This is the hub of all activity that I do, serving as both an application backend / transactional database as well as an analytical database. Each are neatly separated out into different schemas. I use DBeaver as a SQL client for ad-hoc querying, manipulation, setup, etc. Since a single Postgres instance can scale to vastly larger workloads than I’d ever be able to throw at it, this seemed like a good backbone for everything.
    • Apache Airflow - In it’s simplest form, I’m using this basically as a scheduled job engine. Building on my earlier post about Airflow, I’ve expanded my usage of it to a significant variety of different DAGs. This uses the same Postgres database as earlier.
    • Jupyter Notebooks - Interactive analytical code/markdown files served up in your web browser. Jupyter is the ultimate tool in interactive analytical computing, with RStudio being the only major other option I considered (went with Jupyter, if for nothing else, because I find python more enjoyable to write). There’s great options available if you want to run this on a server, I found that I like just running it locally the best.
    • Apache Superset - Web-based Dashboards. Superset, similar to Airflow, is based on Python/Flask and can be run in a single Docker container if you so desire. Since I need some way of sharing some of these results with a significant other (namely finance + house shopping data), I needed to have some sort of dashboarding software in order to have that visible to someone who will access it via web browser.

    My Workflow

    Since the pieces have been set in place now, this is my general workflow:

    Process workflow from Airflow task to Jupyter notebook and then to Superset dashboard.
    1. Find a new datasource. Generally, this is something scraped from the web or pulled from a service I use.
    2. Write an Airflow task to pull that data on a regular schedule (normally nightly).
    3. In Postgres, create schema / table in order to store info.
    4. Use downstream in:
      1. Superset dashboard. This is normally when I’ve created a datasource that I’m interested in (i.e. personal spending data) and want to look at from a variety of angles.
      2. Jupyter notebook. Normally, this is to either create a more in-depth analysis (any modelling required) or to investigate one hunch / idea that I’ve got. In the event that I something useful to replicate, I ship this upstream into an Airflow task.
      3. Exported to some sort of printable / shareable thing. Good example for this is pre-draft research for fantasy sports.

    Tips

    This was a fun adventure to go on, but I’m happy that it’s stabilized by this point.

    • It’s fun stuff. It really is. Using data to analyze parts of your own life is something people rarely get to do.
    • There’s a tradeoff between having a robust, feature-filled system and an easy one. As your needs change, your tools tend to change too. My aims for this outgrew my cron/jupyter/Google Sheets hackjob before trying to take on any of this.
    • Getting data is the hardest part. Significantly more time for me is spent writing data fetching tasks than anything else.

    Building Dynamic Dungeons using Unity Tilemaps

    written by Zach Morrissey on 2019-02-11

    In one of the last projects I’m doing for school, I’m working on a game wherein everything in the game is procedurally generated using a dungeon-building algorithm.

    Fresh Content Each Time

    One of the main draws of building a Roguelike game is figuring out how to procedurally generate dungeons. While it’s difficult to replicate the high-quality hand-crafted content of titles like Legend of Zelda: A Link to the Past, you can approximate a lot of the same experience while coming up with a new layout each time the game is loaded.

    Achieving this is, as you may imagine, pretty difficult.

    There are a lot of resources online, but most of them follow the same pattern - create a set of rooms, connect them, place items within them. There’s a significant amount of wiggle room within those, but that’s the basics.

    The Dungeon of Deneb logo and intro scene

    Unity Set Up

    The dungeon I built is based on a Unity Tilemap, which is effective a grid of 1x1 squares that you can place tiles within. For the game that I’m working on, I made each tile 16x16 pixels (you can set them to the density that you’re planning on making your tile sprites).

    The unity objects involved:

    • Two tilemaps (Created by Game Object > 2D Object > Tilemap). Spawning a tilemap will create a Grid object as its parent. I created two Tilemaps and connected them to the same parent. I attached my Dungeon.cs building script as a component to this parent.
      • Floor tilemap - This is what I use to tile the floor background, and this is the main data for the dungeon layout (since walls are just created using the floor layout).
      • Walls (Collidable) tilemap - This one has a collider added to it, so the player can’t walk through them.
    • Tile Palette - This contains all of the floor / wall tiles that I use to build the dungeon. It’s nice to have one of these so you can manually paint some on if you want.
    The "Dungeon Simplified" tile palette in the Unity Sidebar.

    The Algorithm

    Since dungeon generation algorithms can fall into a number of ways that they can end up in an infinite loop where they will never end up with a good dungeon layout, it’s best to program each step of this in the following way:

    • While not at my limit of tries
    • Try
    • If failed, reset everything and increment how many tries I’ve done.

    Steps

    1. Place a number of rooms in a grid.
    2. Derive a graph for the grid. The recommended way I found to do this was to create a Delauney Triangulation. This is somewhat complicated, so it’s recommended to find an algorithm or package that will do this for you.
    3. From that graph, derive a minimum spanning tree. I found that this was easy enough to do using Prim’s Algorithm, which is greedy and runs quickly.
    4. Verify that the graph meets your desired gameplay goals. For me, I wanted to make sure that I had at least two non-overlapping subtrees that comprised >25% of the nodes in the tree, so that I could use these as specific gameplay areas.
    5. Connect all the parent/child relationships in the tree with corridors.
    6. Once that’s done, add some optional corridors for the nodes within the same subtree, so that you can remove the perfect “tree”-ness of the layout and add some variety to it.
    7. Choose a boss room. For mine, I simply chose a leaf node in the larger subtree with a higher-than-average floor area.
    8. Choose rooms to place keys in that will open locked doors.
    9. Lock the doors.
    10. Place items and enemies within those rooms according to some sort of sliding difficulty scale. You can easily derive this difficulty on a number of simple heuristics (i.e. bigger room = more enemies, further depth in tree = harder enemies, etc).

    Go!

    Layouts Generated

    First, a key, so you can understand what you’re looking at:

    • Rooms:
      • Green / White = starting room / starting area
      • Red = boss room
      • Light blue = harder-than-starter area, Dark Blue = advanced area
      • Yellow = room with a key
    • Corridors:
      • White - Parent/child relationship in tree
      • Yellow - Optional corridor added between two rooms in the same subtree.
    Dungeon Layout 1

    The key to the light blue area spawns in the green / white area, the key to the dark blue area spawns in the light blue area. The key to the boss room is placed within the dark blue area.

    Dungeon Layout 2

    As you can see in this one, sometimes the room with a key in it spawns directly next to the boss (in the first example above), but sometimes it’s located fairly far away in a different corner (in this example, on the bottom left).

    Dungeon Layout 3

    In this example, I’ve included the minimum spanning tree generated during the dungeon building process. If you note the yellow corridors between rooms, those are randomly added between rooms within the same subtree.

    Applying Tiles

    The TileMap.SetTile function can then be used to programmatically fill out which tiles are supposed to go where. Once you’ve got the room data figured out, it’s pretty easy to determine where to set tiles (floor tiles set within any room or corridor, wall tiles set in any empty space that borders a corridor or room).

    The result:

    Dungeon Layout with tiles applied to the TileMap

    From here, you can generate the items and enemies how you see fit.

    Switching Careers to Software Engineering: Six Months Update

    written by Zach Morrissey on 2019-01-07

    Lessons Learned

    Now that I’ve been in my role for a while, it’s time to reflect on that period and try to put it in context of being in a fully different role roughly 1 year ago.

    The skinny:

    • I don’t regret it at all.
    • There’s a lot I don’t know.

    …and that’s basically it. I’m learning a lot every day. If you were expecting some sort of sea change in who you are after you get the job, it’s likely not going to be there. It’s a new set of challenges.

    You’re Good Enough

    The thing that’s taken me aback most of all upon switching to a full time software engineering role was how comfortable it feels. There’s a lot of practices I held near and dear to my heart on a personal level that have translated well into an engineering practice.

    My takeaways:

    • You’re likely a good enough engineer.
    • You probably know more about engineering as a job than you think that you do.
    • Your teammates are happiest when you’re resourceful. Research your questions before bugging people. Look for novel solutions. Use what’s available to you.
    • It doesn’t hurt to be open about what you’re looking for / dealing with.
    • Don’t be too entrenched in your opinions.
    • The real world of engineering is different than whatever source of information you’re reading about online.

    Don’t Expect Things to be Easy

    Like any job, software engineering comes with its own set of frustrations. You’re likely going to have to use software you don’t want to. You’re probably going to have to revive long-dead code that’s uncommented and unloved. You may even use unsexy languages / frameworks / approaches.

    That’s well-known enough that it should’ve been obvious, but until you’re there it’s much too easy to write things off.

    I’ve noticed the things that I struggle with the most aren’t even technical challenges. Sure, those are there. The majority of the issues, however, come from trying to be productive as a team. Solving things not just for myself but for the entirety of the organization. If I find a script that helps me debug something, that’s great, but it may only help me. Whereas a little bash script you use on your own machine may have been helpful before, there’s a lot of added difficulty if you want to share it with others, automate it, or make it a standard practice.

    Motivating and working with groups of people is not an easy practice, and there will always be discrepancies in what people believe is the best approach.

    Don’t Stop Learning

    Software engineering is a role where you can find yourself in a life of learning. Finding the job doesn’t mean that part is over, it means you’ve just passed the first level.

    There’s a lot you can do in your spare time, but the best thing you can do is occassionally challenge yourself to learn something new and try a different technology. I’ve found that the times I’m most productive are when I can recognized a pattern in one approach based on something I’ve done somewhere else. It’s a satisfying thing to see your curiosity rewarded, doubly so in programming.

    The obvious suggestions:

    • Take a course on Coursera, Udemy, etc.
    • Build a personal project.
    • Try to see if you can improve something you wrote in the past in a new framework / language / paradigm.
    • Benchmark something and see if you can make it faster.
    • Refactor an old set of code for maintainability. Make it easier for others.

    This is not comprehensive, nor does it fit everyone, but it should give you an idea of things you can do that aren’t just the “Get ticket, do ticket” type engineering it’s easy to fall into. People love to see you excited about things, and being passionate will make you happier.

    What’s next?

    Frankly, I don’t know. I’m optimistic.

    I’ll follow this up with another check-in at the one year mark.