Adventures in Self-hosting

Well folks, I’m back! Unfortunately it’s because I got laid of and thus I have a lot more free time. I’ve decided to use the time off to try to finish up some personal projects, and brush up on a few skills to add to my resume.

Today I’m going to give a tour of my self-hosting project. This has been on ongoing project in my house for the past year or so. I originally started it as a way to make sure my home automation systems could function even if the Internet connection went down. It’s grown since then, and I now have nearly every service I use on a daily basis hosted locally.

Without further adieu, let’s dive in!

My Data Center

Originally, all of my servers were located in my home office, and connected to an ethernet switch in the basement. This was convenient for doing work on the servers, but it made my tiny office very crowded, and the heat was difficult to tolerate in the summer. I realized I simply had to put this stuff somewhere else; the question was where? There was no really good place on the main floor, and the upstairs does not have any network drops. In the end I realized the only place for it all to go was the basement.

My house is old (at least 120 years), and the basement is only partially finished. To protect the equipment I ended up building a tiny (16 square foot) room just big enough for a rack. The floor and back wall have been epoxy sealed and painted to resist moisture. It isn’t fully enclosed yet, but my plan is to mostly enclose it, and add a blower to blow filtered air into the room and use the resulting positive air pressure to keep basement dust out.

For power I ran a dedicated 20-amp circuit to a single outlet on the back wall. Using a 20-amp circuit gives me more headroom, plus it means I can potentially upgrade to a beefier UPS down the road.

Here’s how it looks as of 11/16/2024:

Please pardon the mess; I still have some rewiring to do!

The rack is a four-post 27U unit I picked up on Amazon (I would have preferred a full-size rack, but the basement doesn’t have enough clearance). In this rack I have the following:

  1. A 48-port cat 7 patch panel. All of the existing ethernet drops in the house will terminate here eventually.
  2. A Juniper EX3400 48-port PoE switch, with four 10GBase-T modules installed, and dual power supplies.
  3. An Intel N5105 rack mount server with six 2.5Gbase-T ports, running OPNsense.
  4. My NAS. This is a custom Intel i3-9100F server with 64 GB of RAM. It has five 9.1 TB spinning rust drives.
  5. Another custom build, this one is an Intel i7-13700K with 64 GB of RAM and an RTX4090 GPU
  6. Two refurbished Dell PowerEdge R630s with dual 28-core Xeon processors, 128 GB of RAM, dual power supplies, and iDRAC 8s.
  7. A small NUC I decommissioned a while back and now use as a CheckMK server.
  8. An Eaton 1500VA UPS. It can’t power the entire rack for long, so everything shuts down gracefully when power fails.

The i7-13700k and the two R630s comprise a three-node Proxmox cluster. All user-facing services run as VMs on this cluster. Storage for the VMs is provided by the NAS, which runs TrueNAS Scale. This setup allows me to move services between nodes easily.

To monitor everything I run CheckMK. This is running on a dedicated NUC and is not dependent on any other services being up to function. Alerts go to my email account and a Slack channel.

Virtual Machines

User-facing services are provided by virtual machines running in the Proxmox cluster. The services I currently host are:

Some services are on dedicated VMs, such as AgentDVR, Emby, and FreePBX. The less resource-intensive services run as Docker containers on a single VM. Most of these services are behind the Caddy reverse-proxy, which provides SSL for my services through a Lets Encrypt wildcard certificate.

Fault tolerance is provided in two ways. Critical services (FreeIPA and DNS) are setup as pairs of servers bound to separate Proxmox nodes. They utilize local storage on the nodes, so they continue to function in some capacity if the NAS is down. For everything else the shared storage is used and the services can migrate to another Proxmox node if necessary. The exception is services marked with “(GPU)” above; they are not redundant as only one node has a GPU.

All VMs are backed up daily by Proxmox to a dedicated TrueNAS share. From there, TrueNAS backs itself up to a set of Backblaze buckets.

Final Thoughts

My setup is good, but still has a few weak points: the switch, firewall, and NAS. Of these three, the only ones I have any immediate plans to fix is the NAS. Once I am gainfully employed again I’d like to pick up a couple more PowerEdge servers and load them up with NAS-grade SSDs to use as a pair of TrueNAS servers.

I’d also like to upgrade the UPS down the road, preferably to a rack-mount 20-amp model. I would like to eventually be able to keep the rack on battery power long enough to switch the circuit over to a generator. This is going to involve me adding a second service panel, and moving select circuits over to it. This panel would have a generator switch and an outdoor plug for connecting my generator. I had planned that for this past summer, but I ran out of time, so it will probably be next summer now.

As for software, there are several other things I’d like to add. In the very near term I will be setting up a Romm instance for playing retro games via a browser. Medium term I’d also like to set up an Authentik instance to provide single sign-on to as many services as possible.

Finally, and most importantly, I am working on thoroughly documenting everything, in case something unfortunate happens to me. I’ll go into more detail on that in a future post.

In my next post I will go more into my network configuration, including how I provide secure outside access.

If you have any questions. comments, or suggestions, you can comment below, or contact me via BlueSky at @area73.bsky.social.

November 16, 2024 by Projects 0

A Long Overdue Update

It’s been about 14 months since I last posted an update, and I want to check in here and let everyone know the the blog isn’t dead; I’ve just been incredibly busy.

Early in 2023 I started to realize that my job of ten years was just not a good fit for me anymore. I was stuck maintaining and extending old code and subject to the whims of what features the clients wanted. So, in the spring, I decided to jump ship, and took a position at a new startup called Positron AI. Like most startups the work hours are longer and so I haven’t had as much energy for personal projects. But,it’s also much more rewarding work and I don’t regret the decision one bit.

JRC-1

First and foremost, the name of the OS for this project was changed last year from JR/OS to jrcOS. The OS itself is not entirely “retro” anymore, being a Unix clone, so the new name is more fitting. I did have a brief blog post about this last year, but the rest of the post was so bad I ended up deleting it.

Though my new job has more or less destroyed my original jrcOS schedule, I have continued to work on it, albeit in small bursts. In fact, about three months ago I made a major decision — I decided to do the thing I said I wouldn’t do and start writing non-critical jrcOS code in C, using the Calypsi 65816 C compiler.

I thought about decision this long and hard, especially because I felt like I was giving up on my self-imposed challenge, and it’s the right decision. In the end, it wasn’t that doing everything in assembly was beyond my ability, but rather I just don’t have the patience for it (I’m rather neurodivergent). I’ve also had more time to look at the code Calypsi generates, and I’m decently happy with it. Changing over to C is going to let me meet my goals much faster and with less hassle.

Look for more jrcOS updates between now and spring.

XGS

This project, unfortunately, is now on the far back burner. I hesitate to call it officially dead, because I do have plans for the code base, but I’m not actively working on it at the moment. Tentative plans for some future date include a port that runs on a bare metal Raspberry Pi, with a native ARM CPU emulation.

Home Automation

“Wait,” you are probably now saying, “when did you ever blog about home automation?” The answer is I haven’t…yet.

Over the last year I’ve largely rebuilt my entire home network, with the goal of being more fault-tolerant of both power fluctuations and Internet outages. This is quite a bit harder than it sounds with so many products these days being cloud-based.

As part of my revamp I also wanted to delve more into home automation. I’ve dabbled in it before in the form of Phillips Hue light bulbs, but I wanted to start automating even more. I also wanted to add a voice assistant like Alexa, but without relying on a cloud connection.

At this point I’ve achieved my initial goals, thanks to Home Assistant and its large ecosystem of compatible hardware and open source projects. Even my voice assistant, Wintermute (IYKYK), runs on local hardware.

Throughout 2024 I’m hoping to start blogging more about the progress I’ve made and my future plans, including some of the products I’ve found that work with HA.

Closing Thoughts

I have some other, smaller projects mulling around in my brain that I may or may not get to this year. Anything I do I will try to remember to document here. Stay tuned.

February 26, 2024 by Electronics JRC-1 News Projects XGS

An Updated Roadmap for JR/OS

The origins of JR/OS date back to my ill-fated COLE-2 project. I had originally envisioned it as being a simple single-tasking OS that could run on a variety of hardware. There would be a BIOS layer that interfaced to the hardware, with the actual operating system interfacing to the BIOS. JR/OS would not provide much functionality other than device and file system management.

Over the last year it has become apparent that there will probably never will be a plethora of 65816-based systems. It’s just not as popular a target for hobbyists compared to the 6502. I decided that the only systems I should worry about supporting are my own. Thus the BIOS/OS separation fell away to become just JR/OS.

Freed from the constraints of having to run on a variety of differing hardware, a more ambitious goal for JR/OS began to crystallize. JR/OS will be a full multitasking OS with memory protection, virtual memory, and a POSIX user space environment. It goes without saying that this is a much bigger project than my original plans. However, I think the journey will be fun and educational even if it doesn’t get to its destination.

To maximize my chance of success I am going to do this in phases. The idea is to make JR/OS usable for user space programs as quickly as possible. I will then add functionality in manageable chunks. Each phase will add tangible functionality, which should hopefully keep me motivated to keep moving to the next phase.

Required Features

Before I delve into the phases of this road map i want to go over some requirements I have. These are in addition to the core OS features such as multitasking and memory protection:

  1. The system must be usable at power-on even without booting from storage. This means booting into a ROM-based BASIC interpreter.
  2. The BASIC interpreter must support storage (LOAD, SAVE, OPEN, etc).
  3. If storage is detected at boot time the system must attempt to boot from it.
  4. The system monitor must always be available.

If it is not obvious I am basing my design on the Apple II (or more accurately the IIGS). This should not be surprising, given my stated desire to create a modern evolution of the IIGS.

With these requirements out of the way let’s take a look at how I’m planning to build all this.

Phase 1: User space environment

The first phase is the easiest, and the one on which I will be starting work this week. In this phase I will get the user space environment fleshed out enough that it’s possible to write functional applications.

For me to consider this phase “done” the user space must provide the following minimum features:

  1. Full console I/O
  2. Memory management (allocate and deallocate memory)
  3. Device access (serial ports, SPI devices, SD cards)

Much, if not most, of this is already implemented in some form, but there are some gaps to plug. The largest of these is that memory allocation needs to be wrapped in some sort of standard library. This is because memory management will eventually move to user space, and I want the change to be transparent.

Phase 2: File system support

The next step will be to implement a file system. The ability to load and save files is critical for writing any truly usable software. Since the hardware only supports SD cards it makes sense to use FAT32 as my file system. The current JR/OS code already has some support for this: FAT32 partitions are detected and enumerated at boot time.

It is in this phase that I’ll also be adding the ability to automatically boot from storage. If no boot media is found then the built-in BASIC will be launched.

Phase 3: Preemptive multitasking

This is where things will start getting really interesting. In this phase JR/OS will gain support for multiple processes and preemptive multitasking, but without virtual memory or memory protection. In this sense it will resemble GNO/ME.

It is also in this phase that JR/OS will implement multiple user IDs and multiple privilege levels. Theoretically at this point JR/OS could support multiple simultaneous users by spawning a console on the second serial port. However, I am not actively planning on implementing this at this point.

One thing I will need to iron out in this phase is how to support the system monitor going forward. Currently the monitor is basically a user application running with root privileges. It uses the JR/OS direct page and stack, and directly accesses some JR/OS internals. The monitor will need converted to a regular application here and divorced as much as possible from JR/OS itself.

Phase 4: Virtual memory support

This phase will begin once JRC-2 hardware is available, as it will require a PMMU. In this phase I will move memory management into the user space runtime. JR/OS will continue to manage its own memory allocations, but only provide VM support for user space.

Phases 5+: ???

Beyond phase 4 my road map is currently undefined. A major part of these phases will likely be making fleshing out POSIX support in user space.

I may add networking support here, though it would probably be limited to a PPP link. I don’t even know where to begin building an Ethernet card.

Another very lofty goal that would fit in here somewhere is to implement a IIGS-compatibility mode in JR/OS. This is purely a pie-in-the-sky idea right now. I have barely scratched the surface of how this might be accomplished.

Final Thoughts

As I mentioned at the start, this is a very ambitious road map. I can’t even begin to estimate how long it will take, especially considering my unpredictable periods of burnout. Only time will tell…

December 29, 2022 by Electronics JRC-1 Projects 0

JRC-1, One Year Later

JRC-1

It has been just over one year since I assembled my first build of JRC-1. It’s also been that long since my last significant post here, but despite the radio silence I have been making steady progress and I’d like to share that now.

Firmware Updates

First, and most importantly, the firmware has seen a lot of improvement. This includes:

  • I made the ROM code 16-bit clean. Much of the ROM code originated on my COLE-1 board, which was a plain 65C02, and had been updated just enough to work on the 65816. Now, everything except low-level driver code runs in 16-bit mode by default.
  • I implemented a device manager and registered the built-in hardware as devices.
  • There is now a functioning SD card driver for reading and writing blocks.
  • Implemented the initial scaffolding of a DOS, including scanning devices at startup and reading DOS-style partition tables.
  • Created a built-in mini assembler reminiscent of the Apple IIGS mini assembler. There is still work to be done here but it is already usable for writing small programs.
  • I created an Apple IIGS-style memory manager to manage allocation of memory to both the OS and user space programs
  • The code base got a fair amount of reorganization as the amount of code continues to grow. This includes not only physical file layout but also changes to how the various layers of the operating system (core OS, device drivers, DOS, and the system monitor) interoperate.
  • Added a Makefile target for the memSIM2 ROM emulator. This USB device emulates the physical EEPROM and allows me to upload new code directly to the board (including a reset afterward) with no fuss. Testing a new ROM image can now be done in just seconds.

Hardware Updates

There have only been two hardware changes in the last year:

  • I installed a bodge wire from the 65SPI’s external clock input to PB7 on the VIA. SD cards (at least the one I am using) cannot reliably be initialized without starting with a slower SPI clock. The bodge allows the VIA to be programmed to provide a slow initialization clock for SD card initialization; once the card is initialized it switches back to the faster, PHI2-based clock.
  • I installed a 16 MHz crystal in the board, bringing the system clock up to 8 MHz. The board has been rock-solid at this speed for several months now. This is already better than I expected, and it’s possible I might be able to squeeze out another 1-2 MHz down the road.

Next Steps

My top priority right now is to finish the mini assembler, as having the ability to easily write code directly on the system makes testing ideas much easier. I am hoping to have this completed by the new year.

After the mini assembler I have a number of subprojects that need attention:

  • Building out the scaffolding for supporting task switching (multiple processes).
  • Fleshing out the user space environment so that actual user space programs can be written
  • Implementing a FAT32 driver
  • Developing a BASIC interpreter application. This will require the user space environment to be usable.

I also have an updated road map for the future of JR/OS as a whole. I had originally had that here, but this post was getting rather long, so look for another new post very shortly.

December 28, 2022 by Electronics JRC-1 Projects 0

The blog has moved (a little bit)

When I first transitioned my personal blog to the Area 73 name, the only reasonable domain name available was “area73.us”. At the time I was fine with it; it is short and easy to remember. However, it recently came to my attention that the “area73.org” and “area73.net” names have become available as well. Of course I immediately purchased them both, and how I am happy to report that this blog officially lives at area73.org! In fact, you can access it at any of the three domains, though they will all redirect to area73.org.

For the time being I have no plans to decommission the original area73.us, so any page links out in the wild will continue to work. However I still suggest updating your bookmarks to use the new canonical site name.

Now if only area73.com would become available…

November 5, 2022 by News

JRC-1 Has Officially Arrived!

I’m happy to report that the JRC-1 boards arrived a few days ago, and exactly as I predicted I had a build up and running by Christmas. Here it is, in all of its black solder mask glory:

JRC-1 as of 2021-12-23

I guess the third time’s a charm, because this is my third full build, and the board booted up with no hardware issues. It’s currently running at 4 MHz; it should run faster but I have not tried yet.

The board is not perfect; there are a couple minor problems with the layout. The footprints for the DS1813s (the reset and NMI controllers) are reversed, and the footprints for the expansion slots don’t fit the slot connectors I have. Neither problem is critical and both have easy workarounds, so at this point I am considering this design final. All of the the hardware design files (KiCad project and CPLD Verilog project), as well as the latest firmware code, are in the JRC-1 GitHub repository.

With the board up and running I am turning my attention to the firmware. Currently I’m working on cleaning up the code so that I can start the major work. Next up is getting the SD card driver working, and then work on JR/OS. That’s where the real fun begins!

December 27, 2021 by Electronics JRC-1 Projects 0

A Quick JRC-1 Update

Well folks, today the JRC-1 project reached a major milestone: the design is complete, and parts and PC boards have been ordered!

KiCad 3-D rendering of the final board design.

I should to have everything in my hands in about a week, and with some luck, a working assembled board a few days after that.

Ordering PC boards is always a bit nerve wracking for me. One can never be 100% sure that a board is going to work until you get them back from the fab house and try assembling one. In addition, the final JRC-1 board comes in at 200×120 mm, which is considerably larger (and more expensive) than any of my previous designs. Still, I am very relieved to have finally hit this milestone, and assuming no major problems arise with the design I can shift my focus to creating the BIOS and operating system.

Over the next few days I will be copying most of the information from my private JRC-1 planning document over to the JRC-1 home page. This will include information such as the memory layout, component information, and details on the expansion slots and user port. Stay tuned.

December 11, 2021 by Electronics JRC-1 Projects 0

Expansion Bus Woes

When I first start planning the JRC-1, my main goal was to produce a simple base design with which I could experiment with more advanced features down the road. Something akin to the original IBM PC, except with a 65816. Meeting that goal meant I needed some sort of expansion bus.

The Design

Before I started any design work I ran my original plans by the folks at the 6502 forums. There are a lot of very smart people there who have experience with these things, and I trust their advice. The recommendation was that I not bring the system buses out of the slots. Instead, I should create an expansion bus behind a 6522 VIA.

After a few iterations I settled on a bus design. My bus looked very similar to the typical bus you might find on a RAM or ROM chip. It has six address lines, eight data lines, a R/W signal, a slot enable, and an I/O strobe. A pulse on the I/O strobe signals the card to perform a transaction. This design was chosen to interface to as many available peripheral chips as possible with minimal additional logic. It also fits on a single VIA, so I could test the design on COLE-1+.

The Experiment

At this point I need to mention that the deal-breaker application for any expansion bus is that I can create a video card for it. Sp. I adapted the TIVI code that I wrote for the TinyFPGA board to run on my ULX3S and hooked it up to COLE-1+. The results were….less than encouraging.

The whole setup was terribly unreliable. Sustained writes would frequently result in missed or duplicated bus transactions and thus garbled frame buffer data. Hooking up a scope did not reveal anything wrong at the signal level. The signals were very clean, and there was very little ringing or cross-talk. Manually slowing things down in code did not help at all, either.

I have spent many hours in the past week trying to fix this issue without success. My working theory is that the relatively high rise/fall times on the strobe line may be problematic. At 50 ns it might very well be causing metastability issues inside the FPGA. Given enough time I could probably make this work, but my frustration level has reached maximum.

Moving Forward

After a lot of thought I’ve come to realization that my expansion bus is turning into the kind of project-killing feature that doomed COLE-2. The irony here is that I was trying to avoid this exact issue by having expansion slots in the first place!

If I want JRC-1 to actually be finished and built, then I need to move past this. Instead of trying to come up with the perfect design on my first try, I’m just going to build what I really want, and if it is unstable, or fails outright, then I learn from that and design a better version.

And so I have decided to go with an ISA-style bus that provides access to buffered versions of the main board buses. This should work fine for at least a couple of MHz, and anecdotal evidence from folks using RC2014 and ISA backplanes suggests I may be able to hit 8 MHz without too much trouble. The only way to tell for sure is to build the dang thing and see what happens!

September 26, 2021 by Electronics JRC-1 Projects 2

COLE-1+ is alive

I’m happy to report that the first build of COLE-1+ is up and running! JLCPCB created some beautiful boards and delivered them almost exactly a week after I put in my order. Here’s a photo of the assembled board:

There were only two fixes I had to make to the board after delivery. One is to add the missing pullup resistor on the ACIA IRQ line that I mentioned in my previous post; the other was a bodge wire to bring the A11 address line to the GAL. With those two fixes the board works exactly as expected.

Not only does this board run its original firmware just fine, it also runs a slightly modified version of the COLE–2 firmware, and this is the firmware version I am going to use going forward. My plan is to make this a universal BIOS for all of my future 65816 builds. This will allow me to continue development on the firmware while I plan out and design JRC-1.

In the end I am very happy with this build. I may do a re-spin of it down the road to correct the two errors I had to correct, and possibly add a power switch, just so that I can turn the unit off to pull the EEPROM without having to disconnect the USB serial cable. But, for now I’m going to concentrate on the firmware and on the JRC-1 design.

August 20, 2021 by COLE-1+ COLE-2 Electronics JRC-1 Projects 0

Introducing JRC-1…and COLE-1+

Apologies for the long silence on the blog; I’ve had work and health issues that ate up a considerable amount of time during the first half of the year. Fortunately that’s all over and I am returning to my many projects.

JRC-1

First, I’d like to introduce the JRC-1, or Josh’s Retro Computer 1. This is a direct successor to COLE-2, with the aim of actually bringing the design to completion. Along those lines I ditched some of the more complex features of COLE-2, including built-in video and the game/joystick ports. Instead, it has an expansion bus with three slots that can be used to add features down the road.

Unlike its predecessor, JRC-1 is a 5V design. I had previously moved to 3.3V in order to gain access to larger RAMs, but it added complexity, as some parts of the design still needed to be 5V.

At the moment JRC-1 is still in the early planning stage. Once I am confident my design can be implemented I will begin work on the schematic and PC board. I am planning to skip the home prototyping stage for this project, and instead go directly to a manufactured PCB.

COLE-1+

As part of planning out JRC-1 I have been testing some design ideas by using my original COLE-1 board. However, that board is a bit problematic: it has power instability issues, and the footprint for the 6850 is wrong and required an ugly hack.

Since having a good test bed for my design ideas is important, I decided to spend a few hours fixing up the COLE-1 design a bit:

  • The 6850 now has the correct footprint
  • Added a large electrolytic capacitor to help smooth out the power supply
  • Replaced all of the discrete logic with a single 22V10D GAL. This also allowed me to tighten up the address decoding to make all but the bottom 1K of the ROM available.
  • Swapped out the CPU for a 65C816, because it was very easy to do so.
  • Removed the dedicated 1.8432 MHz crystal for the 6850. Instead, the CPU and the 6850 now share a 3.6864 MHz crystal.
  • Reduced the serial port speed to 57.6k bps, due to the use of a double-speed clock crystal.
  • Made the board layout more compact, and made all ICs align vertically.
  • Added some silkscreen to label the pins on the expansion headers.

In the interest of time I used the FreeRouting autorouter to route the board. It actually did a pretty decent job, and I only needed to make a few tweaks to some via locations. I may give it a try for JRC-1 down the road just to see how well it works on a larger design.

The final result of this work is officially the COLE-1+, as the feature set is almost identical to COLE-1. Boards are already on order from JLCPCB; I should have them in about a week. Hopefully I didn’t mess up anything this time around!

(Nope! It seems I left off a pullup resistor for the 6850’s IRQ line. Sigh. Fortunately this will be a very simple and relatively clean fix during assembly.)

August 10, 2021 by COLE-1 COLE-1+ COLE-2 Electronics JRC-1 Projects 0