2023-07-23

Self hosting Firefox's syncserver-rs

With Google's recent proposal for "Web Environment Integrity", which is a fundamentally terrible proposal, I no longer feel comfortable utilizing a Blink based browser (Blink is Chrome's engine and has basically been adopted by everyone). According some statistics that I quickly found, Blink has ~61% market share, followed by WebKit at ~34%. That's 95% of web traffic covered by Google's and Apple's browser engines. Unfortunately, WebKit is only really utilized by Apple as most of the alternative Browsers (like Opera, Brave, etc) have switched to Blink. Since many of my devices are not Apple devices, this leaves me with FireFox. I wish that I could say that Mozilla was a shining star of the open web but, they're complicit with the takeover of the web by the tech monopolies. With the other two options being worse, I am stuck with FireFox.

As I want to self host all of my data, the thing that came up was how to self-host the FireFox sync service. Searching for it will quickly come up with many results which point to syncserver. Right at the top of that project's readme is this:

Note that this repository is no longer being maintained. Use this at your own risk, and with the understanding that it is not being maintained, work is being done on its replacement, and that no support or assistance will be offered.

Luckily, there is an issue asking about what the replacement is with a link to another issue that describes what the replacement is. Following down that rabbit hole sends you to syncstorage-rs. As for how to self-host it, the documentation is rather sparse. Some intrepid self-hosters have figured out how to run it in docker in one of the issues. This does have most of what is needed however, the instructions are out of date since the 0.13 release of syncserver-rs. There are instructions in that thread for how to fix it but, it is unclear which of the values need to be changed to the new format and as a result, it took me a number of hours to get everything working.

This is what I've ended up with for environment variables:

RUST_LOG=trace
SYNC_HUMAN_LOGS="1"
SYNC_SYNCSTORAGE__DATABASE_URL=mysql://$(DB_USERNAME):$(DB_PASSWORD)@$(DB_HOST):3306/$(DB_NAME)
SYNC_SYNCSTORAGE__RUN_MIGRATIONS="true"
SYNC_HOST=0.0.0.0
SYNC_MASTER_SECRET=$(sync_key)
SYNC_SYNCSTORAGE__MASTER_SECRET=$(sync_key)
SYNC_TOKENSERVER__ENABLED="true"
SYNC_TOKENSERVER__RUN_MIGRATIONS="true"
SYNC_TOKENSERVER__NODE_TYPE=mysql
SYNC_TOKENSERVER__DATABASE_URL=mysql://$(USER_DB_USERNAME):$(USER_DB_PASSWORD)@$(DB_HOST):3306/$(USER_DB_NAME)
SYNC_TOKENSERVER__FXA_EMAIL_DOMAIN=api.accounts.firefox.com
SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL=https://oauth.accounts.firefox.com/v1
SYNC_TOKENSERVER__FXA_METRICS_HASH_SECRET=$(metrics_key)
SYNC_TOKENSERVER__ADDITIONAL_BLOCKING_THREADS_FOR_FXA_REQUESTS="10"

Now, for an explainer on what these thing control: - RUST_LOG sets the logging level. By default, I wasn't getting any logging output which made troubleshooting difficult. - SYNC_HUMAN_LOGS="1" sets the log format to be human readable. I would argue that these are not particularly readable even with this setting - SYNC_SYNCSTORAGE__DATABASE_URL=mysql://$(DB_USERNAME):$(DB_PASSWORD)@$(DB_HOST):3306/$(DB_NAME) sets the database url. Note that syncserver-rs only supports mysql and spanner. Spanner is off the table for self-hosters so, that leaves mysql. Mysql is an unfortunate regression, for me, from syncserver as I have a very slick setup for Postgres databases but not for mysql. - SYNC_SYNCSTORAGE__RUN_MIGRATIONS="true" ensures that the database migrations are run for the storage engine. I don't believe this is necessary. - SYNC_MASTER_SECRET=$(sync_key) and SYNC_SYNCSTORAGE__MASTER_SECRET=$(sync_key) this is the key for encrypting the data set to the storage engine. Reading the configuration, I'm unclear whether this needs to be in the namespaced naming format or not so, I just added it twice. - SYNC_TOKENSERVER__ENABLED="true" enables the token service. This is required for the self-hosted sync service to function. - SYNC_TOKENSERVER__RUN_MIGRATIONS="true" enables the token service to run its own migrations. - SYNC_TOKENSERVER__NODE_TYPE=mysql sets the database type to mysql. See previous discussion about the supported databases - SYNC_TOKENSERVER__DATABASE_URL=mysql://$(USER_DB_USERNAME):$(USER_DB_PASSWORD)@$(DB_HOST):3306/$(USER_DB_NAME) sets the token service's database url. Given that both services need to run migrations, it seems that this should be a separate database. - SYNC_TOKENSERVER__FXA_EMAIL_DOMAIN=api.accounts.firefox.com sets the trusted domain for logins to the official FireFox accounts service. See discussion below - SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL=https://oauth.accounts.firefox.com/v1 again sets the oauth server url to the official FireFox accounts service. - SYNC_TOKENSERVER__FXA_METRICS_HASH_SECRET - A separate secret for hashing metrics. I'm unclear what purpose this serve - SYNC_TOKENSERVER__ADDITIONAL_BLOCKING_THREADS_FOR_FXA_REQUESTS - Allocates additional threads for the token service. This allocates threads so that the OAuth public JWK can be fetched. 10 is likely excessive however, it is what the NixOS module sets it to

As mentioned this setup trusts the FireFox account service. At some point, I might go down the rabbit-hole of self hosting the accounts service as well. For now, it i fine for my setup to trust the official accounts service and store the data in my lab environment.

Of course, that is only one piece of getting the syncserver-rs running. For some people, that might be enough but, I'm going to provide the kubernetes manifest that I used to deploy syncserver-rs. This will likely not be applicable to many people as I don't believe that many self-hosters run kubernetes. In spite of that, this is the exact configuration I'm running and I don't want to provide something that isn't actually being used.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: default-policy
spec:
  egress:
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: '53'
              protocol: UDP
          rules:
            dns:
              - matchPattern: '*'
    - toEndpoints:
        - {}
  endpointSelector: {}
  ingress:
    - fromEndpoints:
        - {}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: truenas-encrypted-nfs
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: mariadb-conf
data:
  50-server.cnf: >
    #
    # These groups are read by MariaDB server.
    # Use it for options that only the server (but not clients) should see

    # this is read by the standalone daemon and embedded servers
    [server]

    # this is only for the mariadbd daemon
    [mariadbd]

    #
    # * Basic Settings
    #

    #user                    = mysql
    pid-file                = /run/mysqld/mysqld.pid
    basedir                 = /usr
    #datadir                 = /var/lib/mysql
    #tmpdir                  = /tmp

    # Broken reverse DNS slows down connections considerably and name resolve is
    # safe to skip if there are no "host by domain name" access grants
    skip-name-resolve

    # Instead of skip-networking the default is now to listen only on
    # localhost which is more compatible and is not less secure.
    #bind-address            = 127.0.0.1

    #
    # * Fine Tuning
    #

    #key_buffer_size        = 128M
    max_allowed_packet     = 1G
    #thread_stack           = 192K
    #thread_cache_size      = 8
    # This replaces the startup script and checks MyISAM tables if needed
    # the first time they are touched
    #myisam_recover_options = BACKUP
    #max_connections        = 100
    #table_cache            = 64

    #
    # * Logging and Replication
    #

    # Both location gets rotated by the cronjob.
    # Be aware that this log type is a performance killer.
    # Recommend only changing this at runtime for short testing periods if needed!
    #general_log_file       = /var/log/mysql/mysql.log
    #general_log            = 1

    # When running under systemd, error logging goes via stdout/stderr to journald
    # and when running legacy init error logging goes to syslog due to
    # /etc/mysql/conf.d/mariadb.conf.d/50-mariadb_safe.cnf
    # Enable this if you want to have error logging into a separate file
    #log_error = /var/log/mysql/error.log
    # Enable the slow query log to see queries with especially long duration
    #log_slow_query_file    = /var/log/mysql/mariadb-slow.log
    #log_slow_query_time    = 10
    #log_slow_verbosity     = query_plan,explain
    #log-queries-not-using-indexes
    #log_slow_min_examined_row_limit = 1000

    # The following can be used as easy to replay backup logs or for replication.
    # note: if you are setting up a replica, see README.Debian about other
    #       settings you may need to change.
    #server-id              = 1
    #log_bin                = /var/log/mysql/mysql-bin.log
    expire_logs_days        = 10
    #max_binlog_size        = 100M

    #
    # * SSL/TLS
    #

    # For documentation, please read
    # https://mariadb.com/kb/en/securing-connections-for-client-and-server/
    #ssl-ca = /etc/mysql/cacert.pem
    #ssl-cert = /etc/mysql/server-cert.pem
    #ssl-key = /etc/mysql/server-key.pem
    #require-secure-transport = on

    #
    # * Character sets
    #

    # MySQL/MariaDB default is Latin1, but in Debian we rather default to the full
    # utf8 4-byte character set. See also client.cnf
    character-set-server  = utf8mb4
    collation-server      = utf8mb4_general_ci

    #
    # * InnoDB
    #

    # InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
    # Read the manual for more InnoDB related options. There are many!
    # Most important is to give InnoDB 80 % of the system RAM for buffer use:
    # https://mariadb.com/kb/en/innodb-system-variables/#innodb_buffer_pool_size
    #innodb_buffer_pool_size = 8G

    # this is only for embedded server
    [embedded]

    # This group is only read by MariaDB servers, not by MySQL.
    # If you use the same .cnf file for MySQL and MariaDB,
    # you can put MariaDB-only options here
    [mariadbd]

    # This group is only read by MariaDB-11.0 servers.
    # If you use the same .cnf file for MariaDB of different versions,
    # use this group for options that older servers don't understand
    [mariadb-11.0]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    service: mysql
  name: mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      service: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        service: mysql
        app.kubernetes.io/name: mysql
    spec:
      containers:
        - env:
            - name: MARIADB_USER
              value: firefox_sync
            - name: MARIADB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: firefox-sync
                  key: DB_PASSWORD
            - name: MARIADB_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: firefox-sync
                  key: ROOT_PASSWORD
            - name: MARIADB_DATABASE
              value: firefox_sync
          image: mariadb:11.0.2-jammy
          name: mysql
          ports:
            - containerPort: 3306
              name: mysql
          volumeMounts:
            - mountPath: /var/lib/mysql
              name: mysql-data
            - mountPath: /etc/mysql/mariadb.conf.d/50-server.cnf
              name: mariadb-conf
              subPath: 50-server.cnf
              readOnly: true
          resources:
            requests:
              cpu: 1m
              memory: 125Mi
            limits:
              memory: 125Mi
      restartPolicy: Always
      volumes:
        - name: mysql-data
          persistentVolumeClaim:
            claimName: mysql-data
        - name: mariadb-conf
          configMap:
            name: mariadb-conf
---
apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
    - name: "mysql"
      port: 3306
      targetPort: mysql
  selector:
    service: mysql
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: firefox-sync-policy
spec:
  egress:
    - toFQDNs:
        - matchName: oauth.accounts.firefox.com
      toPorts:
        - ports:
            - port: "443"
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: firefox-sync
  ingress:
    - fromEndpoints:
        - matchLabels:
            app.kubernetes.io/name: traefik
            io.kubernetes.pod.namespace: traefik
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: firefox-sync
spec:
  replicas: 1
  selector:
    matchLabels:
      app: firefox-sync
  template:
    metadata:
      labels:
        app: firefox-sync
        app.kubernetes.io/name: firefox-sync
    spec:
      containers:
        - name: firefox-sync
          image: mozilla/syncstorage-rs:0.13.6
          env:
          - name: RUST_LOG
            value: trace
          - name: DB_USERNAME
            value: firefox_sync
          - name: DB_PASSWORD
            valueFrom:
              secretKeyRef:
                name: firefox-sync
                key: DB_PASSWORD
          - name: DB_NAME
            value: firefox_sync
          - name: DB_HOST
            value: mysql
          - name: SYNC_HUMAN_LOGS
            value: "1"
          - name: SYNC_SYNCSTORAGE__DATABASE_URL
            value: mysql://$(DB_USERNAME):$(DB_PASSWORD)@$(DB_HOST):3306/$(DB_NAME)
          - name: SYNC_SYNCSTORAGE__RUN_MIGRATIONS
            value: "true"
          - name: SYNC_HOST
            value: 0.0.0.0
          - name: SYNC_MASTER_SECRET
            valueFrom:
              secretKeyRef:
                name: firefox-sync
                key: SYNC_SECRET
          - name: SYNC_SYNCSTORAGE__HOST
            value: 0.0.0.0
          - name: SYNC_SYNCSTORAGE__MASTER_SECRET
            valueFrom:
              secretKeyRef:
                name: firefox-sync
                key: SYNC_SECRET
          - name: SYNC_TOKENSERVER__ENABLED
            value: "true"
          - name: SYNC_TOKENSERVER__RUN_MIGRATIONS
            value: "true"
          - name: SYNC_TOKENSERVER__NODE_TYPE
            value: mysql
          - name: USER_DB_USERNAME
            value: firefox_sync_user
          - name: USER_DB_PASSWORD
            valueFrom:
              secretKeyRef:
                name: firefox-sync
                key: USER_DB_PASSWORD
          - name: USER_DB_NAME
            value: firefox_sync_user
          - name: SYNC_TOKENSERVER__DATABASE_URL
            value: mysql://$(USER_DB_USERNAME):$(USER_DB_PASSWORD)@$(DB_HOST):3306/$(USER_DB_NAME)
          - name: SYNC_TOKENSERVER__FXA_EMAIL_DOMAIN
            value: api.accounts.firefox.com
          - name: SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL
            value: https://oauth.accounts.firefox.com/v1
          - name: SYNC_TOKENSERVER__FXA_METRICS_HASH_SECRET
            valueFrom:
              secretKeyRef:
                name: firefox-sync
                key: USER_SYNC_SECRET
          - name: SYNC_TOKENSERVER__ADDITIONAL_BLOCKING_THREADS_FOR_FXA_REQUESTS
            value: "10"
          ports:
            - name: http
              containerPort: 8000
              protocol: TCP
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: '1m'
              memory: 200Mi
          livenessProbe:
            httpGet:
              path: "/__heartbeat__"
              port: http
              scheme: HTTP
            initialDelaySeconds: 30
            timeoutSeconds: 8
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: "/__heartbeat__"
              port: http
              scheme: HTTP
            initialDelaySeconds: 30
            timeoutSeconds: 8
            periodSeconds: 10
            successThreshold: 5
            failureThreshold: 3
          imagePullPolicy: IfNotPresent
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app: firefox-sync
              topologyKey: kubernetes.io/hostname
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 100%
---
apiVersion: v1
kind: Service
metadata:
  name: firefox-sync
spec:
  ports:
    - name: http
      protocol: TCP
      port: 8000
      targetPort: http
  selector:
    app: firefox-sync
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: firefox-sync
spec:
  dnsNames:
    - firefox-sync.example.com
  issuerRef:
    kind: ClusterIssuer
    name: digital-ocean-production
  secretName: firefox-sync-tls
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: strip-prefix
spec:
  stripPrefix:
    prefixes:
      - /token
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: firefox-sync
spec:
  entryPoints:
    - tailscaletls
  routes:
    - kind: Rule
      match: Host(`firefox-sync.example.com`)
      middlewares:
        - name: strip-prefix
          namespace: firefox-sync
      services:
        - name: firefox-sync
          port: http
  tls:
    secretName: firefox-sync-tls

This does exclude a critical component, it requires a secret name firefox-sync containing the follow keys:

  • DB_PASSWORD - the password for the storage engine's db user
  • USER_DB_PASSWORD - the password for the token engine's db user
  • ROOT_PASSWORD - the password assigned for the mariadb root user
  • SYNC_SECRET - The previously mentioned key for encrypting the data
  • USER_SYNC_SECRET - The previously mentioned key for encrypting metrics

The database setup is less than ideal and requires a few manual steps to get going. It is less than ideal as it doesn't include any form of HA or backup. If it were a postgres database, I would get these things for free but, it is not. At this point in time, I'm unwilling to put more effort into this setup so, it will have to be as is. For me, losing this data is acceptable as my browsers have the same data locally and it can be recreated at some point in the future from the data that the browsers have locally.

As for the manual steps, there are a few. The environment variables on the database: MARIADB_USER, MARIADB_PASSWORD and MARIADB_DATABASE should result in that user being created with the specified password and the specified database should exist. It did not happen for me. I needed to manually create a second database and user for the token service anyways so, I just created them from the cli. The easiest way to do this is to shell into the mariadb container and run mariadb -u root which will get you into a sql shell. First is the setup for the storage server:

    CREATE DATABASE firefox_sync;
    CREATE USER 'firefox_sync'@'%' IDENTIFIED BY '{{password_in_secret}}';
    GRANT CREATE, ALTER, DROP, INSERT, UPDATE, DELETE, SELECT, REFERENCES on firefox_sync.* TO 'firefox_sync'@'%';

Then the token service databse:

    CREATE DATABASE firefox_sync_user;
    CREATE USER 'firefox_sync_user'@'%' IDENTIFIED BY '{{password_in_secret}}';
    GRANT CREATE, ALTER, DROP, INSERT, UPDATE, DELETE, SELECT, REFERENCES on firefox_sync_user.* TO 'firefox_sync_user'@'%';

Don't forget to do a FLUSH PRIVILEGES; after configuring the database users to load the changed permissions. We'll also need to configure some thing inside of the database in order to make it work. The database must be migrated before you can complete these steps so, make sure to check that firefox-sync is actually running. You can continue to use the same sql shell that you have been using:

    USE firefox_sync_user;
    INSERT INTO `services` (`id`, `service`, `pattern`) VALUES ('1', 'sync-1.5', '{node}/1.5/{uid}');
INSERT INTO `nodes` (`id`, `service`, `node`, `available`, `current_load`, `capacity`, `downed`, `backoff`) VALUES ('1', '1', 'https://firefox-sync.example.com', '1', '0', '1', '0', '0');

A quick explanation of what that is doing, we need to tell the sync engine what it is running. The data inserted into services should not be changed. The data inserted into nodes will need to be customized for your usage. The first 1 is the id of the node, this can be left as is. The second 1 is the id of the service and should also be left as is. https://firefox-sync.example.com is the host that your sync engine is available on and must be customized to match the accessible address. It does support specifying a port number and http if those are things that you require. The third 1 indicates how many accounts are available to be created. It needs to be at least one and could be more if you're providing service to multiple people. The first 0 is the number of accounts that are currently being utilized. This should be left at 0 and will be updated as accounts are created. The fourth 1 is the total number of accounts that this node will support. On creation the number that you should use is the same as the previous 1. The remaining values should be left as is.

Client Configuration

Desktop

The desktop configuration went smoothly for me. Just head to about:config (put it in the address bar), accept the warning that it gives you and then search for identity.sync.tokenserver.uri. Set this to https://firefox-sync.example.com/1.0/sync/1.5 (customizing the hostname / protocol to match your environment). After that, you can sign into your FireFox account and it will sync to your service. You can go to about:sync-log. If there are any errors with syncing with your sync service, they will be listed there in files named something like error-sync-1690097170544.txt. You can also check that data is being synced by running the following sql query:

    use firefox_sync;
    SELECT * FROM bso;

The data is encrypted so, it won't show anything that is particularly informative however, it should not show the result as empty set.

Android

Setting up the sync for Android was much less smooth and it took me nearly 6 hours working on it off and on. Mozilla, in their infinite wisdom has disabled the about:config interface in the Android version. Instead, you must follow a convoluted process to get into the Sync Debug menu. This post will walk you through how to get into the Sync Debug menu (tap the Firefox Browser header in the about screen in setting 5 times). The field that you want to change is Custom Sync server. That post will instruct you to use https://firefoxsync.example.com/token/1.0/sync/1.5 this is not correct, as you'll notice, it is not what we configured for the desktop. We don't want the token part. After many hours of testing, I finally noticed that the Android version seems to be hard coded to send to /token/1.0/sync/1.5 regardless of what the url in that menu is actually set to. I have mine set to simple https://firefox-sync.example.com. As a result of that hard coding, I inserted a strip prefix into my Traefik proxy which removes the /token from the url before forwarding the request to syncserver-rs (see the manifest from before to see how that is configured). After changing the sync server, you will have to restart FireFox before the setting will have an effect. Under the third option, there will be a button to do just that after you've made the change.

After restarting FireFox, you can sign into your FireFox account. In the Account settings screen in settings, there is a button to Sync now. If the Last synced: is not something along the lines of 0 minutes ago, there is something wrong with the sync. Before introducing the prefix strip, mine would alternate between Last synced: Never (after killing FireFox) and Last synced: December 31st, 1969 after hitting the Sync now button. Debugging any sync failures is difficult as Mozilla has removed the about:sync-log from the mobile version. The only way to get information about why sync is failing, that I am aware of, is to enable USB Debugging for your device and then use adb logcat to see the logs that your device is generating. Hopefully, the previous steps will allow your sync to just work without any debugging.

Discussion

Lemmy Post

2023-06-12

Leaving Reddit

I've always had mixed feelings about Reddit. One the one hand, it has always felt like a hold over from the older, weirder and less commercial times of the internet. On the other hand, it is a company and publishing on a company's product always has its risks. I've always tried to minimize the amount of work that I put into something that, ultimately, isn't under my control as there is always the possibility that they will do something shitty.

Apparently, Reddit's time to shit on the community is now. I won't detail the entire saga, instead I'll simply link to the Apollo Developer's announcement of shutting down Apollo. Reddit has seemed rather ambivalent towards its users. They have been pushing in directions that many people don't agree with. These are things like the "new" Reddit and pushing people from the web browser into the official app. Up until now, these have mostly been annoyances as you can get around them in some way. old.reddit.com still exists, instead of using the terrible official Reddit app, you could use an alternative, like Apollo. This ambivalence has now ended and is now replaced with hostility. With Apollo's departure, I am also departing Reddit.

The thing that has always seemed off about the way that Reddit has communicated this is they've indicated both that this is an insignificant part of their user base and that the opportunity cost of these people using a platform where they can't serve ads is enormous (see the breakdown in the Apollo thread). These points seem contradictory, how can they both be true simultaneously? The other thing, which has been pointed out continuously, is how completely unaware Reddit seems. Inherently, every social network depends on people's contributions to it. Of the big social networks, Reddit relies on their users more heavily than anyone else. Users are responsible for both content and nearly all moderation (I would say all but, Reddit does sometimes get involved with things). It seems incredibly unwise to poke these people in the eye.

This all feels incredibly unfortunate. Reddit is the last corporate that I have any investment in (emotionally, not monetarily). The loss of me, is wholly insignificant. My contributions to Reddit have been minimal. The collective of people that they've alienated is not so insignificant. Time will tell if Reddit is right that they'll be unharmed by doing this. It seems incredibly unlikely, to me, that it will turn out that way.

While I feel this loss on a personal level, I also feel like the internet as a whole is losing a valuable resource. Fairly frequently, Reddit has an answer to some obscure thing that I am looking for. This is likely, in part, due to their previous status as a holdover to a previous internet time. I see fairly frequent recommendations from people to append site:reddit.com to searches in order to get good results. On multiple occasions, I have used this trick to find things that I wouldn't have been able to find otherwise. Much can be said about whether the internet at large made a good decision by putting all of this work into a company's product but, it happened. None of that is to say that all of Reddit is great. An overwhelming majority of content on Reddit, isn't useful (which is perfectly fine).

Reddit can likely coast for a while, given all of their historical content, even with the ongoing blackout. I think it is important to send a message. My initial plan was to replace all of my contributions to Reddit with place holder content indicating my protest, should Reddit somehow meet my personal and totally reasonable demands, I could use the api to restore it. It didn't quite go as planned. The tool that I was using, Power Delete Suite doesn't appear to have worked properly. While it seemed to indicate that it had edited all of my contributions, it did not. Some of them were edited, most of them were not. Given how unlikely it is that they'll ever turn into the kind of place that I'd be willing to return to, I decided to just delete them instead. I first archived everything (again, it wasn't much) using reddit-user-to-sqlite. I then removed everything with Shreddit, specifically this fork. I've also removed Reddit from my search results, which is something that Kagi makes very simple.

I would highly suggest that you do the same. It is important to send a message. That message is something along the lines of "you don't get to earn money on people's content while shitting on them". While this is a fairly extreme measure, I believe that it is necessary. I also feel bad that this will mean that somethings will no longer be findable on search engines. Some of which may not be available anywhere else (other than archives of Reddit). This is a truly unfortunate consequence of the risk that we collectively took on when pouring so much work into Reddit. I believe that now is the time that we need to pull the only real lever that we have, removing our content. While the blackout will certainly hurt in the short term, the shutdown subreddits will either have their moderators changed out for people willing to work with them or whole new subreddits will appear in place of the old ones. Removing the existing content will make for a much longer term impact.

My Personal Demands

Finally, these are my totally reasonable requirements for me to return to Reddit:

  1. Give up on IPO, restructure into a Non-Profit in service of the community
  2. Reverse course and listen to the community
  3. Complete replacement of the executive team

See, these are all perfectly reasonable requests. It would take hardly any effort for them to accomplish this. Yeah, this is never going to happen so, I won't be returning.

So what is next? I'll probably spin up a Lemmy or Kbin instance. I'm really not interested in letting anyone else control my work.

Discussion

2023-04-22

Replacing Proxmox

I have been using Proxmox for about 5 years. In Feburary, I decided to stop using Proxmox. If you're expecting some compelling reason, you're not going to find one. I use my homelab for many different purposes. One of those is enjoyment, frankly I was bored with Proxmox and I wanted a change. That being said, are some of the things that I didn't like.

It is based on Debian. I don't have an issue with Debian or any traditional Linux distribution, I just think that using one is a poor choice for a hypervisor. Every manual change has the possibility of living forever. That config file you edited because a particular version of a package had a bug and you needed to work around it, did you remember to revert it when you updated the packages? Over time, the system drifts off of the image that Proxmox provides. Of course, there are ways to control this but, this is my homelab. I don't want to apply that level of rigor. I know that one of my hypervisors had a ceph service continuously restarting since I last used Ceph (a subject for another time).

As a result of the aforementioned accumulation of changes, I always found upgrades to be perilous. I had very few issues during upgrades, I could probably count the number of times that an upgrade caused an issue on one hand. It is more that each upgrade was incredibly stressful. Rolling back is not a simple task and spending hours troubleshooting an issue after an upgrade is just not something I'm particularly interested in doing.

This is highly subjective but, the UI always bothered me. It certainly works and a great deal of functionality is exposed but, I didn't enjoy it. The placement of various settings always felt a little strange to me and some operations would require me to go to many different pages to make the changes. I think the main problem, for me, is that the way the settings are grouped is not how I think about them. I did learn where everything was at but, it always felt suboptimal. Finally, I just feel like the interface is dated. Sure, a hypervisor doesn't need to be shiny but, this is also my homelab. I want to enjoy using it.

There has been great work done on a proxmox provider for Terraform but it is hampered by Proxmox's perplexing decisions around cloud-init. It makes some sense that the Proxmox console only exposes some commonly used cloud-init parameters. The API allows you to use a custom file and set whatever options you want and the terraform provider allows you to utilize this. The issue is that it needs to be a file. There isn't an api to upload the file. You have to place the file in one of Proxmox's stores and then set a reference to it in the vm configuration.

Another one of these Terraform limitations is related to cloning. You can clone a vm to any node that has access to the same storage. So, if you have a vm on shared storage, you can clone it to any member of your cluster. If it is using local storage, you can only clone it to the same node. Once it is cloned, you can migrate it to another node but that is two separate operations. This makes utilizing Terraform a bit difficult if you're trying to spin up multiple vms over multiple nodes. You either have to make a duplicate template on each member of your cluster or utilize shared storage. I preferred to utilize local storage for these things as the primary reason for utilizing Terraform was experimentation with different technologies. Being able to rapidly remake the things being managed was the draw. Duplicating the template over all of the members of the cluster was an annoyance. Proxmox could alleviate this by having a compound api action that first does the clone and then migrates to the targeted node.

It is incredibly likely that this isn't exhaustive. I probably should have written closer to the time that I started getting off of it but, the processes of getting off of it wasn't the smoothest and I wanted to have my replacement in place before writing this. If anything, the struggles in setting up my replacement and the subjectivity of my complaints is a ringing endorsement for Proxmox. I would not hesitate to recommend Proxmox to someone looking to start their homelab. It worked very well for me over the years.

So what did I end up with? The first thing was xcp-ng. Before I moved everything onto it, I thought it was great. Then, I ran into some problems and figured out that it does not work in the way that I want it to but, that is, again, a subject for another time. What I did settle on was k3s on top of nixos utilizing KubeVirt. This gets me no UI (at least built-in), a clunky Terraform experience, no built-in way to move local storage between nodes and a fairly convoluted networking setup. I fixed none of the issues and yet I like the change. A wholly irrational choice and sometimes, the homelab is about making irrational choices.

Discussion

2022-07-19

Running static sites from minio

One aspect of S3-like stores that I have found useful is the ability to serve static websites. Unlike S3 and Ceph RGW, doing this in minio requires some additional tooling. This website is hosted on my minio cluster and these are the steps that I took to make that happen.

  1. Create the bucket in minio
  2. Create a user that only has permissions to this bucket. (This isn't strictly required but, I like having a user per bucket).

    • Create a policy for the user, I give full permissions on the bucket (you can simply replace gnuherder.how with the name of your bucket):

      {
         "Version": "2012-10-17",
         "Statement": [
             {
                 "Sid": "VisualEditor0",
                 "Effect": "Allow",
                 "Action": [
                     "s3:GetBucketPolicyStatus",
                     "s3:GetEncryptionConfiguration",
                     "s3:GetLifecycleConfiguration",
                     "s3:PutObject",
                     "s3:DeleteObjectVersion",
                     "s3:GetBucketLocation",
                     "s3:GetBucketNotification",
                     "s3:GetObjectVersion",
                     "s3:GetObjectVersionTagging",
                     "s3:ListBucket",
                     "s3:ListBucketVersions",
                     "s3:ReplicateObject",
                     "s3:GetBucketObjectLockConfiguration",
                     "s3:GetBucketPolicy",
                     "s3:GetBucketVersioning",
                     "s3:ListBucketMultipartUploads",
                     "s3:DeleteObject",
                     "s3:GetObjectRetention",
                     "s3:GetObjectTagging",
                     "s3:GetObjectVersionForReplication",
                     "s3:ListMultipartUploadParts",
                     "s3:PutObjectRetention",
                     "s3:AbortMultipartUpload",
                     "s3:GetBucketTagging",
                     "s3:GetObject"
                 ],
                 "Resource": [
                     "arn:aws:s3:::gnuherder.how",
                     "arn:aws:s3:::gnuherder.how/*"
                 ]
             }
         ]
      }
      
    • Create the user and assign them to the group with this policy

  3. Install the mc utility

  4. Add an alias for your minio cluster mc alias set minio {{url to minio cluster}} {{username}} '{{password}}'
  5. Set the bucket policy to download to allow public downloads mc policy set download minio/gnuherder.how (again, replacing gnuherder.how with the name of your bucket)
  6. Upload the content to the bucket. I use the the aws cli to do so but, I'm not going to cover how to configure. I run AWS_PROFILE=rgw aws --endpoint-url http://minio.tobolaski.lan:9000 s3 sync --delete public/ s3://gnuherder.how/ where the AWS_PROFILE is the name of the profile I have configured with the minio credentials, --endpoint-url is the host where my minio cluster is running, public/ is the source directory of the content and s3://gnuherder.how/ is the bucket to put the content into.
  7. Now you'll need to have a reverse proxy in order to serve the content at the public dns address, i.e. gnuherder.how. I use nginx for this, this is my configuration:

    server {
        listen 0.0.0.0:443 http2 ssl ;
        listen [::0]:443 http2 ssl ;
        server_name gnuherder.how ;
        # Rule for legitimate ACME Challenge requests (like /.well-known/acme-challenge/xxxxxxxxx)
        # We use ^~ here, so that we don't check any regexes (which could
        # otherwise easily override this intended match accidentally).
        location ^~ /.well-known/acme-challenge/ {
            root /var/lib/acme/acme-challenge;
            auth_basic off;
        }
        ssl_certificate /var/lib/acme/gnuherder.how/fullchain.pem;
        ssl_certificate_key /var/lib/acme/gnuherder.how/key.pem;
        ssl_trusted_certificate /var/lib/acme/gnuherder.how/chain.pem;
        location / {
            rewrite ^/$ /index.html last;
            rewrite ^(.*)/$ $1/index.html last;
            proxy_set_header HOST object-store.net.tobolaski.com;
            proxy_set_header CONNECTION '';
            proxy_set_header AUTHORIZATION '';
            proxy_hide_header Set-Cookie;
            proxy_hide_header 'Access-Control-Allow-Origin';
            proxy_hide_header 'Access-Control-Allow-Methods';
            proxy_hide_header 'Access-Control-Allow-Headers';
            proxy_hide_header x-amz-id-2;
            proxy_hide_header x-amz-request-id;
            proxy_hide_header x-amz-meta-server-side-encryption;
            proxy_hide_header x-amz-server-side-encryption;
            proxy_hide_header x-amz-bucket-region;
            proxy_hide_header x-amzn-requestid;
            proxy_ignore_headers Set-Cookie;
            proxy_pass https://object-store.net.tobolaski.com:9001/gnuherder.how/;
        }
        proxy_buffering off;
    }
    

    The rewrite line is to handle going from url that end with / to /index.html which is how the files are stored in minio. It also removes a number of cookies that minio will add to the response which are not required for people to use the site. Be sure to do an nginx -t in order to test the configuration and then a systemd reload nginx to make the server active.

    One gotcha that I ran into was that nginx does not use the system's dns settings. In order to resolve a hostname to proxy to, you need to have resolver x.x.x.x; (where x.x.x.x is an ip address) set in a block. I have it set in the http block as I want it to be used for everything.

Note 2023-06-12

This site is no longer a static site running out of Minio. This was the last configuration I used before switching. It will likely work for quite a while but, eventually, some configuration option will change and it will stop working.

Discussion