Headless Obsidian Sync for an AI Vault

Fri, 29 May 2026 00:00:00 +0000

I wanted my agent to write into the same Obsidian vault I use on my laptop, without giving the container direct access to my desktop. The setup ended up being pretty simple:

a host-owned vault directory on the VPS
a headless Obsidian Sync container
the Hermes container mounting that same vault path
a couple of basic permission guards

That keeps Hermes inside the same note-taking flow I already use, while still keeping the host boundary clean.

The storage model

Hermes mounts the host tree directly into the container:

volumes:
 - ${HERMES_HOME_DIR}:/opt/data

The vault lives under that tree, so anything Hermes writes to /opt/data/vaults lands in the shared vault directory.

The sync bridge

The actual sync layer is a headless Obsidian Sync container:

services:
 obsidian-sync:
 image: ghcr.io/belphemur/obsidian-headless-sync-docker:0.0.8@sha256:...
 restart: unless-stopped
 cap_drop:
 - ALL
 cap_add:
 - CHOWN
 - SETUID
 - SETGID
 - DAC_OVERRIDE
 security_opt:
 - no-new-privileges:true
 environment:
 OBSIDIAN_AUTH_TOKEN: ${OBSIDIAN_AUTH_TOKEN}
 VAULT_NAME: ${VAULT_NAME}
 VAULT_PASSWORD: ${VAULT_PASSWORD:-}
 PUID: ${PUID:-1000}
 PGID: ${PGID:-1000}
 VAULT_PATH: ${VAULT_PATH:-/vault}
 DEVICE_NAME: ${DEVICE_NAME:-obsidian-docker}
 CONFLICT_STRATEGY: ${CONFLICT_STRATEGY:-merge}
 volumes:
 - /home/hermes/vaults:/vault
 - ${CONFIG_HOST_PATH:-./config}:/home/obsidian/.config

This is the bridge to my Obsidian Sync peers. It mounts the same vault path Hermes uses, so edits move both ways:

laptop changes sync to the VPS
Hermes-written notes sync back to the laptop

A small but important detail

Before the containers start, the host directories are created with the right ownership. That avoids the usual first-run Docker problem where a root-owned directory breaks later writes.

The security boundary

The important part is that the agent is not running as my admin user on the host.

hermes is an unprivileged host user
vault and config dirs are owned by hermes
the sync container has no-new-privileges
capabilities are trimmed aggressively
the mount is local to the host, not exposed over a public port

So the note flow stays convenient without making the vault a loose shared filesystem.

The core idea

In practice, the architecture is:

laptop Obsidian
 ↕ Obsidian Sync
headless sync container on VPS
 ↕ vault directory
Hermes container

The point is simple: the agent and the human editor write into the same vault, but through a controlled host-side bridge.

A great pattern for anyone who want an agent to work inside your notes when the agent runs remotely and you don’t share a filesystem

Implementing Scalable GitOps With Argo CD and ApplicationSets: A Case Study

Sun, 28 Apr 2024 00:00:00 +0000

This infrastructure was developed at Nexxen, where I’m currently employed.

Transitioning from the inefficient “App of Apps” model, we adopted GitOps with Argo CD and ApplicationSets for improved efficiency and reliability. The old system, with its hunderds of declarative Argo Applications and frequent manual changes them, was operationally taxing, error-prone and not scalable.

The shift to a declarative GitOps approach using ApplicationSets has significantly minimized manual interventions, enhancing the scalability and accuracy of our deployment processes.

In this blog post, I’ll explore our approach to establishing a scalable and maintainable GitOps infrastructure using the Argo CD ApplicationSets CRD.

I will delve into how we:

Bootstrap and manage Argo CD configurations.
Structure our ApplicationSets and GitOps deployments repository for scalability (utilizing the git generator).
Leverage ApplicationSets to automate deployment processes.
Manage and scale our deployments across multiple clusters and environments.

I provided real examples, from two teams, Octo and Datamine2, to give you a better understanding of how we structure our deployments.

Introduction to Our GitOps Setup

We divided our configurations into two repositories.

The argocd-management repository, accessible only to system administrators, contains sensitive Argo CD configurations.
Developers mainly use the gitops-deployments repository for deployment manifests.

This strategy safeguards our management configurations and aligns with the principle of least privilege.

Our GitOps framework is built around two main repositories:

`argocd-management` repository:

Manages Argo CD’s foundational configurations, including AppProjects and ApplicationSets.

Example directory structure

├── app-of-appprojects.yml
├── app-of-appsets.yml
├── appprojects
| ├── [more team based directories]
│ ├── octo
│ │ ├── appproject-infra-non-prod.yml
│ │ ├── appproject-infra-prod.yml
│ │ ├── appproject-orphans-non-prod.yml
│ │ ├── appproject-orphans-prod.yml
│ │ └── kustomization.yaml
│ ├── sunflower
│ │ ├── appproject-datamine2-non-prod.yml
│ │ ├── appproject-datamine2-prod.yml
│ │ └── kustomization.yaml
│   └── kustomization.yaml
├── appsets
| ├── [more team based directories]
│ ├── octo
│ │ ├── appset-infra-non-prod.yml
│ │ ├── appset-infra-prod.yml
│ │ ├── appset-orphans-non-prod.yml
│ │ ├── appset-orphans-prod.yml
│ │ └── kustomization.yaml
│ ├── sunflower
│ │ ├── appset-datamine2-non-prod.yml
│ │ ├── appset-datamine2-prod.yml
│ │ └── kustomization.yaml
│   └── kustomization.yaml
└── bootstrap.yml

`gitops-deployments` repository:

Acts as the single source of truth for all declarative deployments within our organization.

This is a monorepo gitops repository that contains all the Kubernetes manifests that are used/deployed across all the company.

Example directory structure

deployments
│
├── [more team based directories]
├── octo
│ ├── infra
│ │   ├── devlake
│ │   │   └── envs
│ │   │   └── prod
│ │   │   ├── appset_config.json
│ │   │   ├── kustomization.yaml
│ │   │   ├── ui_deployment_patch.yml
│ │   │   └── values.yml
│ │   └── vault-auth
│ │   └── envs
│ │   └── prod
│ │   ├── appset_config.json
│ │   ├── kustomization.yaml
│ │   └── resources.yml
│ └── orphans
│ ├── aws-search
│ │   ├── base
│ │   │   ├── deployment.yml
│ │   │   ├── kustomization.yaml
│ │   │   └── service.yml
│ │   ├── envs
│ │   │   └── prod
│ │   │   ├── appset_config.json
│ │   │   └── kustomization.yaml
│ │   └── variants
│ │   └── prod
│ │   ├── deployment.yml
│ │   └── kustomization.yaml
│ └── another-app
│    ├── base
│    │   ├── deployment.yml
│    │   └── kustomization.yaml
│    └── envs
│    └── prod
│    ├── appset_config.json
│    ├── consul_configmap.yml
│    ├── consul_deployment.yml
│    └── kustomization.yaml
│
└── datamine2
 ├── datamine-amoeba-appprd
 │   ├── base
 │   │   ├── deployment.yml
 │   │   ├── kustomization.yaml
 │   ├── envs
 │   │   ├── non-prod-sandbox
 │   │   │   ├── appset_config.json
 │   │   │   ├── kustomization.yaml
 │   │   ├── non-prod-staging
 │   │   │   ├── kustomization.yaml
 │   │   └── prod
 │   │   ├── appset_config.json
 │   │   ├── kustomization.yaml
 │   └── variants
 │   ├── prod
 │   │   ├── env.yml
 │   │   ├── kustomization.yaml
 │   ├── sandbox
 │   │   ├── env.yml
 │   │   ├── kustomization.yaml
 │   └── staging
 │   ├── env.yml
 │   ├── kustomization.yaml
 └── datamine-amoeba-catalog
    ├── base
    │   ├── deployment.yml
    │   ├── kustomization.yaml
    ├── envs
    │   ├── non-prod-sandbox
    │   │   ├── appset_config.json
    │   │   └── kustomization.yaml
    │   ├── non-prod-staging
    │   │   └── kustomization.yaml
    │   └── prod
    │   ├── appset_config.json
    │   └── kustomization.yaml
    └── variants
    ├── prod
    │   ├── env.yml
    │   ├── kustomization.yaml
    ├── sandbox
    │   ├── env.yml
    │   ├── kustomization.yaml
    └── staging
    ├── env.yml
    ├── kustomization.yaml

Our Approach to Bootstrapping and Managing Argo CD Configurations

Although ApplicationSets offer significant reductions in time and manual errors by automating deployment processes, integrating them with Kustomize has elevated their effectiveness.

By managing ApplicationSets and AppProjects through Argo CD applications themselves (app-of-appsets and app-of-appprojects) we have automated even the application of these configurations to the cluster. This meta-management layer ensures that all changes are self-maintaining and self-applying, which enhances automation and reduces the need for manual oversight even further.

We utlize the power of the Application CRD to manage the AppProjects, and the Application factories, the ApplicationSets!

Below, I’ll explain our approach more in detail:

We start our setup with a bootstrap operation defined in the bootstrap.yml file, which contains all necessary configurations for initializing Argo CD.

At the time of writing this post, it only contains an AppProject manifest (named management), that we use for the two root YAMLs I will mention below.
In order to reconstruct our setup, the only manual steps we need to perform are the following:
1. Apply the bootstrap configuration:
```
kubectl apply -f bootstrap.yml
```
2. Set up management for AppProjects and ApplicationSets:
```
kubectl apply -f app-of-appprojects.yml
kubectl apply -f app-of-appsets.yml
```
To make changes (add, delete, or edit), simply update the appprojects and appsets directories using the kustomize structure. Each change will be automatically detected by either the app-of-appprojects or app-of-appsets respectively.

Diagram: How It’s Working Together

Understanding Our ApplicationSet Approach

Generators: Using generators, the ApplicationSet constructs a dynamic list of Argo CD applications to be managed. These generators use patterns and parameters defined in configuration files (like appset_config*.json that you’ll see below) to determine which applications should exist (or be deleted).
Scalability: This approach enables the ApplicationSet to scale efficiently, as it can automatically adjust the number of managed applications based on the repository’s current state. New applications are created, and outdated ones are removed without manual intervention.
Considerations:
- Any directory that won’t have the appset_config*.json file or that won’t match its appropriate ApplicationSet, will be ignored by the ApplicationSet.
- It’s also important to make sure that no other ApplicationSet is targeting the same directory, as it may lead to conflicts and unexpected behavior.

Our ApplicationSets Strategies

We have two main strategies for our ApplicationSets:

GitLab Subgroup-based `ApplicationSet`s

ApplicationSets are primarily organized per GitLab subgroup, automatically generating Argo CD Applications for each service or application within the subgroup.

“Orphans” ApplicationSets (for applications/repos that do not belong to any subgroup) As we have many applications that do not belong to any subgroup, we handle them as “orphans” (both in terms of ApplicationSets and in the gitops-deployments repo).

Rationale: We encountered issues when creating an ApplicationSet that watches the root hierarchy of the group, as it did not behave as expected. Instead, we decided to separate the “orphans” into a different directory and handle the path logic in the CI using simple bash code. For more details on how we resolved this issue, refer to the Q&A section.

Prod and Non-Prod Environments

The production environments are handled by ApplicationSets that target directories with a prod prefix in their paths, and non-production environments with a non-prod prefix.

Rationale: By using prefixes for prod* and non-prod*, we can effectively manage different types of deployments in separate ApplicationSets. This approach also ensures that the user only needs to add these prefixes when they want the deployments to be picked up and watched by the corresponding ApplicationSet for deployment.

Examples `ApplicationSet` Manifests

Real example ApplicationSets config that we use for our own DevOps team (“Octo”):

Infra:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
 name: octo-infra-prod
 namespace: argocd
spec:
 generators:
 - git:
 repoURL: git@git.company.com:octo/gitops-deployments.git
 revision: main
 files:
 - path: deployments/octo/infra/**/envs/prod*/appset_config*.json
 template:
 metadata:
 name: "octo-{{path[3]}}-{{cluster.name}}-{{env}}"
 finalizers:
 - resources-finalizer.argocd.argoproj.io
 namespace: argocd
 spec:
 project: octo-infra-prod
 source:
 repoURL: git@git.company.com:octo/gitops-deployments.git
 targetRevision: main
 path: "{{path}}"
 destination:
 server: "{{cluster.address}}"
 namespace: "{{namespace}}"
 syncPolicy:
 automated:
 selfHeal: true
 prune: true
 syncOptions:
 - CreateNamespace=true

Orphans:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
 name: octo-orphans-prod
 namespace: argocd
spec:
 generators:
 - git:
 repoURL: git@git.company.com:octo/gitops-deployments.git
 revision: main
 files:
 - path: deployments/octo/orphans/**/envs/prod*/appset_config*.json
 template:
 metadata:
 name: "octo-{{path[3]}}-{{cluster.name}}-{{env}}"
 finalizers:
 - resources-finalizer.argocd.argoproj.io
 namespace: argocd
 spec:
 project: octo-orphans-prod
 source:
 repoURL: git@git.company.com:octo/gitops-deployments.git
 targetRevision: main
 path: "{{path}}"
 destination:
 server: "{{cluster.address}}"
 namespace: "{{namespace}}"
 syncPolicy:
 automated:
 selfHeal: true
 prune: true
 syncOptions:
 - CreateNamespace=true

These ApplicationSets targets the orphans/infra directory within the octo subgroup in the gitops-deployments repo, focusing on the prod environment.

For more details about the appset_config*.json files, refer to the Q&A section.

Scale Strategies with ApplicationSets

Our architecture leverages ApplicationSets to facilitate scalability in the following ways:

Deploying New Services: Adding a new directory in the gitops-deployments, to a path crawled by a specific ApplicationSet will make the ApplicationSet add another crawl (thanks to the globbing ** in the path), and so, if the rest of the path is matching the regex/path in the rest of the path, it will add your new service, including all its environments.
Deploying the Same Service to Multiple Clusters: Adding a new JSON configuration file (matching the regex appset_config*.json) in the relevant envs directory automatically triggers the creation of a new Argo Application tailored to the specific parameters defined within the file.

This makes scaling out to additional clusters or deployment environments seamless.
Multi-environment Management (Prod/Non-Prod): By defining separate paths for different environments (like prod, non-prod-staging, non-prod-sandbox), ApplicationSets can manage deployments across multiple environments efficiently, creating distinct sets of applications for each environment.

Example of Environments (`envs`) Structure

The envs directory structure plays a crucial role in how ApplicationSets manage deployments:

envs/
├── non-prod-sandbox/
│ ├── appset_config-a.json
│ ├── appset_config-b.json
│ ├── kustomization.yml
├── non-prod-staging/
│ ├── appset_config-c.json
│ ├── kustomization.yml
├── prod/
 ├── appset_config-d.json
 ├── kustomization.yml

Q&A

Q: How did you overcome the difference between the GitLab repositories heirarcy and the gitops-deployments repo heirarcy?
A: we implemented in the CI, a logic that checks if the repository is in the “root” / group level, and then adds to its path in gitops-deployments, the orphans/ directory, so it will hit the right path in the gitops repo.
- Snippet from our Gitlab CI’s before_script logic:
```
if [[ $CI_PROJECT_PATH =~ ^[^/]+/[^/]+$ ]]; then
 FULL_PROJECT_PATH="$CI_PROJECT_NAMESPACE/orphans/$CI_PROJECT_NAME"
else
 FULL_PROJECT_PATH="$CI_PROJECT_PATH"
fi
```

Q: What are the appset_config*.json files?
A: The appset_config*.json files provide additional parameters for the ApplicationSet; that way we can combine both the ones we get from the git generator and the ones we define in the JSON file.
- Example appset_config.json file:
  - taken from deployments/sunflower/datamine2/datamine-amoeba-appprd/envs/prod, from the gitops-deployments repo
  - Of course, you can structure your ApplicationSet and your JSON file in any way that suits your organization’s needs.
```
{
 "env": "prod",
 "cluster": {
 "name": "dc1-november",
 "address": "https://10.1.1.1:6443",
 "dc": "dc1"
 }
}
```

Q: How do we integrate Helm in our deployment approach?
A: Our deployment approach integrates Helm, allowing us to manage Helm-based and native Kubernetes deployments seamlessly. Key integration points include:
- Enabling Helm in Kustomize: We modify the argocd-cm ConfigMap to include the --enable-helm flag in the Kustomize build command.
- Example Kustomization Config:
```
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
helmCharts:
 - name: devlake
 repo: https://apache.github.io/incubator-devlake-helm-chart
 releaseName: devlake
 namespace: devlake
 version: 1.0.0-beta3
 valuesFile: values.yml
```

Q: How do we integrate Helm in our deployment approach?
A: Helm deployments work in a similar way to normal applications, where the entrypoint is the environment’s kustomization.yml file. This file should include a helmCharts section. Key integration points include:

This enables flexibility by allowing Helm-based deployments to be patched and to create native K8s resources along the Helm releases.
- Enabling Helm in Kustomize: We modify the argocd-cm ConfigMap to include the --enable-helm flag in the Kustomize build command.
- Example Kustomization Config:
```
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
helmCharts:
 - name: devlake
 repo: https://apache.github.io/incubator-devlake-helm-chart
 releaseName: devlake
 namespace: devlake
 version: 1.0.0-beta3
 valuesFile: values.yml
```

Obsidian on aviadhaham.me