iteggo

I Think Everything's Gonna Go Offline

Self-hosted website archiver for offline reading. Linux only.

Install

Prerequisites — Go 1.26+, gcc, sqlite3 dev headers:

# Debian / Ubuntu
$ sudo apt install golang-go gcc libsqlite3-dev

From source:

$ CGO_ENABLED=1 go install gitlab.com/whiteboardguy/iteggo/cmd/iteggo@latest

Using install script (recommended — detects distro, installs deps, offers service setup):

$ git clone https://gitlab.com/whiteboardguy/iteggo.git
$ cd iteggo
$ ./internal/scripts/install.sh

System service — install script offers, or manually:

# systemd
$ sudo ./internal/scripts/service/service_systemd.sh
# OpenRC
$ sudo ./internal/scripts/service/service_openrc.sh

Pre-built binaries not yet available. Binary installed to ~/.local/bin/iteggo (local) or /usr/local/bin/iteggo (system-wide). See README for manual build instructions.

Quick Start

# one-time setup
$ iteggo init

# archive a site (crawls homepage + linked pages within depth)
$ iteggo add --depth=1 https://example.com

# list your archives
$ iteggo list

# browse archives on localhost:8000
$ iteggo serve

Archives saved to ~/.local/share/iteggo/archives/. Config in ~/.config/iteggo/iteggo.conf.

Usage

iteggo init

Creates config directory, iteggo.conf, data directory, and SQLite database. Run once.

$ iteggo init

Dev mode — uses ./iteggo.conf.dev and ./data/ for local testing:

$ iteggo init dev

iteggo add

Archives one or more URLs. Crawls pages within --depth hops. Downloads linked assets (CSS, JS, images, video, fonts). Rewrites HTML and CSS to relative paths so archives work offline.

# archive homepage and immediate sub-pages
$ iteggo add --depth=1 https://example.com

# archive single page only (no crawling)
$ iteggo add --depth=0 https://example.com

# deep crawl with verbose output
$ iteggo add --depth=3 --debug https://example.com

# archive with per-URL blacklist (inline syntax)
$ iteggo add https://example.com blacklist/ https://ads.com https://tracker.net

iteggo list

Lists all archives in a table: URLs, depth, last updated, hash, job ID, finished status.

$ iteggo list

iteggo serve

Starts minimal archive browser on ITEGGO_SERVE_HOST:ITEGGO_SERVE_PORT (default 127.0.0.1:8000). Auto-refreshes every 10 seconds. Browse and delete archives.

$ iteggo serve

iteggo webui

Full web interface with HTMX — 3-panel layout (topbar, content, sidebar). Create archives, manage blacklists, view jobs, browse archive files. Keyboard shortcuts for power users.

$ iteggo webui

Keyboard shortcuts:

Key Action
Escape Close overlays / close sidebar
Ctrl+K Toggle keyboard shortcuts help
n / + Open new archive overlay
a Show archives page
j Show jobs page
b Show blacklists page
Delete Delete selected item in sidebar

iteggo delete

Deletes archive(s) by hash prefix. Interactively confirms before removing archive directory and database rows.

$ iteggo delete a1b2c3d4e5

iteggo config

View and update configuration.

# print resolved config with source path
$ iteggo config show

# get a single key
$ iteggo config get ITEGGO_SERVE_PORT

# set a key
$ iteggo config set ITEGGO_SERVE_PORT 9000

Flags

Flag Commands Description
-d, --depth add Crawl depth (default 1, max 10)
--debug add, serve, webui Enable debug logging / Gin debug mode

Configure

Config file: ~/.config/iteggo/iteggo.conf (or $XDG_CONFIG_HOME/iteggo/iteggo.conf). Dev mode: ./iteggo.conf.dev. Simple KEY=VALUE format, # for comments.

Blacklists directory: ~/.config/iteggo/blacklists/ (or $XDG_CONFIG_HOME/iteggo/blacklists/). Each file is a plain .txt with one URL pattern per line. Prepend DISABLED_BLACKLIST\n to disable.

Key Default Description
ITEGGO_DATA_DIR ~/.local/share/iteggo Where archives and database are stored.
ITEGGO_SERVE_PORT 8000 HTTP server port for serve and webui.
ITEGGO_SERVE_HOST 127.0.0.1 Bind address. Use 0.0.0.0 to expose on network.
DEBUG_MODE false Enable debug logging.
DEBUG_HARD_MODE false Verbose debug logging (implies DEBUG_MODE=true).
ASSET_RETRY_DEPTH 3 Max retries for 404 assets, re-resolving href up one directory each time (0 disables).
USER_AGENT Mozilla/5.0 ... Chrome/125.0.0.0 Safari/537.36 User-Agent header on all HTTP requests.

How It Works

Each archive is a directory in the filesystem with an ID derived from a random UUID. A SQLite database tracks metadata. Pages are crawled, assets downloaded, and URLs rewritten to relative paths so the site works offline.

Archive ID generation

Iteggo generates a random UUID v4, SHA-256 hashes it, and takes the first 5 bytes (10 hex chars). The first 4 chars become the first directory level, chars 6+ become the second level (char 5 skipped). Result: archives/a1b2/c3d4e5/. No two archives share the same path.

Crawling

Built on Colly. Starting from each target URL, iteggo visits linked pages within the configured depth boundary. External host hops count toward the depth limit. Every page is parsed for asset URLs (CSS, JS, images, video, fonts, iframes) which are downloaded and saved alongside the page. Assets with 404s are re-resolved up the directory tree up to ASSET_RETRY_DEPTH times.

Normalization

After all pages and assets are downloaded, iteggo walks the archive directory. HTML files are parsed and all href, src, srcset, and other URL attributes are rewritten to relative paths pointing to the downloaded copies. CSS files are similarly rewritten for url() and @import. The result is a fully self-contained offline snapshot.

Storage

Files are saved at <archive_path>/<hostname>/<path>. Extensionless HTML gets .html appended. Directory indexes land on index.html (or custom landing_page). The SQLite database stores the URL-to-path mapping, asset checksums, job status, and per-archive blacklist.

Serving

Both serve and webui use the Gin HTTP framework. Requested paths are resolved against the archive directory with path traversal protection — filepath.Clean is verified to stay within the archive root. Directories auto-serve their index.html. The default bind address is 127.0.0.1 for local-only access.

Technical Details

Security

No authentication. The serve and webui commands have no access control. Anyone who can reach the port can list, view, create, and delete archives. Default bind to 127.0.0.1 limits exposure. Front with nginx, Caddy, or similar for authentication and TLS.

Security headers are applied to all responses: X-Content-Type-Options: nosniff, Referrer-Policy: no-referrer, and a strict Content-Security-Policy allowing only same-origin resources with inline scripts and styles.

Path traversal is blocked. All archive file requests are cleaned with filepath.Clean and checked to stay within the archive root directory.

No TLS. Use a reverse proxy for HTTPS.

Database

SQLite with two tables managed by goose migrations.

Table Key Columns
jobs id (UUID), finished, created_at, updated_at, deleted
archives id (UUID), job_id, urls (JSON), archive_depth, archive_hash (unique), archive_path (unique), blacklisted_urls (JSON), landing_page, assets (JSON), urlToPath (JSON), pathToURL (JSON)

File format

Archived files are stored in their original format. HTML files have URL attributes rewritten to relative paths. CSS files have url() and @import references rewritten. No bundling or re-encoding — pages render as close to the original as possible.

Roadmap

  • CLI support
  • Local archive serving
  • Blacklists
  • Multi-URL archives
  • WebUI mode
  • System service (systemd / OpenRC)
  • JS-heavy page archives
  • Release v0.2.0 binaries
  • Incremental re-crawl
  • Search across archives
  • Windows support