Mosh (Mobile Shell) is a userland remote terminal application that provides resilient connections surviving network changes and interruptions. Unlike traditional system daemons, Mosh runs entirely in userspace, providing natural security boundaries and simplified deployment.

How Mosh Works

Userland Architecture

Mosh operates entirely in userspace, which is fundamentally different from traditional SSH:

  1. Initial Connection

    • Uses existing SSH for authentication
    • SSH launches mosh-server as your user (not root)
    • Server picks available UDP port
    • Communicates port and key back through SSH
    • Local mosh-client establishes UDP connection

  2. Security Benefits

    • No root daemon needed
    • Per-user process isolation
    • Sessions die with user logout
    • Uses unprivileged ports (>1024)
    • Natural user permission boundaries

This contrasts with sshd which runs as root, forks, and drops privileges.

Installation

# Debian/Ubuntu
sudo apt install mosh

# RHEL/CentOS
sudo dnf install mosh

# EndeavourOS
sudo pacman -S mosh

Tailscale Integration

Since Mosh runs in userland, it’s perfect for Tailscale integration. This restricts access to your VPN network only.

Server Configuration

Tell mosh-server to bind only to Tailscale:

# Add to /etc/profile.d/mosh.sh or ~/.bashrc
export MOSH_SERVER_NETWORK_TMPL="tailscale0"

Firewall Configuration

UFW (Ubuntu/Debian)

# Allow Mosh UDP only on Tailscale interface
sudo ufw allow in on tailscale0 proto udp from any to any port 60000:61000

# More restrictive - only from Tailscale IPs
sudo ufw allow in on tailscale0 proto udp from 100.0.0.0/8 to any port 60000:61000

firewalld (RHEL/CentOS/Fedora/EndeavourOS)

# Allow Mosh UDP only on Tailscale interface
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" interface=tailscale0 port port="60000-61000" protocol="udp" accept'
sudo firewall-cmd --reload

Session Management

Since mosh-server runs as your user, managing sessions is straightforward.

Listing User Sessions

# List your mosh-servers
pgrep -u $USER -a mosh-server

# More detail
ps aux | grep "[m]osh-server"

Terminating Sessions

# Kill specific session
kill <pid>

# Kill all your sessions
pkill -u $USER mosh-server

Automatic Cleanup

Prevent abandoned userland sessions:

# Add to crontab -e
# Kill idle mosh-servers after 24 hours
0 * * * * pkill -u $USER --full "mosh-server.*idle timeout"

Session Behavior

Mosh sessions terminate when:

  • Client disconnects cleanly
  • Timeout expires (default: 1 week)
  • User logs out (unless in screen/tmux)
  • Manual termination
  • System shutdown

Additional Security

fail2ban Configuration

Even though Mosh runs in userland, you can monitor connection attempts:

# Create /etc/fail2ban/filter.d/mosh.conf
[Definition]
failregex = ^.*Did not find request packet ID.*from <HOST>.*$
            ^.*Unknown message type.*from <HOST>.*$
ignoreregex =

# Add to /etc/fail2ban/jail.local
[mosh]
enabled = true
filter = mosh
logpath = /var/log/syslog
maxretry = 5
findtime = 600
bantime = 3600
port = 60000:61000
protocol = udp

Configuration

SSH Integration

Mosh uses your existing SSH config:

# ~/.ssh/config
Host myserver
    HostName 203.0.113.10
    Port 2222
    User myname
    IdentityFile ~/.ssh/special_key

Environment Variables

# Add to .bashrc or .profile
# Restrict UDP ports
export MOSH_UDP_PORT=60000:60010

# Remove [mosh] from terminal title
export MOSH_TITLE_NOPREFIX=1

Command Options

# Custom SSH port
mosh --ssh="ssh -p 2222" server

# Specific UDP port
mosh --server="mosh-server new -p 60001" server

# Custom idle timeout (1 hour)
mosh --server="mosh-server new -i 3600" server

Best Practices

  1. Userland Security

    • Remember Mosh runs as your user
    • Regular session audits
    • Use shortest necessary timeouts
    • Monitor user resource limits

  2. Network Security

    • Run only on Tailscale interface
    • Restrict UDP port range
    • Use fail2ban monitoring
    • Regular firewall rule audits

  3. Session Management

    • Implement automatic cleanup
    • Monitor active sessions
    • Set appropriate timeouts
    • Check resource usage

Mosh and tmux: Understanding the Difference

Mosh and tmux solve different problems and work exceptionally well together.

Mosh Capabilities

  • Network roaming (handles IP changes)
  • Survives sleep/wake cycles
  • Local echo for responsive typing
  • Handles intermittent connectivity
  • Syncs terminal state over unreliable networks
  • Limitations:
    • Can’t reattach if client crashes
    • Session dies if you close your laptop/terminal
    • No session sharing

tmux Capabilities

  • Session persistence across disconnections
  • Multiple windows/panes management
  • Session sharing between users
  • Survives complete client crashes
  • Limitations:
    • No local echo
    • No handling of network changes
    • Requires stable connection

Why Use Both?

The tools complement each other perfectly:

  • Mosh handles the network layer
  • tmux handles session persistence
  • Together: resilient AND persistent sessions

Common use cases for combining them:

# Long-running processes that can't fail
mosh server -- tmux new -s compile
make && make install  # Safe from network/laptop issues

# Shared development sessions
# Developer A:
mosh server -- tmux new -s pair
# Developer B:
mosh server -- tmux attach -t pair

# Multiple project workspaces
mosh server -- tmux new -s project1
# Later:
mosh server -- tmux attach -t project1

Best Practices for Combined Usage

  1. Session Organization

    • Use meaningful tmux session names
    • One project per tmux session
    • Use tmux windows for subtasks

  2. Resource Management

    • Remember each mosh+tmux combination uses two processes
    • Monitor both mosh-server and tmux memory usage
    • Clean up unused sessions regularly

  3. Team Usage

    • Document session naming conventions
    • Consider tmux user permissions
    • Use separate sessions for different tasks

Troubleshooting

Enable verbose logging:

mosh --ssh="ssh -v" server

Common issues:

  1. Connection Problems

    • Check SSH connectivity
    • Verify Tailscale status
    • Check UDP port availability
    • Confirm user permissions

  2. Resource Issues

    • Check user process limits
    • Monitor UDP port usage
    • Verify system resources
    • Check user session limits

Conclusion

Mosh’s userland architecture provides a secure, user-friendly approach to remote terminal access. Combined with Tailscale, it creates a robust, secure remote access solution that’s both resilient and easy to manage. The userspace implementation means simpler security models, natural process isolation, and straightforward session management.

Remember:

  • Regular session audits
  • Keep software updated
  • Monitor user resources
  • Review security settings
  • Leverage Tailscale ACLs

This setup provides a secure, reliable remote terminal solution that respects Unix process and security models while providing modern features like roaming and intermittent connectivity support.