Troubleshooting
Fix InfluxDB database is locked on a network share
Short answer
The mounted share is the first suspect: InfluxDB 2.5.1 opens its bolt file on the share, then fails SQL migrations with database is locked because SQLite file locking is not dependable on NFS or SMB. Stop every process that can reach the paths, confirm the active bolt-path and engine-path, prove the same configuration on local storage, then keep the engine local and use the share only for backups.
On this page
The failure signature points at the mount, not at your data or your permissions. InfluxDB v2.5.1 starts, opens /mnt/influxdb/influxd.bolt, grows it to 64 KB, creates a 0-byte influxd.sqlite, then stops in Failed to apply SQL migrations with database is locked. The share accepted the opens and the bolt writes but could not give SQLite the lock and write ordering its transactions need, and NFS and SMB mounts routinely cannot. Stop every process that can reach those paths, confirm which bolt-path and engine-path the running process actually uses, prove the same configuration on local storage, then leave the engine on local disk and use the share for backups.
What the startup log pins down #
Read one complete controlled start before changing anything. journalctl for the service, or the console if you launched the binary by hand, gives you the phase:
Resources openedwith"service": "bolt"and your path: the process found the mount and opened the bolt file. Reachability and permissions to that point are good.- The following
Resources openedforinfluxd.sqlite: the metadata database was created and opened at 0 bytes. Bringing up metadata migrations, thenBringing up SQL migrations: startup moved past file opens into migration work.Failed to apply SQL migrationswithdatabase is locked: the immediate fault is a SQLite lock, not a missing file and not a permission denial.
A full bolt file beside an empty sqlite file is the state that failed run leaves behind. Record it, capture the first error and the exit status, and do not restart into the same directory again and again hoping it clears.
Why the share is the first suspect #
SQLite serializes every writer with file locks and a journal: take the exclusive lock, write the journal, commit, release. On a local filesystem the kernel guarantees those steps happen in order and that every other process sees them. On a network mount, lock handling and cache visibility depend on the protocol version and the mount options, and databases on such mounts routinely report locks that never clear or writes that land out of order.
InfluxDB 2.x has kept an SQLite metadata file beside the bolt since 2.1.0, so every start runs SQLite transactions during migration, exactly the phase your start dies in. The bolt writes got through; the first SQLite transaction did not.
Stop everything that can reach the paths #
Before touching any database file, work this list:
- Stop the packaged service, then look for a manually started
influxdand end it too. A service unit and your own console start against one directory is the classic way to producedatabase is lockedeven on local disk. - Confirm the mount exists before the service starts. A unit that starts before
/mnt/influxdbis mounted writes into the VM's local mount-point directory instead, and the logged paths will tell you. - Check traversal and read/write access as the service's operating-system user.
influxdbownership on the share is what got the files opened; it says nothing about locking. - Record which executable ran, how it was launched, and which configuration it read. Your package setup separated metadata at
/mnt/influxdb/influxdb/meta, TSM data at/mnt/influxdb/influxdb/dataand WAL at/mnt/influxdb/influxdb/wal; the binary setup instead uses--bolt-pathand--engine-path. Those are two different layouts, so verify which one is live before drawing conclusions. - Do not delete
influxd.sqliteor its journal file to clear the lock: the journal can be part of SQLite's recovery state, and removing it can corrupt the metadata database.
Run the controlled local-storage comparison #
- With everything stopped, snapshot or copy the share's contents and leave the original untouched.
- Create a new empty directory on the VM's own disk and give the service user access to it.
- Start exactly one process with the same two flags pointed at that local directory. Drop the
string=from your first attempt, which read--bolt-path string=/mnt/influxdb/influxd.bolt; the flag takes the path directly, as your later invocationinfluxd --bolt-path /mnt/influxdb/influxd.bolt --engine-path /mnt/influxdb/enginealready did. - Watch the log pass
Bringing up SQL migrationsand stay up, and confirm the local files appear and grow. - Stop it cleanly and start it once more; the same paths should open with no lock error.
If it passes locally, the mounted filesystem or its mount configuration is the difference, and no amount of permission work on the share will fix a migration that needs a real lock. Lock-related mount options exist on some NFS implementations, but they are not a supported way to run a live database directory. If it fails locally with the same error, the share is innocent: hunt the second process, or the config file the running process is not the one you edited.
Working the v2.5.1 start: 64 KB bolt, 0 KB sqlite, 50 ms opens #
Inputs, each with its source:
- InfluxDB v2.5.1, commit
5b6fdbf05d, from the startup banner in your console capture. - Ubuntu 22.04 LTS VM, share mounted at
/mnt/influxdb, ownershipinfluxdb, from your setup notes. influxd.boltat 64 KB after the failed start andinfluxd.sqliteat 0 KB, from your file listing.- Banner logged at
2023-03-20T13:19:23.724534Z, boltResources openedat2023-03-20T13:19:23.774769Z, the sqliteResources openedfractionally later, from the same capture. - SQLite metadata file present in the 2.x line since 2.1.0, so every start performs this migration work.
Rule in words: a 2.x start opens the bolt, opens the sqlite, then applies SQL migrations. If the opens complete in tens of milliseconds and every byte written before the failure went to the bolt, then the mount's read, write and permission paths all work, and the failure is isolated to lock acquisition on the first SQLite transaction.
Arithmetic:
- Banner to bolt open: 774,769 µs − 724,534 µs = 50,235 µs ≈ 50 ms.
- Bolt open to sqlite open: 774,900 µs − 774,769 µs = 131 µs.
- Bytes written before the failure: 64 KB × 1,024 = 65,536 bytes to the bolt, plus 0 bytes to the sqlite, all 65,536 of them delivered through the share.
Result: the mount delivered sub-second opens and 65,536 bytes of writes, then failed the first operation that requires an exclusive lock held across a commit.
Decision: when the opens land in tens of milliseconds, the bolt sits at its full initial size and the sqlite file is empty, stop adjusting ownership and credentials and move the data directory to local disk. That signature is filesystem semantics, not access control.
Keep the engine on local disk #
The permanent arrangement is bolt-path, the influxd.sqlite file beside it, and engine-path on the box's own disk, with the share demoted to a backup target that receives a stopped-directory copy or exported archives on a schedule. Your reason for the share was local capacity, so size the local disk from the write rate and retention, the way sizing a self-hosted Node-RED InfluxDB Grafana historian works it through; at high write rates the disk's endurance matters as much as capacity, the trade-offs laid out in choosing edge storage for high-rate vibration data. To move what exists: stop everything, copy the whole directory while the service is down, start once against the copy, and keep the share copy until the local instance has restarted cleanly.
Verify before the writers come back #
- The service survives a clean stop and start and opens the same bolt, sqlite and engine paths.
- Buckets, users and API tokens are present after the move, because they live in the copied files.
- A test point written through the API comes back from a query.
- No lock error repeats across at least two restart cycles.
- Then repoint the three systems that were sending data and confirm arrival; if points stop appearing downstream, work the checks in troubleshooting missing data in a Node-RED InfluxDB Grafana stack.
Questions asked
Why did InfluxDB run fine on the share for a day and then lock?
The first start can finish its migrations before anything else touches the files, so a share with weak locking looks healthy until the next start. Overnight, a service restart, a second influxd process, or the mount coming up late puts recovery and migration back on the failing path, and the lock error returns. The share did not change; the startup sequence did.
Can I fix it by deleting the empty influxd.sqlite file?
No. The file regenerates empty and fails at the same first transaction, so deleting it buys one identical failure. Worse, buckets, users and API tokens live in the bolt and sqlite files, so deleting them on a directory that ever held data destroys that metadata, and removing a journal file can break recovery state.
Is an SMB share safer than NFS for the InfluxDB directory?
Not as a fix. SQLite needs the same two things on any network filesystem: a lock that holds across a commit and writes that land in order. Choose the share for its backup and retention qualities and keep the live data directory on local disk; the engine and the metadata files move together.
Will moving to local disk lose the buckets and tokens I already created?
No. Stop the service, copy the whole directory with the service down, bolt, sqlite and engine together, and start once against the copy. Buckets, users and API tokens live in those files and travel with them. Keep the share copy untouched until the local instance has restarted cleanly and answered a query.
What if the local-storage test fails with the same error?
Then the share is not the cause. Look for a second influxd still running, a service unit and a manual start both pointing at the same paths, or a process reading a different configuration than the file you edited. The comparison is only valid with one process, one verified configuration and one controlled start.