@fastfinge possibly use a ram disk as the local end? haven't tried it though
Samuel Proulx @fastfinge@interfree.ca@ireneista Isn't the issue caused by how slow the remote end is, though? Because Rclone is pretending to be a filesystem, so the kernel slows IO down because it's constantly waiting for data from rclone. So writing to disc isn't the actual bottleneck.
@fastfinge @ireneista I can't put my finger on why, but this doesn't sound quite right. If you're syncing data down from a remote with the sync command, that's just a bulk download as many apps carry out. Why would your wider system delay I/O because of an expectation of some files being written to? And I didn't follow the "pretending to be a filesystem" bit.
Samuel Proulx @fastfinge@interfree.ca@jscholes @ireneista So my understanding is that rclone mounts the cloud using fuse, then treats it as a virtual filesystem. So when sync is touching files to get time/date/size etc, you get high iowait, slowing the entire system.
@fastfinge That's what `rclone mount` does, not `rclone sync`. If you're using the latter, you can tweak it quite a bit, including telling it to sequence checks, transfers then deletions, instead of mixing them all together. If you're using the former to sync your files, I wouldn't. @ireneista
Samuel Proulx @fastfinge@interfree.ca@jscholes @ireneista I'm using rclone sync to move a little over 900 gigs, of files between 20 kb and 5 mb in size. It's nowhere near maxing out CPU, bandwidth, or memory. But it brings the server to its knees anyway. Anything other than rclone that wants to do any IO takes multiple seconds.
@fastfinge Is it a VPS?
Samuel Proulx @fastfinge@interfree.ca@jscholes Nope! Debian 12 directly installed on a server right here beside me. It's on 3 gig symmetrical fiber, with 64 gigs of ram, and a quad-core AMD CPU.
@fastfinge Interesting. What's the Rclone command line invocation you're using?
Samuel Proulx @fastfinge@interfree.ca@jscholes rclone sync --interactive icloud /Family/backups:/mnt/backups
Samuel Proulx @fastfinge@interfree.ca@jscholes The destination disc is a solid state NVME drive.
@fastfinge You mentioned it's a quad core CPU, but do you know how many threads it has?
Samuel Proulx @fastfinge@interfree.ca@jscholes I do not. At least, not off the top of my head.
Samuel Proulx @fastfinge@interfree.ca@jscholes However, I do know that stuff that doesn't need to read or write to the filesystem is just fine. Like, there's no lag typing into SSH. But if I try to write to a file, then there is.
@fastfinge @jscholes What happens if you tell it to use idle I/O priority? ionice -c 3 rclone sync
@fastfinge Is this an initial sync, or an update to a previous one? And if the latter, do you happen to remember if the slowdowns happened when initially downloading the files to an empty local destination?
Samuel Proulx @fastfinge@interfree.ca@jscholes This is initial. Destination started off completely empty.
Samuel Proulx @fastfinge@interfree.ca@jscholes Also, apparently I was wrong about the CPU. according to lscpu:
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 1