How do I speed up processing many files using all CPU cores?

Quick Linux Tip #7:

Try: find /images -name '*.jpg' | parallel -j+0 'convert {} -resize 800x600 {}.resized'

Info: GNU parallel executes commands in parallel across all CPU cores. Dramatically speeds up batch processing of files, downloads, or any repeatable task.

Examples:

  • $ parallel -j 4 gzip ::: *.log
  • $ cat urls.txt | parallel -j 10 wget {}
  • $ find . -name '*.mp4' | parallel ffmpeg -i {} -vcodec h264 {.}.mp4

Note: Install with apt install parallel. -j+0 uses all cores, -j N uses N processes, and ::: passes arguments directly. Perfect for CPU-intensive tasks.




LinuxTeck.com
linuxteck@ubuntu:~$ find /images -name '*.jpg' | parallel -j+0 'convert {} -resize 800x600 {}.resized'

Processing 5000 files across 8 cores...
Completed in 2m 15s (would take 18m 30s serially)
Speedup: 6.8x

linuxteck@ubuntu:~$

PREVIOUS ARTICLE Quick Linux Tip #06: What files is a specific process using? NEXT ARTICLE Quick Linux Tip #08: How do I who modified a critical file and when?
About John Britto

John Britto Founder & Chief-Editor @LinuxTeck. A Computer Geek and Linux Intellectual having more than 20+ years of experience in Linux and Open Source technologies.

View all posts by John Britto →

Leave a Reply

Your email address will not be published.