Git 2.20.1 is super slow on macOS Mojave on my work Mac

Update: I just upgraded my personal mac to 2.20.1, and am experiencing none of the slowdown I had on my work Mac. So something else is afoot. Maybe some of the 'spyware-ish' software that's installed on the work mark is making calls like lstat() super slow? Looks like I might be profiling some things on that machine anyways :)

I regularly use Homebrew to switch to more recent versions of CLI utilities and other packages I use in my day-to-day software and infrastructure development. In the past, it was necessary to use Homebrew to get a much newer version of Git than was available at the time on macOS. But as Apple's evolved macOS, they've done a pretty good job of keeping the system versions relatively up-to-date, and unless you need bleeding edge features, the version of Git that's installed on macOS Mojave (2.17.x) is probably adequate for now.

But back to Homebrew—recently I ran brew upgrade to upgrade a bunch of packages, and it happened to upgrade Git to 2.20.1.

Later in the day when I was doing some heavy Git activity, I noticed everything felt... so... sluggish. And it was even sluggish on tiny git repos with less than 100 files, so either something was seriously wrong with my filesystem—which should've shown problems elsewhere—or Git was being funny.

So I benchmarked it:

$ time git status
On branch master
nothing to commit, working tree clean

real 0m1.037s
user 0m0.004s
sys 0m0.009s

That's... suspicious. A full second just to check the working directory? And I ran git gc and git status again, no difference. I ran it a bunch of times and it was always ± 0.1% of that timing.

I went ahead and unlinked the Brew-installed Git:

brew unlink git

Then opened a new Terminal window, and ran the same test:

$ time git status
On branch master
nothing to commit, working tree clean

real 0m0.018s
user 0m0.005s
sys 0m0.010s

Ahh... much better. Special thanks to Steven Wichers for putting the idea in my head that the Git version could be the problem—I remember a couple weeks ago in a tech chat at Acquia he mentioned something similar. If it weren't for that, I would've spent a few hours trying to profile and debug git traces, most likely! Seconds count, when you run like 5,000 git commands a day!

Further reading

git
macos
performance
homebrew
cli
terminal

Comments

Thanks for this post; was running in to similar performance issues on an 2011 iMac on High Sierra after Homebrew upgraded git. 10+ minute brew installs, 5 minute git status results... ugh.

Unlinked brew's git and everything was back to normal.

I also experienced this and used `sample git -wait -mayDie -file ~/Desktop/git_status.sample.txt` to see what was going on. When comparing to colleagues who weren't having this problem on a similar system setup, the difference was an order of magnitude more lstat calls. Still not sure why it's happening.

Interesting, I experienced the opposite: using the version of Git that was installed from installing XCode's command line tools, git was painfully slow. Specifically this was in evaluating the __git_ps1 bash function that people often use for having the git branch show up in their bash prompt. Doing a quick time of that function it was taking about 1.7 seconds to evaluate.

Brew installed the latest version and much faster, down to 0.06 seconds to evaluate.

I received a 20% or so speedup by removing brew's updated git and using the system default. That said, I'm running my terminal through the x86 emulation mode and still get full real times in the ~.06s range, 10-20x slower than my old 'nix machine with RHEL 8.3 and with a slightly older version of Git than the mac (2.30.1 [mac] vs 2.27.0 [rhel]).

It appears that lstat requires a global lock per thread within APFS and no matter what one does, one will be held back by at least a factor of ten for large IO operations versus linux.

I've filed a bug with Apple, however, this is probably deep and internal with APFS and I have no expectation of it being fixed or worked on by Apple. I'm not sure that even makes sense for their business, as I'd guess they chose this tradeoff to better server their primary customer--the consumer. And I am assuming consumer usage doesn't typically call the slow-er filesystem APIs like a single git merge (with many changed files) or git status would.

I guess this is my big fat reward for working on a consumer-focused closed source OS ;-)