linguist README

The project's README from github.com/github-linguist/linguist. As one markdown file: README.md.

Linguist

Actions Status

Open in GitHub Codespaces

This library is used on GitHub.com to detect blob languages, ignore binary or vendored files, suppress generated files in diffs, and generate language breakdown graphs.

Documentation

Installation

Install the gem:

gem install github-linguist

Dependencies

Linguist is a Ruby library so you will need a recent version of Ruby installed. There are known problems with the macOS/Xcode supplied version of Ruby that causes problems installing some of the dependencies. Accordingly, we highly recommend you install a version of Ruby using Homebrew, rbenv, rvm, ruby-build, asdf or other packaging system, before attempting to install Linguist and the dependencies.

Linguist uses charlock_holmes for character encoding and rugged for libgit2 bindings for Ruby. These components have their own dependencies.

  1. charlock_holmes
  2. rugged

You may need to install missing dependencies before you can install Linguist. For example, on macOS with Homebrew:

brew install cmake pkg-config icu4c

On Ubuntu:

sudo apt-get install build-essential cmake pkg-config libicu-dev zlib1g-dev libcurl4-openssl-dev libssl-dev ruby-dev

Usage

Application usage

Linguist can be used in your application as follows:

require 'rugged'
require 'linguist'

repo = Rugged::Repository.new('.')
project = Linguist::Repository.new(repo, repo.head.target_id)
project.language       #=> "Ruby"
project.languages      #=> { "Ruby" => 119387 }

Command line usage

The github-linguist executable operates in two distinct modes:

  1. Git Repository mode - Analyzes an entire Git repository (when given a directory path or no path)
  2. Single file mode - Analyzes a specific file (when given a file path)

Git Repository

A repository's languages stats can be assessed from the command line using the github-linguist executable. Without any options, github-linguist will output the language breakdown by percentage and file size.

cd /path-to-repository
github-linguist

You can try running github-linguist on the root directory in this repository itself:

$ github-linguist
66.84%  264519     Ruby
24.68%  97685      C
6.57%   25999      Go
1.29%   5098       Lex
0.32%   1257       Shell
0.31%   1212       Dockerfile

Additional options

--rev REV

The --rev REV flag will change the git revision being analyzed to any gitrevisions(1) compatible revision you specify.

This is useful to analyze the makeup of a repo as of a certain tag, or in a certain branch.

For example, here is the popular Jekyll open source project.

$ github-linguist jekyll

70.64%  709959     Ruby
23.04%  231555     Gherkin
3.80%   38178      JavaScript
1.19%   11943      HTML
0.79%   7900       Shell
0.23%   2279       Dockerfile
0.13%   1344       Earthly
0.10%   1019       CSS
0.06%   606        SCSS
0.02%   234        CoffeeScript
0.01%   90         Hack

And here is Jekyll's published website, from the gh-pages branch inside their repository.

$ github-linguist jekyll --rev origin/gh-pages
100.00% 2568354    HTML
--breakdown

The --breakdown or -b flag will additionally show the breakdown of files by language.

You can try running github-linguist on the root directory in this repository itself:

$ github-linguist --breakdown
66.84%  264519     Ruby
24.68%  97685      C
6.57%   25999      Go
1.29%   5098       Lex
0.32%   1257       Shell
0.31%   1212       Dockerfile

Ruby:
Gemfile
Rakefile
bin/git-linguist
bin/github-linguist
ext/linguist/extconf.rb
github-linguist.gemspec
lib/linguist.rb
…
--strategies

The --strategies or -s flag will show the language detection strategy used for each file. This is useful for understanding how Linguist determined the language of specific files. Note that unless the --json flag is specified, this flag will set the --breakdown flag implicitly.

You can try running github-linguist on the root directory in this repository itself with the strategies flag:

$ github-linguist --breakdown --strategies
66.84%  264519     Ruby
24.68%  97685      C
6.57%   25999      Go
1.29%   5098       Lex
0.32%   1257       Shell
0.31%   1212       Dockerfile

Ruby:
  Gemfile [Filename]
  Rakefile [Filename]
  bin/git-linguist [Extension]
  bin/github-linguist [Extension]
  lib/linguist.rb [Extension]
  …

If a file's language is affected by .gitattributes, the strategy will show the original detection method along with a note indicating whether the gitattributes setting changed the result or confirmed it.

For instance, if you had the following .gitattributes overrides in your repo:


*.ts linguist-language=JavaScript
*.js linguist-language=JavaScript

the output of Linguist would be something like this:

100.00% 217        JavaScript

JavaScript:
  demo.ts [Heuristics (overridden by .gitattributes)]
  demo.js [Extension (confirmed by .gitattributes)]
--json

The --json or -j flag output the data into JSON format.

$ github-linguist --json
{"Dockerfile":{"size":1212,"percentage":"0.31"},"Ruby":{"size":264519,"percentage":"66.84"},"C":{"size":97685,"percentage":"24.68"},"Lex":{"size":5098,"percentage":"1.29"},"Shell":{"size":1257,"percentage":"0.32"},"Go":{"size":25999,"percentage":"6.57"}}

This option can be used in conjunction with --breakdown to get a full list of files along with the size and percentage data.

$ github-linguist --breakdown --json
{"Dockerfile":{"size":1212,"percentage":"0.31","files":["Dockerfile","tools/grammars/Dockerfile"]},"Ruby":{"size":264519,"percentage":"66.84","files":["Gemfile","Rakefile","bin/git-linguist","bin/github-linguist","ext/linguist/extconf.rb","github-linguist.gemspec","lib/linguist.rb",...]}}

NB. The --strategies flag has no effect, when the --json flag is present.

Single file

Alternatively you can find stats for a single file using the github-linguist executable.

You can try running github-linguist on files in this repository itself:

$ github-linguist grammars.yml
grammars.yml: 884 lines (884 sloc)
  type:      Text
  mime type: text/x-yaml
  language:  YAML

Additional options

--breakdown

This flag has no effect in Single file mode.

--strategies

When using the --strategies or -s flag with a single file, you can see which detection method was used:

$ github-linguist --strategies lib/linguist.rb 
lib/linguist.rb: 105 lines (96 sloc)
  type:      Text
  mime type: application/x-ruby
  language:  Ruby
  strategy:  Extension

If a file's language is affected by .gitattributes, the strategy will show whether the gitattributes setting changed the result or confirmed it:

In this fictitious example, it says "confirmed by .gitattributes" since the detection process (using the Filename strategy) would have given the same output as the override:

.devcontainer/devcontainer.json: 27 lines (27 sloc)
  type:      Text
  mime type: application/json
  language:  JSON with Comments
  strategy:  Filename (confirmed by .gitattributes)

In this other fictitious example, it says "overridden by .gitattributes" since the gitattributes setting changes the detected language to something different:

test.rb: 13 lines (11 sloc)
  type:      Text
  mime type: application/x-ruby
  language:  Java
  strategy:  Extension (overridden by .gitattributes)

Here, the .rb file would normally be detected as Ruby by the Extension strategy, but .gitattributes overrides it to be detected as Java instead.

--json

Using the --json flag will give you the output for a single file in JSON format:

$ github-linguist --strategies --json  lib/linguist.rb
{"lib/linguist.rb":{"lines":105,"sloc":96,"type":"Text","mime_type":"application/x-ruby","language":"Ruby","large":false,"generated":false,"vendored":false}}

NB. The --strategies has no effect, when the --json flag is present.

Docker

If you have Docker installed you can either build or use our pre-built images and run Linguist within a container:

$ docker run --rm -v $(pwd):$(pwd):Z -w $(pwd) -t ghcr.io/github-linguist/linguist:latest
66.84%  264519     Ruby
24.68%  97685      C
6.57%   25999      Go
1.29%   5098       Lex
0.32%   1257       Shell
0.31%   1212       Dockerfile
Building the image
$ docker build -t linguist .
$ docker run --rm -v $(pwd):$(pwd):Z -w $(pwd) -t linguist
66.84%  264519     Ruby
24.68%  97685      C
6.57%   25999      Go
1.29%   5098       Lex
0.32%   1257       Shell
0.31%   1212       Dockerfile
$ docker run --rm -v $(pwd):$(pwd) -w $(pwd) -t linguist github-linguist --breakdown
66.84%  264519     Ruby
24.68%  97685      C
6.57%   25999      Go
1.29%   5098       Lex
0.32%   1257       Shell
0.31%   1212       Dockerfile

Ruby:
Gemfile
Rakefile
bin/git-linguist
bin/github-linguist
ext/linguist/extconf.rb
github-linguist.gemspec
lib/linguist.rb
…

Contributing

Please check out our contributing guidelines.

License

The language grammars included in this gem are covered by their repositories' respective licenses. vendor/README.md lists the repository for each grammar.

All other files are covered by the MIT license, see LICENSE.

linguist (github-linguist/linguist)

Searching Database…

13.7K ·GitHub
Linguist
What is this?Linguist is the library GitHub.com uses to detect the programming language of files (blobs), ignore binary or vendored files, suppress generated files in diffs, and build the language breakdown graphs shown on repository pages.
Is this a library or a command-line tool?Both: it's a Ruby gem you can require in your own app, and it also ships the 'github-linguist' executable for analyzing a whole git repository or a single file from the terminal.
How do I get it running?Install the gem, then run github-linguist with no arguments to get the language breakdown by percentage and size for the current git repo, or pass a file path to analyze just that file.
What native dependencies do I need before installing?You need a recent Ruby plus native libraries: charlockholmes requires cmake, pkg-config, ICU and zlib, while rugged requires libcurl and OpenSSL.
Any install pitfalls I should know about?The Ruby bundled with macOS/Xcode is known to break installation of these native dependencies, so the project recommends installing Ruby through Homebrew, rbenv, rvm, ruby-build or asdf first — or just use the provided Docker images instead.
What's the difference between --breakdown and --strategies?--breakdown lists every file with its detected language and size; --strategies additionally shows which detection method (filename, extension, heuristics, etc.) was used, including whether .gitattributes confirmed or overrode it.
Can I use Linguist without installing the Ruby native dependencies myself?Yes, Linguist provides Docker images you can build or pull, letting you run the analysis in a container instead of installing charlockholmes and rugged locally.
Can I override how a specific file gets classified?Yes, you can set overrides in .gitattributes, and running with --strategies will tell you whether your override matched (confirmed) or changed (overrode) the automatic detection.
Can I check language stats for an old tag or branch instead of the current checkout?Yes, the --rev flag accepts any gitrevisions(1)-compatible revision, so you can see the language makeup as of a specific tag or branch.
What are similar projects?bat, hub, difftastic, perch, alfred-workflows, vimrc, diff-so-fancy, tmuxinator
Which topics is it in?graphs, code-analysis, text-processing, syntax-highlighting, ruby, static-analysis, homebrew, github, git, command-line, terminal, library
Creator
github-linguist
Size
41.4 MB
Stars
13.7K
Updated
CommitOct 14Releasev9.7.0Aug 26
Age
15y
Commits · 90 Days